Well it sounds like there may be a problem with your amp, your power or the sensitivity of your hearing. Ever notice any annoying clicks when you are out and about in the summertime at night?
Since this is the second time in recent history you have mentioned fairly audible background hiss (previously the Shanling) methinks you should search out some other component(s) in the chain or on the circuit.
I'm using the previous generation Mac Mini with the slower processor, 1Gb Ram and the 80Gb HD. It is more than fast enough and I have noticed absolutely nothing that would indicate any need to upgrade anything except for perhaps the shitty monitor I'm using because I was trying to do this as cheaply as possible.
Now it makes make sense as to why I got my Mac Mini so cheap. Nice upgrades but nothing to make me lust for one since I use it as a music server and to have a Mac available at work.