Well, that's one way to look at it. The other way to look at it is that hearing something in context is a lot different than hearing it at all. Dither, for example, sounds completely different than, for example, road noise, or thermal noise, especially in the context of a digital recording, but at the lowest levels alone, they all sound the same to an untrained ear. If you overdrive your ADC at the recording point at very low levels, it might sound tolerable, but put it on top of a recording and it will sound awful.
There's also the nature of the distortion -- that whole even-order harmonics vs. odd-order harmonics thing.