To get bit perfect output on the mac:
In the application "Audio MIDI Settings" which is in the Applications->Utilities directory, plug in your optical cable, set the output to digital output, and set the bit rate of the digital output to the rate of the files you're going to be playing. Then launch your playback application. That's all there is to it.
You also have the option of letting itunes do the upsampling, which it does a pretty good job at doing. In that case, just set the sample rate in Audio MIDI Settings to the rate of your choice (probably 96/24). Then launch the playback application.
If you launch the application before changing the settings, you get much worse results.