My experience with the duet says the best setting is to go for about 1/2 volume on line out (not instrument out). Use your ears and find the level that works best for the input on your amp.
For the mac, go to audio midi settings, set it to 24/96, and just leave it there. Core audio upsampling is pretty darn good. Always launch itunes AFTER you change those settings. or you'll get the resampling done inside itunes, which is terrible.