The only downside to dual primaries is cost.
Birgir, how are you implementing the latching switches? I thought that most of these weren't capable of switching the voltage and/or current levels that we need them to. Or am I just over thinking this?
[edit]I just looked at the specs of one of the 22mm Bulgin latching switches, clearly I'm wrong since they're rated for 250VAC/5A which should be sufficient on the primary side of most headphone amps.