This is a good point, and should rule out any devices that Head-Fi considers as all-in-one options like the Benchmark (even though the new DAC does sound great) and Lavry DA11 and Grace. There is a T+A DAC 8 reviewed on Compute Audiophile, but he didn't know how to work it most of the time he had it and the review is not that helpful. The main advantage for your search is the size of the T+A (rimshot).
I think the Weiss DAC2 or a Metric Halo ULN-2 would be great, but would not solve the problem of other media like USB sticks and SD cards. Why can't you just drag them onto your computer and play them from there, which would sound better anyway?