Yes, I live in a noisy environment -- do you live in a monestary? Or a ...wherever it is that nuns are? Or an abandoned velvet-lined stone fortress? Because short of that, headphones are almost always quieter than a speaker environment. And even when it is, you had to have gone to a lot of effort to achieve that, which adds a psychological, "waiting for the other shoe to drop" pressure. To me, anyway.
I listen on relatively low volumes with both.
And yes, I totally agree that there's the whole isolation vs. background noise thing, but to me, it's more than that. I have heard excellent speakers on a well-controlled room, and they still seemed more diffuse than headphones. I mean, when you're straining yourself to understand lyrics, don't you subconsciouly do the ear equivalent of squinting your eyes, and try to lean up closer to the speaker? You're already doing that with headphones.
I'm not saying that speakers aren't more enjoyable for you (and Jacob and most of the rest of the population), but you touched a nerve when you said "critical" listening.