This.
The way I understand it - and I understand nothing - is that your brain is used to compensating for the shape of your head (your HRTFs) but is designed to compensate for a single-point source of sound somewhere well outside of your head. Speakers aren't a single-point source but for HRTF purposes they aren't that different, so when listening to speakers, people hear similar things. Headphones are very far from a single point source, the sound originates from drivers just outside your ears and enclosed by earcups besides, so a lot of your head isn't a factor, and the same compensation curve now leads you astray into hearing a different sound, from person to person. And with IEMs, where the sound basically originates inside the ear canal, the differences are more pronounced.
Now what I wonder is if the brain adjusts to the fact that the sound originates not from a single point source and comes up with some new means of compensation.
I generally agree with others on how speakers sound like. When it comes to headphones, stuff like the HD650 and SR-007 actually sounds closer to a speaker system that's tuned to be flat 20-20, for me, than a lot of other things people call neutral. When it comes to IEMs, I have no clue what others are smoking, but if my crackpot theory has merit it's to be expected. People with whom I've listened to and compared IEMs have heard rather different things with regards to FR, but similar things with regards to detail, speed, dynamics, etc.
From time spent on forums I've learned more or less how to compare myself to others, i.e. compared to Tyll, I gravitate towards a sound that is slightly, but not significantly warmer. Given those differences, I can kind-of extrapolate how things sound based on what I read but it's never easy or reliable, and even less so with IEMs.
Of course, acclimatization is a powerful force, too. Listening to the L700 after the 007, it sounds bright. Listening to the 007 after the L700, it sounds dark. And over time, both start to sound more or less normal, though the actual relative shapes of their FR - like the L700's midbass hump and small treble spikes - are always audible.