Because it doesn't add anything to the sound. Specifically, it doesn't add anything good to the sound. Involving music will still be involving; boring music will come through as boring.
Then why did you say neutral is boring?
Think about it like a party. The hostess is boring, the furniture is boring, the music is boring, the plants are boring, the view is boring, the drink is boring.
But you found one chick there that was interesting -- makes it all worthwhile, dunnit? Doesn't change the fact that everything else was boring.