I have found being able to sight read relatively easily to unlock a vast trove of music I'd like to be able to play which would have been harder to pick up purely by ear. It's definitely worth learning the things you're talking about. I found the surest way to get good at that kind of playing is to play with other people. The time dependence of having to keep up with everyone makes the feedback really tight.

Playing with other people also highlights other perhaps unexpected skills. I played in a band for a while and I still retain the skill of reading chords off other player's hands. You also need to be able to respond to someone just shouting "OK, let's go to C minor" in ways that only matter in that context. When you're listening or sight reading, you don't need names.