This is a big motivation for me as well. I live in Salt Lake City and all winter long I can ski at dawn before every day of work, and all summer long I can climb 11k' mountains 20 minutes from my house - benefits I'd be hard-pressed to recreate anywhere else. Moving to NY, SF, etc would be miserable for me as well (I've lived in NY). There are plenty of reasons remote workers prefer to be remote, not just the obvious ones! (Family, grew up there, etc).
The rest of my team is scattered across Europe, so the time difference can make an impact, but once everyone trusts one another I don't notice it making a big difference.