Yep. It's the autonomy part that is the problem in nearly all cases. Choosing your own spot on that spectrum is the key part. And your spot may have to change over time, based on other parts of your life. Being forced to spot X is where all the contention comes in, it seems.
I did miss occasional lunches, dinners, break room chats with folks, but now run a coworking spot in town. I have most of the socialization aspects I missed, without them being tied to work (no office politics with these folks, just... socialization).