Then there are the people who program best in a fugue state and the idea of having to constantly break that to transform their thoughts into words and human interaction is anathema.
I say this as someone who just woke up in the wee hours of the morning when nobody else is around so I can get some work done (:
The biggest downside to me was that it forces a level of engagement exceeding the most heads down solo work I’ve done. I’d come home and feel mentally exhausted in a way I didn’t usually.