This may be viewed as a benefit rather than a downside by a surprisingly large number of people.

I'm not passing judgment here, just speaking empirically. For men especially, the office and work are the default place to hide from unhappy marriages, dull domestic life, housework, and [perceived as] perfunctory obligations that they want to avoid (due to lack of interest) but which it is not socially acceptable, or perhaps politically correct, to reject. Not everyone is in a happy marriage, and not everyone actually enjoys spending time with their wife. Furthermore, not all fathers have the right combination of personality, emotional constitution and intellectual disposition to have fruitful interactions with all kids, at all ages, and all times. "I love spending time with my kids more than anything else in the world!" is an obligatory refrain nowadays, so few people will admit otherwise—even to themselves—because it's very non-comme il faut. But the truth is that 2 1/2 year-olds can be pretty unstimulating company. For a lot of busy, career-minded, highly accomplished, worldly people, ten minutes before bedtime sounds about right. That sounds cruel and terrible now to a lot of people now, but would have been quite uncontroversial half a century ago.

That has been one of the most important functions of the organisation office in the 20th century. It's a cover that gets many men—often fairly inscrutably—out of having to deal with things they don't really want to deal with. Feminism picked up on this a long time ago, and that's what's behind a lot of the "second shift" double standard critique. The charge is that men still have an easier time cloaking themselves in inscrutability when they attend to their careers. In this version of events, nobody really has a right to question why they're at the office so much, what they do there, or why they're working so hard.

I'm not talking about men that lie that they are "at the office" when they're in fact at the hotel with their mistress. I'm talking about men that are legitimately ambitious and interested enough in their work that they take on more of it—a lot more of it—than they absolutely have to willingly, because they are secretly not so interested in achieving "work-life balance", as other aspects of their life are not fulfilling to them. There's a lot you can get out of if you have to stay late at the office all the time due to meet a legitimately ambitious shipping schedule. Hard to argue with.

Another, related problem is that most non-digiratti don't really understand computer-based self employment. I'm sure we have all dealt with the problem of older relatives, more traditionally-minded people, or unenlightened spouses perceiving that we are simply sitting on the couch "playing on the computer", much as many of us have always done since childhood. In the mind of such people—a vast segment of American society—work is a place one goes, with a briefcase, and which keeps defined business hours. Intellectually, they understand that the money comes from somewhere, and that you are not actually paid to "play on the computer". But instinctively, they can't process that. The appearance of "playing on the computer" means you don't really do much of anything and are always available to chat, run errands, walk the dog, feed the baby, etc.

Some will say, "But that's why I have a house, with a door that closes, and everyone around me knows not to ever bother me when I'm in there!" Yeah, okay. My experience hasn't been that this works. The first time your wife walks in there anyway and sees you browsing Reddit instead of feverishly parrying phone calls and struggling to right critical infrastructure to prevent impending apocalypse, that's done. From that point onward, you're just hiding in there to play on your computer and are preemptible by processes with nominally high priority. :-)

That's another class of problem that going to an office solves.