From the PoV of an engineer, the assumptions we have as to why most companies are now choosing these inhuman workspaces are:
- money: it costs more to have separate cubicles or worse, offices with doors. It costs more both in actual hardware and construction costs, but also in square footage: you can pack more people into little 4x4 desk quads by giving them less personal space. Full-size cubicles need more space. This is probably the #1 reason, though managers will never admit it.
- "collaboration": this is the reason typically given by managers. It's true: having low or no walls between employees makes it easy for them to chatter with each other. If their job needs a lot of chatter, this can be a good thing. If their job doesn't, and needs a lot of intense concentration, then it's a bad thing. So stockbrokers for instance would be well-suited to open-office environments. Engineers and programmers, not so much. But managers are "people people" who spend all their time flapping their lips and not doing any real work, so they don't understand this, and think everyone likes to talk all day. Strangely, a lot of them seem to have walled offices with doors, while they tout the benefits of "collaboration" and open offices.
Finally, it doesn't help that a LOT of the younger generation of programmers seem to just love these open-offices. You might think it's a version of Stockholm Syndrome, but I think it's because the personality type attracted to programming jobs has completely changed in the last 15 years. Instead of attracting introverts, now it's attracting the "brogrammer" crowd.
Also, I imagine a lot of these managers would disagree with you about the output; they'll claim that collaboration boosts output, improves quality, etc. Of course, it's all just hand-waving with no evidence. To get any real evidence, you'd have to do some kind of study doing identical or similar projects in both environments. And even then, different people are different: some people might actually be more productive in an open environment. It probably really depends a lot on the personality type of the programmer, and on how they prefer to work. A lot of today's hipster programmers really do like open offices.
So, what we have done at our office is to just have to more or less equally size open spaces: one library-silent, one meeting-room-noisy. It's not perfect, but cheaper than individual offices, while still allowing you to concentrate when you most need to.
