At the end of the day, the "subordinates" are the ones actually doing the far majority of the work, and they far outnumber the managers. I don't see why the workers shouldn't at least have a seat at the table when the decisions are being debated that would affect them. Often the managers are so removed from the work that they lack a lot of the insights from the front lines.
The most common rebuttal I hear when I bring this up is, like you said, "too many people makes things inefficient". Bullshit. Sure a decision ultimately needs to be made, but there's no reason one shouldn't be allowed to provide input.
And in my experience, I do think it's more about people wanting to feel powerful, and fear of losing this power (eg. like the other commenter who mentioned fear of "power inversion"). The people in power are generally the most sycophantic, and thus have the least interest in interacting with or relinquishing power to subordinates because it doesn't personally benefit their position of power or ladder-climbing question.
If a bunch of executives make a bad decision, then "oh well, there's no way we could've known". If a bunch of executives make a bad decision after a meeting with subordinates who questioned their decision, then it makes the executives look bad and threatens their judgement and position of authority. So from a selfish perspective, it's better to keep the minions out.
