A good PM will discuss options with the team, but they will serve as the interface through which the customer communicates with the team. The customer does not care how things get done, only that they get done on time.
Why does it have to be like that? This sort of silo thinking is probably what created the disconnect in the first place.
The customer might care about how things get done, if it means the difference between getting 85% of their wish list done or 50%, all depending on one obscure feature.
Agile/SCRUM can make the potential complexity of a feature more visible because it's a team estimate. It also can make the decision process less laggy because the customer/product owner sits in on the team. Finally it can make the end goal more visible to the Devs, resulting in less of a disconnect.
My 2p
There is no such thing as "no PM". Such a concept can't exist (for the same reason that "no developers" can't exist). There's actual work there, that someone has to do, or nothing gets built.
Now, it's totally fine if you say "I think it's best if all of our team members spend 75% of their time as developers, and 25% of their time as project managers". That's what you've just described above, and (in my opinion) there's nothing inherently wrong with that.
But your misleading yourself if you claim "PM's aren't needed because Agile", or "PM's aren't needed because the one's I've met 'silo' us off". All you've done is shift the PM burden from a specific person, to some/all of your developers. Your developers are now actually "part-time developers, part-time PM's"
There's also some weird assumptions that your making around PM's. PM's do not have to silo developers away from the client (and good ones don't). PM's do not have to estimate for devs (good ones always use team estimates).
I've seen developers write buggy code, or brittle tests. But I would never say "developers aren't needed". You've clearly encountered bad PM's, and that's very unfortunate. But that doesn't mean "PM isn't needed", and as you yourself just described, you already have to pick up that work on your own since you lack one on your team.