My recollections of Linux audio issues are more from the days when one thing might want to use OSS and another ALSA and you might have to fiddle around and enable software mixing in some ALSA configuration files to get non-exclusive sound to work. And then some issues right around when Pulse first started to be a thing. But really it's been a "Just Works" for me for years and years.
I use Pulse for some pretty complicated stuff, too; I'll pipe individual streams to my speakers, if I want to show something to my SO (e.g., a reddit video) but want to keep, like, my music or my backgrounded game on the headset. I can do that with PA, and honestly pretty easily. AFAIK, no other system supports that, though admittedly I haven't used Windows in some time.
Far more complicated, I also have TTS bot I use in Discord sometimes, when I can't speak. PA allows me to push that into Discord; this is another thing I don't even begin to know how to set up on other systems. (The bot emits sound, so it's like a program playing sound, not a recording device. Discord thus doesn't recognize it as such. So I end up creating a fake input device, of sorts, and wiring the bot's output to that fake output (so it goes to Discord) but also to my own headset, so I can also hear the bot, as the TTS is far from perfect and will sometimes just make hilarious garbage of words.)
I also use Bluetooth, and that usually works. It certainly works more reliably with PA on Linux than it does w/ macOS, though it is my no means flawless. But that seems to be how bluetooth is: it sucks on every OS. There will be days it won't connect to Linux, and there will be days it won't connect to macOS.
I will say that Pipewire looks like it more solidly hits the nail: the thing I don't like is that PA doesn't really model audio as a graph of nodes. (PA's model is equivalent to a graph of nodes, but that it isn't just a graph of nodes makes it a lot harder to deal with and intuit.) For that reason I'm excited about Pipewire; I need to try it one of these days. PA would also be served by a better API; it is difficult to script against.
Also, systemd units are hands-down a more solidly engineered thing than the mess of shell scripts they replaced. I will take systemd+journald any. day. of. the. week. People that … want shell back — I honestly can't comprehend this. I've literally not had to deal for years now with a service wedging itself in because some daemon orphaned itself somehow between some unreliable mess of shell, pidfiles, and other hacks. Is systemd/journald perfect? No, I've spent time fighting it too. But it's a lot more productive than what came before, and the design is engineered to solve the problem at hand… which the shell mess was not.
PA is ok if you use consumer software for consumer tasks. As soon as you go into the production side of things (or even do it live and/or for money) things can fall appart pretty quickly.
The vocal PA critics I know don't even complain about the way pulseaudio does things. They complain about the persistent problems this widely deployed piece of software still has after years in the field. And some of those problems are of conceptual nature.
I don't Pöttering really deserves the heat he gets (no dev would), but I sometimes wish that type of (usually male) programmer would just sometimes stay with the trouble and not ride of into the sunshine when things aren't a beautiful fresh sheet of white paper anymore, but a tangled mess.
(I'll use a bluetooth headphone, if I want audio while pushing something to HDMI. Though I could see using TRS headphones being a nice to have, and I honestly don't know how to configure that, or if it is even possible, since it's all represented as one card in PA.)
