But who cares?
Even if there were, hypothetically, no possible way to make XMPP work, and Google really needed a custom protocol, they could still have released public documentation for that protocol, and open source client and/or server code.
Indeed, that's sort of what they did in 2009 by creating Jingle, an XMPP extension for voice chat, and releasing libjingle. But no such luck for the new stuff.
I could understand giving up federation, maybe, since it creates a spam problem. Though for Google to throw up their hands on spam is a bit ironic, given that Hangouts sits inside Gmail, which has world-class spam filtering. But federation was never really necessary: most XMPP clients support multiple accounts, and it's not like there were a lot of huge XMPP services - only one, Google. You just add that as an account in addition to whatever smaller-scale/organizational XMPP servers you use. (You know, the thing people used to use before Slack.)
Similarly, most XMPP clients support multiple protocols, so for Hangouts to use a custom protocol is not the end of the world. Annoying, but if the benefits are clear enough… it's fine.
But no. Like with most messaging services these days, Hangouts' protocol isn't documented, and nobody has bothered to reverse engineer it. So you're stuck with the official clients. Maybe not the end of the world, since unlike the bad old days of Windows-only clients that left Linux users in the cold, today any desktop OS can use the web client. (And non-top-two mobile OSes don't have enough market share for people to care about their not being supported, and many of them can run Android apps.)
Not the end of the world, unless you want all your messaging services in one place - people used to see a lot of benefit in that, and there are a lot more services now.
Or you don't like the official design. Or you don't like bloated web clients. Or you prefer free software.
Sigh.
If by world class you mean huge amounts of false positives and destroying the ability for people to send messages reliably between one another, then yes. It's totally world class .. a world class failure.
(I also remember being IMed by spambots using XMPP federation a few times.)
This does come at the cost of letting Google see all my email. I'm not happy about that, but I'm skeptical that a fully encrypted solution could provide filtering as effective. Certainly a self-hosted solution cannot.
(Except it costs money so most of my WhatsApp contacts will not switch.)