This applies to centralised/proprietary systems as well, including those 8 walled gardens that you use.

When Slack was created, nobody was using it. When Signal came out, nobody was using it. Same with Telegram, Skype, Twitter and all the other centralised/proprietary systems.

Some of those did manage to gain enough users for positive network effects to kick in, so "nobody uses it" isn't specific to FOSS/decentralised/federated systems, it applies to messaging systems in general. The fact that so many messaging systems keep being created, and so many users hop from one to another, shows that it's not a particularly strong argument either.

The real problems for systems like XMPP are the levels of investment required in marketing, UI design, etc. to compete with the proprietary systems.