Is it only that XMPP was overcomplicated and if managed better it would survive? Or is it something else?
* Onboarding was difficult. There was no obvious choice of server or client to use.
* Adding friends was difficult. You needed to send a subscription request to a contact, and they needed to send one to you. If anything happened during this process, you couldn't chat.
* Popular XMPP clients, like Pidgin, also supported the other chat services (AIM, ICQ, MSN, Yahoo, etc) so people just continued using those.
* Network effect. You need to convince a mass of people its better, otherwise nobody's using it because no-one uses it.
* No obvious benefit to the user. It's decentralized sure, but there weren't many improvements over AIM that people actually used.
* A lack of good iPhone XMPP clients.
In 2005 Google added XMPP support to Google Talk/GMail Chat and they were federated, but nobody federated back and they closed off its successor (Hangouts).
As someone whose company used XMPP prior to the iPhone, this is what killed it for us, and I suspect it’s a major contributor to federated XMPP losing any public inertia it had.
iPhone push notifications could only be sent with a signing key tied to the same developer account as used to publish the client application.
That meant it was impossible to send push notifications from your own XMPP server to a generic XMPP client written by someone else.
I don't think "lack of federation back" was a big driver in this decision. Everyone who used XMPP was able to chat with gchat folks and despite some weird changes Google made to their integration, it worked reasonably well. You didn't need to do anything special to federate with another XMPP server. It just worked like email does.
I expect either product complexity (like having to support non-gchat addressing which then requires full JIDs instead of short names) or de-prioritizing features that weren't directly driving their growth thesis were probably more relevant. Ie, why spend two engineers to fix integration concerns and deal with federation everywhere, when you can just retask those headcount onto some new feature that will drive growth.
And no server based push, though I'm sure someone invented an xmpp extension to fix that eventually :)
IRC has bitlbee (and probably others) to fulfill that role https://www.bitlbee.org/main.php/news.r.html
Matrix has "bridges" to interop with other protocols https://matrix.org/bridges/
But it's very hard, technically, to maintain interfaces to a changing set of evolving third-party protocols, and provide a good user experience. Especially if those people don't want you bridging to them. Open networks tend to fare better in the long term, but generally the user experience of an app bespoke for the target protocol is nearly always better.
Not only that. I'm putting the downfall of XMPP right about where most people started having a smartphone (2009-12), so it certainly didn't help - also on Android there weren't any great ones, the ones we know today are all younger.
SMTP is an old protocol; it really predates much of what we'd think of as the Internet (e.g., it predates DNS or even IPv4). This means that during the big initial growth phase of the Internet, SMTP was already an established standard that could be used to transfer email. Indeed, for email routing on the internet, you kind of had to support SMTP anyways, which makes it difficult for any other protocol to gain traction.
That's not to say that there weren't alternative protocols. The biggest of these was X.400, which many in the 90s saw as the eventual replacement of SMTP. But this was hampered by the already existing install base of SMTP. Other failures leading to the failure of X.400 was its reliance on the OSI stack, which lagged behind in implementation compared to the TCP/IP stack, and the rather cumbersome addressing model of X.400 compared to SMTP. The value added in supporting X.400 in addition to SMTP wasn't worth the cost of implementing X.400, and it wasn't really feasible to support only X.400 given the already widespread use of SMTP.
In this model, XMPP actually works closer to X.400 than it does to SMTP. It came about after the protocols it aimed to replace were already in existence--and wide use. (See also IRC, which is still alive and active). So implementing XMPP means you need to justify the value-add of XMPP over existing legacy protocols. If you were implementing your own, new chat service, it might make sense to build it on top of XMPP instead of a custom protocol. But replacing existing chat protocols with XMPP was again a costly move with benefits rarely justifying the move. The further federation goal of XMPP would have been an anti-goal to many of the chat implementations.
Email address routing used to be ugly too, but the consolidation on the <user>@<host> standard really helped there.
E.G. wasn't advertising basic client online / busy / etc janky between different siloed implementations?
I don't know about IPv4... IPv4 was RFC 760 (Jan 1980) while SMTP was RFC 821 (Aug 1982). DNS was certainly later, RFC 1034 (Nov 1987).
IPv4 is defined by RFC 760 and RFC 791, although I don't know if RFC 760 actually implements something that looks like modern IPv4 (one of the challenges with old RFCs is that they were actually requests for comment, and I don't have a good sense of when this practice actually stopped).
I was going by Wikipedia's dates, which give the actual use of IPv4 on ARPANET as January 1983, which I believe postdates the actual use of SMTP on ARPANET.
naming. The biggest failure of X.400 was naming. X.400/X.500 style naming was and remains an unqualified disaster. Not that the pre-DNS alternative (UUCP) was great, but that name@domain turns out to be infinitely better than X.400.
According to the XMPP offical website to add some numbers to what you mentioned:
~800 million WhatsApp
~200 million Zoom
~4 million Grindr
I wouldn't at all be surprised if there's chat services that have you connect your local app with XMPP to, and get a simple UI layer that only connects to specific servers and other provides other config settings.
I've also heard of other uses besides chat, like for server management or video game multiplayer games for example. It's a great protocol that's provides delivery logic, accounts, etc. but keeps the actual messages simple and flexible for whatever you want to do.
I think SMTP "won" from being focused on email management instead of a more general protocol for multiple uses. (not that you can't do wacky things with SMTP, of course)
The client side Cisco Jabber app is Cisco's version of it that works with their VOIP.
Believe it is both Client & Server, and that is an interoperable implementation that looks like a reskinned pidgin somewhat in older versions, but has a great deal more functionality now, and looks capable of doing VTC and other things [2]
Considering how popular Cisco VOIP is, I think lots more people have XMPP functionality than they realize
[1] https://www.cisco.com/c/en/us/products/unified-communication...
[2] https://www.cisco.com/c/en/us/products/unified-communication...
Matrix is not built on any existing Internet Standards for messaging and is not compatible with XMPP in a meaningful way.
It most definitely is not built on XMPP in any way.
In short, data-at-rest still can remains unencrypted for majority of the messagings on the WhatsApp’s server for all those who may be interested in them … UNLESS you enabled Signal/Encrypted option for two-way.
Group chat for WhatsApp?, it’s still the proverbial pants’ down.
The environment outside of major providers was (it definitely is) kind of janky. If you can set up a server application it's not hard but it's not exactly easy to get started, or get people on it without handholding (from my own experience).
The biggest problem I remember was that XMPP was (is?) about message delivery, while what we really wanted was "synchronized message log".
So multi device did not work: One device was designated "active" and would receive the messages, others would not.
There was no history sync of any sort: If you had checked messages from home, they would not appear in your work computer's history next morning. If you replied from home, you won't be able to see your own messages at work PC.
Anything mobile (mobile phone, laptop in coffee shops) was also unusable -- you cannot start app and catch up on all missing messages. You had to be online to receive them.
For mobile access, I ended up doing custom Pidgin plugin (in perl!) which would export messages to text documents served over webserver, and then I'd visit that page from my phone to read the messages.
---
I still don't know what was wrong with that system, was this our admin not spending enough time on the server, bad documentation, or bad clients -- but this did not really matter, this experience really destroyed XMPP brand. We used Google Talk and Slack later, and they were "just working", no batteries required, no messages lost.
The funny thing I believe that Google Talk might have used XMPP under a hood at some point -- but it worked really well. So it is definitely possible to build working, feature-full things on XMPP... you just had to extend it and use it as underlying layer. Kinda like TCP and IP in a way.
You're spot on with your memory of early XMPP's behaviour. While it was one of the first messaging protocols to support simultaneous login, the focus was originally about routing messages to the right device. In the context of that period in tech, that was excusable and even desirable. As people got more devices connected, more permanently yet intermittently (i.e. smartphones), it stopped making any sense. We switched to a broadcast-and-sync approach, but it took a long time (years) for all the long tail of smaller internal deployments to roll that out (if they ever did). Another big issue was that one of the most popular clients was Pidgin, which ran into developer resource issues around the same time and still hasn't caught up yet. To many people though, this was XMPP.
I'm hoping a new generation of XMPP implementations can solve these kinds of issues, which was what inspired my "Products vs Protocols" [1] talk/article a couple of years ago, and is why I work on Snikket (and we have many happy users who are both new and not-so-new to XMPP).
All of these companies chose to kill federation for their own advantage, making the world poorer.
Controlling spam inside a closed network is drastically easier than controlling spam in an open network, because in a closed network you have lots of ways to make bans stick, whereas in an open network you really can't. Faced with the choice of building a massive Gmail style heuristic spam filter in which basically all the messages it was processing would be spam, or, just turning off federation, they went for the latter. It was probably the right call. Very few people even noticed, let alone cared. The idea of an SMTP-style global network of federated chat servers was dead by that point.
https://www.slideshare.net/InfoQ/how-slack-works
It may have subsequently turned into something more like how IRC servers work, but early on it doesn't appear to have been.
Of all the original internet protocols, SMTP is the ice that needs replacement the most IMO.
> companies no longer send confidential info through it.
It depends, B2B absolutely still do.
What I meant was that Email as a whole should be overhauled, though SMTP is by far the weakest link there. All the kludges and workarounds like DMARC, SPF etc don't fix the issues, and this is why so many companies are inventing the wheel. For example, Microsoft won't let your server deliver mail to them if you don't have enough 'reputation', meaning you must send a certain number of emails that are not spam. Otherwise you will find yourself in the doghouse (I've commented on this more detailed before). Even if you never sent spam in your entire life and your server isn't on any blocklist!
SMTP is decentralised yes and that is a great thing. But such measures like MS are doing totally undermine this decentralisation. And it will continue to do so, it's already super hard to run your own small-time server for the issues mentioned below. This decentralisation is already disappearing. It's time for a new protocol that is ready for the future.
> It depends, B2B absolutely still do.
They do for now because people involved in sales are usually hungry for business and don't care about the risks. They want to have low barriers for the customers, which is understandable. However there are significant risks depending on how the customer does their mail. We're on O365 and comms with other companies that are also on that are obviously pretty secure because they never go unencrypted through the public internet. Again a point that Microsoft (and Google) make into a selling point for their services, further undermining decentralisation.
We in the cybersecurity realm are imposing rules, e.g. any confidential info must be shared through secure filesharing. But it's hard to avoid this happening.
- multiple devices sharing a mailbox
- store in plaintext?
- if store in ciphertext, how to index, search?
- how to share decryption, signing keys among devices?
- how to enroll devices?
- how to exchange keys?
In terms of "End to End" I would consider the user the endpoint, not the device. There's many techniques that can make that happen. But this is more on the receiver side of course.
PGP is obviously not simple. S/MIME is also not simple. This is why nobody bothers using either (though S/MIME is used internally within a lot of corporations).
One is that XMPP is an overcomplicated nightmare. The protocol was verbose, flabby, and hard to implement, and the server software was very hard to set up and run especially for novices. A person could not just install a server and start chatting, even if the server was just stand-alone let alone linked to anyone else.
The second and IMHO more fundamental reason is that the Internet is a dark forest. Any open system that becomes sufficiently popular will be destroyed by abuse.
If you offer a chance to make any amount of money whatsoever online millions of hustlers will rush into the void like gas molecules invading a cracked vacuum bell and will scramble all over each other to suck every last fraction of a penny out until nothing of value remains. So far only centralized managed systems have been (somewhat) successful at keeping the barbarians at bay. I'm not saying a decentralized system could never succeed here, but I don't think it's been done yet. (Cryptocurrency isn't an example as it's already been throughly destroyed for its original vision and use case by scammers. It's a great case study in exactly what I mean.)
(Edit: the goal is not always money either. Read money as "value." Political propaganda, cult recruitment, weaponized disinformation, or just trolling for lulz all count as extracting value of some kind at the expense of the commons.)
SMTP along with Usenet was one of the first casualties of this phenomenon, and I would argue that it did fail as an open system. You can run your own SMTP server but it's not for novices or people without time on their hands. You'll have to fight constantly to keep your IP out of blacklists and to keep spam away from your users. Spammers are in an arms race against huge companies with massive training data sets and entire teams dedicated to spam filtering, so this only gets harder over time. An independent SMTP server is easy prey.
99% or more of users use one of several large mail hosts. These are mostly Google, Microsoft, and Apple in the USA. E-mail hosting is cheap to free and the vast majority of people would rather someone else deal with the pain of defending them from spam.
It's even worse than that because just a very small group (dozens, or even a single individual) can really screw over an entire system because doing things digitally is so cheap and easy. For example the world has 8 billion people, and sending every single one of them an email message (assuming everyone has an email address, which they don't) is easy, cheap, and can be done relatively quickly. With just 100 spammers on the entire planet your email system would still be screwed.
This is a harsh truth that I'm always surprised that so many HNers haven't accepted yet.
It's not that most people are bad. It's that it doesn't take very many people behaving badly to ruin a common platform on the Internet since one bad actor can use cloud compute and bots and other resource pools to launch huge attacks. Add anonymity to the mix and you've got a recipe for easy high-impact low-risk abuse.
Chat didn't get super-useful until everyone had a screen in front of them or on them just about every waking hour. People used it before (I know, I was there, and I did) but it was more of a "see who's on and chat with them" than what it is now.
SMTP made these all interoperate and ultimately replaced them all.
It's just that getting federation, interoperability and backwards compatibility is much harder than inventing your own new thing + users gain the freedom to move to a different provider (without loosing all your contacts).
I tried XMPP early on, and found the software to have a user experience that was utter garbage, so after that I didn't give it a second thought.
At the enterprise level, though, garbage UX is the norm. SMTP is hard to deploy, but you have paid full-time sysadmins whose job it is to figure out the necessary parts of sendmail.cf or whatever else. You tell them "make it so" and they do.
I suspect there was no enterprise level push for XMPP anywhere.
At the about the time when federated XMPP could have gained momentum at the enterprise level, 2005-2010, SaaS and then cloud began to take off. Enterprises began to shy away from hosting their own services. Even though many were still hosting their own e-mail and web servers, it was already too late for XMPP, especially given all the other factors--IM wasn't yet a killer business app, etc.
Without a large, diverse group of players effectively forcing federation, there was nothing countering the incentives for big players to create walled gardens.
I worked at company, Barracuda Networks, that acquired an enterprise XMPP appliance product around 2005. They also hired the main developer of one of the most popular open source Windows XMPP clients to help them integrate enterprise features (LDAP, etc). After a few years they basically put the project on life support. If there was more enthusiasm, and Barracuda worked harder with other vendors to build a large installed base, improving the overall value-add and "creating" a market, maybe things would have worked out differently. But that's a big ask for a bunch of medium-sized, short-term profit-seeking companies. Eventually startups like Slack ended up doing something like that, but using proprietary, walled-garden, hosted services, a model where they could capture entire markets--i.e. a big enough potential pay-day to make it worthwhile. :(
Another alternative scenario: maybe if anti-trust regulators came down hard on Google when they began to close-off federation. GTalk could have been (and briefly was) an important anchor tenant for a nascent federated IM ecosystem. Not sure it would have been legally viable, though, to restrain Google that way given the state of IM at the time.
Gmail and Facebook adopted XMPP. But it seems like it wasn't a good fit for their business model of shutting users in and collecting as much data as possible, so they shut it down eventually.
> because someone’s choice to use an XMPP client or server that doesn’t support video or some other arbitrary feature doesn’t only affect them
It was actually even worse than that, because clients also have complete freedom in which audio and video codecs to support. So you would end up with client 1 and 2 both supporting video, but not the right protocol, so it didn't work, and it was hard to figure out why.
All of these federated protocols: XMPP and even Mastodon to an extent, are smaller groups of humble nerds running instances. They aren't massively advertised, large-scale billion dollars of investment companies like Discord or Facebook, trying to build network effects and/or advertising revenue.
-----------
When competing against profitable companies, its not sufficient to merely exist. You're competing against advertising and eyeballs. Facebook is big because its big, because it advertises, because it is constantly pushing for more-and-more users. Same with Twitter, same with Discord.
This leads to little advantages: a $10-million+ UI overhaul every few years. Designers to simplify the interface and onboard faster. Paying Google/Apple absurd amounts of money to access the push-notification APIs on phones. Etc. etc.
Smaller XMPP instances already fail at the push-notification thing. Who will pay for that? And without push-notifications, do you really have a modern chat platform? Other companies can afford the costs.
At the time of writing this, 90% of 363 tested XMPP domains support push notifications.
I'm not disagreeing with your overall point, mind. There is lots to compete with, and due to the lack of a good business case for open networks, most XMPP projects are by open-source volunteers and the majority of public XMPP services are similarly operated by volunteers.
There do exist projects with funding though, and there are multiple companies actively engaged in paid XMPP projects and deployments. But these all tend to happen behind closed doors, they're typically not deployments for the general public, or the XMPP is quietly under the hood of a larger service/product (such as happened in Zoom, WhatsApp, and many online games such as EVE and Fortnite).
As far as any individual user is concerned, loss of the server(Which their identity is tied to) would be a hassle. Just like email.
SMTP got really entrenched really early. It's what gmail uses. They can't really do a proprietary mail protocol, SMTP was around way before them and nobody wanted to switch.
Instant messaging for the masses was proprietary from the start, as far as I can tell.
There was IRC, but it seems like by the time the internet was everywhere, people were already on AOL and ICQ and a bazillion other proprietary ones offering various extras and integrations.
None of them seem that interested in open protocols.
Universities and others using mainly UUCP also migrated because of the benefits.
There wasn't really a huge benefit for the big instant messaging players to integrate, and the dirty secret is cross-platform "instant" messaging has always existed - it's SMTP! Many people still use email as a form of a chat tool.
XMPP also suffers from the "oh shit security might be important" that SMTP originally didn't deal with (it's horribly insecure by modern standards) which greatly increased the complexity.
More so, absence of security is what allowed SMTP to proliferate.
SMTP was early enough and good enough to get widespread adoption. IRC in its original form was never good enough (esp. re: scaling), so services like AIM were able to step in and were good enough and open enough[1] solution while the open source world stuck with IRC for too long because it 'works for us'. So open standards like IRC never got entrenched the way SMTP did in proprietary solutions and by the late 90's the open source world had become pretty weak on developing new Internet standards that would even get widespread adoption in the open source world.
It's worth noting that there's nothing etched in stone to say that SMTP has won indefinitely. Try setting up a purely open source SMTP server on the Internet these days and see who you can talk to. Google has been making it harder and harder[2] to use non-Google clients with their servers. Slowly but surely, business interests seem to be enveloping SMTP with their own walled gardens... it's just taking longer.
[1] Well, sort of. They actually seemed to want to be proprietary about it. But every time they changed the protocol, the apps that worked with it were able to reverse engineer their changes quickly and AOL didn't fight too hard to prevent them from doing so.
[2] Or at least enough of a chore so that most people won't bother trying.
The trick I remember the most was really quite clever - the server started sending hash challenges to the clients i.e. a random byte range into the client binary. The client had to respond with a hash within a timeout or else the server disconnected them. The trick was, because the challenges were randomly generated you couldn't make a database of them, but also, the copyright license on the client forbade people redistributing it. Eventually it was worked around of course, people set up servers to answer the oracle queries that clients could relay to, but it really disrupted the alt messenger scene for quite a while. And in turn that meant it screwed people who wanted to use Linux because AIM was the number 1 social network in the USA during this period (though it's all forgotten now and was never as big in Europe, where MSN dominated).
Anecdotally I experimented with an XMPP server for awhile and found it far more difficult to start up than an IRC server. It was far more enterprise oriented, assuming you had control over the DNS (only one server per domain of course) and with lots of complexity around LDAP or AD integration. This is as opposed to an IRC server where you downloaded the software, optionally exchanged keys with a peer, and started it up.
Using XML as a message format was also a mistake, but not one so bad that it would sink the entire project. ASN.1 would have been a better choice.
Then we tried Slack but most people didn't come there too much. Today we have a Telegram group and it's so much better UX/UI wise...
I thought it's what everyone used for messaging and push notifications. I know Google extended it beyond compatibility around when Hangouts came out and they started doing browser notifications. Apple stopped providing sources for ejabberd in the last few years, so I assume they moved off that for their iChat/messaging backend, but isn't iChat still an XMPP client?
Aren't most SMS messages routed via XMPP?
Google has also discontinued Hangouts and who knows what you're supposed to use now. Their chat stuff has been so chaotic I really can't keep track. (Allo? Duo? Plus? Google Talk? Wave? Chat?)
That's really cool. I would love to send XMPP messages to my friends who use iPhones without having them register on a public chat service and downloading a client.
XMPP:
- is designed for chatting and P2P and VoIP over persistent HTTP connections
- requires everyone to be online
- servers have to implement 50 different specs
- works at the frontend and backend
- is designed to send one large message in bulk over a regular short-lived TCP session
- is a store-and-forward messaging system, highly resilient to slow or inconsistent networks and server issues
- is simple and the ecosystem is layered, so the core server doesn't have to implement a dozen specs
- is practically backend-only
I'd not really describe it as fragile or bloated either, having used it for twenty years or so with no real problems except the now-solved absence of e2e encryption (OTR, OMEMO).
Still, clients could be better.
- XML
- XHTML
- vCard
- Bookmarks
- Message Archive
- P2P
- VoIP
- Multi-user Chat
- File Transfer
- Data Forms
- Service Discovery
- Pub-Sub & Personal Eventing
- Bidirectional Synchronous HTTP Streams
- WebSockets
- Push Notification
- Protocol Gateways
- TLS
- OTR
- PGP
- SMTP
- TLS
24 SMTP specifications (https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#...)
As developers we look too often for technical reasons for a limitation or failure but usually the social reasons are more significant.
Getting a client installed on a computer was socially difficult like convincing some one to install an app on their phone. Having a browser installed on every computer and phone made it trivial to get a client on a computer the only down side is being forced to use http and websockets as the protocol.
DNS and WHOIS used to be protocols in their own right but both are being replaced by http, I wouldn't be surprised if http came for smtp next.
This is an under-appreciated observation, and adds useful context to a lot of the other answers given here.
> I wouldn't be surprised if http came for smtp next.
I hate to be the one to tell you, but:
That might be the biggest contribution matrix has to offer.
The json encoding is large and seemingly under defined. Many things only seem to work with element and the python server.
Doesn't sound very complicated, yet?
> The json encoding is large and seemingly under defined.
Examples?
I vaguely recall there being something else as well as ICQ but we’re talking back to high school for me so well out of the range of good/clear memory.
I think messenger also came into existence at around the same time as jabber so concurrent development presumably meant completely independent protocol, so it wouldn’t even be subject to embrace/extend/extinguish - this was the height of MS monopoly so presumably it meant essentially “everyone” instantly had messenger?
So by the time jabber and xmpp came around there were giant existing networks: AIM, ICQ, Messenger. Obviously none of them had any business reason to adopt an open, etc protocol.
Additionally even if they wanted to, actually doing it would likely have been a significant engineering challenge, due to the need to maintain compatibility between client versions.
Aren't open protocols and interoperability great?
At work (operations for a space mission), we use an XMPP-based chat system for tactical communications along with several other systems, but Slack was introduced and more and more communcations are being handled over Slack. I'm advocating for replacing several systems with Jitsi, which is XMPP-based, because it allows us better control and customizability. It's an uphill battle.
There are probably multiple reasons and we can't replay history to know which reason contributes the most but one key difference is that email accounts (SMTP) were pushed onto the regular non-techie consumers whereas XMPP was more of a geek tool that users had to pull and opt into.
E.g. New students at a university automatically got a ".edu" email address. Residents got a free email account (ISP) with their cable service. Employees got a corporate email address. In other words, millions effortlessly got the utility of email/SMTP even without installing AOL CDROMS.
To further reinforce email/STMP, if consumers want to order something from Amazon, they needed an email address to create an account instead of an XMPP address.
Other forms of communication like XMPP/IRC don't have that widely disseminated self-reinforcing utility cycle so they stay a niche tool. What critical service in normal life requires an XMPP address? I can't think of any.
But, like commenter tetraca mentioned, XMPP has been embraced and extended, and it's not very interoperable, as implemented.
SMTP has been and will always be around forever, albeit eventually viewed like FAX is today. I personally view it like that today.
I'm not entirely certain why this was, but I suspect that valuing extensibility over interoperability (or the perception of it) was part of it. Optional extensions and interop with hosts that used a different set simply gave hosts/operators too much to think about in terms of deploying the service for users.
In the 90s email addresses along with domain names became part of professional fashion. Everyone wanted @mycompany.com addresses, and that created one of the obstacles for walled garden email. And ISPs often offered email-addresses to subscribers.
"The Microsoft Network" was an attempt at a walled garden that failed.
Hotmail etc had no choice but to support the existing open protocols. Hotmail was just one of many free webmails.
No instant messaging network used "global identities" (or hierarchical identities/addresses like email). The early-mover ICQ was a walled garden, and it died quickly when it became outdated. ICQ logins/identities had no relevance to anything else.
If instant messaging platforms that wished to form a network had existed, then adopting some protocol would have been trivial. Creating new a network of services/operators competing with already an established platform is not trivial.
It's 2022, XMPP is 23 years old, and I still can't use it as a primary solution for my community. There is no client/server environment available that meets the most demanded purposes and runs on all common devices.
In 5 years I have tried to set up an infrastructure 3 times to have a look, I always come back to Matrix (just because I have more hope that it will turn out better in the future).
XMPP is ready to do too many things, that's a good point, but the most common practices that everyone finds on centralized solutions are not implemented on all platforms, clients, and servers.
The server documentation doesn't give any advice on which clients to use, and the clients doesn't advise which server to use. Everyone has their head in the sand.
It's been at least 15 years since it's easy to set up an email infrastructure by yourself, that everyone can use on all their devices.
If there were an alternative to SMTP I believe it would be NNTP given it is similar in concept and just introduces threaded messaging which SMTP clients try to mimic now. There are several potential rabbit hole discussions of why moving away from SMTP are highly unlikely to ever occur.
Essentially, XMPP's federation goal was antithetical to the vested interests of messaging providers so it was ignored. But the protocol itself was relatively well designed and so formed the basis of a lot of what we use today.
They never managed to promote pleasant onboarding and finally shut it down ('nobody wants such') months before WhatsApp rocketed - with ejabberd.
SMTP was first so one registered accounts with email and not XMPP. Companies send stuff like bills via email.
There were various popular providers -- today Gmail is the most popular one but others like AOL were popular before. It's hard to change one's address.
Sometimes, your ISP gives you an email address.
Email is formal, so job application happens also over email (at least in not very technical jobs).
Probably because email was so important, there were and are still more people who host themselves, including companies (companies with their own xmpp setup are rare).
The rest of email is today (mostly) at Facebook, Twitter or LinkedIn.
Eventually, for various reasons, Google decided that an open xmpp based system didn't work for them, and they closed it and extended it. This was effectively the end of xmpp as a distributed tech that many people use today.
Interesting, in a lot of ways SMTP is no longer an open system. It is extremely hard to operate a mail server in such a way that your valid emails get delivered correctly and not marked as spam. So for example in the name of fighting spam, it is extremely difficult to participate in this "open" system.
Same issue with Mastodon, in my opinion. People want to grab something from the Store or Play, sign in, and be where everyone else is.
SMTP dates from the earliest days of the internet and you don't really see new decentralized/federated protocols after about 1995, which is when IM started to be developed. But that isn't by itself an answer, it just raises the question of why open protocol development mostly stopped in that era.
I think a big reason why was the advancement of networks and hardware made it feasible to actually build and run large internet connected datacenters for the first time. In turn that meant you could run a server farm to which everyone could connect to, without running massive modem banks and the like.
In the early days of the internet that wasn't the case. Servers were machines running in people's offices and the ecosystem was extremely fragmented. There were lots of different kinds of network and email system, SMTP was really a sort of hack to bridge different systems together. The internet was congealing, being patched together out of lots of existing stuff, and big companies mostly didn't care. Networks were often only intermittently connected, the internet was for hobbyists and researchers, not 'serious' companies like AOL or Microsoft, who didn't "get" the internet at all until some famous memo by Bill Gates (they were pushing their own closed network). SMTP reflects this world with its handling of offline servers, lowest-common-denominator simplicity etc.
Once big companies realized the internet mattered and datacenter / networking technology got far enough along, the era of decentralized federated infrastructure died out pretty quick. This is in many ways very sad, but it's also in some ways inevitable. The decentralized cooperative standards-based approach sounds romantic, but it can be described in other much less noble ways e.g. as a form of committee-controlled pure communism in which nobody owns anything.
Phrased that way it's maybe obvious why that model of development has failed: there's no incentive to do anything good. SMTP isn't good. It's like an MVP email protocol. XMPP isn't that good either in many ways; you certainly wouldn't design a protocol that way today. And of course the issue here is not just the protocol but the entire stack - the servers, and most especially the clients. Every federated protocol system always suffers badly from a fragmented and min-viable client landscape, which certainly held back XMPP a lot.
Who is incentivized to make something good? Well, people who have an ownership stake in the results. The prospect of a big payout is what lets you attract teams of the most skilled people who will work crazy hours to build a service people love. It's the capitalist model - wall off the garden and charge an entrance fee. Suddenly finding gardeners isn't a problem anymore, because you can just pay them instead of asking them to volunteer.
The lack of ownership causes all sorts of subtle problems in the federated protocol landscape, beyond the obvious ones like a proliferation of low quality clients written in people's spare time. Probably the worst is that there's little innovation. People become motivated to build a federated/open protocol mostly when they see a successful proprietary product and they'd like to have the benefits of that product without the loss of freedom, dependence on a single vendor, having to pay for it etc. These communities are always going where the puck was, not where it's going now. By the time they finish cloning one cool feature their proprietary competition has, it's too late, that competition already moved ahead and added something else. This really nailed XMPP because networks like MSN were constantly adding features that XMPP couldn't easily respond to, and you see it again with stuff like stickers in Telegram. No open source dev is going to spontaneously decide to add stickers if they haven't seen that feature before, because it'd require recruiting a ton of artists and would seem trivial. Proprietary software companies are much more likely to come up with popular stuff like that. They find it much easier to build multi-disciplinary teams.