This is not apples to apples but Whatsapp is a product that entirely ran on 16 servers at the time of acquisition (1.5 billion users). It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently. Twitter was unprofitable during acquisition and spent around half their revenue on compute, maybe some of these features were not really necessary (but were just burning money)?
Whats app is not an applicable comparison because messages and videos are stored on the client device. Better to look at Pinterest and snap, which spend a lot on infra as well.
The issue is storage, ads, and ML to name a few. For example, from 2015:
“ Our Hadoop filesystems host over 300PB of data on tens of thousands of servers. We scale HDFS by federating multiple namespaces.”
You can also see their hardware usage broken down by service as put in their blog.
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
https://blog.twitter.com/engineering/en_us/a/2015/hadoop-fil....
- 450m DAUs at the time of facebook acquisition [0]
- Twitter is not just DMs or Group Chat.
> It really begs the question why Twitter uses so much compute if there are companies that have operated significantly more efficiently.
A fair comparision might have been Instagram: While Systrom did run a relatively lean eng org, they never had to monetize and got acquired before they got any bigger than ~50m?
[0] https://www.sequoiacap.com/article/four-numbers-that-explain...
Twitter, while still not profitable (maybe it was in some recent quarters?) was much closer to it, having all the components necessary to form a reasonable ad business. For ads, analytics is critical, plus all the ad serving, plus it’s a totally different scale of compute being many to many rather than one to ~one.
> This is not apples to apples but Whatsapp
And yeah, whatsapp isn't even close to an apt comparison. It's a completely different business model with vastly different engineering requirements.
Is Twitter bloated? Perhaps, but it's probably driven by business reasons, not (just) because engineers just wanted to make a bunch of toys and promo projects (though this obviously always plays some role).
In terms of what Twitter uses compute on, I'd guess analytics (Twitter measures "everything" for ad serving - go explore ads.twitter.com and analytics.twitter.com) and non-chronological timeline mixing both takes orders of magnitude more resources than the basic functionality.
1. Lots of batch jobs. Sometimes it's unclear how much value they produce / whether they're still used.
2. Twitter probably made a mistake early on in taking a fanout-on-write approach to populate feeds. This is super expensive and necessitates a lot of additional infrastructure. There is a good video about it here: https://www.youtube.com/watch?v=WEgCjwyXvwc
WhatsApp is mostly a bent pipe connecting user devices & relying on device storage. If WhatsApp had to implement Twitter functionality and business model (with same Engineers and stack), they'd need a lot more servers too. I'd hazard the number of servers would be in the same order of magnitude
I don’t know enough about Twitter to assess their infrastructure, but I know that it easy to run lean until there’s a problem, and then you get trapped.
Erlang is particularly well-suited to building distributed systems because it was designed to handle failures at the process level rather than the hardware level. This means that if one part of a distributed system built with Erlang fails, the rest of the system can continue to operate without interruption.
This is critical for a messaging platform like WhatsApp, which needs to be able to handle millions of users sending messages simultaneously without experiencing any downtime. Additionally, Erlang's concurrency features allow it to support many thousands of simultaneous connections on a single machine, which is also important for a messaging platform that needs to be able to handle a large volume of traffic.
