Yes, they made certain engineering decisions now because the decisions they made back then were stupid, and they have to dig themselves out.
That Zuck wrote the first version of thefacebook.com in PHP, the language he was the most productive at?
That the initial team didn't rewrite Facebook in Python/Perl/Ruby/Haskell during the fast growth phase? If you have ever experienced the growth phase, you understand how ludicrous the idea of rewrite would be. I've personally experienced and heard only horror stories about rewrites. We underestimate how much hidden wisdom a production code base has and that the messiness is often there for good reasons.
And it was also written in PHP. And we rewrote in Ruby (not rails, at the time it was not mature in the ways we needed it to be). We made a lot of great technical achievements in doing this and our codebase became much much better. On a technical level I don't regret us doing that one bit. We went long on schedule and we made mistakes, but some of the best work of my career went into that and I'm immensely proud of it.
But then in a matter of months of Facebook going to being open to people who weren't at a university or big organization (remember that?) our users, who got bored with our site not changing while we rebuilt all the tech, completely abandoned us and we went from profitable (having never taken any outside investment) to dead in another couple of years.
This isn't to say that this wouldn't have happened to Facebook had they done something like this, but it is always a huge risk, even if you do everything right.
Using PHP to begin with was a bad technical decision. Failing to establish a reasonable migration strategy was a bad business decision likely rooted in bad engineering management that fell out of starting with bad technical decisions.
It's much harder to hire people that can pull you out of a mess like PHP when, at the same time, you have to hire people that can keep writing PHP for you.
> That the initial team didn't rewrite Facebook in Python/Perl/Ruby/Haskell during the fast growth phase?
That would have been a good time to bring on new engineering blood as part of scaling out, which would have provided opportunities to enact mitigation and transition strategies. Imagine if the massive amount of talent currently devoted to HHVM had been devoted to Facebook's actual business?
There are migration strategies other than "rewrite everything immediately", and in fact, I'd bet that's exactly what HHVM is. It's just a shame they waited so long that the most cost-effective migration strategy was to tackle an enormously difficult computer science and engineering problem that the world's biggest software companies already invest hundreds of millions of dollars on and provide to the world largely for free.
> If you have ever experienced the growth phase, you understand how ludicrous the idea of rewrite would be.
Yes, and I've also (repeatedly) been the team brought in to rewrite the mess of a code base that was about to torpedo the growth phase.
There's not much correlation between funding, initial success, and engineering talent. Which is why you so often wind up with a mess that has to be cleaned up once you can hire people who know what the hell they're doing, instead of the ones you happened to be stuck with because you didn't know how to grow an engineering team.
> We underestimate how much hidden wisdom a production code base has and that the messiness is often there for good reasons.
Messiness is never there for good reasons other than that replacing it is more expensive than not touching it. You don't strive for hidden wisdom and inescapable messiness -- that's just what you get when you let engineering slip up.
Which language would have been a good technical decision in 2002-2003? It needs to be fast enough in terms of iteration. It needs to not require more resources than PHP. It must be easy to onboard people who don't know it onto. It needs to be easy to operate, and not be costly to deploy on the tens, hundreds, and then thousands of servers necessary. (Spending time learning a new technology that others think is cool or which seems cool, while trying to also build a product, has sunk more than a few startups...)
What was bad about the decision to keep the reasonably well-performing and reasonably suited-to-purpose PHP code for front-end code, and peel off suitable tasks into services like the feed, typeahead, messages, and so forth into languages like C++, Java, and so forth. What was bad with the decision to let hundreds of software engineers continue to build the PHP code-base while a much smaller group of people work on projects to improve the efficiency of both the execution environment but also the tooling and developer efficiency on that code-base? Their contribution there is multiplied out by the improved efficiency of those hundreds of developers and the code they wrote.
Seemingly Facebook survived its growth phase fairly well and didn't need a tiger team from outside to handle it - and without the reliability problems others who chose to use languages other than PHP for their front-end and decided to rewrite their much smaller surface area in other languages.
As much as people may dislike PHP (and I'm one of them), it was definitely "good enough". Many languages may not have been, even if they are nicer languages in some objective way.
(Disclaimer: I actually have to write code in Facebook's PHP code-base every once in a while. But most of the code I work with is Python, Java, or C++.)
Facebook is a ten year old company with a market capitalization of $170 billion, so "bad" is probably not the most accurate word to describe their technical and business decisions.
Facebook is surprisingly easy to underestimate even as someone using it a fair bit.
Just try find every single interface in the front-end. For your own profile. Feed is front-and-center, as is timeline. Then look at events, groups, &c.. Look at messaging.
Then look at the interfaces for managing your privacy and permissions. The apps that you're using and information about when last they requested information. Security like login approvals. Then think about the flows involved in reporting content as abusive or inappropriate. For verifying your identity if you've forgotten your password. For adding more security if you log in from a new computer or from a new location.
Then look at pages, including insights and scheduled posts and so forth. Then look at advertising - boosting individual posts, creating campaigns, &c.. Then look at the interfaces for developers. For translators. Interest lists.
The backends for most of this are in C++ and Java. There's a large amount of data processing happening to track hidden things like spam and scam prevention. But the front-end surface area is quite clearly an order of magnitude or maybe two larger than Reddit (at least where it was when this happened).
Rewrite is more likely to succeed if you are not in a high growth phase, but even then it's risky.
Trying to build a reasonable forward-looking high-performance managed language runtime platform is hard.
Trying to build it atop of PHP is harder.
The only way I could see this as being a smart long-term strategy is if the eventual goal is to isolate and retire PHP projects entirely (and PHP usage itself) over time.
However, even then, with PHP gone, and Hack no longer necessary, you're still stuck maintaining your own incompatible VM / runtime. Is Facebook signing up to reproduce the CLR? Or do they have long term plans to somehow bridge the gap between HHVM and more established VMs/runtimes, where they can better share the maintenance load with the wider industry.
I detest PHP, but I've still more than once made the choice to do apps in PHP motivated by developer availability alone.
It's not a great language, but with some discipline it is also not nearly as awful as some people like to think, and you can make up for a lot of awful with the difference in ability to hire experienced engineers who know PHP vs. many of the less common languages.
