I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebases at their scale?
I've found it to be hugely useful in safely working in a large Python codebase :)
(NB: I work at Dropbox)
Sounds like a reasonable guess.
> which is interesting given most providers are writing new infrastructure code largely in Go
Dropbox apparently uses Go a lot, see e.g. https://about.sourcegraph.com/go/go-reliability-and-durabili...
They also use Rust for some performance-critical stuff, e.g. https://news.ycombinator.com/item?id=11282948
That being said, AFAIK originally Dropbox was written mostly in python. Probably there's still lots of that left.
I'm not sure this is true. There's a lot of Java around, Scala is pretty popular too, in many industries C/C++ are the norm still for this sort of code.
Go might be the trend in open source infrastructure projects right now, but a significant amount of that is likely to be inertia from Docker and Kubernetes.
I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products.
"Using the network database and configuration tool developed by our Network Reliability Engineering (NRE) team,"
Another custom tool? Network inventory and config management tools do exist already...
"Rather than having engineers manually running tests using playbooks, Pirlo performed an automated sequential battery of tests that reduced the need for hands-on attention and concurrently increased diagnostic accuracy."
Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.
Most recently I've been rebuilding my own CI stack and never really gave much thought to going back to Jenkins. So I've been asking around and one of the only common complaints I've heard so far is that getting the initial configuration done is painful and generally orthogonal to automation. Plus the documentation is atrocious.
No off-the-shelf product will be a perfect fit, but with CI software I was truly surprised at just how large the gaps were.
So, sure, if you're Dropbox and you want to automate everything Jenkins is almost certainly not the right tool for the job. If Dropbox already had a supported, mature in-house job queue system, why not use it?
Conversely at megacorp, they spent 4+ years claiming to work on deploying a Jenkins (CloudBees) cluster and still came up with bupkis. Our own internal job and message queuing systems were astoundingly bad (and support for internal tools was almost entirely forbidden).
At megacorp I absolutely decried any sort of home grown solution. But if Dropbox were to actually tackle the problems of internal testing and support and come up with mature solutions, why shouldn't they use them?
Maybe, just maybe, the engineers in question considered mature open source tools and decided they didn't serve their needs.
I think the part "Leveraging in-house primitives" is the important bit here. It sounds like they have an in-house task queue, so introducing another external one like Celery might add a lot of operational complexity rather than piggy backing on existing infrastructure. As for the reason for the existing infrastructure not being Celery, I suspect at Dropbox's scale tools like Celery start to break down.
> Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.
They likely haven't written their own job queue, they've just used an existing one. Again, the operational overhead of Jenkins is unfortunately very non trivial.
Not to mention the overhead of running Jenkins at the kind of scale they're talking about here. We're not talking about small numbers here.
^ perfect band name
