How to scale a distributed system [pdf] (opens in new tab)

(cdn.oreillystatic.com)

214 pointsHenryR7y ago10 comments

10 comments

Another quick hint: always make things work first then profile and optimize it. This happens when my colleagues were building a stream processing engine that scales up. At first, they start with all those fancy fine-grained locks but it turns out the final result just does not scale at all. They have to retreat to the version with some big locks and then figure out what was happening and how to solve it.

hardwaresofton7y ago

Am I the only one that found that there wasn't really much actual actionable information in here? There's basically no specific information about how they scaled along the axes they established were important

HenryROP7y ago

That’s totally a fair point. I could have given a more concrete talk about how I’ve done these things in real systems (and the original version of this talk, presented internally to my company, had more real details and spoke very candidly about mistakes that I’d made along the way). Here I wanted to give more of a “here’s how you might want to structure your thinking, along with some basic design principles” kind of talk - I wish I’d had time to give more detail!

hardwaresofton7y ago

Yeah it was an good talk from the perspective of laying out the landscape and I found myself agreeing with everything, only to find that I was close to the end of the slide deck :(

Is there a blog somewhere where you've written a bit more about it?

1 more reply

carapace7y ago

It's a bit dated but "Scalable Internet Architectures" by Theo Schlossnagle is still good (IMHO)

https://www.goodreads.com/book/show/853963.Scalable_Internet...

I see he did a talk on it: https://www.youtube.com/watch?v=2WuT2rdLK5A

scarface747y ago

I think of high level talks like these emphasizing timelessness. Technologies change over time and with in different contexts. But the theory remains as a way of thinking abstractly about a problem.

Besides, at the level of jobs I apply to now, interviewers care much more about my thought process than specific technologies.

jhabdas7y ago

I did a screen cast for these guys once. The majority of questions I got in reasons was where's the video.

lalwanivikas7y ago

anybody has a link to the video? google gives me only 3 min preview to the video :(

agacera7y ago

I think it is only available through Safari

j / k navigate · click thread line to collapse

10 comments

matachuan7y ago

hardwaresofton7y ago

HenryROP7y ago

hardwaresofton7y ago

Yeah it was an good talk from the perspective of laying out the landscape and I found myself agreeing with everything, only to find that I was close to the end of the slide deck :(

Is there a blog somewhere where you've written a bit more about it?

1 more reply

carapace7y ago

It's a bit dated but "Scalable Internet Architectures" by Theo Schlossnagle is still good (IMHO)

https://www.goodreads.com/book/show/853963.Scalable_Internet...

I see he did a talk on it: https://www.youtube.com/watch?v=2WuT2rdLK5A

scarface747y ago

I think of high level talks like these emphasizing timelessness. Technologies change over time and with in different contexts. But the theory remains as a way of thinking abstractly about a problem.

Besides, at the level of jobs I apply to now, interviewers care much more about my thought process than specific technologies.

jhabdas7y ago

I did a screen cast for these guys once. The majority of questions I got in reasons was where's the video.

lalwanivikas7y ago

anybody has a link to the video? google gives me only 3 min preview to the video :(

agacera7y ago

I think it is only available through Safari

j / k navigate · click thread line to collapse