undefined | Better HN

0 pointsOperyl8y ago0 comments

Sounds like a question you might be better off asking their sales team.

0 comments

3 comments · 1 top-level

eropple8y ago· 2 in thread

You realize their team is in this thread, right?

Yep, and they didn't answer. It's a question that sales would be able to answer. No need to downvote me for a legitimate response, right?

brianwski8y ago

> they didn't answer

That was a strange oversight, sorry! Here is an explanation of our architecture regarding downloads:

Backblaze files are served up from a set of load balanced "download servers". Think of the download servers as "caches". The FIRST time you request a URL, the download server has to go fetch the file from the vault. The file must be reconstructed since it is striped across 20 separate pods (as soon as any 17 of these pods respond, it is enough to reconstruct the file).

The download servers have 10 Gbit networking and a fast SSD. The file is cached on the local download server's fast SSD. So the SECOND time you fetch the URL from one download server it can go really fast.

Ok, so the download servers don't talk to each other and there isn't any IP address affinity. So take the example where we have 20 download servers serving files from the area your one file comes from. If there is a lot of "hot" activity on one file (like 500 threads fetching the same one file in the same minute) then each download server fetches it ONCE from the vault, which means out of the 500 threads, about 20 early threads will get "lower performance" as the vault reconstructs the file, so maybe 2 - 5 Mbits/sec for the first file fetch on each download server? Then 480 threads will be fetching the file directly from the 10 Gbit/sec fast SSD cache servers. If the threads are spread evenly across the 20 download servers, so each download server serves up around 24 threads, which is easy.

Now some disclaimers: Backblaze has a TON of experience storing and writing files into our datacenter. However, we are still learning about serving files OUT of our datacenter. Recently we realized the SSDs in the download servers were a little small and files weren't getting cache hits enough. In other words, we could deploy fewer download servers if we gave them more local SSD space. So we're fixing that.

We also like to listen to customers and work with them. So if a knowledgeable customer experiences slow downs or they see an anomaly we can take a look and figure out how to improve.

j / k navigate · click thread line to collapse