How Netflix deploys 2 Terabits in 12 hours [pdf] (opens in new tab)

(nanog.org)

14 pointsaditya13y ago3 comments

3 comments

3 comments · 2 top-level

bradhe13y ago· 1 in thread

Amazing to see the on-prem chops of a company who is so cloud-oriented.

Yes, I thought Netflix was completely cloud located. I realize now that I've been naive in my thoughts. It looks like they developed their own CDN, which makes me wonder what it was about the existing solutions they weren't happy with. Too slow? Too expensive? Not enough control?

Ah, I bet its the control one. The slides mentioned that they actually knew what the customer wanted to watch before the customer did. This allows them to push content down to a local node in advance of the customer requesting it. Wow, that's amazing and scary all at the same time.

donavanm13y ago

i didnt catch it in the pdf but a single deployment is supposed to push 200tb/s. Id ball park the capital cost aaround 1.25M USD. the original 4u boxes do/did ~9gb/s of prod traffic each. peak/lab util was about 15gb/s. IIRC two of these boxes could cache nearly the entire library for a region.

The interesting bit is the new 2u hosts used specifically for hot content. only using 3x 10gig interfaces because the bus is too narrow. im guessing these hosts are pushing 20-24gb/s at peak. theyll have to be using a zeroish copy like sendfile() to just dma everything from disks to nic. looks like the 14 ssds can push 450-500MB/s, so theres the bus limit again. read scheduling will probably suck at that rate. but id wager that most of the bits served are super hot content being dmad straight from memory pages.

j / k navigate · click thread line to collapse