Zuck’s photos from Facebook’s futuristic Arctic data center (opens in new tab)

I think the lesson came in earlier in the NUMA and MPP machines where they kept trying to cram more stuff on boards that were themselves pluggable into the larger system. This convergence has happened from several directions. It's not all the different from the earlier one that started in the 1960's where they fought cost and inefficiency by getting as few components per box sharing as much as possible. Moores Law temporarily reversed it (transistors and memory are free!) then reality check hits that this seems to be a fundamental principle.

My design a while back was to put it all on PCI cards on a PCI backplane. I saw backplanes that basically look like motherboards full of PCI slots that load into racks. I wanted to make the cards nothing but CPU and memory whose software communicated over efficient networking (not TCP/IP) through PCI DMA. My design had IO/MMU functionality in the backplane or PCI cards. At least one card having full-featured stack for management and at least one I/O card for external interface. I figured the backplane itself could be extended for that, too, with a dedicated port like motherboards do integrated GigE. Management and I/O could come through remote DMA over dedicated wires like many servers do with Ethernet so all the PCI slots could be dedicated to compute.

Dumbest thing about Facebook's model is them destroying drives. The first thing to notice, due to Ross Anderson's Security Engineering, is that those pieces still contain a lot of data if they weren't degaussed first. Next is to remember the fastest way to destroy data: use clustered, encrypting filesystems so that secrets never touch the drive. Then, you just have to delete the keys to loose the secrets. No need to trash the drives at all. The crypto can happen at the storage manager or at hardware interface with HW acceleration available for both types. I'm surprised they haven't already built this with all the smart people they have working on big-data stacks.

The number of physical connections (power and numerous data cables) that can be seen in the 5th slide that each machine has makes this seem infeasible for now, but considering the article mentions there's "only one technician for every 25,000 servers" and Facebook's FBAR software (hey, that's up from 20k), it seems like the next step in "the new data center" is having a robot to unrack (and rack) entire machines and bring them to a servicing bay.

But then he wouldn't succeed in cultivating the nickname he always wanted!

Good old Zuck!

nah, a much better 'villain' data center actually exists in Stockholm.

https://www.wired.com/2012/11/bahnhof/

Are you sure it isn't?

These are farms of cheap Linux boxes.

They go out of their way to design and manufacture their own motherboards and equipment[1] to reduce their capital and operational expenses.

While the motherboard and enclosures are probably custom made, the components inside (CPU, memory, etc) are all "commodity".

[1] http://www.opencompute.org/

The hydroelectric dams are taking energy from the water system by slowing its descent. Presumably the friction of faster moving water would heat the riverbeds and surroundings more than the waste heat from the data centers does. The thermal impact should be negligible compared to releasing energy stored in hydrocarbons or atoms. However, pouring a lot of concrete can release enormous amounts of carbon dioxide.

I wonder whether they are using the heat for surrounding buildings. Otherwise it would just be wasted, which is sad.

We were thinking along similar lines. Except I just had an amusing observation about how I'm seeing some people telling me the Arctic might melt due to technological activity and another where critical datacenters are being built in the Arctic. One better hope the other isn't right. ;)

Thx. The slideshow presentation is also crap because the banner ad on top makes the whole page jump around.

And zuckerberg's money went to tax payers in terms of taxes and salary and tax on salary and spending from salary, so a profitable deal overall.

From what I understand, there are a few issues, including:

1) The head has to move further to read the next sector of interest, which is particularly problematic for fragmented data.

2) It is more difficult to manufacture high data density (AKA high-precision) disks in large formats, as some surface defects are cumulative, and get worse as the disk gets larger.

3) Manufacturing defects which occur pseudo-randomly increase proportionally to the surface area of the disk, so the reject rate increases as a square of the radius.

4) Smaller drives can be spun much faster, allowing for higher data rates, as the centripetal accelerations in the disk are proportional to the square of the radius (and I believe the stresses are proportional to the cube of radius).

For these reasons and many others, HDDs have been moving to smaller and smaller form factors.

You know how hard it was to make the lens for the Hubble Space Telescope? Now imagine that you need to make it even more fine-grained in precision and accuracy as well as being electromagnetically within tolerance across that whole surface.

The cost of making high-precision surfaces goes up exponentially with the surface area of each unit. It's much cheaper to make lots of smaller ones than one large one.

The same probably applies for things like the actuator for the heads, etc.

The I/O bandwidth would have to be tremendous. I'm not sure the technology even exists.

Or else split the I/O into a zillion little independent ports.

It seems unlikely considering their acquisition and later shutdown of Parse.