Facebook open sources its servers and data centers (opens in new tab)

This isn't a response to Greenpeace, this is a response to the real costs that it takes to run compute infrastructure at the scale of Facebook, Amazon, Google, etc.

If a datacenter can run at 1.07 PUE (as Facebook's new datacenter does), then it directly translates to massive power cost savings. Environmental efficiency just comes along with it.

This is true. However, the services that startups use to host their servers (Amazon Web Services, Rackspace, etc.) could adopt these practices in new datacenters because the specs are open. This could lower costs for the startups that use these services, and be good for the environment.

Seems like everyone wins.

The question comes up a lot when you're building very large scale applications. When does it make sense to have your own infrastructure vs using someone else's (like Amazon's).

If you're running 'one' thing (like say a hadoop farm) and you can optimize out the things you don't need, there can be a pretty durable benefit in building your own machines.

People like Rackable, HP, Dell, or IBM who sell servers need to build them able to do 'anything' you might want, in order to do that cost effectively they often put things on the mother board (lowest marginal cost) which are perhaps not useful in all cases. However, when you're using lots of machines you have to power and cool those unused sound chips and USB hub chips, and may firewire ports that aren't really all that useful to a web app.

I talk about it as 'rack level' blades, basically motherboards on a cookie sheet that only have network and storage interfaces. Taking away a size constraint makes building them a lot easier (you don't need a custom backplane for example, you just plug cables in)

The only other motive it suggests is garnering as much publicity for their innovation/company as they can. Facebook is notorious for being able to magnify a small announcement into a big event and reap the dividends -- remember the announcement of a redesign via 60 minutes? If you can get media and influences to come pay attention to cool stuff you're doing, why wouldn't you?

blip.tv builds and maintains our own servers and data centers. We started before the Cloud though :)

It could be. I always thought that an interesting way to compete with Google would be to index the web and then sell map/reduce access to that index.

Because Google could never benefit from Facebook's designs too? Google does a lot of things bespoke, but now they have a model to compare with that is using more off-the-shelf tech. That can only help them in the buy-vs-build decision process.

I think you're going overboard in thinking that everything the big players do is about killing the other guy. Sometimes they just want to reduce costs. If you decide that part of your infrastructure isn't strategic to own, it always makes sense to be open.

A proliferation of scalable data centers could also enable the next generation of social networking sites ... I don't see why it's limited to helping Google's competitors and not Facebook's.

That is a good point, that you have raised. But I wonder, why well established companies like microsoft, apple etc. are not doing this.

One of the most important fact form this article which is likely to be overlooked is : "the Prineville facility runs at 85°F with a 65 percent relative humidity". Running at such a high humidity + Temp combination is impressive esp when you look at the money saved in environmental costs.

Any idea on how much would it cost to produce a small quantity of those boards? At which scale does it starts to make sense to use a custom made motherboard instead of a off-the-shelf PC?

This "sorry ARM" stuff is mostly just the media's need to turn every story into a horse race. But there is some technical detail here: Facebook said they're not ever willing to use 32-bit and ARM won't have 64-bit for years. Also notice that Facebook's servers are "conventional" 2-socket with lots of DIMM slots, not 1-socket "microservers".

No they haven't. The closest you would come would be a tegra2 which is dual core and I believe the omap4 is dual core as well. Though iirc, you can't currently use both cores in Linux. Freescale have announced the imx6 which should have quad, dual and single cores but availability won't be until q4 (though they are pushing hard to get it before then I think.) You still are currently limited with ram on an ARM machine which I think is the biggest hinderance to going in to servers. Presently almost all server software works on an ARM machine with a few exceptions like spidermonkey (so no running Launchpad) but plenty of people use them as personal servers.

Amir Michael who worked on the hardware team said, "The utility provides us with AC voltage which we then convert to DC very close to the motherboard in the power supply. Our goal was to bring the high voltage as close as possible to the load to minimize IR losses. We could do an efficient AC –DC conversion in the rack and then distribute DC to the individual servers but that would mean several feet of low voltage conductors which would be made from large copper bars and a would have higher IR losses."

The UPS doesn't need an inverter - the PSUs have a DC in.

I'm also puzzled why they don't go all-DC. This thread doesn't appear to be getting any traction here. Is there a HN for datacenter geeks?

> In August 2008, Intel conducted a 10-month study to assess the effectiveness of using only outside air to cool a data center. The temperature range was 64°F to 92°F. Humidity varied from 4% to over 90% and changed rapidly at times. No increase in server failure was observed.

http://www.energystar.gov/index.cfm?c=power_mgt.datacenter_e...

A certain amount of humidity helps to prevent issues with static electricity. And unless you're pouring water directly across electrical components, there's really not a problem.

I don't know whether 65% humidity is going to hurt a server, but evaporative cooling doesn't work well if the relative humidity is high. So maybe it's a good idea in dry climates, but I can't see it being very effective in the midwest of the US in July/August.

But it's great for all those plants they have hidden behind the false rack fronts.

A list of Facebook's open-sourced applications and their major contributions to other open-source projects can be found here: http://developers.facebook.com/opensource/

"Please take a look, tell us what we did wrong and join us in working together to make every data center more efficient."

Mad Props.

I'd say it's more of a refinement than something new. I haven't looked at much more than what they're saying in the video, but it boils down to:

- new featureset mobos: stripped down, no parallel ports among other things I'm sure. there looked to be some SATA/Mini-USB type sockets. - fancy new power supplies - the triple racks don't seem to be anything special at face value, convenient maybe, but with the power features above two of these triple racks are serviced by one battery rack. So, you'd need, oh, 400+ servers to take full advantage at this level.

NOW HOW MUCH WOULD YOU PAY?

- Fancy power distribution and cabling and stuff, mostly part of the racks, possibly involving wacky connectors on the compute side. For smaller operations this would be like having a 60-outlet power strip.

- 95% efficient from plug to party-time. This helps with the bills.

- Using the room/building as the cooling mechanism. While this has likely been done before in some manner (at least an intentional use of stack-effect in HVAC), that weird slot that one of the boxes gets pushed into makes me think they have some kind of sealed thingy from floor to ceiling that interfaces with the racks, basically using the building cooling to push in and suck out air forced through the racks. at a basic level it's all about CFMs, after all. This could also be done with forced current from front to back, or vice versa.

whatever is going on here, you first have to get to the "your own server room" part of business. this can be had at smaller companies, too, but for the room-cool ducting you'd need to cut into walls and stuff in the server room to pipe that stuff in. Spendy.

So, this leaves the smalltimer to save $15 on their monthly power bill by using new rack servers that don't need or have an NVidia GeForce 9000+ and 10-drive RAID. Stripped down BIOS and maybe no more IDE support, that kind of thing. Hot Rod rack servers of the skeleton/pure-compute variety, not the AlienWare one.

I can give you an interesting data point: our power bill is only about $600/mo (2.5KWh), our bandwidth bill is about $3,500/mo (1.2Gbps). So power is a smaller consideration.

Pretty standard in telecoms. Never quite understood why Google went per server.

I found that interesting too, thought it was a joke at first.

The only details I was able to dig up are in the Data Center specs [1], and they're pretty brief:

> Energy-efficient LED lighting is used throughout the data center interior. / Innovative power over Ethernet LED lighting system. / Each fixture has an occupancy sensor with manual override. / Programmable alerts via flashing LEDs.

I wondered what the justification for PoE lighting could possibly be, sounds like all the lighting is also functional as instrumentation.

Anyone know more?

[1] http://opencompute.org/specs/Open_Compute_Project_Data_Cente...

Way ahead of you: http://www.google.com/corporate/datacenter/efficiency-measur...

Are you serious? Facebook open sources so much of their code.

Check http://developers.facebook.com/opensource/

Each server rack sits atop the shallow grave of a HN commenter who said something bad about Facebook.

As much as I like James and his commentary, I have this thing about primary sources: http://opencompute.org/