undefined | Better HN

0 pointsbcantrill2y ago0 comments

Even more galling about that 20-30%: it is really, really hard to measure. Believe it or not, you can't measure the draw even in fans in a traditional rack-and-stack server enclosure (that is, they are either not on their own voltage regulators or not regulators that have PMBus support, or the PMBus support is not plumbed through the BMC -- or all of these). And these aren't the only fans! In a traditional rack-and-stack, each server has AC power supplies -- and these supplies also have fans! These supplies are often entirely dark: they don't have PMBus support at all -- and definitely don't provide insight into how much power they are dissipating in their own fans. (Just empirically: if you don ear protection and walk up to your local 1RU/2RU server under load, those power supply fans will be cranking, and you will feel a hot, stiff breeze from just the power supplies alone!) This is where all of these effects start to reinforce one another: the per-server conversion is stupidly inefficient and the geometry is stupidly inefficient (power supplies are forced to have even smaller fans than the servers!) -- all of which results in more draw, none of which is observable.

The Oxide rack is really the opposite in all of these dimensions: we do our AC/DC conversion in a power shelf and run DC on a busbar; the geometry of the sleds allows for very efficient 80mm fans; and (importantly!) every regulator is observable through PMBus and plumbed through our service processor and management network -- so you can see where it all goes.

And if you think I'm worked up about this problem, you should talk to sufferers of at-scale rack-and-stack who have been burned by this. ;)

0 comments

gary_02y ago

I worked on designing computerized machinery in the past, and having sensors for as much stuff as possible just seemed like a good idea. We had quite a few current sensors for various components. Then we could do stuff like log in to the unit remotely when a customer had an issue, and diagnose based on if something was using too much current or zero current, and that beats having to send out a technician to go poke and prod. There are a lot of uses for that kind of data. And you've got a CPU right there to send it to.

> These supplies are often entirely dark

Good grief. What happens if a fan dies? The whole rack mysteriously shuts down? Or does the power supply just turn into a Minecraft lava block?

bcantrillOP2y ago

In the traditional rack-and-stack, if a power supply fan dies, the power supply will shut down -- which is why a server has two of them. (And indeed, in commodity servers, the fan in the power supply is what is most likely to fail.) All of this becomes a bit of a self-fulfilling prophesy: because there are so damned many of them, the power supplies in rack-and-stack servers are really not great -- which means they are more likely to fail. This is all sidestepped by having conversion consolidated in the rack, running high voltage (54V) DC up and down the busbar that is converted by each sled. (That converter throws off heat -- but can then leverage the high quality fans in the sled.)

j / k navigate · click thread line to collapse