undefined | Better HN

0 pointsjeffbee5y ago0 comments

miclosoft.com is only one bit away from microsoft.com. Used to see these problems all the time when I worked on gmail.

At Google even with ECC everywhere there wasn't enough systematic error detection and correction to prevent the global database of monitoring metrics from filling up with garbage. /rpc/server/count was supposed to exist but also in there would be /lpc/server/count and /rpc/sdrver/count and every other thing. Reminded me daily of the terrors of flipped bits.

0 comments

thu21115y ago

Ahaha. Reminds me of when I worked there. One day a large service tanked in some datacenter because BigTable replication in that location just stopped. Digging in, it turned out the BigTable should have been replicating from YQ but had started trying to use QQ instead, which didn't exist. Q being one bit away from Y. Or it was something like that, I don't remember exactly. There'd been a bit flip in the exact part of memory that contained the name of the database cluster to replicate from!

jeffbeeOP5y ago

Beautiful.

As long as we're swapping war stories there was once a frontend server at that company which died of bit flip related causes but just before it did it managed to charge some product group for 2^63 bytes worth of network traffic in the internal quota system, which set off every budgetary alarm that capacity planners had.

j / k navigate · click thread line to collapse

0 pointsjeffbee5y ago0 comments

miclosoft.com is only one bit away from microsoft.com. Used to see these problems all the time when I worked on gmail.

0 comments

thu21115y ago

jeffbeeOP5y ago

Beautiful.

j / k navigate · click thread line to collapse