Ask HN: Anyone else having bitflips in Azure Network traffic?

after a full day of searching for a possible cause, someone from microsoft suggested that they try to move the express route gateways (which are seemingly VMs inside of Azure) to a different hypervisor. For now it seems like the first gateway was flipping bits and on new hardware it is running normal again... That was a very interesting day troubleshooting wise

yeah, that's what we did. We captured it at the last place in our network (a router) and then compared the tcp payloads to the packets that we received in azure in our firewall. Next step was to get the dumps from as much inbetween as possible, it seems that is only the edge router of Azure and our express route gateway. After looking at the pcap of the azure edge myself I verified that the package is still okay there, the MS support verified it's broken in the express route gateway, so it has to be flipped somewhere inside their Azure Germany network stack. They are still searching where ... :)

A much more common source of bitflips is from heat. If you run a desktop computer with more than about 32GB of RAM you probably have bitflips from the heat alone- which is part of why DDR5 adds a weaker form of error correction as standard.

Bitflips in network packets are often stuck bits in networking equipment. Replacing the offending equipment tends to be easy, once it's identified. But you can only do so much when you have no control.