Google Cloud Europe service disruption (opens in new tab)

2015 Chennai (South India) Floods. It was the flood of a century. [1]

Our DC was intact, but the building and access was cut-off. We lost the backup diesel power generators in the flooding. Of course, grid power was cut-off.

Our DC operating team managed to shutdown all the servers and racks cleanly before UPS power was completely drained. The 4 engineers and 2 security guards then swam out of the compound in chest high waters. (I am not kidding).

When the rains subsided and the flood waters receded after a couple of days, we had to plan the restart. The facility still had to be certified by health and safety, but we needed to get the datacenter back up.

A secondary operations site that would remote-connect to the DC was brought up in 1 week since we estimated the rains to potentially continue for a few more days and cause interruptions. But the critical item for the plan to work was getting a new backup power setup. We rolled in a truck-mounted diesel generator and positioned it in the highest point in the campus (also close to our building tower that had the DC) and ran power cables to it (we had to source this and it was a challenge to do it with the time crunch and the rains).

We moved staff to other cities by bus (airport was shutdown) as part of our recovery plan, but we still needed connectivity to our DC for some of the critical processes.

Long story short, it worked.

I'll never forget the experience and the scars from this war story.

[1]: https://en.wikipedia.org/wiki/2015_South_India_floods

I once was a customer of a DC who's roof drainage was clogged, turning it into a lake after a couple of rain storms. It then proceeded to rain inside the DC as the roof started to leak from all the pressure.

"Servers are down, I'll head over to the DC" turned into "Um... it's raining _in the DC_. Get me some tarps and get us cut over to the backup in the office".

Ah, the glory days of running out of a single co-lo across the parking lot with our "backup site" being a former broom closet.

The machines are not industry standard stuff, and they don't auction, they destroy for customer security. See here: https://www.datacenterknowledge.com/google-alphabet/robots-n...

I'm not sure what the disk encryption story is in Google Cloud but I'd rather it didn't end up on Ebay. Mind you, "flooded" covers a wide range of possibilities and a surprisingly small amount of water ingress would trip a breaker while leaving the racks in good order.

Better than when Planet's DC actually exploded [1].

Restoration is hard when health and safety are in question. Good luck to these ops folks <3

[1] https://www.datacenterknowledge.com/archives/2008/06/01/expl...

A long time ago, one server room (located in the basement of the university building) of SPB-IX was flooded. It was a fun day for engineers whom unplugged survived equipment standing knee-deep in water

It was before dam (1) was built and floods were a huge problem in SPB

[1]: https://en.wikipedia.org/wiki/Saint_Petersburg_Dam

Umm thoughts and prayers? It's not as if their house is being washed away :) They just have a busy day at work. Keeps things exciting :P

I doubt they would let anyone have access to their hardware. There is a ton of proprietary stuff in there

> but are they going to auction off the flooded hardware?

I wonder how many inches/feet we're talking here? The hardware on the top (unless it experienced electrical short) is most likely fine?

Likely not. It’s also not Google’s first dc flood/water intrusion causing a GCP incident.

Also, consider everyone either automatically or manually trying to make up for the lost capacity in eu.

Every customer of affected region try to restore data/compute in other regions. It's quite known and expected issue in case of region loss.

Cloud Console is having issues related to the outage in europe-west9

> Customer using Cloud Console globally are unable to open and view the Compute Engine related pages like: Instance creation page Disk creation page Instance templates page Instance Groups page

https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPf...

Same – was unable to create new VMs in all regions between 7:15am and 11:41am UK time. Not limited to France.

My impression, from reading the docs around Google's "premium-tier network routing" — and just from the "feeling" of deploying GCLB updates — is that when you're configuring "a" Google Cloud Load Balancer, you're actually configuring "the" Google Cloud Load Balancer. I.e., your per-tenant virtual LB config resources, get baked down along with every other tenants' virtual LB config resources, to form a single real config file, across all of GCP (maybe all of Google?), which then gets deployed to not only all of Google's real border network switches, across all their data centers; but also to all their edge network switches, in every backbone transit hub they have a POP in.

(Why not just the switches for the DC(s) your VPC is in? Because GCLB IP addresses are anycast addresses, with BGP peers routing them to their nearest Google POP, at which point Google's own backhaul — that's the "premium-tier networking" — takes over delivering your packets to the correct DC. Doing this requires all of Google's POP edge switches to know that a given GCLB-netblock IP address is currently claimed by "a project in DC X", in order to forward the anycast packets there.)

To ensure consistency between deployed GCLB config versions across this huge distributed system — and to avoid that their switches constantly being interrupted by config changes — it would seem to me that at least one — but as many as four — of the following mechanisms then take place:

1. some distributed system — probably something Zookeeper-esque — keeps global GCLB state, receiving virtual GCLB resource updates at each node and consensus-ing with the nodes in other regions to arrive at a new consistent GCLB state. Reaching this new consensus state across a globally-distributed system takes time, and so introduces latency. (But probably very little, because the resources being referenced are all sharded to their own DCs, so the "consensus algorithm" can be one that never has to resolve conflicts, and instead just needs to ensure all nodes have heard all updates from all other nodes.)

2. Even after a consistent global GCLB state is reached, not every one of those new consistent global states get converted into a network-switch config file and pushed to all the POPs. Instead, some system takes a snapshot every X minutes of the latest consistent state of the global-GCLB-config-state system, and creates and publishes a network-switch config file for that snapshot state. This introduces variable latency. (A famous speedrunning analogy: you can do everything else to remediate your app problems as fast as you like, but your LB config update arrives at a bus stop, and must wait for the next "config snapshot" bus to come. If it just missed the previous bus, it will have to wait around longer for the next one.)

3. Even after the new network-switch config file is published, the switches might receive it, but only "tick over" into a new config file state on some schedule, potentially skipping some config-file states if they're received at a bad time. Or, alternately, the switches might themselves coordinate so that only when all switches have a given config file available, will any of them go ahead and "tick over" into that new config.

4. Finally, there is probably a "distributed latch" to ensure that all POPs have been updated with the config file that contains your updates, before the Google Cloud control plane will tell you that your update has been applied.

No matter which of these factors are at fault, it's a painfully long time. I've never seen a GKE GCLB Ingress resource take less than 7 minutes to acquire an IP address; sometimes, it takes as much as 17 minutes!

And while there's definitely some constant component to the time that this config rollout takes, there's also a huge variable component to it. At least one of #2, #3, or #4 must be happening; possibly multiple of them.

---

You might ask why load-balancer changes in AWS don't suffer from this same problem. AWS doesn't have nearly as complex a problem to solve, since AFAIK their ALBs don't give out anycast IPs, just regular unicast IPs that require the packets be delivered to the AWS DC over the public Internet. (Though, on the other hand, AWS CDN changes do take minutes to roll out — CloudFront at least distributed-version-latched for rollouts, and might be doing some of the other steps above as well.)

You might ask why routing changes in Cloudflare don't suffer from this same problem. I don't know! But I know that they don't give their tenants individual anycast IP addresses, instead assigning tenants to 2-to-3 of N anycast "hub" addresses they statically maintain; and then, rather than routing packets arriving at those addresses based purely on the IP, they have to do L4 (TLS SNI) or L7 (HTTP Host header) routing. Presumably, doing that demands "smart" switches; which can then be arbitrarily programmed to do dynamic stuff — like keeping routing rules in an in-memory read-through cache with TTLs, rather than depending on an external system to push new routing tables to them.

What’s more obscure and less tested than figurative plumbing? Literal plumbing!

It's still DNS

Droplets Nuking Servers

We just use a simple cloud function for that.

GCP has multiple zones in the same physical building. Not all cloud providers have distinct physical buildings for each Availability Zone.

AFIK all zones (a, b, and c) have been reported to be down. I'd love to understand ehat happened.

Probably some dependencies they did not plan for

It happened at GlobalSwitch Clichy, near Paris. From what I gathered from a french forum[1], it started with a flood and then a fire. No rooms have been affected, apparently.

[1]: https://lafibre.info/datacenter/incendie-maitrise-globalswit...

Yes, it would be less confusing to say “down”.

GCP doesn't operate the same way as Google consumer products. We are a paid customer for over 5 years and I also have only good things to say about GCP and their support

TBH, GCP isn't operating like Google does.

I'm a long time customer and have only good things to tell so far.

A memorable observation from idlewords is that Googlers will organize for better conditions for Googlers but they never organize for better conditions for their users.

that’s how GCP does zones, firewalled off with separate networks/power in the same physical location.

At least Paris is red, what's the last time you saw more than a green dot with a little exclamation mark on the AWS status page?

There's network alert on the whole of EU because one of the regions in EU is out.

A case of "need to drill down"