Google cloud outage (opens in new tab)

(status.cloud.google.com)

74 pointsthomassharoon6y ago33 comments

33 comments

26 comments · 6 top-level

tagux6y ago· 10 in thread

"We had a router failure in Atlanta".

WHAT? You kidding us?

Urs Hölzle, technical infrastructure at Google Cloud senior vice president, said, "We're very sorry about that! We had a router failure in Atlanta, which affected traffic routed through that region. Things should be back to normal now. Just to make sure: This wasn't related to traffic levels or any kind of overload, our network is not stressed by COVID-19."

thedance6y ago

Wrong outage.

neonate6y ago

https://twitter.com/uhoelzle/status/1243217659690278912

ocdtrekkie6y ago

Was it like... a hardware failure? If you serve more than 100 people you probably should have redundant routers. Was it a configuration issue that replicated over to multiple devices at least, I hope?

toast06y ago

Have you worked with redundant routers? They certainly reduce the number of outages, but sometimes the hardware (or software) fails in exciting ways that doesn't engage the redundancy, or doesn't engage it properly, and you still get an outage (or you get an outage that wouldn't have happened). Or sometimes, one circuit is out of service for repair or upgrade, and the other circuit is connected to the router that failed. And routing for the AS that travels on that circuit was set not to fallback to transit because the last time that happened, it caused major issues.

I have no specific knowledge of today's events, but this sort of thing happens. You can get the number of incidents down pretty low, but not to zero.

nineteen9996y ago

I remember seeing a Security Analyst for the DC I worked for take down 6 racks worth of Cisco Catalyst 12000 series router hardware once.

They had a HSRP interface set up at the .1 address, and the security analyst set his laptop up with the same static .1 IP address and plugged it in. Instant outage.

ocdtrekkie6y ago

I have. I am just highlighting that the problem surely should be more complex than described. Or that their redundancy for these events was not adequately devised.

1 more reply

packetslave6y ago

yes, because OBVIOUSLY Google is too stupid to know about redundant routers. /s

https://twitter.com/uhoelzle/status/1243259280410554368

"When routers fail cleanly (say, power out) failover is quick, so you never hear about these. This wasn't such a simple case. We have "many" (not just two) routers in Atlanta so it wasn't an issue of missing redundancy."

dodobirdlord6y ago

Networks are harder than everyone thinks. The 2018 CenturyLink outage on the west coast was caused by 1 bad network card that started writing malformed packets.

https://www.geekwire.com/2018/report-huge-centurylink-outage...

AdamJacobMuller6y ago

Not that simple as you sometimes need to manually isolate the faulty hardware and remove it from service.

thanksforfish6y ago

Surely the "100 people" metric is too low although I agree at some point (and certainly Google-scale) a redundant router makes sense.

qmarchi6y ago· 9 in thread

Heyo Googler here.

The problem was a mix between another cloud provider and GCP.

Dare I say, there should be little customer impact as of 13:37 PST.....

The status dashboard is going to be your best idea on information.

svacko6y ago

Is the another cloud provider AWS? I could see tons of connection timeoutes between GCP & S3/Elasticsearch service.

Hope everything is resolved now for good.

judge20206y ago

Seems AWS, connection to gmail's smtp relay also started timing out.

gigatexal6y ago

Oh man I had no idea the big cloud providers have dependencies on other clouds like this.

lima6y ago

They do not, according to the dashboard, this issue merely affected connectivity between GCP and other cloud providers.

There was a different outage yesterday, which has nothing to do with the one discussed in this thread.

dodobirdlord6y ago

Given how much trans-continental/trans-oceanic network cable the major cloud providers own, they almost certainly have special trans-cloud network traffic infrastructure. Especially since so much of "The Cloud" is within a few 10s of square miles in a field in Virginia. I can easily see how one provider could majorly disrupt another provider by accidentally breaking inbound traffic on one of those links.

qmarchi6y ago

The bigger issue is that there's a lot of customers where they have split cloud deployments, which means the customers hurt even if they are stable within the clouds themselves.

1 more reply

gigatexal6y ago

Yeah, I see that now. Makes total sense.

the-dude6y ago

This can't be real.

unbeli6y ago

[removed]

1 more reply

kgraves6y ago· 1 in thread

This is extremely concerning as somebody looking to move or build on top of GCP for the long term. I wonder why anyone would choose GCP if outages are occurring on a regular basis.

pgodzin6y ago

Any evidence they happen more frequently that the other clouds?

1 more reply

nammi6y ago

We were seeing timeouts in east-1. I don't know what "normal" looks like, but Pingdom's map seems to show the whole east coast as affected https://livemap.pingdom.com/

svacko6y ago

yeah, our GKE pods running in us-east1 were dying ~90minutes ago like crazy... hope they are gonna resolve this soon. not the luckiest day for Google, nor us

x__x6y ago

I was bummed out when Siteground moved all their cloud accounts over G, without telling their customers beforehand

j / k navigate · click thread line to collapse

33 comments

26 comments · 6 top-level

tagux6y ago· 10 in thread

"We had a router failure in Atlanta".

WHAT? You kidding us?

thedance6y ago

Wrong outage.

neonate6y ago

https://twitter.com/uhoelzle/status/1243217659690278912

ocdtrekkie6y ago

toast06y ago

I have no specific knowledge of today's events, but this sort of thing happens. You can get the number of incidents down pretty low, but not to zero.

nineteen9996y ago

I remember seeing a Security Analyst for the DC I worked for take down 6 racks worth of Cisco Catalyst 12000 series router hardware once.

They had a HSRP interface set up at the .1 address, and the security analyst set his laptop up with the same static .1 IP address and plugged it in. Instant outage.

ocdtrekkie6y ago

I have. I am just highlighting that the problem surely should be more complex than described. Or that their redundancy for these events was not adequately devised.

1 more reply

packetslave6y ago

yes, because OBVIOUSLY Google is too stupid to know about redundant routers. /s

https://twitter.com/uhoelzle/status/1243259280410554368

dodobirdlord6y ago

Networks are harder than everyone thinks. The 2018 CenturyLink outage on the west coast was caused by 1 bad network card that started writing malformed packets.

https://www.geekwire.com/2018/report-huge-centurylink-outage...

AdamJacobMuller6y ago

Not that simple as you sometimes need to manually isolate the faulty hardware and remove it from service.

thanksforfish6y ago

Surely the "100 people" metric is too low although I agree at some point (and certainly Google-scale) a redundant router makes sense.

qmarchi6y ago· 9 in thread

Heyo Googler here.

The problem was a mix between another cloud provider and GCP.

Dare I say, there should be little customer impact as of 13:37 PST.....

The status dashboard is going to be your best idea on information.

svacko6y ago

Is the another cloud provider AWS? I could see tons of connection timeoutes between GCP & S3/Elasticsearch service.

Hope everything is resolved now for good.

judge20206y ago

Seems AWS, connection to gmail's smtp relay also started timing out.

gigatexal6y ago

Oh man I had no idea the big cloud providers have dependencies on other clouds like this.

lima6y ago

They do not, according to the dashboard, this issue merely affected connectivity between GCP and other cloud providers.

There was a different outage yesterday, which has nothing to do with the one discussed in this thread.

dodobirdlord6y ago

qmarchi6y ago

The bigger issue is that there's a lot of customers where they have split cloud deployments, which means the customers hurt even if they are stable within the clouds themselves.

1 more reply

gigatexal6y ago

Yeah, I see that now. Makes total sense.

the-dude6y ago

This can't be real.

unbeli6y ago

[removed]

1 more reply

kgraves6y ago· 1 in thread

This is extremely concerning as somebody looking to move or build on top of GCP for the long term. I wonder why anyone would choose GCP if outages are occurring on a regular basis.

pgodzin6y ago

Any evidence they happen more frequently that the other clouds?

1 more reply

nammi6y ago

We were seeing timeouts in east-1. I don't know what "normal" looks like, but Pingdom's map seems to show the whole east coast as affected https://livemap.pingdom.com/

svacko6y ago

yeah, our GKE pods running in us-east1 were dying ~90minutes ago like crazy... hope they are gonna resolve this soon. not the luckiest day for Google, nor us

x__x6y ago

I was bummed out when Siteground moved all their cloud accounts over G, without telling their customers beforehand

j / k navigate · click thread line to collapse