For smaller users I can imagine this sort of thing happens pretty regularly with every cloud, even smaller ones.
Please reply if you had a TAM and were spending that much! I'd be personally interested to hear that was the case.
We first had no idea what was going on. GCP support just told us "it seems you deleted your CC". Eventually, we figured out what happened.
Set up a new payment profile and started migrating our GCP projects to it. Eventually had to create multiple of them, because there is an arbitrary quota of how many projects you can have per payment profile (~4), and support told us it would take days to increase it.
Fortunately, all our data was still there. However, support had initially told us that it's "all gone".
Doing the same with service accounts as you can do with a personal account takes weeks before you can even get started, and informs the whole management chain what you're doing, which means it informs essentially every manager that could complain about it of exactly the right time to complain to be maximally obstructionist about it.
Or to put it perhaps less ...: using service accounts requires the processes in the company to be well thought out, well-resourced with people who understand the system (which this issue shows: they don't even have those at Google itself), well-planned, and generally cooperative. Often, there will be a problem somewhere.
> Google Cloud accidentally deletes UniSuper’s online account due to ‘unprecedented misconfiguration’
which is a lot more alarming.
I've heard of sudden and unwarranted bans before, but never an accidental deletion of a customer who they only just convinced to migrate to Google Cloud last year!
Just incredible that their entire account, spanning two geographies, was entirely deleted.
Large financial pension fund with advertised $124 billion in funds under management so not some toy cat gif startup has account deleted accidentally by Google. That can very easily wipe out a company using cloud as cloud vendors advertise you to. From the article it sounds like they are lucky they had offsite backups but still potential for data loss and restoring offsite backups likely a task in itself.
It's a major incident, I feel for the ops team who'll be working under massive pressure trying to get everything back up.
Lesson learned: back your GCP data up to a real cloud like Azure or AWS.
It reminds me of "I use box/iCloud/some-other-cloud-drive-service so don't need backups", but don't understand the model is "I deleted/broke the data on my machine, and then synced that data loss to every machine"
that means not on the same cloud infra as the rest of it, but also means different creds and different access paths.
if a massive client came and said "hey our thing is completely broken", then there would have been a war room of SREs and SWEs running 24/7 over two continents until it wasn't.
https://www.googlecloudpresscorner.com/2023-06-05-UniSuper-A...
This article doesn’t challenge the assertion by Google that this is a once-off, which is really sloppy journalism.
Luckily they were able to restore from backups, but it took a full day and there was still significant data loss (the delta since last backup).
Since those servers were used to host the companies own cloud managed offering it ended up affecting all of their own cloud customers as well.
