Gitlab is down (opens in new tab)

Your impression is correct. We use GitLab.com and notice these incidents as they happen.

The delay in updating status is a result of our Incident Management process [0]. We have a Communications Manager on Call (CMOC) who leads communication throughout an incident. One of their responsibilities includes updating the status page. The slight delay between noticing the issue and updating the status page is a result of the time it takes for the CMOC to get alerted, assess the situation, and write the communication that is shared on the status page.

I'm not sure how the "updated a few seconds ago" messages are generated but I'll try to find out once the incident has been resolved.

0 - https://about.gitlab.com/handbook/engineering/infrastructure...

After you notice I assume you have to declare an incident, get a call going, assess the extent of the issues, get the needed people involved, and then you'd announce on the status page. 13 minutes isn't amazing but it also isn't terrible. Perhaps you have better ways of keeping status pages updated much faster while also not ending up ramping up the posting of false positives.

It doesn't matter if each individual detects the outage because they'll start blame at the local source and move further up the tree rather than assign blame to a full system failure right off the bat. 99.9% of the time it's going to be a local failure affecting the individual.

Also, most alerting systems like check multiple times before declaring a public outage, many times 2 to 3 failures some seconds apart are needed.

yes, you're right.

I was just working on gitlab, so I would say the status page reflected the issue about 5 minutes later

Exactly. That is the whole point. I keep telling that for GitHub since that goes down once a month. [0][1] GitLab SaSS is the same but a self-hosted backup is better.

[0] https://news.ycombinator.com/item?id=29901564

[1] https://news.ycombinator.com/item?id=29379648

Not sure if this is irony (I often don't identify irony as such).

But I fatfingered a lot of self hosted stuff in my time.

It doesn't, but I can fix it as opposed to waiting for their team to do it.

Also at gitlab.com scale the problems they face are very different from a typical deployment.

It is like having maintaining your car and using the train.

On average if you can fix your car (or hire a good mechanic i.e. consulting) you would probably have a better experience than public transport breaking down, that you are powerless to do anything about.

I would rather run a business depending on my car than the train ?

And even if it goes down you might have more options to get it back to work.

They could, if you stuck to the yak shaving full-time.

Well, my GitLab instance at some point started to have its Prometheus eat 100% CPU all the time until I disabled the Prometheus component altogether, so there’s that. A cursory glance at the tracker just now says the issue is still open. That’s the kind of problems you get for self-hosting, it’s not all rainbows and unicorns.

Must be caching issue - it shows "System Wide Outage" for me.