GitHub was down again (opens in new tab)

As a CTO responsible for an often failing eCommerce website that lost millions when down and which I took over, I fell in the same trap of trying to sweep things under the rug.

Until I decided to no longer do that and my life improved considerably.

What's the point in having a status page if it's a political artifact? In that case, it serves zero customer value.

3 words: Service Level Agreement

I've caught a big cloud provider not reporting a degraded service, I assume they knew but politics and $ come in and it's easier to just gaslight everyone. I get it, but my frustration is worth loosing a trailing 9.

I think there should be some 3rd party continuously testing APIs. Degraded states are downtime!

Honestly I think this can be true at any size organization. Small startups often take the approach of "let's hope no one noticed while we try to fix it", it's just that they have fewer users to notice so it's more likely to work.

Just a friendly correction: `SNAFU:' Situation Normal: All Fucked Up

Paying Saucelabs customer here.

It's gotten more reliable over time (especially selenium events being dropped on the floor causing tests to stall and fail), but I used to have to babysit it quite a bit and there were quite a number of times where IE instances just would not spool up (with a multiple minute timeout set). Sometimes it was a one-shot thing, other times it went on for hours.

During these incidents the average allocation times listed on their status page would double for Windows VMs (I don't recall the exact numbers but they were on the order of 10 seconds vs 5) but nothing would be red, and most of the time nothing ever did go red.

And that's what you get for using averages for things and divide infinity by n improperly.

This is weird because we haven't encountered any real issues with agents in Azure DevOps pipelines. I think we maybe had a single downtime in last 6 months. They recently removed .NET Core 2.2 SDK without any notice and broke our builds but that's another thing.

Honestly I've had the opposite experience. With so many community actions available, I've had little trouble finding anything I could dream up. Sure, some of the actions features are a little immature but they are improving with time. The uptime issues are annoying and I feel like the lack of transparency is not helping that situation, but as far as CI solutions go, I feel like my move to actions has been a great way to get up and running with far less effort than other offerings like Code Pipeline.

Gitlab is probably at the top of the list of candidates if we go down this road. I don't necessarily need it to be in the same binary as my devops tools, but certainly no further than localhost or another machine on the same network.

Gitea has a lot of those features already: https://docs.gitea.io/en-us/comparison/

Issues with a green 'X' means they link to a feature on their issue tracker.

And, as far as I know, they are working on integrating CI/CD right now. They already have support for other non-integrated CI/CD platforms: https://docs.gitea.io/en-us/ci-cd/

For those that don't know, Gitea forked from Gogs a while back and they are very much being developed with two different philosophy. If you take take a look at the active contributors for Gitea and Gogs, you can tell how much they differ now.

https://imgur.com/ZExNVV4

https://imgur.com/v0fGXgv

There are two active contributors for Gogs, while Gitea has 27. Note, the number of contributors can't tell you if one has higher quality or not, I just wanted to point out the difference in development philosophy.

Given that Gitea has significantly more active developers working on it, we can probably assume it can add functionality faster than Gogs though.

I cannot say if this comment is being sarcastic, but for the record I found it hilarious. Thanks :)

Hilarious. GitHub has their fingers on the pulse of what developers and their customers really want. Not stability, but pretending to do things to help POCs through mindless censorship.

> But that could be all a part of coordinated effort to be more transparent about their service status, an effort that should be applauded.

Microsoft could be pushing for transparency. Or people are more relaxed about transparency now that GitHub has its exit. How long did GitHub know they were looking to be acquired? Maybe this analysis should look at a longer time interval..

From the first two graphs it looks like they are a lot less liberal about using "down" instead of "warn".

Currently it does, indeed. From my Slack logs, at 15:00 UTC I noticed problems. I'm pretty sure that message is manually created, at least 41 minutes after the fact.

Github had been in growth mode up until the acquisition. If Github stops being the nirvana for developers that it once was, it will be another dark mark in the history of MS acquisitions. Moreover, considering how sentiment influenced the stock market is at the moment, continued news of one of their products having outages could easily shed a considerable amount of Microsoft’s valuation, ~1%. The say stocks only go up nowadays, but when everything goes up, whoever grows at the slowest rate is really going down. I‘d assume that the Microsoft executive team won’t be happy with the new perception of Github.

Wow, it used to be so much more detailed! I get they probably can't have that level of "casual" disclosure now that they are so big, but man the current status updates just feel so... useless and unhelpful in comparison.

I think they switched to Atlassian StatusPage[1] a while ago.

[1] https://www.atlassian.com/software/statuspage

I posted this on another thread. If you only want the commits, something like this works,

    ssh user@git.example.com
    mkdir project-1.git
    cd project-1.git
    git init —-bare
    exit
    git remote add alternate user@git.example.com:project-1.git

All you need is SSH

That's fine for your code repository.

However git has more: - Bug tickets - PR - Wiki - many projects use their githubpages as primary homepage - newly GitHub actions

And then: Often collaborators are only known and identified by their GitHub handle. Running an own server requires some mechanism to identify them again and creating a way to handle their access credentials (ssh key etc.)

Moving a mildly successful project isn't easy. Good if more people plan for that eventuality, even if they stay on GH for the time being.

I’ve had this before. Closing and reopening the PR does seem to do the trick sometimes.

Why not MercurialHub?

Yes, I was. However, I was pushing to master of a fork. Maybe thats the reason.

> Thats why the self hosted options are there -- and why GitLab has a competitive advantage in this sense.

GitHub has GitHub Enterprise

> Cloud solutions are great -- however they have a golden rule, don't go down ever.

Oh where oh where can I sign up for this mythical unicorn cloud service?

I think the OP was being sarcastic.

At work I'm managing a gitlab instance for 15k users and 5k projects. Uptime is 100% since 1 year except for few minutes of planned downtimes every month for the monthly upgrade. To be honest I expected it to be a lot harder and run into troubles ... But I always find answers quickly in gitlab doc or forum