Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.
They should install OpenClaw for that as well.
Not at all, you merely move the goal post of at what layer the "root cause" actually could come from! At that speed, it's always something short and sweet, while when you actually want to long-term address things, you have to have time to even investigate organizational issues or whatever the actual problems stem from.
But you have half a day? "Post-mortem: Push X wasn't properly analyzed before deployment, in future more testing" and call it a day.
https://damrnelson.github.io/github-historical-uptime/
Unfortunately, it doesn't look like it's being updated with new data. But it wouldn't look any better for GH if it was.
Is it possible that there has been a change in the way the data are collected/recorded that even partially accounts for this sudden onset?
IMO as a github-watcher, I think they changed their definition of what constitutes a sev-0 between sev-1 for the better. In particular, they had a few "sev-1"'s around the turn of the year that would be classified as sev-0's if they happened today.
Pre-4/22 GitHub sev-1 was a normal SaaS company's sev-0, imo. So I think their new system is more reflective of reality. My guess is that a few of their big customers bullied them to have more accurate SEV categorization.
At this point you would get better uptime by just self-hosting your own GitLab, Forgejo or Codeberg instance instead of dealing with Github's unreliablity.
There is no defending them with their clear neglet and carelessness of the platform.
I’m working on an open source Forgejo browser called Joui. It’s coming along nicely, and is so much snappier than GitHub in every single way.
People may have had complaints about functionality, features, commercial issues, but the thing used to at least have a decent uptime until recently.
The user profile / contributions and PR UX is pretty much the entire "hub" product since git is a fully separate offline app.
Is it? Seems a text description of "Make a website outlining 'How cooked GitHub' is with a modern style" to basically any LLM would produce exactly that UI and design, literally nothing of that design a human had any influence on, besides the ones selecting what training data the used LLMs was trained with.
I think most of us who've tried using LLMs for web-design can recognize that style and design at this point, regardless of model actually used.
Moving everything from GitHub to Forgejo and Tangled for now. These outages haven’t effected me for the past month because of this.
Most part screen is taken by picture. Contrast ratio is really low. Hard to read Should they remove that useless banner, current status which is the most interesting part coud've been made visible right away.
I would call this whole thing highly un-ergonomic
I get downtime on Supabase every few weeks. Even Cloudflare. And now Github
Less care on process, or quality, more focus on "just ship".
I've also seen and heard from peers this happening in multiple smaller outsorcing companies.
Has worked wonders for me :)
Gitea (what Forgejo forked from) recently stole the sidebar on repos from GitHub and I think that would be great for Forgejo to steal too…
Forgejo themed by Codeberg: https://codeberg.org/forgejo/forgejo (the codeberg theme is extremely low contrast)
Forgejo default: https://v15.next.forgejo.org/pparaxan/quark
Forgejo themed by Lix: https://git.lix.systems/lix-project/lix
Gitea: https://gitea.com/gitea/awesome-gitea
Gitea themed by Blender: https://projects.blender.org/blender/blender
I personally like Blender’s Gitea theme better than the rest but I guess that’s subjective. In dark mode I do not like the low contrast Codeberg theme or the default Forgejo theme, but all of the instances custom themes look great.
As far as Git forges go in general though.. tangled is very pretty https://tangled.org/tangled.org/core I think more power user oriented software should be comfortable with compact interfaces
Yes, it's still Microsoft, but they've forgotten about it, so it runs entirely adequately and is actually a surprisingly okay github replacement. It does nothing special, but it does do everything, just in a way you often would rather it wouldn't. It doesn't have the flexibility of JIRA for the ticketing, and the deployment machinery doesn't have the fanciness ( and vendor threat ) of chaining github actions, but it does handle both.
I haven't used gitlab, so I'm curious to hear what makes it a "complete mess" too.
* Microsoft's headless chicken naming strategy in full force, it's a miracle they haven't yet renamed and rebranded it to align with copilot yet.
Multiple companies are trying to create new versioning primitives/architectures which can handle machine-level code generation - 1 commit per second per repo.
It's like switching from horse buggies to automobiles, the whole worlds needs re-architecturing to handle the new load.
The age of boutique hand-coding is being replaced by the age of industrial software factories.
This is not a particularly novel level of scale. Facebook's mercurial backend had to handle >5,000 developers committing to the singular monorepo long before LLMs were a thing
Is the scaling issue with git or github?
New PR: disable new user signups for 6 months
HR initiative: all future KPIs automatically require three-nines availability; all bonuses are forfeited, regardless of accomplishments, if annual availability falls below target
HR initiative: fire CEO and CTO
Personally, I think they'd have more luck actually attacking the source, what that might be. Somehow I think Microsoft's push for "Every developer only use AI for development, no manual thinking/coding from now on" is the detrimental step, seemingly many companies are still discovering the right approach. Put a freeze to that, and I'm fairly sure you'd see less downtime pretty much immediately, unless all real engineers already left the company, I'm sure I would have at this point.
If you want to upload to GitHub, you should pay. The days of charitably giving away compute for the “open source communities” are over.
Grandfather existing public repositories in, then cut it off. Stop the bleeding. It doesn’t have to be forever.
It was just yesterday [0] that GA was down and another incident today? I am convinced that Copilot and Tay.ai are destroying GitHub and there is no CEO of GitHub to contact.
Now will you please self-host as I said 6 years ago? [1]
This time they've just scrubbed the evidence outright?
Today's is green, even though there was an outage.
Your choice is to accept the product that exists on the market, switch to another product that exists on the market (such as Codeberg or self-hosted Forgejo), make your own product, or not use any.
They’re trying to scale from 1 billion commits last year to over 14 billion this year. I have zero desire to try and manage that scaling. Basically being DDOS’d by agents all day now.
Last time I checked, this is called "doing their job".
You are free to choose what company you want to work for, but for github, if they are offering the services, often as paid services, they have an obligation to keep things running which is called SLA. Otherwise people will leave (and are leaving)
https://mrshu.github.io/github-statuses/
Seems more accurate with my experience of GitHub.
It's the same issue as the other day - display message at the top admitting that cache needs to be refreshed (or whatever the wording was)
maybe it's time to revert back to the central idea of git & not centralize around a particular provider.
for issues - mailing list will do. you can always slap a beautiful ui if you want to or a tui (as is the fad) these days.
actions can also be decentralized via an API spec & webhooks.
wild that there is a large pattern forming up of unreliable software being pushed
Facepalmed and decided that this is it for today.
even comparing branches were giving 500 in the web ui. even github copilot in their new session limits now also mention if github is going through an incident, which is plainly comical.
fun times!
OK now people can layoff even more engineers and feed the tasks to AI.
/s
