GitHub issue - resolved (opens in new tab)

My company uses a self-hosted Gerrit instance. A bit of a learning curve, but the code review experience is SO much better than PRs (one commit == one unit of review, clean intra-diffs between different revisions of a patch, stacking of reviews simply by having multiple commits on a branch, UI is very snappy and responsive...).

Self-hosting Gerrit is easy[1] because all its internal state is fully transactional, including reviews and configs, and is simply stored as normal Git commits inside the Git repos on the filesystem.

In fact, our instance had much better uptime over the past year than GitHub despite being migrated to another server once!

[1]: ... unless you need a complex high availability setup or replicas. But 99% of projects are fine with just a single instance and backups.

Gitea[1]. It's a self hosted Github alternative, and it's quite lightweight too.

[1] https://gitea.io

If you're into self hosting, soft-serve is a really cool CLI based git server from Charmbracelet that is about a half step above a bare git repo over SSH

https://github.com/charmbracelet/soft-serve

I have a vps with bare git repos, pushing via ssh. A selection of these has hooks set up to mirror all pushes onto github/gitlab/$platform, but these are write-only.

I've been very happy with SourceHut (git.sr.ht). It's fast and slick. Does what you need, none of the cruft. Among other things, I really like: I can push to any URL under my username and it creates a repo automatically, and that it reminds me to put a license file on public repos.

I really like how the author is open about both the development and the business side.

At my previous work almost everything was self-hosted including Git and servers to run machine learning models. The only exception was Jira. The company owned the servers and rented the space in a datacenter. For code reviews we had Critic.

At the new work we use BitBucket, but for reviews its UI is strictly worse than Critic. And on top of it it is strictly more expensive than self-hosting experience including paying for a competent sysadmin.

I understand that cloud allows to offload a lot of headaches, but I really see no point in using cloud services for development. Even for a small company a dedicated server with another to spare in case of failures will be cheaper and it’s administration will be trivial.

I'm migrating from GitHub to CodeCommit. My project has pretty strict security guidelines, and GitHub doesn't have a high enough accreditation. GitHub tries to handwave this by saying good current practice means there's no personally identifiable information, etc. in Git repositories, but I'm not willing to entrust the code behind moderately sensitive infrastructure to a service that thinks they don't need to implement (and more importantly, prove they implement) better security standards and practices.

Recent developments have only reinforced my feelings on this matter.

I’ve been happy with SourceHut

We still use self-hosted GitLab. It's crazy expensive but we're completely locked in to their CI/CD pipelines plus we still enjoy using it ¯\_(ツ)_/¯

Gitlab is very good and feels almost the same.

Everytime something has degraded performance or is down for me:

1. check my internet connection

2. check HN

3. check official statuspage

I find downdetector to be quite reliable. You can pick up on a spike of reports very quickly.

HN though comes with all the commentary.

it'd be cool if HN would check the status of Github periodically and turn into a "we're partying now" kind of UI when Github is down.

Same here haha. StackOverflow informed me that it means GitHub is down, and indeed it was.

I have to wonder if Git could somehow report this better. I guess it depends on exactly how GitHub is down, but "fatal error in commit_refs" made me worry that my local repo was somehow hosed.

Same here! Was going crazy what I was doing wrong. Renamed my branch to see if that was causing anything.

For me it went down just as I was logging in, I thought I was banned or something :')

I was rebasing and thought I had committed some kind of git sin that I wasn't aware of.

I truly doubt you are running at the scale of GitHub both in terms of users, complexity, and amount of data.

I'm sure your scale is similar.

Only GitHub truly knows. But everyone here knows that since Microsoft acquired it, it has degraded to the point where it goes down every month.

It is so frequent and unreliable, you just might as well self-host at this point. You would likely have better up time than GitHub over the past three years since this prediction. [0]

[0] https://news.ycombinator.com/item?id=22867803

I find this strange too. GH seems to have more major incidents lately...

ChatGPT overloading it... (scanning repos)

Seems better done than a lot of status pages, runs on separate infra, updated, has a way to subscribe, etc.

However, saying "degraded performance" when you know it's "down for everyone" is an industry phrasing thing that's irritating. AWS also has "elevated response times" when everyone is seeing 5xx errors, or infinite response times.

Because if you don't spend multiple days per year managing your own "backup" to GitHub, you might be left unable to push to GitHub for a few hours per year?

btw, I hope none of your CI system relies on build steps that might include pulling code from GitHub or downloading packages from GitHub Package registry. Often when GitHub is down, my CI system on GitLab is broken too.

Depends on the context. I need to push to a repo and QA needs to test the app, but since I can't push, they can't. No other practical way (other than setting up another origin on Bitbucket or GitLab, or emailing/zipping sending etc).

For those who missed the context of this joke: https://news.ycombinator.com/item?id=35285390

Can’t get hacked if you’re offline.

Git avoids the problem where the central service being down gets in the way of local development. Even in Git, that central service going down means degraded collaboration.

Normally a Git remote is just an ssh-accessible machine, and so pretty resilient. But GitHub is a lot more complex, so apparently that simple service went down, along with all the features they built on top of it

I’m not a big Git/GitHub user but presumably distributed version control is still superior to Systems like Subversion that don’t provide a full clone of the repo, as you can still work during GitHub downtime and then use the good branch merging tools to merge back after the outage?

...Github has gotten significantly better.

I think you meant "I dunno what's down..."

"GitHub is down" probably wouldn't even make the top 10 stupidest reasons that I couldn't work. Is the idea here that you're downloading brand new code? I would imagine most people would have a stale from Friday copy of the repo locally.

You just clone the repo from one of your colleagues'.

I'd long assumed the opposite, but maybe Azure DevOps is the Git web interface they're betting on long-term.

Depends on if it’s at all related

The site is up but git push still doesn’t work for me as of now.

Given that looking at repos is just about the only thing that I can do then I'd say that seems unlikely.

NB: GitHub was purchased by MS 5 years ago. All these events are in the last month.

It's apparant that it wasn't without issues prior to acquisition (e.g. a quick search for GitHub issues prior to 2018 gives this: https://techcrunch.com/2017/07/31/github-goes-down-and-takes...) - reporting issues in 2017, 2015, and 2012.

I don't have the data to comment on whether it was better before or after MS acquisition, but would suggest this isn't the best sample size to base any conclusions on.

You can keep issues within git itself: https://github.com/MichaelMure/git-bug

Clicking "Subscribe To Updates" top right on https://www.githubstatus.com/ gives me a whole load of options for getting updates, one of those is RSS

Does anyone have real, contemporary numbers on this? Your observation matches my intuition, and I've seen some stats around the first two years after the acquisition¹, but I don't know any good up-to-date analysis on the question.

My intuition aligns too closely with my known biases here for me to be satisfied with that alone.

1: https://statusgator.com/blog/has-github-been-down-more-since...

You can always sync over local (or remote) ssh or fileshares or even zip up smaller repos, sneakernet them over to those who need them, unpack it there and pull from the unzipped folder.

The pros who maintain git uses email(!), but I think that would take more time than just waiting out the outage.