* on 16th, https://www.githubstatus.com/incidents/fpk08rxnqjz2
* on 17th, https://www.githubstatus.com/incidents/sksd097hm0y5
* on 22nd, https://www.githubstatus.com/incidents/83lq7ftk19r5
* on 23rd, https://www.githubstatus.com/incidents/tyc8wpsgr2r8
* on 24th, https://www.githubstatus.com/incidents/y5hdmv0p49x3
but damn, since the microsoft acquisition, they are having more issues more frequently
We've reached the stage where architectural decisions are being questioned. Nobody should need to ask me "well why are we using github?" but here we are.
They needed to fix this two weeks ago. The next time I'm in charge of stack decisions I will be evaluating competitors. This is exactly how Slack beat Hipchat.
Don't let weekends dilute your view of the situation. There's only 23 weekdays in March and Github has not been reliable for 5 of them.
Long read but describing it neatly: https://danluu.com/nothing-works/
A decision between - do you pay your personnel for doing it - or - do you give others money and hope they do it in a way which is good enough. Outsourcing as trend failed because it was driven by capitalism isn't a new finding. And as convenient removte web services are, the cloud is outsourcing.
Maybe after another doubling we will have these options available. We should be a perfect fit at this stage for a hosted solution. Github is making themselves unviable for a company at this stage (low $XX million ARR)
Hoping to migrate away our little chunk of github to a private island by end of next week.
Hackernews talks a lot of shit about how big boys can run their computers better than us lowly startup peasants. This is a fortunate situation where that stale argument starts to fall apart really quickly. I have many other systems with far higher availability than what github offers in their public cloud or even as part of the contractual 99.5% enterprise SLA.
Sure, github is really complicated and hard. Yes, it's an incredible tool. No, it's not OK to rest on your laurels and let COTS database technology kill your product when you are a billion dollar technology company with the ability to write your own systems from scratch several times per year using parallel teams and other investor money furnaces.
Oh and because cloud forces continuous payment whereas prior many customers simply bought a one year license and went on without renewing support.
I've had experiences where devops minded people I've worked with have wanted to self-host services such as Bitwarden. Sure, it will be cheaper and you will have noone to blame but yourself, but once things go bad they go really bad. It's also another thing to keep eyes on.
I guess similar argument could also be extended to self-hosted clouds. Seems like it could take away a lot of focus and energy from working on the product itself.
No. You update it on patch day (or when a big CVE comes out that you are actually impacted by) and know exactly what goes wrong and when. If you can't solve it, you roll back. When Github (or a part of it) goes down, you know nothing and with persistent issues there's no way to solve it either.
A company of the size where "but we have to scale" is an actual issue should self-host. A SaaS solution is a risk that you cannot mitigate.
Self-hosting inherently mitigates that problem because you now need to support less than 0.1% of the load of the worldwide service.
It also puts you in control of maintenance and updates - you can choose to make changes outside of business hours so that nobody is affected if you screw up. Developers at SaaS services can't easily do that because it's always business hours in some parts of the world, and may not be motivated to do it anyway even if it was possible with some effort.
Our company uses GitHub actions and other features for deployments so every one of these outages stops us from putting any work on production.
When, in reality, it's one service having issues and not the whole site. These incidents also seem to be resolved quickly.
Downtime is not the end of the world.
> Downtime is not the end of the world.
What if you needed to push that critical change and it is down and all you could do is wait?
What if you hosted your website on GitHub Pages? Maybe you use GitHub Actions (I assume most do here and are paying for it for their teams). Surely people use it for pull requests and issue management as well as for the webhooks and basic git operations.
There are those that went 'all in' on GitHub and use everything on it and are now crying that it is unreliable. This is where going 'all in' makes no sense. (Especially without a backup/self-hosted system somewhere.) Or centralizing everything on it as predicted years ago. [2]
[0] https://news.ycombinator.com/item?id=30841070
> Downtime is not the end of the world.
Tell that to our customers & support...
I did move my repos to "free" basic git hosting services on the net.
What would you need for today "development" (broad definition) of "popular" components: a web front end, an issue tracker, a mailing-list, a git server (ssh-ed write, http/https read).
The web front end is static html.
noscript/basic (x)html is beyond than enough for an "issue tracker" (cf bugzilla), including account creation (google javascript only-captcha is really nasty).
The mailing-list would need to implement grey listing for subscription (which would be disabled once the subscription process is done). The naive usage of spamhaus lists by many sysadmins of email servers is really toxic for self-hosted ppl or users of small email services.
The git server, well, the git server.
This is significant upfront work, not to mention the maintenance with the permanent "attacks" from corpo/state-sponsored paid/brain-washed saboters to deal with.
I understand why many devs are caving-in: they endup using those gitlab-like free and ready development hosting services because of the "path of least resistance". This is a very toxic pitfall, because those hosting services forces ppl to use the absurdely huge and grostesquely complex google(blink/geeko) or apple(webkit) based browsers where noscript/basic (x)html browsers should be more than enough for most, if not all, core online development functions.
The answer is no. Other UIs are much worse in my experience. I haven't given Gitlab a good shake in a while though. That said, the network effects of Github are pretty significant. Everybody's on it, lots of major repos and orgs use it.
>>> Hopefully GitHub won't have another outage in a week's (or even a months) time. [0]
Yet another incident less than a week / 4 days later with GitHub Actions if one was to go 'all in' on GitHub. Would have to reset the counter once again.
You can see in the whole comment chain [0] and in [1] why I was totally right in the 'long term' of not 'centralizing everything' on GitHub since 2020. [1]
Hello Microsoft executives, we would like some transparency.
Was discussed on HN too last week: https://news.ycombinator.com/item?id=30783051
Anyone skimmed their enterprise SLA docs to see how much we are due back?
Developers ain't free.
