https://www.federalregister.gov/documents/2023/06/09/2023-12...
On the surface a product like managed git repos would seem to be relatively straightforward to deal with, but these same regulated firms are also under tremendous scrutiny for access management, change management, SDLC, etc etc. They also have a huge software footprint which just increases the impact.
Self-hosting is obviously the traditional answer, but that's not exactly a simple one either.
Just an interesting problem.
Of all the many dependencies on cloud services, git is by far the last I’d worry overly much about.
In particular, I've moved a CI for a large repository between different CI systems. It was anything but trivial: you want to believe "it's just a YAML that runs commands, right? Translate the format, right?" but it's really not; differences between how CI systems map commands to machines, external integrations (e.g., in this case, into Action's artifacts system, or output system) etc. all make it more complicated.
And it doesn't protect you from a "forced exit" either. Github could terminate your contract, and change the terms of the license in a way that you found unacceptable, or even go out of business, and being self hosted would leave you in no better position than if you had used cloud with external backups. You can somewhat mitigate this risk by self hosting an open source solution, so that in the worst case scenario, you can fork the project and maintain it yourself, but there is still risk that the project could be abandoned, or possibly have the license changed in future versions.
To be clear, I'm not saying that you shouldn't self host and SaaS is always better. But it isn't a magic bullet that solves these problems.
Of course we have a small and mostly unchanging number of users, don't have to deal with DDoS attacks, and can schedule the fairly-infrequent updates during maintenance windows that are convenient for us (since we don't need 100% availability outside of US working hours).
I don't have the metrics in front of me, but I would say we've easily exceeded github.com's uptime in the last 12 months.
Kinda eliminates all those pennies saved (in theory) for outsourcing to "the cloud" if you have to duplicate your infra.
Hybrid has always seemed the most optimal approach, but there's always someone in charge who thinks spending money on safety nets is just wasted money.
you shift it from a problem of software reliability to a problem of physical infrastructure. at some point in the chain somebody has to do that, but i'd prefer the person doing that was somebody with DEEP experience in it that could give you some nice confident assurances.
>at some point in the chain somebody has to do that, but i'd prefer the person doing that was somebody with DEEP experience in it that could give you some nice confident assurances.
Yes. This is part of the reason why services like git are being moved outside the datacenter. Most of the product offerings on the market don't scale well, have terrible reliability and are still very expensive to run.
I mean seriously people, a "cloud" is just someone elses' data center.
Am I missing something?
When I'm trying to explain to people what it's like to work on this kind of software, I like to use an analogy: it's as though I have my own personal brick, or group of bricks, in the great pyramids of Egypt, just a a tiny piece of a stupefyingly, inconceivably larger whole -- and when I twist my chisel and strike my block just so, at exactly the right (or rather the wrong) angle, I can shake the very foundations of Egypt.
Is the risk of your git repos higher than a chip shortage causing you to lose access to the infrastructure you need? So many factors to consider. A chip shortage doesn't seem that unlikely with geopolitics.
The list of scenarios you mitigate for seem like they could very easily be an arbitrary list of the of scenarios a single person came up with.
The risk calculations are very primitive at the moment, I'm guessing they will be refined over time as industry feedback starts to resonate.
They are replacing everything with Github actions. I wonder what they are going to do when Github is down.
But you are right that I want reliable and easy-to-use services. And centralisation is often one way to go there.
As an interesting counterpoint: Git itself is decentralised and replaced centralised services like Subversion. And that made git easier to use, especially easier to get started with: no need for a server, no need to be online, just do `git init` in any old directory.
A GitHub-clone could be more decentralised, but they'd need to use that decentralisation to drive those other features that people actually care about day to day.
- A canonical name and place on the web;
- Access policy enforcement (who can commit and when);
- The whole pull request thing, with tags, issues, review, discussion, etc linked to it;
- Code review, linked to said policy enforcement;
- Issue tracking, even as basic as what GitHub offers;
- A trusted store for signing keys, so commits are verified;
- CI/CD triggers and runners;
- A page with releases, including binary releases, and a CDN allowing to use the download links without fear.
This is way more than distrusted version tracking. Actually the above is not even married to Git; it could be as valuable with Mercurial, or even Perforce.
This is a large product, actually a combination of many potentially self-contained products. It should not be compared to Git, but rather to Gitea or BitBucket. Not all of this can be reasonably decentralized, though quite a bit can.
Interestingly, what GitHub mostly enforces is where your branches point to. Not who can make commits. That's mostly because of how git works, not because of any grand design on GitHub's part.
I've used up 17h of CI time these two (slow) January weeks, for free, testing stuff across ~20 different OS/CPU combinations.
That's on just one "personal" project; a bigger dependency of that, of which I'm a maintainer, spends an order of magnitude more.
Can you (GP post, people complaining, not parent) blame us? Should we instead self host everything and beg for donations just to cover the costs?
As a professional software developer, I want tools that just work that I can rely on. GitHub 99.99% uptime is something I can rely on.
Daily: 8.6s
Weekly: 1m 0.48s
Monthly: 4m 21s
Quarterly: 13m 2.4s
Yearly: 52m 9.8s
If you assume that "uptime" means all tools are available
https://statusgator.com/services/github
this appears to be 45 minutes in just one day
Incident with Git Operations 30m Jan 14, 2025 9:01 AM Down
Incident with Git Operations 10m Jan 14, 2025 8:51 AM Down
Incident with Git Operations 5m Jan 14, 2025 8:46 AM Down
Not much margin to hit four 9's left for the rest of the year.
Their enterprise level SLA is only 99.9% (measured quarterly) and the remedy is a 10% credit, increasing to a 25% credit if they drop below 99%.
github basically shoves a webby frontend and workflows on top of someone else's work. That's all fine and good but github is not git.
As a professional IT consultant, I want tools, I use lots of other's and I also create my own and I also generally insist on hosting my own. I'm also a fair carpenter and several other trades. I have tools for all my trades and I look after all of them. That way I can guarantee quality - or at least reproducible and documented quality.
I'm sure you are fine with abrogating responsibility for stuff that you are not able to deal with - that's all good: Your choice.
EDIT: Sorry, forgot to say: "Yay cloud"
Whenever you do a clone or an npm install or apt get or pip install, etc...
You choose github because your dependencies chose git
Not your point, really, but fortunately, git is easily extensible. This in-repo issue tracker is surprisingly feature complete: https://github.com/git-bug/git-bug. Anyone else given it a whirl?
I believe Gitea has support for it, not sure to what extent.
AFAICT the internet was built on negativity.
Here's the 2nd post from a random USENET group I found:
https://www.usenetarchives.com/view.php?id=comp&mid=PDQ5ajZp...
Except if you have a release planned but most don't at that time, statistically.
Problem is that people get comfortable with pushing to branch -> deploying in dev and testing from there.
Nothing is built into git to let it actually run decentralized: there's no server or protocol where someone can register say, an identifying public key and then just have a network of peers all communicate updates to each other. It's even pretty damn unsafe to just run a repo through basic file-sync (I do this for myself with syncthing with bare repos in a special folder, which seems to work fine but I'm hardly loading it up to chase down why it doesn't).
Well there is https://github.com/git-bug/git-bug
Email and mailing lists?
Otherwise you can claim Facebook is distributed because you can email people links to Facebook pages.
git@github.com: Permission denied (publickey).
This is bad.
But yeah, also status pages seem to be under the domain of non-engineers who are too concerned with how things look, vs. conveying useful information in a timely manner, and ultimately, fail at both.
fatal: clone of 'https://github.com/libsdl-org/freetype.git' into submodule path '~/src/SDL/SDL_ttf/external/freetype' failed
...
fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output
...
fatal: clone of 'https://github.com/libsdl-org/harfbuzz.git' into submodule path '~/src/SDL/SDL_ttf/external/harfbuzz' failed Failed to clone 'external/harfbuzz'. Retry scheduled
...
Failed to clone 'external/freetype' a second time, aborting
For a problem that's supposedly "fixed" that's a whole lot of errors...
Then went to github status and calmed down.
Developer "snow day".
Update: They promise >99.9% on a quarterly basis for enterprise customers - https://github.com/github/docs/blob/main/content%2Fsite-poli...
The front-end is glacial nowadays and frequently has issues actually loading the page, actions frequently has some kind of panic attack and breaks or just grinds along a glacial speeds. The UX has gotten worse (why no merge-queue button?).
"We've identified a cause of degraded git operations, which may affect other GitHub services that rely upon git. We're working to remediate."
For example, I doubt you would be able to easily merge any pull-requests or use the same CI/CD code for the same services without hacky solutions.
In a wonderful twist, we are relying on a couple modules served from GitHub!
if there's an error/timeout, they'll do a check of their status page so you don't get the standard 'error' but rather a 'dont worry, youre not doing anything wrong. its borked, our bad' message.
There's innumerable causes of this kind of failure that aren't rooted cloud service provider shenanigans
(Admittedly the duration of the outage does "feel" like an infra outage)
ducks
eats popcorn waiting for explanation of why they didn't catch it in non-prod
~$25/mo