Did they get a waiver from the git team to name it as such?
Per the trademark policy, new “git${SUFFIX}” names aren’t allowed: https://git-scm.com/about/trademark
>> In addition, you may not use any of the Marks as a syllable in a new word or as part of a portmanteau (e.g., "Gitalicious", "Gitpedia") used as a mark for a third-party product or service without Conservancy's written permission. For the avoidance of doubt, this provision applies even to third-party marks that use the Marks as a syllable or as part of a portmanteau to refer to a product or service's use of Git code.
Where does this fit into a product? Maybe I am blind, but while this is cool, I don't really see where I would want this.
What if these bug reporting platforms could create a branch and tag it for each issue.
This would be particularly useful for point and time things where you have an immutable deployment branch. So it could create a branch off that immutable deployment branch and tag it, so you always have a point in time code reference for bugs.
Would that be useful? I feel like what you’re doing here isn’t that different if I get what’s going on (basically creating one repository per bug?)
I like this system in general, but I don't understand why scaling the number of repos is treated as a pinch point? Are there git hosts that struggle with the number of repos hosted in particular? (I don't think the "Motivation" section answers this, either.)
There are many reasons not to do this, perhaps this scratches away at one of them.
It’s unlikely any Git providers struggle with the number of repos they're hosting, but most are larger companies.
Currently, we're a bootstrapped team of 2. I think our approach changes the kind of product we can build as a small team.
Unless, of course, your product is infinite git repos with cf workers.
You can still sync to a platform like GitHub or BitBucket after all users close their tabs.
A long time ago, I looked into using isomorphic-git with lightning-fs to build light note-taking app in the browser: pull your markdown files in, edit them in a rich-text-editor a la Notion, stage and then commit changes back using git.
> We ended up creating our own Emscripten filesystem on top of Durable Objects, which we call DOFS.
> We abandoned the porting efforts and ended up implementing the missing Git server functionality ourselves by leveraging libgit2’s core functionality, studying all available documentation, and painstakingly investigating Git’s behavior.
Using a ton of great open source & taking it all further. Would sure be great if ya'll could contribute some of this forward!
Libgit2 is GPL with Linking Exception, and Emscripten MIT so I think legally everything is in the clear. But it sure would be such a boon to share.
I believe our changes are solid, but they’re tailored specifically to our use case and can’t be merged as-is. For example, our modifications to libgit2 would need at least as much additional code to make them toggleable in the build process, which requires extra effort.
Very cool project. I hope Cloudflare workers can support more protocols like SSH and GRPC. It's one of the reasons why I prefer Fly.io over Cloudflare worker for special servers like this.
Choose these values:
* P, pack "Planck" size, e.g. 100kB
* N, branching factor, e.g. 8
After each write:
1. iterate over each pack (pack size is S) and assign each pack a class C which is the smallest integer that satisfies P * N^C > S
2. iterate variable c from 0 to the maximum value of C that you got in step 2
* if there are N packs of class c, repack them into a new pack, new pack is going to be at most of class c+1
If only 20% of the content gets changed, the rolling hash that Borg does to chunk files could identify the 80% common parts and then with its deduplication it would store just a single compressed copy of those chunks. And as a bonus, it's designed for handling historical data.
adamc@router> show arp | display xml <rpc-reply xmlns:JUNOS="http://xml.juniper.net/JUNOS/15.1F6/JUNOS"> <arp-table-information xmlns="http://xml.juniper.net/JUNOS/15.1F6/JUNOS-arp" JUNOS:style="normal"> <arp-table-entry> <mac-address>0a:00:27:00:00:00</mac-address> <ip-address>10.0.201.1</ip-address> <hostname>adamc-mac</hostname> <interface-name>em0.0</interface-name> <arp-table-entry-flags> <none/> </arp-table-entry-flags> </arp-table-entry> </arp-table-information> <cli> <banner></banner> </cli> </rpc-reply>
...daily? monthly? how many versions do you have to keep around?
I'd look at a simple zstd dictionary based scheme, first. Put your history/metadata into a database. Put the XML data into file system/S3/BackBlaze/B2, zstd compressed against a dictionary.
Create the dictionary : zstd --train PathToTrainingSet/* -o dictionaryName Compress with the dictionary: zstd FILE -D dictionaryName Decompress with the dictionary: zstd --decompress FILE.zst -D dictionaryName
Although you say you're fine with it being not that storage efficient to a degree, I think if you were OK with storing every version of every XML file, uncompressed, you wouldn't have to ask right?
As a hobbyist, “free” is pretty appealing. I’m pretty sure my repos on GitHub won’t cost me anything, and that’s unlikely to change anytime soon. Not sure about the new stuff.
This could be a fantastic building block for headless CMS and the like.
what if there are two users who wants to access the same DO repo at the same time, one in the US and the other in Singapore? the DO must live either in US servers or SG servers, but not at the same time. so one of the two users must have high latency then?
then after some time, a user in Australia accesses this DO repo - the DO bounces to AU servers - US and SG users will have high latency?
but please correct me if i'm wrong
I must be getting old but building a gigantic house of card of interlinked components only to arrive to a more limited solution is truly bizarre to me.
The maintenance burden for a VPS: periodically run apt update upgrade. Use filesystem snapshots to create periodic backups. If something happens to your provider, spin up a new VM elsewhere with your last snapshot.
The maintenance burden for your solution: Periodically merge upstream libgit2 in your custom fork, maintain your custom git server code and audit it for vulnerabilities, make sure everything still compiles with emscripten, deploy it. Rotate API keys to make sure your database service can talk to your storage service and your worker service. Then I don't even know how you'd backup all this to get it back online quickly if something happened to cloudflare. And all that only to end up with worse latency than a VPS, and more size constraints on the repo and objects.
But hey, at least it scales infinitely!
And make sure it reboots for kernel upgrades (or set up live-patching), and make sure that service updates don't go wrong[0], and make sure that your backups work consistently, and make sure that you're able to vertically or horizontally scale, and make sure it's all automated and repeatable, and make sure the automation is following best-practices, and make sure you're not accidentally configuring any services to be vulnerable[1], and ...
Making this stuff be someone else's problem by using managed services is a lot easier, especially with a smaller team, because then you can focus on what you're building and not making sure your SPOF VPS is still running correctly.
[0] I self-host some stuff for a side-project right now, and packages updates are miserable because they're not simply `apt-get update && apt-get upgrade`. Instead, the documented upgrade process for some services is more/less "dump the entire DB, stop the service, rm -rf the old DB, upgrade the service package, start the service, load the dump in, hope it works."
[1] Because it's so easy to configure something to be vulnerable because it makes it easier, even if the vulnerability was unintentional.
There's only a difference here because there exist off-the-shelf git packages for traditional VPS environments but there do not yet exist off-the-shelf git packages for serverless stacks. The OP is a pioneer here. The work they are doing is what will eventually make this an off-the-shelf thing for everyone else.
> Rotate API keys to make sure your database service can talk to your storage service and your worker service.
Huh? With Durable Objects the storage is local to each object. There is no API key involved in accessing it.
> Then I don't even know how you'd backup all this
Durable Object storage (under the new beta storage engine) automatically gives you point-in-time recovery to any point in time in the last 30 days.
https://developers.cloudflare.com/durable-objects/api/storag...
> And all that only to end up with worse latency than a VPS
Why would it be worse? It should be better, because Cloudflare can locate each DO (git repo) close to whoever is accessing it, whereas your VPS is going to sit in one single central location that's probably further away.
> and more size constraints on the repo and objects.
While each individual repo may be more constrained, this solution can scale to far more total repos than a single-server VPS could.
(I'm the tech lead for Cloudflare Workers.)
I would love to use this to serve as a live/working automatic backup for my github repos on CF infrastructure.
I am ending up with AWS lambdas. Not only that solves the Wasm issue but you can have up to 10Gb of memory on a single instance. That is close to enough for most use cases. 100Mb? Not really.
Between libgit2 on emscripten, the number of file writes to DO, etc, how is performance?
It provides client and server API. The latter is used by Gerrit for its server. https://www.gerritcodereview.com
Not sure what the Java to WASM story is if that is a requirement for what they need.
Unfortunately, the entrepreneur in me continues that thought with "work that could have gone into finding customers instead". Now you have a system that could store "infinite" git repos, but how many customers?
But I can't figure out what makes this an AI company. Seems like a collaboration tool?
i prototyped a similar serverless git product recently using a different technique.
i used aws lambda holding leases in dynamo backed by s3. i zipped git binaries into the lambda and invoked them directly. i used shallow clone style repos stored in chunks in s3, that could be merged as needed in lambda /tmp.
lambda was nice because for cpu heavy ops like merging many shallow clones, i could do that in a larger cpu lambda, and cache the result.
other constraints were similar to what is described here. mainly that an individual push/pull cannot exceed the api gateway max payload size, a few MB.
i looked at isomorphic, but did not try emscripten libgit2. using cloudflare is nice because of free egress, which opens up many new use cases that don’t make sense on $0.10/GB egress.
i ended up shelving this while i build a different product. glad to see others pursuing the same thing, serverless git is an obvious win! do you back your repos with r2?
for my own git usage, what i ended up building was a trustless git system backed by dynamo and s3 directly. this removes the push/pull size limit, and makes storage trustless. this uses git functionality i had no idea about prior, git bundle and unbundle[1]. they are used for transfer of git objects without a server, serverless git! this one i published[2].
good luck with your raise and your project. looking forward to the next blog. awesome stuff.
