Advantages of Monorepos (2015) (opens in new tab)

I don't think version control system is the major differentiator for whether people like monorepos or not, tbh. Having a good incremental build/test system is far more important to developer experience, IMHO.

The biggest dissonance when it comes to the purported benefits of monorepos is that a "good" monorepo generally assumes very good interface design skills across all teams, but in reality, the path of least resistance is tacking on more and more unique codepaths (e.g. forking/"rewriting" existing things), so in effect, likability often comes down to how well a team is able to isolate itself from global changes (by choosing stable/boring APIs, inventing their own abstractions in their own little corner, or what have you).

Honestly I would 100% do a monorepo every single time if there was good tooling for incrementally building and testing libraries. Having to rebuild every image from scratch for every single change scales miserably. Things like Bazel exist, but you basically have to have a team dedicated to operating it (maybe the difficulty varies by language, but it was a major pain when I tried to use it to build some relatively simple Python projects a few years ago).

feel like the root problem companies run into when you don't have monorepo is shit gets locked down - e.g. I didn't even know this repo existed b/c I couldn't see it/clone it b/c of permissions. the other thing is lets say we have microservices - now I need to call your service - and most places are terrible at documenting things - especially if it's a new service which it probably is I'm trying to connect w/ it for the first time - now I have to figure out how to call your service - I can do that on my own my cloning your project and reading the code and bugging you but I'm way more less likely to bug you if it's part of the mono and I just need to open the code in the existing probject. I think this leads to a second point is mono does lead to more consistency and better knowledge sharing across codebases.

It's more than that. When you have to make changes that touch a lot of dependencies, it's much easier if all those dependencies

  - are in the same repo (making it easy to
    find and change all of them)

  - are in the same universe of build/test/deploy
    services (making integration of your changes
    atomic)

Atomicity of integration is essential, especially in organizations that move fast and make lots of breaking interface changes. Where it's to make a breaking interface change, it will be OK IFF you can make that change atomic.

Conversely, if you want to be able to make breaking interface changes, the integration and deployment of those has to be atomic.

Not having a monorepo & monobuild means that you have to have stringent interface backwards-compatibility commitments. That's fine if you're shipping an operating system, say, but it's usually too painful if you're not shipping anything to third parties.

For me, the atomicity feature is the killer feature of monorepos.

This is massively oversimplified.

- the cost of having access to more than you need: cognitive load and tooling for filtering, larger repositories require more tooling work to be performant

- there's also the atomicity of change and past changes which one can see/understand

Years ago, Google had a gcheckout tool that would trace the dependency information for whatever project you were working on, and then selectively grab the portions of the monorepo that you were going to need for it. Maybe they still have that or it's evolved into something else; I dunno, I haven't been there in a really long time.

Anyway, it seemed like such an obvious complement to the the perforce/monorepo style of working that I came away surprised that perforce wouldn't have hoisted such a thing into their product as a first-class feature. Tracing dependencies across a lot of different build systems is obviously not trivial, but it's not intractable, particularly if the tool is pluggable so that orgs can provide modules to handle their own particular approach.

This almost feels like a version of Conway's Law: you inevitably ship the org structure.

Your dev tooling also influences the shape of the thing that you write. If you have a monorepo then it encourages you to ship a monolith that freely interoperates with itself. If you have multiple repoes that need to be versioned against each other, you will ship components with more stable APIs.

So this means that if you ship a product within which customers are free to update portions of them at will, then using a monorepo will make things more difficult than necessary.

And if you ship a single unversioned monolith to the world, then using multiple repoes adds unnecessary friction to working within the company.

Google, at one point, had component versioning that was not just "build everything from the latest commit". Libraries within the tree would get tagged releases, and everything else would build from the latest tag of those libraries.

This practice was abandoned, but I don't know the reasoning for why it was abandoned.

Right, smaller repos add more friction to dependencies, that is certain, but flipside view of that is that it enforces API boundaries and thinking about systems building as SOLID components in their own right.

That friction sometimes helps: If it is painful to update Dependency A because it usually means upstreaming changes to A's Dependency B first, for instance, that can often indicate a tight-coupling problem that in a class diagram someone might easily discover and refactor over lunch but in a systems diagram was non-obvious without that "update hell" pain. Solving such tight-coupling problems is hard, and it may mean living with the pain for some time, and while monorepos make that pain go away they never solve those coupling problems (and arguably make it far easier to strongly couple systems that you likely don't want coupled). It's a lot like turning off all the Warnings in your compiler; it makes the immediate dev experience a lot nicer, but it risks missing things that while not problems now may be problems in the future.

I think there are also some benefits to using the same dependency managers for first-party components/libraries as for third-party components. The auto-updating of first-party versions is seen as a benefit to monorepos, but if recent and current CVEs have taught us anything you need to audit and update your third-party components quite regularly. Needing to also update first-party components/libraries with the same dependency managers has some benefits in terms of forcing a regular dependency update cadence, that then also benefits additional developer eyes on third party update rhythms. (Especially as increasingly more dependency managers pick up auto-auditing/security and CVE awareness tooling that runs on each update. There's more likely developer eyeballs on those audit reports if frequently run for first-party components and third-party components.) Dependency managers are their own friction in the process, but necessary friction for third-party components, and there are benefits to first-party components needing the same friction.

As with most software development practices there is no objectively "right" answer here. Monorepos have less friction in a large org. Friction and pain are sometimes useful tools, despite few people "want" them in their developer experience. Systems design is hard and tight-coupling is often an easy solution. Looser coupling is often better, more resilient design that is easier to work with at the boundaries and the "I can trust this other team's repo to be a black box and they let me file bug reports as if they were a second-party vendor" level, which can be its own tool for avoidance of mental fatigue.

Whether that's an upside depends. Mostly I think it's a downside.

Hm can you give an example of that? Are they wondering if the features they care about are deployed?

The linear version numbers could have an advantage in that regard. If you want to know if CL 12345 is deployed, and you know the current deployment is running as of CL 12350, then it should be in there. Conversely if it's less than that number, it's definitely not in there.

git hashes also have good properties but I'm wondering how non-technical employees use them. Do they know how to dig through the git history?

> Everyone can speak this language. The semantics are ridiculously simple.

Similar to the engineering/BOM-oriented semantics of everything is a drawing with a matching part number?

At least, I would like to read more about it. A while ago, I collected some information [0] but I don't know it first-hand. One Amazon developer told me that Nix is the closest thing in the Open Source world.

[0] http://beza1e1.tuxen.de/amazon_manyrepo_builds.html

What is proper decoupling? In a properly-tooled monorepo, project A can't take a dependency on project B unless B is a public library or explicitly gives access to A [1]. Authors have full control over coupling.

Sure, avoid changing the API contract. But when the time comes to change the API, you can 1) make the change backwards compatible and maintain both methods forever; 2) release a new major version and maintain both versions forever; or 3) just migrate the callers and immediately be free of all technical debt that would've been accrued in 1 and 2. This assumes internal clients whom you presumably can't break.

1: https://bazel.build/concepts/visibility

Not having to submit and coordinate PRs across a dozen repos is a pretty tangible benefit.

> unless you invest a ton of resources into proprietary tooling that requires a bunch of maintenance, since all the open source solutions are terrible at this and the whole data model is built around splitting out individual project repositories.

Agree that there's a bunch of tooling needed to operate a monorepo, but there's also a bunch of tooling to sanely manage dozens of "microrepos" as well (when an upstream library changes, update downstream libraries' dependency manifest files to take the new version, run tests, report errors back to upstream, etc). I don't know of any open source tools that manage this problem, but I'm guessing they aren't high-quality if only due to the complex nature of the problem space.

> And then after all that effort, why wouldn’t you just use tooling the way it was intended, and the way it’s used in the open source model, so you can partition your CI/CD without a bunch of hacks, and don’t run into bizarre scaling issues with your VCS.

Because the tooling sucks, as previously mentioned. Many changes require touching many repos, which means coordinating many pull requests and manually changing dependency manifest files and so on.

Ultimately, the "repo" concept is limiting. We need tooling that is aware of dependencies one way or another, and sadly all such open source tooling sucks whether it assumes the relevant code lives in a single repo or across many repos.

Amazon has Brazil versionsets and workspaces, which solve many of the same problems monorepos do. I really liked the way those extra resources let you organize "ad-hoc monorepos" from smaller repositories, but it's infrastructure that I haven't seen elsewhere.

Since leaving Amazon, I've mostly worked with monorepos and wouldn't go back to multirepos without Brazil-style tooling.

* Why is ownership management a mess in a monorepo? Can just decide “this team owns this folder hierarchy” etc. * ‘Contain blast radius of changes’ - is that actually difficult? Isn’t there tooling that figures out what changed and what dependencies need rebuilding? (eg Facebook buck)

For context, I was for a very long time at FB so am definitely used to the monorepo way, and recently switched to place which uses github + many repos, and it feels so much worse.

Honest question - how do you actually effectively share code between many repos? Example: How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? It should be a compile/buildtime error for the other project, but how does that work if everything is in its own little repository?

> It perplexes me people advocate for this strategy. All I can think is it's another one of those cargo-cult ideas that everyone is doing because Google did it (So it must be good).

Not sure if it is a generic comment or a comment on TFA:

i) If the latter, I'm compelled to point out that TFA doesn't nearly advocate for monorepos as much as it lists reasons why a few SV companies use it, how they use it, and what they get out of it.

ii) If the former, then this blog post makes for a good read: https://tailscale.com/blog/modules-monoliths-and-microservic...

Let's not pretend that Amazon hasn't spent significant effort over decades building and maintaining their own (non-monorepo) build systems and tooling.

I don't agree with having libraries in their own repositories. For us one of the biggest costs to fixing bugs in libraries particularly was having to then update all dependent projects with the new version. By moving our libraries into a monorepo alongside all the consuming projects we got away from that busywork. It really streamlined things for us.

> I agree that monorepos are great if you're using version control systems in their current state. But I can't help but wonder if it's a question of monorepos being good, or version control/tooling inhibiting other options.

Ultimately the problem is that we need tooling which is aware of dependencies, and the repo abstraction isn't. Whether that code lives in a single repo or in many repos is fairly irrelevant, but keeping the code in a single repo is usually a fair bit easier for many things (especially when you're working in a single language since the language's build tools are usually well-suited for this basic case) and you don't need to manually update dependency manifest files, test how a given upstream change affects every downstream package, or coordinate half a dozen PRs for every change.

That's factually untrue. Google, Microsoft, Facebook, Twitter, Airbnb all have a huge monorepo. Splitting it out is obviously not necessary with the right tools.

Legal obligation and unnecessary are mutually exlusive

> Additionally, how do you avoid doing pointless builds when new features are pushed? I can only imagine what the `.github` folder in a monorepo looks like.

It's simple, with proper tooling, you know exactly the dependencies, so you know which test depend on the affected files and can run those tests, the rest shouldn't be impacted. And that tooling exists. It's not the one you may be using, but it exists, and not just in FAANG.

Have you used git submodules before? I've only used them once and vowed to never use them again.

It's effectively just a pointer to a hash, and ends up being useless for versioning + a really nice footgun for tracking upstream updates.

The monorepo vs manyrepo tradeoff boils down to this:

Do you want more complicated build + deploy tooling or do you want more complicated dependency management?

If the former, pick monorepo. If the latter, pick manyrepo.

There are a bunch of downsides, although they are often just the opposite problem from what the monorepo solves.

For example, the article states:

> [In the other direction,] Forcing dependees to update is actually another benefit of a monorepo.

What happens when the other teams that depend on your work don’t have the time/priority to update their code to work with the new version of your code? The ideal case that monorepo proponents tout is that the team updating the code that is depended on can just update the code for everyone who depends on them… however, that update is not always trivial and might require rework for code deeper inside the other teams projects. Maybe they are depending on functionality that is going away and it requires major work to change to the new functionality, and the team is working on other high priority things and can’t spend the time right now to update their code.

What does the team do? Do they wait until every team who depends on them is ready to work on updating? Do they try to work out how the depending team is using their code so they can update it themselves? How does this work if there are dozens of teams that use the dependency? You cant have every team that creates core shared code be experts on every other team’s work. You can end up stuck waiting for every team to be ready to work on updating this dependency.

Imagine if this was how all dependencies in your code worked, and every build task used the latest release of every dependency regardless of major version bumps. You might wake up on a Tuesday and your build fails and now you have to spend a week updating your project to use the latest version. Multiply this by all the dependencies and your priority list is no longer your own, you are forced to spend your time fixing dependencies.

This is why we specify versions in our dependencies, so we can update on our own schedule.

Of course, the downside of this is now you have to support multiple versions of your code, which is the trade off and the problem a monorepo solves.

You are going to end up with downsides either way, the question is which is worse.

A monorepo assumes all your IP is either open or closed or you need a very reliable way to extract the OSS bits and publish them to a mirror without putting exposure of closed source IP at risk.

The main downside that people always mention is it takes a long time to clone or pull a large repo. This is actually a flaw of git, not a flaw of the monorepo as a concept.

Principle of least privilege springs to mind but I'm not familiar with the other issues.

Perforce is ok. It has roughly the right model, it scales pretty well, and the tools are... good enough.

Microsoft and Twitter, at least, have very publicly invested a lot of engineering work in making git a tool for monorepos.

Not sure why you got downvoted.

The problem is with your deploy system. You can consider each of the clusters to be a service. Thus, a change in Service A (cluster A), should not trigger a deployment of Service B (cluster B).

My pipeline is split in 2:

1. on bitbucket, we run a pipeline that builds "build artefacts", docker images and "packaged" cloudformation templates.

Each of these artefacts has a list of triggers, either base docker images or source code. I'm building the relevant docker image or cf package based on the triggers (it's quite a naïve glob() use).

2. On aws side, I have something I call AWS Apps, in short a Stack Name, along with a set of triggers (the above build artefacts). On merge to main, I only deploy the AWS Apps affected by new build artefacts.

I worked on a team that was doing embedded software inside a monorepo with Linux and cloud stuff, so we needed to version our stuff because we were not doing "continuous deployment" flashing the uCs all the time a change on master happened. We just had big "feature branches" and those got rebased and merged weekly. For the cloud stuff, there was just one version and that version was what other team was striving to continuously deploy into staging and then production.

It's not ideal but it was handy to have access to all the cloud and application code on the embedded side for stuff like interface definitions for communication protocols and stuff like that. On the same company I worked on another project where the definition files for the cloud interface where on a different repo we had to use a submodule and I preferred the monorepo.

As other commenter said, we used bazel and there was indeed a smallish team that gave build support. Ramping up new hires on the build system was one of the more painful processes, I had to give support myself to teammates that had like a six month tenure only that they hadn't been there from day one.

I think the most common option is you don't. The organizations that Dan gives as examples mostly don't produce public facing libraries, and when they do it's a separate process that lives on GitHub.

Instead, everything is based around the idea that you check out the state of the world at some commit, do your build, whatever validation you need, and send it to production. You do this pretty often, ideally multiple times a week. Very occasionally you have an emergency where you want prod + cherrypick, and you generally build tooling that allows saying "build at this commit, but also with these later specific commits merged in".

Technically in git you can namespace tags to your heart's content, it's a relatively free form naming structure, just like branch names. Basically the same rules even to the point that if you use slash separators (example: mylib/v0.0.1) some UIs will even give you a directory structure of tag lists. (On the flipside, some UIs get very confused, but that's not git's fault.)

What monorepos I've seen rarely bother with tags in practice, in part because they rarely individually version libraries, but at a technical level you can do it in git, if you need to.

At least at Google: you don't. There is just one current version of your library. There are exceptions, but for the vast majority of things there is no concept of versioning.

Yes! I once had to merge a dozen or so repos into a mono repo. I don't have my script handy but git allows merging repos with unrelated histories into one repo while preserving the history.

If I remember correctly, this is how you do it:

  1. Create a new empty repo for the monorepo
  2. For each repo, 'git mv' all of the contents into a new directory with the repo's name
  3. Add the repos to the mono repos as remotes
  4. Run 'git merge --allow-unrelated-histories' for each repo

You will now have a monorepo with preserved history with each old repo existing inside of a sub-directory in the new monorepo

You just want to merge the repositories, there are plenty of guides online. You need to realize that git has pretty simple internals, so the procedure looks like this: 1. Inside one repository you add a reference to the other repository with `git remote add`. 2. When you do a fetch, git will just download all the objects from the other repository. 3. Then you check the files out and make commit. You can do it in a few ways, for example, you might wish to preserve tags from all the repositories, but put them in their own namespaces, so you don't get conflicts. I wrote an answer on SO explaining exactly this: https://stackoverflow.com/questions/1425892/how-do-you-merge...

I've not used it extensively, but Josh can probably help here.

https://github.com/josh-project/josh

It's designed for making multiple Git repos from a monorepo, but I think you should be able to make a skeleton repo that represents your desired final monorepo layout and push your individual repos to the Josh subviews of that repo to combine them all.

(A big advantage of this approach over the multiple unrelated histories is that you don't have the mass move commits since Josh will rewrite all history as if the files were always in that folder, so you don't have to worry about history of individual files getting broken.)

Git as a system has no objection to having more than one 'initial commit'. It will happily take that branching history and merge it together. With a bit of branch renaming you can add extra remotes to your repo so both 'masters' are present. Commit to both to make the resulting directory structure not overlap, and then just merge. You'll end up with full history of both.

I did a quick google and these instructions seem about right (without the delete step): https://gist.github.com/msrose/2feacb303035d11d2d05

With Git I’m pretty sure you can literally just add a new remote to a completely unrelated repo, fetch it and then “merge” any branch from the other two into the new mono repo (ie. git checkout master; git merge other-remote/master). If all your projects are otherwise in a top level directory inside the monorepo this should merge cleanly and then can just live beside eachother in the checkout.

Lerna is only needed if you're publishing multiple packages from the monorepo. If you're consuming your packages only within the monorepo for your various services, Yarn Workspaces is generally all that's needed.

Just don't use Yarn + Lerna :) pnpm is amazing