Though I always wonder - how do Google, Microsoft, Facebook etc. deal with developing code near the root of their dependency tree? Utility libraries for example. Technically you're going to have every change you make there building all the code and running all the tests, which is obviously unworkable. What do they do?
But you still have library code that's depended on so much that you can't easily run all the tests for each change. So you run a train system so that you can run tests for a whole bunch of changes (those on the same train) at once. Developers have to schedule changes to get on the next train. Other code analyzes the failures from the train to try to apportion blame correctly.
Then, because getting on trains can be cumbersome and tests you care about can fail because of other changes, you build random test sampling and smoke test subsets so that you can get some immediate results to attach to a review, etc.
It roughly works, as well as anything can at that scale.
Merge trains/queues don't really help with the problem I'm describing - they just prevent race conditions between commits. You still need to run all the tests for each PR/MR first before joining the queue/train.
Of course it has consequences and makes some things harder, you really have to go in on API-first even for things like UI widgets if different ones live in different repos, but it at least alleviates the common problem of trying to get multiple repo commits merged at the same time and have everything keep agreeing. I still prefer the monorepo approach.
Or you do what the .NET team at Microsoft did, create a separate monorepo that rolls up individiual repos: https://github.com/dotnet/dotnet
If you are having to keep two or more modules/libs/packages/repos in sync, those should be unified.
It doesn't change the fact other people will think: "why didn't you just build it out of normal bricks to save yourself all that trouble?".
If everyone loved it, there'd be no Hacker News mobile apps.
I'm working my way through them but it takes quite a bit of time and checking in with people on what is actually used.
Besides that there is some prestige in saying that you handle 2000 amount of repos instead of it being "we have 20 prod repos and 1980 personal playground repos with one commit"
(This is assuming GP is correct. I have my doubts)
In our monorepo, everyone passes around django orm objects and boundaries are practically non-existent. N+1 queries abound. Tests are full of patching and mocking and are _slow_. Our build takes over an hour to run tests. Someone on team A can and absolutely will mess up what someone on team B is doing. We are now having to spend quarter upon quarter as we define and enforce domain boundaries within the python code base. It is all bolted on checks. Tests are getting worse and people are actively trying to figure out ways around the testing system because it sucks.
Compare to my last gig. We had several hundred production repos. Each repo starts from a template with its own build pipeline. All production repos are gated so that any PR must pass tests before it can merge. Any merge has to pass tests before it could be deployed. As the base build processes matured, teams could, at their leisure, pull their services up to the latest and greatest. We even migrated from Jenkins to Buildkite; yeah, it took N pulls into N repos. Not a big deal. Most projects' tests and builds could get code out to production in under 10 minutes, including all those checks. Due to the network boundary, you couldn't accidentally get around someone's abstraction. And if one team blew up their build doing something dumb? No problem, it only affects that one team.
The argument is "gah, managing all those services!" Keep data behind APIs. Keep APIs backwards compatible. Keep dependencies acyclic. This is _possible_ with monorepos, but you have to do extra work compared to networked services -- yes, when any particular team/service can deploy in minutes due to low build system complexity you are winning. Can you get that wrong and make strange cyclic dependencies and introduce performance issues due to network hops? Yeah, of course. However, we were processing, literally, 10s of billions of api requests on this system and teams could work untethered from one another. The new gig does eerily similar software, but is several orders of magnitude slower in their ability to process data and their ability to move new features.
yes, yes, you could have networked services and a monorepo and you can leverage tooling like Pants to minimize the testing to only account for changed files. It is just fighting what I have found to be a better model. Keep things separate. Keep things fast to change.
- Dev-time analysis. This is, types, linters and tests that will throw errors and block merges before they happen.
- A good code-review environment. Not one that's so tedious that everyone just seeks the stamp to move on, but also no one that's so lax there's no code review at all. One in which code-review is really an opportunity to improve the code before it's merged, people are committing to unblocking others, and everyone assumes their responsibility in having the right people look at changes.
- A better team structure. One where ownership of the code is shared, and code quality goals are present.
Usually if computer systems, peers and the org at large all agree in creating a good dev environment, dev environments tend to not be bad.
In my experience not having a monorepo doesn't fix the issue, it just hides it away. Repos don't have the same code-quality, and overall everyone tries to stay in their little island and not deal with problems and disagreements outside of it. You lose the opportunity to talk about these issues. There isn't that learning and teaching dynamic in the wider org, and the whole company is worse off as a consequence.
Plus, of course, there's a lot of extra-overhead in developing with APIs between each component.
Monorepos have their drawbacks (and I personally don't like them either), but being unable to trust devs is not one of them.
If you can't make monorepos work, what makes you think you can get "microrepos" to work?
At some point, I don't know, maybe when you cross the 100 repos mark, you've gotta ask yourself "maybe we could try a different approach?"
It's not like reddit has been known for its wonderful stability over the years
I'm sure the scale here is completely unlike anything I've ever worked on, but how hard can it be to write a sane implementation of a message board?
I'd be curious how much of this problem is caused by the junk that is "new reddit". I've been there since 2007... The day old.reddit.com is the day I abandon it for good
Hope I’m wrong.
A small part of me actually hopes they kill off old reddit, so I have a reason to completely stop using it :)
It always seemed so perverse to me to promote this sort of tacit approval--as you say--passing it off as "cute" instead of... improving things.
I know they did alot with git to make it manageable, hopefully what ever they did makes it to the open source world eventually so we can all avoid these crazy thousand repo worlds.
I started eyeballing two of them to combine because they were dominating stack trace frames, and after a number of rounds of feature toggle cleanup and refactoring to use other modules, had dwindled down to about 60% of a reasonable module size.
Not a lot of this sort of work gets done.
I’ve worked at companies with large monorepos and I’ve worked on teams with more repos than engineers. At large scale, it takes good tooling to make it work well, and that’s true both for monorepos and for multirepos.
I do think that the monorepo tooling is better, but I think the difference isn’t so large. You can have a completely miserable experience in a monorepo or multirepo setup, or a good experience.
I don't think someone knows what "repository" means.
At least they're bringing in Sourcegraph. That tool's helped me make sense of some chaos. Not 2000 repos' worth of chaos, but still, some chaos.
The nice thing about 2024 is that it’s not 1999. But every once in a while I run into people making 25 year old mistakes.
A lot of the old guard have left the company though and our main product moved from four repos to just one. The threat from the legal team to have enforced OWNERS files — essentially replicating the divisive politics of the old repos but in the monorepo — thankfully withered on the vine. We still audit what goes into each release but it’s no longer part of any active permissions thing. We trust our developers but verify, for legal reasons, that nothing went wrong.
You either want one engineering team to act in unison behind your company’s mission, or you want to live a divisive narrative that you are actually multiple teams “working” together with none of the advantages of living under one roof and all the disadvantages of hard repository boundaries crisscrossing your intellectual property.
So many factors threaten to curdle your team dynamic: multiple offices, multiple floors, work from home hermits, bad management, etc. It’s simply org entropy and it takes much effort to keep the weeds out of the garden. Multiple repositories is one less bullet you can keep out of your feet while fighting all the other battles that threaten to turn your team from 1990s Sun Microsystems into 2010 Sun Microsystems.
