In the end, I never got things working the way I wanted. The hardest thing was getting Typescript (and worse, VScode) to recognize code across modules. The second hardest thing was getting GraphQL schema types into the frontend and backend. There's a huge ecosystem around GraphQL development, especially if you're using JS/TS, but it's still all so clunky. I ended up using a handful of tools to 1) generate the schema from the backend code, 2) serve it via introspection on the backend dev server (thank goodness for hot reloading), and 3) watch said backend schema and generate static type files for the frontend.
Did it work, sure. Is it elegant and straight forward? Definitely not. What a mess!
Each [1] tool [2] seemed [3] to break differently, and needed some form of manual massaging to make it work. That manual massaging meant learning a new configuration file syntax, multiple times.
When it worked, it felt magical. Weaving together an entire web app, powered by a small bit of GraphQL schema [4] means building at a high level of abstraction (hence can be very productive). The only issue is the muddy forest of the NPM ecosystem you're surrounded by: any step towards upgrading your external dependencies seems to cost far more time than promised.
[1] Yarn3/PnP seems to assume all packages define their dependencies correctly. Unfortunately, this isn't true in the real world. I spent hours massaging dependencies in https://github.com/ThomasRooney/reamplify/blob/master/.yarnr...
[2] Getting TypeScript to work cleanly both in an IDE (IntelliJ) and when imported across backend/frontend packages was really cumbersome: I ended up just emitting .gitignored JS files next to their associated TS.
[3] Whispering into the IDE to make it understand GraphQL required learning the .graphqlconfig syntax, and fine-tuning it.
[4] https://github.com/ThomasRooney/reamplify/blob/master/packag...
import {IUser} from '@shared/interfaces/users/IUser'Thinking on it, not much different than having a protobuf layer generate data objects across apps. Combined with any client generation logic, we are basically there.
Just, realize that these being separated in repo is also a benefit, since that mirrors how they are separated by deployment.
It's honestly embarrassing that modern web development (an incredibly "dumb" network protocol at its core) doesn't have an easier onboarding/development process.
Learna needed a new maintainer and is now moving to Nrwl who develops Nx and seems to be the better choice for future projects?
I set up several projects in a monorepo recently using Yarn 3, and while the end result was awesome, the path to success was dark and weedy.
It seemed I was constantly 95% of the way there, but some detail or another wasn't quite right. Yarn has no solution for this, because the problem is the integration of tooling. eslint, typescript, prettier, jest, relay, you name it. They each want to work together and integrate with each other in different ways, using various sources of truth, and so on. Then your IDE needs to adhere to the same protocols, but various extensions have different opinions about default configurations.
Along the way you really need to know your tooling deeply, or you're in for some suffering.
Relatively simple projects with full buy-in of Yarn 3's tooling are probably pretty easy. Once you get relatively complex though, I don't know, I don't think any of this is easy.
It does sound like Nx handles a bunch of this stuff for you, which I'd love to try out. I really like that Yarn isn't particularly opinionated though. Once I had things set up, there was very little to change or that could break. The scaffolding was fairly plain to see, easy to work with, and we owned it. With something like Nx where things reportedly "just work", I worry it would be an unsettling blend of black boxes and black magic making my tooling and code work... Until it didn't, at which point I'd wish I used something less opinionated.
This is especially valuable when you have a chain of dependencies (A -> B -> C ->... Z). With package-based dependencies living in individual repos, this is tedious work - branch A, modify package.json, build, branch B, update A version to new build, give B new version number, build,..., branch Z. Submodules don't particularly help.
In contrast, with a monorepo and dependencies based on your local copies, you just create a new branch of the repo, and now everything is automatically pointing to the branched code. The benefits can be major.
If you also integrate something to generate nice branched version numbers, things living outside the monorepo won't even notice - you branch, then you start a new build for each library, and any external user can get the new version, while you still get the simple branching support.
I have seen several articles addressing the folder/file structure. The author has done an excellent job and taken it a bit further. Now I have questions to research about deployment. What are best practice for isolation of code for Docker images which are to become k8s Deployments and Pods? Does it matter to stuff it all into a mega image? Are Monoimages a thing and what susceptibilities to performance do these yield?
If you're fine with apps using an older version of the library then you don't need this. However, you might want to think about what you'd do about a security bug when many apps are using old versions of a library and it would take a lot of work to bring them up to date.
For the hobbyist development I do now, I'm fine with old apps using out of date libraries. I'm mostly not maintaining the apps and I'm not writing shared libraries.
- library - app-a - app-b
and you need to make a change to `library` to support some new thing in `app-a`. If you can publish a new version of `library` and update `app-a` to use it, it's really easy to make a change that's incompatible with `app-b`. Even with a comprehensive `library` test suite, it's easy for Hyrum's law to make an appearance and now you've got some unexpected corner case that's depended on.
With a monorepo, you can immediately see `app-b`'s tests failing and either fix the usage, or re-think your `library` changes.
that said, monorepos solve a certain set of real problems for organizations by allowing contributions across code bases from one VCS repository, and by reframing the release/deploy pipeline around singular atomic refs. the impact is multiplicative and happens in downstream tooling and processes. they squash some problems in one place in the org (product development) that bubble up elsewhere (devops/release engineering). depending on the org this is a sensible/cost effective approach.
submodules don’t really accomplish either of those things, unless I’m missing something about your workflow.
Generally with submodules we run into issues like each dev having to maintain the setup on their own machine, and unique commands to work with module repositories.
For WAY more writeup: https://codingkilledthecat.wordpress.com/2012/04/28/why-your...
Whole tools were created to get around these issues (git-subtree for ex).
I really feel there needs to be like a git --pull-config option or something to pull all project configs (including submodules, commit hooks, etc). Or perhaps move those things into the top-level folder and allow them to be git-add'd.
If you are writing a sufficiently large application you are just better off switching to a mature tech stack than dealing with how awful the TS ecosystem is.
Ideally something with a good build system, good incremental compilation, proper test framework integration (so tests only run when input classes/objects are changed) etc.
Arguably Golang is decent here but it's not my cup of tea for other reasons but you do have to admit it has a very fast compiler and the way it's package system works makes for good incremental build support.
.NET has always been very good in this regard.
Rust is a bit slower on the compiler end but it's still very good, incremental support is good etc.
All of these languages can also either produce code that works on all targets (JVM/.NET) or cross-compile natively (Go, Rust). This matters for packaging and deployment as half of devs use MacOS but deployment target is usually Linux and increasingly containers. Being able to construct Docker containers directly from artifacts without needing Dockerfiles is a huge win and all of the above languages support that via one tool or another (Bazel, Jib, etc).
Literally any of these blow TS out of the water for DX, performance and tooling. Unless you are chained to TS for browser reasons or isomorphic code requirements it's just not worth it on the server.
I do not like the prospect of having to use a TS/JS specific build tool, because of wanting to use monorepo, but fortunately, I did not yet have to do that, as I only ever developed extensions, and did not fork JupyterLab, to change anything core. Lerna and the whole setup brimborium is definitely something that scares me away from even trying to change things in the core.
This way, we can use babel-node or ts-node to transparently run our TS files, no transpilation needed.
Wouldn’t that mean the shared packages tsconfigs aren’t respected if you changed something like strict options? And also that a clean build of the whole monorepo is going to recompile each shared file for every app project rather than just once?
Moving forward, I'm going to use wireit for these things. Pure modules get built with tsc. At the highest level (e.g. where it needs to be embedded in a page), make a bundle with rollup.
wireit has two nice properties: incremental building and file-system-level dependencies. Within a repo, you can depend on ../package-b. However, if you have multiple monorepos that often get used together, you can also depend on ../../other-project/packages/package-b. No matter where in your tree you make changes, wireit knows when to make a new bundle.
I've just started with wireit (it was only launched recently), but it seems to be a nice solution to wrangling dependencies between related JS libraries.
https://github.com/pnpm/pnpm/blob/main/.meta-updater/src/ind...
I think that a well structured monorepo might make a move away from all-encompassing full-stack frameworks and plugins to libraries, tools and special purpose frameworks easier to get close to a best of both worlds situation.
The background is deploying up to a dozen or less new sites and apps per year as a small team while continuously maintaining the old ones and wanting to merge in new improvements found in newer development.
---
Rationale:
Big web frameworks and similar give you per-project productivity, structure and are batteries included, but come with downsides, such as less control, less flexibility, unneeded complexity and abstraction, legacy cruft and gotchas, generally poor performance that you "fix" with caching where you can etc. You end up patching over things with overrides, workarounds, and strip functionality that gets in the way, and in some cases you bypass the framework entirely. You are generally more dependent on framework specific solutions instead of general solutions and can often not do things from first principles without considering the hairball of integration issues that often comes with it.
On the other end of the spectrum you have the possibility stich something together with specialized tools, libraries etc. But you can easily get into the danger of inventing your own framework because you want that common structure. Also once you found some good ways to do something you want to enable straight forward reuse and maintainence. Refactorings, regression testing, performance improvements and possibly new features should benefit everything as easily as possible. All of this is _hard_ if your codebase is spread across many repos, primarily because you don't have a hollistic workspace that helps with these structural changes.
---
My hope is that my learnings and experiments with monorepos lead to a way out, so we can make incremental, cross cutting changes with more confidence and faster feedback loops.
Does anything of the above sound familiar to you? Or do you think I might be looking at this the wrong way?
I don’t think it caught much traction; sadly people seem to prefer doing the easy thing over the simple thing.
npx create-tamagui-app@latest
[0] https://github.com/tamagui/starters/tree/main/next-expo-soli...
We have a monorepo with about equal gigantic parts Rust & TypeScript. The Rust part builds ~1100 crates in ~8 minutes, and runs all the unit tests after another ~9 min. (~17 min total.) The yarn install part of our CI takes ~39 minutes. (Not really sure how to do a "# of crates" style comparison.) (And at 39 minutes, this is very much on CI's critical path. The Rust stuff … isn't. The irony of a compiled language beating the pants off one that is only sort-of isn't lost on us.)
You can save the contents of the yarn cache directory between builds. Set it with the YARN_CACHE_FOLDER environment variable.
Build an intermediate docker image with all packages built. Later images can be based from that and don't need to rebuild.
This is true for any programming language. (Also, successful monorepos can be polyglot.)
If you don't have a dedicated team, you will eventually end up with all the downsides of a monorepo and few of the benefits. Builds will break frequently, impacting many teams. Dependency management will become a nightmare.
Open-source tooling like Bazel will only get you so far -- you will need in-house tooling too, but more than that, you will need an in-house culture of behaving well in a monorepo. Unless most of your engineers have done it before, you will need strong leadership to build that culture.
If you can't dedicate a team to that purpose and really follow through with it, then don't even try having a monorepo. Do a repo per team, or a repo per project.
Or is monorepo more of a "place to put all the code" not necessarily correct or working.
I like multiple repos because it's easier to assume that the main branch of each is "correct and tested and excellent quality".
I tried https://nx.dev/ in the past and it helped with many things. You should check it out.
A monorepo is not just a bunch of projects thrown together into one repo. It's the philosophy of having all code of a bigger organization in one repository.
When smaller teams inside of companies start creating "monorepos" for a hand full of projects, they end up with many "monorepos". This approach combines the worst of both worlds: you get the tooling complexity and scalability problems of monorepo combined with the inability to make atomic changes over multiple projects. You get none of the benefits.
If you are thinking about moving to a monorepo, do it in a way that
- has everything required to build a deployable unit into the repo, no dependencies to other repos
- under no circumstances have code in another repo depend on code inside the monorepo
- avoids ending up with dozens of monorepos
The problem with my plan is that I know from experience that making changes to the shared Docker layer is a pain in the ass to get it to propagate across your other projects as you're developing it, at first. Once you learn the incantations to chant, it's quick.
I just don't want to have to teach my team the incantations. It takes time!
If we get another round of funding and/or I find out I'll need to care about this project for more than a few additional months, I will likely make the switch, but at this point I can probably white knuckle my way into whatever exit we end up with.
And honestly, I think this is how the decision should be made; entirely dependent on a) your team and b) your anticipated future state of your work environment. No right answers here, just more or less complex ones with better or worse tradeoffs.
AFAIK this is due to some limit in TypeScript’s project references. It’s not possible to add a typing lib to a particular package without the checker merging all global namespaces.
We use NPM v8 for packages. Yarn v1 is falling behind and v2 is on a different planet. The other package managers have too much trickery.
Turborepo is fast, simple to use, and very active at the moment.
NPM v8 with Turborepo has eliminated all of our custom monorepo tooling.
I swear by pnpm + rush (from Microsoft). Fast installs. Good caching. Keeps every dependency in sync. Handles the local workspace builds well if you buy into their build tool, heft (which I have).
Eg throw out eslint and prettier.