Yes, it's complicated, but it's also quite rigorous, and the rigor pays off.
(We at Square had already found a Blaze-alike necessary. We are currently busy converting our Java build from Pants to Bazel.)
Both approaches take a huge amount of work and tooling.
The big selling point of a monorepo is that the time and effort taken to follow strict versioning and upgrade discipline for multiple interdependent projects can be somewhat avoided. On the code side.
Essentially: version skew across numerous artifacts in a large organization starts to look like the version skew across an industry or ecosystem. The aggregate cost of dealing with it project by project is probably higher, at least that is what most of the biggest tech companies have concluded, than dealing with it at the source level using a monorepo and single-version policy.
Having looked seriously at both options, I think the monorepo world is the right one, but it presently lacks good tooling to sanely model your dependency graph AND create custom build rules while still being affordable for small or medium-sized orgs. Git/hub simply isn’t designed for this kind of modeling and everything I’ve seen built atop it is either way too manual or a kludge. Maybe the “kludge” solutions are actually reasonable, but my confidence is low.
Bazel is the right idea, but it’s execution disappoints. The documentation is abysmal, last I checked they advertised Python 3 support, but it’s been broken for years with no signs of progress. Building custom rules also looked hopelessly complex (by which I mean, “not something our organization can afford to implement and maintain”) but maybe there’s some undocumented happy path that I’m missing out on? These things seem easy enough to implement. We’re using Pants right now, and for it’s many similar problems (bugs, documentation, poor code base, difficult extensibility), it at least does a passable job at building Python projects.
I’ve thought about it a fair amount, and I think it’s reasonable to build something simpler that might not meet Google’s use case, but would at least enable small and medium sized shops to play the monorepo game.
we don't use bazel (yet), because dotnet is not that supported.
You will soon experience:
- dependencies hell due to transitive and conflicting dependencies
- one back-incompatible change in some obscure library end up breaking some other unknown service that happens to transitively depend and it
- the entire codebase will become a mess due to inconsistent code styles and formatting because hey we are developers and we can never agree on anything. Thus each team lead will have its own opinion
- each team will have to maintain its own CI/CD jobs
- heterogeneous builds: maven, node, sbt, webpack, etc ...
the list goes on ...
All (or most of) this mess is solved by centralizing the codebase in a monorepo.
Using Bazel with external packages on the other hand is one of the most tedious and frustrating endeavors imaginable. If you can vendor all of your source it's much less frustrating. This is extremely manageable in the C and C++ worlds where there aren't really any package mangers and you end up needing to do that anyway.
Edit: I don't want to come off as too negative about Bazel. But I really think it needs more time and is nowhere near something that I would call a 1.0 let alone a 2.0
I’ve been using it for the last 2 years, and I am not doing any project again without it.
Bazel is not that complex if you start a project with it. Migrating to it and learning it at the same time will be hard though since you’re likely to uncover a lot of skeletons.
For very large organizations with the capability of assigning one or more teams to tooling it may very well be the right choice.
But which one? Are there any other (non blaze-like) build systems enabling hermetic (possibly remote) builds and caching?
If you don't need these properties and have a mono language project, the language's native build system sure fits and may be a better choice.
You can of course try to use the native solution on each but that makes it more difficult for people to jump between projects/languages as the syntax for describing the build changes. Moreover for centralized build infra this becomes more difficult to orchestrate/co-ordinate because now you have to add remote caching/parallel compilation & whatnot to multiple places (with all the associated challenges of trying to upstream the same set of logical changes into many different projects with their own maintenance schedules/philosophies).
If you have a small project that you can rebuild in a couple of minutes, Bazel is probably an overkill.
I feel like I see the pattern of people on HN being disappointed in some of the tools that come out of Google and other large engineering orgs, when they don't work out really well in orgs that are not operating at the same scale. People have similar complaints about the complexity of other projects that come out of Google. K8s comes to mind as one such example. Often times these tools must be robust to such a large variety of uses that they are simply overkill for smaller organizations. I'll readily admit that I could be wrong and Bazel is simply poorly designed, but it is perhaps worth considering that the build system used by an engineering team of 50 need not be as complex as the build system used by one of the largest engineering orgs in the world. My guess is we'd see a lot less backlash if people tried to step off the hype train for a moment and critically evaluate whether they really need to use something like Bazel or K8s when something simpler would suffice.
> Build and test software of any size, quickly and reliably
Given the comments here, a tagline focusing on its strengths for large orgs/projects probably would be better marketing.
I don't blame the bazel authors, but the development process it was designed for is not the development process of 99% of companies out there. Maintaining BUILD files for all of your vendored dependencies is expensive for your company. You need a full time team working on it.
Unless hermeticity and build correctness issues are absolutely killing your team's productivity (and at a certain size, they might be!) think twice before adding bazel to your maintenance overheads. You can always move to it later if you need it, and it might be more stable and have a better 3rd party package story by then
The net result is a Bazel (and Blaze) that are less burdened by the baggage of legacy, but the cost is a faster treadmill to keep pace with changes.
(I work on Bazel.)
This is accurate. A few of the biggest breaking change themes are:
1) Converting functionality linked within the Bazel binary into the extensibility mechanism implemented in Starlark. An example includes converting the native Java, C++, Android, Python, Protobuf, Obj-C and packaging rules into rules_java, rules_cc, etc. Many languages now are already implemented exclusively in Starlark. See rules_scala, rules_rust, rules_go and rules_haskell.
2) Starlark and Build API cleanups that accumulated over organic growth and development within Google for the past decade.
3) New build system features to support seamless integration with other build systems and package managers.
That it can't differentiate between those two cases is because it's not meant to. It's like complaining that the blurb of a novel is "next to useless" because it doesn't tell you the complete story in a detailed way over several hundred pages.
Theoretically every version change can introduce a bug, which leads to an implicit API change and as such require being a major version bump.
Also, fixing a bug can also introduce an API change, because the API can behave differently with and without the bug.
SemVer just covers the intent, not what's actually happening, which makes it kinda useless in most scenarios. I guess Elm gets it right, tho'.
Hopefully in the coming years something will eclipse semver which solves both problems sufficiently. I don't know of any candidates offhand though.
You could do something like "LoC from last version" + SemVer.
So, 1.2.3k to indicate Major 1, minor 2, 3k lines of code changed from 1.1. It would also possibly be a good way to say "2.0.3" meaning, we moved from 1.2 to 2.0, but only changed 3 lines of code. The breaking change is likely not going to affect you but it is there.
That might make magnitude changes easier to communicate.
I'm not sure how useful this would be for automated build tools though. Would you set bounds on how far drift would go before automated updates?
This is a bizarre naming. They could call it Blaze installer 2.0 for Blaze 1.
This is how Bazel rolls out incompatible changes:
- Introduce a new behavior behind a flag
- Wait
- If there was no push back (and key projects could migrate), flip the flag to enable the new behavior by default.
The goal is to give an opportunity for users to update their code in advance, and get feedback about migration issues.
- Do you have 200+ developers working on a monorepo?
- Are you willing to vendor all of your dependencies and maintain their builds yourself?
If so, consider it. The productivity you're losing to unnecessary rebuilding and re-running unchanged unit tests will probably be paid back if you can contort your development process to the one Bazel expects.
If you're a small shop, the benefits Bazel is going to provide over, say, Make (or whatever standard build system your primary language uses), are going to be minimal. And the overhead of maintaining Bazel is going to cost you a ton of developer time you may not be able to afford.
Gazelle is wonderful and it doesn’t belong in Bazel core. Bazel is a build system for every language, and Gazelle is for a subset of Go developers. Since it’s not part of Bazel core, you can always replace it with something else.
Given that the Python community is really oriented around the PyPi registry and pip packaging tool having a good Bazel-native packaging solution is near essential, but right now it’s not quite there.
Provenance is a word I first saw advertised in a platform called dotscience.io — that I find fundamentally interesting. And it seems quite relevant to hermetic builds.
Provenance is about giving any state derived from an arbitrary computation an identity that is derived from the content hash of the inputs needed to re-compute that state ... in dotscience they achieve this by instrumenting io and creating zfs filesystem snapshots when computing new provenance artifacts.
I think this concept could be the ultimate building block for a build system — and it could become the job of oses/containers/runtimes/databases to Coordinate to allow this abstraction to be tracked with sufficient efficiency that programmers would feel allowed to freely use the concept of provenance when building ... it seems to me like provenance could provide all the information needed to support a distributed build cache? You wouldn’t actually need a build language at all — just an api in each language to ask for the saving of provenance artifacts. The artifact would hold all the info needed to be able to recompute the artifact with the same state — which is also all the info needed to decide when the artifact is out of date ...?
I wonder what's the real usage of Bazel (not Blaze) in Google.
Edit: I'm referring to building the framework itself (e.g. to contribute a fix). Building an Angular project with the CLI works quite well.
<script type="module"> import Vue from 'https://unpkg.com/vue@2.6.0/dist/vue.esm.browser.min.js'; new Vue({ ... }); </script>
It is utter hell when you have tons of third party libraries (internal or external to the company) that you don’t have the source to and it is especially painful when trying to integrate bazels behavior against other build systems. Also bazels packaging and use of internal symlink renaming was a constant source of suffering. Bazel pretty much destroys a number of totally valid work linux commands for looking for so files.
Bazel might be useful in the case of a monorepo with a massive engineering pool AND a massive cloud infrastructure backing that repo to handle all the artifact sharing, but after having used cmake, premake, waf, random perl and ruby scripts, or just checking vs projects into perforce manually, I’d pick any of those before bazel for most projects. I say that having worked on code bases from a few 10s of thousands to 25+ million LoC with teams small, large, and distributed.
Bazel probably has its place but I have yet to find it.
That said, the major sore point with Bazel for me is the general lack of expertise about how to work with it sanely. Depending on what part you’re looking at, it’s somehow both “too opinionated” and “too flexible” at the same time.
I think it will capture a big chunk of the mindshare for build systems over the next few years, and you’ll see more and more of it. Over that time, people will develop the expertise and best practices for different development problems.
For managing third-party dependencies specifically, Bazel gives you a ton of options, including options that only really make sense for huge orgs like Google. Google vendors their third-party libraries directly into the monorepo. If that doesn’t make sense for your org, Bazel lets you work with external Git repos, with artifact repositories, with package repositories like NPM, or with tools like pkg-config.
The thing that makes this hell, right now, is that few people how to use it well and the documentation is rough. I’m personally very happy with it, even for small codebases, but I’ve used it a lot.
Also building distributable packages with bazel never seemed to work well due to the constant aliasing of so files. Things that would work in the direct bazel build would fail in packages and vice versa so now we had even more pain.
Trying to suck up just header files and multiple so files was always arcane bullshit as well.
We did work with git and other such functionality, but if you had to build a package from another build system to bring into bazel there were always annoying pain points.
Also bazel managed to bring in implicit dependencies in our system so that clearly isn’t something bazel magically handles but was rather a product of your expertise.
After reading build systems ala carte I am just more convinced bazel is not the build system that I would really ever need. I’m not sure that build system exists yet to be honest :). But in the work I do other systems solve my problems better.
With that, we end up paying so much extra time building everything over and over without need, and then not building things that we ought to.
So that's one reason to switch, but at the same time lots of people simply do not get it. To them it seems intrusive, new, opinionated, and makes them not happy to use it. I've used it for 2+ years at google, and yes initially - was WTF is this? Then it hit me... And I'm sure the same is for buck, pants, please.build, gn and other similar systems.
At the end of the day, you need way to express "end to end" your build graph, from any single individual source file, shell script, or configuration downto building your executables, deploying them, etc.
It's an industry tool, that needs to be looked, and if it takes 5 people to support it, then it takes 5 people to support it, but you won't be wasting other peeople's time on issues like - "Why this build in the CI did not trigger?", why it takes, and wastes my time (waiting for presubmit), etc.
Yes, it does not come for free, but it's worth knowing and trying it out at least.
If nothing else, here is the takeaway - Try to use a system with static graph, where relationships are known before you start building things. It's not always there, e.g. your #include "header.h" file is dynamic, but bazel forces you to express even that, and later it catches it whether you've done it, and breaks unless it's fixed.
There’s an exercise you can do where you design a build system on the basis that it shouldn't do unnecessary work (which can be very slow and frustrating in practice).
My personal experience is that you can really quickly get to the point where just reading the entire graph into memory gets expensive. People talk about how Google is huge… but long before you get to that scale, you can end up with a build graph that just takes forever to parse and evaluate. (At Google's scale, it doesn't even fit in memory any more.)
So you decide that, as a hard design requirement, you should be able to only load the portion of the repository that you are building. And then you want to make this cacheable, so you can change the repository and know what’s changed in some quick / reasonable way.
If you go down this path, you end up rediscovering some of the big design decisions behind Bazel, Buck, Pants, Please, and GN.
I was wondering how to make sure Bazel doesn’t rebuild something it has built previously? (Caching)
I'm always weary of build tools that try to do multiple languages. On Scala projects I use SBT, and for people who have tried to hack on SBT itself or its plugins, you know it's a big mess under there. On other projects I've tried using Gradle with Scala, but I found a lot of times Gradle just wasn't setup for a Scala workflow or was missing essential tooling to make it as effective as SBT (although its configuration is considerably more sane). Most of the tooling and plugins around Scala are built around SBT as well (for better or for worse).
I try to stick with the major tool for a given language; cargo for Rust, SBT for Scala, the built-in tooling for go, with the exception of Java projects where I'd gladly take gradle over the hellscape that is Maven.
That can be fine but can make it even more of an efficiency loss for someone switching projects/contributing partially to another project. Uniformity reduces costs on many fronts but like anything else it's a tradeoff. Now you need a team to maintain your Bazel/Buck/etc for each language & it may not jive 100% well with languages that have opinionated package managers/build systems alread (Node, Cargo, SBT, etc). On the other hand you'd probably end up having to create teams to maintain your company's Node, Cargo, SBT builds anyway except now you need to hire domain experts who not only understand each language but also how it should integrate within your larger infrastructure. A single uniform build system framework makes that easier.
> Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users.
I don't think people will ever fundamentally all agree on:
- static vs dynamic configuration
- custom language vs piggy back on existing
- intelligent, deeply integrated / understands code
it is building vs "language agnostic" but
necessarily shallow integration