Check out Bazel for Rust.
It allows:
* caching of artifacts.
* shareable caches between {developers, build jobs} based on hashes.
* remote distributed builds (on very many cores).
Something else is that most projects tend to build everything from source, even protobuf dependencies, so it takes me an hour to get the initial build of envoy done.
Any example?
Not nearly as flexible or powerful as Bazel, but also vastly simpler to setup if all you want is caching.
This is not a complaint about Bazel specifically, its fantastic, and easily my favourite build system bar none.
However it cannot cross compile Rust. This means if I’m developing on my MacBook, and I want to compile a Rust binary and put it in an Ubuntu docker container, I can’t do it on my host machine. I need to copy the source into the container and build it there, using multistage builds.
But this is -extremely slow- because it cannot take advantage of Rusts build caching. I’m talking 10-15 minutes for my small Rust project.
Has anyone run into this? How do you work around it?
I’ve considered running a Bazel remote execution server on a local Ubuntu VM, but this feels like so much extra complexity just to use Rust, Bazel and containers.
Apple ld doesn't support Linux as an output target, so you need to use GNU ld or LLVM lld instead.
Code examples at https://john-millikin.com/notes-on-cross-compiling-rust#baze...
I've been using them locally of late, and they're excellent for storing state that's not obviously identical across invocations of the same layer.
There are two common ways i'm familiar with. Cargo Chef and Manually. Manually can be a bit involved, but you basically just build with empty source and prune your fake-lib files from the cache dir. Cargo Chef is very easy.
My docker images build as fast as my local images. Though if you have many crates in a workspace it can start to become difficult.
The problem is that Google is well-known for unceremoniously dumping stuff.
So my first question about any Google tool is: "Does it have enough non-Google people supporting it?" because, if it doesn't, it will die as soon as it becomes a political football.
My second question about any Google tool is: "Are there small projects using it and singing its praises?" Can I use the tool to do what I need to a handful of files after reading a couple of web pages for 15 minutes and have it work. I don't need to scale to 4 gazillion foobars--I do need to get stuff done without an IT staff of 100 people. From a cursory look, Bazel fails that question.
My final question about any tool is more personal: "Does it use C++?" because, if it does, then integrating with anything other than C++ is going to be a gigantic PITA. At this point there are plenty of fine alternatives to C++ that don't suck. If you don't have a clean, pleasant way of talking to things that only understand C library conventions, you fail and I'm moving on.
Per the article: bazel + rules_rust should have the flexibility to override the linker flags that Cargo may take as required since that would be a property of the bazel toolchain used.
It's a nice amalgamation of how cargo works and how bazel works.
In general bazel supports hermetic builds, multiple toolchains, cross complilation, and ways to compile multi-language projects.
I still wish that Cargo.toml didn't support build.rs as it can cause a lot of system-dependent problems that bazel sidesteps entirely by being hermetic.
I need this in my life.
I see the linked resources are to docs only.
If it's any help, here's a bazelized Rust demo implementation:
https://github.com/mihaigalos/code_templates/tree/master/rus...
[1] https://developers.facebook.com/blog/post/2021/07/01/future-...
This sounds like something that nix is optimised for. The inputs into building each package is captured so having different feature flags would just create different artifacts.
I’m all about writing code once. But compiling a few times doesn’t seem like that big of a deal to me?
The article says it runs on “six operating systems” but I can’t find them listed?
Basically every Go executable is dynamically linked against libc these days unless the builder goes seriously out of their way to get a static binary.
> This post describes a proof-of-concept Python executable (2.7.18 and 3.6.14) built on Cosmopolitan Libc, which allows it to run on six different operating systems (Linux, Mac, Windows, NetBSD, FreeBSD, OpenBSD)
It would have been nice if the OP had spent a few words on the motivation here.
So some subset of those
There are two machines that you care about with a compiler: the machine the compiler is running on ("Host"), and the machine the compiler is producing code for ("Target").
Generally we use a compiler with the same Host and Target - if you use Rust on x64-Windows you get a binary that runs on x64-Windows. If you use it on ARM-Linux you get a binary that runs on ARM-Linux. What you are talking about is making a compiler that would run on all Hosts, but it would take different work to make it be able to produce code for all Targets. So you'd produce a compiler that targeted x86-Windows and it would run on x86-Linux but still produce code for x86-Windows. It would also NOT be able to run on ARM-Linux.
[For completeness there's actually three machines we talk about with compilers - in addition to Host and Target there is also "Build". This allows you to cross-compile your compiler. For example you want to build your compiler on x86, you want the resulting compiler to run on ARM, and when it runs it produces code for RISC-V. Here Build is x86, Host is ARM and Target is RISC-V.]
This is already how rustc works, it is not like GCC. Any rustc can (cross-)compile for any target, as long as you have the rlibs for that target, your libllvm has those targets enabled, and you have the appropriate linker. Rustup usually manages all that for you.
I help maintain a kernel in C that runs on nine architectures, some of which don't even have LLVM backends, much less stable rust toolchains.
"Portable" means rather different things. This blog post is focused on the easy stuff.
Cosmopolitan is an incredibly cool project that does more than you think.
I don’t think you comparing your (unlinked and unnamed) kernel to this is very constructive. It feels like you’re gate-keeping.
portable = able to port
In this sense, we cannot say "C is portable"? Since we need to compile it for each platform; the same for java, we need to run the bytecode on each platform's virtual machine, and the compiler/virtual machine isn't built-in along with most os.
Then back to node.js, since v8/bun is not built-in on most os, so we cannot say it's portable right?
Like OP I hit two walls: libunwind, and linking. For libunwind, I ended up downloading/compiling manually; and for linking there is auditwheel[1]. Although it is a Python tool, I did actually end up using it for Ruby (by creating a "fake python package", and then copying the linked dependencies).
It was at that time that I learned about linking for dynamic libraries, patchelf and there is really no single/established tool to do this. I thought there should be something but most people seem to install the dependencies with any certain software. I also found, the hard way, that you still have to deal with gcc/c when working with Rust. It does isolate you from many stuff, but for many things there is no work around.
There is a performance hit to this strategy, however, since shared dynamic libraries will be used by all the running programs that need them; whereas my solution will run its own instance. It made me wonder if wasm will come up with something similar without affecting portability.
Finally, the project is open source and you can browse the code here: https://github.com/pyroscope-io/pyroscope-rs
I’d change a configuration flag, some part of std would break because my
flag was wrong, and I’d learn something new about Rust and how std worked.
The library this is built on does have a bit of a weakness with respect to GUI software https://github.com/jart/cosmopolitan/issues/35 if this can be fixed this will be an amazing tool for building simple cross platform utilities and tools.
Software distribution is by its nature a very platform specific problem; even if we accept the premise of an x64 world, an universal binary solves just a very small portion of the practical problems of portable deployment.
Ironically, the best use case I can imagine is creating an universal binary installer that can run on any Unix system and then proceed to make platform-specific decisions and extract files contained in itself, sort of like Windows binary installers work. But that's an utterly broken distribution model compared to modern package managers.