Earthly is a modern build solution that is similar to a sandboxed makefile. It works with your existing CI and the #1 reason people use it today is to able to reproduce a CI build locally.
Today we promoted a number of important features to GA status in our 0.6 release including:
- WITH DOCKER : Earthly can execute any number of containers in parallel, via an isolated Docker daemon that it starts up within.
- User-Defined Commands: Extract common repetitive build steps into a command you can use across your projects.
- Shared Cache: Earthly v0.6 now provides shared caching. This extends our existing build caching to work in ephemeral CI runners by storing the cache in regular container registries.
Let me know what you think! Feature requests always welcome :)
One thing I haven’t gotten right is caching. Some of my builds have targets they depend on that are very expensive but rarely change, and I haven’t been able to make sense of when those targets are (re)computed and when they aren’t. This results in many builds taking a long time, when the targets could be reused from a cache.
Earthly is also missing something that I find to be a glaring omission from make—give me a command to simply list the targets in an Earthfile. ‘earthly ls’? Maybe have a column or two that indicates whether the target saves an image or artifact. When building a project I didn’t author, it’d be so helpful for the targets to be discoverable.
Private would not be callable from command line and not be listed in `earthly ls`?
Cache misses can be a bit inscrutable. It could be the buildkit GC is running, because disk space is getting scarce, or that some arg or file change caused the cache to be considered invalid.
Caching is an area we will continue to improve. We have a proposal for extended cache mounts here[1].
Thanks for using earthly!
I think some of my cache misses were a rookie mistake: I was building a few expensive targets on a `FROM ...:latest` base. The base image changed often enough to make the cache misses annoying, but infrequent enough to make me think it was GC. My builds are much happier now! :)
> This helps avoid cases where tweaks to the CI script take long interaction cycles to test (git commit -m "try again..." anyone?)
Now you have my attention. I haven't done much CI work and the long iteration times are the main reason why. I have to edit a file, commit, push, wait for CI to start, wait for CI to complete, search through 10k lines of logs to find the error, repeat... seriously? This is the best we can do? I assumed that I was missing something.
I should be able to just `gh run-ci` in a repository on my machine. Or take one of those fancy online dev environments that everyone's making now and specialize it for editing CI files. Open an editor, let me click Run to run CI on-demand, have live error messages for syntax and missing variables, show a pane with a description of the CI jobs, show what config options are available for each job as it's being edited. Then once the changes are working I can commit and open a PR.
You can see this in action with the Elixir Phoenix framework. They have an earthfile[1] and use github actions[2] to run it for CI.
Even if you don't use Earthly, using a vendor neutral format to describe your build process can help with iteration time.
[1] https://github.com/phoenixframework/phoenix/blob/master/Eart...
[2] https://github.com/phoenixframework/phoenix/actions/runs/146...
Bookmarked for later. Glad someone is thinking about this problem space!
Just my opinions.
Unfortunately, we rejected it at our organization due to how verbose the commands are. I want my users to be able to say `earthly plan-stage` for a terraform repo, and for it to work. But you cannot.
You cannot have earthly automatically use environment variables from your env, you have to explicitly list the commands yourself using flags:
$ earthly \
--build-arg AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
--build-arg AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
--build-arg AWS_DEFAULT_REGION=us-west-2 \
--build-arg AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \
--build-arg AWS_SECURITY_TOKEN=$AWS_SECURITY_TOKEN \
+plan-stage
Compare this to `task plan-stage` or `make plan-stage`.
It is much too verbose and will only get worse as we add features to our pipelines.
earthly \
--secret-file config=/home/user/.aws/config \
+build
Earthly seems very useful to work around issues like cross-compiling. A simple example that I'd expect a lot of people to run into is running into libc/stdlibc++ differences with Alpine containers. That is easy to come across when doing something as simple as pulling in a Python pip dependency that wants to compile native C/C++ code. I find this extremely difficult to learn to do in Bazel.
However if you're in the league of complex cross-compiling, then you may already be on CMake and/or Bazel and have the dev resources to define the toolchains as needed.
I think you’re better of using something like taskfile w/ earthly for a really good workflow, it gives you the best of both worlds.
(Actually I think I saw this on their slack)
They sell it as a make replacement. Bazel constantly says "you don't need hazel, it is a big tool" and earthly does not. Which is fine.
Either way, I already said earthly is a great tool. This is just my feedback around why it doesnt fit my use case.
Worth watching the reported issue at https://github.com/earthly/earthly/issues/707
Thumbs-ups on tickets and comments help us prioritize features.
We already have a PR up for the `earthly ls` feature.
Dockerfiles are reason #1 for docker's popularity.
Shipping 'the lot' so the artifact works everywhere is #2. This is incredibly inefficient for most use cases.
`docker pull random-shit` is #3.
Everything else it gives is largely immaterial, or maybe even a negative.
You guys are solving #1 and #3, and providing a better alternative to #2. I'm very excited for you and your project!
I've actually been looking into nixos to generate docker containers after that repl post the other day.
Aside from the software itself I wanted to compliment you on this image.
https://earthly.dev/blog/assets/images/earthly-v0-6/earthly-...
I really appreciate how much context is added there for what earthly actually does or is trying to do. I'll probably save that just to have it as an example to ask people to recreate when I'm baffled by their landing page.
https://docs.earthly.dev/basics
https://github.com/earthly/earthly/tree/main/examples
Do those help?
Having done a lot of work in the past with Mesos and Docker and having done the deep dive on Earthly I just don't see the corollary between Mesos and this. Maybe there's more context you can provide :)
