Build time is important. It's one of the questions in the Joel Test:
If the process takes any more than one step, it is prone to errors. And when you get closer to shipping, you want to have a very fast cycle of fixing the “last” bug, making the final EXEs, etc. If it takes 20 steps to compile the code, run the installation builder, etc., you’re going to go crazy and you’re going to make silly mistakes.
Our build process looked like this when I started:
While this worked well when we were starting out, and the team was small, we eventually found three big problems with this setup:
Just before I started at Replay I read about and tried Earthly, a tool that purports to solve these problems. I decided to put it through its paces by throwing our codebase at it. Here's what I found.
This is Earthly's headlining feature and it delivers. Everything in Earthly runs in containers, with deterministic Dockerfile-like instructions. For an example, here's one of our Earthly rules that builds one of our TypeScript apps:
webpack-backend: FROM +deps COPY src/build/webpack.config.ts ./build/ COPY --dir src/control src/dispatch src/host src/instance src/kube src/channel src/processing src/protocol src/shared src/fuzzer ./ RUN ./node_modules/.bin/ts-node ./node_modules/webpack/bin/webpack --config build/webpack.config.ts RUN chmod +rw /out/*.js SAVE ARTIFACT /out
Earthly will use the version of
At first running our builds in containers seemed to make things slower because we use macOS for local development and Docker on macOS is slow. However, because Earthly analyzes the structure of your build it can run unrelated things in parallel and even skip doing work altogether with caching. This more than makes up for the Docker overhead.
For instance: our C++ builds and TypeScript builds are mostly unrelated, except that they both need to be done before we can build Docker images. Earthly knows this and runs them in parallel.
The result is that, while first builds in Earthly are much slower than our
| Command | No Code Changes | Code Changes |
| --- | --- | --- |
Still, 57 seconds to get a Docker image when everything is cached felt like a long time. It turns out that as much as 15 seconds of Earthly build times is devoted to exporting the image from Earthly in to your laptop's Docker daemon. We've opened an issue to figure out how to improve this.
The fastest, most reproducibile build isn't worth that much if you need to be an expert in the build system to change it.
Changes to the build system happen all the time, especially when you want caching and reproducibility. In order to make things cacheable, you need to tell the build tool about a thing's dependencies. Earthly makes that easy. In fact, most of our engineers already knew how to do write it: it's just Dockerfiles!
If we look at our previous TypeScript example it's easy to see what the dependencies are:
webpack-backend: ### Begin dependencies COPY src/build/webpack.config.ts ./build/ COPY --dir src/control src/dispatch src/host src/instance src/kube src/channel src/processing src/protocol src/shared src/fuzzer ./ ### End dependencies RUN ./node_modules/.bin/ts-node ./node_modules/webpack/bin/webpack --config build/webpack.config.ts RUN chmod +rw /out/*.js SAVE ARTIFACT /out
If you're familiar with Docker, it's easy to imagine how you'd change this build rule to add your new code:
--- a/Earthfile +++ b/Earthfile @@ -29,6 +29,7 @@ webpack: COPY src/build/webpack.config.ts ./build/ COPY --dir src/control src/dispatch src/host src/instance src/kube src/channel src/processing src/protocol src/shared src/fuzzer ./ + COPY src/myThing . RUN ./node_modules/.bin/ts-node ./node_modules/webpack/bin/webpack --config build/webpack.config.ts RUN chmod +rw /out/*.js SAVE ARTIFACT /out
Additionally Earthly gives structured output when running so it's easy to see where failures happen:
What failed? that
Earthly has worked so well for us that we've begun to use it for everything. Earthly has become our development "menu": want to run a test? It's in Earthly. Want to do a deploy? It's in Earthly. Want to run the autoformatter? You get the idea.
There is still room for improvement. A lot of our C++ builds are still done in node.js scripts. So while Earthly helps cache those things, if their dependencies change, we build the entire binary with no caching of intermediate artifacts. It would be great to use something like the user defined commands proposal to create a generic target that we can use to build any given C++ file, with a linker target that waits for all those intermediate artifacts to exist before linking them together.
But that's my favorite part about Earthly: we didn't have to bite off that complexity immediately in order to start getting some of the benefits. All in due time.
Replay is the first tool that lets you record your application and inspect it later! If that sounds interesting to you we're looking for beta testers.