Using Earthly To Build Replay

2020-12-30

Using Earthly to Build Replay

Late last year I started working at Replay. We build a JavaScript debugger that is offered as a website. There's several different components, some in C++ and others in TypeScript, that go in to making this work. The first problem I took on was fixing our slow, error-prone build process.

Build time is important. It's one of the questions in the Joel Test:

If the process takes any more than one step, it is prone to errors. And when you get closer to shipping, you want to have a very fast cycle of fixing the “last” bug, making the final EXEs, etc. If it takes 20 steps to compile the code, run the installation builder, etc., you’re going to go crazy and you’re going to make silly mistakes.

Our build process looked like this when I started:

Manually ensure that some repos are neighbors in a directory
Run node build in one of them
Run the docker image that creates
Replay is running on your laptop!

While this worked well when we were starting out, and the team was small, we eventually found three big problems with this setup:

Reproducibility: Non-determinism could be introduced in to this build by having different versions of the other repos checked out and different versions of node/npm/libraries installed.
Speed: While node build called Docker to do some things, which does some caching, no caching was done when building the C++ code. All binaries were compiled from scratch each time.
Understandable: Editing the node build script required deep knowledge of both the TypeScript code and the C++ code. It was hard to know what to change if you were familiar with one or the other. If something went wrong it was hard to know what failed, and which build it was a part of.

Just before I started at Replay I read about and tried Earthly, a tool that purports to solve these problems. I decided to put it through its paces by throwing our codebase at it. Here's what I found.

Reproducibility

This is Earthly's headlining feature and it delivers. Everything in Earthly runs in containers, with deterministic Dockerfile-like instructions. For an example, here's one of our Earthly rules that builds one of our TypeScript apps:

webpack-backend:
    FROM +deps
    COPY src/build/webpack.config.ts ./build/
    COPY --dir src/control src/dispatch src/host src/instance src/kube src/channel src/processing src/protocol src/shared src/fuzzer ./
    RUN ./node_modules/.bin/ts-node ./node_modules/webpack/bin/webpack --config build/webpack.config.ts
    RUN chmod +rw /out/*.js
    SAVE ARTIFACT /out

Earthly will use the version of ts-node installed in the container to run the webpack build, and it will be the same for every engineer.

Speed

At first running our builds in containers seemed to make things slower because we use macOS for local development and Docker on macOS is slow. However, because Earthly analyzes the structure of your build it can run unrelated things in parallel and even skip doing work altogether with caching. This more than makes up for the Docker overhead.

For instance: our C++ builds and TypeScript builds are mostly unrelated, except that they both need to be done before we can build Docker images. Earthly knows this and runs them in parallel.

The result is that, while first builds in Earthly are much slower than our node build script, subsequent builds were faster. And, by breaking out our build in to smaller component parts, we drastically improved the time it takes for an engineer to see the result of their change.

Command	No Code Changes	Code Changes
`node build`	1m30s	1m34s
`earth +all`	1m8s	3m14s
`earth +service`	15s	57s

Still, 57 seconds to get a Docker image when everything is cached felt like a long time. It turns out that as much as 15 seconds of Earthly build times is devoted to exporting the image from Earthly in to your laptop's Docker daemon. We've opened an issue to figure out how to improve this.

Understandable

The fastest, most reproducibile build isn't worth that much if you need to be an expert in the build system to change it.

Changes to the build system happen all the time, especially when you want caching and reproducibility. In order to make things cacheable, you need to tell the build tool about a thing's dependencies. Earthly makes that easy. In fact, most of our engineers already knew how to do write it: it's just Dockerfiles!

If we look at our previous TypeScript example it's easy to see what the dependencies are:

webpack-backend:
    ### Begin dependencies
    COPY src/build/webpack.config.ts ./build/
    COPY --dir src/control src/dispatch src/host src/instance src/kube src/channel src/processing src/protocol src/shared src/fuzzer ./
    ### End dependencies

    RUN ./node_modules/.bin/ts-node ./node_modules/webpack/bin/webpack --config build/webpack.config.ts
    RUN chmod +rw /out/*.js
    SAVE ARTIFACT /out

If you're familiar with Docker, it's easy to imagine how you'd change this build rule to add your new code:

--- a/Earthfile
+++ b/Earthfile
@@ -29,6 +29,7 @@ webpack:
     COPY src/build/webpack.config.ts ./build/
     COPY --dir src/control src/dispatch src/host src/instance src/kube src/channel src/processing src/protocol src/shared src/fuzzer ./
+    COPY src/myThing .
     RUN ./node_modules/.bin/ts-node ./node_modules/webpack/bin/webpack --config build/webpack.config.ts
     RUN chmod +rw /out/*.js
     SAVE ARTIFACT /out

Additionally Earthly gives structured output when running so it's easy to see where failures happen:

What failed? that RUN, which is part of the +linker-version rule. Easy!

Concluding thoughts, next steps

Earthly has worked so well for us that we've begun to use it for everything. Earthly has become our development "menu": want to run a test? It's in Earthly. Want to do a deploy? It's in Earthly. Want to run the autoformatter? You get the idea.

There is still room for improvement. A lot of our C++ builds are still done in node.js scripts. So while Earthly helps cache those things, if their dependencies change, we build the entire binary with no caching of intermediate artifacts. It would be great to use something like the user defined commands proposal to create a generic target that we can use to build any given C++ file, with a linker target that waits for all those intermediate artifacts to exist before linking them together.

But that's my favorite part about Earthly: we didn't have to bite off that complexity immediately in order to start getting some of the benefits. All in due time.