Layers and the Build Cache

A Skylane team once asked me why their CI builds had crept from forty seconds to eight minutes. Their Dockerfile looked like the one you wrote last lesson: COPY . . and then npm ci. Every commit changed some file, so every build reinstalled 900 packages from the internet. One reordering brought it back to forty seconds.

Layers and cache keys

Each FROM, COPY, ADD and RUN produces a layer. When you rebuild, BuildKit walks the Dockerfile top to bottom and asks of each step: “have I run exactly this before, with exactly these inputs?”

• For RUN, the key is the command text plus the layer it runs on.
• For COPY and ADD, the key includes a checksum of the files being copied.

The first step whose key doesn’t match is a cache miss, and from then on every later step rebuilds, because each one runs on top of a layer that’s now different. Cache hits are a prefix of the file. Order is everything.

Reorder for the cache

Dependencies change weekly; source changes on every commit. So copy the dependency manifests on their own, install, and only then copy the rest:

# syntax=docker/dockerfile:1
FROM node:24-slim
WORKDIR /app

# Changes rarely: only when dependencies change
COPY package.json package-lock.json ./
RUN npm ci

# Changes on every commit
COPY . .
RUN npm run build

EXPOSE 3000
CMD ["node", "dist/server.js"]

Edit src/server.ts and rebuild. The output shows which steps were reused:

 => CACHED [2/6] WORKDIR /app
 => CACHED [3/6] COPY package.json package-lock.json ./
 => CACHED [4/6] RUN npm ci
 => [5/6] COPY . .
 => [6/6] RUN npm run build

The general rule: sort instructions from least to most frequently changing. System packages, then language dependencies, then your code, then anything derived from your code.

docker history notes-api:dev lists the layers with their size and the instruction that created them. Use it when an image is bigger than you expect; the culprit is usually one fat layer.

The same rule applies to build arguments. A line like ARG GIT_SHA near the top, used in a later RUN or LABEL, gives every RUN step below it a new input on every commit, because build arguments are visible to every later RUN as environment variables. Declare such values as late as possible, right before the one step that uses them.

COPY keys on file contents, not timestamps, so a fresh git clone on a build server hits the same cache as your laptop as long as the bytes match. That’s also why a committed, up-to-date package-lock.json matters: it is the input that decides whether the expensive install is reused.

System packages: one RUN, cleaned up

If you need OS packages, install them in a single RUN:

RUN apt-get update \
 && apt-get install -y --no-install-recommends ca-certificates \
 && rm -rf /var/lib/apt/lists/*

--no-install-recommends skips optional extras, and deleting the package lists in the same step keeps them out of the layer. Deleting them in a later RUN doesn’t shrink anything: the earlier layer still contains them.

Cache mounts: faster misses

Sometimes the install must rerun because you added a dependency. A cache mount keeps npm’s download cache in the builder between builds, without putting it in the image:

COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

npm ci still deletes and recreates node_modules, but the tarballs come from local disk instead of the registry. On notes-api that turns a 70-second reinstall into about 15.

Caches in CI

Your laptop keeps its cache between builds. Most CI runners start empty, so every build is cold however well you order the file. Export the cache somewhere durable and import it on the next run:

docker buildx build \
  --cache-from type=registry,ref=ghcr.io/skylane/notes-api:buildcache \
  --cache-to type=registry,ref=ghcr.io/skylane/notes-api:buildcache,mode=max \
  -t ghcr.io/skylane/notes-api:dev .

mode=max stores intermediate layers too, which matters once you have several build stages. GitHub Actions users can use type=gha instead of a registry.

Builds are fast now. Next you’ll get comfortable with the running side: ports, environment variables, logs, and what really happens when a container stops.