How Docker Layer Caching Actually Works (And Why You're Probably Doing It Wrong)

Docker caching is one of the most misunderstood mechanisms in modern development workflows. It’s not just a speed boost; it’s an engineering culture. When done right, it can cut build times in half, reduce image size, and save thousands in CI compute.

Yet, most teams still write Dockerfiles like they’re writing shell scripts, casually and without understanding what’s actually happening under the hood.

And when their builds are slow, they assume their app is too complex for Docker or blame it on the CI being slow.

Dockerfile = Layered Build System

Every instruction in a Dockerfile (FROM, COPY, RUN, etc.) creates a layer. These layers are cached, meaning Docker won’t rebuild them unless something changes.

But what exactly counts as a change?

Each layer is cached using a content-based hash, which includes:

• The instruction string (e.g., RUN apt-get update)

• The input files involved (e.g., for COPY, the file contents and metadata)

• The build context (the directory sent to the Docker daemon)

Docker computes a unique hash for each instruction using this data. If any part changes, even subtly, the resulting hash is different, which invalidates the cache for that layer.

Why a Single Cache Miss Invalidates All Layers After It

Docker builds layers sequentially. Each layer depends on the one before it. So if Layer 3 is invalidated, Layers 4, 5, 6, etc. can’t reuse their previous cache, because they were built on top of a now-different Layer 3.

Think of it like a linked list. Yes, that thing they taught you in college that the internet swears is never used in real life. But here, it is. Break one node, and you invalidate the rest of the chain.

This cascading invalidation is why one small change early in your Dockerfile (like a COPY . .) can force a full rebuild downstream.

Think of it like a chain of snapshots: break one, and everything downstream becomes obsolete.

Common Cache Killers

Let’s look at the things that silently invalidate your cache:

Wildcard COPY

COPY . .

This copies everything in your working directory, including README files, .env, test folders, and more. If you don’t have a .dockerignore, every small change nukes your cache.

Dynamic Shell Output

RUN echo $(date)

Every time this runs, it produces different outputs, meaning a different hash. Same goes for:

• uuidgen

• Random strings

• Timestamps

That’s also one of the reasons to use proper instructions in your Dockerfiles. If you want metadata, use proper instructions for it. You think there are only 5 Dockerfile instructions? Check this post where we walk through all 17, including labels, metadata, and much more.

Changing ENV or ARGs

ARG NODE_ENV=production
ENV NODE_ENV=$NODE_ENV

Even if you’re not using these values directly in the command, changing them invalidates the cache because they alter the build context.

Caching Best Practices

Here’s how to work with the cache instead of against it:

1. Copy Dependencies First

COPY package.json package-lock.json ./
RUN npm ci
COPY . .

Why? Because dependencies change less often than source code. This means the npm ci layer gets reused unless your deps actually change.

And yes, use npm ci instead of npm install for reproducible builds in CI and production images.

2. Use .dockerignore

Create a proper .dockerignore to exclude files you don’t want copied into the build context.

.git
node_modules
*.log
.env

Without this, Docker copies everything, even your .git folder, into the image.

3. Avoid Random or Non-Deterministic Commands

RUN echo "Built at $(date)"  # Don't

Instead, pass build metadata as labels or use runtime logging. If you don’t know how to add metadata as labels to your images, check this post where we go through all 17 Dockerfile instructions, including labels and metadata.

4. Chain Commands in a Single RUN

RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*

If you split these into separate RUN lines, they each become a layer and may cache independently, causing inconsistencies and bloat.

5. Keep Your Docker Context Minimal

Docker sends the entire build context (i.e., your current directory) to the daemon. If it’s bloated with unnecessary files, you’re increasing transfer time and risk of unintentional cache invalidation, even if those files aren’t directly used in your build.

Multi-Stage Builds: Powerful but Dangerous

Yes, multi-stage builds reduce image size. But they don’t fix bad caching, they just hide it.

FROM node:18 AS builder
COPY . .
RUN npm run build

FROM node:18-slim
COPY --from=builder /dist /app

If COPY . . in the builder stage includes unnecessary files, or if your cache gets invalidated early, your final stage rebuilds every time.

Use multi-stage builds to separate build-time vs runtime, not to mask caching mistakes.

Debugging Cache Behavior

Use these flags to see what Docker is doing:

docker build --progress=plain --no-cache
DOCKER_BUILDKIT=1 docker build --progress=plain .

And inspect layers with:

docker image history <image_id>
docker image inspect <image_id>

Test different build orders and watch how layer caching behaves to get real performance gains.

Bonus: BuildKit and Advanced Caching

Enable BuildKit and use cache mounts to persist directories between builds:

BuildKit is way more than just caching, it unlocks powerful features like cache mounts, secrets management, parallelism, and remote cache exports. We'll go deep into BuildKit in one of the next posts, stay tuned.

# syntax=docker/dockerfile:1.4
RUN --mount=type=cache,target=/root/.npm npm ci

This caches your NPM directory across builds, dramatically reducing install times if dependencies haven't changed.

You can also mount multiple caches:

RUN --mount=type=cache,target=/root/.cache \
    --mount=type=cache,target=/root/.npm \
    npm ci

Useful for pip, yarn, go modules, apt cache, and more.

To retain intermediate layer cache between CI runs or across machines, add:

--build-arg BUILDKIT_INLINE_CACHE=1
--cache-from=type=registry,ref=my-app:buildcache

This allows BuildKit to embed inline cache metadata into the image and pull from prior builds pushed to your registry. You can chain this with:

docker buildx build --cache-from=type=registry,ref=my-app:buildcache \
                     --cache-to=type=registry,ref=my-app:buildcache,mode=max \
                     -t my-app:latest .

This enables full remote cache sharing, which is great for distributed teams or ephemeral CI runners.

We’ll go deep into BuildKit internals and tricks in a future post, stay tuned.

Final Thoughts

If you’re not thinking in layers, you’re just making your CI and your teammates suffer every time someone hits docker build.

Docker caching isn’t magic. It’s an engineering culture.
And once you master it, you’ll never go back to blindly rebuilding the world.

---

If this post saved your CI from slow builds, inconsistent images, or cloud waste.
That's what I do for engineering teams every day.

I help teams build smarter systems with minimal images, clean Dockerfiles, and bulletproof container pipelines.

→ Follow me on LinkedIn for brutally practical Docker, Kubernetes, and CI insights
→ Subscribe to this blog for deep dives that skip the fluff
→ Reach out if you want a no-BS review of your Dockerfiles or CI setup

No fluff. No copy-paste YAML. Just production-grade containers that work.