Smaller, Faster, Cheaper: The Practical Guide to Go Docker Optimization

How I brought down build times from 25 minutes to under 2, and cut image sizes by 95%

TL;DR

This is a real-world story of how I optimized the Docker build and deployment pipeline for a Golang-based service:

• Reduced build time from ~25 minutes to under 2

• Dropped image size from 2GB+ to just ~80MB

• Improved Kubernetes pod startup and hotfix deployment speed

• Cut ECR storage costs and made CI pipelines faster & more reliable

• Shifted to secure, minimal base images with stripped, statically compiled Go binaries

• Enabled BuildKit + remote caching via ECR for cache-friendly, repeatable builds

• Standardized reusable Docker patterns across multiple services

💬 Heads-up: This is a long one — but if you're building Go apps for production, it's probably the only guide you’ll ever need to run them in containers efficiently, securely, and at scale.

---

Let’s be honest, no one likes waiting 25 minutes for a Docker image to build, especially when you're in the middle of development and just want to test a small change or worse, when your production system is down and you need to push a quick fix, fast.

That was exactly the situation I found myself in.

We had Golang-based microservices that worked well, but everything around them was painfully slow. Docker builds took forever. The image sizes were close to 2GB. And our CI/CD pipeline felt sluggish - every build and deploy step added unnecessary delays. On top of that, Kubernetes pod and ECS task startups were slow, and our ECR storage costs kept going up.

At first, it seemed like a few minor issues. But once I dug in, I found all the usual suspects: outdated Dockerfile, no caching, heavy base images, and no attention to what was actually getting packed into the final container.

This blog is a real-world story of how I brought down Docker build times from 25 minutes to under 2, reduced image size by 95%, made deployments faster, and saved on storage costs - all with small, focused changes that had a big impact.

Let me walk you through how it all happened.

---

The Setup: When Things Took Forever

When I first looked at the repos, everything seemed fine on the surface. The Go codebases were clean, the microservices worked well, and the functionalities were solid. But the moment I triggered a build, I knew something wasn’t right.

• Build time: 20 to 25 minutes

• Docker image size: ~2GB

• CI/CD: painfully slow

• ECR storage: growing steadily

• Kubernetes pod and ECS task spin-up: delayed every time due to heavy image pulls

Whether the team was pushing a regular feature or trying to ship a hotfix under pressure, it felt like the pipeline was working against us. And it wasn’t just about time - it was mentally exhausting. Waiting 25 minutes just to test a minor change kills momentum, and during production incidents, it made things worse.

We were using AWS EKS and AWS ECS to deploy services, and naturally, the large image size made everything downstream slow too - from image pulls to container starts. Over time, the impact started showing up in our cloud bills as well, thanks to the bloated ECR usage.

That’s when I decided to dig deeper and figure out what was really going on inside the Dockerfiles.

---

Legacy State: What Was Wrong

Once I dug deep into the Dockerfiles, it became pretty clear why things were so slow.

It had all the classic red flags of a first-draft or "it-just-works" setup - something likely written in a hurry or by someone new to Docker best practices (we’ve all been there).

Here’s what I found:

1. Heavy base images

   The build used a full-fledged golang:1.xx image, and sometimes even an ubuntu base layer stacked below it - which pulled in hundreds of MBs of unnecessary tools and packages.

2. Improper multi-stage build

   Technically, there were multiple stages defined - but they weren’t used the right way.
   In the end, everything - including build tools, test binaries, source files, and Go module cache - was copied into the final stage anyway. The whole point of isolating build and runtime environments was lost, which made the final image just as bloated as a single-stage build.

3. No .dockerignore

   Every time the image was built, it copied the entire repo - including .git, local configs, test files, and sometimes even large assets. That bloated the build context and slowed down the docker build process.

4. Dependencies rebuilt every time

   The build didn’t cache go mod download or any dependencies, so even the tiniest code change triggered a full re-download of all modules. This added several minutes to every build and made CI unnecessarily slow.

5. Too many layers, unoptimized instructions

   Each RUN, COPY, and ADD instruction created a new image layer. And many of them were repetitive or could have been merged to reduce overhead. No layer caching strategy meant the whole image rebuilt far too often.

All of this combined made the image big, the build slow, and the pipeline fragile. And because it wasn’t modular or maintainable, any optimization felt risky - which is often why these setups stay broken for so long. But with just a few focused improvements, I knew we could fix all of it.

---

Starting the Fix: Where I Began

Once I had a clear picture of what was going wrong, I didn’t try to solve everything at once. I started with the low-hanging fruit - small changes that I knew would give immediate results without breaking anything.

1. Add a .dockerignore

   This was the simplest win. Until then, our Docker builds were copying everything - including .git, test files, README docs, local configs, and even random developer junk lying around.

   By adding a proper .dockerignore file, it reduced the build context dramatically. This alone cut ~15–20 seconds from every build and reduced noise inside the image.

    # .dockerignore
    .git
    *.md
    test/
    *.log
    *.local
    node_modules/
   

2. Clean up and restructure the Dockerfile

   Next, I reorganized the Dockerfile to follow a proper multi-stage structure.

   Earlier, we had multiple stages defined - but the final stage copied almost everything again, defeating the purpose. I fixed that by making sure the final image only contained the stripped, statically compiled Go binary, nothing else.

   This also meant removing all build tools, C dependencies, and temp files from the final image - instantly shrinking the size.

3. Build a custom base image for heavy dependencies

   One of the biggest time sinks in our builds was installing C libraries like librdkafka, libssl, and other native packages - every single time.

   To solve this, I created a custom Docker base image that included all these heavy dependencies pre-installed. This way, our actual app builds could start from a pre-baked image that didn’t need to reinstall or download anything at runtime.

   It took some upfront effort, but the payoff was huge:

   • Reduced build time by several minutes

   • Made the builds more stable and reproducible

   • Allowed us to reuse the base image across multiple Go services

4. Switch to a minimal final base image

   Instead of shipping everything inside a golang or ubuntu image, I switched to using scratch or distroless/static as the base for the final stage.

   This change alone dropped our final image from over 2GB to ~80MB. It also made the image more secure by removing shells, compilers, and unnecessary binaries.

5. Enable Go build caching

   Previously, our builds were re-downloading modules and rebuilding everything from scratch. To fix this, I added Docker cache mounts to persist Go’s build and module cache between builds:

    ENV GOCACHE=/root/.cache/go-build
    RUN --mount=type=cache,target=/root/.cache/go-build \
        go build -o app .
   

   How this works:

   • GOCACHE tells Go where to store its compiled build artifacts (object files, dependency build outputs, etc.)

   • The --mount=type=cache,target=... directive is a special Docker BuildKit feature that preserves that cache between builds

   • This cache is persisted outside the image layers, so Docker can reuse it even when the source changes slightly

   • On subsequent builds, Go can detect unchanged packages and skip rebuilding them entirely - drastically cutting down build time

This made a huge difference - especially in CI pipelines where minor code changes were triggering full builds earlier. Now, unchanged dependencies didn’t add to build time anymore.

6. Strip the binary and build statically

   To make the binary as lean as possible and compatible with minimal base images like scratch, I added flags to:

   • Strip debug info and symbols

   • Ensure it was statically compiled (very important when using native C dependencies like librdkafka)

       CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
       go build -ldflags="-s -w" -o app .
     

     • CGO_ENABLED=0 ensures no dynamic linking

     • -ldflags="-s -w" strips debugging symbols for smaller binaries

⚠️ Note: In cases where native C libraries like librdkafka are actually needed at runtime, then ensure CGO_ENABLED=1 and use a custom base image to handle linking cleanly.

7. Configure Docker BuildKit in CI/CD

   Most of these optimizations rely on Docker BuildKit, so I made sure it was explicitly enabled in the CI pipeline.

   For example, in GitLab CI:

    variables:
      DOCKER_BUILDKIT: 1
      BUILDKIT_INLINE_CACHE: 1
   

   This unlocked:

   • Inline cache support

   • Parallel layer execution

   • Cache mounts for Go build cache

Without this, none of the caching improvements would work consistently in CI/CD.

8. Remote layer caching using ECR

   To make builds even faster in our GitLab CI runners, I set up remote cache pushing to Amazon ECR. This allowed subsequent pipeline runs to pull existing layers instead of rebuilding everything from scratch - even across branches and commits.

    docker build \
      --cache-from=type=registry,ref=<account>.dkr.ecr.region.amazonaws.com/my-app:buildcache \
      --cache-to=type=registry,mode=max,ref=<account>.dkr.ecr.region.amazonaws.com/my-app:buildcache \
      -t my-app:latest .
   

   This worked especially well when builds ran on ephemeral runners (k8s runners), where local layer caching wasn’t persistent. By using ECR as a remote cache store, I preserved all heavy layers like base images, Go modules, and C library installs - and skipped them on every incremental build.

---

Iterating & Fine-Tuning

Once the big issues were out of the way - oversized base images, missing .dockerignore, and cache-less builds - things were already much better. But I didn’t stop there.

This is where I went a bit deeper - looking at what else could be optimized to squeeze out every second and make the setup truly production-grade.

1. Reordered Dockerfile layers for better cache hits

   In Docker, the order of instructions matters a lot when it comes to caching. I made sure that:

   • COPY go.mod and COPY go.sum came before the full source code

   • go mod download was in its own layer

This ensured that if I didn’t change dependencies, Docker would reuse that layer and skip re-downloading modules.

    COPY go.mod go.sum ./
    RUN go mod download

    COPY . .
    RUN go build -o app .

This simple reorder reduced 30–60 seconds on average per build in CI.

2. Removed unused packages and binaries

   Once I moved to a custom base image and clean multi-stage builds, I ran docker history (very handy command) on the final image to inspect what was still there - and found some hidden leftovers.

   I removed:

   • Unused tools (like curl, jq, etc. mistakenly left from early setups)

   • Redundant COPY steps

   • Old static files that weren’t used at runtime

Every MB mattered - especially in environments like EKS and ECS, where smaller images meant faster pod spinups and fewer image pull throttles.

3. Split test/lint/build stages in CI

   To speed up feedback in CI pipelines, I split the stages:

   • Run lint + tests first (fast fail if needed)

   • Build and push image only if tests pass

This avoided wasting time building/pushing a container if tests were going to fail anyway.

4. Tagged and reused base layers across services

   Since I had multiple Go services with similar dependencies (Kafka, Redis, etc.), I reused the same custom base image across them.

   This made:

   • Build times more predictable

   • ECR usage more efficient

   • CI logs less noisy (Docker could reuse layers)

These refinements weren’t dramatic on their own, but together they added polish and consistency. The whole system felt lighter, faster, and more maintainable - and most importantly, everyone in the team could feel the difference.

---

Results: What We Gained

Once all the optimizations were in place, the impact was immediately visible - not just in numbers, but in the overall developer experience, CI speed, and even how fast things shipped to production.

Here’s what we achieved:

There were multiple incidents where something broke in prod - and thanks to the optimized build pipeline and smaller images, we were able to patch the issues, run tests and build & deploy to production - all within a few minutes

This directly helped us maintain uptime and stay on top of incident SLAs - something that would have been impossible with the old 25-minute build pipeline.

When things go wrong, speed matters. And this speed gave us control, not chaos.

---

Final Thoughts and Reflections

Looking back, this wasn’t about chasing build speed for the sake of it. It was about removing friction - the kind that silently slows down shipping, wears down developers, and becomes a hidden tax on every deploy.

The original setup wasn’t “wrong” - it was just what happens when things are built fast, without time to reflect on best practices. It was the natural outcome of time constraints, unclear ownership, and the all-too-common “it works for now” mindset. But once I started cleaning it up - step by step, the benefits were massive… especially during a production outage.

It wasn’t magic. Just:

• Respecting Docker’s caching model

• Keeping the final image as lean as possible

• Building with clarity and purpose

What really made the difference was treating Docker as an active part of the development experience, not just a deployment afterthought.

---

💡 Tips for Teams Starting Out

If you're building Go apps and containerizing them for production, here are a few things I wish we had nailed from day one:

• Don’t ignore Docker best practices: The defaults usually “work”, but they rarely work well. Spend time structuring your Dockerfile properly, it pays back quickly.

• Bake in security and reproducibility from day one: Use minimal base images, strip sensitive tools, and lock down environments early. Avoid surprises later.

• Use multi-stage builds religiously: Never ship compilers, tools, or test data in your final image. It’s not just about size, it’s about safety and clarity.

• Static Go binaries are a gift - use them well: CGO_ENABLED=0 + scratch or distroless can take you a long way. Simpler, safer, smaller.

• Enable BuildKit early: It's not optional anymore - it unlocks all the caching and performance improvements you need for modern CI/CD.

And when you ensure these things, your future self (and your team) will thank you :)