What I Learned Rebuilding My App During a Live Docker Hub Outage

On July 8, 2025, Docker Hub went into read-only mode — and my app went down with it. I was deploying my containerized Go-based application, CronOps, on Minikube, using a public image from Docker Hub. Everything had been working smoothly, until suddenly…

ImagePullBackOff
denied: requested access to the resource is denied

Kubernetes couldn’t pull my image from Docker Hub, and pods started crashing. I quickly realized: Docker Hub was down. I had no GHCR, no ECR, no private registry. Just… panic.

---

💥 What Broke

Everything was working fine… until it wasn’t.

I pushed a new version of my Docker image — nsahil992/overkill-cronops:v1 — to Docker Hub and updated my Kubernetes deployment to use that tag. As usual, I applied the manifest:

kubectl apply -f cronops-deployment.yaml

Seconds later, the pods started crashing. Running kubectl get pods, I saw:

ImagePullBackOff

And when I described the pod:

kubectl describe pod cronops-deployment-xxxx

I saw this in the logs:

Failed to pull image "nsahil992/cronops:v1": 
denied: requested access to the resource is denied

At first, I thought it was a permission issue, maybe a typo in the image name or a problem with Docker Hub credentials. But after checking the registry manually, it became clear: Docker Hub was down and in read-only mode, not serving fresh image pulls. I couldn't even view my image on the Docker UI.

---

⛓️‍💥 The Real Problem

My Kubernetes cluster was completely dependent on Docker Hub to pull container images during deployments. No image meant no pod. And no pod meant the application was dead.

Worse, I had no fallback:

• No images mirrored on GitHub Container Registry (GHCR)

• No ECR setup on AWS

• No local registry cache

• No smart imagePullPolicy defaults

My deployment was brittle — completely at the mercy of Docker Hub’s availability.

What should have been a routine deploy turned into a full-on incident. My app couldn’t recover because it had no way to fetch the image it needed to run. I checked on for Docker community forum and got to know that users were experiencing the same issue.

Well, rimelek on the Docker forum was partially correct that there was nothing to do but wait for a fix. However, I decided to solve my application's problem myself.

---

🤕 The Recovery Plan

When external systems fail, you only have two choices: wait or work. I chose the latter.

Docker Hub was unreachable, but I still had everything I needed to recover — the Dockerfile, the source code, and a local Minikube cluster. Here’s how I bypassed Docker Hub entirely and brought my app back online.

Step 1: Build the Image Locally

Since I couldn’t pull from Docker Hub, I built the image directly on my machine:

docker build -t cronops:local .

This gave me a local image tagged cronops:local, ready to be injected into the cluster.

Step 2: Point Docker to Minikube’s Internal Environment

Minikube runs its own internal Docker daemon. To make Kubernetes aware of the local image, I needed to tell my shell to use Minikube’s Docker engine:

eval $(minikube docker-env)

Then, I rebuilt the image inside Minikube:

docker build -t cronops:local .

This ensured that the image would be available directly to the Kubernetes runtime without any need to pull it from a remote registry.

Step 3: Modify the Deployment Manifest

Next, I edited my Kubernetes Deployment YAML:

image: cronops:local
imagePullPolicy: Never

The key here is imagePullPolicy: Never. This instructs Kubernetes not to attempt pulling the image from any registry. It will simply use the one already present in the local Docker daemon (which is now Minikube’s Docker).

Step 4: Reapply the Deployment

With the updated deployment file in place, I redeployed:

kubectl apply -f cronops-deployment.yaml

Kubernetes recreated the pods. This time, it found the image locally, didn’t attempt any network pull, and launched successfully.

---

🤩 The Result

In a matter of minutes:

• The app was back online

• Pods were running again

• I had bypassed the Docker Hub outage completely

No GHCR. No ECR. Just a local image and some DevOps instincts.

---

🤔 Why I Chose Local Over ECR or GHCR and the Trade-Offs

In an ideal world, my app images would be mirrored across ECR, GHCR, or a private registry for redundancy. But during this incident, time was critical. I had a choice:

🐇 Fix the issue fast using a local image
or
🐌 Spend more time setting up and pushing to another registry

I chose local recovery because it was the fastest, most reliable, and self-contained option in the moment.

Here’s the breakdown of the trade-offs:

---

🧠 Why "Local" Made Sense Right Now

• Trial: I had already worked with ECR and wanted to try on local to check how it works.

• Speed: I already had the Dockerfile and source code. Building and injecting locally took <1 minute.

• Focus: I didn’t need to worry about authentication tokens, registry permissions, or push latency.

• Scope: I was running Minikube locally for testing/dev — not a shared or production cluster.

• Independence: By going local, I eliminated all external dependencies, which was the root cause of the failure.

---

💡 But What If This Were Production?

Then local would not be a viable solution. In that case, I would have needed:

• Multi-region registry backups (ECR + GHCR)

• CI/CD pipeline pushing to multiple registries

• Pull secrets with fallbacks in my Kubernetes manifests

For production-grade systems, redundancy and registry independence aren’t optional — they’re mandatory.

---

📒 Lessons Learned

This wasn’t just a Docker outage, it was a wake-up call.

Even though I recovered fast, the incident highlighted some deep architectural flaws in my workflow. Here’s what I learned:

1. Never Depend on a Single Registry

   I was relying 100% on Docker Hub for every Kubernetes deployment. That’s like deploying your app with no version control backup, you’re one outage away from disaster.

2. Simulate Failure Before Failure Happens

   I had never tested what would happen if Docker Hub went down. I assumed it would “just work.”
   That false confidence cost me deployment time. Now, I know better.

3. Document Your Recovery Steps

   If I had not known about Minikube’s internal Docker, or how imagePullPolicy worked, I would’ve been stuck.

---

So if you have reached here, thank you for your time. If you like my blogs and they help you, consider subscribing to my newsletter to get my blogs directly in your mailbox 📬