Skip to main content
ToolsRunbooksBlogDocs

rawops.dev

P2

ImagePullBackOff — Kubernetes Troubleshooting Guide

Diagnose why Kubernetes can't pull a container image. Covers image name typos, registry auth, pull secrets, network issues, and rate limits.

10 min8 steps
kubernetesimagepullbackoffregistrydockerpull-secrets
Progress: 0/8 steps
0%

Find all pods with ImagePullBackOff or ErrImagePull status.

kubectl get pods -A | grep -E 'ImagePullBackOff|ErrImagePull|ErrImageNeverPull'
Expected: List of pods stuck pulling images. ErrImagePull is the initial error, ImagePullBackOff means it's retrying with exponential backoff.

The events section contains the specific pull error message.

kubectl describe pod POD_NAME -n NAMESPACE | grep -A10 'Events:'
Expected: Look for: 'manifest unknown' (wrong tag), 'unauthorized' (auth issue), 'connection refused' (network), 'toomanyrequests' (rate limit).

Check for typos in the image reference — the most common cause.

kubectl get pod POD_NAME -n NAMESPACE -o jsonpath='{.spec.containers[*].image}' && echo
Expected: Full image reference (registry/repo:tag). Verify: registry URL is correct, repo name is spelled right, tag exists.
Using ':latest' tag is unreliable — images may be overwritten or not pushed. Always use specific tags or digests.

SSH to the node and try pulling the image directly.

# On the node:
crictl pull IMAGE:TAG
# Or with Docker:
docker pull IMAGE:TAG
Expected: If it fails here too, it's a node-level issue (network, auth, disk). The error message will be more detailed than K8s events.

Verify the pod has the correct pull secret for private registries.

# Check if pod has imagePullSecrets:
kubectl get pod POD_NAME -n NAMESPACE -o jsonpath='{.spec.imagePullSecrets}' && echo

# List available secrets:
kubectl get secrets -n NAMESPACE | grep docker
Expected: Pod should reference a docker-registry secret. If empty and using a private registry, this is the problem.

Decode the pull secret to check if credentials are valid.

kubectl get secret SECRET_NAME -n NAMESPACE -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .
Expected: Shows registry URLs and auth tokens. Verify the registry URL matches the image reference and credentials aren't expired.

Docker Hub limits anonymous pulls to 100/6h and free accounts to 200/6h.

# Check remaining pulls (from node):
TOKEN=$(curl -s 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/alpine:pull' | jq -r .token) && curl -sI -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/library/alpine/manifests/latest 2>&1 | grep -i ratelimit
Expected: RateLimit-Remaining shows pulls left. If 0, wait or authenticate. Consider using a registry mirror or upgrading Docker Hub plan.

If the issue is missing or invalid credentials, create a new pull secret.

kubectl create secret docker-registry regcred -n NAMESPACE \
  --docker-server=REGISTRY_URL \
  --docker-username=USERNAME \
  --docker-password=PASSWORD \
  --docker-email=EMAIL

# Patch the default service account to use it:
kubectl patch serviceaccount default -n NAMESPACE -p '{"imagePullSecrets": [{"name": "regcred"}]}'
Expected: Secret created. Delete the stuck pod to trigger a fresh pull attempt: kubectl delete pod POD_NAME -n NAMESPACE