Staging ButItWorkedLocal

Category: Containers

Troubleshooting: Docker Image Pull Limit Exceeded on CentOS Stream / Rocky Linux
Resolve 'Docker image pull limit exceeded' errors on CentOS Stream or Rocky Linux. Learn to authenticate, configure registries, and optimize pulls.

Introduction

As a Systems Administrator or DevOps engineer managing containerized applications on CentOS Stream or Rocky Linux, encountering Docker image pull limit exceeded errors can halt deployments, break CI/CD pipelines, and prevent local development. This issue typically manifests as docker pull or podman pull commands failing to retrieve images from Docker Hub, often accompanied by messages indicating rate limits for anonymous users. This guide provides a comprehensive technical breakdown and actionable steps to diagnose and resolve this common problem.

Symptom & Error Signature

When you attempt to pull an image from Docker Hub without authenticating, or if your authenticated pulls exceed your plan's limits, you will observe errors similar to the following in your terminal or build logs:

Docker Engine Error Output:
```
[root@host ~]# docker pull nginx:latest
Using default tag: latest
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may resume pulls in a short while or upgrade your account. See https://docs.docker.com/docker-hub/rate-limit/ for more information.
```
Or, for an anonymous pull attempt:
```
[root@host ~]# docker pull someuser/someimage:latest
Error response from daemon: pull access denied for someuser/someimage, repository does not exist or may require 'docker login': denied: You have reached your pull rate limit. You may resume pulls in a short while or upgrade your account.
```
Podman Error Output (Common on Rocky/CentOS):
```
[root@host ~]# podman pull docker.io/nginx:latest
Trying to pull docker.io/nginx:latest...
Error: initializing source docker.io/nginx:latest: pinging container registry registry-1.docker.io: GET https://registry-1.docker.io/v2/: unexpected status code 429 Too Many Requests: toomanyrequests: You have reached your pull rate limit. You may resume pulls in a short while or upgrade your account. See https://docs.docker.com/docker-hub/rate-limit/ for more information.
```
Root Cause Analysis

The underlying reason for the "Docker image pull limit exceeded" error is Docker Hub's rate limiting policy. Docker Hub imposes limits on the number of image pulls based on whether a user is authenticated or anonymous, and these limits are enforced per IP address over a rolling 6-hour window.
1. Anonymous Pull Limits: Unauthenticated users (those not logged in to Docker Hub) are typically limited to 100 pulls per 6 hours per IP address.
2. Authenticated Pull Limits: Users logged in with a free Docker ID account are typically limited to 200 pulls per 6 hours per IP address. Paid subscriptions offer significantly higher limits or unlimited pulls.
Several scenarios frequently lead to these limits being hit:
- Shared IP Addresses: In cloud environments, data centers, or behind corporate NATs/proxies, multiple servers or users might share a single public IP address. This means all their pull requests contribute to the same rate limit bucket, causing the limit to be reached much faster than anticipated.
- CI/CD Pipelines: Automated build and deployment pipelines often perform numerous docker pull operations as part of their workflow (e.g., pulling base images, caching layers, deploying new versions). Without proper authentication or caching, these can quickly exhaust pull limits, especially if multiple pipelines run concurrently.
- Frequent Image Updates: If your applications rely on rapidly changing base images or if you frequently rebuild images that pull many layers, you can hit the limits.
- Development Environments: Teams of developers frequently pulling images for local development can collectively exhaust shared IP limits.
- Lack of Authentication: The most common immediate cause is simply not being logged in to Docker Hub. Many system-level container orchestrators or build scripts might operate without explicit docker login commands, leading to anonymous pulls.
Step-by-Step Resolution

Here are the recommended steps to resolve and mitigate Docker Hub pull rate limit issues on CentOS Stream and Rocky Linux.

1. Verify Current Rate Limit Status (Optional)

Before attempting a fix, you can optionally check your current rate limit status using curl against Docker Hub's API. This provides the X-RateLimit-Limit and X-RateLimit-Remaining headers.
```
[root@host ~]# REGISTRY_AUTH_HEADER="$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "YOUR_DOCKER_USERNAME", "password": "YOUR_DOCKER_PASSWORD"}' https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull | jq -r .token)"
[root@host ~]# curl -s -I -H "Authorization: Bearer $REGISTRY_AUTH_HEADER" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i 'ratelimit'
```
Replace YOURDOCKERUSERNAME and YOURDOCKERPASSWORD with your actual Docker Hub credentials. If you are an anonymous user, you can omit the authentication part and just try:
```
[root@host ~]# curl -s -I https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i 'ratelimit'
```
The output will show headers like: ` ratelimit-limit: 100;w=21600 ratelimit-remaining: 99;w=21600 ` w=21600 represents the 6-hour window in seconds.

2. Authenticate to Docker Hub

The most straightforward and recommended solution is to authenticate your Docker daemon or Podman environment with a Docker Hub account. This immediately increases your pull limit from 100 to 200 pulls per 6 hours and is a prerequisite for higher limits.

For Docker Engine:
```
[root@host ~]# docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: your_docker_username
Password:
Login Succeeded
```
This command stores your credentials in ~/.docker/config.json. If you run it as root, it stores them in /root/.docker/config.json. Ensure the user running docker pull has access to these credentials.

[!IMPORTANT] If running Docker in a CI/CD pipeline, consider using environment variables for DOCKERUSERNAME and DOCKERPASSWORD and piping them to docker login for security. echo $DOCKERPASSWORD | docker login --username $DOCKERUSERNAME --password-stdin

For Podman (on CentOS Stream / Rocky Linux):

Podman manages registry authentication similarly to Docker.
```
[root@host ~]# podman login docker.io
Authenticating with existing credentials for docker.io
Username: your_docker_username
Password:
Login Succeeded!
```
Podman stores credentials in ~/.config/containers/auth.json (for rootless users) or /run/containers/0/auth.json (for root users).

3. Purchase a Docker Subscription

If 200 pulls per 6 hours is insufficient for your needs, you will need to upgrade your Docker Hub account to a paid subscription (e.g., Pro, Team, Business). These plans offer significantly higher or virtually unlimited pull rates.
- Visit Docker Hub Plans for detailed information.
- Once subscribed, ensure you docker login (or podman login) with the account associated with the subscription.
4. Configure a Private Docker Registry or Mirror

For environments with high pull volumes, multiple servers, or strict network policies, setting up a local caching registry mirror is an excellent long-term solution. This reduces external pulls, speeds up image retrieval, and provides greater control.

You can run your own registry using the docker/distribution image or utilize cloud-native registry services (e.g., Red Hat Quay.io, AWS ECR, Azure Container Registry, Google Container Registry).

Setting up a Docker Hub Mirror (using docker/distribution):
1. Start the Registry Container:
2. On your CentOS Stream/Rocky Linux host, run the registry. Choose a host with ample storage and network bandwidth.
```
    [root@host ~]# docker run -d -p 5000:5000 --restart=always --name registry 
      -v /var/lib/registry:/var/lib/registry 
      registry:2
    ```
```
1. Configure Docker Daemon to Use the Mirror:
2. Edit or create the Docker daemon configuration file /etc/docker/daemon.json.
```
    [root@host ~]# vi /etc/docker/daemon.json
```
Add the following content, replacing YOURREGISTRYMIRROR_IP with the IP address or hostname of your registry mirror:
```
    {
      "registry-mirrors": ["http://YOUR_REGISTRY_MIRROR_IP:5000"],
      "log-level": "info"
    }
    ```
```
1. Restart Docker Service:
```
    [root@host ~]# systemctl daemon-reload
    [root@host ~]# systemctl restart docker
```
Verify the configuration by checking daemon info: `bash [root@host ~]# docker info | grep 'Registry Mirrors' Registry Mirrors: http://127.0.0.1:5000 `

Now, when docker pull is executed, it will first attempt to pull from your local mirror. If the image isn't found there, the mirror will pull it from Docker Hub, cache it, and then serve it to your Docker daemon. Subsequent pulls of the same image will come from the local mirror.

[!WARNING] For production environments, ensure your private registry is secured with TLS/SSL and authentication. Running an unauthenticated HTTP registry in production poses significant security risks. You would typically need to add --insecure-registries YOURREGISTRYMIRROR_IP:5000 to daemon.json if not using HTTPS, but this is highly discouraged for security reasons.

Configuring Podman to Use a Registry Mirror:

Podman's configuration for registries is managed via /etc/containers/registries.conf.d/.
1. Create a Configuration File:
2. `bash
3. [root@host ~]# vi /etc/containers/registries.conf.d/00-mirror.conf
4. `
1. Add Mirror Configuration:
2. `toml
3. [[registry]]
4. location = "docker.io"
5. mirror = ["YOURREGISTRYMIRROR_IP:5000"]
6. `
7. Replace YOURREGISTRYMIRROR_IP with the IP address or hostname of your registry mirror.
1. No service restart is typically needed for Podman, it reads the configuration directly.
5. Optimize Image Pulling Strategy

Beyond direct authentication and mirroring, optimizing how you pull images can significantly reduce the overall pull count.
- Cache Images in CI/CD Runners: For CI/CD systems like GitLab CI, Jenkins, or GitHub Actions, configure runners to persist Docker image caches across jobs. This avoids repeated pulls of base images.
- Minimize Base Image Changes: Avoid frequent updates to base images unless absolutely necessary. Pinning to specific digests (myimage@sha256:digest) instead of tags (myimage:latest) can improve determinism, though it won't directly reduce rate limits if the image still needs to be pulled.
- Leverage Multi-Stage Builds: Use multi-stage Dockerfiles to build your application and then copy only the necessary artifacts into a minimal final image. This reduces the size and number of layers that need to be pulled for the final deployable image.
- Prune Less Frequently: If storage allows, consider cleaning up old Docker images less aggressively. Keeping frequently used images locally can prevent unnecessary re-pulls. Use docker image prune -a cautiously.
6. Utilize an Alternative Container Registry

If Docker Hub's policies or technical limitations consistently hinder your operations, consider migrating your images to an alternative container registry. Many cloud providers offer highly scalable and integrated registries:
- Red Hat Quay.io: A robust registry solution, particularly popular in enterprise and OpenShift environments.
- AWS Elastic Container Registry (ECR): Seamlessly integrates with other AWS services.
- Azure Container Registry (ACR): Integrated with Azure, supports geo-replication.
- Google Container Registry (GCR) / Artifact Registry: Integrated with GCP, high performance.
- GitLab Container Registry: Built into GitLab, excellent for CI/CD integration.
These registries typically offer their own rate limits, which are often more generous or tied to your cloud service subscription, and provide better control over image storage and distribution. You would push your images to these registries and then configure your docker pull commands or daemon.json to pull from the alternative registry.
July 19, 2026
Troubleshooting Kubernetes CrashLoopBackOff: Container Startup Failures on Ubuntu 20.04 LTS
Resolve Kubernetes CrashLoopBackOff errors on Ubuntu 20.04 LTS. This guide details common causes and provides expert step-by-step fixes for container startup failures.

When deploying applications on Kubernetes, encountering a CrashLoopBackOff status indicates a fundamental issue preventing your container from starting and remaining healthy. This status means Kubernetes is repeatedly trying to start the container, it's crashing immediately, and then Kubernetes waits for an exponentially increasing back-off duration before the next retry. This guide provides a highly technical, step-by-step approach to diagnose and resolve CrashLoopBackOff errors on an Ubuntu 20.04 LTS-based Kubernetes cluster.

Symptom & Error Signature

The primary symptom of a CrashLoopBackOff is a pod that never reaches a Running state, instead cycling through ContainerCreating, Crashing, and CrashLoopBackOff.

You will typically observe this when checking your pods:
```
kubectl get pods
```
```
NAME                             READY   STATUS             RESTARTS   AGE
my-app-deployment-78f9f7584f-abcd1   0/1     CrashLoopBackOff   5          2m3s
another-pod-xyz-12345            1/1     Running            0          10m
```
Further inspection using kubectl describe pod reveals the container's last state and relevant events:
```
kubectl describe pod my-app-deployment-78f9f7584f-abcd1
```
```
...
Containers:
  my-app:
    Container ID:  containerd://a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0
    Image:         my-registry/my-app:v1.0.0
    Image ID:      my-registry/my-app@sha256:fedcba9876543210
    Port:          8080/TCP
    Host Port:     0/TCP
    State:         Waiting
      Reason:      CrashLoopBackOff
    Last State:    Terminated
      Reason:      Error
      Exit Code:   1
      Started At:  Thu, 18 Jul 2024 10:30:15 -0400
      Finished At: Thu, 18 Jul 2024 10:30:16 -0400
    Ready:         False
    Restart Count: 5
    Environment:
      DB_HOST: mysql-service
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-abcde (ro)
...
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Pulled     2m4s (x6 over 3m)   kubelet            Container image "my-registry/my-app:v1.0.0" already present on host
  Normal   Created    2m4s (x6 over 3m)   kubelet            Created container my-app
  Normal   Started    2m3s (x6 over 3m)   kubelet            Started container my-app
  Warning  BackOff    14s (x7 over 2m4s)  kubelet            Back-off restarting failed container
```
The most crucial information here is the Exit Code (often non-zero, indicating an error) and the Events section which shows the BackOff reason.

Root Cause Analysis

A CrashLoopBackOff signifies that the container's main process exited prematurely. The underlying causes are diverse but generally fall into these categories:
1. Application Errors: The application code itself has a bug, an unhandled exception, or fails to initialize correctly (e.g., cannot connect to a database, missing configuration).
2. Incorrect command or args: The command or args specified in the Pod definition do not correctly execute the application's entrypoint or pass incorrect parameters.
3. Missing Dependencies/Files: The application expects certain files, libraries, or environment variables that are not present in the container image or mounted volumes.
4. Resource Constraints (OOMKilled): The container tries to consume more memory than allocated by resources.limits.memory, leading to the Linux OOM (Out Of Memory) killer terminating the process.
5. Permissions Issues: The application lacks the necessary file system or network permissions to operate (e.g., trying to write to a read-only volume, binding to a privileged port without appropriate capabilities).
6. Misconfigured Liveness/Readiness Probes: Probes are configured too aggressively, failing immediately upon startup, causing Kubernetes to restart a perfectly healthy container.
7. Volume Mount Problems: Persistent Volume Claims (PVCs) are not bound, the subPath is incorrect, or the underlying storage is inaccessible or has permission issues.
8. Bad Container Image: The container image itself is corrupted, incompatible with the underlying kernel, or fundamentally broken.
9. Network Configuration Issues: While less common for immediate startup crashes, if the application critically depends on an immediate network connection to an external service or a service mesh sidecar fails, it can cause an early exit.
Step-by-Step Resolution

Follow these steps to systematically diagnose and resolve the CrashLoopBackOff error.

1. Inspect Pod Status and Container Logs

The logs are your primary source of truth. They contain the direct error message from your application.
```
# Get a detailed overview of the failing pod
```
Check the current logs of the container kubectl logs <your-pod-name> -c <your-container-name>

If the container has already restarted, check logs from the previous instance kubectl logs <your-pod-name> -c <your-container-name> –previous `

[!IMPORTANT] The kubectl logs --previous command is invaluable. Since the container keeps crashing and restarting, the "current" logs might be empty or just show the initial startup. The --previous flag retrieves logs from the last terminated instance, which often contains the actual crash reason.

Analyze the log output for stack traces, error messages (e.g., "Connection refused," "File not found," "Segmentation fault," "OutOfMemoryError"), or any explicit reasons for termination.

2. Verify Pod Definition (YAML)

Retrieve the YAML definition of your failing pod to scrutinize its configuration.
```
kubectl get pod <your-pod-name> -o yaml > pod-debug.yaml
less pod-debug.yaml
```
Pay close attention to these sections in the pod-debug.yaml:
- image: Is the correct image and tag specified?
- command and args: Do these correctly invoke your application's entrypoint? Are there any typos or incorrect parameters?
- env: Are all necessary environment variables passed, and are their values correct?
- volumeMounts and volumes: Are all required volumes mounted correctly, and are the mountPath and subPath configurations accurate?
- resources.limits: Is there a memory limit that might be too low, leading to OOMKilled?
- livenessProbe and readinessProbe: Are these configured correctly, or could they be too aggressive, causing premature restarts?
3. Test Container Image Locally

This step helps isolate whether the issue is with your application/image or the Kubernetes environment itself.

First, identify the image used by the failing pod:
```
kubectl get pod <your-pod-name> -o jsonpath='{.spec.containers[0].image}'
```
Then, pull and run the image locally on your Ubuntu machine (assuming Docker is installed):
```
docker pull <your-image-name>:<tag>
docker run --rm -it --name debug-container <your-image-name>:<tag>
```
If your application requires specific environment variables or mounted volumes, replicate them in the docker run command:
```
docker run --rm -it 
  -e DB_HOST=localhost 
  -v /path/on/host:/path/in/container 
  --name debug-container <your-image-name>:<tag>
```
Observe the output. If it crashes locally, the problem lies within your application or container image configuration. Debug it as you would a standalone application. If it runs successfully locally, the issue is likely Kubernetes-specific (e.g., incorrect command/args in K8s manifest, K8s volume/secret issues, resource constraints, network policies).

4. Review Resource Limits (Especially Memory)

If kubectl describe pod or the container logs indicate OOMKilled (Out Of Memory Killed) as the reason for termination, your container is exceeding its allocated memory.
```
# Example from kubectl describe pod output
State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137
```
To address this:
1. Adjust resources.limits.memory: Increase the memory limit in your Deployment/Pod manifest.
```
    # my-app-deployment.yaml
    spec:
      containers:
      - name: my-app
        image: my-registry/my-app:v1.0.0
        resources:
          limits:
            memory: "512Mi" # Increase from e.g., 256Mi
            cpu: "500m"
          requests:
            memory: "256Mi"
            cpu: "250m"
```
```
    kubectl apply -f my-app-deployment.yaml
```
[!WARNING] Blindly increasing resource limits can mask underlying memory leaks in your application and lead to inefficient cluster resource utilization. It's best to profile your application's memory usage to determine appropriate limits.

5. Check Liveness and Readiness Probes

Misconfigured probes can cause a healthy container to be restarted.

If your pod definition includes livenessProbe or readinessProbe, try commenting them out or simplifying them temporarily to see if the container starts successfully.
```
# my-app-deployment.yaml
spec:
  containers:
  - name: my-app
    image: my-registry/my-app:v1.0.0
    # ... other config ...
    # livenessProbe: # Temporarily comment out or remove
    #   httpGet:
    #     path: /healthz
    #     port: 8080
    #   initialDelaySeconds: 10
    #   periodSeconds: 5
    #   failureThreshold: 3
    # readinessProbe: # Temporarily comment out or remove
    #   httpGet:
    #     path: /ready
    #     port: 8080
    #   initialDelaySeconds: 5
    #   periodSeconds: 3
    #   failureThreshold: 1
```
If removing them allows the pod to start, the probes were the issue. Reintroduce them with more lenient initialDelaySeconds, periodSeconds, and failureThreshold values, and ensure the probe endpoints are actually implemented and reachable within the container.

6. Examine Volume Mounts and Permissions

Incorrect volume mounts or permissions within the container are common causes.
1. Verify PVC Status:
2. `bash
3. kubectl get pvc
4. `
5. Ensure your PersistentVolumeClaims are in a Bound state. If not, the PV/PVC provisioner is failing.
1. Check subPath: If you're mounting a specific subdirectory of a volume using subPath, ensure the path exists within the volume.
1. Inspect Permissions Inside a Running Pod (if possible):
2. If you have another similar pod running successfully, or if you can temporarily make the failing pod run by fixing a different issue, exec into it to check paths and permissions.
```
    kubectl exec -it <running-pod-name> -- /bin/bash # or /bin/sh
    ls -la /path/to/mounted/volume
    whoami
```
If your application expects to write to a volume mounted as read-only, or if the user running the application inside the container doesn't have permissions, it will fail. You may need to specify securityContext in your Pod definition to run as a specific user or group.
```
    # my-app-deployment.yaml
    spec:
      containers:
      - name: my-app
        image: my-registry/my-app:v1.0.0
        securityContext:
          runAsUser: 1000 # Example: run as user ID 1000
          runAsGroup: 3000 # Example: run as group ID 3000
          allowPrivilegeEscalation: false
```
7. Debugging with an Init Container or Temporary Sidecar

For complex startup issues, an initContainer can help diagnose problems before your main application starts.
```
# my-app-deployment.yaml
spec:
  initContainers:
  - name: debug-init
    image: busybox:latest # A lightweight image with basic tools
    command: ["sh", "-c", "echo 'Checking network connectivity...' && ping -c 3 mysql-service && echo 'Checking file permissions...' && ls -la /app/config && sleep 10"]
    # Mount volumes if needed for checks
    volumeMounts:
    - name: app-config-volume
      mountPath: /app/config
  containers:
  - name: my-app
    image: my-registry/my-app:v1.0.0
    # ... main app config ...
```
The initContainer will run to completion. If it fails, kubectl describe pod will show its error. If it succeeds, the main container will start. You can also use a temporary sidecar container with debugging tools (e.g., net-tools, strace) that shares the main container's namespace.

8. Rebuild and Retag Image

If you suspect issues with the container image itself, try:
1. Clean Cache: Clear your Docker build cache (docker builder prune) if you're building locally.
2. Verify Base Image: Ensure the base image (e.g., ubuntu:20.04, node:16-alpine) is stable and compatible.
3. Rebuild: Rebuild your application image completely and push it with a new tag.
4. Update Deployment: Update your Kubernetes Deployment to use the new image tag.
```
    kubectl set image deployment/<your-deployment-name> <your-container-name>=<new-image-name>:<new-tag>
```
By systematically applying these debugging techniques, you can pinpoint the root cause of your CrashLoopBackOff error and restore stability to your Kubernetes applications.
July 19, 2026
Fixing ‘invalid volume mount specification’ for Docker Compose Relative Paths on macOS
Resolve Docker Compose volume mount errors on macOS when using relative paths. Understand Docker Desktop file sharing and path resolution.

Docker Compose is an indispensable tool for orchestrating multi-container applications, especially in local development environments. However, macOS users often encounter perplexing issues when attempting to mount local host directories into containers using relative paths, leading to errors like "invalid volume mount specification" or unexpectedly empty directories within containers. This guide delves into the specifics of why this occurs on macOS and provides robust, technical solutions.

Symptom & Error Signature

When running docker-compose up or docker-compose run with a docker-compose.yml file that utilizes relative paths for volume mounts (e.g., ./data:/app/data), you might observe one of the following:
1. Direct Error Message:
2. Your docker-compose command fails immediately with an error indicating an invalid volume mount.
```
    $ docker-compose up -d
    ERROR: for web Cannot start service web: invalid volume mount specification: '/Users/youruser/myproject/./data:/var/www/html'
    ERROR: Encountered errors while bringing up the project.
    ```
    Or, more generically:
    ```bash
    $ docker-compose up
    ERROR: for my_service_1  Cannot start service my_service: invalid volume mount specification: '/var/lib/docker/volumes/my_project_my_service_data/_data:/usr/src/app/data:rw'
```
1. Container Starts, but Directory is Empty/Incorrect:
2. The service appears to start without an immediate error, but when you inspect the container, the mounted directory is empty or contains an unexpected default.
```
    $ docker-compose up -d
    Creating network "myproject_default" with the default driver
    Creating myproject_db_1 ... done
```
$ docker exec -it myprojectweb1 ls -la /var/www/html/data total 0 drwxr-xr-x 2 root root 64 Oct 26 10:00 . drwxr-xr-x 1 root root 220 Oct 26 10:00 .. ` Expected files from the host are conspicuously absent.

Example docker-compose.yml snippet causing issues:
```
version: '3.8'
services:
  web:
    image: nginx:latest
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf # Relative file mount
      - ./html:/var/www/html               # Relative directory mount
      - ./data:/var/www/html/data          # Another relative directory mount
    depends_on:
      - db
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
```
volumes: db_data: `

Root Cause Analysis

The core of this issue stems from the architectural differences between Docker on Linux and Docker Desktop on macOS (and Windows).
1. Docker Desktop's Linux VM Abstraction:
2. Unlike native Linux installations where Docker interacts directly with the host kernel and filesystem, Docker Desktop on macOS runs a lightweight Linux virtual machine (VM). This VM is the actual Docker host. Your containers run inside this VM.
1. Filesystem Sharing Mechanism:
2. For containers within the Docker Desktop VM to access files on your macOS host, the host filesystem must be explicitly shared with the VM. Docker Desktop uses a mechanism (historically osxfs, now primarily VirtioFS on newer macOS versions/Docker Desktop) to expose specified host directories to the VM.
1. "Invalid Directory" Context:
2. When Docker Compose encounters a volume mount like - ./data:/app/data:
3. * It resolves the . (current directory) to an absolute path on the macOS host (e.g., /Users/youruser/myproject/data).
4. * It then attempts to instruct the Docker daemon (running in the Linux VM) to mount this path.
5. If /Users/youruser/myproject (or a parent directory like /Users/youruser) has not been explicitly added to Docker Desktop's "File Sharing" settings, the Linux VM does not see this path*. From the VM's perspective, the directory simply doesn't exist, leading to an "invalid volume mount specification" error or an empty mount because it cannot locate the source. The path exists on macOS but is not mapped into the VM's /Users namespace.
1. Permissions and UID/GID Mismatch (Secondary Cause):
2. While less common for the "invalid directory" error itself, permission issues can compound the problem or manifest as "permission denied" errors after the mount is established. Docker Desktop attempts to map macOS user/group IDs to the VM's docker user, but mismatches (especially if a process inside the container runs as a specific UID/GID) can lead to access failures even if the mount is technically valid.
Step-by-Step Resolution

The primary solution involves correctly configuring Docker Desktop's file sharing and ensuring your paths are resolved correctly.

1. Verify and Configure Docker Desktop File Sharing

This is the most frequent culprit. Ensure the directory containing your docker-compose.yml (and thus your source files) is shared with the Docker Desktop VM.
1. Open Docker Desktop Settings: Click the Docker icon in your macOS menu bar, then navigate to Settings (or Preferences on older versions).
2. Navigate to Resources > File Sharing: In the Settings window, go to Resources -> File Sharing.
3. Add Your Project Directory:
4. * You will see a list of directories shared with the Docker VM.
5. * If your project directory (e.g., /Users/youruser/myproject) is not listed, click the + button and add it.
6. * Alternatively, you can add a higher-level directory like /Users/youruser to share all projects within your home directory, though it's generally best practice to share only what's necessary.
7. Apply & Restart Docker Desktop: After adding or modifying shared directories, click Apply & Restart. This is crucial as Docker Desktop needs to remount these directories within its VM.
[!IMPORTANT] > Always restart Docker Desktop after making changes to "File Sharing" settings. Failure to do so will result in the changes not taking effect.

2. Use Absolute Paths or ${PWD} for Robustness

While relative paths generally work once file sharing is correctly configured, using absolute paths or the ${PWD} (Present Working Directory) environment variable can enhance robustness and clarity, especially in scripts or CI/CD pipelines.
1. Using $(pwd) or ${PWD}:
2. Docker Compose intelligently resolves . to the directory where docker-compose.yml resides. Explicitly using $(pwd) or ${PWD} provides the absolute path and is often preferred.
```
    version: '3.8'
    services:
      web:
        image: nginx:latest
        ports:
          - "80:80"
        volumes:
          - ${PWD}/nginx.conf:/etc/nginx/nginx.conf
          - ${PWD}/html:/var/www/html
          - ${PWD}/data:/var/www/html/data
    ```
```
1. Using Absolute Paths Directly (Less Portable):
2. You can specify the full absolute path, but this makes your docker-compose.yml less portable across different developer machines or environments.
```
    version: '3.8'
    services:
      web:
        image: nginx:latest
        ports:
          - "80:80"
        volumes:
          - /Users/youruser/myproject/nginx.conf:/etc/nginx/nginx.conf
          - /Users/youruser/myproject/html:/var/www/html
```
3. Ensure Source Directories/Files Exist

Sometimes the error isn't about Docker's configuration, but simply that the host path you're trying to mount doesn't exist.
1. Check Host Directory: Before running docker-compose, verify the directories and files referenced in your volumes section actually exist on your macOS host.
```
    $ ls -la ./data
    # If this returns "No such file or directory", create it:
    $ mkdir -p ./data
```
4. Inspect Container Mounts and Contents

After applying the fixes, verify the mounts are correct inside the container.
1. Start Services:
2. `bash
3. $ docker-compose up -d
4. `
5. Inspect Container: Get the full mount details from the Docker daemon.
6. `bash
7. $ docker ps
8. # Copy the CONTAINER ID for your web service, e.g., b0e1a2f3c4d5
$ docker inspect b0e1a2f3c4d5 | grep -A 5 "Mounts" ` Look for entries under "Mounts" that show Type: "bind", the Source (host path as seen by the VM), and Destination (container path). The Source path here should be accessible within the Docker Desktop VM.
1. Check Inside Container: Log into the container and list the contents of the mounted directory.
2. `bash
3. $ docker exec -it b0e1a2f3c4d5 ls -la /var/www/html/data
4. # You should now see the files from your host machine.
5. `
5. Address Potential Permissions Issues (Advanced)

While less common for the "invalid directory" error, if you still face "permission denied" errors after fixing the mount path, it might be a UID/GID mismatch.
1. Host Permissions: Ensure the macOS host directory has appropriate read/write permissions for your user.
2. `bash
3. $ chmod -R 755 ./data
4. $ chown -R $(whoami):staff ./data
5. `
6. Container User: Identify the user running the process inside the container. You might need to adjust the container's user or explicitly map UIDs/GIDs.
7. * Often, adding user: "${UID}:${GID}" to your docker-compose.yml service definition can help, especially for development.
```
    services:
      web:
        image: nginx:latest
        user: "${UID}:${GID}" # Map host UID/GID to container
        # ... other configurations
    ```
    > [!WARNING]
```
6. Clean Up and Rebuild

If issues persist, a clean slate can often resolve lingering problems.
1. Stop and Remove Containers/Volumes:
2. `bash
3. $ docker-compose down -v
4. `
5. The -v flag removes named volumes (like db_data in the example), ensuring a fresh start. Be cautious if you have critical data in named volumes.
1. Rebuild Images (if applicable): If your Dockerfile copies local content or its build context depends on local files, rebuild the images.
2. `bash
3. $ docker-compose up –build -d –force-recreate
4. `
5. --build forces images to be rebuilt. --force-recreate forces containers to be recreated even if their configuration hasn't changed.
By meticulously following these steps, particularly ensuring Docker Desktop's file sharing is correctly configured for your project path, you should successfully resolve "invalid volume mount" errors and achieve reliable local development with Docker Compose on macOS.
July 17, 2026
Troubleshooting Docker Container Exited with Code 137: OOM Killed on Alpine Linux
Resolve Docker containers exiting with code 137 due to OOM kills on Alpine Linux. Learn to identify, diagnose, and fix out-of-memory issues effectively.

When a Docker container abruptly stops functioning, exhibiting an "Exited (137)" status, it's a strong indicator that the Linux kernel's Out-Of-Memory (OOM) killer has terminated the process. This guide provides a comprehensive, expert-level approach to diagnose and resolve these critical memory-related issues, specifically in environments utilizing Alpine Linux base images for their Docker containers.

Symptom & Error Signature

The most prominent symptom is a container that repeatedly crashes, fails to start, or becomes unresponsive, leading to service disruption. Your application, whether it's a web server like Nginx, a database, or a custom microservice, will cease to function.

You'll typically observe the following:
1. Container Status:
2. `bash
3. $ docker ps -a
4. CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5. a1b2c3d4e5f6 my-alpine-app:latest "node /app/index.js" 2 minutes ago Exited (137) 5 seconds ago web_app
6. `
1. Docker Daemon Logs (on the host system):
2. These logs often provide the first explicit mention of an OOM event.
3. `bash
4. $ sudo journalctl -u docker.service | grep -i "oom|killed process"
5. Jul 15 10:30:45 hostname dockerd[1234]: containerd/task/v2/sender/pipe.go:39: "OOM killer killed containerida1b2c3d4e5f6"
6. Jul 15 10:30:45 hostname dockerd[1234]: shimexit: containerid_a1b2c3d4e5f6 exit status 137
7. `
1. Kernel Logs (dmesg on the host system):
2. The definitive evidence comes from the kernel itself, detailing the OOM event.
3. `bash
4. $ sudo dmesg -T | grep -i "oom-killer|killed process"
5. [Wed Jul 15 10:30:44 2026] node invoked oom-killer: gfpmask=0x100cca(GFPHIGHUSERMOVABLE|GFPCOMP), order=0, oomscoreadj=0
6. [Wed Jul 15 10:30:44 2026] oom-kill:constraint=CONTAINER,nodemask=(null),cpuset=docker/a1b2c3d4e5f6,memsallowed=0,globaloom,task_memcg=/docker/a1b2c3d4e5f6,task=node,pid=5678,uid=0
7. [Wed Jul 15 10:30:44 2026] Memory cgroup out of memory: Killed process 5678 (node) total-vm:450000kB, anon-rss:400000kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:800kB oomscoreadj:0
8. [Wed Jul 15 10:30:44 2026] Tasks state (memory and swap accounts):
9. [Wed Jul 15 10:30:44 2026] [ pid ] uid tgid totalvm rss nrptes nrpmds swapents oomscore_adj name
10. [Wed Jul 15 10:30:44 2026] [ 5678] 0 5678 112500 100000 200 0 0 0 node
11. [Wed Jul 10:30:44 2026] oom-killer: Kill process 5678 (node) score 1000 or sacrifice child
12. [Wed Jul 10:30:44 2026] Killed process 5678 (node) total-vm:450000kB, anon-rss:400000kB, file-rss:0kB, shmem-rss:0kB
13. `
14. This output clearly shows oom-killer being invoked and targeting process 5678 (node) within the specified Docker cgroup.
Root Cause Analysis

The "Exited with code 137" error specifically means the container process received a SIGKILL signal (signal 9). When this is combined with "OOM killed," it unambiguously indicates that the Linux kernel's Out-Of-Memory (OOM) killer decided to terminate the container's main process.

Here's a breakdown of the underlying reasons:
- Linux OOM Killer: This is a critical kernel mechanism designed to maintain system stability. When the system's memory (RAM + swap) is exhausted, the OOM killer steps in to free up resources by terminating one or more processes. It uses an internal heuristic (oom_score) to decide which processes are "least important" to kill.
- Docker Memory Limits (cgroups): Docker leverages Linux Control Groups (cgroups) to isolate and limit resource usage for containers.
- * mem_limit (or -m flag): This sets the maximum amount of RAM a container can use.
- * memswap_limit (or --memory-swap flag): This defines the combined limit for RAM and swap space.
- If a container's processes attempt to allocate memory beyond its memlimit, the OOM killer is triggered within that container's cgroup*, effectively killing the container without affecting other processes on the host or in other containers (unless the host itself is completely out of memory). If memswaplimit is hit, and there's no more swap available, the OOM killer will also be triggered.
- Application Memory Profile:
- * Memory Leak: The application has a bug that causes it to continuously consume more memory over time without releasing it.
- * Memory Spike: The application might have a legitimate, but unexpected, memory spike during certain operations (e.g., loading large datasets, complex computations, garbage collection cycles), especially during startup or peak load.
- * Incorrect Configuration: Application-specific memory settings (e.g., JVM heap size -Xmx, Node.js --max-old-space-size, PHP memory_limit) are misconfigured or too high for the allocated container limits.
- * Unoptimized Code/Libraries: Using inefficient data structures or libraries that are memory-hungry.
- Host System Memory Exhaustion: While less common when cgroup limits are in place, if the host system itself runs out of memory (e.g., too many containers, other demanding host processes, insufficient swap), the OOM killer might target container processes even if they haven't explicitly hit their Docker limits, or if limits are unset/very high.
- Alpine Linux Specifics: While Alpine's small base image size (often just a few MB) reduces the baseline memory footprint, it doesn't fundamentally change how the OOM killer or cgroups operate. The application inside the Alpine container is still subject to the same memory demands. Often, developers choose Alpine for its small size and then forget to properly account for the application's actual memory needs, leading to hitting tighter limits more quickly.
Step-by-Step Resolution

Resolving OOM issues requires a systematic approach, combining host-level diagnostics, container-level inspection, and application-specific tuning.

1. Confirm OOM Kill and Identify the Victim Process

Before making any changes, confirm the container was indeed OOM killed and identify the exact process within the container that was targeted.
- Check dmesg output on the host:
- `bash
- sudo dmesg -T | grep -E "oom-killer|killed process"
- `
- Look for entries similar to the "Symptom & Error Signature" section, specifically mentioning Memory cgroup out of memory and the container's cgroup path (e.g., /docker/a1b2c3d4e5f6). Note the pid and name of the killed process.
- Check Docker daemon logs:
- `bash
- sudo journalctl -u docker.service | grep "OOM killer killed container"
- `
- This confirms Docker's daemon saw the OOM event.
2. Inspect Container Memory Limits and Usage

Understand what memory limits, if any, were applied to the container, and review historical usage if possible.
- Check configured memory limits:
- `bash
- docker inspect <containeridor_name> | grep -E "Memory|Swap"
- `
- Look for Memory and MemorySwap under HostConfig. Values of 0 typically mean unlimited, but this is relative to the host's physical memory.
- Example output:
- `json
- "Memory": 0,
- "MemorySwap": 0,
- "MemoryReservation": 0,
- "KernelMemory": 0,
- "OomKillDisable": false,
- "OomScoreAdj": 0,
- `
- If Memory is 0, the container can consume all available host memory, making the host's overall memory the limiting factor. If it's a specific value (e.g., 536870912 for 512MB), that's your hard limit.
- Monitor live memory usage (if the container starts but crashes later):
- `bash
- docker stats <containeridor_name>
- `
- This command provides real-time statistics, including memory usage, against the configured limits. Run this frequently to observe memory trends before a crash.
3. Analyze Application Memory Footprint and Configuration

This is often the most critical step. You need to understand how much memory your application genuinely needs.
- Profile the application:
- * Run with generous limits: Temporarily remove or significantly increase memory limits (-m 2g --memory-swap -1) on a test environment to see how much memory the application consumes at peak load and steady state.
- * Use in-container tools: If your Alpine image has tools like ps or top (you might need to install procps), exec into a running container:
- `bash
- docker exec -it <container_id> sh
- / # apk add procps # if not already installed
- / # ps aux
- / # top
- `
- Monitor the VSZ (virtual size) and RSS (resident set size) columns.
- * Language-specific profiling:
- * Java: Tune JAVA_OPTS with -Xmx (max heap size) and -Xms (initial heap size). Remember that JVM also uses off-heap memory.
- * Node.js: Check for memory leaks using built-in profiling tools or Chrome DevTools. Adjust --max-old-space-size if needed via NODE_OPTIONS.
- * PHP: Adjust memorylimit in php.ini or via iniset().
- * Python: Libraries like memory_profiler can help.
- * Go: Go applications are often very efficient, but Goroutine leaks can lead to memory growth.
- * Review application logs: Look for errors or warnings that might indicate resource exhaustion, large data operations, or unhandled exceptions that could trigger memory spikes.
4. Adjust Docker Container Memory Limits

Based on your analysis, you will likely need to increase the memory allocated to your container.
- Increase mem_limit: This directly increases the available RAM.
- * Docker CLI:
- `bash
- docker run -d –name my-app –restart always -m 768m my-alpine-app:latest
- `
- This sets the RAM limit to 768MB.
- * Docker Compose: This is the recommended approach for defining services.
- `yaml
- # docker-compose.yml
- version: '3.8'
- services:
- my_app:
- image: my-alpine-app:latest
- container_name: my-app
- restart: always
- mem_limit: 768m # Max RAM usage
- mem_reservation: 512m # Soft limit, Docker tries to keep usage below this
- memswap_limit: 768m # Total RAM + SWAP. Set to -1 for unlimited swap
- `
- > [!IMPORTANT]
- > If memswaplimit is set to the same value as memlimit, it means the container has no swap available. If the application hits its memlimit, it will immediately be OOM killed. Setting memswaplimit to -1 allows the container to use unlimited swap on the host if its memlimit is reached. While this prevents OOM kills, it can lead to severe performance degradation if the application frequently swaps. A better approach is to set memswaplimit higher than mem_limit (e.g., 1g for 512m RAM limit) to allow for some controlled swapping.
- Consider memreservation: This is a "soft limit." Docker will try to keep the container's memory usage below this value, but will allow it to exceed it up to memlimit if the host has spare memory. This is useful for resource scheduling.
5. Optimize the Application and Dockerfile

Reduce your application's memory footprint or ensure it's configured to respect container limits.
- Application-level tuning:
- * JVM: Reduce -Xmx if it's too high for the container's memlimit. Ensure the container's memlimit is greater than -Xmx to account for off-heap memory.
- * Node.js: NODE_OPTIONS=--max-old-space-size=512 (for 512MB) can prevent Node from attempting to use too much memory.
- * PHP: Set memorylimit in php.ini to a value below your Docker container's memlimit.
- * Caching: Implement efficient caching strategies to reduce repetitive memory-intensive operations.
- * Data Structures: Review code for inefficient data structures or algorithms that cause excessive memory allocation.
- * Garbage Collection: Tune garbage collection parameters for languages like Java or Node.js.
- Dockerfile optimization (especially for Alpine):
- * Multi-stage builds: Use multi-stage builds to create extremely lean final images that only contain the necessary runtime dependencies, removing build-time tools and intermediate files.
- `dockerfile
- # Example Multi-stage Dockerfile for Node.js on Alpine
- # Stage 1: Build dependencies
- FROM node:18-alpine AS builder
- WORKDIR /app
- COPY package*.json ./
- RUN npm ci –production –frozen-lockfile
- COPY . .
- RUN npm run build # if you have a build step
Stage 2: Runtime image FROM node:18-alpine WORKDIR /app COPY –from=builder /app/nodemodules ./nodemodules COPY –from=builder /app/dist ./dist # Assuming build output is in dist COPY –from=builder /app/package.json ./package.json COPY –from=builder /app/index.js ./index.js # Your main app file EXPOSE 3000 CMD ["node", "index.js"] ` * Minimalist packages: When installing packages with apk add, only install what's absolutely necessary. Use apk add --no-cache to prevent package caches from increasing image size.

6. Increase Host System Memory or Swap (If Necessary)

If multiple containers are consistently hitting OOM issues, or the host itself is frequently low on memory, it might be time to provision more RAM for your server.
- Add more RAM: The most straightforward solution for overall system capacity.
- Increase host swap space: While not a substitute for RAM, sufficient swap space can prevent the host's global OOM killer from being invoked during temporary memory spikes.
- `bash
- > [!IMPORTANT]
- > Relying heavily on swap can severely degrade application performance due to disk I/O. Use it as a last resort or for minor overflow, not as a primary memory extension.
Example: Add a 4GB swap file on Ubuntu/Debian sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile

Make swap persistent across reboots by adding to /etc/fstab: echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab `

7. Adjust OOM Score (Advanced – Use with Extreme Caution)

For highly critical containers that must stay alive at all costs (e.g., a monitoring agent), you can adjust their oomscoreadj. A lower score makes the process less likely to be chosen by the OOM killer.
- Docker CLI:
- `bash
- docker run -d –name critical-service –oom-score-adj -500 my-alpine-app:latest
- `
- Values range from -1000 (least likely to be killed) to 1000 (most likely).
[!WARNING] > Lowering the OOM score for a memory-hungry process can lead to the OOM killer targeting other, potentially more critical, system processes or even the Docker daemon itself, potentially destabilizing the host system. Use this only for processes that are truly essential and where all other alternatives for memory optimization and provisioning have been exhausted.

8. Implement Robust Monitoring and Alerting

Proactive monitoring can help identify memory pressure before it leads to a full OOM kill.
- Memory Usage: Monitor container memory usage (RSS, cache, swap) using tools like Prometheus, Grafana, cAdvisor, or specialized APM solutions.
- Container Status: Set up alerts for containers exiting with a 137 status.
- Host System Metrics: Monitor overall host memory usage, swap usage, and dmesg output for OOM events.
- Application Metrics: If your application exposes memory metrics, integrate them into your monitoring stack.
By following these steps, you can effectively diagnose and resolve Docker container OOM issues, ensuring the stability and reliability of your containerized applications on Alpine Linux.
July 15, 2026
Troubleshooting ‘Docker Registry Push Failed: Authentication Token Expired’ on Debian 12
Fix 'authentication token expired' errors when pushing Docker images on Debian 12 Bookworm. Learn to re-authenticate, clear caches, and troubleshoot common causes.

When working with Docker, pushing images to a remote registry is a fundamental operation in development and CI/CD pipelines. Encountering an "authentication token expired" error during a docker push operation on your Debian 12 Bookworm server can halt your workflow. This guide provides a detailed breakdown of the issue, its common causes, and a systematic approach to resolve it, ensuring your image pushes succeed.

Symptom & Error Signature

You attempt to push a Docker image to a registry (such as Docker Hub, a private registry, AWS ECR, GCP Container Registry, etc.) using the docker push command, and it fails with an error message indicating an authentication issue.

The typical error output will resemble one of the following:
```
$ docker push myuser/myimage:latest
The push refers to repository [docker.io/myuser/myimage]
...
denied: requested access to the resource is denied
unauthorized: authentication required
Error: pushing to remote repository failed: unauthorized: authentication required
```
Or, more specifically, the error explicitly stating the token expiration:
```
$ docker push myuser/myimage:latest
The push refers to repository [docker.io/myuser/myimage]
...
Error: pushing to remote repository failed: unauthorized: docker.io/myuser/myimage: authentication token expired
```
Root Cause Analysis

The "authentication token expired" error means that the credentials Docker is attempting to use for authenticating with the registry are no longer valid. Here's a breakdown of the underlying reasons:
1. Expired Session Token: When you execute docker login, Docker authenticates with the registry and receives a short-lived session token (often a JSON Web Token – JWT). This token is cached locally (typically in ~/.docker/config.json) and used for subsequent docker push or docker pull operations. For security reasons, these tokens have a limited lifespan (e.g., hours or days). If your session has been active for too long, or you haven't re-logged in recently, the cached token will expire.
2. System Clock Skew (NTP Issues): A significant time difference between your Debian 12 server and the Docker registry's servers can cause problems. If your system's clock is ahead of the registry's clock, a newly issued token might appear expired to the registry. Conversely, if your clock is significantly behind, the registry might validate a token as expired even if it's still technically within its valid window on the client side.
3. Stale Credential Helper Cache: If you're using a Docker credential helper (e.g., docker-credential-pass, docker-credential-desktop, or cloud-provider specific helpers like docker-credential-ecr-login), it might be caching an expired token or failing to refresh it correctly, leading to the same authentication failure.
4. Misconfigured Registry Access Policy: While less common for an "expired" message, sometimes changes in registry access policies or user permissions could invalidate existing tokens or prevent new ones from being issued correctly.
5. Network or Proxy Issues Interfering with Refresh: Although the error specifically points to token expiration, underlying network instability or misconfigured proxy settings could, in rare cases, prevent the docker login process from successfully obtaining or refreshing a valid token.
Step-by-Step Resolution

Follow these steps to diagnose and resolve the "authentication token expired" error on your Debian 12 Bookworm system.

1. Re-authenticate with docker login

This is the most common and straightforward solution. Your existing session token has likely expired, and simply logging in again will obtain a new, valid token.
```
docker login
```
You will be prompted for your username and password.

[!IMPORTANT] If you are pushing to a specific private registry (e.g., registry.example.com), make sure to specify it during login: `bash docker login registry.example.com ` For cloud registries like AWS ECR, GCP Container Registry, or Azure Container Registry, the authentication method often involves provider-specific commands to retrieve temporary credentials, which are then piped to docker login.

Example for AWS ECR: `bash aws ecr get-login-password –region <your-region> | docker login –username AWS –password-stdin <awsaccountid>.dkr.ecr.<your-region>.amazonaws.com ` Always refer to your cloud provider's documentation for the most accurate and secure authentication method.

After successfully logging in, retry your docker push command.

2. Verify System Clock Synchronization (NTP)

An out-of-sync system clock can cause cryptographic and authentication issues. Ensure your Debian 12 server's clock is synchronized with a reliable NTP (Network Time Protocol) server.
1. Check current time status:
2. `bash
3. timedatectl status
4. `
5. Look for NTP service: active and System clock synchronized: yes. If NTP service is inactive or System clock synchronized is no, your clock might be off.
1. Install/Enable NTP service (if not active):
2. Debian 12 typically uses systemd-timesyncd by default. If you prefer a more robust NTP client like ntp or chrony:
3. `bash
4. sudo apt update
5. sudo apt install -y ntp # Or chrony
6. `
1. Ensure NTP service is running:
2. `bash
3. sudo systemctl enable –now ntp # Or chrony, or systemd-timesyncd
4. sudo systemctl restart ntp # Or chrony, or systemd-timesyncd
5. sudo systemctl status ntp # Verify status
6. `
7. After confirming NTP synchronization, retry your docker push.
3. Clear Docker Credential Cache

If docker login doesn't seem to fix the issue, a corrupted or persistently stale credential entry in Docker's configuration might be the culprit.
1. Inspect your Docker configuration:
2. `bash
3. cat ~/.docker/config.json
4. `
5. You will see a JSON structure with a "auths" section, listing registries and their associated authentication details (often just an auth field which is base64 encoded username:password).
1. Remove specific registry entries (safer):
2. If you only want to affect a specific registry, you can manually edit ~/.docker/config.json and remove the entry for the registry causing issues (e.g., docker.io or registry.example.com).
1. Clear the entire Docker credential file (more drastic):
2. > [!WARNING]
3. > Deleting config.json will require you to docker login again for all registries you've previously authenticated with. Use this if you suspect global corruption or if targeted removal doesn't work.
4. `bash
5. rm ~/.docker/config.json
6. `
7. After removal, perform a docker login (as per Step 1) for the affected registry and then retry the push.
4. Troubleshoot Credential Helpers

If you're using a docker-credential-helper, the issue might lie within its cached credentials.
1. Identify your credential helper:
2. Look at the credsStore or credHelpers entries in your ~/.docker/config.json.
3. `json
4. {
5. "auths": {
6. "docker.io": {}
7. },
8. "credsStore": "desktop" // Example: using docker-credential-desktop
9. }
10. `
11. Or:
12. `json
13. {
14. "auths": {
15. "docker.io": {}
16. },
17. "credHelpers": {
18. "registry.example.com": "ecr-login", // Example: ECR helper for a specific registry
19. "gcr.io": "gcloud" // Example: gcloud helper for GCR
20. }
21. }
22. `
1. Clear credential helper cache:
2. * docker-credential-pass (for pass utility):
3. `bash
4. pass docker-credential-helpers/docker-credential-pass/erase docker.io
5. # Or for a specific registry:
6. # pass docker-credential-helpers/docker-credential-pass/erase registry.example.com
7. `
8. * Other helpers: Consult the documentation for your specific credential helper. Often, a re-login using docker login will force the helper to refresh its tokens. If a desktop-based helper is used, restarting the Docker Desktop application might clear its cache.
5. Check Docker Daemon Status and Logs

While unlikely to be the primary cause for "token expired", ensuring the Docker daemon itself is running correctly can rule out other potential issues.
```
systemctl status docker
journalctl -u docker.service --since "10 minutes ago"
```
```
6. Review CI/CD Pipeline Configuration

If this error occurs within a CI/CD pipeline, the resolution strategies are similar but require modifying the pipeline's scripts or environment.
- Explicit Re-authentication: Ensure your CI/CD script includes a docker login command before any docker push operations.
- Short-Lived Tokens: Avoid using long-lived personal access tokens (PATs) directly in CI/CD. Instead, leverage service accounts, OIDC, or cloud-provider specific temporary credentials (like AWS ECR get-login-password) that are regularly refreshed.
- Environment Variables: If using environment variables for credentials (e.g., DOCKERUSERNAME, DOCKERPASSWORD), ensure they are correctly set and not accidentally expired or revoked at the source.
By systematically working through these steps, you should be able to identify and resolve the "Docker registry push failed authentication token expired" error on your Debian 12 Bookworm system, restoring your ability to push Docker images.
July 12, 2026
Resolving ‘Port is Already Allocated: bind failed’ Error in Docker Compose on Debian 12 Bookworm
Troubleshoot and fix the common 'port is already allocated' bind error when running Docker Compose services on Debian 12 Bookworm. Learn to identify and free up conflicting ports.

When deploying or restarting Docker Compose services, encountering a "port is already allocated bind failed" error is a common frustration for systems administrators. This issue prevents your Docker containers from binding to the host's specified network ports, leading to service startup failures. On a Debian 12 Bookworm system, this typically means another process or container is actively listening on the port your Docker service is trying to claim. Resolving this requires identifying the rogue process and either terminating it or reconfiguring your Docker Compose application to use an available port.

Symptom & Error Signature

When you attempt to start your Docker Compose services using docker-compose up or docker compose up, your containers will fail to start, and you will see an error message similar to the following in your terminal output:
```
ERROR: for my-web-app_web_1 Cannot start service web: driver failed programming external connectivity: Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
```
Or, if the error occurs during a docker run command:
```
docker: Error response from daemon: driver failed programming external connectivity: Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use.
```
The key indicator is bind: address already in use, specifically referencing the IP address (0.0.0.0 for all interfaces) and the port (80 in this example) that Docker tried to use.

Root Cause Analysis

The "port is already allocated bind failed" error fundamentally indicates a network port conflict. When a Docker container is configured to expose a port to the host machine (e.g., -p 80:80 or ports: - "80:80" in docker-compose.yml), it attempts to bind to that specific port on the host's network interface. If another process has already claimed and is listening on that port, the bind operation fails.

Common root causes include:
1. Another Docker Container: A different Docker container (perhaps from a different docker-compose.yml project, a standalone container, or a lingering container from a previous deployment) is already running and exposing the same port.
2. Native System Service: A system service, such as a web server (Nginx, Apache), database (PostgreSQL, MySQL), or another application, is configured to listen on the conflicting port. On Debian 12, Nginx or Apache are prime suspects for ports 80 and 443.
3. Orphaned Process: A previously running application or even a Docker container might have crashed or been improperly shut down, leaving its process in a state where it's still holding the port, even if the service itself is no longer functional.
4. Misconfiguration in docker-compose.yml: While less common for this specific error, an incorrect port mapping could unintentionally target an already used port, or multiple services within the same docker-compose.yml might be configured to use the same host port.
Step-by-Step Resolution

Follow these steps to diagnose and resolve the port conflict on your Debian 12 Bookworm server.

1. Identify the Conflicting Process

The first step is to determine which process is currently occupying the port Docker is attempting to use. We'll use lsof or netstat. If you don't have lsof or net-tools (which provides netstat), install them:
```
sudo apt update
sudo apt install lsof net-tools -y
```
Now, use lsof to find the process:
```
sudo lsof -i :80
```
Replace 80 with the port number reported in your error message.

Alternatively, using netstat:
```
sudo netstat -tulnp | grep :80
```
Look for output similar to this (example for port 80):
```
COMMAND     PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nginx     98765   root    6u  IPv4 123456      0t0  TCP *:http (LISTEN)
```
or
```
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      98765/nginx
```
From this output, you can identify: * PID: The Process ID (e.g., 98765). * COMMAND: The name of the process (e.g., nginx). * USER: The user running the process (e.g., root).

2. Determine if it's Another Docker Container

If the COMMAND from the previous step is docker-proxy, containerd, or if the process name is generic and doesn't immediately indicate a system service, it might be another Docker container.

List all running and stopped Docker containers:
```
docker ps -a
```
Look for any containers that expose the problematic port (e.g., 0.0.0.0:80->80/tcp). Pay close attention to the PORTS column. If you find a container mapping to the conflicting port, that's likely your culprit.

If you are using Docker Compose, you might have multiple Compose projects. Navigate to other project directories and check their status:
```
cd /path/to/another/docker-compose-project
docker-compose ps -a
```
3. Stop and Remove the Conflicting Docker Container (If Applicable)

If you identified another Docker container as the source of the conflict, you have a few options:
- Stop and Remove the Container: If the container is not needed, stop and remove it.
```
    docker stop <container_id_or_name>
    docker rm <container_id_or_name>
```
Replace <containeridor_name> with the actual ID or name found in docker ps -a.
- Stop a Conflicting Docker Compose Project: If the container belongs to another docker-compose.yml, stop that project.
```
    cd /path/to/conflicting/docker-compose-project
    docker-compose down
```
[!WARNING] Ensure you understand the impact of stopping and removing Docker containers, especially in a production environment. Data loss can occur if volumes are not properly managed.

4. Stop or Reconfigure the Conflicting Native System Service (If Applicable)

If the conflicting process is a native system service (e.g., Nginx, Apache, a custom application), you have two main approaches:
- Temporarily Stop the Service: For troubleshooting or if your Docker container is replacing the service.
```
    sudo systemctl stop <service_name>
```
For Nginx: sudo systemctl stop nginx For Apache2: sudo systemctl stop apache2

[!IMPORTANT] > Stopping critical system services can disrupt other applications or websites running on your server. Proceed with caution.
- Permanently Disable and Mask (If no longer needed): If the Docker container is a permanent replacement and the system service should never run again.
```
    sudo systemctl disable <service_name>
    sudo systemctl mask <service_name>
```
disable prevents it from starting on boot, mask prevents it from being started manually or by other services.
- Reconfigure the System Service: If you need both the system service and your Docker application to run, you must change the port that one of them uses. For a system service like Nginx, edit its configuration file:
```
    sudo nano /etc/nginx/sites-available/default
    # or /etc/nginx/nginx.conf
```
Change listen 80; to listen 8080; (or another available port). Then restart the service:
```
    sudo systemctl restart nginx
```
5. Adjust Docker Compose Port Mapping (If Necessary)

If stopping the conflicting process isn't an option (e.g., it's a critical service that must run on that port), or if you prefer to give your Docker service a different host port, modify your docker-compose.yml file.

Open your docker-compose.yml:
```
nano docker-compose.yml
```
Locate the ports section for the service that failed. Change the host port (the first number) to an available one, for example, from 80:80 to 8080:80. The container port (the second number) should remain the same unless your application inside the container expects a different port.
```
version: '3.8'
services:
  web:
    image: my-web-app-image:latest
    ports:
      - "8080:80" # Changed host port from 80 to 8080
    # ... other configurations
```
[!IMPORTANT] After modifying docker-compose.yml, you may need to update any external configurations (like Nginx reverse proxies) that were pointing to the old host port.

6. Restart Your Docker Compose Services

Once you've freed up the port or reconfigured your docker-compose.yml, attempt to start your services again:
```
docker-compose up -d
```
The -d flag runs the services in detached mode. If you want to see the logs directly, omit -d.

Check the status of your services:
```
docker-compose ps
```
All services should now show Up.

7. Implement restart Policy (Preventive Measure)

To make your Docker services more resilient to restarts or host reboots, consider adding a restart policy to your docker-compose.yml services.
```
version: '3.8'
services:
  web:
    image: my-web-app-image:latest
    ports:
      - "80:80"
    restart: unless-stopped # Add this line
    # ... other configurations
```
- no: Do not automatically restart.
- on-failure: Restart only if the container exits with a non-zero exit code.
- always: Always restart, even if it was stopped manually (unless it's explicitly stopped by docker stop).
- unless-stopped: Always restart unless the container was stopped manually (or by docker-compose down). This is often the recommended policy for production services.
This policy helps ensure that if your host reboots or Docker itself restarts, your containers will attempt to come back online, reducing the chances of a port being left allocated or a service not starting.
July 12, 2026
Troubleshooting Kubernetes ImagePullBackOff: Private Registry Authentication on Debian 12 Bookworm
Resolve 'ImagePullBackOff' errors in Kubernetes on Debian 12 when pulling images from private registries due to incorrect secret authentication.

When deploying applications to Kubernetes that utilize images from private container registries, you might encounter ImagePullBackOff errors. This specific guide focuses on scenarios where these errors stem from authentication failures with the private registry, particularly when running your Kubernetes cluster on Debian 12 (Bookworm). Understanding and correctly configuring ImagePullSecrets is crucial for seamless private image deployments.

Symptom & Error Signature

The primary symptom is that your Pods will remain in a Pending or CrashLoopBackOff state, and upon inspection, show an ImagePullBackOff status. The underlying error indicates that Kubernetes could not pull the required container image from the specified registry.

You'll typically observe this using kubectl get pods:
```
kubectl get pods
```
```
NAME                          READY   STATUS             RESTARTS         AGE
my-app-deployment-78f9xxxxxx-abcde   0/1     ImagePullBackOff   0                2m
```
Further investigation with kubectl describe pod will reveal more specific details about the failure:
```
kubectl describe pod my-app-deployment-78f9xxxxxx-abcde
```
```
...
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m                   default-scheduler  Successfully assigned default/my-app-deployment-78f9xxxxxx-abcde to k8s-worker-01
  Normal   Pulling    1m (x3 over 2m)      kubelet            Pulling image "your-private-registry.com/my-image:latest"
  Warning  Failed     1m (x3 over 2m)      kubelet            Failed to pull image "your-private-registry.com/my-image:latest": rpc error: code = Unknown desc = Error response from daemon: unauthorized: authentication required
  Warning  Failed     1m (x3 over 2m)      kubelet            Error: ErrImagePull
  Normal   BackOff    45s (x4 over 2m)     kubelet            Back-off pulling image "your-private-registry.com/my-image:latest"
  Warning  Failed     25s (x6 over 2m)     kubelet            Error: ImagePullBackOff
```
The key message here is unauthorized: authentication required, explicitly pointing to a credentials issue. Sometimes, it might be pull access denied if the credentials are valid but lack permissions for the specific image.

Root Cause Analysis

The ImagePullBackOff error, when accompanied by "unauthorized: authentication required" or "pull access denied," directly indicates that the Kubernetes runtime (typically containerd on modern Debian 12 setups) could not successfully authenticate with your private container registry.

The underlying reasons for this authentication failure can include:
1. Missing ImagePullSecrets: The Pod or its associated ServiceAccount does not have an imagePullSecrets field referencing a secret containing the registry credentials.
2. Incorrect ImagePullSecret Name: The name of the ImagePullSecret referenced in the Pod/Deployment specification does not match an existing secret in the same namespace.
3. Invalid Secret Content:
4. * Wrong Credentials: The username, password, or token stored within the docker-registry secret (or kubernetes.io/dockerconfigjson type) is incorrect, expired, or has insufficient permissions.
5. * Incorrect Registry URL: The docker-server (or the key in .dockerconfigjson) in the secret does not exactly match the registry URL used in the image path of the Pod specification (e.g., my-registry.com vs https://my-registry.com).
6. * Malformed Secret: The base64-encoded config.json is corrupted or incorrectly formatted.
7. Secret in Wrong Namespace: The ImagePullSecret exists but is in a different namespace than the Pod trying to use it. Secrets are namespace-scoped.
8. Network/DNS Issues (Less Common for Authentication): While authentication-specific errors usually point to credentials, underlying network connectivity or DNS resolution issues preventing the Kubelet/container runtime from reaching the registry can sometimes manifest similarly.
Step-by-Step Resolution

Follow these steps to diagnose and resolve private registry authentication issues leading to ImagePullBackOff on your Debian 12 Kubernetes cluster.

1. Verify the Error Details and Node Logs

Start by re-confirming the error message and identifying the problematic node and image.
```
kubectl get pods -n your-namespace
kubectl describe pod <problematic-pod-name> -n your-namespace
```
Note the exact Failed to pull image message and the node where the pod is scheduled. SSH into the identified worker node (e.g., k8s-worker-01) and check the containerd logs for more low-level details:
```
sudo journalctl -u containerd -f
```
Look for errors related to image pulling or authentication.

2. Confirm Private Registry Access Manually from a Node

To rule out credential issues and confirm network connectivity from a Kubernetes node, attempt to log in and pull the image manually using the Docker CLI (even if containerd is your runtime). This verifies that the registry is reachable and your credentials are valid.

[!IMPORTANT] This step uses the docker client for testing purposes only. Kubernetes will use containerd and its configured ImagePullSecrets. The goal here is to isolate if the credentials themselves are the problem.
1. SSH into a worker node where the problematic pod is scheduled.
2. Install the Docker client (if not already present). Debian 12 uses apt.
```
    sudo apt update
    sudo apt install -y docker.io
    ```
```
```
    docker login your-private-registry.com
    ```
    You will be prompted for your username and password. If this fails, your credentials are definitively incorrect or expired. Resolve this with your registry administrator.
```
```
    docker pull your-private-registry.com/your-image:tag
    ```
```
3. Inspect Existing ImagePullSecret Configuration

If you already have an ImagePullSecret configured, verify its contents and ensure it's correctly formatted and in the right namespace.
1. Check if the secret exists and is in the correct namespace:
```
    kubectl get secret <your-secret-name> -n your-namespace -o yaml
    ```
    Replace `<your-secret-name>` with the name referenced in your Pod/Deployment `imagePullSecrets` and `your-namespace` with the namespace where your application runs. If the secret is not found, you need to create it (proceed to step 4).
2.  **Examine the secret's content:**
```
```
    kubectl get secret <your-secret-name> -n your-namespace -o jsonpath='{.data..dockerconfigjson}' | base64 -d | jq .
    ```
```
```
    {
      "auths": {
        "your-private-registry.com": {
          "auth": "dXNlcm5hbWU6cGFzc3dvcmQ=", // base64 encoded username:password
          "email": "[email protected]"
        }
      }
    }
    ```
    > [!WARNING]
    > The `auth` field is base64-encoded `username:password`. Decoding it will expose your credentials. Be cautious in shared environments.
```
1. Verify details:
2. Registry URL: Ensure your-private-registry.com in the auths block exactly* matches the registry prefix in your image name (e.g., your-private-registry.com/my-image:tag). Sometimes, https:// is added in one place but not the other, causing a mismatch.
3. * Credentials: Confirm the decoded username and password are correct and match what you used in the manual docker login test.
4. * Format: Ensure the JSON is well-formed.
4. Create or Recreate the ImagePullSecret

If your secret was missing or incorrect, create or recreate it.

[!IMPORTANT] Ensure the secret is created in the same namespace where your Pods will run.

Option A: Recommended – Using kubectl create secret docker-registry This is the simplest and most robust method.
```
kubectl create secret docker-registry my-registry-secret 
  --docker-server=your-private-registry.com 
  --docker-username=your-username 
  --docker-password='your-password' 
  [email protected] 
  -n your-namespace
```
*   Replace `my-registry-secret` with your desired secret name.
*   Replace `your-private-registry.com` with the exact URL of your registry.
*   Replace `your-username`, `your-password`, and `[email protected]` with your actual credentials.
```
Option B: Manually creating a .dockerconfigjson and applying This method involves generating the .dockerconfigjson locally and then creating the secret from it.
1. Log in locally with Docker:
2. `bash
3. docker login your-private-registry.com
4. # Enter username and password when prompted
5. `
6. This will create/update ~/.docker/config.json with your credentials.
7. Extract and base64 encode the config.json:
```
    cat ~/.docker/config.json | base64 -w 0
    ```
    Copy the entire base64-encoded output.
```
```
    apiVersion: v1
    kind: Secret
    metadata:
      name: my-registry-secret
      namespace: your-namespace # Ensure this matches your application's namespace
    type: kubernetes.io/dockerconfigjson
    data:
      .dockerconfigjson: <PASTE_BASE64_ENCODED_OUTPUT_HERE>
    ```
    Replace `<PASTE_BASE64_ENCODED_OUTPUT_HERE>` with the string from step 2.
```
```
    kubectl apply -f my-registry-secret.yaml
```
5. Link ImagePullSecret to Pod/Deployment

Once the secret is correctly created, you must inform your Pods or Deployment to use it.

Option A: Specify imagePullSecrets in your Deployment/Pod manifest (Recommended) Modify your Deployment, StatefulSet, or Pod manifest to include the imagePullSecrets field under spec.template.spec:
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deployment
  namespace: your-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-container
        image: your-private-registry.com/my-image:latest # Ensure this image path matches the secret's registry
      imagePullSecrets:
      - name: my-registry-secret # This must match the name of the secret created in step 4
```
```
Option B: Link ImagePullSecret to the ServiceAccount If you want all Pods in a specific namespace that use a particular ServiceAccount (e.g., the default ServiceAccount) to automatically use this secret, you can link it to the ServiceAccount. This avoids adding imagePullSecrets to every Pod spec.
```
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "my-registry-secret"}]}' -n your-namespace
```
This command adds `my-registry-secret` to the `default` ServiceAccount in `your-namespace`. Any *new* Pods created in this namespace that use the `default` ServiceAccount (and don't specify their own `imagePullSecrets`) will automatically inherit this.
```
[!TIP] Linking to a ServiceAccount is convenient for namespaces where most pods pull from the same private registry. However, direct specification in the Pod/Deployment manifest (Option A) provides more explicit control and can be easier to debug for individual applications.

6. Clean Up and Retest

After updating the secret or your Pod/Deployment configuration, you need to ensure Kubernetes attempts to pull the image again.
1. Delete existing problematic pods:
2. If you modified an existing Deployment, delete the old pods to force them to be recreated with the new configuration:
```
    kubectl delete pod <problematic-pod-name> -n your-namespace
    ```
```
```
    kubectl rollout restart deployment <your-deployment-name> -n your-namespace
    ```
```
```
    kubectl get pods -n your-namespace -w
    kubectl describe pod <new-pod-name> -n your-namespace
    ```
```
7. Network and DNS Troubleshooting (If All Else Fails)

If you've exhaustively checked all authentication steps and are still facing issues (though unlikely to show unauthorized errors), verify network connectivity and DNS resolution from your Kubernetes worker nodes to the private registry.
1. SSH into a worker node.
2. Test DNS resolution:
```
    nslookup your-private-registry.com
    ```
    Ensure it resolves to the correct IP address. If not, check your node's `/etc/resolv.conf` and your cluster's DNS configuration (CoreDNS).
```
```
    ping -c 3 your-private-registry.com
    # Or, if HTTP/HTTPS registry:
    curl -v https://your-private-registry.com/v2/ # Replace with your registry's API endpoint
    ```
```
By systematically following these steps, you should be able to identify and resolve the ImagePullBackOff issue caused by private registry authentication failures on your Debian 12 Kubernetes cluster.
July 12, 2026
Troubleshooting ‘Docker Registry Push Failed: Authentication Token Expired’ on Ubuntu 20.04 LTS
Fix Docker registry push errors due to expired authentication tokens on Ubuntu 20.04 LTS. Learn to reauthenticate and resolve Docker login issues quickly.

Introduction

As an experienced Systems Administrator and DevOps engineer, encountering authentication failures when pushing Docker images to a private or public registry is a common yet frustrating issue. One specific error, "authentication token expired," indicates that your Docker client's cached credentials for the registry are no longer valid, preventing successful image uploads. This guide will walk you through diagnosing and resolving this problem on an Ubuntu 20.04 LTS system, ensuring your CI/CD pipelines and manual pushes can proceed without interruption.

Symptom & Error Signature

When attempting to push a Docker image to a remote registry, the operation fails, typically after successfully building or pulling the image. The terminal output will indicate an authentication failure, often explicitly mentioning an expired token or general authorization denial.

Here's a common error signature you might observe:
```
$ docker push myregistry.example.com/my-org/my-app:latest
The push refers to repository [myregistry.example.com/my-org/my-app]
...
denied: authentication required
Error response from daemon: authorization failed
Error response from daemon: unauthorized: authentication token expired
```
Alternatively, if you attempt to docker login and the token has expired, you might see:
```
$ docker login myregistry.example.com
Authenticating with existing credentials...
Error: Your authorization token has expired. Please log in again.
```
Root Cause Analysis

The "authentication token expired" error, while seemingly straightforward, can stem from a few underlying causes:
1. Time-Limited Tokens: This is the most common reason. Docker registry authentication tokens (be it from a docker login session or a CI/CD system's generated token) are often designed to have a finite lifespan for security reasons. Once this period elapses, the token becomes invalid.
2. Password/Credential Change: The username or password associated with the Docker registry account might have been changed or rotated by an administrator since the last successful login. The client's locally cached credentials (~/.docker/config.json) would then no longer match the active credentials on the registry.
3. Token Revocation: An administrator might have explicitly revoked the specific authentication token or session linked to your client for security or operational reasons.
4. Network/Proxy Interruption During Re-authentication (Less Common for "Expired"): While not directly causing an "expired" token, an intermittent network issue or misconfigured proxy could prevent the Docker client from successfully renewing or re-establishing an authenticated session, making it seem like the old token is the sole issue.
5. Stale Docker Daemon Session: In rare cases, a misbehaving or stale Docker daemon might hold onto outdated authentication information, preventing it from correctly using newly supplied credentials.
Step-by-Step Resolution

Follow these steps to diagnose and resolve the "authentication token expired" error.

#### 1. Verify Current Docker Login Status

First, let's check what Docker thinks its current authentication status is for the registry in question.
```
cat ~/.docker/config.json
```
This command will display your Docker configuration file, which stores cached registry credentials. Look for an entry under "auths" that corresponds to myregistry.example.com (or your specific registry hostname). If an auth field exists, it indicates Docker has some credentials cached, but they might be expired or invalid.
```
{
  "auths": {
    "myregistry.example.com": {
      "auth": "Zn...oZGl" // Base64 encoded username:password
    },
    "docker.io": {
      "auth": "YWJ...sdf",
      "credsStore": "desktop"
    }
  },
  "HttpHeaders": {
    "User-Agent": "Docker-Client/20.10.7 (linux)"
  }
}
```
The presence of an auth token or a credsStore reference confirms Docker has attempted to store credentials.

#### 2. Re-authenticate with the Docker Registry

The most direct solution for an expired token is to simply log in again. This forces the Docker client to obtain a fresh authentication token from the registry.
```
docker login myregistry.example.com
```
You will be prompted to enter your username and password for the registry.
```
Username: <your-username>
Password: <your-password>
Login Succeeded
```
[!IMPORTANT] * Ensure you use the correct username and a currently valid password or access token for the registry. If your registry uses Multi-Factor Authentication (MFA), you might need to use an API token or an application-specific password rather than your primary account password. Consult your registry's documentation for MFA login procedures. * After successful login, immediately try your docker push command again.

#### 3. Clear Stale Docker Credentials (If Re-authentication Fails)

If a direct re-authentication doesn't resolve the issue, or if you suspect corrupted local credentials, it's best to explicitly log out and then log in again.
1. Log out from the specific registry:
2. `bash
3. docker logout myregistry.example.com
4. `
5. This command removes the cached credentials for myregistry.example.com from ~/.docker/config.json.
1. Manually verify ~/.docker/config.json (Optional but recommended):
2. After docker logout, inspect the ~/.docker/config.json file again to ensure the entry for your registry has been removed or updated. If you find any residual entries or suspect corruption, you can manually edit the file.
[!WARNING] > Backup your ~/.docker/config.json before manual editing. Incorrectly modifying this file can disrupt Docker's ability to interact with any registry. > `bash > cp ~/.docker/config.json ~/.docker/config.json.bak > nano ~/.docker/config.json > ` > Carefully remove the entire block related to myregistry.example.com under the "auths" key.
1. Attempt docker login again:
2. `bash
3. docker login myregistry.example.com
4. `
5. Enter your username and password when prompted.
#### 4. Check Registry Service Health and Connectivity

Sometimes, the issue isn't with your token but with the registry being unreachable or having its own issues.
1. Test network connectivity:
2. `bash
3. ping -c 4 myregistry.example.com
4. `
5. Ensure you receive responses. If not, check your local network configuration, DNS settings, and firewall rules.
1. Test HTTPS connectivity (basic):
2. `bash
3. curl -v https://myregistry.example.com/v2/
4. `
5. This command attempts to access a common Docker registry API endpoint. A successful connection will show HTTP headers, possibly an empty JSON response, but importantly, no connection errors. If you see certificate errors (SSLCTXsetdefaultverify_paths), your system might be missing root certificates or your registry uses a self-signed certificate not trusted by your system.
[!IMPORTANT] > If your network requires a proxy for outgoing connections, ensure your Docker daemon and client are configured to use it. > * For Docker daemon: Configure proxy settings in /etc/systemd/system/docker.service.d/http-proxy.conf or similar. > * For Docker client: Set HTTPPROXY, HTTPSPROXY, and NO_PROXY environment variables.

#### 5. Verify User Permissions on the Registry

While "token expired" specifically points to authentication, it's always good to confirm that the user account you're using still has the necessary permissions to push to the target repository. This usually involves checking the registry's web interface or contacting your registry administrator.
- Confirm push permissions: Ensure the user account is authorized to push images to the specific repository path (e.g., my-org/my-app).
- Team/Role assignments: Check if the user is part of the correct team or role that grants push access.
#### 6. Restart Docker Daemon (If All Else Fails)

In rare cases, a stale Docker daemon process might be holding onto old session information. Restarting the Docker daemon can clear its internal state and force it to pick up fresh authentication details.
```
sudo systemctl restart docker
sudo systemctl status docker
```
[!WARNING] Restarting the Docker daemon will stop all running containers on your system. Ensure this is acceptable for your environment or schedule it during a maintenance window. If you have critical services running, proceed with caution.

After the daemon has restarted, attempt your docker login and docker push commands again.

By systematically working through these steps, you should be able to resolve the "Docker registry push failed: authentication token expired" issue on your Ubuntu 20.04 LTS system.
July 11, 2026
Kubernetes CrashLoopBackOff on Alpine Linux: Troubleshooting Container Startup Failures
Diagnose and resolve Kubernetes CrashLoopBackOff errors for Alpine Linux containers. Debug startup issues, misconfigurations, and resource constraints causing repeated crashes.

The CrashLoopBackOff status in Kubernetes indicates that a container within a pod is repeatedly starting, crashing, and then restarting after a back-off delay. While common across all Linux distributions, troubleshooting this on Alpine Linux containers presents unique challenges due to its minimalist design, use of musl libc, and often a different default shell (ash instead of bash). This guide provides a highly technical, step-by-step approach to diagnose and resolve CrashLoopBackOff errors specific to Alpine-based containers in a Kubernetes environment.

Symptom & Error Signature

Users will observe their application being unavailable or intermittently failing to respond. When inspecting the Kubernetes cluster, pods will report a CrashLoopBackOff status.

You can observe this status using kubectl get pods:
```
kubectl get pods
```
```
NAME                          READY   STATUS             RESTARTS        AGE
my-alpine-app-xxxxxxxxx-yyyyy   0/1     CrashLoopBackOff   5               2m30s
```
Further details can be gleaned from kubectl describe pod, which will often show a series of Back-off restarting failed container events:
```
kubectl describe pod my-alpine-app-xxxxxxxxx-yyyyy
```
```
...
Status:         Failed
Reason:         CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    137 # or other non-zero code
  Started:      Fri, 10 Jul 2026 09:00:15 -0700
  Finished:     Fri, 10 Jul 2026 09:00:16 -0700
Ready:          False
Restart Count:  5
...
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Pulled     2m45s (x6 over 5m)   kubelet            Container image "my-repo/my-alpine-app:latest" already present on host
  Normal   Created    2m45s (x6 over 5m)   kubelet            Created container my-alpine-app
  Normal   Started    2m45s (x6 over 5m)   kubelet            Started container my-alpine-app
  Warning  BackOff    10s (x6 over 4m)     kubelet            Back-off restarting failed container
  Warning  Unhealthy  5s (x10 over 4m)     kubelet            Liveness probe failed: HTTP GET http://10.42.0.10:8080/healthz fail: connection refused
```
The crucial information lies within the Exit Code and the pod's logs. An Exit Code: 137 typically indicates an OOMKilled (Out Of Memory Killed) event, while other non-zero exit codes point to application-specific failures.

Root Cause Analysis

A CrashLoopBackOff on Alpine Linux containers usually stems from one or more of the following issues:
1. Application Code Errors: Unhandled exceptions, logic errors, or incorrect startup sequences within the application itself causing it to exit prematurely.
2. Missing Dependencies or Files:
3. * Alpine Specifics: The minimalist nature of Alpine often means common utilities or libraries (e.g., bash, procps, glibc) expected by your application or startup scripts might be missing. Alpine uses apk for package management.
4. * Incorrect paths for configuration files, data volumes, or dynamically loaded libraries.
5. * Missing ConfigMaps or Secrets that the application needs to start.
6. Permissions Issues:
7. * The container's entrypoint script or application binaries lack execute permissions.
8. * The application attempts to write to a path where its user lacks permissions, especially common with non-root users or scratch images with added files.
9. Resource Constraints:
10. * Out Of Memory (OOMKilled): The container tries to consume more memory than specified by its limits.memory in the Pod's YAML, leading the kernel to terminate it. This is frequently indicated by Exit Code: 137.
11. * CPU throttling due to low limits.cpu causing startup processes to time out.
12. Incorrect Entrypoint/Command:
13. * The ENTRYPOINT or CMD in the Dockerfile, or the command and args in the Pod's YAML, refer to a non-existent executable or script within the Alpine container.
14. * Expecting bash when only sh (BusyBox shell) is available on Alpine.
15. * The entrypoint script fails due to syntax errors or incorrect environment variable usage.
16. Network Issues: The application might crash if it attempts to connect to an essential external service (database, message queue, API) that is unreachable or misconfigured at startup time.
17. Liveness/Readiness Probe Misconfiguration:
18. * Probes are configured to fail too quickly, before the application has fully initialized.
19. * The probe path or port is incorrect.
20. * The probe target itself is faulty.
Step-by-Step Resolution

Troubleshooting requires a systematic approach, starting with inspecting the most immediate symptoms.

1. Inspect Pod Status and Events

Begin by getting a high-level overview of the pod's state and recent events.
```
kubectl get pods my-alpine-app-xxxxxxxxx-yyyyy
```
```
kubectl describe pod my-alpine-app-xxxxxxxxx-yyyyy
```
[!IMPORTANT] Pay close attention to the Events section in the kubectl describe output. Look for Back-off restarting failed container, OOMKilled, or other specific error messages that might point to resource issues or immediate failures. The Exit Code under Last State is also critical.

2. Review Container Logs

This is the most crucial step. The container's logs will often contain the exact error message or stack trace explaining why it crashed.
```
kubectl logs my-alpine-app-xxxxxxxxx-yyyyy
```
If the container has already crashed and restarted, the current logs might be empty or only show the latest attempt. To see logs from previous failed attempts, use the --previous (or -p) flag:
```
kubectl logs my-alpine-app-xxxxxxxxx-yyyyy --previous
```
[!WARNING] If your pod has multiple containers, specify the container name using -c <container-name> to get logs for the correct container. kubectl logs my-alpine-app-xxxxxxxxx-yyyyy -c my-alpine-container --previous

Look for: * Application-specific error messages. * "command not found" errors, often due to missing binaries or incorrect shell (bash vs sh). * Permission denied errors. * Memory allocation failures. * Output from startup scripts.

3. Validate Entrypoint/Command and Environment

Alpine Linux is minimal. Confirm that your Dockerfile's ENTRYPOINT or CMD, or your pod's command and args, are correctly configured for Alpine.
- Shell Compatibility: If your entrypoint script uses bash-specific syntax (e.g., [[ ... ]], source), it will fail if only sh (BusyBox) is available. Either rewrite the script for sh or explicitly install bash in your Dockerfile (apk add bash).
```
    # Dockerfile Example for bash
    FROM alpine:3.18
    RUN apk add --no-cache bash
    COPY startup.sh /usr/local/bin/startup.sh
    RUN chmod +x /usr/local/bin/startup.sh
    ENTRYPOINT ["/usr/local/bin/bash", "/usr/local/bin/startup.sh"] # Use bash explicitly
```
- Executable Paths: Ensure the executable path is correct. If your ENTRYPOINT is /app/start.sh, verify /app/start.sh exists and has execute permissions (chmod +x /app/start.sh in Dockerfile).
4. Check Resource Requests and Limits (OOMKilled)

If kubectl describe pod showed Exit Code: 137 or the logs indicate out-of-memory errors, your container is likely being terminated by the kernel.
- Review Pod YAML: Examine the resources section for your container.
```
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi" # <-- This is often the culprit
        cpu: "500m"
```
- Increase Limits: Temporarily increase limits.memory and limits.cpu to see if the issue resolves. If it does, you've found a resource constraint. Then, work on optimizing your application's resource usage or find a suitable, permanent limit.
[!IMPORTANT] > Increasing limits without understanding why the application uses so much memory is a temporary fix. Profile your application to identify memory leaks or inefficient resource consumption.

5. Verify Configuration (ConfigMaps, Secrets, Environment Variables)

Ensure all external configurations are correctly injected into the container.
- ConfigMaps & Secrets: Check that they are mounted as files or injected as environment variables as expected by your application.
```
    kubectl get configmap my-config -o yaml
    kubectl get secret my-secret -o yaml # Be careful with sensitive data in output
```
- Environment Variables: Confirm that all necessary environment variables are set and have the correct values, especially paths or connection strings.
```
    env:
      - name: DB_HOST
        value: "my-database-service"
      - name: APP_CONFIG_PATH
        value: "/etc/config/app.conf"
```
6. Test Container Locally with Docker

Isolate the problem from Kubernetes by running the container image locally with docker run. This helps differentiate between an issue with your container image versus a Kubernetes configuration problem.
```
docker run --rm -it 
  -e DB_HOST="my-database-service" 
  -v /path/to/local/config:/etc/config  # Mount relevant configs if needed
  my-repo/my-alpine-app:latest sh # Or bash, if installed.
```
If you can sh into the container and execute the entrypoint command manually (/usr/local/bin/startup.sh in the example), but it still crashes locally, the issue is likely within the application or the image build.

7. Diagnose Liveness/Readiness Probes

If the pod is marked Unhealthy in kubectl describe events, your probes might be failing.
- Adjust initialDelaySeconds: Give your application enough time to start before the probes begin.
- Check periodSeconds and timeoutSeconds: Ensure probes aren't too aggressive or timing out too quickly.
- Verify Probe Endpoint: If using HTTP probes, GET the endpoint manually from within a working container (using kubectl exec) or locally to ensure it responds correctly.
```
    kubectl exec -it my-alpine-app-xxxxxxxxx-yyyyy -c my-alpine-container -- sh
    # Inside the container:
    apk add --no-cache curl # Install curl if not present
    curl -v http://localhost:8080/healthz
```
8. Network Connectivity (if applicable)

If your application depends on external services, check network connectivity from within the container.
```
kubectl exec -it my-alpine-app-xxxxxxxxx-yyyyy -c my-alpine-container -- sh
# Inside the container:
ping <database-host> # or other external service
curl <external-api-endpoint>
```
If DNS resolution fails, check your Pod's dnsPolicy and dnsConfig.

By methodically working through these steps, focusing on the logs and Alpine-specific considerations, you can effectively diagnose and resolve CrashLoopBackOff issues in your Kubernetes deployments.
July 10, 2026
Resolve ‘Docker compose port is already allocated bind failed’ Error on macOS
Fix 'port is already allocated bind failed' errors with Docker Compose on macOS by identifying and terminating conflicting processes. Master container networking.

Introduction

As an experienced DevOps engineer, encountering bind: address already in use errors when launching Docker Compose applications is a common scenario, especially in local development environments on macOS. This typically manifests as your Docker containers failing to start because a port they're configured to expose is already being used by another process on your host machine. This guide will walk you through a highly technical and precise methodology to diagnose and resolve this issue, ensuring your development workflow remains uninterrupted.

Symptom & Error Signature

When you attempt to start your Docker Compose services using docker compose up -d (or docker-compose up -d for older Docker Compose v1 installations), one or more services fail to initialize, displaying an error similar to the following in your terminal output:
```
[+] Running 1/2
 ⠿ Container myapp-backend-1  Stopped                                                                                                                                                                                           0.0s
 ⠿ Container myapp-frontend-1  Error                                                                                                                                                                                             0.0s
Error response from daemon: driver failed programming external connectivity: Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
```
Or, if a database container fails:
```
Error response from daemon: driver failed programming external connectivity: Error starting userland proxy: listen tcp 0.0.0.0:5432: bind: address already in use
```
The key phrase here is bind: address already in use, indicating a network port conflict on the host system.

Root Cause Analysis

The "Docker compose port is already allocated bind failed error" fundamentally means that a specified TCP or UDP port, which your Docker Compose service intends to use on the host machine (the left side of HOSTPORT:CONTAINERPORT mapping), is currently occupied by another active process.

Common culprits for this conflict on macOS include:
1. Another Docker Container: A previously running Docker Compose stack, or individual Docker container, that was not properly shut down (e.g., using docker compose down or docker stop) might still be binding to the port.
2. Local Development Servers: Applications like Node.js servers, Python Flask/Django, Ruby on Rails, PHP's built-in web server, or even local proxy servers (e.g., Nginx, Apache installed via Homebrew) are often configured to listen on common development ports (e.g., 3000, 8000, 8080, 5000).
3. Database Servers: Local installations of PostgreSQL (5432), MySQL (3306), Redis (6379), or MongoDB (27017) are common.
4. System Services: Less common for typical web ports, but macOS can have services using specific ports. For instance, sometimes launchd or cupsd might interfere.
5. Browser Extensions/Proxies: Rarely, but certain browser extensions or system-wide proxies might bind to ports.
6. Incomplete Shutdown: A previous docker compose up command that was interrupted (e.g., Ctrl+C) might leave processes in a zombie state or resources partially allocated.
The problem specifically arises when Docker's userland proxy attempts to bind the HOST_PORT on the macOS host, and the operating system rejects this attempt because the port is already in use by a different process, identified by its Process ID (PID).

Step-by-Step Resolution

Follow these steps meticulously to identify and resolve the port conflict.

1. Identify the Conflicting Process

The first step is to pinpoint exactly which process is using the desired port on your macOS system. We'll use the lsof (list open files) utility, which is invaluable for network troubleshooting on Unix-like systems.
```
# Replace 80 with the specific port reported in your error message
sudo lsof -i :80
```
Example Output (for port 80):
```
COMMAND   PID     USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
nginx    1234  myuser    7u  IPv4 0x1234567890abcdef      0t0  TCP *:http (LISTEN)
```
In this example, nginx with PID 1234 is listening on port 80.

[!IMPORTANT] If lsof returns no output, it means the port is not currently allocated by a long-running process. In such a rare case, the conflict might be transient, or another Docker container is the culprit which you'll address in step 3.

2. Terminate the Conflicting Process

Once you've identified the PID (Process ID) of the conflicting process, you can terminate it.
```
# Replace 1234 with the PID identified in the previous step
kill -9 1234
```
[!WARNING] Using kill -9 (SIGKILL) forces immediate termination of a process, which can lead to ungraceful shutdowns and potential data loss if the process was performing critical operations. Always try kill PID first (sends SIGTERM, allowing for graceful shutdown) and only resort to kill -9 if the process persists. However, for development servers, kill -9 is generally acceptable.

After killing the process, re-run sudo lsof -i :<PORT> to confirm the port is now free.

3. Check for Lingering Docker Containers

Sometimes, the conflict is with another Docker container or a previous instance of your current stack that didn't shut down cleanly.

a. List All Running and Exited Containers
```
docker ps -a
```
Look for any containers that might be mapping the problematic port. If you see containers related to a previous project or an improperly stopped service, you might need to stop and remove them.

b. Stop and Remove All Containers (Use with Caution!)

If you're unsure which container is the culprit, or if you want a clean slate for your Docker environment, you can stop and remove all currently active and exited containers.
```
docker stop $(docker ps -aq) # Stops all running containers
docker rm $(docker ps -aq)   # Removes all stopped containers
```
[!WARNING] This command will stop and remove all containers on your system. Ensure you have no critical containers running (e.g., production databases) that you didn't intend to stop. For a development machine, this is generally safe.

c. Clean Up Docker Networks

Less common for port conflicts, but good practice to clear out old network configurations.
```
docker network prune -f
```
d. Prune Docker System (Last Resort)

For a complete cleanup of dangling images, containers, volumes, and networks, use docker system prune. This is a more aggressive cleanup.
```
docker system prune -a
```
[!IMPORTANT] docker system prune -a will remove all stopped containers, all dangling images, all unused networks, and all build cache. Use with extreme caution as it will free up significant disk space but require re-pulling images for your projects.

4. Modify Docker Compose Port Mapping

If repeatedly killing processes is tedious, or if the conflicting process is a system service you don't want to stop, consider changing the host port your Docker Compose service binds to.

Open your docker-compose.yml file and modify the ports section for the affected service.

Original (conflict on port 80):
```
services:
  web:
    image: nginx:latest
    ports:
      - "80:80" # Host port 80 mapped to container port 80
```
Modified (using host port 8080):
```
services:
  web:
    image: nginx:latest
    ports:
      - "8080:80" # Host port 8080 mapped to container port 80
```
Now, your service will be accessible via http://localhost:8080 instead of http://localhost. Remember to update any application code or configurations that expect the service on the old port.

5. Restart Docker Desktop

Occasionally, Docker Desktop itself might enter an inconsistent state where ports aren't released correctly, or its internal proxy is malfunctioning. A full restart can often resolve these transient issues.
- Click the Docker Desktop icon in your macOS menu bar.
- Select "Quit Docker Desktop".
- Wait a few seconds, then reopen Docker Desktop from your Applications folder.
6. Verify Firewall Rules (Advanced)

While rare for bind failed errors specifically on a local environment, incorrect firewall rules could theoretically block Docker's ability to bind to ports. On macOS, this involves pfctl.
```
sudo pfctl -s rules # Display current firewall rules
sudo pfctl -s anchor docker # Display Docker's specific anchor rules
```
[!NOTE] Do not modify pfctl rules unless you are highly experienced with macOS networking and understand the implications. Incorrect pfctl configurations can severely impact your network connectivity. For this specific error, firewall issues are almost never the root cause; lsof is the definitive diagnostic tool.

After performing the necessary steps, retry starting your Docker Compose services:
```
docker compose up -d
```
Your Docker application should now launch successfully without port allocation errors.
July 10, 2026