Staging ButItWorkedLocal

Tag: rockylinux

  • Troubleshooting: Docker Image Pull Limit Exceeded on CentOS Stream / Rocky Linux

    Resolve 'Docker image pull limit exceeded' errors on CentOS Stream or Rocky Linux. Learn to authenticate, configure registries, and optimize pulls.

    Introduction

    As a Systems Administrator or DevOps engineer managing containerized applications on CentOS Stream or Rocky Linux, encountering Docker image pull limit exceeded errors can halt deployments, break CI/CD pipelines, and prevent local development. This issue typically manifests as docker pull or podman pull commands failing to retrieve images from Docker Hub, often accompanied by messages indicating rate limits for anonymous users. This guide provides a comprehensive technical breakdown and actionable steps to diagnose and resolve this common problem.

    Symptom & Error Signature

    When you attempt to pull an image from Docker Hub without authenticating, or if your authenticated pulls exceed your plan's limits, you will observe errors similar to the following in your terminal or build logs:

    Docker Engine Error Output:

    [root@host ~]# docker pull nginx:latest
Using default tag: latest
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may resume pulls in a short while or upgrade your account. See https://docs.docker.com/docker-hub/rate-limit/ for more information.

    Or, for an anonymous pull attempt:

    [root@host ~]# docker pull someuser/someimage:latest
Error response from daemon: pull access denied for someuser/someimage, repository does not exist or may require 'docker login': denied: You have reached your pull rate limit. You may resume pulls in a short while or upgrade your account.

    Podman Error Output (Common on Rocky/CentOS):

    [root@host ~]# podman pull docker.io/nginx:latest
Trying to pull docker.io/nginx:latest...
Error: initializing source docker.io/nginx:latest: pinging container registry registry-1.docker.io: GET https://registry-1.docker.io/v2/: unexpected status code 429 Too Many Requests: toomanyrequests: You have reached your pull rate limit. You may resume pulls in a short while or upgrade your account. See https://docs.docker.com/docker-hub/rate-limit/ for more information.

    Root Cause Analysis

    The underlying reason for the "Docker image pull limit exceeded" error is Docker Hub's rate limiting policy. Docker Hub imposes limits on the number of image pulls based on whether a user is authenticated or anonymous, and these limits are enforced per IP address over a rolling 6-hour window.

    1. Anonymous Pull Limits: Unauthenticated users (those not logged in to Docker Hub) are typically limited to 100 pulls per 6 hours per IP address.
    2. Authenticated Pull Limits: Users logged in with a free Docker ID account are typically limited to 200 pulls per 6 hours per IP address. Paid subscriptions offer significantly higher limits or unlimited pulls.

    Several scenarios frequently lead to these limits being hit:

    • Shared IP Addresses: In cloud environments, data centers, or behind corporate NATs/proxies, multiple servers or users might share a single public IP address. This means all their pull requests contribute to the same rate limit bucket, causing the limit to be reached much faster than anticipated.
    • CI/CD Pipelines: Automated build and deployment pipelines often perform numerous docker pull operations as part of their workflow (e.g., pulling base images, caching layers, deploying new versions). Without proper authentication or caching, these can quickly exhaust pull limits, especially if multiple pipelines run concurrently.
    • Frequent Image Updates: If your applications rely on rapidly changing base images or if you frequently rebuild images that pull many layers, you can hit the limits.
    • Development Environments: Teams of developers frequently pulling images for local development can collectively exhaust shared IP limits.
    • Lack of Authentication: The most common immediate cause is simply not being logged in to Docker Hub. Many system-level container orchestrators or build scripts might operate without explicit docker login commands, leading to anonymous pulls.

    Step-by-Step Resolution

    Here are the recommended steps to resolve and mitigate Docker Hub pull rate limit issues on CentOS Stream and Rocky Linux.

    1. Verify Current Rate Limit Status (Optional)

    Before attempting a fix, you can optionally check your current rate limit status using curl against Docker Hub's API. This provides the X-RateLimit-Limit and X-RateLimit-Remaining headers.

    [root@host ~]# REGISTRY_AUTH_HEADER="$(curl -s -H "Content-Type: application/json" -X POST -d '{"username": "YOUR_DOCKER_USERNAME", "password": "YOUR_DOCKER_PASSWORD"}' https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull | jq -r .token)"
[root@host ~]# curl -s -I -H "Authorization: Bearer $REGISTRY_AUTH_HEADER" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i 'ratelimit'

    Replace YOURDOCKERUSERNAME and YOURDOCKERPASSWORD with your actual Docker Hub credentials. If you are an anonymous user, you can omit the authentication part and just try:

    [root@host ~]# curl -s -I https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest | grep -i 'ratelimit'

    The output will show headers like: ` ratelimit-limit: 100;w=21600 ratelimit-remaining: 99;w=21600 ` w=21600 represents the 6-hour window in seconds.

    2. Authenticate to Docker Hub

    The most straightforward and recommended solution is to authenticate your Docker daemon or Podman environment with a Docker Hub account. This immediately increases your pull limit from 100 to 200 pulls per 6 hours and is a prerequisite for higher limits.

    For Docker Engine:

    [root@host ~]# docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: your_docker_username
Password:
Login Succeeded

    This command stores your credentials in ~/.docker/config.json. If you run it as root, it stores them in /root/.docker/config.json. Ensure the user running docker pull has access to these credentials.

    [!IMPORTANT] If running Docker in a CI/CD pipeline, consider using environment variables for DOCKERUSERNAME and DOCKERPASSWORD and piping them to docker login for security. echo $DOCKERPASSWORD | docker login --username $DOCKERUSERNAME --password-stdin

    For Podman (on CentOS Stream / Rocky Linux):

    Podman manages registry authentication similarly to Docker.

    [root@host ~]# podman login docker.io
Authenticating with existing credentials for docker.io
Username: your_docker_username
Password:
Login Succeeded!

    Podman stores credentials in ~/.config/containers/auth.json (for rootless users) or /run/containers/0/auth.json (for root users).

    3. Purchase a Docker Subscription

    If 200 pulls per 6 hours is insufficient for your needs, you will need to upgrade your Docker Hub account to a paid subscription (e.g., Pro, Team, Business). These plans offer significantly higher or virtually unlimited pull rates.

    • Visit Docker Hub Plans for detailed information.
    • Once subscribed, ensure you docker login (or podman login) with the account associated with the subscription.

    4. Configure a Private Docker Registry or Mirror

    For environments with high pull volumes, multiple servers, or strict network policies, setting up a local caching registry mirror is an excellent long-term solution. This reduces external pulls, speeds up image retrieval, and provides greater control.

    You can run your own registry using the docker/distribution image or utilize cloud-native registry services (e.g., Red Hat Quay.io, AWS ECR, Azure Container Registry, Google Container Registry).

    Setting up a Docker Hub Mirror (using docker/distribution):

    1. Start the Registry Container:
    2. On your CentOS Stream/Rocky Linux host, run the registry. Choose a host with ample storage and network bandwidth.
        [root@host ~]# docker run -d -p 5000:5000 --restart=always --name registry 
      -v /var/lib/registry:/var/lib/registry 
      registry:2
    ```
    1. Configure Docker Daemon to Use the Mirror:
    2. Edit or create the Docker daemon configuration file /etc/docker/daemon.json.
        [root@host ~]# vi /etc/docker/daemon.json

    Add the following content, replacing YOURREGISTRYMIRROR_IP with the IP address or hostname of your registry mirror:

        {
      "registry-mirrors": ["http://YOUR_REGISTRY_MIRROR_IP:5000"],
      "log-level": "info"
    }
    ```
    1. Restart Docker Service:
        [root@host ~]# systemctl daemon-reload
    [root@host ~]# systemctl restart docker

    Verify the configuration by checking daemon info: `bash [root@host ~]# docker info | grep 'Registry Mirrors' Registry Mirrors: http://127.0.0.1:5000 `

    Now, when docker pull is executed, it will first attempt to pull from your local mirror. If the image isn't found there, the mirror will pull it from Docker Hub, cache it, and then serve it to your Docker daemon. Subsequent pulls of the same image will come from the local mirror.

    [!WARNING] For production environments, ensure your private registry is secured with TLS/SSL and authentication. Running an unauthenticated HTTP registry in production poses significant security risks. You would typically need to add --insecure-registries YOURREGISTRYMIRROR_IP:5000 to daemon.json if not using HTTPS, but this is highly discouraged for security reasons.

    Configuring Podman to Use a Registry Mirror:

    Podman's configuration for registries is managed via /etc/containers/registries.conf.d/.

    1. Create a Configuration File:
    2. `bash
    3. [root@host ~]# vi /etc/containers/registries.conf.d/00-mirror.conf
    4. `
    1. Add Mirror Configuration:
    2. `toml
    3. [[registry]]
    4. location = "docker.io"
    5. mirror = ["YOURREGISTRYMIRROR_IP:5000"]
    6. `
    7. Replace YOURREGISTRYMIRROR_IP with the IP address or hostname of your registry mirror.
    1. No service restart is typically needed for Podman, it reads the configuration directly.

    5. Optimize Image Pulling Strategy

    Beyond direct authentication and mirroring, optimizing how you pull images can significantly reduce the overall pull count.

    • Cache Images in CI/CD Runners: For CI/CD systems like GitLab CI, Jenkins, or GitHub Actions, configure runners to persist Docker image caches across jobs. This avoids repeated pulls of base images.
    • Minimize Base Image Changes: Avoid frequent updates to base images unless absolutely necessary. Pinning to specific digests (myimage@sha256:digest) instead of tags (myimage:latest) can improve determinism, though it won't directly reduce rate limits if the image still needs to be pulled.
    • Leverage Multi-Stage Builds: Use multi-stage Dockerfiles to build your application and then copy only the necessary artifacts into a minimal final image. This reduces the size and number of layers that need to be pulled for the final deployable image.
    • Prune Less Frequently: If storage allows, consider cleaning up old Docker images less aggressively. Keeping frequently used images locally can prevent unnecessary re-pulls. Use docker image prune -a cautiously.

    6. Utilize an Alternative Container Registry

    If Docker Hub's policies or technical limitations consistently hinder your operations, consider migrating your images to an alternative container registry. Many cloud providers offer highly scalable and integrated registries:

    • Red Hat Quay.io: A robust registry solution, particularly popular in enterprise and OpenShift environments.
    • AWS Elastic Container Registry (ECR): Seamlessly integrates with other AWS services.
    • Azure Container Registry (ACR): Integrated with Azure, supports geo-replication.
    • Google Container Registry (GCR) / Artifact Registry: Integrated with GCP, high performance.
    • GitLab Container Registry: Built into GitLab, excellent for CI/CD integration.

    These registries typically offer their own rate limits, which are often more generous or tied to your cloud service subscription, and provide better control over image storage and distribution. You would push your images to these registries and then configure your docker pull commands or daemon.json to pull from the alternative registry.

  • CentOS Stream / Rocky Linux: UFW Blocking Docker Container Port Exposures

    Troubleshoot and resolve UFW firewall conflicts preventing Docker containers from exposing ports on CentOS Stream or Rocky Linux systems. Reclaim access.

    CentOS Stream / Rocky Linux: UFW Blocking Docker Container Port Exposures

    When deploying containerized applications with Docker on CentOS Stream or Rocky Linux, it's common to use a firewall to secure the host system. While firewalld is the native solution, some administrators might opt for UFW (Uncomplicated Firewall) for its perceived simplicity. However, UFW's default configuration can often clash with Docker's internal networking, leading to frustrating scenarios where your Docker containers appear to be running correctly, but their exposed ports are inaccessible from outside the host. This guide will walk you through diagnosing and resolving these conflicts, restoring connectivity to your containerized services.

    Symptom & Error Signature

    The primary symptom is a lack of external connectivity to a Docker container's exposed port, even though docker ps indicates the port is mapped and the container is running.

    Typical observations include:

    • Browser/Client:
    • * "Connection Refused"
    • * "ERRCONNECTIONTIMED_OUT"
    • * "Site cannot be reached"
    • curl from an external machine:
    • `bash
    • curl: (7) Failed to connect to <yourserverip> port <exposed_port> after XXX ms: Connection refused
    • `
    • curl from the Docker host itself (often works):
    • `bash
    • # This might work if accessing via localhost, confirming the service is running
    • curl http://localhost:<exposed_port>
    • `
    • Docker Container Logs: The container's logs (e.g., docker logs <container_id>) usually show no errors related to networking or port binding, as the issue lies with the host's firewall.
    • ss or netstat output:
    • `bash
    • # On the Docker host, showing docker-proxy listening on the exposed port
    • sudo ss -tulpn | grep <exposed_port>
    • # Example output for a container exposing port 8080:
    • # tcp LISTEN 0 4096 0.0.0.0:8080 0.0.0.0:* users:(("docker-proxy",pid=12345,fd=4))
    • `
    • This indicates docker-proxy is correctly listening, but external access is blocked by the host firewall.

    Root Cause Analysis

    The conflict between UFW and Docker stems from how both tools manage iptables rules on the Linux host.

    1. Docker's iptables Management: When you expose a port from a Docker container (e.g., -p 8080:80), Docker creates a set of iptables rules. These rules perform Network Address Translation (NAT) to redirect incoming traffic from the host's exposed port to the container's internal IP and port. It also creates rules in the FORWARD chain to allow this traffic. Docker typically inserts its rules into specific chains like DOCKER and DOCKER-USER.
    1. UFW's iptables Management: UFW is a user-friendly frontend for iptables. By default, UFW often sets its DEFAULTFORWARDPOLICY to DROP in /etc/default/ufw. This means any traffic that is forwarded through the host (which includes traffic destined for Docker containers) will be blocked unless explicitly allowed by UFW rules.
    1. The Conflict: The core problem is rule order and chain traversal. UFW often places its DROP rules very early in the FORWARD chain of the filter table. Docker's rules, while present, may be jumped to after UFW has already decided to DROP the packet. This prevents external traffic from ever reaching the DOCKER or DOCKER-USER chains where the allow rules for your containers reside.
    1. CentOS/Rocky Linux Specifics: While UFW is not native to CentOS/Rocky (which uses firewalld), users who install UFW on these systems might inadvertently create more complex iptables interactions if firewalld isn't fully disabled or if they are unfamiliar with the iptables structure on these distributions. The fundamental UFW-Docker conflict, however, remains consistent across distributions.

    Step-by-Step Resolution

    There are several ways to resolve this, ranging from simple but less secure, to more integrated and robust. We'll start with the recommended approach for UFW users.

    #### 1. Initial Diagnostics & Verification

    Before making changes, confirm UFW is active and Docker is running:

    # Check UFW status (look for "Status: active" and "Default: deny (forward)")

    Expected output example: # Status: active # Logging: on (low) # Default: deny (incoming), deny (outgoing), deny (forward) # New profiles: skip

    Check Docker container status and port mapping docker ps

    Confirm the service is listening on the host (e.g., for port 8080) sudo ss -tulpn | grep 8080

    Attempt external connection from another machine (or via curl <yourserverip>:<port>) # This should fail if the issue persists curl -v http://<YOURSERVERIP>:<CONTAINER_PORT> `

    #### 2. Method A: Recommended Docker/UFW Integration (More Secure)

    This method involves adjusting UFW's iptables rules to ensure Docker's traffic is processed before UFW's default DROP policy on forwarded packets. This allows Docker to manage its own port forwarding rules effectively without compromising overall firewall security.

    1. Edit UFW's after.rules:
    2. UFW uses before.rules and after.rules to apply iptables rules before or after UFW's automatically generated rules. We need to insert rules that prioritize Docker's FORWARD requirements.

    Open the after.rules file for editing: `bash sudo vim /etc/ufw/after.rules # If you also use IPv6, repeat for: sudo vim /etc/ufw/after6.rules `

    1. Add Docker-specific iptables Rules:
    2. Locate the filter section. Add the following rules before the COMMIT line, ideally at the very end of the filter section. These rules explicitly allow FORWARD traffic that Docker needs.
        # BEGIN UFW AND DOCKER INTEGRATION
    *filter
    :ufw-user-forward - [0:0] # Ensure this chain exists
    # Allow Docker's internal network to reach the host and the outside world
    -A FORWARD -i docker0 -j ACCEPT
    # Allow traffic from anywhere to Docker's internal network (docker0)
    # This ensures external requests reaching docker-proxy are forwarded
    -A FORWARD -o docker0 -j ACCEPT
    # Allow established/related connections through the firewall
    -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
    # Jump to the DOCKER-USER chain, allowing Docker to insert its own allow rules
    -A FORWARD -j DOCKER-USER
    COMMIT
    # END UFW AND DOCKER INTEGRATION
    ```
    > [!IMPORTANT]
    1. Adjust DEFAULTFORWARDPOLICY (Optional, but often necessary if issues persist):
    2. If the above changes don't fully resolve the issue, UFW's DEFAULTFORWARDPOLICY might still be too restrictive. While the rules in after.rules should ideally precede the default policy, iptables rule processing can be complex.

    Open /etc/default/ufw: `bash sudo vim /etc/default/ufw ` Change the line: `diff – DEFAULTFORWARDPOLICY="DROP" + DEFAULTFORWARDPOLICY="ACCEPT" ` > [!WARNING] > Setting DEFAULTFORWARDPOLICY="ACCEPT" significantly lowers your host's default forwarding security. It means all traffic being routed through your server is allowed by default. This is generally discouraged for maximum security. Only use this if the more granular rules above do not work, and ensure you understand the security implications. If you set this to ACCEPT, the rules in after.rules are even more critical to manage what is forwarded, but the default "drop everything" fallback is removed.

    1. Reload UFW and Restart Docker:
    2. For the changes to take effect, UFW needs to be reloaded, and then Docker should be restarted to ensure it rebuilds its iptables rules with the new UFW context.
        sudo systemctl restart ufw
    sudo systemctl restart docker
    1. Test Connectivity:
    2. Attempt to access your Docker container's exposed port from an external machine again.
        curl -v http://<YOUR_SERVER_IP>:<CONTAINER_PORT>
    ```

    #### 3. Method B: Open Specific Ports in UFW (Simple but Manual)

    If you only have a few Docker containers with static port exposures, you can simply open the required ports in UFW for the host. This is less dynamic but straightforward.

    # Allow TCP traffic on port 8080 (example)

    Allow TCP traffic on port 443 (example) sudo ufw allow 443/tcp

    Reload UFW to apply changes sudo ufw reload

    Restart Docker (optional, but good practice after firewall changes) sudo systemctl restart docker ` > [!NOTE] > This method primarily opens ports in the INPUT chain, allowing traffic to the host. While docker-proxy listens on the host and Docker's NAT rules handle the redirect, the FORWARD chain interaction is still critical. If this method doesn't work consistently, Method A is likely required as it directly addresses the FORWARD chain conflict.

    #### 4. Method C: Switch to firewalld (Native for CentOS/Rocky Linux)

    For CentOS Stream and Rocky Linux, firewalld is the default and recommended firewall management tool. It often integrates more smoothly with Docker without requiring manual iptables rule manipulation. If you're encountering persistent issues with UFW, consider migrating.

    1. Disable UFW:
    2. `bash
    3. sudo ufw disable
    4. sudo systemctl stop ufw
    5. sudo systemctl disable ufw
    6. `
    1. Enable and Start firewalld:
    2. `bash
    3. sudo systemctl start firewalld
    4. sudo systemctl enable firewalld
    5. sudo firewall-cmd –state
    6. # Expected output: running
    7. `
    1. Add Docker to firewalld (or allow specific ports):
    2. Docker usually creates its own firewalld zone (docker) and manages its rules when firewalld is active during Docker service start. If not, you might need to add rules manually.
        # Option 1: Reload firewalld and restart Docker (often sufficient)
    # Docker automatically adds rules to firewalld if firewalld is active upon Docker start.
    sudo systemctl reload firewalld

    Verify if Docker has added its rules sudo firewall-cmd –get-active-zones # Look for 'docker0' or similar interfaces under a zone like 'docker'

    Option 2: Manually add specific ports to the public zone (example for 8080/tcp) sudo firewall-cmd –zone=public –add-port=8080/tcp –permanent sudo firewall-cmd –reload ` > [!IMPORTANT] > After switching to firewalld, ensure that any previous iptables rules manually added by UFW (if not fully purged) do not conflict. A clean reboot after disabling UFW and enabling firewalld can sometimes help to clear old iptables states.

    Choose the method that best fits your security posture and management style. Method A provides a good balance of security and Docker integration for users who prefer UFW on CentOS/Rocky systems, while Method C offers the native and often more seamless solution.