Staging ButItWorkedLocal

Tag: ubuntu

Troubleshooting Kubernetes CrashLoopBackOff: Container Startup Failures on Ubuntu 20.04 LTS
Resolve Kubernetes CrashLoopBackOff errors on Ubuntu 20.04 LTS. This guide details common causes and provides expert step-by-step fixes for container startup failures.

When deploying applications on Kubernetes, encountering a CrashLoopBackOff status indicates a fundamental issue preventing your container from starting and remaining healthy. This status means Kubernetes is repeatedly trying to start the container, it's crashing immediately, and then Kubernetes waits for an exponentially increasing back-off duration before the next retry. This guide provides a highly technical, step-by-step approach to diagnose and resolve CrashLoopBackOff errors on an Ubuntu 20.04 LTS-based Kubernetes cluster.

Symptom & Error Signature

The primary symptom of a CrashLoopBackOff is a pod that never reaches a Running state, instead cycling through ContainerCreating, Crashing, and CrashLoopBackOff.

You will typically observe this when checking your pods:
```
kubectl get pods
```
```
NAME                             READY   STATUS             RESTARTS   AGE
my-app-deployment-78f9f7584f-abcd1   0/1     CrashLoopBackOff   5          2m3s
another-pod-xyz-12345            1/1     Running            0          10m
```
Further inspection using kubectl describe pod reveals the container's last state and relevant events:
```
kubectl describe pod my-app-deployment-78f9f7584f-abcd1
```
```
...
Containers:
  my-app:
    Container ID:  containerd://a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0
    Image:         my-registry/my-app:v1.0.0
    Image ID:      my-registry/my-app@sha256:fedcba9876543210
    Port:          8080/TCP
    Host Port:     0/TCP
    State:         Waiting
      Reason:      CrashLoopBackOff
    Last State:    Terminated
      Reason:      Error
      Exit Code:   1
      Started At:  Thu, 18 Jul 2024 10:30:15 -0400
      Finished At: Thu, 18 Jul 2024 10:30:16 -0400
    Ready:         False
    Restart Count: 5
    Environment:
      DB_HOST: mysql-service
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-abcde (ro)
...
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Pulled     2m4s (x6 over 3m)   kubelet            Container image "my-registry/my-app:v1.0.0" already present on host
  Normal   Created    2m4s (x6 over 3m)   kubelet            Created container my-app
  Normal   Started    2m3s (x6 over 3m)   kubelet            Started container my-app
  Warning  BackOff    14s (x7 over 2m4s)  kubelet            Back-off restarting failed container
```
The most crucial information here is the Exit Code (often non-zero, indicating an error) and the Events section which shows the BackOff reason.

Root Cause Analysis

A CrashLoopBackOff signifies that the container's main process exited prematurely. The underlying causes are diverse but generally fall into these categories:
1. Application Errors: The application code itself has a bug, an unhandled exception, or fails to initialize correctly (e.g., cannot connect to a database, missing configuration).
2. Incorrect command or args: The command or args specified in the Pod definition do not correctly execute the application's entrypoint or pass incorrect parameters.
3. Missing Dependencies/Files: The application expects certain files, libraries, or environment variables that are not present in the container image or mounted volumes.
4. Resource Constraints (OOMKilled): The container tries to consume more memory than allocated by resources.limits.memory, leading to the Linux OOM (Out Of Memory) killer terminating the process.
5. Permissions Issues: The application lacks the necessary file system or network permissions to operate (e.g., trying to write to a read-only volume, binding to a privileged port without appropriate capabilities).
6. Misconfigured Liveness/Readiness Probes: Probes are configured too aggressively, failing immediately upon startup, causing Kubernetes to restart a perfectly healthy container.
7. Volume Mount Problems: Persistent Volume Claims (PVCs) are not bound, the subPath is incorrect, or the underlying storage is inaccessible or has permission issues.
8. Bad Container Image: The container image itself is corrupted, incompatible with the underlying kernel, or fundamentally broken.
9. Network Configuration Issues: While less common for immediate startup crashes, if the application critically depends on an immediate network connection to an external service or a service mesh sidecar fails, it can cause an early exit.
Step-by-Step Resolution

Follow these steps to systematically diagnose and resolve the CrashLoopBackOff error.

1. Inspect Pod Status and Container Logs

The logs are your primary source of truth. They contain the direct error message from your application.
```
# Get a detailed overview of the failing pod
```
Check the current logs of the container kubectl logs <your-pod-name> -c <your-container-name>

If the container has already restarted, check logs from the previous instance kubectl logs <your-pod-name> -c <your-container-name> –previous `

[!IMPORTANT] The kubectl logs --previous command is invaluable. Since the container keeps crashing and restarting, the "current" logs might be empty or just show the initial startup. The --previous flag retrieves logs from the last terminated instance, which often contains the actual crash reason.

Analyze the log output for stack traces, error messages (e.g., "Connection refused," "File not found," "Segmentation fault," "OutOfMemoryError"), or any explicit reasons for termination.

2. Verify Pod Definition (YAML)

Retrieve the YAML definition of your failing pod to scrutinize its configuration.
```
kubectl get pod <your-pod-name> -o yaml > pod-debug.yaml
less pod-debug.yaml
```
Pay close attention to these sections in the pod-debug.yaml:
- image: Is the correct image and tag specified?
- command and args: Do these correctly invoke your application's entrypoint? Are there any typos or incorrect parameters?
- env: Are all necessary environment variables passed, and are their values correct?
- volumeMounts and volumes: Are all required volumes mounted correctly, and are the mountPath and subPath configurations accurate?
- resources.limits: Is there a memory limit that might be too low, leading to OOMKilled?
- livenessProbe and readinessProbe: Are these configured correctly, or could they be too aggressive, causing premature restarts?
3. Test Container Image Locally

This step helps isolate whether the issue is with your application/image or the Kubernetes environment itself.

First, identify the image used by the failing pod:
```
kubectl get pod <your-pod-name> -o jsonpath='{.spec.containers[0].image}'
```
Then, pull and run the image locally on your Ubuntu machine (assuming Docker is installed):
```
docker pull <your-image-name>:<tag>
docker run --rm -it --name debug-container <your-image-name>:<tag>
```
If your application requires specific environment variables or mounted volumes, replicate them in the docker run command:
```
docker run --rm -it 
  -e DB_HOST=localhost 
  -v /path/on/host:/path/in/container 
  --name debug-container <your-image-name>:<tag>
```
Observe the output. If it crashes locally, the problem lies within your application or container image configuration. Debug it as you would a standalone application. If it runs successfully locally, the issue is likely Kubernetes-specific (e.g., incorrect command/args in K8s manifest, K8s volume/secret issues, resource constraints, network policies).

4. Review Resource Limits (Especially Memory)

If kubectl describe pod or the container logs indicate OOMKilled (Out Of Memory Killed) as the reason for termination, your container is exceeding its allocated memory.
```
# Example from kubectl describe pod output
State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137
```
To address this:
1. Adjust resources.limits.memory: Increase the memory limit in your Deployment/Pod manifest.
```
    # my-app-deployment.yaml
    spec:
      containers:
      - name: my-app
        image: my-registry/my-app:v1.0.0
        resources:
          limits:
            memory: "512Mi" # Increase from e.g., 256Mi
            cpu: "500m"
          requests:
            memory: "256Mi"
            cpu: "250m"
```
```
    kubectl apply -f my-app-deployment.yaml
```
[!WARNING] Blindly increasing resource limits can mask underlying memory leaks in your application and lead to inefficient cluster resource utilization. It's best to profile your application's memory usage to determine appropriate limits.

5. Check Liveness and Readiness Probes

Misconfigured probes can cause a healthy container to be restarted.

If your pod definition includes livenessProbe or readinessProbe, try commenting them out or simplifying them temporarily to see if the container starts successfully.
```
# my-app-deployment.yaml
spec:
  containers:
  - name: my-app
    image: my-registry/my-app:v1.0.0
    # ... other config ...
    # livenessProbe: # Temporarily comment out or remove
    #   httpGet:
    #     path: /healthz
    #     port: 8080
    #   initialDelaySeconds: 10
    #   periodSeconds: 5
    #   failureThreshold: 3
    # readinessProbe: # Temporarily comment out or remove
    #   httpGet:
    #     path: /ready
    #     port: 8080
    #   initialDelaySeconds: 5
    #   periodSeconds: 3
    #   failureThreshold: 1
```
If removing them allows the pod to start, the probes were the issue. Reintroduce them with more lenient initialDelaySeconds, periodSeconds, and failureThreshold values, and ensure the probe endpoints are actually implemented and reachable within the container.

6. Examine Volume Mounts and Permissions

Incorrect volume mounts or permissions within the container are common causes.
1. Verify PVC Status:
2. `bash
3. kubectl get pvc
4. `
5. Ensure your PersistentVolumeClaims are in a Bound state. If not, the PV/PVC provisioner is failing.
1. Check subPath: If you're mounting a specific subdirectory of a volume using subPath, ensure the path exists within the volume.
1. Inspect Permissions Inside a Running Pod (if possible):
2. If you have another similar pod running successfully, or if you can temporarily make the failing pod run by fixing a different issue, exec into it to check paths and permissions.
```
    kubectl exec -it <running-pod-name> -- /bin/bash # or /bin/sh
    ls -la /path/to/mounted/volume
    whoami
```
If your application expects to write to a volume mounted as read-only, or if the user running the application inside the container doesn't have permissions, it will fail. You may need to specify securityContext in your Pod definition to run as a specific user or group.
```
    # my-app-deployment.yaml
    spec:
      containers:
      - name: my-app
        image: my-registry/my-app:v1.0.0
        securityContext:
          runAsUser: 1000 # Example: run as user ID 1000
          runAsGroup: 3000 # Example: run as group ID 3000
          allowPrivilegeEscalation: false
```
7. Debugging with an Init Container or Temporary Sidecar

For complex startup issues, an initContainer can help diagnose problems before your main application starts.
```
# my-app-deployment.yaml
spec:
  initContainers:
  - name: debug-init
    image: busybox:latest # A lightweight image with basic tools
    command: ["sh", "-c", "echo 'Checking network connectivity...' && ping -c 3 mysql-service && echo 'Checking file permissions...' && ls -la /app/config && sleep 10"]
    # Mount volumes if needed for checks
    volumeMounts:
    - name: app-config-volume
      mountPath: /app/config
  containers:
  - name: my-app
    image: my-registry/my-app:v1.0.0
    # ... main app config ...
```
The initContainer will run to completion. If it fails, kubectl describe pod will show its error. If it succeeds, the main container will start. You can also use a temporary sidecar container with debugging tools (e.g., net-tools, strace) that shares the main container's namespace.

8. Rebuild and Retag Image

If you suspect issues with the container image itself, try:
1. Clean Cache: Clear your Docker build cache (docker builder prune) if you're building locally.
2. Verify Base Image: Ensure the base image (e.g., ubuntu:20.04, node:16-alpine) is stable and compatible.
3. Rebuild: Rebuild your application image completely and push it with a new tag.
4. Update Deployment: Update your Kubernetes Deployment to use the new image tag.
```
    kubectl set image deployment/<your-deployment-name> <your-container-name>=<new-image-name>:<new-tag>
```
By systematically applying these debugging techniques, you can pinpoint the root cause of your CrashLoopBackOff error and restore stability to your Kubernetes applications.
July 19, 2026
Fixing Apache ‘client denied by server configuration’ 403 Forbidden on Windows WSL2 Ubuntu
Troubleshoot and resolve Apache 403 Forbidden errors ('client denied by server configuration') on your WSL2 Ubuntu setup, caused by misconfigured access controls or permissions.

When developing or hosting web applications on a Windows machine using the Windows Subsystem for Linux 2 (WSL2) with Ubuntu and Apache2, encountering a 403 Forbidden error is a common hurdle. While a 403 status generally indicates that the server understands the request but refuses to authorize it, the specific Apache error message "client denied by server configuration" points directly to an access control issue within Apache's configuration files rather than file system permissions, though the latter can also result in a 403. This guide will help you systematically diagnose and resolve this frustrating error.

Symptom & Error Signature

When you attempt to access your web application or a specific directory in your browser, you'll be greeted with:
You don't have permission to access this resource. `

Or, more generically:
```
403 Forbidden
```
In your Apache error.log (typically located at /var/log/apache2/error.log), you will see entries similar to these:
```
[Sat Jul 18 10:30:00.123456 2026] [core:error] [pid 1234] [client 127.0.0.1:54321] AH00035: access to /var/www/html/index.html denied (filesystem path '/var/www/html/index.html') because search permissions are missing on a component of the path
[Sat Jul 18 10:30:00.123456 2026] [core:error] [pid 1234] [client 127.0.0.1:54321] AH01797: client denied by server configuration: /var/www/html/
[Sat Jul 18 10:30:00.123456 2026] [authz_core:error] [pid 1234] [client 127.0.0.1:54321] AH01630: client denied by server configuration: /var/www/html/
```
The key phrase to look for is client denied by server configuration.

Root Cause Analysis

The "client denied by server configuration" error specifically points to Apache's access control directives. While general file system permissions can also cause a 403, this particular error message isolates the problem to how Apache itself is configured to grant or deny access based on the requesting client or the requested resource's path.

Common root causes include:
1. Strict Apache Directory Configuration:
2. * Require all denied: This is the default in many Apache installations for the root directory (/), meaning you must explicitly grant access to your document root.
3. * Order Deny,Allow (Legacy): Older Apache 2.2 style configurations using Deny from all without a corresponding Allow from statement.
4. * IP-based Restrictions: Your configuration might be set to only allow access from specific IP addresses (e.g., Require ip 192.168.1.0/24) and your client's IP doesn't match.
1. Incorrect Options Directive: The Options directive within a <Directory> block controls what features are available for that directory.
2. * Options -Indexes: If you try to access a directory without an index file and Indexes is disabled, Apache will not list the directory contents and may issue a 403 if it can't find an index file and no other DirectoryIndex is configured.
3. * Missing FollowSymLinks: If your DocumentRoot or other accessed paths involve symbolic links, and FollowSymLinks is not enabled, Apache will deny access.
1. Mismatched DirectoryIndex: If your DirectoryIndex directive specifies index.html but your actual entry file is index.php (or vice-versa), and directory listing is disabled, Apache will return a 403.
1. WSL2-Specific File System Behavior (Secondary): While client denied by server configuration usually isn't directly a file permission issue, how WSL2 handles permissions for files mounted from Windows (/mnt/c/) can sometimes lead to an inability for Apache to read necessary files, which can also manifest as a 403. This is less common for the exact error message but worth considering.
Step-by-Step Resolution

Let's systematically troubleshoot and fix the Apache 403 Forbidden error on your WSL2 Ubuntu instance.

1. Verify Apache Service Status and Basic Connectivity

Ensure Apache is running and listening on the expected port.
```
sudo systemctl status apache2
```
You should see output indicating active (running). If not, start it:
```
sudo systemctl start apache2
```
Now, try to access localhost from within your WSL2 terminal to confirm Apache is serving requests locally:
```
curl http://localhost
```
If this returns HTML (e.g., the default Ubuntu Apache page), then Apache is basically functional. If it hangs or shows a connection refused, you have a more fundamental network/service issue, not a 403.

2. Analyze Apache Error Logs

The error.log is your best friend for diagnosing Apache issues. Keep an eye on it while you attempt to reproduce the error.
```
sudo tail -f /var/log/apache2/error.log
```
Access your site in the browser (e.g., http://localhost/yourproject or http://<WSL2IP>/your_project) and observe the logs. Look for the AH01797 or AH01630 errors specifically, as these confirm the "client denied by server configuration" problem.

3. Review Apache Virtual Host and Directory Configuration

This is the most critical step for resolving "client denied by server configuration". You need to examine the Apache configuration files that apply to your document root.
1. Locate your Virtual Host Configuration:
2. Your primary virtual host is likely in /etc/apache2/sites-available/000-default.conf or a custom .conf file you created (e.g., /etc/apache2/sites-available/your_site.conf).
3. If you've made changes, ensure the site is enabled:
4. `bash
5. sudo a2ensite your_site.conf
6. sudo systemctl reload apache2
7. `
1. Examine the DocumentRoot and <Directory> Directives:
2. Open the relevant virtual host file with a text editor (e.g., nano or vim):
3. `bash
4. sudo nano /etc/apache2/sites-available/000-default.conf
5. `
6. (Replace 000-default.conf with your actual virtual host file if different).
Look for the DocumentRoot directive, which specifies where your website files are located. Then, find the <Directory> block that corresponds to your DocumentRoot or the path you're trying to access.

Example (Incorrect Configuration):
```
    <VirtualHost *:80>
        ServerAdmin webmaster@localhost
```
This block might be missing or explicitly deny <Directory /var/www/html> # Incorrect or overly restrictive: Require all denied # Or legacy style: # Order Deny,Allow # Deny from all </Directory>

ErrorLog ${APACHELOGDIR}/error.log CustomLog ${APACHELOGDIR}/access.log combined </VirtualHost> `

Example (Corrected Configuration for Local Development):

You need to explicitly grant access within the <Directory> block. For local development on WSL2, granting access to all is generally acceptable.
```
    <VirtualHost *:80>
        ServerAdmin webmaster@localhost
```
<Directory /var/www/html> Options Indexes FollowSymLinks MultiViews AllowOverride All Require all granted # <– This is the key fix for Apache 2.4+ # For Apache 2.2 style (less common in modern Ubuntu): # Order Allow,Deny # Allow from all </Directory>

ErrorLog ${APACHELOGDIR}/error.log CustomLog ${APACHELOGDIR}/access.log combined </VirtualHost> `

[!IMPORTANT] > * Require all granted is the modern Apache 2.4+ directive for allowing access to a directory. > * Options Indexes FollowSymLinks MultiViews: > * Indexes: Allows directory listing if no DirectoryIndex is found. Useful for browsing during development. > * FollowSymLinks: Allows Apache to follow symbolic links. Crucial if your document root or subdirectories are symlinks. > * MultiViews: Content negotiation for multiple views. > * AllowOverride All: Allows .htaccess files to override configurations for this directory. Useful for frameworks and local .htaccess rules.

If your files are on the Windows filesystem (e.g., /mnt/c/Users/youruser/Documents/website):

The configuration would look similar, but the DocumentRoot and <Directory> paths would reference the mounted Windows drive.
```
    <VirtualHost *:80>
        ServerAdmin webmaster@localhost
```
<Directory /mnt/c/Users/youruser/Documents/website> Options Indexes FollowSymLinks MultiViews AllowOverride All Require all granted </Directory>

ErrorLog ${APACHELOGDIR}/error.log CustomLog ${APACHELOGDIR}/access.log combined </VirtualHost> `

[!WARNING] > While hosting files from /mnt/c/ in WSL2 for development is convenient, it can introduce performance overhead and complex permission issues, as standard Linux chown/chmod commands do not directly apply to the underlying Windows filesystem in the same way. For production-like environments or more robust setups, it's highly recommended to store your web files directly within the WSL2 Linux filesystem (e.g., /var/www/html or ~/projects).

4. Check File and Directory Permissions (Secondary, but related)

Even though client denied by server configuration specifically points to Apache access rules, incorrect file permissions can also lead to a 403. Apache (running as the www-data user on Ubuntu) needs read access to your files and execute (search) access to directories leading to your files.
1. Identify the Apache User:
2. On Ubuntu, Apache typically runs as the www-data user and group.
1. Check Permissions for your DocumentRoot:
2. Navigate to your DocumentRoot (e.g., /var/www/html or /mnt/c/Users/youruser/Documents/website) and check permissions.
```
    ls -ld /var/www/html
    ls -l /var/www/html/index.html
```
Expected output for directories should have at least rwx for the owner and rx for group/others (e.g., drwxr-xr-x). Files should have at least r for the owner and r for group/others (e.g., -rw-r--r--).
1. Adjust Permissions (if necessary, for files within WSL2 filesystem):
2. If your files are within the WSL2 Linux filesystem (e.g., /var/www/html), you can adjust permissions:
```
    sudo chown -R www-data:www-data /var/www/html
    sudo find /var/www/html -type d -exec chmod 755 {} ; # Directories read/write/execute for owner, read/execute for others
    sudo find /var/www/html -type f -exec chmod 644 {} ; # Files read/write for owner, read for others
```
[!NOTE] > If your files are on the Windows filesystem (/mnt/c/...), chown does not behave as expected for the underlying Windows files. The permissions shown by ls -l are often a result of how WSL2 mounts the Windows drive, and they might appear as root-owned with broad permissions by default. In such cases, the Apache Directory configuration (Step 3) is paramount. If you must manage permissions for Windows-mounted files, consider adding uid and gid options to your /etc/fstab for the mounted drive, or setting specific umask options during mount. However, for most users, resolving the Apache Directory directives is the primary solution.

5. Ensure DirectoryIndex is Present

If you're accessing a directory (e.g., http://localhost/my_project/) without explicitly specifying a file (like index.html), Apache looks for a DirectoryIndex file. If it doesn't find one and Options Indexes is not enabled, it will return a 403.
1. Verify DirectoryIndex:
2. In your virtual host file (.conf) or apache2.conf, ensure DirectoryIndex is defined and includes your entry file (e.g., index.html, index.php).
```
    DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm
    ```
```
6. Test Configuration and Restart Apache

After making any changes to Apache configuration files, always test the syntax before restarting to avoid service disruption.
```
sudo apache2ctl configtest
```
You should see Syntax OK. If there are errors, Apache will tell you where they are. Fix them before proceeding.

Once Syntax OK, restart Apache:
```
sudo systemctl restart apache2
```
Now, try accessing your site in the browser again.

7. Firewall Considerations (Briefly)

While usually not the cause of "client denied by server configuration", ensure no firewalls are blocking access. * WSL2 UFW (Uncomplicated Firewall): If you've enabled ufw within your WSL2 instance, ensure port 80 (and 443 for HTTPS) are open. `bash sudo ufw status sudo ufw allow 80/tcp sudo ufw reload ` Windows Firewall: For accessing your WSL2 Apache server from your Windows host browser, the Windows Firewall typically doesn't block localhost connections to WSL2. However, if you're trying to access it from another machine on your network*, you might need to create an inbound rule in Windows Firewall for the WSL2 IP address.

By following these steps, you should be able to identify and resolve the "client denied by server configuration" 403 Forbidden error on your Apache WSL2 Ubuntu setup. The vast majority of the time, adjusting the <Directory> directives to include Require all granted will solve the problem.
July 18, 2026
Troubleshooting Caddy Reverse Proxy TLS Handshake Failed Verification on Ubuntu 20.04 LTS
Resolve Caddy reverse proxy TLS handshake failures with upstream servers on Ubuntu 20.04 LTS. Diagnose certificate validation, SNI, and CA issues for seamless reverse proxy operations.

Introduction

Caddy has established itself as a powerful, easy-to-use web server and reverse proxy, renowned for its automatic HTTPS capabilities. However, when operating Caddy as a reverse proxy, you might encounter situations where it fails to establish a secure (TLS) connection with its upstream backend server. A "TLS handshake failed verification" error signifies that Caddy initiated a TLS connection but was unable to validate the certificate presented by the upstream server. This guide provides a highly technical, step-by-step approach to diagnose and resolve this issue specifically on Ubuntu 20.04 LTS environments.

### Symptom & Error Signature

When Caddy experiences a TLS handshake failure with an upstream, client requests attempting to reach the proxied service will typically receive a 502 Bad Gateway error. In the Caddy server logs, you'll find entries indicating certificate validation problems.

To view Caddy's logs, use journalctl:
```
sudo journalctl -u caddy -f --no-hostname
```
Typical error signatures you might observe include:
```
{
  "level": "error",
  "ts": 1678886400.000,
  "logger": "http.log.error",
  "msg": "x509: certificate signed by unknown authority",
  "trace": "...",
  "request": {
    "method": "GET",
    "uri": "/api/data",
    "proto": "HTTP/2.0",
    "remote_addr": "192.168.1.100:54321",
    "host": "your-caddy-domain.com",
    "headers": {
      "User-Agent": ["Mozilla/5.0 ..."],
      "Accept": ["*/*"]
    }
  },
  "error": "reverse proxy: selected upstream is unhealthy: connection to upstream timed out after 10s"
}
```
Or a more generic handshake failure:
```
{
  "level": "error",
  "ts": 1678886405.000,
  "logger": "http.handlers.reverse_proxy",
  "msg": "aborting with incomplete response",
  "error": "remote error: tls: handshake failure",
  "duration": 0.0001
}
```
Or an SNI related error:
```
{
  "level": "error",
  "ts": 1678886410.000,
  "logger": "http.log.error",
  "msg": "tls: failed to verify certificate: x509: certificate is valid for example.com, not your-upstream-hostname.internal",
  "trace": "...",
  "error": "reverse proxy: selected upstream is unhealthy: connection to upstream timed out after 10s"
}
```
### Root Cause Analysis

The "TLS handshake failed verification" error fundamentally means that Caddy, acting as a TLS client to the upstream server, could not trust the certificate presented by that server. The most common underlying reasons for this include:
1. Untrusted Certificate Authority (CA): The upstream server's certificate is signed by a CA that is not present or trusted in Caddy's (or the underlying system's) trust store. This is common with self-signed certificates, private enterprise CAs, or newly issued CA roots not yet widely distributed.
2. Certificate Name Mismatch (SNI/SAN):
3. * Common Name (CN) / Subject Alternative Name (SAN) Mismatch: The hostname Caddy is attempting to connect to (or the Host header it's sending) does not match the CN or SANs listed within the upstream server's certificate.
4. * Incorrect Server Name Indication (SNI): Caddy might not be sending the correct SNI hostname during the TLS handshake, causing the upstream server (especially if it hosts multiple virtual hosts with different certificates) to present the wrong certificate or refuse the connection.
5. Expired or Invalid Certificate: The upstream server's certificate has expired, is not yet valid, or has been revoked.
6. Network/Firewall Issues: While less direct for "verification failure," network problems or restrictive firewalls between Caddy and the upstream could interrupt the TLS handshake process, leading to a perceived failure.
7. System Time Skew: Significant clock drift on either the Caddy server or the upstream server can cause certificate validity checks (e.g., notBefore, notAfter dates) to fail.
### Step-by-Step Resolution

Follow these steps to diagnose and resolve the TLS handshake verification error.

1. Verify Upstream Server's Certificate Directly

Before troubleshooting Caddy, confirm the upstream server's certificate is valid and correctly configured from the perspective of the Caddy server's host.
```
# Replace 'your_upstream_hostname' with the actual hostname Caddy connects to
# Replace 'upstream_ip_address' with the upstream server's IP
# Replace 'upstream_port' with the port the upstream listens on (e.g., 443, 8443)
openssl s_client -connect your_upstream_ip_address:upstream_port -servername your_upstream_hostname </dev/null
```
Inspect the output carefully: * Verify return code: 0 (ok): Indicates the certificate chain is trusted by the system's CA store. If you see anything else (e.g., 20 (unable to get local issuer certificate), 21 (unable to verify the first certificate)), it points to an untrusted CA. * Not Before / Not After: Check if the certificate is within its valid date range. * Subject Common Name (CN) / Subject Alternative Name (SAN): Ensure yourupstreamhostname is listed here. If not, you have a hostname mismatch. * depth and issuer: Review the certificate chain to identify the issuing CA.

2. Update System CA Trust Store (for Untrusted CAs)

If openssl s_client returned an error indicating an untrusted issuer (e.g., Verify return code: 20), you need to add the upstream's CA certificate to Caddy's system trust store. This is common for self-signed certificates or private enterprise CAs.
1. Obtain the CA Certificate:
2. If you have the upstream CA's .crt file, copy it. If not, you can often extract it using openssl:
```
    # Connect to the upstream and extract the full certificate chain
```
Open 'upstream_chain.pem' and identify the root CA certificate. # It's usually the last certificate in the chain and self-signed (Issuer and Subject are the same). # Copy its content (including —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–) # into a new file, e.g., 'upstream_ca.crt'. `
1. Add the CA Certificate to System Trust:
```
    sudo cp /path/to/upstream_ca.crt /usr/local/share/ca-certificates/
    sudo update-ca-certificates
```
You should see output similar to: Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
1. Restart Caddy:
```
    sudo systemctl restart caddy
    sudo journalctl -u caddy -f --no-hostname # Check logs for success
```
[!IMPORTANT] Ensure the CA certificate has a .crt extension when copied to /usr/local/share/ca-certificates/. The update-ca-certificates utility only processes files with this extension in that directory by default.

3. Configure Caddy for Specific Upstream TLS Trust (Caddyfile Directive)

For more granular control, especially if you only want Caddy to trust a specific custom CA for a particular upstream and not globally, you can configure this directly in your Caddyfile.
1. Place CA Certificate: Store the upstream CA certificate file (e.g., upstream_ca.crt) in a secure, Caddy-readable location, typically /etc/caddy/certs/.
1. Modify Caddyfile:
2. Add or modify the transport block within your reverse_proxy directive:
```
    your-caddy-domain.com {
        reverse_proxy your_upstream_ip_address:upstream_port {
            transport http {
                # Trusts the specified CA certificate for this upstream only
                tls_trusted_ca /etc/caddy/certs/upstream_ca.crt
                # Optional: specify the server name if different from the Caddy domain
                # This ensures the correct certificate is requested via SNI
                tls_server_name your_upstream_hostname
            }
        }
    }
```
1. Validate and Apply Caddyfile Changes:
```
    sudo caddy validate --config /etc/caddy/Caddyfile
    sudo systemctl reload caddy
    sudo journalctl -u caddy -f --no-hostname # Monitor logs
```
4. Address Hostname/SNI Mismatch

If openssl s_client revealed a hostname mismatch (CN/SAN mismatch) or if your upstream requires a specific SNI hostname to present the correct certificate:
1. Caddyfile reverse_proxy Configuration:
2. Ensure Caddy is sending the correct Host header and tlsservername (SNI) for the upstream.
```
    your-caddy-domain.com {
        reverse_proxy your_upstream_ip_address:upstream_port {
            # This sets the Host header sent to the upstream
            # Crucial if upstream expects a specific hostname
```
transport http { # This sets the SNI hostname for the TLS handshake # Required if the upstream server serves multiple certificates tlsservername yourupstreamhostname } } } ` Replace yourupstreamhostname with the actual hostname that the upstream's certificate is valid for. This is often the internal FQDN of the backend server.
1. Validate and Apply Caddyfile Changes:
```
    sudo caddy validate --config /etc/caddy/Caddyfile
    sudo systemctl reload caddy
    sudo journalctl -u caddy -f --no-hostname
```
5. Verify System Time Synchronization

Significant time differences can cause certificates to appear invalid. Ensure both Caddy and the upstream server are synchronized with an NTP server.
```
timedatectl
```
If time is unsynchronized, ensure systemd-timesyncd is enabled or install ntp:
```
sudo systemctl enable --now systemd-timesyncd
# Or for legacy NTP daemon:
# sudo apt update && sudo apt install ntp
# sudo systemctl enable --now ntp
```
6. Bypass TLS Verification (Development/Testing ONLY)

[!WARNING] This step disables all TLS certificate verification. It should NEVER be used in production environments as it makes your system vulnerable to Man-in-the-Middle attacks. Use this only for debugging in isolated development environments or if you fully understand and accept the security implications.

If all else fails and you need to quickly confirm if TLS verification is the root cause, you can temporarily disable it:
```
your-caddy-domain.com {
    reverse_proxy your_upstream_ip_address:upstream_port {
        transport http {
            tls_insecure_skip_verify
            # Optionally, still specify server name for SNI
            # tls_server_name your_upstream_hostname
        }
    }
}
```
Validate and apply, then immediately remove this directive once you've confirmed the issue.

7. Check Upstream Server's Certificate Expiration/Validity

If openssl s_client revealed an expired or not-yet-valid certificate on the upstream, the solution lies with the upstream server's administrator. They need to renew or correctly install a valid certificate. Caddy cannot fix an invalid certificate presented by its backend.

This comprehensive guide should help you systematically troubleshoot and resolve the "Caddy reverse proxy TLS handshake failed verification" error on your Ubuntu 20.04 LTS system. Remember to always apply the least permissive fix possible for security.
July 16, 2026
Caddy Reverse Proxy: Resolving TLS Handshake Failed Verification (x509) on Windows WSL2 Ubuntu
Troubleshoot 'TLS handshake failed verification' when Caddy in WSL2 acts as a reverse proxy. Learn to manage certificate trust for backend services.

Caddy Reverse Proxy: Resolving TLS Handshake Failed Verification (x509) on Windows WSL2 Ubuntu

When operating Caddy as a reverse proxy within a Windows Subsystem for Linux 2 (WSL2) Ubuntu environment, you might encounter errors related to TLS handshake verification. This typically happens when Caddy attempts to establish a secure connection to your backend service (e.g., a web application, API, or another internal server) but fails to validate the backend's TLS certificate. The result is an inability for Caddy to proxy requests, leading to unresponsive services or error pages for your users.

This guide will walk you through diagnosing and resolving the common causes of "TLS handshake failed verification" errors, particularly focusing on certificate trust issues within the WSL2 environment.

Symptom & Error Signature

Users attempting to access your service through Caddy will likely see generic browser errors such as NET::ERRCERTINVALID, ERRSSLPROTOCOL_ERROR, or 502 Bad Gateway if Caddy cannot successfully connect to the backend.

The most telling sign will be in your Caddy logs. You'll observe entries similar to these, indicating Caddy's struggle to trust the backend certificate:
```
ERROR   http.log.error  reverse proxy: selected backend is unhealthy: TLS handshake failed: remote error: tls: handshake failure
ERROR   http.log.error  reverse proxy: x509: certificate signed by unknown authority
ERROR   http.log.error  reverse proxy: tls: failed to verify certificate: x509: certificate signed by unknown authority
ERROR   http.log.error  reverse proxy: dial tcp 172.17.0.2:443: connect: connection refused
```
The specific messages x509: certificate signed by unknown authority or tls: failed to verify certificate are key indicators of a certificate trust problem.

Root Cause Analysis

The "TLS handshake failed verification" error, especially with x509: certificate signed by unknown authority, means that Caddy, acting as a client to your backend server, received a certificate but couldn't validate its authenticity using its list of trusted Certificate Authorities (CAs). This usually stems from one or more of the following:
1. Self-Signed Certificates: The backend server is using a self-signed TLS certificate. These certificates are not issued by a publicly trusted CA and therefore are inherently untrusted by default.
2. Internal CA Certificates: The backend server's certificate is issued by an internal or private Certificate Authority (CA) specific to your organization or development setup. WSL2's Ubuntu instance, by default, only trusts widely recognized public CAs.
3. Missing Intermediate Certificates: The backend server might be serving its certificate without providing the full chain of intermediate certificates, preventing Caddy from building a trusted path to a known root CA.
4. Incorrect Caddy Configuration: The reverse_proxy directive might be pointing to the wrong IP address or port, or is missing necessary TLS configuration options.
5. Network/Firewall Issues: Though less likely to cause a "TLS handshake failed verification" specifically (which implies a connection was made and a certificate received), underlying network connectivity issues or firewall blocks could result in similar high-level errors like 502 Bad Gateway and precede this specific TLS error, showing as connection refused initially.
6. Expired or Invalid Certificates: The backend's certificate might be expired, revoked, or have a hostname mismatch, leading to validation failure.
The most prevalent causes for WSL2 scenarios involving internal services are self-signed or internal CA certificates, as WSL2's Ubuntu environment needs explicit instruction to trust them.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the TLS handshake verification error.

1. Verify Caddy Configuration

First, ensure your Caddyfile's reverse_proxy directive is correctly configured to point to your backend service's address and port.
```
yourdomain.com {
    reverse_proxy https://your_backend_ip_or_hostname:port {
        # Other directives if needed
    }
}
```
Make sure you're using https:// if your backend is serving over TLS, which is the assumption for this error.

2. Inspect Caddy Logs for Detailed Error Messages

The Caddy logs are your primary diagnostic tool. If Caddy is running as a systemd service, you can retrieve logs using journalctl.
```
sudo journalctl -u caddy -f --since "5 minutes ago"
```
Look for the exact error message string. x509: certificate signed by unknown authority is a strong indicator that Caddy does not trust the CA that signed your backend's certificate.

3. Determine Your Backend Certificate's Origin

You need to know if your backend's TLS certificate is: * Issued by a publicly trusted CA (e.g., Let's Encrypt, DigiCert). * Self-signed. * Issued by an internal/private CA.

You can inspect the certificate directly from WSL2 using openssl. Replace yourbackendiporhostname:port with your actual backend's address.
```
openssl s_client -connect your_backend_ip_or_hostname:port -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep "Issuer:|Subject:|Not Before:|Not After:"
```
Pay close attention to the Issuer field. If it's your own company, a custom name, or the same as Subject, it's likely a self-signed or internal CA certificate.

4. Add the CA Certificate to WSL2's Trust Store (Recommended)

This is the most robust solution for internal or self-signed certificates. You need to obtain the root or intermediate CA certificate that signed your backend's certificate and add it to your WSL2 Ubuntu instance's system-wide trust store.

a. Obtain the CA Certificate File

You might receive this file from your internal IT department, or you can extract it if you have access to the backend server. If you only have the backend server's leaf certificate, you'll need the CA certificate that signed it.

[!IMPORTANT] The certificate file must be in PEM format (base64-encoded ASCII with -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- headers) and typically have a .crt extension.

If you don't have the .crt file directly, you can often extract it from the backend server's certificate chain. For example, if you can curl your backend (even if it fails validation, you might get the cert):
```
# This command tries to fetch the certificate chain from your backend
# and save it as backend_chain.pem. You'll then need to identify
# and extract the CA cert from this chain.
openssl s_client -showcerts -verify 5 -connect your_backend_ip_or_hostname:port < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ print $0 }' > backend_chain.pem
```
```
b. Install the CA Certificate in WSL2 Ubuntu
1. Copy your .crt file (e.g., my-internal-ca.crt) into the /usr/local/share/ca-certificates/ directory within your WSL2 Ubuntu instance.
```
    sudo cp /path/to/my-internal-ca.crt /usr/local/share/ca-certificates/
```
1. Update the system's CA certificate store.
```
    sudo update-ca-certificates
```
You should see output similar to Adding new cert into /etc/ssl/certs/ and 1 added, 0 removed; done..
1. Restart Caddy to pick up the new trust store.
```
    sudo systemctl restart caddy
```
Check Caddy logs again (sudo journalctl -u caddy -f) to ensure the error is resolved.

5. Configure Caddy with tlstrustedca_certs (Alternative/Specific)

If you prefer not to add the CA certificate system-wide, or if you need more granular control, you can specify the trusted CA certificate directly in your Caddyfile for a specific reverse_proxy block.
```
yourdomain.com {
    reverse_proxy https://your_backend_ip_or_hostname:port {
        tls_trusted_ca_certs /path/to/my-internal-ca.crt
    }
}
```
[!NOTE] Ensure the /path/to/my-internal-ca.crt is accessible by the Caddy user and specifies the path within your WSL2 filesystem.

After modifying your Caddyfile, reload Caddy:
```
sudo systemctl reload caddy
# Or if reload fails to pick up all changes:
sudo systemctl restart caddy
```
6. Temporarily Skip TLS Verification (Development/Last Resort)

[!WARNING] Using tlsinsecureskip_verify is highly discouraged in production environments. It completely disables TLS certificate validation, making your connection vulnerable to Man-in-the-Middle (MITM) attacks and compromising the security of your data. Only use this for development or debugging where security implications are fully understood and accepted.

If you're in a development environment and absolutely need to proceed without certificate validation, you can instruct Caddy to skip verification:
```
yourdomain.com {
    reverse_proxy https://your_backend_ip_or_hostname:port {
        tls_insecure_skip_verify
    }
}
```
Reload or restart Caddy after this change. Remove this directive as soon as you implement a proper certificate trust solution.

7. Verify Network Connectivity and DNS Resolution

If you're still facing issues, or if Caddy logs initially showed connection refused before TLS handshake failed, verify network connectivity from your WSL2 instance to the backend.
1. Ping the backend:
```
    ping your_backend_ip_or_hostname
```
1. Test direct HTTPS connection with curl from WSL2:
```
    curl -v https://your_backend_ip_or_hostname:port/your_health_check_path
    ```
```
1. Check DNS Resolution: If using a hostname, ensure WSL2 can resolve it. Add an entry to /etc/hosts if it's an internal hostname not resolved by your DNS server.
```
    # Example /etc/hosts entry
    192.168.1.100   your_backend_hostname
```
8. Check Backend Certificate Expiry and Hostname

Ensure the backend server's certificate is not expired and that its Subject Alternative Name (SAN) or Common Name (CN) matches the hostname or IP address Caddy is using to connect.
```
openssl s_client -connect your_backend_ip_or_hostname:port -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -E "Subject Alternative Name|Not Before:|Not After:"
```
If the certificate is expired or the hostname doesn't match, you'll need to renew or re-issue the backend's certificate.

By systematically following these steps, you should be able to diagnose and resolve the "TLS handshake failed verification" error with Caddy acting as a reverse proxy in your Windows WSL2 Ubuntu environment. Prioritize adding trusted CA certificates to the system store for a secure and robust solution.
July 16, 2026
Resolving Git ‘fatal: refusing to merge unrelated histories’ on Ubuntu 20.04 LTS
Fix the Git 'refusing to merge unrelated histories' error when pulling or merging disparate repositories on Ubuntu 20.04. Learn the root cause and resolution.

This guide addresses a common Git error encountered by developers and system administrators on Ubuntu 20.04 LTS servers: "fatal: refusing to merge unrelated histories". This error typically arises when attempting to merge or pull changes from a remote repository into a local one that Git deems to have no common ancestral commits. While seemingly a blocking issue, Git provides a clear mechanism to resolve it, ensuring your project histories can be aligned.

Symptom & Error Signature

When you attempt to integrate changes from a remote Git repository into your local working copy, typically using git pull or git merge, you might encounter the following error message in your terminal:
```
$ git pull origin main
From github.com:your-organization/your-repository
 * branch            main       -> FETCH_HEAD
fatal: refusing to merge unrelated histories
```
Or, if you're trying a direct merge:
```
$ git merge origin/main
fatal: refusing to merge unrelated histories
```
This error halts the pull/merge operation, leaving your local repository in its current state without integrating the remote changes.

Root Cause Analysis

The "fatal: refusing to merge unrelated histories" error is a safety mechanism introduced in Git version 2.9 (released June 2016). Prior to this version, Git would attempt to merge any two branches you specified, even if they had no common commit history, potentially leading to a confusing and possibly destructive merge.

Git's core principle is to build a directed acyclic graph (DAG) of commits. When you try to merge two branches, Git normally expects to find a common ancestor commit from which both branches diverged. If no such common ancestor exists, Git considers their histories "unrelated."

Common scenarios that lead to unrelated histories include:
1. Initializing an Empty Local Repo, Pulling from Existing Remote: You create a new project directory, run git init, and then try to git pull from an existing remote repository that already has commits (e.g., a README.md or initial project files). Your local HEAD has no commits, and the remote HEAD has a history, so they don't share a common ancestor.
2. Remote Repository Initialized Separately: You create an empty repository on a platform like GitHub or GitLab, and then initialize it directly on the web interface by adding a README.md, .gitignore, or license file. Simultaneously, you create a local repository with git init and some initial commits without cloning the remote. When you later try to git pull from the remote into your local repo, their histories are unrelated.
3. Restoring from a "Working Directory" Backup: If you only backed up the .git folder along with the working directory, and then restored it, or if you only backed up the working directory and then re-initialized Git with git init, the new local .git history might not align with the remote's history graph if not handled carefully.
4. Accidental Parallel Development: Two separate projects were started, both initialized as Git repositories, and later an attempt is made to merge them as if they were branches of a single project.
In essence, Git is preventing what it perceives as an attempt to merge two entirely distinct timelines, which could obscure project history or introduce unexpected changes.

Step-by-Step Resolution

The primary and recommended solution involves explicitly telling Git to allow merging of unrelated histories using the --allow-unrelated-histories flag.

1. Understanding the allow-unrelated-histories Flag

This flag instructs Git to proceed with the merge even if it cannot find a common ancestor between the two histories. When this flag is used, Git treats the first commit of one history and the first commit of the other history as their common base for the merge, effectively creating a merge commit that bridges the two distinct histories.

[!IMPORTANT] Use this flag with caution. While it resolves the immediate error, ensure you understand why the histories are unrelated. This action cannot be easily undone without rewriting history, which can be problematic in collaborative environments.

2. Safely Merging Unrelated Histories (Recommended Approach)

This method preserves both local and remote histories, combining them into a new merge commit.

2.1 Verify Current Status and Remote Configuration

Before proceeding, ensure your local repository is clean and correctly configured to point to the remote.
```
# Check for any uncommitted local changes (commit or stash them if necessary)
```
Verify your remote configuration git remote -v `

Expected output for git remote -v:
```
origin  https://github.com/your-organization/your-repository.git (fetch)
origin  https://github.com/your-organization/your-repository.git (push)
```
2.2 Fetch Remote Changes (Optional, but Good Practice)

It's often good practice to first fetch all remote changes without merging them. This allows you to inspect the remote's history if needed.
```
git fetch origin
```
2.3 Perform the Merge with --allow-unrelated-histories

Now, execute the git pull or git merge command with the crucial flag. Assuming you want to pull from the main branch of your origin remote:
```
git pull origin main --allow-unrelated-histories
```
If you prefer to fetch and then merge manually:
```
git merge origin/main --allow-unrelated-histories
```
Git will then attempt to merge the two histories.

2.4 Resolve Merge Conflicts (If Any)

[!IMPORTANT] It is highly probable that you will encounter merge conflicts after using --allow-unrelated-histories, especially if both histories have different files or different versions of the same files at their respective roots (e.g., two different README.md files).

Your terminal will indicate any conflicting files:
```
Auto-merging README.md
CONFLICT (add/add): Merge conflict in README.md
Automatic merge failed; fix conflicts and then commit the result.
```
1. Open the conflicting files in your preferred text editor (e.g., nano, vim, vscode).
2. Identify and resolve conflicts: Git marks conflicts with <<<<<<<, =======, and >>>>>>>.
3. `markdown
4. <<<<<<< HEAD
5. This is the content from my local repository.
6. =======
7. This is the content from the remote repository.
8. >>>>>>> origin/main
9. `
10. Edit the file to include the desired content.
11. Stage the resolved files:
12. `bash
13. git add . # Or git add <conflicting-file-1> <conflicting-file-2>
14. `
15. Commit the merge: Git will typically provide a default merge commit message. You can accept it or modify it.
16. `bash
17. git commit -m "Merge remote main with –allow-unrelated-histories and resolved conflicts"
18. `
2.5 Push Changes to Remote (If Necessary)

Once the merge is complete and committed locally, you can push the new, unified history to your remote repository.
```
git push origin main
```
3. Alternative: Reinitialize Local Repository (Use with Caution)

This approach discards your local repository's history and state, effectively making it a fresh clone of the remote. This is suitable if your local repository has no important uncommitted changes, or if its history is completely irrelevant compared to the remote.

3.1 Backup Local Changes (If Any)

If you have any uncommitted or local-only committed changes that you wish to preserve, back them up.
```
# Stash uncommitted changes
```
Or, simply copy your entire project directory cp -r /path/to/your/project /path/to/your/projectbackupdate +%Y%m%d%H%M%S `

3.2 Remove Local .git Directory

Navigate into your project directory and remove the hidden .git folder. This effectively de-initializes your local repository.
```
cd /path/to/your/project
rm -rf .git
```
3.3 Re-clone the Remote Repository

Navigate to the parent directory of your project and clone the remote repository anew.
```
cd /path/to/parent/directory
git clone https://github.com/your-organization/your-repository.git your_project_name
cd your_project_name
```
3.4 Restore and Reapply Local Changes (If Any)

If you stashed changes in step 3.1, you can now try to apply them. Be aware that conflicts might still occur if the remote has diverged significantly.
```
# If you stashed changes and are in the newly cloned repo:
```
Alternatively, manually copy back specific files from your backup cp /path/to/your/projectbackup/yourfile.js . `

4. Alternative: Force Push (Use with EXTREME Caution)

This option is generally NOT recommended in collaborative environments. It should only be used if you are absolutely certain that your local repository's history should completely overwrite the remote's history, and no one else is working on that branch.

4.1 Ensure Local Repository is Exactly What You Want

Make sure your local branch (main in this example) contains the exact history and files you want to be on the remote.

4.2 Force Push
```
# Safer variant: only forces if the remote branch hasn't been updated since your last pull/fetch
```
More aggressive variant: forces regardless of remote updates, can overwrite others' work # Use with extreme caution, only if you are absolutely sure. # git push –force origin main `

[!WARNING] The git push --force-with-lease or git push --force commands will overwrite the remote branch history! All commits on the remote that are not present in your local branch will be permanently lost. This can cause significant problems for other developers working on the same branch. Only use this if you are absolutely certain of the consequences and have communicated with your team, or if you are the sole contributor and are certain the remote history is undesirable.

By understanding the root cause and carefully applying one of these resolution methods, particularly the --allow-unrelated-histories flag, you can effectively overcome the "fatal: refusing to merge unrelated histories" error and continue your development workflow on Ubuntu 20.04 LTS.
July 15, 2026
Resolving Apache SSL/TLS Protocol & Cipher Suite Mismatch on WSL2 Ubuntu
Fix Apache SSL protocol version and cipher suite errors on Windows WSL2 Ubuntu. This guide covers common TLS negotiation failures and provides step-by-step configuration fixes for a secure setup.

A common challenge for developers and system administrators working with Apache on Windows Subsystem for Linux 2 (WSL2) Ubuntu is encountering SSL/TLS errors related to protocol version or cipher suite mismatches. These errors prevent secure connections (HTTPS) from being established between a client (your browser) and the Apache web server running inside your WSL2 instance. This guide will walk you through diagnosing and resolving these frustrating TLS handshake failures by correctly configuring Apache's SSL modules.

Symptom & Error Signature

When experiencing an SSL protocol or cipher suite mismatch, you will typically see the following in your web browser:
- Google Chrome: NET::ERRSSLPROTOCOLERROR or ERRSSLVERSIONORCIPHERMISMATCH
- Mozilla Firefox: SSLERRORPROTOCOLVERSIONORCIPHERMISMATCH
- Microsoft Edge: NET::ERRSSLPROTOCOL_ERROR
In Apache's error logs (located at /var/log/apache2/error.log within your WSL2 Ubuntu instance), you might find entries similar to these, indicating a failure during the TLS handshake:
```
[timestamp] [ssl:error] [pid XXXX] [client 172.XX.XX.XX:XXXXX] AH02039: Certificate Verification: Error, `SSL_do_handshake() failed: error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher`
[timestamp] [ssl:error] [pid YYYY] [client 172.XX.XX.XX:XXXXX] AH02039: Certificate Verification: Error, `SSL_do_handshake() failed: error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol`
```
The key phrases to look for are no shared cipher and unsupported protocol, which directly point to the core issue.

Root Cause Analysis

This error occurs when the client (your web browser) and the server (Apache running in WSL2 Ubuntu) cannot agree on a common TLS protocol version or a mutually supported cryptographic cipher suite during the initial TLS handshake. The primary reasons for this discrepancy often include:
1. Outdated Apache/OpenSSL Configuration: The Apache server, or its underlying OpenSSL library within WSL2, might be configured to use or prefer older, insecure TLS protocols (like TLSv1.0 or TLSv1.1) or weak, deprecated cipher suites. Modern browsers have dropped support for these to enhance security, leading to a mismatch.
2. Overly Restrictive Server Configuration: Less commonly, the Apache server might be configured to only allow very new TLS protocols (e.g., TLSv1.3 exclusively) or highly specific cipher suites that your client's browser might not yet fully support or have enabled.
3. System Outdatedness: The WSL2 Ubuntu distribution itself or Apache/OpenSSL packages might be outdated, lacking support for modern TLS versions or strong cipher suites.
4. Incorrect Virtual Host Configuration: If you have multiple virtual hosts, one might be incorrectly configured, overriding global SSL settings.
The goal of the resolution is to ensure your Apache configuration uses modern, secure, and widely supported TLS protocols (primarily TLSv1.2 and TLSv1.3) and robust cipher suites that are compatible with contemporary web browsers.

Step-by-Step Resolution

Follow these steps within your WSL2 Ubuntu environment to resolve the Apache SSL/TLS protocol and cipher suite mismatch.

1. Update Your WSL2 Ubuntu System and Apache Packages

Ensure your system and Apache are up-to-date. This often pulls in newer OpenSSL versions and patches that support modern TLS features.
```
sudo apt update
sudo apt upgrade -y
sudo apt autoremove -y
```
2. Backup Your Apache SSL Configuration

Before making any changes, create backups of your critical Apache SSL configuration files. This allows for easy rollback if anything goes wrong.
```
# Backup the main SSL module configuration
```
Backup your primary SSL virtual host configuration (adjust path if needed) # Common locations: /etc/apache2/sites-available/default-ssl.conf or your custom site file sudo cp /etc/apache2/sites-available/000-default-ssl.conf /etc/apache2/sites-available/000-default-ssl.conf.bak.$(date +%Y%m%d%H%M%S) `

3. Inspect and Modify Apache SSL Configuration Files

The primary files to check are /etc/apache2/mods-available/ssl.conf (which is often symlinked to mods-enabled/ssl.conf) and your virtual host configuration file for SSL (e.g., /etc/apache2/sites-available/your-site-ssl.conf).

Open ssl.conf for editing:
```
sudo nano /etc/apache2/mods-available/ssl.conf
```
Or, if your SSL configuration is primarily within your virtual host file:
```
sudo nano /etc/apache2/sites-available/000-default-ssl.conf
```
Within these files, locate or add the SSLProtocol and SSLCipherSuite directives.

4. Adjust SSLProtocol Directive

The SSLProtocol directive controls which TLS versions Apache will accept. You need to disable older, insecure versions and ensure modern ones are enabled.

Find the SSLProtocol line and modify it as follows:
```
# Recommended configuration for modern security
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
```
[!IMPORTANT] The all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1 directive enables all supported protocols except SSLv2, SSLv3, TLSv1.0, and TLSv1.1. This ensures that only TLSv1.2 and TLSv1.3 (if supported by your OpenSSL version) are used, which are considered secure by modern standards.

5. Adjust SSLCipherSuite Directive

The SSLCipherSuite directive defines the list of cryptographic algorithms that Apache will use for encrypted connections. Using an overly restrictive or outdated list can cause handshake failures. Adopt a modern, secure cipher suite string. The Mozilla SSL Configuration Generator is an excellent resource for this. For intermediate compatibility and strong security, use something like this:

Find the SSLCipherSuite line and modify it:
```
# Recommended intermediate compatibility cipher suite (Mozilla recommended)
SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
```
[!WARNING] Do not use ALL for SSLCipherSuite, as it includes weak and insecure ciphers. Always specify a curated list. The example above provides a strong, widely compatible set.

You may also want to set SSLHonorCipherOrder On to ensure Apache prefers the server's order of cipher suites over the client's.
```
SSLHonorCipherOrder On
```
6. Configure Header always set Strict-Transport-Security (HSTS) – Optional but Recommended

For enhanced security, consider enabling HSTS to force browsers to interact with your site only over HTTPS for a specified duration.
```
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"
```
[!NOTE] For HSTS to work, the mod_headers Apache module must be enabled. You can enable it using sudo a2enmod headers.

7. Save Changes and Test Apache Configuration

Save the changes to the configuration file (Ctrl+O, Enter, Ctrl+X in nano). Before restarting Apache, always test your configuration syntax to catch errors.
```
sudo apache2ctl configtest
```
If you see Syntax OK, you can proceed to restart Apache. If there are errors, review your changes carefully.

8. Restart Apache Service

Apply the new configuration by restarting Apache:
```
sudo systemctl restart apache2
```
[!IMPORTANT] If Apache fails to restart, check sudo systemctl status apache2 and /var/log/apache2/error.log for specific error messages. Your backup files will be useful for reverting if necessary.

9. Test Your Website

Now, try accessing your website via HTTPS in your browser. The SSL/TLS protocol and cipher suite mismatch error should be resolved, and your site should load securely.

You can also use the openssl command-line tool from within your WSL2 instance to test the server's SSL configuration:
```
openssl s_client -connect 127.0.0.1:443 -tls1_2
openssl s_client -connect 127.0.0.1:443 -tls1_3
```
Replace 127.0.0.1 with the IP address Apache listens on if different (e.g., your WSL2 internal IP if accessing from Windows host). These commands will show you the protocol and cipher suite negotiated, as well as the certificate details. If the connection is successful, it confirms Apache is correctly configured for modern TLS versions.

By following these steps, you will have successfully configured your Apache server on WSL2 Ubuntu to use modern, secure TLS protocols and strong cipher suites, resolving the handshake errors and ensuring a secure connection for your web applications.
July 13, 2026
Troubleshooting ‘Docker Registry Push Failed: Authentication Token Expired’ on Ubuntu 20.04 LTS
Fix Docker registry push errors due to expired authentication tokens on Ubuntu 20.04 LTS. Learn to reauthenticate and resolve Docker login issues quickly.

Introduction

As an experienced Systems Administrator and DevOps engineer, encountering authentication failures when pushing Docker images to a private or public registry is a common yet frustrating issue. One specific error, "authentication token expired," indicates that your Docker client's cached credentials for the registry are no longer valid, preventing successful image uploads. This guide will walk you through diagnosing and resolving this problem on an Ubuntu 20.04 LTS system, ensuring your CI/CD pipelines and manual pushes can proceed without interruption.

Symptom & Error Signature

When attempting to push a Docker image to a remote registry, the operation fails, typically after successfully building or pulling the image. The terminal output will indicate an authentication failure, often explicitly mentioning an expired token or general authorization denial.

Here's a common error signature you might observe:
```
$ docker push myregistry.example.com/my-org/my-app:latest
The push refers to repository [myregistry.example.com/my-org/my-app]
...
denied: authentication required
Error response from daemon: authorization failed
Error response from daemon: unauthorized: authentication token expired
```
Alternatively, if you attempt to docker login and the token has expired, you might see:
```
$ docker login myregistry.example.com
Authenticating with existing credentials...
Error: Your authorization token has expired. Please log in again.
```
Root Cause Analysis

The "authentication token expired" error, while seemingly straightforward, can stem from a few underlying causes:
1. Time-Limited Tokens: This is the most common reason. Docker registry authentication tokens (be it from a docker login session or a CI/CD system's generated token) are often designed to have a finite lifespan for security reasons. Once this period elapses, the token becomes invalid.
2. Password/Credential Change: The username or password associated with the Docker registry account might have been changed or rotated by an administrator since the last successful login. The client's locally cached credentials (~/.docker/config.json) would then no longer match the active credentials on the registry.
3. Token Revocation: An administrator might have explicitly revoked the specific authentication token or session linked to your client for security or operational reasons.
4. Network/Proxy Interruption During Re-authentication (Less Common for "Expired"): While not directly causing an "expired" token, an intermittent network issue or misconfigured proxy could prevent the Docker client from successfully renewing or re-establishing an authenticated session, making it seem like the old token is the sole issue.
5. Stale Docker Daemon Session: In rare cases, a misbehaving or stale Docker daemon might hold onto outdated authentication information, preventing it from correctly using newly supplied credentials.
Step-by-Step Resolution

Follow these steps to diagnose and resolve the "authentication token expired" error.

#### 1. Verify Current Docker Login Status

First, let's check what Docker thinks its current authentication status is for the registry in question.
```
cat ~/.docker/config.json
```
This command will display your Docker configuration file, which stores cached registry credentials. Look for an entry under "auths" that corresponds to myregistry.example.com (or your specific registry hostname). If an auth field exists, it indicates Docker has some credentials cached, but they might be expired or invalid.
```
{
  "auths": {
    "myregistry.example.com": {
      "auth": "Zn...oZGl" // Base64 encoded username:password
    },
    "docker.io": {
      "auth": "YWJ...sdf",
      "credsStore": "desktop"
    }
  },
  "HttpHeaders": {
    "User-Agent": "Docker-Client/20.10.7 (linux)"
  }
}
```
The presence of an auth token or a credsStore reference confirms Docker has attempted to store credentials.

#### 2. Re-authenticate with the Docker Registry

The most direct solution for an expired token is to simply log in again. This forces the Docker client to obtain a fresh authentication token from the registry.
```
docker login myregistry.example.com
```
You will be prompted to enter your username and password for the registry.
```
Username: <your-username>
Password: <your-password>
Login Succeeded
```
[!IMPORTANT] * Ensure you use the correct username and a currently valid password or access token for the registry. If your registry uses Multi-Factor Authentication (MFA), you might need to use an API token or an application-specific password rather than your primary account password. Consult your registry's documentation for MFA login procedures. * After successful login, immediately try your docker push command again.

#### 3. Clear Stale Docker Credentials (If Re-authentication Fails)

If a direct re-authentication doesn't resolve the issue, or if you suspect corrupted local credentials, it's best to explicitly log out and then log in again.
1. Log out from the specific registry:
2. `bash
3. docker logout myregistry.example.com
4. `
5. This command removes the cached credentials for myregistry.example.com from ~/.docker/config.json.
1. Manually verify ~/.docker/config.json (Optional but recommended):
2. After docker logout, inspect the ~/.docker/config.json file again to ensure the entry for your registry has been removed or updated. If you find any residual entries or suspect corruption, you can manually edit the file.
[!WARNING] > Backup your ~/.docker/config.json before manual editing. Incorrectly modifying this file can disrupt Docker's ability to interact with any registry. > `bash > cp ~/.docker/config.json ~/.docker/config.json.bak > nano ~/.docker/config.json > ` > Carefully remove the entire block related to myregistry.example.com under the "auths" key.
1. Attempt docker login again:
2. `bash
3. docker login myregistry.example.com
4. `
5. Enter your username and password when prompted.
#### 4. Check Registry Service Health and Connectivity

Sometimes, the issue isn't with your token but with the registry being unreachable or having its own issues.
1. Test network connectivity:
2. `bash
3. ping -c 4 myregistry.example.com
4. `
5. Ensure you receive responses. If not, check your local network configuration, DNS settings, and firewall rules.
1. Test HTTPS connectivity (basic):
2. `bash
3. curl -v https://myregistry.example.com/v2/
4. `
5. This command attempts to access a common Docker registry API endpoint. A successful connection will show HTTP headers, possibly an empty JSON response, but importantly, no connection errors. If you see certificate errors (SSLCTXsetdefaultverify_paths), your system might be missing root certificates or your registry uses a self-signed certificate not trusted by your system.
[!IMPORTANT] > If your network requires a proxy for outgoing connections, ensure your Docker daemon and client are configured to use it. > * For Docker daemon: Configure proxy settings in /etc/systemd/system/docker.service.d/http-proxy.conf or similar. > * For Docker client: Set HTTPPROXY, HTTPSPROXY, and NO_PROXY environment variables.

#### 5. Verify User Permissions on the Registry

While "token expired" specifically points to authentication, it's always good to confirm that the user account you're using still has the necessary permissions to push to the target repository. This usually involves checking the registry's web interface or contacting your registry administrator.
- Confirm push permissions: Ensure the user account is authorized to push images to the specific repository path (e.g., my-org/my-app).
- Team/Role assignments: Check if the user is part of the correct team or role that grants push access.
#### 6. Restart Docker Daemon (If All Else Fails)

In rare cases, a stale Docker daemon process might be holding onto old session information. Restarting the Docker daemon can clear its internal state and force it to pick up fresh authentication details.
```
sudo systemctl restart docker
sudo systemctl status docker
```
[!WARNING] Restarting the Docker daemon will stop all running containers on your system. Ensure this is acceptable for your environment or schedule it during a maintenance window. If you have critical services running, proceed with caution.

After the daemon has restarted, attempt your docker login and docker push commands again.

By systematically working through these steps, you should be able to resolve the "Docker registry push failed: authentication token expired" issue on your Ubuntu 20.04 LTS system.
July 11, 2026
Troubleshooting PostgreSQL shared_buffers Memory Allocation Crash on Ubuntu 20.04 LTS
Resolve critical PostgreSQL startup failures on Ubuntu 20.04 due to misconfigured shared_buffers exceeding system or kernel shared memory limits.

When managing high-performance PostgreSQL databases, optimizing memory usage through parameters like shared_buffers is crucial. However, misconfiguring this setting can lead to a critical PostgreSQL service crash upon startup, presenting as a "Cannot allocate memory" error specifically related to shared memory segments. This guide will walk you through diagnosing and resolving this issue on Ubuntu 20.04 LTS systems.

Symptom & Error Signature

The most common symptom is that your PostgreSQL service fails to start, leading to applications being unable to connect to the database. When checking the service status or logs, you'll encounter FATAL errors indicating a failure to allocate shared memory.

You can observe this by checking your PostgreSQL logs or journalctl:
```
sudo systemctl status postgresql
```
```
● postgresql.service - PostgreSQL RDBMS
     Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-10-27 10:00:05 UTC; 5s ago
    Process: 12345 ExecStart=/usr/lib/postgresql/12/bin/pg_ctlcluster --skip-systemctl-redirect 12 main start (code=exited, status=1/FAILURE)
```
Oct 27 10:00:05 webserver systemd[1]: Starting PostgreSQL RDBMS… Oct 27 10:00:05 webserver pg_ctlcluster[12345]: FATAL: could not create shared memory segment: Cannot allocate memory Oct 27 10:00:05 webserver pg_ctlcluster[12345]: DETAIL: Failed system call was shmget(key=5432001, size=16777216000, 0x1FF). Oct 27 10:00:05 webserver pg_ctlcluster[12345]: HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available system memory or kernel limits. Oct 27 10:00:05 webserver pg_ctlcluster[12345]: The PostgreSQL process would require 16777216000 bytes of shared memory. Oct 27 10:00:05 webserver pgctlcluster[12345]: Check for other PostgreSQL instances running. If not, try reducing the request size (e.g., by reducing sharedbuffers or max_connections), or check system limitations. Oct 27 10:00:05 webserver pg_ctlcluster[12345]: LOG: database system is shut down Oct 27 10:00:05 webserver systemd[1]: postgresql.service: Main process exited, code=exited, status=1/FAILURE Oct 27 10:00:05 webserver systemd[1]: postgresql.service: Failed with result 'exit-code'. Oct 27 10:00:05 webserver systemd[1]: Failed to start PostgreSQL RDBMS. `

Or by examining the PostgreSQL log file directly (e.g., for PostgreSQL 12):
```
sudo tail -n 50 /var/log/postgresql/postgresql-12-main.log
```
```
2023-10-27 10:00:05.123 UTC [12345] LOG:  starting PostgreSQL 12.16 (Ubuntu 12.16-0ubuntu0.20.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, 64-bit
2023-10-27 10:00:05.123 UTC [12345] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2023-10-27 10:00:05.123 UTC [12345] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-10-27 10:00:05.124 UTC [12345] FATAL:  could not create shared memory segment: Cannot allocate memory
2023-10-27 10:00:05.124 UTC [12345] DETAIL:  Failed system call was shmget(key=5432001, size=16777216000, 0x1FF).
2023-10-27 10:00:05.124 UTC [12345] HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available system memory or kernel limits.  The PostgreSQL process would require 16777216000 bytes of shared memory.  Check for other PostgreSQL instances running. If not, try reducing the request size (e.g., by reducing shared_buffers or max_connections), or check system limitations.
2023-10-27 10:00:05.124 UTC [12345] LOG:  database system is shut down
```
Root Cause Analysis

The FATAL: could not create shared memory segment: Cannot allocate memory error, specifically with the shmget system call failure, indicates that PostgreSQL attempted to allocate a shared memory segment that either exceeded the physical memory available on the system or, more commonly, violated the Linux kernel's System V IPC (Interprocess Communication) shared memory limits.

PostgreSQL uses a significant portion of its memory for shared_buffers, which acts as a cache for database pages. This memory is allocated from the kernel's shared memory pool. The Linux kernel has two primary parameters that govern System V shared memory:
1. kernel.shmmax: This parameter defines the maximum size (in bytes) of a single shared memory segment that the kernel will allow. If PostgreSQL's shared_buffers (which is typically one large shared memory segment) configured value exceeds kernel.shmmax, the allocation will fail.
2. kernel.shmall: This parameter defines the total amount of shared memory (in pages) that the system can use at once. If the sum of all shared memory segments requested by all processes (including PostgreSQL) exceeds kernel.shmall multiplied by the system's PAGESIZE, allocation will fail. On x86-64 systems, PAGESIZE is typically 4096 bytes (4KB).
The crash typically occurs when: * You've set shared_buffers in postgresql.conf to a value too high for your system's physical RAM. * The shared_buffers value, even if reasonable for RAM, exceeds the current kernel.shmmax limit. * The overall shared memory footprint requested (primarily shared_buffers) exceeds the total allowed by kernel.shmall.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the PostgreSQL shared buffers memory allocation crash.

1. Verify PostgreSQL Service Status and Examine Logs

First, confirm that the PostgreSQL service is indeed failing and check the logs for the exact error message.
```
# Check service status
```
View recent PostgreSQL specific logs (PostgreSQL 12) sudo journalctl -u postgresql –no-pager | grep -i "fatal"

Alternatively, check the main PostgreSQL log file sudo tail -n 100 /var/log/postgresql/postgresql-12-main.log `

Confirm the presence of the FATAL: could not create shared memory segment: Cannot allocate memory error. Note the size mentioned in the DETAIL line, as this is the amount PostgreSQL tried to allocate.

2. Identify Current shared_buffers Configuration

Locate your postgresql.conf file and check the configured shared_buffers value. For PostgreSQL 12 on Ubuntu 20.04, it's typically located at /etc/postgresql/12/main/postgresql.conf.
```
# Display the shared_buffers line from your config
grep -E '^shared_buffers' /etc/postgresql/12/main/postgresql.conf
```
3. Determine Current Kernel Shared Memory Limits

Check the current values of kernel.shmmax and kernel.shmall on your system.
```
# View current kernel shared memory maximum segment size
```
View current kernel shared memory total pages sudo sysctl kernel.shmall

Get system page size (usually 4096 bytes or 4KB) getconf PAGESIZE `

[!IMPORTANT] The HINT in the PostgreSQL error message will often provide the exact byte size that PostgreSQL attempted to allocate. This is your target value for kernel.shmmax. For kernel.shmall, you'll need to divide that byte size by your system's PAGE_SIZE (e.g., 4096).

4. Option A: Reduce shared_buffers (Recommended First Approach)

If your shared_buffers value is excessively high (e.g., more than 25-30% of your total system RAM on a dedicated DB server, or even less if running other services), reducing it is often the simplest and safest fix.
1. Edit postgresql.conf:
2. `bash
3. sudo nano /etc/postgresql/12/main/postgresql.conf
4. `
5. Adjust shared_buffers:
6. Find the shared_buffers line and set it to a more conservative value. A common recommendation is 25% of your system's total RAM. For example, if you have 8GB of RAM, 2GB would be a good starting point. If you have 4GB RAM, 1GB is appropriate.
```
    # Example: Reduce shared_buffers to 2GB
    shared_buffers = 2GB
    ```
    > [!WARNING]
```
1. Save and Exit:
2. Press Ctrl+X, then Y to save, and Enter to confirm the filename.
1. Restart PostgreSQL:
2. `bash
3. sudo systemctl restart postgresql
4. `
1. Verify Status:
2. `bash
3. sudo systemctl status postgresql
4. `
5. If successful, the service should now be active (running).
5. Option B: Increase Kernel Shared Memory Limits (If Reduction is Not Desirable)

If you have ample physical RAM and genuinely require a larger shared_buffers value (e.g., 4GB or more on systems with 16GB+ RAM), you can increase the kernel's shared memory limits. This requires calculating appropriate shmmax and shmall values.
1. Determine Required Values:
2. Let's assume you want to set shared_buffers = 4GB.
- kernel.shmmax: This should be at least the value of shared_buffers in bytes.
- 4 GB = 4 1024 1024 * 1024 = 4294967296 bytes
- So, kernel.shmmax = 4294967296
- kernel.shmall: This is calculated as the desired total shared memory in bytes, divided by the system's page size, then rounded up. Assuming PAGE_SIZE = 4096 bytes:
- 4294967296 bytes / 4096 bytes/page = 1048576 pages
- So, kernel.shmall = 1048576
1. Create/Edit a sysctl.d Configuration File:
2. It's best practice to create a new .conf file in /etc/sysctl.d/ rather than modifying /etc/sysctl.conf directly.
```
    sudo nano /etc/sysctl.d/99-postgresql-shm.conf
```
1. Add/Modify Parameters:
2. Add the calculated values to the file.
```
    # PostgreSQL shared memory settings
    kernel.shmmax = 4294967296
    kernel.shmall = 1048576
```
1. Save and Exit:
2. Press Ctrl+X, then Y to save, and Enter to confirm the filename.
1. Apply New Kernel Parameters:
2. `bash
3. sudo sysctl -p /etc/sysctl.d/99-postgresql-shm.conf
4. `
5. You should see output confirming the new values:
6. `
7. kernel.shmmax = 4294967296
8. kernel.shmall = 1048576
9. `
10. These changes are persistent across reboots due to being in /etc/sysctl.d/.
1. Verify New Kernel Parameters:
2. `bash
3. sudo sysctl kernel.shmmax kernel.shmall
4. `
1. Adjust shared_buffers in postgresql.conf (if not already done):
2. If you increased kernel limits to support a larger shared_buffers value, ensure postgresql.conf reflects this.
```
    sudo nano /etc/postgresql/12/main/postgresql.conf
    ```
```
1. Restart PostgreSQL:
2. `bash
3. sudo systemctl restart postgresql
4. `
1. Verify Status:
2. `bash
3. sudo systemctl status postgresql
4. `
5. The service should now be active (running).
[!IMPORTANT] Always ensure that the sum of sharedbuffers and other memory-intensive PostgreSQL settings (like workmem, maintenanceworkmem, walbuffers, and memory for maxconnections) does not exceed your system's physical RAM to prevent excessive swapping and performance degradation. As a general rule for shared_buffers, start with 25% of system RAM and never exceed 50% without very specific reasons and careful monitoring.
July 11, 2026
Resolving MySQL Error 1040: Too Many Connections on Ubuntu 20.04 LTS
Troubleshoot and fix MySQL Error 1040 'Too many connections' on Ubuntu 20.04 LTS. Learn to adjust system limits and MySQL configurations for optimal performance.

When your web application or service experiences intermittent downtime or slow responses, and you encounter error messages like "Error establishing a database connection," it often points to an underlying database issue. MySQL Error 1040, specifically "Too many connections," is a common indicator that your database server has reached its capacity for handling simultaneous client connections, leading to service disruption. This guide will walk you through diagnosing and resolving this critical issue on Ubuntu 20.04 LTS, addressing both MySQL's internal limits and the underlying operating system configurations.

Symptom & Error Signature

Users typically experience one of the following:
- Website/Application unavailability: A generic database connection error message displayed in the browser or application UI (e.g., "Error establishing a database connection" for WordPress).
- Application Log Entries: Specific error messages within your application's logs (e.g., PHP, Python, Java stack traces) indicating a connection failure.
- MySQL Client Error: When attempting to connect to MySQL from the command line or an application, you receive:
```
    ERROR 1040 (HY000): Too many connections
```
- MySQL Error Log Entries: The MySQL server error log (/var/log/mysql/error.log or viewable via journalctl -u mysql.service) will contain entries like:
```
    [ERROR] [MY-00000] [Server] Too many connections
```
Root Cause Analysis

MySQL Error 1040 indicates that the server has reached the maximum number of simultaneous client connections it can handle. This can be due to several factors, often a combination of them:
1. max_connections Limit: This is the primary MySQL configuration parameter that defines the maximum number of concurrent client connections allowed. The default value is often set conservatively (e.g., 151), which can be easily exceeded by busy applications.
2. systemd LimitNOFILE: Each connection to MySQL, along with open tables and other internal server operations, consumes file descriptors. The mysqld process, when started by systemd, is subject to the LimitNOFILE (Maximum number of open file descriptors) setting in its systemd unit file (mysql.service). If this limit is too low, MySQL won't be able to open new files or accept new connections, even if max_connections is high.
3. sysctl fs.file-max: This is a global system-wide kernel parameter that defines the maximum number of file handles the Linux kernel can allocate. If this limit is hit, no process on the system, including MySQL, can open new files or sockets.
4. Application Behavior:
5. * Connection Leaks: Applications failing to properly close database connections after use, leading to a build-up of open connections.
6. * Inefficient Queries/Long-running Transactions: Slow queries or transactions that hold connections open for extended periods, consuming slots.
7. * Spikes in Traffic: Sudden, unmanaged surges in user traffic or bot activity overwhelming the database.
8. * Lack of Connection Pooling: Applications constantly opening and closing new connections instead of reusing a pool of existing ones.
9. Resource Exhaustion: While not a direct cause of "too many connections," high CPU, RAM, or I/O utilization can slow down query processing, causing connections to remain active longer, exacerbating the connection limit issue.
Step-by-Step Resolution

This section outlines a structured approach to diagnose and resolve MySQL Error 1040. We'll start with diagnostics and then move to increasing limits at different levels.

1. Diagnose Current Status

Before making changes, understand the current state of your system and MySQL configuration.
- Check MySQL max_connections and active connections:
```
    mysql -u root -p -e "SHOW VARIABLES LIKE 'max_connections';"
    mysql -u root -p -e "SHOW STATUS LIKE 'Threads_connected';"
    mysql -u root -p -e "SHOW STATUS LIKE 'Max_used_connections';"
```
Maxusedconnections shows the highest number of connections that have been in use simultaneously since the server started. This is crucial for determining how high you need to set max_connections.
- Check mysqld process open file limits:
```
    # Find the process ID (PID) of the MySQL server
    PID=$(pgrep mysqld)
    # View its current limits
    sudo cat /proc/$PID/limits | grep "Max open files"
```
This will show you the Limit: Max open files for the running mysqld process, which is often constrained by systemd.
- Check system-wide file descriptor limits:
```
    cat /proc/sys/fs/file-max
```
This displays the global maximum number of file handles the kernel can allocate.
- Review MySQL error logs:
```
    sudo journalctl -u mysql.service --since "1 hour ago" -p err
    # Or, if using a separate log file:
    # tail -f /var/log/mysql/error.log
```
Look for repeated "Too many connections" errors or other issues indicating instability.

2. Adjust MySQL max_connections

This is the most common fix, directly increasing the number of connections MySQL will accept.
1. Edit MySQL configuration:
2. The main MySQL configuration file is typically /etc/mysql/mysql.conf.d/mysqld.cnf or /etc/my.cnf. On Ubuntu, it's often a file within the /etc/mysql/mysql.conf.d/ directory (e.g., mysqld.cnf).
```
    sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf
```
1. Locate or add max_connections:
2. Under the [mysqld] section, add or modify the maxconnections parameter. A good starting point is often 200-500, but it should be tailored based on Maxused_connections from your diagnostics and available RAM. Each connection consumes RAM, so setting it excessively high without sufficient memory can lead to swapping and performance degradation.
```
    [mysqld]
    # ... other configurations ...
    max_connections = 500
```
[!IMPORTANT] > Carefully consider your server's RAM. Each connection requires memory (e.g., buffer sizes, thread stack). Setting max_connections too high without sufficient RAM can cause the server to run out of memory, crash, or swap excessively, severely impacting performance.
1. Save and exit the editor.
1. Restart MySQL service:
```
    sudo systemctl restart mysql
```
1. Verify the change:
```
    mysql -u root -p -e "SHOW VARIABLES LIKE 'max_connections';"
```
3. Increase Open File Limits for MySQL (systemd LimitNOFILE)

If increasing max_connections doesn't resolve the issue, or if Max open files for mysqld is too low, you need to adjust the systemd service unit for MySQL.
1. Create a systemd override file:
2. Use systemctl edit to safely create an override file for the mysql.service unit. This prevents direct modification of the main unit file, making upgrades cleaner.
```
    sudo systemctl edit mysql.service
```
1. Add/modify LimitNOFILE:
2. Add the following lines to the override file. The value for LimitNOFILE should be significantly higher than maxconnections. A common practice is maxconnections * 2 or more, to account for other file descriptors used by MySQL. For example, if max_connections is 500, set LimitNOFILE to 2048 or even 4096.
```
    [Service]
    LimitNOFILE=4096
```
[!WARNING] > Setting LimitNOFILE to an extremely high or incorrect value can prevent the MySQL service from starting. Start with a moderately high value and increase if necessary. This value must also not exceed the system-wide fs.file-max.
1. Save and exit the editor. systemd will automatically reload the daemon when you save the file.
1. Restart MySQL service:
2. This is crucial for the new systemd limits to take effect.
```
    sudo systemctl restart mysql
```
1. Verify the new LimitNOFILE:
```
    PID=$(pgrep mysqld)
    sudo cat /proc/$PID/limits | grep "Max open files"
```
Ensure the "Max open files" value reflects your change.

4. Adjust System-wide File Descriptor Limit (sysctl)

If fs.file-max (system-wide limit) is too low, it can prevent any process from opening new files. This is less common but important to check, especially on systems with many services or high file I/O.
1. Edit sysctl.conf:
```
    sudo vim /etc/sysctl.conf
```
1. Add or modify fs.file-max:
2. Add or modify the following line. The value should be higher than the sum of all LimitNOFILE values for all processes on your system. A common value for busy servers is 500,000 to 1,000,000.
```
    fs.file-max = 1048576 # Set to 1 million as an example
```
[!IMPORTANT] > This is a global kernel parameter. While a high value is generally safe on modern systems, ensure it's reasonable for your server's total workload.
1. Save and exit the editor.
1. Apply the changes immediately:
```
    sudo sysctl -p
```
1. Verify the new limit:
```
    cat /proc/sys/fs/file-max
```
5. Analyze and Optimize Application Behavior (Advanced)

While increasing limits provides immediate relief, addressing application-level issues is crucial for long-term stability and performance.
1. Review Application Code for Connection Leaks:
2. Ensure that database connections are properly closed after use, especially in loops or error handling blocks. Use try-finally constructs or similar mechanisms in your programming language.
1. Implement Connection Pooling:
2. For high-traffic applications, use a database connection pool (e.g., built into frameworks, or a dedicated proxy like ProxySQL for MySQL). Connection pooling reuses existing connections, significantly reducing the overhead of opening and closing connections and managing the total number of active connections more efficiently.
1. Optimize Slow Queries:
2. Long-running or inefficient queries hold connections open, contributing to the "too many connections" error.
3. * Enable the MySQL slow query log (slowquerylog = 1, longquerytime = 1) to identify problematic queries.
4. * Use EXPLAIN to analyze query execution plans and add appropriate indexes.
5. * Consider query caching or redesigning schema/queries.
1. Adjust waittimeout and interactivetimeout (MySQL):
2. These parameters in my.cnf define how long the server waits for activity on a non-interactive/interactive connection before closing it. Reducing these values (e.g., to 60 or 120 seconds) can help release idle connections faster, but exercise caution as too low a value can prematurely disconnect legitimate idle connections.
```
    [mysqld]
    # ...
    wait_timeout = 120
    interactive_timeout = 120
```
1. Traffic Management:
2. Implement load balancing (e.g., Nginx, HAProxy) to distribute traffic across multiple application servers, reducing the load on a single database. Consider rate limiting for potentially abusive traffic.
6. Monitoring and Further Tuning

Ongoing monitoring is essential to ensure the solution is effective and to identify future bottlenecks.
- Monitor Threadsconnected and Maxused_connections: Regularly check these status variables to understand your connection usage patterns.
- System Resource Monitoring: Keep an eye on CPU, RAM, and I/O utilization to ensure your server has enough resources. Tools like htop, atop, nmon, or dedicated monitoring solutions (e.g., Prometheus with Grafana, Percona Monitoring and Management) are invaluable.
- Database-specific Metrics: Monitor cache hit rates, query execution times, and other relevant database performance metrics.
By systematically working through these steps, you can effectively resolve MySQL Error 1040 and ensure your database server can handle its workload efficiently on Ubuntu 20.04 LTS.
July 11, 2026
GitHub Actions `checkout` Action Repository Authentication Failed on Ubuntu 22.04 LTS
Troubleshoot 'authentication failed' errors with `actions/checkout` in GitHub Actions on Ubuntu 22.04. Resolve common `GITHUB_TOKEN` permission issues and PAT misconfigurations.

When your GitHub Actions workflow fails during the actions/checkout step with an authentication error on an Ubuntu 22.04 LTS runner, it typically indicates that the workflow lacks the necessary permissions or the provided credentials (e.g., GITHUB_TOKEN or a Personal Access Token) are invalid or insufficient to access the repository. This issue can abruptly halt your CI/CD pipeline, preventing code deployment, testing, or build processes. This guide provides a detailed analysis and step-by-step resolution for these common authentication failures.

Symptom & Error Signature

The most common symptom is a workflow job failing at the actions/checkout step. You will observe error messages in the workflow run logs similar to these:
```
Run actions/checkout@v4
  with:
    repository: my-org/my-private-repo
    token: ***
    fetch-depth: 1
  env:
    # ...
    
Error: The process '/usr/bin/git' failed with exit code 128
Error: fatal: could not read Username for 'https://github.com': No such device or address
```
Or, a more direct authentication failure message:
```
Run actions/checkout@v4
  with:
    repository: my-org/my-private-repo
    token: ***
    fetch-depth: 1
  env:
    # ...
    
Error: The process '/usr/bin/git' failed with exit code 128
Error: fatal: Authentication failed for 'https://github.com/my-org/my-private-repo/'
```
These errors indicate that the Git client, running on the Ubuntu 22.04 LTS runner, was unable to authenticate successfully with GitHub to clone or fetch the repository.

Root Cause Analysis

The underlying reasons for actions/checkout authentication failures are typically tied to credential or permission misconfigurations within the GitHub Actions workflow context.
1. Insufficient GITHUBTOKEN Permissions: The default GITHUBTOKEN provided to each workflow run has limited permissions. If the workflow attempts to perform actions requiring elevated privileges (e.g., fetching a deep history (fetch-depth: 0) on a private repository, checking out specific branches, or pushing changes back to the repository), the default token might not suffice.
2. Incorrect token Parameter Usage:
3. * Sometimes, users attempt to use a custom Personal Access Token (PAT) but either misconfigure the token input for actions/checkout or provide an invalid secret name.
4. Using GITHUB_TOKEN directly for operations that it isn't designed for, such as checking out another* private repository.
5. Attempting to Check Out Another Private Repository: The GITHUBTOKEN is scoped only to the repository where the workflow is running. If your workflow needs to checkout a different private repository, the GITHUBTOKEN will always fail, and a Personal Access Token (PAT) with appropriate scopes is required.
6. Expired or Revoked Personal Access Token (PAT): If a custom PAT is used, it might have expired, been revoked, or its permissions (scopes) are insufficient for the required Git operations.
7. Repository Visibility/Existence: The target repository might be private, not exist, or the workflow's configured credentials simply do not have access to it due to organizational settings or repository deletion.
Step-by-Step Resolution

Follow these steps to diagnose and resolve the GitHub Actions checkout authentication failure.

1. Verify and Adjust Workflow permissions for GITHUB_TOKEN

The GITHUB_TOKEN is a temporary credential generated for each workflow run. By default, it has read-only access to contents and packages. For many checkout scenarios, particularly those involving private repositories or specific Git operations, you might need to explicitly grant additional permissions.
- Understanding GITHUB_TOKEN Permissions:
- * contents: read (default): Sufficient for cloning public repositories or your own private repository if you only need to read files.
- * contents: write: Required if your workflow needs to push branches, create tags, or perform git checkout operations that involve modifying the local repository state in a way that implies potential writes (e.g., fetch-depth: 0 on a private repo might sometimes require write permission for internal Git operations related to ref updates, though read is often sufficient for fetch-depth > 0).
To set or elevate permissions, add a permissions block at the workflow level or job level in your .github/workflows/*.yml file.

Example: Granting contents: read at the workflow level:
```
# .github/workflows/your-workflow.yml
```
on: [push, pull_request]

permissions: contents: read # Explicitly grant read access to repository contents

jobs: build: runs-on: ubuntu-22.04 steps: – name: Checkout repository uses: actions/checkout@v4 – name: Your build steps… run: | echo "Building…" `

Example: Granting contents: write (if needed) at the job level:
```
# .github/workflows/your-workflow.yml
```
on: push: branches: – main

jobs: update_docs: runs-on: ubuntu-22.04 permissions: contents: write # Grant write access for this job steps: – name: Checkout repository uses: actions/checkout@v4 # If pushing changes, ensure the token is correct for the push operation # e.g., git config user.name "github-actions[bot]" # git config user.email "41898282+github-actions[bot]@users.noreply.github.com" # git commit -am "Update documentation" # git push – name: Your steps that modify/push to the repository… run: | echo "Performing write operations…" `

[!IMPORTANT] Always grant the minimum necessary permissions. Giving contents: write when only read is needed increases the security risk if your workflow is compromised.

2. Ensure Correct token Parameter in actions/checkout

The actions/checkout action has a token input. It's crucial to use the correct secret here.
- Using the built-in GITHUB_TOKEN: This is the most secure and recommended approach for actions within the same repository.
```
    steps:
      - name: Checkout repository with GITHUB_TOKEN
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
```
[!NOTE] > While actions/checkout defaults to secrets.GITHUB_TOKEN if no token is specified, explicitly defining it can sometimes resolve subtle issues or improve clarity.
- Using a Custom Personal Access Token (PAT): If GITHUBTOKEN is insufficient (e.g., for cross-repository access or highly specific permissions not covered by GITHUBTOKEN), you'll need to create a PAT, add it to your repository secrets, and then reference it.
```
    steps:
      - name: Checkout repository with custom PAT
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.MY_REPO_PAT }} # Assuming MY_REPO_PAT is your secret name
```
[!WARNING] > Never hardcode PATs directly in your workflow file. Always store them as GitHub repository or organization secrets.

3. Handling Cross-Repository Checkout for Private Repositories

If you are trying to check out a different private repository than the one the workflow is running in, the GITHUB_TOKEN will not work. You must use a Personal Access Token (PAT) that has access to the target repository.

Steps:
1. Create a Fine-Grained Personal Access Token (Recommended):
2. * Navigate to your GitHub profile settings -> Developer settings -> Personal access tokens -> Fine-grained tokens.
3. * Click "Generate new token".
4. * Give it a descriptive name (e.g., ci-checkout-other-repo).
5. * Set an expiration date.
6. * Under "Resource owner", select your user or organization.
7. Under "Repository access", choose "Only select repositories" and add the target* private repository you wish to check out.
8. * Under "Repository permissions" for the selected target repository, grant Contents: Read permission (or Contents: Write if your workflow needs to push to it).
9. * Generate the token and copy it immediately as it will not be shown again.
1. Add PAT to GitHub Secrets:
2. * Go to your workflow repository -> Settings -> Secrets and variables -> Actions.
3. * Click "New repository secret".
4. * Name it (e.g., CROSSREPOPAT) and paste the PAT you just generated into the "Secret value" field.
1. Use the PAT in your Workflow:
```
    # .github/workflows/cross-repo-checkout.yml
```
on: [push]

jobs: fetchotherrepo: runs-on: ubuntu-22.04 steps: – name: Checkout current repository (optional) uses: actions/checkout@v4 # No token needed if this is the current repo, or use ${{ secrets.GITHUB_TOKEN }}
- name: Checkout another private repository
- uses: actions/checkout@v4
- with:
- repository: my-org/another-private-repo # Specify the target private repo
- token: ${{ secrets.CROSSREPOPAT }} # Use the PAT with access to 'another-private-repo'
- path: another-repo-dir # Optional: checkout into a specific directory
- – name: List contents of the other repo
- run: |
- ls -la another-repo-dir
- `
[!IMPORTANT] > Ensure the CROSSREPOPAT has access to my-org/another-private-repo specifically. If you use a classic PAT, it might need the full repo scope, which is less secure than fine-grained tokens.

4. Check Personal Access Token (PAT) Validity and Scope

If you are using a custom PAT, verify its status and permissions:
1. Access PATs: Go to your GitHub profile settings -> Developer settings -> Personal access tokens.
2. * For Fine-grained tokens: Check the list under "Fine-grained tokens".
3. * For Classic tokens: Check the list under "Tokens (classic)".
4. Review Expiration & Revocation: Ensure the PAT has not expired or been revoked. If it has, generate a new one.
5. Verify Scopes (Classic PATs) / Permissions (Fine-grained PATs):
6. * For Classic PATs: The PAT should have at least the repo scope to access private repositories. For public repositories, public_repo might suffice.
7. * For Fine-grained PATs: Confirm that the token explicitly grants Contents: Read (or Write if needed) permission for the specific repository you are trying to check out.
If any of these are incorrect, update the PAT or generate a new one and update the corresponding secret in GitHub Actions.

5. Confirm Repository Accessibility and Existence

Though less common for an authentication failure specifically, it's a fundamental check:
1. Does the repository exist? Double-check the repository: path in your actions/checkout step (e.g., my-org/my-private-repo). A typo here will lead to access errors.
2. Is the repository private, and do the credentials have access? If the repository is truly private, ensure that the GITHUB_TOKEN (with appropriate permissions) or the custom PAT has explicit permission to access it. If the repository was recently made private or moved, permissions might need re-evaluation.
By systematically going through these steps, you should be able to identify and resolve the GitHub Actions checkout action repository authentication failed error, restoring your CI/CD pipeline's functionality.
July 9, 2026