Resolve Caddy reverse proxy TLS handshake failures with upstream servers on Ubuntu 20.04 LTS. Diagnose certificate validation, SNI, and CA issues for seamless reverse proxy operations.

Introduction

Caddy has established itself as a powerful, easy-to-use web server and reverse proxy, renowned for its automatic HTTPS capabilities. However, when operating Caddy as a reverse proxy, you might encounter situations where it fails to establish a secure (TLS) connection with its upstream backend server. A "TLS handshake failed verification" error signifies that Caddy initiated a TLS connection but was unable to validate the certificate presented by the upstream server. This guide provides a highly technical, step-by-step approach to diagnose and resolve this issue specifically on Ubuntu 20.04 LTS environments.

### Symptom & Error Signature

When Caddy experiences a TLS handshake failure with an upstream, client requests attempting to reach the proxied service will typically receive a 502 Bad Gateway error. In the Caddy server logs, you'll find entries indicating certificate validation problems.

To view Caddy's logs, use journalctl:

sudo journalctl -u caddy -f --no-hostname

Typical error signatures you might observe include:

{
  "level": "error",
  "ts": 1678886400.000,
  "logger": "http.log.error",
  "msg": "x509: certificate signed by unknown authority",
  "trace": "...",
  "request": {
    "method": "GET",
    "uri": "/api/data",
    "proto": "HTTP/2.0",
    "remote_addr": "192.168.1.100:54321",
    "host": "your-caddy-domain.com",
    "headers": {
      "User-Agent": ["Mozilla/5.0 ..."],
      "Accept": ["*/*"]
    }
  },
  "error": "reverse proxy: selected upstream is unhealthy: connection to upstream timed out after 10s"
}

Or a more generic handshake failure:

{
  "level": "error",
  "ts": 1678886405.000,
  "logger": "http.handlers.reverse_proxy",
  "msg": "aborting with incomplete response",
  "error": "remote error: tls: handshake failure",
  "duration": 0.0001
}

Or an SNI related error:

{
  "level": "error",
  "ts": 1678886410.000,
  "logger": "http.log.error",
  "msg": "tls: failed to verify certificate: x509: certificate is valid for example.com, not your-upstream-hostname.internal",
  "trace": "...",
  "error": "reverse proxy: selected upstream is unhealthy: connection to upstream timed out after 10s"
}

### Root Cause Analysis

The "TLS handshake failed verification" error fundamentally means that Caddy, acting as a TLS client to the upstream server, could not trust the certificate presented by that server. The most common underlying reasons for this include:

1. Untrusted Certificate Authority (CA): The upstream server's certificate is signed by a CA that is not present or trusted in Caddy's (or the underlying system's) trust store. This is common with self-signed certificates, private enterprise CAs, or newly issued CA roots not yet widely distributed.
2. Certificate Name Mismatch (SNI/SAN):
3. * Common Name (CN) / Subject Alternative Name (SAN) Mismatch: The hostname Caddy is attempting to connect to (or the Host header it's sending) does not match the CN or SANs listed within the upstream server's certificate.
4. * Incorrect Server Name Indication (SNI): Caddy might not be sending the correct SNI hostname during the TLS handshake, causing the upstream server (especially if it hosts multiple virtual hosts with different certificates) to present the wrong certificate or refuse the connection.
5. Expired or Invalid Certificate: The upstream server's certificate has expired, is not yet valid, or has been revoked.
6. Network/Firewall Issues: While less direct for "verification failure," network problems or restrictive firewalls between Caddy and the upstream could interrupt the TLS handshake process, leading to a perceived failure.
7. System Time Skew: Significant clock drift on either the Caddy server or the upstream server can cause certificate validity checks (e.g., notBefore, notAfter dates) to fail.

### Step-by-Step Resolution

Follow these steps to diagnose and resolve the TLS handshake verification error.

1. Verify Upstream Server's Certificate Directly

Before troubleshooting Caddy, confirm the upstream server's certificate is valid and correctly configured from the perspective of the Caddy server's host.

# Replace 'your_upstream_hostname' with the actual hostname Caddy connects to
# Replace 'upstream_ip_address' with the upstream server's IP
# Replace 'upstream_port' with the port the upstream listens on (e.g., 443, 8443)
openssl s_client -connect your_upstream_ip_address:upstream_port -servername your_upstream_hostname </dev/null

Inspect the output carefully: * Verify return code: 0 (ok): Indicates the certificate chain is trusted by the system's CA store. If you see anything else (e.g., 20 (unable to get local issuer certificate), 21 (unable to verify the first certificate)), it points to an untrusted CA. * Not Before / Not After: Check if the certificate is within its valid date range. * Subject Common Name (CN) / Subject Alternative Name (SAN): Ensure yourupstreamhostname is listed here. If not, you have a hostname mismatch. * depth and issuer: Review the certificate chain to identify the issuing CA.

2. Update System CA Trust Store (for Untrusted CAs)

If openssl s_client returned an error indicating an untrusted issuer (e.g., Verify return code: 20), you need to add the upstream's CA certificate to Caddy's system trust store. This is common for self-signed certificates or private enterprise CAs.

1. Obtain the CA Certificate:
2. If you have the upstream CA's .crt file, copy it. If not, you can often extract it using openssl:

    # Connect to the upstream and extract the full certificate chain

Open 'upstream_chain.pem' and identify the root CA certificate. # It's usually the last certificate in the chain and self-signed (Issuer and Subject are the same). # Copy its content (including —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–) # into a new file, e.g., 'upstream_ca.crt'. `

1. Add the CA Certificate to System Trust:

    sudo cp /path/to/upstream_ca.crt /usr/local/share/ca-certificates/
    sudo update-ca-certificates

You should see output similar to: Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.

1. Restart Caddy:

    sudo systemctl restart caddy
    sudo journalctl -u caddy -f --no-hostname # Check logs for success

[!IMPORTANT] Ensure the CA certificate has a .crt extension when copied to /usr/local/share/ca-certificates/. The update-ca-certificates utility only processes files with this extension in that directory by default.

3. Configure Caddy for Specific Upstream TLS Trust (Caddyfile Directive)

For more granular control, especially if you only want Caddy to trust a specific custom CA for a particular upstream and not globally, you can configure this directly in your Caddyfile.

1. Place CA Certificate: Store the upstream CA certificate file (e.g., upstream_ca.crt) in a secure, Caddy-readable location, typically /etc/caddy/certs/.

1. Modify Caddyfile:
2. Add or modify the transport block within your reverse_proxy directive:

    your-caddy-domain.com {
        reverse_proxy your_upstream_ip_address:upstream_port {
            transport http {
                # Trusts the specified CA certificate for this upstream only
                tls_trusted_ca /etc/caddy/certs/upstream_ca.crt
                # Optional: specify the server name if different from the Caddy domain
                # This ensures the correct certificate is requested via SNI
                tls_server_name your_upstream_hostname
            }
        }
    }

1. Validate and Apply Caddyfile Changes:

    sudo caddy validate --config /etc/caddy/Caddyfile
    sudo systemctl reload caddy
    sudo journalctl -u caddy -f --no-hostname # Monitor logs

4. Address Hostname/SNI Mismatch

If openssl s_client revealed a hostname mismatch (CN/SAN mismatch) or if your upstream requires a specific SNI hostname to present the correct certificate:

1. Caddyfile reverse_proxy Configuration:
2. Ensure Caddy is sending the correct Host header and tlsservername (SNI) for the upstream.

    your-caddy-domain.com {
        reverse_proxy your_upstream_ip_address:upstream_port {
            # This sets the Host header sent to the upstream
            # Crucial if upstream expects a specific hostname

transport http { # This sets the SNI hostname for the TLS handshake # Required if the upstream server serves multiple certificates tlsservername yourupstreamhostname } } } ` Replace yourupstreamhostname with the actual hostname that the upstream's certificate is valid for. This is often the internal FQDN of the backend server.

1. Validate and Apply Caddyfile Changes:

    sudo caddy validate --config /etc/caddy/Caddyfile
    sudo systemctl reload caddy
    sudo journalctl -u caddy -f --no-hostname

5. Verify System Time Synchronization

Significant time differences can cause certificates to appear invalid. Ensure both Caddy and the upstream server are synchronized with an NTP server.

timedatectl

If time is unsynchronized, ensure systemd-timesyncd is enabled or install ntp:

sudo systemctl enable --now systemd-timesyncd
# Or for legacy NTP daemon:
# sudo apt update && sudo apt install ntp
# sudo systemctl enable --now ntp

6. Bypass TLS Verification (Development/Testing ONLY)

[!WARNING] This step disables all TLS certificate verification. It should NEVER be used in production environments as it makes your system vulnerable to Man-in-the-Middle attacks. Use this only for debugging in isolated development environments or if you fully understand and accept the security implications.

If all else fails and you need to quickly confirm if TLS verification is the root cause, you can temporarily disable it:

your-caddy-domain.com {
    reverse_proxy your_upstream_ip_address:upstream_port {
        transport http {
            tls_insecure_skip_verify
            # Optionally, still specify server name for SNI
            # tls_server_name your_upstream_hostname
        }
    }
}

Validate and apply, then immediately remove this directive once you've confirmed the issue.

7. Check Upstream Server's Certificate Expiration/Validity

If openssl s_client revealed an expired or not-yet-valid certificate on the upstream, the solution lies with the upstream server's administrator. They need to renew or correctly install a valid certificate. Caddy cannot fix an invalid certificate presented by its backend.

This comprehensive guide should help you systematically troubleshoot and resolve the "Caddy reverse proxy TLS handshake failed verification" error on your Ubuntu 20.04 LTS system. Remember to always apply the least permissive fix possible for security.