Caddy Reverse Proxy: Resolving TLS Handshake Failed Verification (x509) on Windows WSL2 Ubuntu

Troubleshoot 'TLS handshake failed verification' when Caddy in WSL2 acts as a reverse proxy. Learn to manage certificate trust for backend services.

Caddy Reverse Proxy: Resolving TLS Handshake Failed Verification (x509) on Windows WSL2 Ubuntu

When operating Caddy as a reverse proxy within a Windows Subsystem for Linux 2 (WSL2) Ubuntu environment, you might encounter errors related to TLS handshake verification. This typically happens when Caddy attempts to establish a secure connection to your backend service (e.g., a web application, API, or another internal server) but fails to validate the backend's TLS certificate. The result is an inability for Caddy to proxy requests, leading to unresponsive services or error pages for your users.

This guide will walk you through diagnosing and resolving the common causes of "TLS handshake failed verification" errors, particularly focusing on certificate trust issues within the WSL2 environment.

Symptom & Error Signature

Users attempting to access your service through Caddy will likely see generic browser errors such as NET::ERRCERTINVALID, ERRSSLPROTOCOL_ERROR, or 502 Bad Gateway if Caddy cannot successfully connect to the backend.

The most telling sign will be in your Caddy logs. You'll observe entries similar to these, indicating Caddy's struggle to trust the backend certificate:

ERROR   http.log.error  reverse proxy: selected backend is unhealthy: TLS handshake failed: remote error: tls: handshake failure
ERROR   http.log.error  reverse proxy: x509: certificate signed by unknown authority
ERROR   http.log.error  reverse proxy: tls: failed to verify certificate: x509: certificate signed by unknown authority
ERROR   http.log.error  reverse proxy: dial tcp 172.17.0.2:443: connect: connection refused

The specific messages x509: certificate signed by unknown authority or tls: failed to verify certificate are key indicators of a certificate trust problem.

Root Cause Analysis

The "TLS handshake failed verification" error, especially with x509: certificate signed by unknown authority, means that Caddy, acting as a client to your backend server, received a certificate but couldn't validate its authenticity using its list of trusted Certificate Authorities (CAs). This usually stems from one or more of the following:

1. Self-Signed Certificates: The backend server is using a self-signed TLS certificate. These certificates are not issued by a publicly trusted CA and therefore are inherently untrusted by default.
2. Internal CA Certificates: The backend server's certificate is issued by an internal or private Certificate Authority (CA) specific to your organization or development setup. WSL2's Ubuntu instance, by default, only trusts widely recognized public CAs.
3. Missing Intermediate Certificates: The backend server might be serving its certificate without providing the full chain of intermediate certificates, preventing Caddy from building a trusted path to a known root CA.
4. Incorrect Caddy Configuration: The reverse_proxy directive might be pointing to the wrong IP address or port, or is missing necessary TLS configuration options.
5. Network/Firewall Issues: Though less likely to cause a "TLS handshake failed verification" specifically (which implies a connection was made and a certificate received), underlying network connectivity issues or firewall blocks could result in similar high-level errors like 502 Bad Gateway and precede this specific TLS error, showing as connection refused initially.
6. Expired or Invalid Certificates: The backend's certificate might be expired, revoked, or have a hostname mismatch, leading to validation failure.

The most prevalent causes for WSL2 scenarios involving internal services are self-signed or internal CA certificates, as WSL2's Ubuntu environment needs explicit instruction to trust them.

Step-by-Step Resolution

Follow these steps to diagnose and resolve the TLS handshake verification error.

1. Verify Caddy Configuration

First, ensure your Caddyfile's reverse_proxy directive is correctly configured to point to your backend service's address and port.

yourdomain.com {
    reverse_proxy https://your_backend_ip_or_hostname:port {
        # Other directives if needed
    }
}

Make sure you're using https:// if your backend is serving over TLS, which is the assumption for this error.

2. Inspect Caddy Logs for Detailed Error Messages

The Caddy logs are your primary diagnostic tool. If Caddy is running as a systemd service, you can retrieve logs using journalctl.

sudo journalctl -u caddy -f --since "5 minutes ago"

Look for the exact error message string. x509: certificate signed by unknown authority is a strong indicator that Caddy does not trust the CA that signed your backend's certificate.

3. Determine Your Backend Certificate's Origin

You need to know if your backend's TLS certificate is: * Issued by a publicly trusted CA (e.g., Let's Encrypt, DigiCert). * Self-signed. * Issued by an internal/private CA.

You can inspect the certificate directly from WSL2 using openssl. Replace yourbackendiporhostname:port with your actual backend's address.

openssl s_client -connect your_backend_ip_or_hostname:port -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep "Issuer:|Subject:|Not Before:|Not After:"

Pay close attention to the Issuer field. If it's your own company, a custom name, or the same as Subject, it's likely a self-signed or internal CA certificate.

4. Add the CA Certificate to WSL2's Trust Store (Recommended)

This is the most robust solution for internal or self-signed certificates. You need to obtain the root or intermediate CA certificate that signed your backend's certificate and add it to your WSL2 Ubuntu instance's system-wide trust store.

a. Obtain the CA Certificate File

You might receive this file from your internal IT department, or you can extract it if you have access to the backend server. If you only have the backend server's leaf certificate, you'll need the CA certificate that signed it.

[!IMPORTANT] The certificate file must be in PEM format (base64-encoded ASCII with -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- headers) and typically have a .crt extension.

If you don't have the .crt file directly, you can often extract it from the backend server's certificate chain. For example, if you can curl your backend (even if it fails validation, you might get the cert):

# This command tries to fetch the certificate chain from your backend
# and save it as backend_chain.pem. You'll then need to identify
# and extract the CA cert from this chain.
openssl s_client -showcerts -verify 5 -connect your_backend_ip_or_hostname:port < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ print $0 }' > backend_chain.pem
```

b. Install the CA Certificate in WSL2 Ubuntu

1. Copy your .crt file (e.g., my-internal-ca.crt) into the /usr/local/share/ca-certificates/ directory within your WSL2 Ubuntu instance.

    sudo cp /path/to/my-internal-ca.crt /usr/local/share/ca-certificates/

1. Update the system's CA certificate store.

    sudo update-ca-certificates

You should see output similar to Adding new cert into /etc/ssl/certs/ and 1 added, 0 removed; done..

1. Restart Caddy to pick up the new trust store.

    sudo systemctl restart caddy

Check Caddy logs again (sudo journalctl -u caddy -f) to ensure the error is resolved.

5. Configure Caddy with tlstrustedca_certs (Alternative/Specific)

If you prefer not to add the CA certificate system-wide, or if you need more granular control, you can specify the trusted CA certificate directly in your Caddyfile for a specific reverse_proxy block.

yourdomain.com {
    reverse_proxy https://your_backend_ip_or_hostname:port {
        tls_trusted_ca_certs /path/to/my-internal-ca.crt
    }
}

[!NOTE] Ensure the /path/to/my-internal-ca.crt is accessible by the Caddy user and specifies the path within your WSL2 filesystem.

After modifying your Caddyfile, reload Caddy:

sudo systemctl reload caddy
# Or if reload fails to pick up all changes:
sudo systemctl restart caddy

6. Temporarily Skip TLS Verification (Development/Last Resort)

[!WARNING] Using tlsinsecureskip_verify is highly discouraged in production environments. It completely disables TLS certificate validation, making your connection vulnerable to Man-in-the-Middle (MITM) attacks and compromising the security of your data. Only use this for development or debugging where security implications are fully understood and accepted.

If you're in a development environment and absolutely need to proceed without certificate validation, you can instruct Caddy to skip verification:

yourdomain.com {
    reverse_proxy https://your_backend_ip_or_hostname:port {
        tls_insecure_skip_verify
    }
}

Reload or restart Caddy after this change. Remove this directive as soon as you implement a proper certificate trust solution.

7. Verify Network Connectivity and DNS Resolution

If you're still facing issues, or if Caddy logs initially showed connection refused before TLS handshake failed, verify network connectivity from your WSL2 instance to the backend.

1. Ping the backend:

    ping your_backend_ip_or_hostname

1. Test direct HTTPS connection with curl from WSL2:

    curl -v https://your_backend_ip_or_hostname:port/your_health_check_path
    ```

1. Check DNS Resolution: If using a hostname, ensure WSL2 can resolve it. Add an entry to /etc/hosts if it's an internal hostname not resolved by your DNS server.

    # Example /etc/hosts entry
    192.168.1.100   your_backend_hostname

8. Check Backend Certificate Expiry and Hostname

Ensure the backend server's certificate is not expired and that its Subject Alternative Name (SAN) or Common Name (CN) matches the hostname or IP address Caddy is using to connect.

openssl s_client -connect your_backend_ip_or_hostname:port -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -E "Subject Alternative Name|Not Before:|Not After:"

If the certificate is expired or the hostname doesn't match, you'll need to renew or re-issue the backend's certificate.

By systematically following these steps, you should be able to diagnose and resolve the "TLS handshake failed verification" error with Caddy acting as a reverse proxy in your Windows WSL2 Ubuntu environment. Prioritize adding trusted CA certificates to the system store for a secure and robust solution.