{"id":13,"date":"2026-07-16T00:00:00","date_gmt":"2026-07-16T00:00:00","guid":{"rendered":"http:\/\/\/var\/www\/html\/?p=13"},"modified":"2026-07-21T01:02:30","modified_gmt":"2026-07-21T01:02:30","slug":"caddy-reverse-proxy-tls-handshake-failed-verification-error-on-ubuntu-2004-lts","status":"publish","type":"post","link":"https:\/\/butitworkedlocal.com\/caddy-reverse-proxy-tls-handshake-failed-verification-error-on-ubuntu-2004-lts\/","title":{"rendered":"Troubleshooting Caddy Reverse Proxy TLS Handshake Failed Verification on Ubuntu 20.04 LTS"},"content":{"rendered":"\n

Resolve Caddy reverse proxy TLS handshake failures with upstream servers on Ubuntu 20.04 LTS. Diagnose certificate validation, SNI, and CA issues for seamless reverse proxy operations.<\/strong><\/p>\n\n\n

Introduction<\/h2>\n

Caddy has established itself as a powerful, easy-to-use web server and reverse proxy, renowned for its automatic HTTPS capabilities. However, when operating Caddy as a reverse proxy, you might encounter situations where it fails to establish a secure (TLS) connection with its upstream backend server. A "TLS handshake failed verification" error signifies that Caddy initiated a TLS connection but was unable to validate the certificate presented by the upstream server. This guide provides a highly technical, step-by-step approach to diagnose and resolve this issue specifically on Ubuntu 20.04 LTS environments.<\/p>\n

### Symptom & Error Signature<\/h2>\n

When Caddy experiences a TLS handshake failure with an upstream, client requests attempting to reach the proxied service will typically receive a 502 Bad Gateway<\/code> error. In the Caddy server logs, you'll find entries indicating certificate validation problems.<\/p>\n

To view Caddy's logs, use journalctl<\/code>:<\/p>\n

sudo journalctl -u caddy -f --no-hostname\n<\/code><\/pre>\n
Typical error signatures you might observe include:<\/p>\n
{\n  "level": "error",\n  "ts": 1678886400.000,\n  "logger": "http.log.error",\n  "msg": "x509: certificate signed by unknown authority",\n  "trace": "...",\n  "request": {\n    "method": "GET",\n    "uri": "\/api\/data",\n    "proto": "HTTP\/2.0",\n    "remote_addr": "192.168.1.100:54321",\n    "host": "your-caddy-domain.com",\n    "headers": {\n      "User-Agent": ["Mozilla\/5.0 ..."],\n      "Accept": ["*\/*"]\n    }\n  },\n  "error": "reverse proxy: selected upstream is unhealthy: connection to upstream timed out after 10s"\n}\n<\/code><\/pre>\n
Or a more generic handshake failure:<\/p>\n
{\n  "level": "error",\n  "ts": 1678886405.000,\n  "logger": "http.handlers.reverse_proxy",\n  "msg": "aborting with incomplete response",\n  "error": "remote error: tls: handshake failure",\n  "duration": 0.0001\n}\n<\/code><\/pre>\n
Or an SNI related error:<\/p>\n
{\n  "level": "error",\n  "ts": 1678886410.000,\n  "logger": "http.log.error",\n  "msg": "tls: failed to verify certificate: x509: certificate is valid for example.com, not your-upstream-hostname.internal",\n  "trace": "...",\n  "error": "reverse proxy: selected upstream is unhealthy: connection to upstream timed out after 10s"\n}\n<\/code><\/pre>\n
### Root Cause Analysis<\/h2>\n
The "TLS handshake failed verification" error fundamentally means that Caddy, acting as a TLS client to the upstream server, could not trust the certificate presented by that server. The most common underlying reasons for this include:<\/p>\n
    \n
  1. Untrusted Certificate Authority (CA)<\/strong>: The upstream server's certificate is signed by a CA that is not present or trusted in Caddy's (or the underlying system's) trust store. This is common with self-signed certificates, private enterprise CAs, or newly issued CA roots not yet widely distributed.<\/li>\n
  2. Certificate Name Mismatch (SNI\/SAN)<\/strong>:
      \n
    • Common Name (CN) \/ Subject Alternative Name (SAN) Mismatch<\/strong>: The hostname Caddy is attempting to connect to (or the Host<\/code> header it's sending) does not match the CN or SANs listed within the upstream server's certificate.<\/li>\n
    • Incorrect Server Name Indication (SNI)<\/strong>: Caddy might not be sending the correct SNI hostname during the TLS handshake, causing the upstream server (especially if it hosts multiple virtual hosts with different certificates) to present the wrong certificate or refuse the connection.<\/li>\n<\/ul>\n<\/li>\n
    • Expired or Invalid Certificate<\/strong>: The upstream server's certificate has expired, is not yet valid, or has been revoked.<\/li>\n
    • Network\/Firewall Issues<\/strong>: While less direct for "verification failure," network problems or restrictive firewalls between Caddy and the upstream could interrupt the TLS handshake process, leading to a perceived failure.<\/li>\n
    • System Time Skew<\/strong>: Significant clock drift on either the Caddy server or the upstream server can cause certificate validity checks (e.g., notBefore<\/code>, notAfter<\/code> dates) to fail.<\/li>\n<\/ol>\n
      ### Step-by-Step Resolution<\/h2>\n
      Follow these steps to diagnose and resolve the TLS handshake verification error.<\/p>\n
      1. Verify Upstream Server's Certificate Directly<\/h4>\n
      Before troubleshooting Caddy, confirm the upstream server's certificate is valid and correctly configured from the perspective of the Caddy server's host.<\/p>\n
      # Replace 'your_upstream_hostname' with the actual hostname Caddy connects to\n# Replace 'upstream_ip_address' with the upstream server's IP\n# Replace 'upstream_port' with the port the upstream listens on (e.g., 443, 8443)\nopenssl s_client -connect your_upstream_ip_address:upstream_port -servername your_upstream_hostname <\/dev\/null\n<\/code><\/pre>\n
      Inspect the output carefully:<\/p>\n
        \n
      • Verify return code: 0 (ok)<\/code><\/strong>: Indicates the certificate chain is trusted by the system's CA store. If you see anything else (e.g., 20 (unable to get local issuer certificate)<\/code>, 21 (unable to verify the first certificate)<\/code>), it points to an untrusted CA.<\/li>\n
      • Not Before<\/code> \/ Not After<\/code><\/strong>: Check if the certificate is within its valid date range.<\/li>\n
      • Subject Common Name (CN)<\/code> \/ Subject Alternative Name (SAN)<\/code><\/strong>: Ensure your_upstream_hostname<\/code> is listed here. If not, you have a hostname mismatch.<\/li>\n
      • depth<\/code> and issuer<\/code><\/strong>: Review the certificate chain to identify the issuing CA.<\/li>\n<\/ul>\n
        2. Update System CA Trust Store (for Untrusted CAs)<\/h4>\n
        If openssl s_client<\/code> returned an error indicating an untrusted issuer (e.g., Verify return code: 20<\/code>), you need to add the upstream's CA certificate to Caddy's system trust store. This is common for self-signed certificates or private enterprise CAs.<\/p>\n
          \n
        1. Obtain the CA Certificate<\/strong>:\nIf you have the upstream CA's .crt<\/code> file, copy it. If not, you can often extract it using openssl<\/code>:<\/p>\n
          # Connect to the upstream and extract the full certificate chain\nopenssl s_client -showcerts -connect your_upstream_ip_address:upstream_port -servername your_upstream_hostname <\/dev\/null > upstream_chain.pem\n\n# Open 'upstream_chain.pem' and identify the root CA certificate.\n# It's usually the last certificate in the chain and self-signed (Issuer and Subject are the same).\n# Copy its content (including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----)\n# into a new file, e.g., 'upstream_ca.crt'.\n<\/code><\/pre>\n<\/li>\n
        2. Add the CA Certificate to System Trust<\/strong>:<\/p>\n
          sudo cp \/path\/to\/upstream_ca.crt \/usr\/local\/share\/ca-certificates\/\nsudo update-ca-certificates\n<\/code><\/pre>\n
          You should see output similar to: Updating certificates in \/etc\/ssl\/certs... 1 added, 0 removed; done.<\/code><\/p>\n<\/li>\n
        3. Restart Caddy<\/strong>:<\/p>\n
          sudo systemctl restart caddy\nsudo journalctl -u caddy -f --no-hostname # Check logs for success\n<\/code><\/pre>\n<\/li>\n<\/ol>\n
          Ensure the CA certificate has a .crt<\/code> extension when copied to \/usr\/local\/share\/ca-certificates\/<\/code>. The update-ca-certificates<\/code> utility only processes files with this extension in that directory by default.<\/p>\n<\/blockquote>\n
          3. Configure Caddy for Specific Upstream TLS Trust (Caddyfile Directive)<\/h4>\n
          For more granular control, especially if you only want Caddy to trust a specific custom CA for a particular upstream and not globally, you can configure this directly in your Caddyfile<\/code>.<\/p>\n
            \n
          1. Place CA Certificate<\/strong>: Store the upstream CA certificate file (e.g., upstream_ca.crt<\/code>) in a secure, Caddy-readable location, typically \/etc\/caddy\/certs\/<\/code>.<\/p>\n<\/li>\n
          2. Modify Caddyfile<\/strong>:\nAdd or modify the transport<\/code> block within your reverse_proxy<\/code> directive:<\/p>\n
            your-caddy-domain.com {\n    reverse_proxy your_upstream_ip_address:upstream_port {\n        transport http {\n            # Trusts the specified CA certificate for this upstream only\n            tls_trusted_ca \/etc\/caddy\/certs\/upstream_ca.crt\n            # Optional: specify the server name if different from the Caddy domain\n            # This ensures the correct certificate is requested via SNI\n            tls_server_name your_upstream_hostname\n        }\n    }\n}\n<\/code><\/pre>\n<\/li>\n
          3. Validate and Apply Caddyfile Changes<\/strong>:<\/p>\n
            sudo caddy validate --config \/etc\/caddy\/Caddyfile\nsudo systemctl reload caddy\nsudo journalctl -u caddy -f --no-hostname # Monitor logs\n<\/code><\/pre>\n<\/li>\n<\/ol>\n
            4. Address Hostname\/SNI Mismatch<\/h4>\n
            If openssl s_client<\/code> revealed a hostname mismatch (CN\/SAN mismatch) or if your upstream requires a specific SNI hostname to present the correct certificate:<\/p>\n
              \n
            1. Caddyfile reverse_proxy<\/code> Configuration<\/strong>:\nEnsure Caddy is sending the correct Host<\/code> header and tls_server_name<\/code> (SNI) for the upstream.<\/p>\n
              your-caddy-domain.com {\n    reverse_proxy your_upstream_ip_address:upstream_port {\n        # This sets the Host header sent to the upstream\n        # Crucial if upstream expects a specific hostname\n        header_up Host your_upstream_hostname\n\n        transport http {\n            # This sets the SNI hostname for the TLS handshake\n            # Required if the upstream server serves multiple certificates\n            tls_server_name your_upstream_hostname\n        }\n    }\n}\n<\/code><\/pre>\n
              Replace your_upstream_hostname<\/code> with the actual hostname that the upstream's certificate is valid for. This is often the internal FQDN of the backend server.<\/p>\n<\/li>\n
            2. Validate and Apply Caddyfile Changes<\/strong>:<\/p>\n
              sudo caddy validate --config \/etc\/caddy\/Caddyfile\nsudo systemctl reload caddy\nsudo journalctl -u caddy -f --no-hostname\n<\/code><\/pre>\n<\/li>\n<\/ol>\n
              5. Verify System Time Synchronization<\/h4>\n
              Significant time differences can cause certificates to appear invalid. Ensure both Caddy and the upstream server are synchronized with an NTP server.<\/p>\n
              timedatectl\n<\/code><\/pre>\n
              If time is unsynchronized, ensure systemd-timesyncd<\/code> is enabled or install ntp<\/code>:<\/p>\n
              sudo systemctl enable --now systemd-timesyncd\n# Or for legacy NTP daemon:\n# sudo apt update && sudo apt install ntp\n# sudo systemctl enable --now ntp\n<\/code><\/pre>\n
              6. Bypass TLS Verification (Development\/Testing ONLY)<\/h4>\n
              This step disables all TLS certificate verification<\/strong>. It should NEVER<\/strong> be used in production environments as it makes your system vulnerable to Man-in-the-Middle attacks. Use this only<\/em> for debugging in isolated development environments or if you fully understand and accept the security implications.<\/p>\n<\/blockquote>\n
              If all else fails and you need to quickly confirm if TLS verification is the root cause, you can temporarily disable it:<\/p>\n
              your-caddy-domain.com {\n    reverse_proxy your_upstream_ip_address:upstream_port {\n        transport http {\n            tls_insecure_skip_verify\n            # Optionally, still specify server name for SNI\n            # tls_server_name your_upstream_hostname\n        }\n    }\n}\n<\/code><\/pre>\n
              Validate and apply, then immediately remove this directive once you've confirmed the issue.<\/p>\n
              7. Check Upstream Server's Certificate Expiration\/Validity<\/h4>\n
              If openssl s_client<\/code> revealed an expired or not-yet-valid certificate on the upstream, the solution lies with the upstream server's administrator. They need to renew or correctly install a valid certificate. Caddy cannot fix an invalid certificate presented by its backend.<\/p>\n
              This comprehensive guide should help you systematically troubleshoot and resolve the "Caddy reverse proxy TLS handshake failed verification" error on your Ubuntu 20.04 LTS system. Remember to always apply the least permissive fix possible for security.<\/p>\n","protected":false},"excerpt":{"rendered":"
              Resolve Caddy reverse proxy TLS handshake failures with upstream servers on Ubuntu 20.04 LTS. Diagnose certificate validation, SNI, and CA issues for seamless reverse proxy operations.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[36,41,40,37,27,28,13],"class_list":["post-13","post","type-post","status-publish","format-standard","hentry","category-web-server","tag-caddy","tag-certificate-validation","tag-devops","tag-reverse-proxy","tag-ssl","tag-tls","tag-ubuntu"],"_links":{"self":[{"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/posts\/13","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/comments?post=13"}],"version-history":[{"count":2,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/posts\/13\/revisions"}],"predecessor-version":[{"id":261,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/posts\/13\/revisions\/261"}],"wp:attachment":[{"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/media?parent=13"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/categories?post=13"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/butitworkedlocal.com\/wp-json\/wp\/v2\/tags?post=13"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}