Tag: certificate-validation

  • Fixing OpenSSL Self-Signed Certificate Chain Validation Errors on macOS Local Environments

    Resolve OpenSSL validation issues with self-signed certificates on macOS. Learn to trust local development certificates in Keychain Access and for various tools.

    Introduction

    Developing locally on macOS often involves interacting with services that use self-signed SSL certificates. Whether it's a backend API, a local database, or a custom microservice running in Docker, these certificates are not inherently trusted by your operating system or development tools. This leads to frustrating "certificate validation failed" errors when your applications or curl commands try to establish secure connections. This guide will walk you through the precise steps to properly trust these certificates on macOS, ensuring your local development environment runs smoothly.

    Symptom & Error Signature

    You will typically encounter errors when attempting to connect to your local HTTPS service via a web browser, curl, Node.js, Python, or other client applications. The exact error message can vary but usually points to an untrusted certificate or certificate chain.

    Common Error Outputs:

    • curl:
    • `bash
    • curl: (60) SSL certificate problem: self signed certificate in certificate chain
    • More details here: https://curl.haxx.se/docs/sslcerts.html
    • curl failed to verify the legitimacy of the server and therefore could not
    • establish a secure connection to it. To learn more about this situation and
    • how to fix it, please visit the web page mentioned above.
    • `
    • Node.js (e.g., fetch or https module):
    • `
    • FetchError: request to https://localhost:8443/api failed, reason: self-signed certificate in certificate chain
    • at ClientRequest.<anonymous> (/path/to/node_modules/node-fetch/lib/index.js:1505:11)
    • at ClientRequest.emit (node:events:514:28)
    • at TLSSocket.socketErrorListener (node:httpclient:481:9)
    • at TLSSocket.emit (node:events:514:28)
    • at emitErrorNT (node:internal/streams/destroy:151:8)
    • at emitErrorCloseNT (node:internal/streams/destroy:120:3)
    • at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    • type: 'system',
    • errno: 'DEPTHZEROSELFSIGNEDCERT',
    • code: 'DEPTHZEROSELFSIGNEDCERT'
    • }
    • `
    • Python (requests library):
    • `python
    • requests.exceptions.SSLError: HTTPSConnectionPool(host='localhost', port=8443): Max retries exceeded with url: /api (Caused by SSLError(1, '[SSL: CERTIFICATEVERIFYFAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)'))
    • `
    • Browser (e.g., Chrome/Firefox):
    • You will see a "Your connection is not private" or "Warning: Potential Security Risk Ahead" page with an error code like NET::ERRCERTAUTHORITYINVALID or SECERRORUNKNOWNISSUER.

    Root Cause Analysis

    The core of the problem lies in the trust chain of your SSL certificate. When a client (browser, curl, Node.js app) connects to an HTTPS server, it receives the server's SSL certificate. The client then attempts to validate this certificate by tracing its signing authority back to a trusted Root Certificate Authority (CA).

    1. Self-Signed Certificates: In a local development environment, you often create certificates that are "self-signed," meaning the certificate is signed by its own private key, or signed by a custom Certificate Authority (CA) that you created.
    2. Untrusted Authority: Neither your self-signed certificate nor your custom CA's certificate is recognized or pre-installed in the default trust stores of macOS or the various applications you use. These trust stores contain public certificates from globally recognized CAs (like Let's Encrypt, DigiCert, etc.).
    3. macOS Keychain vs. Application Trust: macOS has its own system-wide trust store (Keychain Access). While many applications, especially GUI ones and those using Apple's Secure Transport framework, will leverage this, others (like curl linked to Homebrew's OpenSSL, Node.js, Python's requests library) might rely on their own bundled CA bundles or specific OpenSSL configurations. This divergence means trusting a certificate in Keychain Access might not automatically resolve issues for all tools.
    4. certificate chain: The error "self signed certificate in certificate chain" indicates that the certificate presented by the server is part of a chain where one or more intermediate certificates, or the root certificate itself, is self-signed and not trusted. Typically, for local development, you create a Root CA and then sign your server certificate with that Root CA. The client then needs to trust your Root CA.

    Step-by-Step Resolution

    The resolution involves two primary phases: ensuring you have a properly generated self-signed certificate (or CA) and then explicitly adding that certificate to the relevant trust stores on your macOS system.

    1. Generate a Self-Signed Root CA and Server Certificate (If you don't have one)

    If you are just using a basic, self-signed server certificate, your client may complain about a "depth 0" error. For a more robust local setup that mimics production, it's better to create your own Root Certificate Authority (CA) and then use it to sign individual server certificates. This allows you to trust your single CA on your development machine, and all certificates signed by it will automatically be trusted.

    [!IMPORTANT] If you are already using a self-signed certificate or have a custom CA certificate (e.g., from a Docker container or another local setup), you can skip this step and proceed to Step 2 with your existing CA certificate. Ensure you have the .crt (or .pem) file of your Root CA.

    We'll use openssl via Homebrew on macOS.

    First, install or ensure OpenSSL is updated via Homebrew: `bash brew update brew install openssl@3 # Link openssl@3 if not already linked (this might prompt for sudo access or specific commands) brew link openssl@3 –force `

    Now, let's create a directory for our certificates and generate the CA: `bash mkdir -p ~/certs/local-ca cd ~/certs/local-ca

    1. Generate CA Private Key openssl genrsa -aes256 -out ca.key 4096

    2. Create CA Certificate Request openssl req -new -x509 -sha256 -days 3650 -key ca.key -out ca.crt -subj "/C=US/ST=CA/O=Local Development/CN=Local Development CA"

    Now generate a server certificate signed by this CA: # We'll use example.com and localhost for the server certificate SERVER_NAME="localhost" # Or your local dev domain, e.g., myapp.local

    1. Generate Server Private Key openssl genrsa -out $SERVER_NAME.key 2048

    2. Create Server Certificate Signing Request (CSR) openssl req -new -key $SERVERNAME.key -out $SERVERNAME.csr -subj "/C=US/ST=CA/O=Local Development/CN=$SERVER_NAME"

    3. Create a V3 Ext File for SAN (Subject Alternative Name) # This is crucial for modern browsers and clients which require SAN. cat <<EOF > $SERVER_NAME.ext authorityKeyIdentifier=keyid,issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment subjectAltName = @alt_names

    [alt_names] DNS.1 = $SERVER_NAME DNS.2 = localhost IP.1 = 127.0.0.1 EOF

    4. Sign the Server Certificate with your CA openssl x509 -req -in $SERVER_NAME.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out $SERVERNAME.crt -days 365 -sha256 -extfile $SERVERNAME.ext

    echo "Certificates generated in ~/certs/local-ca:" ls -l ~/certs/local-ca ` You now have ca.crt (your Root CA certificate) and $SERVER_NAME.crt (your server certificate) along with their respective keys. The ca.crt is what needs to be trusted.

    2. Trust the Root CA Certificate in macOS Keychain Access

    This step adds your custom Root CA to the macOS system trust store, which many applications (especially browsers and those using Apple's Secure Transport) will automatically use.

    # Assuming your CA certificate is named ca.crt and is in ~/certs/local-ca
    sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ~/certs/local-ca/ca.crt
    ```

    After running the command: 1. Open Keychain Access.app (Applications > Utilities > Keychain Access). 2. Select "System" under "Keychains" on the left sidebar. 3. Select "Certificates" under "Category" on the left sidebar. 4. Search for "Local Development CA" (or whatever CN you used). 5. Double-click your CA certificate. 6. Expand the "Trust" section. 7. For "When using this certificate:", select "Always Trust" from the dropdown. 8. Close the window, and you'll be prompted to enter your password again to save changes.

    [!IMPORTANT] Some applications, especially those from Homebrew, might use a different OpenSSL configuration that doesn't directly leverage the macOS Keychain. This requires additional steps.

    3. Configure Applications and Development Runtimes to Use the Trusted CA

    Even after adding to Keychain, some applications might still fail. This is because they might be looking for certificates in specific locations or using their own bundled CAs.

    a. For curl and Homebrew-installed OpenSSL

    Homebrew-installed OpenSSL typically looks for trusted certificates in /usr/local/etc/openssl@3/certs/. You need to symlink your CA certificate there and then rehash the directory.

    # Assuming your CA certificate is ~/certs/local-ca/ca.crt
    CA_CERT_PATH=~/certs/local-ca/ca.crt

    Create a symlink to your CA cert in OpenSSL's certs directory ln -s "$CACERTPATH" "$OPENSSLCERTSDIR/$(openssl x509 -hash -noout -in "$CACERTPATH").0"

    Update OpenSSL's certificate trust store # This command re-scans the directory and creates necessary links /usr/local/opt/openssl@3/bin/crehash "$OPENSSLCERTS_DIR" `

    Now, test curl: `bash curl https://localhost:8443/ # Replace with your local service URL ` It should now connect successfully without -k or --insecure.

    b. For Node.js Applications

    Node.js can be configured using the NODEEXTRACA_CERTS environment variable.

    # Add this to your shell profile (.bashrc, .zshrc, .profile)

    Or set it when running a specific command NODEEXTRACACERTS=~/certs/local-ca/ca.crt node yourapp.js `

    [!NOTE] For Electron apps or webviews, ensuring the certificate is trusted in Keychain Access (Step 2) is usually sufficient, as they often leverage the system's trust store.

    c. For Python Applications (e.g., requests library)

    Python's requests library often uses certifi which has its own bundle. You can specify an additional CA bundle.

    # Add this to your shell profile (.bashrc, .zshrc, .profile)

    Or set it when running a specific script REQUESTSCABUNDLE=~/certs/local-ca/ca.crt python your_script.py `

    Alternatively, you can modify the certifi bundle directly (less recommended for maintainability) or pass the verify parameter to requests (not recommended for general use, but useful for debugging).

    d. For PHP Applications (cURL extension)

    PHP's cURL extension, when compiled against OpenSSL, might need the CURLOPT_CAINFO option or a global curl.cainfo setting.

    <?php
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, "https://localhost:8443/");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    // Specify the path to your CA certificate
    curl_setopt($ch, CURLOPT_CAINFO, "/Users/youruser/certs/local-ca/ca.crt");
    $output = curl_exec($ch);
    if (curl_errno($ch)) {
        echo 'Curl error: ' . curl_error($ch);
    }
    curl_close($ch);
    echo $output;
    ?>

    For a more global approach, you can set the SSLCERTFILE environment variable for the PHP process or ensure the OpenSSL ca-certificates package (if applicable to your PHP setup, e.g., via Homebrew or Docker) is updated with your CA.

    e. For Ruby Applications

    Ruby's Net::HTTP and other libraries that use OpenSSL typically respect the SSLCERTFILE or SSLCERTDIR environment variables.

    # Add this to your shell profile (.bashrc, .zshrc, .profile)

    Or set it when running a specific script SSLCERTFILE=~/certs/local-ca/ca.crt ruby your_script.rb `

    f. For Git

    Git can be configured to trust your CA certificate.

    git config --global http.sslCAInfo ~/certs/local-ca/ca.crt

    [!WARNING] While git config --global http.sslVerify false can temporarily fix issues, it's a security risk as it disables all SSL verification. Only use it for temporary debugging and never in production or for sensitive operations. Always prefer to properly trust the certificate.

    4. Restart Services and Applications

    After making changes to environment variables or adding certificates, it's crucial to restart any applications, terminals, or services that rely on these configurations. A full system reboot might sometimes be necessary to ensure all processes pick up the new trust settings, especially for GUI applications.

    For Docker containers, if your container is the one initiating the connection to an external (or host-local) HTTPS service and needs to trust your CA, you'll need to inject your ca.crt into the container's trust store.

    Example for a Debian/Ubuntu-based Docker container:

    1. Copy your CA certificate into the container's build context.
    2. Modify your Dockerfile:
    3. `dockerfile
    4. # In your Dockerfile
    5. COPY ~/certs/local-ca/ca.crt /usr/local/share/ca-certificates/local-dev-ca.crt
    6. RUN chmod 644 /usr/local/share/ca-certificates/local-dev-ca.crt && update-ca-certificates
    7. `
    8. This adds your CA to the container's trust store, making applications inside the container trust it.

    By following these detailed steps, you should successfully overcome the "self signed certificate in certificate chain validation" errors on your macOS local development environment, leading to a much smoother and more secure development workflow.

  • Troubleshooting Caddy Reverse Proxy TLS Handshake Failed Verification on Ubuntu 20.04 LTS

    Resolve Caddy reverse proxy TLS handshake failures with upstream servers on Ubuntu 20.04 LTS. Diagnose certificate validation, SNI, and CA issues for seamless reverse proxy operations.

    Introduction

    Caddy has established itself as a powerful, easy-to-use web server and reverse proxy, renowned for its automatic HTTPS capabilities. However, when operating Caddy as a reverse proxy, you might encounter situations where it fails to establish a secure (TLS) connection with its upstream backend server. A "TLS handshake failed verification" error signifies that Caddy initiated a TLS connection but was unable to validate the certificate presented by the upstream server. This guide provides a highly technical, step-by-step approach to diagnose and resolve this issue specifically on Ubuntu 20.04 LTS environments.

    ### Symptom & Error Signature

    When Caddy experiences a TLS handshake failure with an upstream, client requests attempting to reach the proxied service will typically receive a 502 Bad Gateway error. In the Caddy server logs, you'll find entries indicating certificate validation problems.

    To view Caddy's logs, use journalctl:

    sudo journalctl -u caddy -f --no-hostname

    Typical error signatures you might observe include:

    {
      "level": "error",
      "ts": 1678886400.000,
      "logger": "http.log.error",
      "msg": "x509: certificate signed by unknown authority",
      "trace": "...",
      "request": {
        "method": "GET",
        "uri": "/api/data",
        "proto": "HTTP/2.0",
        "remote_addr": "192.168.1.100:54321",
        "host": "your-caddy-domain.com",
        "headers": {
          "User-Agent": ["Mozilla/5.0 ..."],
          "Accept": ["*/*"]
        }
      },
      "error": "reverse proxy: selected upstream is unhealthy: connection to upstream timed out after 10s"
    }

    Or a more generic handshake failure:

    {
      "level": "error",
      "ts": 1678886405.000,
      "logger": "http.handlers.reverse_proxy",
      "msg": "aborting with incomplete response",
      "error": "remote error: tls: handshake failure",
      "duration": 0.0001
    }

    Or an SNI related error:

    {
      "level": "error",
      "ts": 1678886410.000,
      "logger": "http.log.error",
      "msg": "tls: failed to verify certificate: x509: certificate is valid for example.com, not your-upstream-hostname.internal",
      "trace": "...",
      "error": "reverse proxy: selected upstream is unhealthy: connection to upstream timed out after 10s"
    }

    ### Root Cause Analysis

    The "TLS handshake failed verification" error fundamentally means that Caddy, acting as a TLS client to the upstream server, could not trust the certificate presented by that server. The most common underlying reasons for this include:

    1. Untrusted Certificate Authority (CA): The upstream server's certificate is signed by a CA that is not present or trusted in Caddy's (or the underlying system's) trust store. This is common with self-signed certificates, private enterprise CAs, or newly issued CA roots not yet widely distributed.
    2. Certificate Name Mismatch (SNI/SAN):
    3. * Common Name (CN) / Subject Alternative Name (SAN) Mismatch: The hostname Caddy is attempting to connect to (or the Host header it's sending) does not match the CN or SANs listed within the upstream server's certificate.
    4. * Incorrect Server Name Indication (SNI): Caddy might not be sending the correct SNI hostname during the TLS handshake, causing the upstream server (especially if it hosts multiple virtual hosts with different certificates) to present the wrong certificate or refuse the connection.
    5. Expired or Invalid Certificate: The upstream server's certificate has expired, is not yet valid, or has been revoked.
    6. Network/Firewall Issues: While less direct for "verification failure," network problems or restrictive firewalls between Caddy and the upstream could interrupt the TLS handshake process, leading to a perceived failure.
    7. System Time Skew: Significant clock drift on either the Caddy server or the upstream server can cause certificate validity checks (e.g., notBefore, notAfter dates) to fail.

    ### Step-by-Step Resolution

    Follow these steps to diagnose and resolve the TLS handshake verification error.

    1. Verify Upstream Server's Certificate Directly

    Before troubleshooting Caddy, confirm the upstream server's certificate is valid and correctly configured from the perspective of the Caddy server's host.

    # Replace 'your_upstream_hostname' with the actual hostname Caddy connects to
    # Replace 'upstream_ip_address' with the upstream server's IP
    # Replace 'upstream_port' with the port the upstream listens on (e.g., 443, 8443)
    openssl s_client -connect your_upstream_ip_address:upstream_port -servername your_upstream_hostname </dev/null

    Inspect the output carefully: * Verify return code: 0 (ok): Indicates the certificate chain is trusted by the system's CA store. If you see anything else (e.g., 20 (unable to get local issuer certificate), 21 (unable to verify the first certificate)), it points to an untrusted CA. * Not Before / Not After: Check if the certificate is within its valid date range. * Subject Common Name (CN) / Subject Alternative Name (SAN): Ensure yourupstreamhostname is listed here. If not, you have a hostname mismatch. * depth and issuer: Review the certificate chain to identify the issuing CA.

    2. Update System CA Trust Store (for Untrusted CAs)

    If openssl s_client returned an error indicating an untrusted issuer (e.g., Verify return code: 20), you need to add the upstream's CA certificate to Caddy's system trust store. This is common for self-signed certificates or private enterprise CAs.

    1. Obtain the CA Certificate:
    2. If you have the upstream CA's .crt file, copy it. If not, you can often extract it using openssl:
        # Connect to the upstream and extract the full certificate chain

    Open 'upstream_chain.pem' and identify the root CA certificate. # It's usually the last certificate in the chain and self-signed (Issuer and Subject are the same). # Copy its content (including —–BEGIN CERTIFICATE—– and —–END CERTIFICATE—–) # into a new file, e.g., 'upstream_ca.crt'. `

    1. Add the CA Certificate to System Trust:
        sudo cp /path/to/upstream_ca.crt /usr/local/share/ca-certificates/
        sudo update-ca-certificates

    You should see output similar to: Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.

    1. Restart Caddy:
        sudo systemctl restart caddy
        sudo journalctl -u caddy -f --no-hostname # Check logs for success

    [!IMPORTANT] Ensure the CA certificate has a .crt extension when copied to /usr/local/share/ca-certificates/. The update-ca-certificates utility only processes files with this extension in that directory by default.

    3. Configure Caddy for Specific Upstream TLS Trust (Caddyfile Directive)

    For more granular control, especially if you only want Caddy to trust a specific custom CA for a particular upstream and not globally, you can configure this directly in your Caddyfile.

    1. Place CA Certificate: Store the upstream CA certificate file (e.g., upstream_ca.crt) in a secure, Caddy-readable location, typically /etc/caddy/certs/.
    1. Modify Caddyfile:
    2. Add or modify the transport block within your reverse_proxy directive:
        your-caddy-domain.com {
            reverse_proxy your_upstream_ip_address:upstream_port {
                transport http {
                    # Trusts the specified CA certificate for this upstream only
                    tls_trusted_ca /etc/caddy/certs/upstream_ca.crt
                    # Optional: specify the server name if different from the Caddy domain
                    # This ensures the correct certificate is requested via SNI
                    tls_server_name your_upstream_hostname
                }
            }
        }
    1. Validate and Apply Caddyfile Changes:
        sudo caddy validate --config /etc/caddy/Caddyfile
        sudo systemctl reload caddy
        sudo journalctl -u caddy -f --no-hostname # Monitor logs

    4. Address Hostname/SNI Mismatch

    If openssl s_client revealed a hostname mismatch (CN/SAN mismatch) or if your upstream requires a specific SNI hostname to present the correct certificate:

    1. Caddyfile reverse_proxy Configuration:
    2. Ensure Caddy is sending the correct Host header and tlsservername (SNI) for the upstream.
        your-caddy-domain.com {
            reverse_proxy your_upstream_ip_address:upstream_port {
                # This sets the Host header sent to the upstream
                # Crucial if upstream expects a specific hostname

    transport http { # This sets the SNI hostname for the TLS handshake # Required if the upstream server serves multiple certificates tlsservername yourupstreamhostname } } } ` Replace yourupstreamhostname with the actual hostname that the upstream's certificate is valid for. This is often the internal FQDN of the backend server.

    1. Validate and Apply Caddyfile Changes:
        sudo caddy validate --config /etc/caddy/Caddyfile
        sudo systemctl reload caddy
        sudo journalctl -u caddy -f --no-hostname

    5. Verify System Time Synchronization

    Significant time differences can cause certificates to appear invalid. Ensure both Caddy and the upstream server are synchronized with an NTP server.

    timedatectl

    If time is unsynchronized, ensure systemd-timesyncd is enabled or install ntp:

    sudo systemctl enable --now systemd-timesyncd
    # Or for legacy NTP daemon:
    # sudo apt update && sudo apt install ntp
    # sudo systemctl enable --now ntp

    6. Bypass TLS Verification (Development/Testing ONLY)

    [!WARNING] This step disables all TLS certificate verification. It should NEVER be used in production environments as it makes your system vulnerable to Man-in-the-Middle attacks. Use this only for debugging in isolated development environments or if you fully understand and accept the security implications.

    If all else fails and you need to quickly confirm if TLS verification is the root cause, you can temporarily disable it:

    your-caddy-domain.com {
        reverse_proxy your_upstream_ip_address:upstream_port {
            transport http {
                tls_insecure_skip_verify
                # Optionally, still specify server name for SNI
                # tls_server_name your_upstream_hostname
            }
        }
    }

    Validate and apply, then immediately remove this directive once you've confirmed the issue.

    7. Check Upstream Server's Certificate Expiration/Validity

    If openssl s_client revealed an expired or not-yet-valid certificate on the upstream, the solution lies with the upstream server's administrator. They need to renew or correctly install a valid certificate. Caddy cannot fix an invalid certificate presented by its backend.

    This comprehensive guide should help you systematically troubleshoot and resolve the "Caddy reverse proxy TLS handshake failed verification" error on your Ubuntu 20.04 LTS system. Remember to always apply the least permissive fix possible for security.