AWS CLI v2 embedded CA bundle does not validate a TLS chain validated by certifi and OpenSSL

[AnthonySt-G]

Describe the issue

Summary

We are observing a TLS validation issue when using AWS CLI v2 against our S3-compatible HTTPS endpoint.

With the default AWS CLI v2 configuration, the command fails with a certificate validation error.

However, the same endpoint certificate chain is successfully validated by:

• the current certifi / Mozilla CA bundle;
• the default OpenSSL / operating system trust store.

Based on these observations, my current understanding is that the issue may be related to the CA bundle embedded in AWS CLI v2 / botocore, rather than to the endpoint certificate chain itself.

This does not appear to be a private or enterprise CA use case. The endpoint uses a public TLS certificate chain.

Possible related issue : #9017

Environment

AWS CLI version: aws-cli/2.34.58

Operating system: Ubuntu

Endpoint: https://custom.s3.server

AWS CLI embedded CA bundle used by default:

/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem

Current behavior

Running AWS CLI v2 against the endpoint fails with a TLS certificate validation error.

command:

aws s3 ls \
  --endpoint-url "https://custom.s3.server" \
  --debug

Error:

botocore.exceptions.SSLError: SSL validation failed for https://custom.s3.server/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1081)

The debug output confirms that AWS CLI uses its embedded botocore CA bundle:

/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem

Validation checks

1. AWS CLI embedded CA bundle

openssl s_client \
  -connect "custom.s3.server:443" \
  -servername "custom.s3.server" \
  -CAfile "/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem" \
  -verify_return_error \
  </dev/null

Result:

FAILED

2. certifi / Mozilla CA bundle

Using certifi as the CA bundle works successfully.

import os

import boto3
import certifi


os.environ["AWS_CA_BUNDLE"] = certifi.where()

s3_client = boto3.client(
    "s3",
    endpoint_url="https://custom.s3.server",
    region_name="us-east-1",
)

response = s3_client.list_buckets()
print(response)

Result:

Bucket list is returned successfully.

3. Default OpenSSL / OS trust store

openssl s_client \
  -connect "custom.s3.server:443" \
  -servername "custom.s3.server" \
  -verify_return_error \
  </dev/null

Result:

Verify return code: 0 (ok)

Observed behavior summary

certifi / Mozilla CA bundle      -> OK
OpenSSL / OS trust store         -> OK
AWS CLI embedded botocore bundle -> FAILED

Expected behavior

My expectation is that AWS CLI v2 should be able to validate a TLS certificate chain that is already trusted by current public CA trust stores, such as certifi / Mozilla and the default OpenSSL / OS trust store.

If this expectation is incorrect, I would appreciate clarification on the intended trust model for AWS CLI v2.

Current workaround

The issue can be worked around by explicitly overriding the AWS CLI CA bundle.

For example:

export AWS_CA_BUNDLE=/path/to/valid/ca-bundle.pem

or:

aws ... --ca-bundle /path/to/valid/ca-bundle.pem

or:

[default]
ca_bundle = /path/to/valid/ca-bundle.pem

Using the OS CA bundle or the certifi CA bundle resolves the issue in our tests.

However, for a customer-facing usage, requiring users to manually configure a CA bundle is not ideal when the certificate chain is already trusted by standard public CA bundles.

Request

If the embedded AWS CLI v2 / botocore CA bundle is expected to track public CA trust sources, could it be refreshed or reviewed so that it validates the same public TLS chains as current certifi / Mozilla and OpenSSL trust stores?

I may be missing some details about the AWS CLI v2 trust model, so any clarification would be very helpful.

Thank you for your help and for maintaining AWS CLI.

Additional Information/Context

No response

CLI version used

aws-cli/2.34.58

Environment details (OS name and version, etc.)

Windows and Ubuntu 24.04.4