fix(duckdb): proactively refresh S3 secrets to prevent STS credential expiry

[glopal]

DuckDB's httpfs extension snapshots STS credentials into a secret at creation time and never re-queries the provider chain, even with refresh: auto configured. When using CREDENTIAL_CHAIN with chain: sts (AssumeRole), DuckDB requests the AWS minimum session duration of 900 seconds (15 minutes). Any SQLMesh plan that takes longer than 15 minutes to complete — such as large incremental backfills — fails with HTTP 400 (Bad Request) from S3 once the token expires.

This was confirmed through controlled experiments (binary search narrowed the threshold to exactly ~15 minutes, and duckdb_secrets() showed the same key_id throughout a run — no refresh ever occurs).

The upstream fix (duckdb/duckdb-httpfs#165) adds credential refresh at the httpfs layer but has not been merged yet.

This patch adds a timer-based secret refresh mechanism to the DuckDB engine adapter as a workaround:

• Before each SQL execution, checks if 12 minutes (80% of the 900s TTL) have elapsed since the last secret creation
• If so, queries duckdb_secrets() for existing S3 secret names, drops them, and recreates from the original config — forcing a fresh STS AssumeRole call
• Uses double-check locking to prevent concurrent refresh when concurrent_tasks > 1
• Zero overhead for configs without S3 secrets (early return on null check) and minimal overhead otherwise (monotonic clock comparison on hot path)

Changes:

• sqlmesh/core/engine_adapter/duckdb.py: Add init, _execute override, and secret refresh methods to DuckDBEngineAdapter
• sqlmesh/core/config/connection.py: Add _extra_engine_config to DuckDBConnectionConfig to pass secrets config to the adapter

This workaround can be removed once the upstream duckdb-httpfs fix lands and we upgrade DuckDB.

Refs: duckdb/duckdb-httpfs#165
Refs: duckdb/duckdb-aws#26

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}