From 396c49a11fd52999a4f375f4a7b4b5db5010f2fa Mon Sep 17 00:00:00 2001 From: Ian Bucad Date: Wed, 1 Jul 2026 10:25:53 +1000 Subject: [PATCH 1/5] [TEEP-5473] Document multi_secret_backends for multiple native secret backends Adds documentation for the multi_secret_backends feature (Agent 7.80+): - Config structure with named backends (type + config per entry) - ENC[backendName;secretKey] handle format and semicolon delimiter rules - Precedence over secret_backend_command / secret_backend_type - Partial resolution behavior and secret output example - Step-by-step migration guide from secret_backend_type Co-Authored-By: Claude Sonnet 4.6 --- .../agent/configuration/secrets-management.md | 105 +++++++++++++++++- 1 file changed, 103 insertions(+), 2 deletions(-) diff --git a/content/en/agent/configuration/secrets-management.md b/content/en/agent/configuration/secrets-management.md index db10e5f7eb3..50547e295f5 100644 --- a/content/en/agent/configuration/secrets-management.md +++ b/content/en/agent/configuration/secrets-management.md @@ -426,7 +426,7 @@ secret_backend_type: azure.keyvault secret_backend_config: keyvaulturl: {keyVaultURL} azure_session: - azure_client_id: {clientID} # User-assigned managed identity client ID; omit this field for system-assigned + azure_client_id: {clientID} # User-assigned managed identity client ID; omit this field for system-assigned ``` When using environment variables, convert the configuration to JSON: @@ -1007,7 +1007,7 @@ secret_backend_config: | `client_cert` | Path to a PEM-encoded client certificate file for mTLS. | | `client_key` | Path to the private key file for the client certificate. | | `tls_server` | Expected server name for TLS SNI verification. | -| `insecure` | Set to `true` to disable TLS certificate verification. Do not use in production. | +| `insecure` | Set to `true` to disable TLS certificate verification. Do not use in production. | {{% /collapse-content %}} @@ -1453,6 +1453,107 @@ secret_backend_config: {{% /collapse-content %}} +### Using multiple native backends simultaneously + +**Available in Agent version 7.80+** + +Instead of a single `secret_backend_type`, you can declare multiple named backends under `multi_secret_backends`. Each backend has its own `type` and `config`, and secrets are routed to a specific backend using a `backendName;` prefix in the `ENC[]` handle. + +**Precedence**: If `secret_backend_command` or `secret_backend_type` is also set, `multi_secret_backends` is ignored and a warning is logged. Remove those settings before using `multi_secret_backends`. + +#### Configuration + +```yaml +# datadog.yaml + +secret_backend_timeout: 30 # applies globally to all backends + +multi_secret_backends: + : + type: + config: + : +``` + +Each `` is an arbitrary identifier you choose. The `type` and `config` fields follow the same schema as `secret_backend_type` and `secret_backend_config` for the corresponding backend. + +#### ENC handle format + +When `multi_secret_backends` is active, prefix `ENC[]` handles with the backend name followed by a semicolon: + +``` +ENC[;] +``` + +Only the **first** semicolon is treated as the delimiter. Secret keys that themselves contain semicolons (for example, Kubernetes-style `namespace/secret-name;key`) continue to work. + +#### Example + +The following configuration reads secrets from two file backends simultaneously: + +```yaml +# datadog.yaml +secret_backend_timeout: 30 + +multi_secret_backends: + yaml_secrets: + type: file.yaml + config: + file_path: /etc/datadog-agent/secrets.yaml + aws_secrets: + type: aws.secrets + config: + aws_session: + aws_region: us-east-1 +``` + +Reference secrets by prefixing with the backend name: + +```yaml +# datadog.yaml +api_key: ENC[yaml_secrets;api_key] +app_key: ENC[aws_secrets;My-Secrets;appKey] +``` + +#### Partial resolution + +If one backend or handle fails, the Agent still substitutes all successfully resolved handles. Failed handles are reported in `datadog-agent secret` output under **Secrets not resolved**: + +``` +=== Secrets stats === +Number of secrets resolved: 2 +Secrets handle resolved: +- 'yaml_secrets;api_key': from datadog.yaml + +Secrets not resolved: + - handle "aws_secrets;nokey": an error occurred while resolving 'nokey': backend does not provide secret key +``` + +#### Migrating from `secret_backend_type` + +To switch from a single `secret_backend_type` to `multi_secret_backends`: + +1. Move `secret_backend_type` and `secret_backend_config` into a named entry under `multi_secret_backends`. +2. Remove `secret_backend_type` and `secret_backend_config` from the top level. +3. Update all `ENC[secretKey]` handles to `ENC[backendName;secretKey]`. + +```yaml +# Before +secret_backend_type: file.yaml +secret_backend_config: + file_path: /etc/datadog-agent/secrets.yaml + +api_key: ENC[api_key] + +# After +multi_secret_backends: + my_yaml: + type: file.yaml + config: + file_path: /etc/datadog-agent/secrets.yaml + +api_key: ENC[my_yaml;api_key] +``` ### Option 2: Using the built-in Script for Kubernetes and Docker From 56425f3c6c874b79eab8c3a75e37f05befa9473f Mon Sep 17 00:00:00 2001 From: Ian Bucad Date: Thu, 2 Jul 2026 20:25:13 +1000 Subject: [PATCH 2/5] Address PR review: fix heading structure, secret count, backend name constraint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rename heading to "Option 1b" to preserve Option 1/2/3 numbered flow - Fix resolved secret count (2 → 1) to match the sample output - Note that backend names cannot contain semicolons Co-Authored-By: Claude Sonnet 4.6 --- content/en/agent/configuration/secrets-management.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/en/agent/configuration/secrets-management.md b/content/en/agent/configuration/secrets-management.md index 50547e295f5..d3fa764f81f 100644 --- a/content/en/agent/configuration/secrets-management.md +++ b/content/en/agent/configuration/secrets-management.md @@ -1453,7 +1453,7 @@ secret_backend_config: {{% /collapse-content %}} -### Using multiple native backends simultaneously +### Option 1b: Using multiple native backends simultaneously **Available in Agent version 7.80+** @@ -1475,7 +1475,7 @@ multi_secret_backends: : ``` -Each `` is an arbitrary identifier you choose. The `type` and `config` fields follow the same schema as `secret_backend_type` and `secret_backend_config` for the corresponding backend. +Each `` is an arbitrary identifier you choose. It cannot contain a semicolon, because `;` is the delimiter used in `ENC[]` handles. The `type` and `config` fields follow the same schema as `secret_backend_type` and `secret_backend_config` for the corresponding backend. #### ENC handle format @@ -1521,7 +1521,7 @@ If one backend or handle fails, the Agent still substitutes all successfully res ``` === Secrets stats === -Number of secrets resolved: 2 +Number of secrets resolved: 1 Secrets handle resolved: - 'yaml_secrets;api_key': from datadog.yaml From e7a0c35d4ca7f1c94a60d92ccc1175d0f08868bd Mon Sep 17 00:00:00 2001 From: Ian Bucad Date: Fri, 3 Jul 2026 10:37:52 +1000 Subject: [PATCH 3/5] Address review: restructure Option 1, clean up multi_secret_backends section - Add Single backend / Multiple backends sub-headings under Option 1 - Remove secret_backend_timeout mentions - Rephrase precedence as numbered list - Drop Partial resolution section - Rename "ENC handle format" to "ENC notation" - Clarify "backend delimiter" wording Co-Authored-By: Claude Sonnet 4.6 --- .../agent/configuration/secrets-management.md | 46 +++++++------------ 1 file changed, 17 insertions(+), 29 deletions(-) diff --git a/content/en/agent/configuration/secrets-management.md b/content/en/agent/configuration/secrets-management.md index d3fa764f81f..3064795c82c 100644 --- a/content/en/agent/configuration/secrets-management.md +++ b/content/en/agent/configuration/secrets-management.md @@ -35,9 +35,13 @@ Instead of hardcoding sensitive values like API keys or passwords in plaintext w **Note**: As of Agent version `7.76` and onwards, native secrets management is available for FIPS-enabled Agents. -Starting in Agent version `7.70`, the Datadog Agent natively supports several secret management solutions. Two new settings have been introduced to `datadog.yaml`: `secret_backend_type` and `secret_backend_config`. +Starting in Agent version `7.70`, the Datadog Agent natively supports several secret management solutions. -`secret_backend_type` is used to specify which secret management solution to use, and `secret_backend_config` holds additional configuration relevant to that solution. +**Note**: If you are running Datadog in a containerized environment, the [Cluster Agent](/containers/cluster_agent/) requires Agent 7.77 or later to support native secrets fetching. For earlier versions, use [Option 2](#option-2-using-the-built-in-script-for-kubernetes-and-docker) or [Option 3](#option-3-creating-a-custom-executable) instead. + +#### Single backend + +Use `secret_backend_type` and `secret_backend_config` in `datadog.yaml` to configure a single secret backend: ```yaml # datadog.yaml @@ -47,8 +51,6 @@ secret_backend_config: : ``` -**Note**: If you are running Datadog in a containerized environment, the [Cluster Agent](/containers/cluster_agent/) requires Agent 7.77 or later to support native secrets fetching. For earlier versions, use [Option 2](#option-2-using-the-built-in-script-for-kubernetes-and-docker) or [Option 3](#option-3-creating-a-custom-executable) instead. - More specific setup instructions depend on the backend type used. See the appropriate section below for further information: @@ -1453,21 +1455,23 @@ secret_backend_config: {{% /collapse-content %}} -### Option 1b: Using multiple native backends simultaneously +#### Multiple backends **Available in Agent version 7.80+** Instead of a single `secret_backend_type`, you can declare multiple named backends under `multi_secret_backends`. Each backend has its own `type` and `config`, and secrets are routed to a specific backend using a `backendName;` prefix in the `ENC[]` handle. -**Precedence**: If `secret_backend_command` or `secret_backend_type` is also set, `multi_secret_backends` is ignored and a warning is logged. Remove those settings before using `multi_secret_backends`. +If more than one of the following is set, the highest-priority setting takes effect and the others are ignored with a warning: + +1. `secret_backend_command` +2. `secret_backend_type` +3. `multi_secret_backends` -#### Configuration +##### Configuration ```yaml # datadog.yaml -secret_backend_timeout: 30 # applies globally to all backends - multi_secret_backends: : type: @@ -1477,7 +1481,7 @@ multi_secret_backends: Each `` is an arbitrary identifier you choose. It cannot contain a semicolon, because `;` is the delimiter used in `ENC[]` handles. The `type` and `config` fields follow the same schema as `secret_backend_type` and `secret_backend_config` for the corresponding backend. -#### ENC handle format +##### ENC notation When `multi_secret_backends` is active, prefix `ENC[]` handles with the backend name followed by a semicolon: @@ -1485,16 +1489,14 @@ When `multi_secret_backends` is active, prefix `ENC[]` handles with the backend ENC[;] ``` -Only the **first** semicolon is treated as the delimiter. Secret keys that themselves contain semicolons (for example, Kubernetes-style `namespace/secret-name;key`) continue to work. +Only the **first** semicolon is treated as the backend delimiter. Secret keys that themselves contain semicolons (for example, Kubernetes-style `namespace/secret-name;key`) continue to work. -#### Example +##### Example The following configuration reads secrets from two file backends simultaneously: ```yaml # datadog.yaml -secret_backend_timeout: 30 - multi_secret_backends: yaml_secrets: type: file.yaml @@ -1515,21 +1517,7 @@ api_key: ENC[yaml_secrets;api_key] app_key: ENC[aws_secrets;My-Secrets;appKey] ``` -#### Partial resolution - -If one backend or handle fails, the Agent still substitutes all successfully resolved handles. Failed handles are reported in `datadog-agent secret` output under **Secrets not resolved**: - -``` -=== Secrets stats === -Number of secrets resolved: 1 -Secrets handle resolved: -- 'yaml_secrets;api_key': from datadog.yaml - -Secrets not resolved: - - handle "aws_secrets;nokey": an error occurred while resolving 'nokey': backend does not provide secret key -``` - -#### Migrating from `secret_backend_type` +##### Migrating from `secret_backend_type` To switch from a single `secret_backend_type` to `multi_secret_backends`: From c62ec969e96283dd87c635a96320636768cc755b Mon Sep 17 00:00:00 2001 From: Ian Bucad Date: Fri, 3 Jul 2026 11:17:38 +1000 Subject: [PATCH 4/5] Address linter: wrap ENC in backticks in heading to satisfy sentence-case rule Co-Authored-By: Claude Sonnet 4.6 --- content/en/agent/configuration/secrets-management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/agent/configuration/secrets-management.md b/content/en/agent/configuration/secrets-management.md index 3064795c82c..51ceaa8afbb 100644 --- a/content/en/agent/configuration/secrets-management.md +++ b/content/en/agent/configuration/secrets-management.md @@ -1481,7 +1481,7 @@ multi_secret_backends: Each `` is an arbitrary identifier you choose. It cannot contain a semicolon, because `;` is the delimiter used in `ENC[]` handles. The `type` and `config` fields follow the same schema as `secret_backend_type` and `secret_backend_config` for the corresponding backend. -##### ENC notation +##### `ENC[]` notation When `multi_secret_backends` is active, prefix `ENC[]` handles with the backend name followed by a semicolon: From 206e527c85c3ea1d5772c6bd422e42303a606653 Mon Sep 17 00:00:00 2001 From: Ian Bucad Date: Tue, 7 Jul 2026 11:00:59 +1000 Subject: [PATCH 5/5] Address review: standardize version callouts to italic pattern, convert prose version notes to bullets Co-Authored-By: Claude Sonnet 4.6 --- .../agent/configuration/secrets-management.md | 36 +++++++++++-------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/content/en/agent/configuration/secrets-management.md b/content/en/agent/configuration/secrets-management.md index 51ceaa8afbb..ad06a7c4f1a 100644 --- a/content/en/agent/configuration/secrets-management.md +++ b/content/en/agent/configuration/secrets-management.md @@ -33,11 +33,11 @@ Instead of hardcoding sensitive values like API keys or passwords in plaintext w ### Option 1: Using native Agent support for fetching secrets -**Note**: As of Agent version `7.76` and onwards, native secrets management is available for FIPS-enabled Agents. - -Starting in Agent version `7.70`, the Datadog Agent natively supports several secret management solutions. - -**Note**: If you are running Datadog in a containerized environment, the [Cluster Agent](/containers/cluster_agent/) requires Agent 7.77 or later to support native secrets fetching. For earlier versions, use [Option 2](#option-2-using-the-built-in-script-for-kubernetes-and-docker) or [Option 3](#option-3-creating-a-custom-executable) instead. +Notes: +- **Agent 7.70+**: Native secrets management support introduced. +- **Agent 7.76+**: Native secrets management available for FIPS-enabled Agents. +- **Agent 7.77+**: The [Cluster Agent](/containers/cluster_agent/) requires Agent 7.77 or later in containerized environments. For earlier versions, use [Option 2](#option-2-using-the-built-in-script-for-kubernetes-and-docker) or [Option 3](#option-3-creating-a-custom-executable) instead. +- **Agent 7.80+**: Support for [multiple backends](#multiple-backends). #### Single backend @@ -631,7 +631,7 @@ spec: {{% collapse-content title="GCP Secret Manager" level="h4" expanded=false id="id-for-gcp" %}} -**Available in Agent version 7.74+** +*Available in Agent version 7.74+* The following GCP services are supported: @@ -1015,7 +1015,7 @@ secret_backend_config: {{% collapse-content title="Kubernetes Secrets" level="h4" expanded=false id="id-for-kubernetes" %}} -**Available in Agent version 7.75+** +*Available in Agent version 7.75+* The following Kubernetes services are supported: @@ -1254,7 +1254,7 @@ override: {{% collapse-content title="Docker Secrets" level="h4" expanded=false id="id-for-docker" %}} -**Available in Agent version 7.75+** +*Available in Agent version 7.75+* The following Docker services are supported: @@ -1410,7 +1410,7 @@ secret_backend_config: {{% tab "TEXT File Backend" %}} -**Available in Agent version 7.75+** +*Available in Agent version 7.75+* **Note**: Each secret must be stored in its own individual text file. @@ -1457,7 +1457,7 @@ secret_backend_config: #### Multiple backends -**Available in Agent version 7.80+** +*Available in Agent version 7.80+* Instead of a single `secret_backend_type`, you can declare multiple named backends under `multi_secret_backends`. Each backend has its own `type` and `config`, and secrets are routed to a specific backend using a `backendName;` prefix in the `ENC[]` handle. @@ -1545,7 +1545,9 @@ api_key: ENC[my_yaml;api_key] ### Option 2: Using the built-in Script for Kubernetes and Docker -For containerized environments, the Datadog Agent's container images include a built-in script `/readsecret_multiple_providers.sh` starting with version v7.32.0. This script supports reading secrets from: +*Available in Agent version 7.32+* + +For containerized environments, the Datadog Agent's container images include a built-in script `/readsecret_multiple_providers.sh`. This script supports reading secrets from: * Files: using `ENC[file@/path/to/file]` * Kubernetes Secrets: using `ENC[k8s_secret@namespace/secret-name/key]` @@ -1819,7 +1821,9 @@ On Windows, your executable must: ## Refreshing secrets at runtime -Starting in Agent v7.67, you can configure the Agent to refresh resolved secrets without requiring a restart. +*Available in Agent version 7.67+* + +You can configure the Agent to refresh resolved secrets without requiring a restart. Set a refresh interval: ```yaml @@ -1854,7 +1858,9 @@ secret_refresh_scatter: false ``` ### Autodiscovery check secrets refresh -Starting in Agent v7.76, scheduled [Autodiscovery][1] checks can refresh secrets at runtime if the template uses the `ENC[]` syntax. +*Available in Agent version 7.76+* + +Scheduled [Autodiscovery][1] checks can refresh secrets at runtime if the template uses the `ENC[]` syntax. ```yaml labels: @@ -1881,7 +1887,9 @@ The Agent can then trigger secrets refresh at either the interval set in `secret ### Automatic secrets refresh on API key failure / invalidation -Starting in Agent version v7.74, the Agent can automatically refresh secrets when it detects an invalid API key. This happens when the Agent receives a 403 Forbidden response from Datadog or when the periodic health check detects an invalid or expired API key. +*Available in Agent version 7.74+* + +The Agent can automatically refresh secrets when it detects an invalid API key. This happens when the Agent receives a 403 Forbidden response from Datadog or when the periodic health check detects an invalid or expired API key. To enable this feature, set `secret_refresh_on_api_key_failure_interval` to an interval in minutes in your `datadog.yaml` file. Set to `0` to disable (default).