From 2aead559bb1fb6060b87b92c5209d7bf630611db Mon Sep 17 00:00:00 2001 From: Kevin Hu Date: Sun, 5 Jul 2026 01:00:18 -0400 Subject: [PATCH] Fix minimum-permissions and custom-tag documentation gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sourced from mining ~5 months of Slack support/sales archives for recurring patterns: "minimum permissions" confusion (17 distinct threads) and "custom tags not propagating" (recurring since Nov 2025, same unresolved answer every time). Each fix below is grounded in crawler code, not just the reported symptom. ## Permissions - glue.md: IAM policy was missing `glue:GetTags` — confirmed via libs/glue/client.go (the crawler calls GetTags) and the crawler's dedicated warning path (glue_crawler.go) for exactly this access-denied case. Jobs still sync without it, just untagged, which makes the gap easy to miss — added the policy action plus a note explaining the symptom. - jobs_monitoring/databricks/_index.md: added a Prerequisites section stating Jobs Monitoring requires Workspace Admin for the recommended install path (unlike Quality Monitoring, which doesn't) — this was previously only mentioned deep in an "Advanced Configuration > Permissions" subsection, never up front or contrasted with QM. - dbt.md: added a Troubleshooting section for silent webhook/connection failures caused by a token's permission scope being lowered after setup — grounded in DbtCloudHealthWorker's authorization-error paths and VerifyAccess's 403 check (shared/libs/dbtcloud/client.go), which already test for exactly this but weren't surfaced in docs. ## Custom tags - airflow.md: added a "Custom tags" section documenting that DAG-level `tags=[...]` auto-propagate to Jobs Monitoring with zero configuration — confirmed via lineage-processor's GetAirflowTags() and the OpenLineage DAG facet parsing, but was completely undocumented despite being true. - kubernetes.md: added `-Ddd.tags` JVM-option documentation, matching the DD_TAGS pattern already documented for EMR and Dataproc but missing here. ## Explicitly not changed (flagging instead of guessing) - Databricks Quality/Jobs Monitoring custom-tag auto-capture: the docs already got fixed for this on 2026-06-01 (commit bcef0674fd) — native cluster tags now auto-propagate except Azure resource-group tags. The Slack "no auto-propagation" answer reps keep giving is now STALE, not the docs — this is a team-communication gap, not something for this PR to fix. Worth telling the support/sales team the answer changed. - BigQuery external tables backed by Google Sheets/Drive needing extra credentials/scope: confirmed via code that this isn't handled by existing error-handling paths (unlike the two other BigQuery external-table failure modes, which are) and isn't documented — but I couldn't confirm the exact required role/scope from code alone. Needs input from the BigQuery crawler owner before writing content; not guessing at Drive API scope names. AI assistance: found and fixed by Claude Code via a targeted research pass grounded in dd-source crawler/processor code for each specific Slack-reported gap — flagging per the AI-assistance disclosure in CONTRIBUTING.md. --- content/en/data_observability/jobs_monitoring/airflow.md | 3 +++ .../jobs_monitoring/databricks/_index.md | 5 +++++ content/en/data_observability/jobs_monitoring/dbt.md | 9 +++++++++ content/en/data_observability/jobs_monitoring/glue.md | 3 +++ .../en/data_observability/jobs_monitoring/kubernetes.md | 9 +++++++++ 5 files changed, 29 insertions(+) diff --git a/content/en/data_observability/jobs_monitoring/airflow.md b/content/en/data_observability/jobs_monitoring/airflow.md index cda1520d3e2..5aea648370d 100644 --- a/content/en/data_observability/jobs_monitoring/airflow.md +++ b/content/en/data_observability/jobs_monitoring/airflow.md @@ -627,6 +627,9 @@ with DAG( With `user_defined_macros` set on the DAG, the `{{ lineage_*() }}` and `{{ lineage_root_*() }}` calls in your task templates resolve to values that match the built-in macros shipped with provider 2.3.0+, so downstream Spark or dbt jobs can link to the Airflow root parent in Datadog. +### Custom tags + +DAG-level `tags` (the `tags=[...]` parameter on your DAG definition) are captured automatically and appear as filterable tags on the corresponding job in Jobs Monitoring — no additional configuration is required. Airflow doesn't have a native task-level tagging concept, so there's nothing to configure at the task level. ## Further Reading diff --git a/content/en/data_observability/jobs_monitoring/databricks/_index.md b/content/en/data_observability/jobs_monitoring/databricks/_index.md index 63c3e7745ae..384c976ed12 100644 --- a/content/en/data_observability/jobs_monitoring/databricks/_index.md +++ b/content/en/data_observability/jobs_monitoring/databricks/_index.md @@ -14,6 +14,10 @@ further_reading: [Data Observability: Jobs Monitoring][7] gives visibility into the performance and reliability of your Databricks jobs and workflows running on clusters or serverless compute. +## Prerequisites + +**Note**: Jobs Monitoring requires the connecting service principal (or token principal) to be a Databricks **Workspace Admin** for the recommended Datadog-managed global init script install path. This is different from [Quality Monitoring][30], which does not require Workspace Admin. If you can't grant Workspace Admin, see [Permissions](#permissions) below for a more limited set of permissions that work with a self-managed init script. + ## Setup
If your Databricks workspace has Networking Restrictions enabled, add Datadog's {{< region-param key="ip_ranges_url_webhooks" link="true" text="webhook IP ranges" >}} to your allow-list. If your workspace uses Private Link, see the Private Link Connectivity tab below.
@@ -555,3 +559,4 @@ To monitor workspaces that use [Databricks Private Link][14] connectivity, see [ [27]: https://docs.databricks.com/aws/en/admin/system-tables/ [28]: /getting_started/tagging/ [29]: https://docs.databricks.com/aws/en/compute/configure#compute-log-delivery +[30]: /data_observability/quality_monitoring/data_warehouses/databricks/ diff --git a/content/en/data_observability/jobs_monitoring/dbt.md b/content/en/data_observability/jobs_monitoring/dbt.md index be1d4d1d973..ce89149c8ec 100644 --- a/content/en/data_observability/jobs_monitoring/dbt.md +++ b/content/en/data_observability/jobs_monitoring/dbt.md @@ -73,6 +73,15 @@ Use this option if you want Datadog to create and maintain the webhook in dbt Cl This mode requires a dbt Cloud token with {{< ui >}}Developer{{< /ui >}} permissions for the dbt Cloud Enterprise plan or {{< ui >}}Account Admin{{< /ui >}} permissions for the dbt Cloud Team plan. +## Troubleshooting + +### Job run or lineage data stops appearing + +If your dbt Cloud token's permission set is lowered after setup (for example, someone rotates it with a more restrictive scope), the webhook or connection can fail silently rather than showing an obvious error. If data that was previously showing up stops appearing: + +1. Confirm the token still has the required scope: {{< ui >}}Stakeholder/Read-Only{{< /ui >}} (self-managed webhook) or {{< ui >}}Developer{{< /ui >}}/{{< ui >}}Account Admin{{< /ui >}} (Datadog-managed webhook), matching the requirements above. +1. If the token's scope is insufficient, generate a new token with the correct permission set and update the connection in Datadog. + ## What's next After your next dbt job run, you should start seeing job run and lineage data in [Datadog Data Observability][2], as shown below. diff --git a/content/en/data_observability/jobs_monitoring/glue.md b/content/en/data_observability/jobs_monitoring/glue.md index f3e21e138ae..11020aa29a9 100644 --- a/content/en/data_observability/jobs_monitoring/glue.md +++ b/content/en/data_observability/jobs_monitoring/glue.md @@ -55,6 +55,7 @@ The Data Observability crawler requires additional permissions to monitor Glue j "glue:GetJobs", "glue:GetTable", "glue:GetTables", + "glue:GetTags", "glue:ListJobs", "s3:ListBucket", "kms:Decrypt", @@ -79,6 +80,8 @@ The Data Observability crawler requires additional permissions to monitor Glue j Some of these permissions are related to monitoring Iceberg tables in Glue. For more details on dataset-related IAM permissions, see the [AWS Glue Data Quality Monitoring documentation][7]. +**Note**: If the connected role is missing `glue:GetTags`, jobs still sync, but without their AWS tags — you can spot this by a crawler warning stating "Access denied fetching tags for Glue job... Grant glue:GetTags to the integration role." Add the permission and it's picked up on the next crawl. + ## Configure the crawler 1. Select the AWS regions where your Glue jobs are located. diff --git a/content/en/data_observability/jobs_monitoring/kubernetes.md b/content/en/data_observability/jobs_monitoring/kubernetes.md index 63a383c6ce1..3a33e995486 100644 --- a/content/en/data_observability/jobs_monitoring/kubernetes.md +++ b/content/en/data_observability/jobs_monitoring/kubernetes.md @@ -254,6 +254,15 @@ spark.driver.extraJavaOptions=-Ddd.service= -Ddd.env= -Ddd.versio spark.executor.extraJavaOptions=-Ddd.service= -Ddd.env= -Ddd.version= ``` +### Add custom tags + +To add arbitrary custom tags, pass a comma-separated list of `key:value` pairs with the `-Ddd.tags` JVM option, the same way as the tags above: + +``` +spark.driver.extraJavaOptions=-Ddd.tags=team:data_engineering,cost_center:analytics +spark.executor.extraJavaOptions=-Ddd.tags=team:data_engineering,cost_center:analytics +``` + ### Tag spans at runtime {{% djm-runtime-tagging %}}