From 1107a4db4959fd416bd39808b2ce23202494e15d Mon Sep 17 00:00:00 2001 From: Kevin Hu Date: Sat, 4 Jul 2026 23:16:00 -0400 Subject: [PATCH 1/2] Add Optimizer (query and job recommendations) documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Optimizer had a full customer-facing surface (JobsRecommendationsPage, per-job/cluster recommendation side panels, a public /api/ui/data-observability-optimizer/recommendations API with a FOR_REVIEW/REVIEWED/IGNORED/RESOLVED status lifecycle, and a "Fix with Bits" AI-assisted-fix action) but zero public documentation, despite being alluded to in the top-level DO overview page's "Optimize cost & performance" bullet. Marked Preview per the DO Product Features tracker: individual recommenders (Snowflake, Databricks SQL, Trino) are gated behind their own feature flags plus a Balto org allowlist, and notifications are separately gated off by default. Deliberately did not claim a specific top-level navigation breadcrumb — confirmed evidence shows recommendations surface inline in Jobs Monitoring job/cluster tables via a side panel (RecommendationsSidePanel, RecommendationsSection), not a fully independent standalone page reachable from primary nav. Flagged as an open item to verify the exact entry point against the live product. AI assistance: drafted by Claude Code, grounded in data-observability-optimizer (backend) and the data-jobs app's Recommendations components (frontend). --- content/en/data_observability/_index.md | 4 +++ content/en/data_observability/optimizer.md | 38 ++++++++++++++++++++++ 2 files changed, 42 insertions(+) create mode 100644 content/en/data_observability/optimizer.md diff --git a/content/en/data_observability/_index.md b/content/en/data_observability/_index.md index a0c23e04d00..0b6b09612e1 100644 --- a/content/en/data_observability/_index.md +++ b/content/en/data_observability/_index.md @@ -17,6 +17,9 @@ further_reading: - link: '/data_observability/cicd/' tag: 'Documentation' text: 'CI/CD' + - link: '/data_observability/optimizer/' + tag: 'Documentation' + text: 'Optimizer' - link: 'https://www.datadoghq.com/about/latest-news/press-releases/datadog-metaplane-aquistion/' tag: 'Blog' text: 'Datadog Brings Observability to Data teams by Acquiring Metaplane' @@ -42,4 +45,5 @@ Data Observability (DO) helps data teams improve the reliability of data for ana {{< nextlink href="/data_observability/quality_monitoring/" >}}Quality Monitoring: Identify data issues before downstream BI and AI applications are impacted.{{< /nextlink >}} {{< nextlink href="/data_observability/jobs_monitoring/" >}}Jobs Monitoring: Observe, troubleshoot, and optimize jobs across your data pipelines.{{< /nextlink >}} {{< nextlink href="/data_observability/cicd/" >}}CI/CD: Prevent data-quality issues before they're merged.{{< /nextlink >}} + {{< nextlink href="/data_observability/optimizer/" >}}Optimizer: Get recommendations to reduce cost and improve performance of jobs and queries.{{< /nextlink >}} {{< /whatsnext >}} diff --git a/content/en/data_observability/optimizer.md b/content/en/data_observability/optimizer.md new file mode 100644 index 00000000000..0fc03c323c9 --- /dev/null +++ b/content/en/data_observability/optimizer.md @@ -0,0 +1,38 @@ +--- +title: Optimizer +description: Get recommendations to reduce the cost and improve the performance of Spark and Databricks jobs, and Snowflake, BigQuery, and Trino queries. +further_reading: + - link: '/data_observability/' + tag: 'Documentation' + text: 'Data Observability Overview' + - link: '/data_observability/jobs_monitoring/' + tag: 'Documentation' + text: 'Jobs Monitoring' +--- + +
Optimizer is in preview. Contact your Datadog representative or support to request access.
+ +## Overview + +Optimizer analyzes your Spark and Databricks jobs, and your Snowflake, BigQuery, and Trino queries, to surface specific recommendations for reducing cost and improving performance — for example, right-sizing cluster configuration or rewriting an inefficient query. + +Each recommendation includes an estimated cost or duration impact. Some job recommendations can be applied directly as a code change through Bits AI. + +## View recommendations + +Recommendations for jobs and clusters appear inline in [Jobs Monitoring][1] job and cluster tables. Select a job or cluster with an available recommendation to open its recommendation details in a side panel. + +Each recommendation has a status you can update as you triage it: + +| Status | Description | +|---|---| +| {{< ui >}}For Review{{< /ui >}} | The default status for a new recommendation. | +| {{< ui >}}Reviewed{{< /ui >}} | You've looked at the recommendation but haven't acted on it yet. | +| {{< ui >}}Ignored{{< /ui >}} | You've decided not to act on this recommendation. | +| {{< ui >}}Resolved{{< /ui >}} | You've applied the recommendation. | + +For recommendations that include a code change, click {{< ui >}}Fix with Bits{{< /ui >}} to have Bits AI apply the change for you. + +**Note**: Whether Optimizer generates recommendations for your Snowflake, BigQuery, Databricks, or Trino workloads depends on which recommenders your organization has enabled — contact your Datadog representative to confirm which are active for your account. + +[1]: /data_observability/jobs_monitoring/ From 51ef964c445beded030acfeb3e9b27a658c53e62 Mon Sep 17 00:00:00 2001 From: Kevin Hu Date: Sun, 5 Jul 2026 01:08:09 -0400 Subject: [PATCH 2/2] Add Optimizer to the Data Observability sidebar navigation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Same class of bug just found and fixed for Airbyte: this PR's new top-level page and its _index.md nextlink were correctly added, but the separate sidebar-nav config (config/_default/menus/main.en.yaml) was not — this branch predates discovering that file exists. Added a top-level entry as a sibling of Quality Monitoring / Jobs Monitoring. --- config/_default/menus/main.en.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 69ba6d57d55..b4583eeea84 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -5387,6 +5387,11 @@ menu: identifier: data_jobs parent: data_observability_heading weight: 80000 + - name: Optimizer + url: data_observability/optimizer/ + identifier: data_observability_optimizer + parent: data_observability_heading + weight: 100000 - name: Databricks url: data_observability/jobs_monitoring/databricks identifier: jobs_monitoring_databricks