[WIP] Document Job, Schema Change, and Source-to-Target monitor types

content/en/monitors/types/data_observability.md

@@ -1,13 +1,16 @@
 ---
 title: Data Observability Monitor
-description: "Monitor freshness, row count, column-level metrics, and custom SQL queries across your data warehouses."
+description: "Monitor freshness, row count, column-level metrics, and custom SQL queries across your data warehouses, and monitor data job run failures."
 further_reading:
   - link: "/data_observability/"
     tag: "Documentation"
     text: "Data Observability Overview"
   - link: "/data_observability/quality_monitoring/"
     tag: "Documentation"
     text: "Quality Monitoring"
+  - link: "/data_observability/jobs_monitoring/"
+    tag: "Documentation"
+    text: "Jobs Monitoring"
   - link: "/monitors/notify/"
     tag: "Documentation"
     text: "Configure your monitor notifications"
@@ -271,6 +274,88 @@
 {{% /tab %}}
 {{< /tabs >}}
 
+## Job monitors
+
+<div class="alert alert-info">Job monitors are in preview. Contact your Datadog representative or <a href="/help/">support</a> to request access.</div>
+
+Job monitors alert you when a data job reaches a threshold of consecutive failed runs. Job monitors require [Jobs Monitoring][11] to be set up with at least one supported job source.
+
+### Monitor creation
+
+To create a job monitor, navigate to {{< ui >}}Monitors{{< /ui >}} > {{< ui >}}New Monitor{{< /ui >}} > {{< ui >}}Job{{< /ui >}}.
+
+### Choose jobs to monitor
+
+1. Select a technology from the {{< ui >}}Select a technology{{< /ui >}} menu: {{< ui >}}Airflow{{< /ui >}}, {{< ui >}}AWS Glue{{< /ui >}}, {{< ui >}}Azure Data Factory{{< /ui >}}, {{< ui >}}Databricks{{< /ui >}}, {{< ui >}}dbt{{< /ui >}} (Job, Model, or Test), {{< ui >}}Spark{{< /ui >}}, or {{< ui >}}Custom Job{{< /ui >}} for jobs that emit [custom OpenLineage events][12].
+    - If you select {{< ui >}}Custom Job{{< /ui >}}, also select the specific operation to monitor from the {{< ui >}}Select an operation{{< /ui >}} menu.
+1. Optionally, narrow the jobs matched by this monitor using the {{< ui >}}Filter by tags{{< /ui >}} field (for example, `env:prod`).
+
+The preview panel shows the job names and recent run statistics that currently match your selection.
+
+<!-- TODO(docs review): add a screenshot of the job-selection step (technology dropdown + tag filter + preview panel), light and dark mode. -->
+
+### Configure alert conditions
+
+1. Under {{< ui >}}Define alerting context{{< /ui >}}, optionally add {{< ui >}}Group by{{< /ui >}} tags. Datadog automatically groups by `job_name`, plus additional tags specific to the selected technology (for example, `workspace_name` for Databricks, `project` for dbt).
+1. Under {{< ui >}}Detection Method - Threshold{{< /ui >}}, set {{< ui >}}Trigger when{{< /ui >}} to the number of consecutive failed runs that should trigger an alert.
+
+The monitor automatically recovers after the next run succeeds.
+
+**Note**: Job monitors alert on consecutive run failures. They do not currently support alerting on job run duration.
+
+### Example notification
+
+{{< code-block lang="text" >}}
+{{#is_alert}}
+Databricks job **{{job_name.name}} ({{workspace_name.name}})** has reached {{value}} consecutive failures.
+
+Error: {{job.run_error}}
+
+[View in Databricks]({{job.run_url}})
+{{/is_alert}}
+{{#is_recovery}}
+Databricks job **{{job_name.name}}** recovered.
+{{/is_recovery}}
+{{< /code-block >}}
+
+## Schema change monitors
+
+<div class="alert alert-info">Schema change monitors are in preview. Contact your Datadog representative or <a href="/help/">support</a> to request access.</div>
+
+Schema change monitors alert you when a database, schema, table, or column is added, removed, renamed, or has its type changed.
+
+### Monitor creation
+
+To create a schema change monitor, navigate to {{< ui >}}Monitors{{< /ui >}} > {{< ui >}}New Monitor{{< /ui >}} > {{< ui >}}Schema Change{{< /ui >}}.
+
+### Choose data to monitor
+
+1. Select a warehouse from the {{< ui >}}Select warehouse{{< /ui >}} menu.
+1. Use {{< ui >}}Choose entities within the warehouse to monitor{{< /ui >}} to filter by `account`, `database`, `schema`, `table`, or `column`.
+
+### Alert conditions
+
+Schema change monitors alert on any matching change (there is currently no configurable threshold). Each alert reports the affected entity's fully-qualified name, the change type (`added`, `removed`, `renamed`, or `type_changed`), and — for renames and type changes — the old and new values.
+
+**Note**: You cannot yet configure a custom alert message template for schema change monitors; alerts use a standard format grouped by change type and entity.
+
+## Source-to-target monitors
+
+<div class="alert alert-info">Source-to-target monitors are in preview. Contact your Datadog representative or <a href="/help/">support</a> to request access.</div>
+
+Source-to-target monitors compare a metric computed on a source table or column against the same metric computed on a target table or column — for example, to confirm that a row count in a source system matches the row count after it lands in a warehouse. Use them to catch data loss or drift introduced during a transformation or copy step.
+
+### Monitor creation
+
+1. Navigate to {{< ui >}}Monitors{{< /ui >}} > {{< ui >}}New Monitor{{< /ui >}} > {{< ui >}}Data Quality{{< /ui >}}, then select {{< ui >}}Source to Target{{< /ui >}}.
+1. Choose a warehouse and a source table or column.
+1. Choose a second warehouse and a target table or column. The target must be the same entity type as the source (table-to-table or column-to-column).
+1. Select a metric type, following the same options as [Quality monitors](#select-your-metric-type).
+1. Choose a comparison format: {{< ui >}}Difference{{< /ui >}} (`target - source`) or {{< ui >}}Percent Difference{{< /ui >}} (`(target - source) / source`).
+1. Set an alert threshold using {{< ui >}}more than{{< /ui >}} or {{< ui >}}less than{{< /ui >}}.
+
+**Note**: Source and target metrics must be computed within 30 minutes of each other, or the comparison for that evaluation is skipped.
+
 ## Annotate bounds
 
 For monitors using the **Anomaly** detection method, you can annotate bound ranges to provide feedback and improve the model over time. Unlike infrastructure metrics, data quality metrics are often business-specific, so use annotations to teach the model what behavior is normal for your data.
@@ -300,3 +385,5 @@
 [8]: /monitors/configuration/?tab=thresholdalert#thresholds
 [9]: /help/
 [10]: /api/latest/data-observability/
+[11]: /data_observability/jobs_monitoring/
+[12]: /data_observability/jobs_monitoring/openlineage/