From 9614793f00985eb90c7d5873e29e8e088ba6c7e0 Mon Sep 17 00:00:00 2001 From: Toma Date: Mon, 22 Jun 2026 15:47:36 -0400 Subject: [PATCH 1/6] Document SCIM role provisioning (#37659) * Document SCIM role provisioning Add role provisioning capability to the main SCIM page, Okta role attribute mapping instructions, and the roles attribute to the Entra attribute-mapping table. Co-Authored-By: Claude Opus 4.8 (1M context) * Move role provisioning behavior after prerequisites; trim capabilities Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- content/en/account_management/scim/_index.md | 15 +++++++++-- content/en/account_management/scim/entra.md | 4 +++ content/en/account_management/scim/okta.md | 26 ++++++++++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) diff --git a/content/en/account_management/scim/_index.md b/content/en/account_management/scim/_index.md index 75c8439e71e..1d9f69a824f 100644 --- a/content/en/account_management/scim/_index.md +++ b/content/en/account_management/scim/_index.md @@ -22,11 +22,11 @@ The System for Cross-domain Identity Management, or [SCIM][9], is an open standa ### Supported capabilities -- Create users in Datadog (Email verification is required for first login, see [email verification][1]) +- Create users in Datadog - Remove users in Datadog when they no longer require access -- Keep user attributes synchronized between the identity provider and Datadog - Single sign-on to Datadog (recommended) - Managed Teams: Create Datadog Teams from identity provider groups and keep membership of the Datadog Teams synchronized with group membership in the identity provider. +- Role provisioning: Provision a user's Datadog role (built-in or custom) from an identity provider attribute, and keep it synchronized. When the attribute changes in your identity provider, the user's Datadog role updates in real time. Datadog implements the SCIM server protocol. Datadog supports using SCIM with the Microsoft Entra ID and Okta identity providers. Other identity providers may work, but are not explicitly supported. @@ -44,6 +44,14 @@ Datadog strongly recommends that you use a service account application key when When using SAML and SCIM together, Datadog strongly recommends disabling SAML just-in-time (JIT) provisioning to avoid discrepancies in access. Manage user provisioning through SCIM only. +### Role provisioning behavior + +When a SCIM request includes one or more roles, Datadog provisions only the roles that match a role in your organization. If none of the roles match, the user falls back to your organization's default role (Standard). Unmatched roles are logged to [Audit Trail][11]. + +SCIM is the source of truth for role assignment and takes precedence over [SAML role mappings][12]. SCIM role provisioning events are recorded in Audit Trail and as StatsD metrics. + +Roles follow the SCIM multi-valued attribute convention defined in [RFC 7643][13]. Both Okta and Microsoft Entra ID support this mapping natively, with no custom scripting required. For setup instructions, see the documentation for your identity provider. + ## Using a service account with SCIM To enable SCIM, you must use an [application key][5] to secure the connection between your identity provider and your Datadog account. A specific user or service account controls each application key. @@ -68,3 +76,6 @@ The service account requires at minimum the `user_access_invite` and `user_acces [8]: /help/ [9]: https://scim.cloud/ [10]: /api/latest/scim/ +[11]: /account_management/audit_trail/ +[12]: /account_management/saml/mapping/#map-saml-attributes-to-datadog-roles +[13]: https://www.rfc-editor.org/rfc/rfc7643.html#section-4.1.2 diff --git a/content/en/account_management/scim/entra.md b/content/en/account_management/scim/entra.md index 28bb696547e..9fa50e1f72c 100644 --- a/content/en/account_management/scim/entra.md +++ b/content/en/account_management/scim/entra.md @@ -78,11 +78,14 @@ When using SAML and SCIM together, Datadog strongly recommends disabling SAML ju | `jobTitle` | `title` | | `mail` | `emails[type eq "work"].value` | | `displayName` | `name.formatted` | +| `roles` | `roles` | {{< img src="/account_management/scim/ad-users-2.png" alt="Attribute mapping configuration, Provision Azure Active Directory Users">}} 7. After you set your mappings, click {{< ui >}}Save{{< /ui >}}. +To provision a user's Datadog role (built-in or custom), map the `roles` attribute as shown above. Roles follow the SCIM multi-valued attribute convention defined in [RFC 7643][9], using the role UUID as `value` and the role name as `display`. If a SCIM request sends multiple roles, Datadog provisions only the roles that match a role in your organization. If none match, the user falls back to the org default role (Standard), and unmatched roles are logged to Audit Trail. For more details, see [SCIM][1]. + ### Group attributes Group mapping is not supported. @@ -95,3 +98,4 @@ Group mapping is not supported. [6]: https://entra.microsoft.com/ [7]: https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/permissions-reference#cloud-application-administrator [8]: https://learn.microsoft.com/en-us/entra/identity/app-provisioning/application-provisioning-config-problem-scim-compatibility#flags-to-alter-the-scim-behavior +[9]: https://www.rfc-editor.org/rfc/rfc7643.html#section-4.1.2 diff --git a/content/en/account_management/scim/okta.md b/content/en/account_management/scim/okta.md index 1f3d6bce8a9..0aaf1494279 100644 --- a/content/en/account_management/scim/okta.md +++ b/content/en/account_management/scim/okta.md @@ -59,6 +59,30 @@ When using SAML and SCIM together, Datadog strongly recommends disabling SAML ju - {{< ui >}}Deactivate Users{{< /ui >}} 8. Under {{< ui >}}Datadog Attribute Mappings{{< /ui >}}, find the mapping of Okta attributes to Datadog attributes already pre-configured. You can re-map them if needed, but map the Okta values to the same set of Datadog values. +### Map the Datadog role attribute + +To provision a user's Datadog role (built-in or custom) through SCIM, add an explicit mapping for the `roles` attribute. Okta does not map this attribute by default. + +Datadog's SCIM role support follows the SCIM multi-valued attribute convention defined in [RFC 7643][8], using the role UUID as `value` and the role name as `display`: + +```json +{ + "roles": [ + { "value": "", "display": "" } + ] +} +``` + +1. In {{< ui >}}Directory{{< /ui >}} > {{< ui >}}Profile Editor{{< /ui >}}, select the Okta user profile, then click {{< ui >}}Add Attribute{{< /ui >}} to create a `roles` attribute: + - {{< ui >}}Data type{{< /ui >}}: **string** + - {{< ui >}}Display name{{< /ui >}}: **Roles** + - {{< ui >}}Variable name{{< /ui >}}: **roles** + - For {{< ui >}}Enum{{< /ui >}}, select {{< ui >}}Define enumerated list of values{{< /ui >}} and add one entry per Datadog role, using the role name as the display name and the role UUID as the value. You can find a role's UUID in the role's URL on your [Organization Settings][9] page. Add any custom roles the same way. +2. In your Datadog application's {{< ui >}}Provisioning{{< /ui >}} > {{< ui >}}To App{{< /ui >}} settings, map the Okta `roles` attribute to the Datadog `roles` attribute. +3. In the app's {{< ui >}}Assignments{{< /ui >}} tab, assign each user the appropriate role from the dropdown. + +If a SCIM request sends multiple roles, Datadog provisions only the roles that match a role in your organization. If none match, the user falls back to the org default role (Standard), and unmatched roles are logged to Audit Trail. For more details, see [SCIM][1]. + ## Configure automatic team provisioning With [Managed Teams][6], you control the core provisioning of a Datadog Team — its name, handle, and membership — through the identity provider. The setup process differs depending on whether the team already exists in Datadog. @@ -133,3 +157,5 @@ This procedure allows you to manage team membership in Datadog instead of Okta a [5]: /account_management/org_settings/service_accounts [6]: /account_management/teams/manage/#manage-teams-through-an-identity-provider [7]: https://app.datadoghq.com/teams +[8]: https://www.rfc-editor.org/rfc/rfc7643.html#section-4.1.2 +[9]: https://app.datadoghq.com/organization-settings/roles From 7422e9649317074c53fe0cb86ef3d5d475192f4e Mon Sep 17 00:00:00 2001 From: Nick Lopez <20211302+nlopez@users.noreply.github.com> Date: Mon, 22 Jun 2026 15:48:45 -0400 Subject: [PATCH 2/6] [DOCS-14689] Add US1 and AP2 PrivateLink VPC Endpoint Service IDs reference tables (#37476) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add AP2 VPC Endpoint Service IDs reference table to PrivateLink docs Co-authored-by: Pi * Move VPC Endpoint Service IDs heading outside site-region block for TOC visibility Co-authored-by: Pi * Fix VPC Endpoint Service IDs TOC visibility for AP2 only - Move heading back inside site-region ap2 block so hideTOCItems() hides the nav entry for US1/AP1 (selector targets direct children of d-none site-region-container) - Extend table-of-contents.html condition to trigger scraped-toc.html for pages using {{% site-region %}} (markdown shortcode), not just {{< site-region >}} (HTML shortcode) — scraped-toc was designed for exactly this case Co-authored-by: Pi * Revert table-of-contents.html site-region condition change Co-authored-by: Pi * Replace 'CNAMEs' with 'maps to' per PR feedback Co-authored-by: Pi * [DOCS-14689] Add US1 PrivateLink VPC Endpoint Service IDs reference table (#37489) * [DOCS-14689] Add US1 PrivateLink VPC Endpoint Service IDs reference table Co-authored-by: Pi * Add orchid (api/synthetics) to US1 PrivateLink table Co-authored-by: Pi * Remove duplicate inline table from cross-region tab; keep single VPC Endpoint Service IDs section Co-authored-by: Pi * Remove legacy api.datadoghq.com --- entry; orchid color anchor confirmed in public DNS Co-authored-by: Pi --------- Co-authored-by: Pi * Remove redundant US1 PrivateLink service names link; update table anchor to vpc-endpoint-service-ids Co-authored-by: Pi * Apply suggestions from code review Co-authored-by: Rosa Trieu <107086888+rtrieu@users.noreply.github.com> * Update content/en/agent/guide/private-link.md Co-authored-by: Rosa Trieu <107086888+rtrieu@users.noreply.github.com> --------- Co-authored-by: Pi Co-authored-by: Rosa Trieu <107086888+rtrieu@users.noreply.github.com> --- content/en/agent/guide/private-link.md | 165 +++++++++++++++++++++++++ 1 file changed, 165 insertions(+) diff --git a/content/en/agent/guide/private-link.md b/content/en/agent/guide/private-link.md index 66f5259fa53..20b055ea2a6 100644 --- a/content/en/agent/guide/private-link.md +++ b/content/en/agent/guide/private-link.md @@ -52,6 +52,7 @@ Datadog exposes AWS PrivateLink endpoints in **{{< region-param key="aws_region" {{< img src="agent/guide/private_link/vpc_service_name.png" alt="VPC service name" style="width:70%;" >}} +{{% site-region region="ap1" %}} | Datadog | PrivateLink service name | Private DNS name | |---------------------------|----------------------------------------------------------------------------------------|------------------------------------------------------------------------| | Logs (Agent HTTP intake) | {{< region-param key="aws_private_link_logs_agent_service_name" code="true" >}} | {{< region-param key="agent_http_endpoint_private_link" code="true" >}} | @@ -64,6 +65,13 @@ Datadog exposes AWS PrivateLink endpoints in **{{< region-param key="aws_region" | Traces | {{< region-param key="aws_private_link_traces_service_name" code="true" >}} | {{< region-param key="traces_endpoint_private_link" code="true" >}} | | Database Monitoring | {{< region-param key="aws_private_link_dbm_service_name" code="true" >}} | {{< region-param key="dbm_endpoint_private_link" code="true" >}} | | Remote Configuration | {{< region-param key="aws_private_link_remote_config_service_name" code="true" >}} | {{< region-param key="remote_config_endpoint_private_link" code="true" >}} | +{{% /site-region %}} +{{% site-region region="us" %}} +For the complete list of US1 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} +{{% site-region region="ap2" %}} +For the complete list of AP2 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} 4. Click {{< ui >}}Verify{{< /ui >}}. If this does not return _Service name found_, reach out to [Datadog support][14]. 5. Choose the VPC and subnets that should be peered with the Datadog VPC service endpoint. @@ -132,6 +140,7 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can ## PrivateLink service names +{{% site-region region="ap1" %}} | Datadog | PrivateLink service name | Private DNS name | |---------------------------|----------------------------------------------------------------------------------------|------------------------------------------------------------------------| | Logs (Agent HTTP intake) | {{< region-param key="aws_private_link_logs_agent_service_name" code="true" >}} | {{< region-param key="agent_http_endpoint_private_link" code="true" >}} | @@ -144,6 +153,14 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can | Traces | {{< region-param key="aws_private_link_traces_service_name" code="true" >}} | {{< region-param key="traces_endpoint_private_link" code="true" >}} | | Database Monitoring | {{< region-param key="aws_private_link_dbm_service_name" code="true" >}} | {{< region-param key="dbm_endpoint_private_link" code="true" >}} | | Remote Configuration | {{< region-param key="aws_private_link_remote_config_service_name" code="true" >}} | {{< region-param key="remote_config_endpoint_private_link" code="true" >}} | +{{% /site-region %}} +{{% site-region region="ap2" %}} +For the complete list of AP2 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} + +{{% site-region region="us" %}} +For the complete list of US1 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} **Note**: Cross-region PrivateLink doesn't emit CloudWatch metrics. See [CloudWatch metrics for AWS PrivateLink][2] for more information. @@ -161,6 +178,7 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can {{< img src="agent/guide/private_link/vpc_service_name.png" alt="VPC service name" style="width:90%;" >}} +{{% site-region region="ap1" %}} | Datadog | PrivateLink service name | |---------------------------|----------------------------------------------------------------------------------------| | Logs (Agent HTTP intake) | {{< region-param key="aws_private_link_logs_agent_service_name" code="true" >}} | @@ -173,6 +191,13 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can | Traces | {{< region-param key="aws_private_link_traces_service_name" code="true" >}} | | Database Monitoring | {{< region-param key="aws_private_link_dbm_service_name" code="true" >}} | | Remote Configuration | {{< region-param key="aws_private_link_remote_config_service_name" code="true" >}} | +{{% /site-region %}} +{{% site-region region="us" %}} +For the complete list of US1 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} +{{% site-region region="ap2" %}} +For the complete list of AP2 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} 4. Click {{< ui >}}Verify{{< /ui >}}. If this does not return _Service name found_, reach out to [Datadog support][1]. @@ -200,6 +225,7 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can Use the list below to map service and DNS name to different parts of Datadog: +{{% site-region region="ap1" %}} | Datadog | PrivateLink service name | Private DNS name | |---------------------------|----------------------------------------------------------------------------------------|------------------------------------------------------------------------| | Logs (Agent HTTP intake) | {{< region-param key="aws_private_link_logs_agent_service_name" code="true" >}} | {{< region-param key="agent_http_endpoint_private_link" code="true" >}} | @@ -212,6 +238,13 @@ Use the list below to map service and DNS name to different parts of Datadog: | Traces | {{< region-param key="aws_private_link_traces_service_name" code="true" >}} | {{< region-param key="traces_endpoint_private_link" code="true" >}} | | Database Monitoring | {{< region-param key="aws_private_link_dbm_service_name" code="true" >}} | {{< region-param key="dbm_endpoint_private_link" code="true" >}} | | Remote Configuration | {{< region-param key="aws_private_link_remote_config_service_name" code="true" >}} | {{< region-param key="remote_config_endpoint_private_link" code="true" >}} | +{{% /site-region %}} +{{% site-region region="us" %}} +For the complete list of US1 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} +{{% site-region region="ap2" %}} +For the complete list of AP2 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} You can also find this information by interrogating the AWS API, `DescribeVpcEndpointServices`, or by using the following command: @@ -303,6 +336,138 @@ The VPCs with Private Hosted Zone (PHZ) attached need to have a couple of settin [16]: /agent/logs/?tab=tailexistingfiles#send-logs-over-https [17]: https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html +{{% site-region region="us" %}} +## VPC Endpoint Service IDs + +US1 uses a two-level DNS architecture for PrivateLink. Every customer-facing DNS record maps to a dedicated `color.intake.datadoghq.com` VPC endpoint address. Setting up a VPC endpoint for a given anchor address covers all customer-facing records that map to it. + +Use the following table to identify which VPC endpoints to set up for the Datadog features you use. Wildcard entries match any subdomain not otherwise listed. + +**Note**: In the table below, `---` indicates a direct VPC service endpoint with no intermediate anchor address. +| Name | Anchor | VPC Endpoint Service ID | +|---|---|---| +| `webhook-intake.datadoghq.com` | `azure.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-02bee2072b5c3c226` | +| `webhooks-http-intake.logs.datadoghq.com` | `azure.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-02bee2072b5c3c226` | +| `*.integrations.otlp.datadoghq.com` | `brown.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-00192e92115cbcc75` | +| `opamp.datadoghq.com` | `brown.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-00192e92115cbcc75` | +| `otlp.datadoghq.com` | `brown.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-00192e92115cbcc75` | +| `mcp.datadoghq.com` | `cornsilk.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-058a75ceea85a9175` | +| `agenthealth-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `ci-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `cicodescan-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `citestcov-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `citestcycle-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `cloudplatform-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `contimage-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `contlcycle-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `cws-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `debugger-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `error-tracking-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `event-management-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `event-platform-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `feed-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `instrumentation-telemetry-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `kubeops-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `llmobs-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `ndm-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `ndmflow-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `netpath-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `ocimetrics-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `resources-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `sbom-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `sds-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `sentry-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `snmp-traps-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `softinv-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `iam-rum-intake.datadoghq.com` | `gray.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0a3b2d86676122d8d` | +| `rum-http-intake.logs.datadoghq.com` | `gray.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0a3b2d86676122d8d` | +| `rum.browser-intake-datadoghq.com` | `gray.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0a3b2d86676122d8d` | +| `data-obs-intake.datadoghq.com` | `lime.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ee865cd1c0a7ba32` | +| `trace.agent.datadoghq.com` | `lime.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ee865cd1c0a7ba32` | +| `network-devices.datadoghq.com` | `olive.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-05e3bfec4501e714d` | +| `*.api.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `*.synthetics.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `api.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `intake.synthetics.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `synthetics.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `agent-http-intake.logs.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-025a56b9187ac1f63` | +| `http-intake.logs.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-0e36256cb6172439d` | +| `metrics.agent.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-09a8006e245d1e7b8` | +| `orchestrator.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ad5fb9e71f85fe99` | +| `process.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ed1f789ac6b0bde1` | +| `intake.profile.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-022ae36a7b2472029` | +| `dbm-metrics-intake.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ce70d55ec4af8501` | +| `config.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-01f21309e507e3b1d` | + +{{% /site-region %}} + +{{% site-region region="ap2" %}} +## VPC Endpoint Service IDs + +AP2 uses a two-level DNS architecture for PrivateLink. Every customer-facing DNS record maps to a dedicated `color.intake.ap2.datadoghq.com` VPC endpoint address. Setting up a VPC endpoint for a given anchor address covers all customer-facing records that map to it. + +Use the following table to identify which VPC endpoints to set up for the Datadog features you use. More specific DNS records take precedence over wildcards—for example, `trace.agent.ap2.datadoghq.com` resolves to `lime.intake.ap2.datadoghq.com` even though `*.agent.ap2.datadoghq.com` points to `beige.intake.ap2.datadoghq.com`. + +| Name | Anchor | VPC Endpoint Service ID | +|---|---|---| +| `gcp-intake.logs.ap2.datadoghq.com` | `aqua.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-01b61a61d21fc7273` | +| `*.agent.ap2.datadoghq.com` | `beige.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06a30d6a016b746ff` | +| `agent.ap2.datadoghq.com` | `beige.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06a30d6a016b746ff` | +| `process.ap2.datadoghq.com` | `bisque.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0c26ca335d93a68b5` | +| `*.integrations.otlp.ap2.datadoghq.com` | `brown.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-04c61207a01a73496` | +| `opamp.ap2.datadoghq.com` | `brown.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-04c61207a01a73496` | +| `otlp.ap2.datadoghq.com` | `brown.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-04c61207a01a73496` | +| `agenthealth-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `awsmetrics-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `ci-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cicodescan-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cireport-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `citestcov-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `citestcycle-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cloudplatform-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `contimage-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `contlcycle-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cspm-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cws-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `debugger-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `error-tracking-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `event-management-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `instrumentation-telemetry-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `intake.profile.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `kubeops-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `llmobs-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `ndm-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `ndmflow-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `netpath-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `ocimetrics-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `resources-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `sbom-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `sds-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `sentry-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `snmp-traps-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `softinv-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `webhook-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `agent-http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `aws-kinesis-http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `eventbridge-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `lambda-http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `obpipeline-intake.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `runtime-security-http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `live.logs.ap2.datadoghq.com` | `indigo.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0545109555aa68e7e` | +| `data-obs-intake.ap2.datadoghq.com` | `lime.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0f3e01f4180b2ae09` | +| `trace.agent.ap2.datadoghq.com` | `lime.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0f3e01f4180b2ae09` | +| `orchestrator.ap2.datadoghq.com` | `linen.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-031da3ffac78ef902` | +| `*.synthetics.ap2.datadoghq.com` | `orchid.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06ec78b291ce8020a` | +| `api.ap2.datadoghq.com` | `orchid.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06ec78b291ce8020a` | +| `quota.browser-intake-ap2-datadoghq.com` | `orchid.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06ec78b291ce8020a` | +| `synthetics.ap2.datadoghq.com` | `orchid.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06ec78b291ce8020a` | +| `sourcemap-intake.ap2.datadoghq.com` | `plum.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-028e4348e80fa73f5` | +| `config.ap2.datadoghq.com` | `violet.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-01f8f80f4cb97bd10` | +| `dbm-metrics-intake.ap2.datadoghq.com` | `white.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-094469ee7a178f448` | +| `dbquery-intake.ap2.datadoghq.com` | `white.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-094469ee7a178f448` | +{{% /site-region %}} + {{% /site-region %}} ## Verify that data is being sent using PrivateLink From a43c365ce4a28fc97b1d6efaf1cd91fa4110a5fb Mon Sep 17 00:00:00 2001 From: arti-arutiunov Date: Mon, 22 Jun 2026 16:09:26 -0400 Subject: [PATCH 3/6] Removed legacy app launch metric (#37667) Added time to initial display and time to full display metrics --- .../en/real_user_monitoring/rum_without_limits/metrics.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/en/real_user_monitoring/rum_without_limits/metrics.md b/content/en/real_user_monitoring/rum_without_limits/metrics.md index 4c98f598cb1..ca2e0b4b1b5 100644 --- a/content/en/real_user_monitoring/rum_without_limits/metrics.md +++ b/content/en/real_user_monitoring/rum_without_limits/metrics.md @@ -29,7 +29,8 @@ Datadog provides the below out-of-the-box metrics for a comprehensive overview o | Metric Name | Description | Dimensions | Platform | |-------------|-------------|------------|----------| -| `rum.measure.app.startup_time` | App startup time | Default, Percentiles breakdown | Mobile only | +| `rum.measure.app.startup_to_full_display` | Time to full display during application launch | Default, Percentiles breakdown | Mobile only | +| `rum.measure.app.startup_to_initial_display` | Time to initial display during application launch | Default, Percentiles breakdown | Mobile only | | `rum.measure.error` | Count of errors | Default, Is Crash, View Name | Mobile & Browser | | `rum.measure.error.anr` | Count of ANRs (an Android freeze) | Default, Is Crash, View Name | Mobile only | | `rum.measure.error.hang` | Count of hangs (an iOS freeze) | Default | Mobile only | @@ -84,4 +85,4 @@ Metrics can be managed through [APIs][3] or Datadog's dedicated [Terraform modul [1]: https://app.datadoghq.com/rum/performance-monitoring [2]: /real_user_monitoring/platform/generate_metrics/ [3]: /api/latest/rum-metrics/ -[4]: https://registry.terraform.io/providers/DataDog/datadog/3.60.0/docs/resources/rum_metric \ No newline at end of file +[4]: https://registry.terraform.io/providers/DataDog/datadog/3.60.0/docs/resources/rum_metric From 75621f1a3bb5db36920ebe09641ccdff4ad9428c Mon Sep 17 00:00:00 2001 From: Devin Marsh Date: Mon, 22 Jun 2026 16:18:34 -0400 Subject: [PATCH 4/6] Add full job lifecycle examples to custom OpenLineage jobs docs (#37414) * Add full job lifecycle examples to custom OpenLineage jobs docs Co-Authored-By: Claude Sonnet 4.6 * Add note explaining why step 2 is optional * Clarify the note about optional for step 2 * Clarify optional RUNNING event note Co-Authored-By: Claude Opus 4.8 (1M context) * Highlight product capabilities in overview Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Sonnet 4.6 --- .../jobs_monitoring/openlineage/_index.md | 386 ++++++++++++++++-- 1 file changed, 349 insertions(+), 37 deletions(-) diff --git a/content/en/data_observability/jobs_monitoring/openlineage/_index.md b/content/en/data_observability/jobs_monitoring/openlineage/_index.md index 09872dab651..c54830bb710 100644 --- a/content/en/data_observability/jobs_monitoring/openlineage/_index.md +++ b/content/en/data_observability/jobs_monitoring/openlineage/_index.md @@ -2,19 +2,25 @@ title: Custom Jobs using OpenLineage description: "Monitor jobs from in-house tools, custom pipelines, and orchestrators that don't have native Datadog integrations." further_reading: - - link: '/data_observability/' - tag: 'Documentation' - text: 'Data Observability Overview' - - link: '/data_observability/jobs_monitoring/openlineage/datadog_agent_for_openlineage/' - tag: 'Documentation' - text: 'Set up Datadog Agent for OpenLineage Proxy' + - link: '/data_observability/' + tag: 'Documentation' + text: 'Data Observability Overview' + - link: '/data_observability/jobs_monitoring/openlineage/datadog_agent_for_openlineage/' + tag: 'Documentation' + text: 'Set up Datadog Agent for OpenLineage Proxy' ---
Custom jobs using OpenLineage is in Preview.
## Overview -Custom jobs use the [OpenLineage][1] standard to send job and lineage events to Datadog. Use custom jobs when you need to: +Custom jobs use the [OpenLineage][1] standard to send job and lineage events to Datadog. With custom jobs, you can: + +- Detect failing and long-running jobs +- Pinpoint and resolve the root cause of failed and long-running jobs +- Understand upstream dependencies and downstream data consumers with data lineage + +Use custom jobs when you need to: - Capture lineage from systems Datadog doesn't integrate with natively, such as in-house tools or custom ETL scripts - Emit lineage events for jobs or orchestrators where a native Datadog integration isn't available @@ -76,7 +82,6 @@ curl -X POST "https://data-obs-intake.datadoghq.com/api/v1/lineage" \ }' ``` - {{% /tab %}} {{% tab "OpenLineage Python client (HTTP transport)" %}} @@ -128,7 +133,6 @@ event = RunEvent( client.emit(event) ``` - {{% /tab %}} {{% tab "OpenLineage Python client (Datadog transport)" %}} @@ -195,11 +199,319 @@ export OPENLINEAGE__TRANSPORT__TYPE=datadog client = OpenLineageClient.from_environment() ``` +{{% /tab %}} +{{< /tabs >}} + +## Step 2: Send a `RUNNING` event (optional) + +**Note**: This step is optional. `RUNNING` events let you see a job's status before it finishes. If you only need to capture job completion status, skip to [Step 3](#step-3-send-a-complete-or-fail-event). + +While the job is in progress, send a `RUNNING` event to track it in Datadog's Jobs Monitoring. Use the same `runId` from the `START` event. + +{{< tabs >}} +{{% tab "Direct HTTP with curl" %}} + +```shell +curl -X POST "https://data-obs-intake.datadoghq.com/api/v1/lineage" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "eventTime": "2024-01-01T10:02:00Z", + "eventType": "RUNNING", + "run": { "runId": "" }, + "job": { + "namespace": "", + "name": "", + "facets": { + "jobType": { + "_producer": "", + "_schemaURL": "https://openlineage.io/spec/facets/2-0-3/JobTypeJobFacet.json", + "processingType": "BATCH", + "integration": "custom", + "jobType": "JOB" + } + } + }, + "producer": "" + }' +``` + +{{% /tab %}} + +{{% tab "OpenLineage Python client (HTTP transport)" %}} + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import job_type_job + +running_event = RunEvent( + eventType=RunState.RUNNING, + eventTime=datetime.utcnow().isoformat(), + run=Run(runId=""), # same runId as START + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(running_event) +``` + +{{% /tab %}} + +{{% tab "OpenLineage Python client (Datadog transport)" %}} + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import job_type_job + +running_event = RunEvent( + eventType=RunState.RUNNING, + eventTime=datetime.utcnow().isoformat(), + run=Run(runId=""), # same runId as START + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(running_event) +``` + +{{% /tab %}} +{{< /tabs >}} + +## Step 3: Send a `COMPLETE` or `FAIL` event + +When the job finishes, send a `COMPLETE` or `FAIL` event using the same `runId` from the `START` event. + +{{< tabs >}} +{{% tab "Direct HTTP with curl" %}} + +**Success** + +```shell +curl -X POST "https://data-obs-intake.datadoghq.com/api/v1/lineage" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "eventTime": "2024-01-01T10:05:00Z", + "eventType": "COMPLETE", + "run": { "runId": "" }, + "job": { + "namespace": "", + "name": "", + "facets": { + "jobType": { + "_producer": "", + "_schemaURL": "https://openlineage.io/spec/facets/2-0-3/JobTypeJobFacet.json", + "processingType": "BATCH", + "integration": "custom", + "jobType": "JOB" + } + } + }, + "producer": "" + }' +``` + +**Failure** + +```shell +curl -X POST "https://data-obs-intake.datadoghq.com/api/v1/lineage" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "eventTime": "2024-01-01T10:05:00Z", + "eventType": "FAIL", + "run": { + "runId": "", + "facets": { + "errorMessage": { + "_producer": "", + "_schemaURL": "https://openlineage.io/spec/facets/1-0-1/ErrorMessageRunFacet.json", + "message": "Job failed: division by zero", + "programmingLanguage": "Python", + "stackTrace": "Traceback (most recent call last):\n File \"job.py\", line 42, in run\n result = total / count\nZeroDivisionError: division by zero" + } + } + }, + "job": { + "namespace": "", + "name": "", + "facets": { + "jobType": { + "_producer": "", + "_schemaURL": "https://openlineage.io/spec/facets/2-0-3/JobTypeJobFacet.json", + "processingType": "BATCH", + "integration": "custom", + "jobType": "JOB" + } + } + }, + "producer": "" + }' +``` + +{{% /tab %}} + +{{% tab "OpenLineage Python client (HTTP transport)" %}} + +**Success** + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run + +complete_event = RunEvent( + eventType=RunState.COMPLETE, + eventTime=datetime.utcnow().isoformat(), + run=Run(runId=""), # same runId as START + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(complete_event) +``` + +**Failure** + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import error_message_run + +fail_event = RunEvent( + eventType=RunState.FAIL, + eventTime=datetime.utcnow().isoformat(), + run=Run( + runId="", # same runId as START + facets={ + "errorMessage": error_message_run.ErrorMessageRunFacet( + message="Job failed: division by zero", + programmingLanguage="Python", + stackTrace="Traceback (most recent call last):\n File \"job.py\", line 42, in run\n result = total / count\nZeroDivisionError: division by zero" + ) + } + ), + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(fail_event) +``` + +{{% /tab %}} + +{{% tab "OpenLineage Python client (Datadog transport)" %}} + +**Success** + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import job_type_job + +complete_event = RunEvent( + eventType=RunState.COMPLETE, + eventTime=datetime.utcnow().isoformat(), + run=Run(runId=""), # same runId as START + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(complete_event) +``` + +**Failure** + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import job_type_job, error_message_run + +fail_event = RunEvent( + eventType=RunState.FAIL, + eventTime=datetime.utcnow().isoformat(), + run=Run( + runId="", # same runId as START + facets={ + "errorMessage": error_message_run.ErrorMessageRunFacet( + message="Job failed: division by zero", + programmingLanguage="Python", + stackTrace="Traceback (most recent call last):\n File \"job.py\", line 42, in run\n result = total / count\nZeroDivisionError: division by zero" + ) + } + ), + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(fail_event) +``` {{% /tab %}} {{< /tabs >}} -## Step 2: Verify in Datadog +## Step 4: Verify in Datadog After sending your events, check the following: @@ -212,16 +524,16 @@ To connect your custom job's lineage to datasets already tracked by Datadog's na Datadog resolves datasets into a hierarchy of account, database, schema, and table. If a name has fewer parts than expected (for example, `database.table` instead of `database.schema.table`), Datadog falls back to the nearest higher-order node in the lineage graph. -| Platform | Namespace | Name | -|---|---|---| -| BigQuery | `bigquery` | `{project}.{dataset}.{table}` | -| Snowflake | `snowflake://{org}-{account}` | `{database}.{schema}.{table}` | -| Redshift | `redshift://{aws_account_id}:{region}:{cluster}` | `{database}.{schema}.{table}` | -| PostgreSQL | `postgres://{host}:{port}` | `{database}.{schema}.{table}` | -| Databricks | `databricks://{workspace-url}` | `{database}.{schema}.{table}` | -| Trino | `trino://{host}:{port}` | `{catalog}.{schema}.{table}` | -| AWS Glue | `arn:aws:glue:{region}:{accountId}` | `{database}.{table}` | -| S3 | `s3://{bucket}` | `{path}` | +| Platform | Namespace | Name | +| ---------- | ------------------------------------------------ | ----------------------------- | +| BigQuery | `bigquery` | `{project}.{dataset}.{table}` | +| Snowflake | `snowflake://{org}-{account}` | `{database}.{schema}.{table}` | +| Redshift | `redshift://{aws_account_id}:{region}:{cluster}` | `{database}.{schema}.{table}` | +| PostgreSQL | `postgres://{host}:{port}` | `{database}.{schema}.{table}` | +| Databricks | `databricks://{workspace-url}` | `{database}.{schema}.{table}` | +| Trino | `trino://{host}:{port}` | `{catalog}.{schema}.{table}` | +| AWS Glue | `arn:aws:glue:{region}:{accountId}` | `{database}.{table}` | +| S3 | `s3://{bucket}` | `{path}` | For platforms not listed here, follow the [OpenLineage naming conventions][8]. @@ -256,17 +568,17 @@ The `jobType` job facet is **required**. It determines how Datadog classifies an Use `custom` for custom jobs. The values below are used by Datadog's native integrations. Using them for custom jobs may produce unexpected behavior. In particular, `SPARK` prevents span generation. -| Value | Platform | -|---|---| -| `custom` | Custom or unsupported platforms | -| `SPARK` | Apache Spark (native integration only; do not use for custom jobs) | -| `AIRFLOW` | Apache Airflow | -| `DBT` | dbt | -| `BIGQUERY` | Google BigQuery | -| `SNOWFLAKE` | Snowflake | -| `TRINO` | Trino | -| `ICEBERG` | Apache Iceberg | -| `TABLEAU` | Tableau | +| Value | Platform | +| ----------- | ------------------------------------------------------------------ | +| `custom` | Custom or unsupported platforms | +| `SPARK` | Apache Spark (native integration only; do not use for custom jobs) | +| `AIRFLOW` | Apache Airflow | +| `DBT` | dbt | +| `BIGQUERY` | Google BigQuery | +| `SNOWFLAKE` | Snowflake | +| `TRINO` | Trino | +| `ICEBERG` | Apache Iceberg | +| `TABLEAU` | Tableau | #### `processingType` values @@ -280,12 +592,12 @@ Common values include `JOB`, `TASK`, `DAG`, `MODEL`, `COMMAND`, and `QUERY`. ### Other supported facets -| Facet | What Datadog does | -|---|---| -| `parent` | Creates parent-child job hierarchy in the lineage graph | -| `errorMessage` | Generates error spans with `error.message` and `error.stack` tags | -| `tags` | Adds span tags to the run; `_dd.ol_service` value maps to the Datadog service name | -| `sql` | Parses and masks the SQL query; generates query events | +| Facet | What Datadog does | +| -------------- | ---------------------------------------------------------------------------------- | +| `parent` | Creates parent-child job hierarchy in the lineage graph | +| `errorMessage` | Generates error spans with `error.message` and `error.stack` tags | +| `tags` | Adds span tags to the run; `_dd.ol_service` value maps to the Datadog service name | +| `sql` | Parses and masks the SQL query; generates query events | ## Further reading From d62ba92f9c1ed29d5b69df4b55459adb2bd64e03 Mon Sep 17 00:00:00 2001 From: Jen Gilbert Date: Mon, 22 Jun 2026 15:33:08 -0500 Subject: [PATCH 5/6] Add copy button to hard-coded API docs (#37628) * Add copy button to hardcoded API pages * Adjust Copy button location * Use a cleaner approach * Simplify comment * Remove underline on hover --- assets/styles/pages/_api.scss | 5 +++++ content/en/api/latest/_index.md | 2 +- content/en/api/latest/rate-limits/_index.md | 2 +- content/en/api/latest/scopes/_index.md | 2 +- content/en/api/latest/using-the-api/_index.md | 2 +- layouts/shortcodes/h2-with-copy-btn.html | 15 +++++++++++++++ 6 files changed, 24 insertions(+), 4 deletions(-) create mode 100644 layouts/shortcodes/h2-with-copy-btn.html diff --git a/assets/styles/pages/_api.scss b/assets/styles/pages/_api.scss index bd3699a1de9..0d10232b05b 100644 --- a/assets/styles/pages/_api.scss +++ b/assets/styles/pages/_api.scss @@ -12,6 +12,11 @@ $ddpurple: #632ca6; font-size: 16px; text-decoration: none; border-bottom: none; + + &:hover { + text-decoration: none; + border-bottom: none; + } } .copy-btn-icon-wrapper { diff --git a/content/en/api/latest/_index.md b/content/en/api/latest/_index.md index 6963791ee67..96887c4bd75 100644 --- a/content/en/api/latest/_index.md +++ b/content/en/api/latest/_index.md @@ -23,7 +23,7 @@ algolia: tags: ['api'] --- -{{< h2 >}}API Reference{{< /h2 >}} +{{< h2-with-copy-btn >}}API Reference{{< /h2-with-copy-btn >}} The Datadog API is an HTTP REST API. The API uses resource-oriented URLs to call the API, uses status codes to indicate the success or failure of requests, returns JSON from all requests, and uses standard HTTP response codes. Use the Datadog API to access the Datadog platform programmatically. diff --git a/content/en/api/latest/rate-limits/_index.md b/content/en/api/latest/rate-limits/_index.md index e9c4e43ff7d..d89a0567892 100644 --- a/content/en/api/latest/rate-limits/_index.md +++ b/content/en/api/latest/rate-limits/_index.md @@ -3,7 +3,7 @@ title: Rate Limits type: api --- -{{< h2 >}}Rate Limits{{< /h2 >}} +{{< h2-with-copy-btn >}}Rate Limits{{< /h2-with-copy-btn >}} Many API endpoints are rate limited. Once you exceed a certain number of requests in a specific period, Datadog returns an error. diff --git a/content/en/api/latest/scopes/_index.md b/content/en/api/latest/scopes/_index.md index e1f137291a4..c3448390de5 100644 --- a/content/en/api/latest/scopes/_index.md +++ b/content/en/api/latest/scopes/_index.md @@ -3,7 +3,7 @@ title: Authorization Scopes type: api disable_sidebar: true --- -## Authorization scopes for OAuth clients +{{< h2-with-copy-btn >}}Authorization scopes for OAuth clients{{< /h2-with-copy-btn >}} Scopes are an authorization mechanism that allow you to limit and define the specific access applications have to an organization's Datadog data. When authorized to access data on behalf of a user or service account, applications can only access the information explicitly permitted by their assigned scopes. diff --git a/content/en/api/latest/using-the-api/_index.md b/content/en/api/latest/using-the-api/_index.md index 54df760694b..d901c331789 100644 --- a/content/en/api/latest/using-the-api/_index.md +++ b/content/en/api/latest/using-the-api/_index.md @@ -3,7 +3,7 @@ title: Using the API type: api --- -{{< h2 >}}Using the API{{< /h2 >}} +{{< h2-with-copy-btn >}}Using the API{{< /h2-with-copy-btn >}} Use the Datadog HTTP API to access the Datadog platform programmatically. You can use the API to send data to Datadog, build data visualizations, and manage your account. diff --git a/layouts/shortcodes/h2-with-copy-btn.html b/layouts/shortcodes/h2-with-copy-btn.html new file mode 100644 index 00000000000..55ff998797b --- /dev/null +++ b/layouts/shortcodes/h2-with-copy-btn.html @@ -0,0 +1,15 @@ +{{- /* + Page-title heading for hardcoded API Overview pages. Renders the same anchored +

as the h2 shortcode, paired with the page Copy button so the two sit + together — mirroring the layout of the generated API pages. + + Use once per page, for the top heading only: it emits the #page-copy-btn + element, so a second use would duplicate that id. +*/ -}} +{{ $anchorized := anchorize .Inner }} +
+

+ {{ .Inner }} +

+ {{ partial "api/api-copy-btn.html" .Page }} +
From 05cc88ac1f28813cff9031076df4617952176acc Mon Sep 17 00:00:00 2001 From: Sylvain Cizaire <90193538+sysy6r@users.noreply.github.com> Date: Mon, 22 Jun 2026 23:39:39 +0200 Subject: [PATCH 6/6] Remove US1-Fed (gov) from Case Management unsupported sites (#37499) Case Management is now GA on US1-Fed. Only US2-Fed (gov2) remains unsupported. Co-authored-by: Claude Opus 4.6 --- config/_default/params.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/config/_default/params.yaml b/config/_default/params.yaml index 792e7bb6bb2..b07ffb6eb18 100644 --- a/config/_default/params.yaml +++ b/config/_default/params.yaml @@ -306,7 +306,7 @@ unsupported_sites: bits_ai_sre: [gov,gov2] bits_data_analysis: [gov,gov2] byoti: [gov, gov2] - case_management: [gov,gov2] + case_management: [gov2] ci_visibility: [gov,gov2] cloudcraft_monitors_overlay: [gov,gov2] cloudprem: [gov,gov2]