diff --git a/assets/styles/pages/_api.scss b/assets/styles/pages/_api.scss index bd3699a1de9..0d10232b05b 100644 --- a/assets/styles/pages/_api.scss +++ b/assets/styles/pages/_api.scss @@ -12,6 +12,11 @@ $ddpurple: #632ca6; font-size: 16px; text-decoration: none; border-bottom: none; + + &:hover { + text-decoration: none; + border-bottom: none; + } } .copy-btn-icon-wrapper { diff --git a/config/_default/params.yaml b/config/_default/params.yaml index 792e7bb6bb2..b07ffb6eb18 100644 --- a/config/_default/params.yaml +++ b/config/_default/params.yaml @@ -306,7 +306,7 @@ unsupported_sites: bits_ai_sre: [gov,gov2] bits_data_analysis: [gov,gov2] byoti: [gov, gov2] - case_management: [gov,gov2] + case_management: [gov2] ci_visibility: [gov,gov2] cloudcraft_monitors_overlay: [gov,gov2] cloudprem: [gov,gov2] diff --git a/content/en/account_management/scim/_index.md b/content/en/account_management/scim/_index.md index 75c8439e71e..1d9f69a824f 100644 --- a/content/en/account_management/scim/_index.md +++ b/content/en/account_management/scim/_index.md @@ -22,11 +22,11 @@ The System for Cross-domain Identity Management, or [SCIM][9], is an open standa ### Supported capabilities -- Create users in Datadog (Email verification is required for first login, see [email verification][1]) +- Create users in Datadog - Remove users in Datadog when they no longer require access -- Keep user attributes synchronized between the identity provider and Datadog - Single sign-on to Datadog (recommended) - Managed Teams: Create Datadog Teams from identity provider groups and keep membership of the Datadog Teams synchronized with group membership in the identity provider. +- Role provisioning: Provision a user's Datadog role (built-in or custom) from an identity provider attribute, and keep it synchronized. When the attribute changes in your identity provider, the user's Datadog role updates in real time. Datadog implements the SCIM server protocol. Datadog supports using SCIM with the Microsoft Entra ID and Okta identity providers. Other identity providers may work, but are not explicitly supported. @@ -44,6 +44,14 @@ Datadog strongly recommends that you use a service account application key when When using SAML and SCIM together, Datadog strongly recommends disabling SAML just-in-time (JIT) provisioning to avoid discrepancies in access. Manage user provisioning through SCIM only. +### Role provisioning behavior + +When a SCIM request includes one or more roles, Datadog provisions only the roles that match a role in your organization. If none of the roles match, the user falls back to your organization's default role (Standard). Unmatched roles are logged to [Audit Trail][11]. + +SCIM is the source of truth for role assignment and takes precedence over [SAML role mappings][12]. SCIM role provisioning events are recorded in Audit Trail and as StatsD metrics. + +Roles follow the SCIM multi-valued attribute convention defined in [RFC 7643][13]. Both Okta and Microsoft Entra ID support this mapping natively, with no custom scripting required. For setup instructions, see the documentation for your identity provider. + ## Using a service account with SCIM To enable SCIM, you must use an [application key][5] to secure the connection between your identity provider and your Datadog account. A specific user or service account controls each application key. @@ -68,3 +76,6 @@ The service account requires at minimum the `user_access_invite` and `user_acces [8]: /help/ [9]: https://scim.cloud/ [10]: /api/latest/scim/ +[11]: /account_management/audit_trail/ +[12]: /account_management/saml/mapping/#map-saml-attributes-to-datadog-roles +[13]: https://www.rfc-editor.org/rfc/rfc7643.html#section-4.1.2 diff --git a/content/en/account_management/scim/entra.md b/content/en/account_management/scim/entra.md index 28bb696547e..9fa50e1f72c 100644 --- a/content/en/account_management/scim/entra.md +++ b/content/en/account_management/scim/entra.md @@ -78,11 +78,14 @@ When using SAML and SCIM together, Datadog strongly recommends disabling SAML ju | `jobTitle` | `title` | | `mail` | `emails[type eq "work"].value` | | `displayName` | `name.formatted` | +| `roles` | `roles` | {{< img src="/account_management/scim/ad-users-2.png" alt="Attribute mapping configuration, Provision Azure Active Directory Users">}} 7. After you set your mappings, click {{< ui >}}Save{{< /ui >}}. +To provision a user's Datadog role (built-in or custom), map the `roles` attribute as shown above. Roles follow the SCIM multi-valued attribute convention defined in [RFC 7643][9], using the role UUID as `value` and the role name as `display`. If a SCIM request sends multiple roles, Datadog provisions only the roles that match a role in your organization. If none match, the user falls back to the org default role (Standard), and unmatched roles are logged to Audit Trail. For more details, see [SCIM][1]. + ### Group attributes Group mapping is not supported. @@ -95,3 +98,4 @@ Group mapping is not supported. [6]: https://entra.microsoft.com/ [7]: https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/permissions-reference#cloud-application-administrator [8]: https://learn.microsoft.com/en-us/entra/identity/app-provisioning/application-provisioning-config-problem-scim-compatibility#flags-to-alter-the-scim-behavior +[9]: https://www.rfc-editor.org/rfc/rfc7643.html#section-4.1.2 diff --git a/content/en/account_management/scim/okta.md b/content/en/account_management/scim/okta.md index 1f3d6bce8a9..0aaf1494279 100644 --- a/content/en/account_management/scim/okta.md +++ b/content/en/account_management/scim/okta.md @@ -59,6 +59,30 @@ When using SAML and SCIM together, Datadog strongly recommends disabling SAML ju - {{< ui >}}Deactivate Users{{< /ui >}} 8. Under {{< ui >}}Datadog Attribute Mappings{{< /ui >}}, find the mapping of Okta attributes to Datadog attributes already pre-configured. You can re-map them if needed, but map the Okta values to the same set of Datadog values. +### Map the Datadog role attribute + +To provision a user's Datadog role (built-in or custom) through SCIM, add an explicit mapping for the `roles` attribute. Okta does not map this attribute by default. + +Datadog's SCIM role support follows the SCIM multi-valued attribute convention defined in [RFC 7643][8], using the role UUID as `value` and the role name as `display`: + +```json +{ + "roles": [ + { "value": "", "display": "" } + ] +} +``` + +1. In {{< ui >}}Directory{{< /ui >}} > {{< ui >}}Profile Editor{{< /ui >}}, select the Okta user profile, then click {{< ui >}}Add Attribute{{< /ui >}} to create a `roles` attribute: + - {{< ui >}}Data type{{< /ui >}}: **string** + - {{< ui >}}Display name{{< /ui >}}: **Roles** + - {{< ui >}}Variable name{{< /ui >}}: **roles** + - For {{< ui >}}Enum{{< /ui >}}, select {{< ui >}}Define enumerated list of values{{< /ui >}} and add one entry per Datadog role, using the role name as the display name and the role UUID as the value. You can find a role's UUID in the role's URL on your [Organization Settings][9] page. Add any custom roles the same way. +2. In your Datadog application's {{< ui >}}Provisioning{{< /ui >}} > {{< ui >}}To App{{< /ui >}} settings, map the Okta `roles` attribute to the Datadog `roles` attribute. +3. In the app's {{< ui >}}Assignments{{< /ui >}} tab, assign each user the appropriate role from the dropdown. + +If a SCIM request sends multiple roles, Datadog provisions only the roles that match a role in your organization. If none match, the user falls back to the org default role (Standard), and unmatched roles are logged to Audit Trail. For more details, see [SCIM][1]. + ## Configure automatic team provisioning With [Managed Teams][6], you control the core provisioning of a Datadog Team — its name, handle, and membership — through the identity provider. The setup process differs depending on whether the team already exists in Datadog. @@ -133,3 +157,5 @@ This procedure allows you to manage team membership in Datadog instead of Okta a [5]: /account_management/org_settings/service_accounts [6]: /account_management/teams/manage/#manage-teams-through-an-identity-provider [7]: https://app.datadoghq.com/teams +[8]: https://www.rfc-editor.org/rfc/rfc7643.html#section-4.1.2 +[9]: https://app.datadoghq.com/organization-settings/roles diff --git a/content/en/agent/guide/private-link.md b/content/en/agent/guide/private-link.md index 66f5259fa53..20b055ea2a6 100644 --- a/content/en/agent/guide/private-link.md +++ b/content/en/agent/guide/private-link.md @@ -52,6 +52,7 @@ Datadog exposes AWS PrivateLink endpoints in **{{< region-param key="aws_region" {{< img src="agent/guide/private_link/vpc_service_name.png" alt="VPC service name" style="width:70%;" >}} +{{% site-region region="ap1" %}} | Datadog | PrivateLink service name | Private DNS name | |---------------------------|----------------------------------------------------------------------------------------|------------------------------------------------------------------------| | Logs (Agent HTTP intake) | {{< region-param key="aws_private_link_logs_agent_service_name" code="true" >}} | {{< region-param key="agent_http_endpoint_private_link" code="true" >}} | @@ -64,6 +65,13 @@ Datadog exposes AWS PrivateLink endpoints in **{{< region-param key="aws_region" | Traces | {{< region-param key="aws_private_link_traces_service_name" code="true" >}} | {{< region-param key="traces_endpoint_private_link" code="true" >}} | | Database Monitoring | {{< region-param key="aws_private_link_dbm_service_name" code="true" >}} | {{< region-param key="dbm_endpoint_private_link" code="true" >}} | | Remote Configuration | {{< region-param key="aws_private_link_remote_config_service_name" code="true" >}} | {{< region-param key="remote_config_endpoint_private_link" code="true" >}} | +{{% /site-region %}} +{{% site-region region="us" %}} +For the complete list of US1 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} +{{% site-region region="ap2" %}} +For the complete list of AP2 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} 4. Click {{< ui >}}Verify{{< /ui >}}. If this does not return _Service name found_, reach out to [Datadog support][14]. 5. Choose the VPC and subnets that should be peered with the Datadog VPC service endpoint. @@ -132,6 +140,7 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can ## PrivateLink service names +{{% site-region region="ap1" %}} | Datadog | PrivateLink service name | Private DNS name | |---------------------------|----------------------------------------------------------------------------------------|------------------------------------------------------------------------| | Logs (Agent HTTP intake) | {{< region-param key="aws_private_link_logs_agent_service_name" code="true" >}} | {{< region-param key="agent_http_endpoint_private_link" code="true" >}} | @@ -144,6 +153,14 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can | Traces | {{< region-param key="aws_private_link_traces_service_name" code="true" >}} | {{< region-param key="traces_endpoint_private_link" code="true" >}} | | Database Monitoring | {{< region-param key="aws_private_link_dbm_service_name" code="true" >}} | {{< region-param key="dbm_endpoint_private_link" code="true" >}} | | Remote Configuration | {{< region-param key="aws_private_link_remote_config_service_name" code="true" >}} | {{< region-param key="remote_config_endpoint_private_link" code="true" >}} | +{{% /site-region %}} +{{% site-region region="ap2" %}} +For the complete list of AP2 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} + +{{% site-region region="us" %}} +For the complete list of US1 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} **Note**: Cross-region PrivateLink doesn't emit CloudWatch metrics. See [CloudWatch metrics for AWS PrivateLink][2] for more information. @@ -161,6 +178,7 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can {{< img src="agent/guide/private_link/vpc_service_name.png" alt="VPC service name" style="width:90%;" >}} +{{% site-region region="ap1" %}} | Datadog | PrivateLink service name | |---------------------------|----------------------------------------------------------------------------------------| | Logs (Agent HTTP intake) | {{< region-param key="aws_private_link_logs_agent_service_name" code="true" >}} | @@ -173,6 +191,13 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can | Traces | {{< region-param key="aws_private_link_traces_service_name" code="true" >}} | | Database Monitoring | {{< region-param key="aws_private_link_dbm_service_name" code="true" >}} | | Remote Configuration | {{< region-param key="aws_private_link_remote_config_service_name" code="true" >}} | +{{% /site-region %}} +{{% site-region region="us" %}} +For the complete list of US1 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} +{{% site-region region="ap2" %}} +For the complete list of AP2 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} 4. Click {{< ui >}}Verify{{< /ui >}}. If this does not return _Service name found_, reach out to [Datadog support][1]. @@ -200,6 +225,7 @@ After the endpoint status is updated to {{< ui >}}Available{{< /ui >}}, you can Use the list below to map service and DNS name to different parts of Datadog: +{{% site-region region="ap1" %}} | Datadog | PrivateLink service name | Private DNS name | |---------------------------|----------------------------------------------------------------------------------------|------------------------------------------------------------------------| | Logs (Agent HTTP intake) | {{< region-param key="aws_private_link_logs_agent_service_name" code="true" >}} | {{< region-param key="agent_http_endpoint_private_link" code="true" >}} | @@ -212,6 +238,13 @@ Use the list below to map service and DNS name to different parts of Datadog: | Traces | {{< region-param key="aws_private_link_traces_service_name" code="true" >}} | {{< region-param key="traces_endpoint_private_link" code="true" >}} | | Database Monitoring | {{< region-param key="aws_private_link_dbm_service_name" code="true" >}} | {{< region-param key="dbm_endpoint_private_link" code="true" >}} | | Remote Configuration | {{< region-param key="aws_private_link_remote_config_service_name" code="true" >}} | {{< region-param key="remote_config_endpoint_private_link" code="true" >}} | +{{% /site-region %}} +{{% site-region region="us" %}} +For the complete list of US1 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} +{{% site-region region="ap2" %}} +For the complete list of AP2 DNS records and VPC service endpoints, see [VPC Endpoint Service IDs](#vpc-endpoint-service-ids). +{{% /site-region %}} You can also find this information by interrogating the AWS API, `DescribeVpcEndpointServices`, or by using the following command: @@ -303,6 +336,138 @@ The VPCs with Private Hosted Zone (PHZ) attached need to have a couple of settin [16]: /agent/logs/?tab=tailexistingfiles#send-logs-over-https [17]: https://docs.aws.amazon.com/secretsmanager/latest/userguide/vpc-endpoint-overview.html +{{% site-region region="us" %}} +## VPC Endpoint Service IDs + +US1 uses a two-level DNS architecture for PrivateLink. Every customer-facing DNS record maps to a dedicated `color.intake.datadoghq.com` VPC endpoint address. Setting up a VPC endpoint for a given anchor address covers all customer-facing records that map to it. + +Use the following table to identify which VPC endpoints to set up for the Datadog features you use. Wildcard entries match any subdomain not otherwise listed. + +**Note**: In the table below, `---` indicates a direct VPC service endpoint with no intermediate anchor address. +| Name | Anchor | VPC Endpoint Service ID | +|---|---|---| +| `webhook-intake.datadoghq.com` | `azure.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-02bee2072b5c3c226` | +| `webhooks-http-intake.logs.datadoghq.com` | `azure.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-02bee2072b5c3c226` | +| `*.integrations.otlp.datadoghq.com` | `brown.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-00192e92115cbcc75` | +| `opamp.datadoghq.com` | `brown.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-00192e92115cbcc75` | +| `otlp.datadoghq.com` | `brown.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-00192e92115cbcc75` | +| `mcp.datadoghq.com` | `cornsilk.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-058a75ceea85a9175` | +| `agenthealth-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `ci-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `cicodescan-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `citestcov-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `citestcycle-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `cloudplatform-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `contimage-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `contlcycle-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `cws-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `debugger-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `error-tracking-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `event-management-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `event-platform-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `feed-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `instrumentation-telemetry-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `kubeops-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `llmobs-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `ndm-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `ndmflow-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `netpath-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `ocimetrics-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `resources-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `sbom-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `sds-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `sentry-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `snmp-traps-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `softinv-intake.datadoghq.com` | `cyan.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0b3292e3efce2a445` | +| `iam-rum-intake.datadoghq.com` | `gray.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0a3b2d86676122d8d` | +| `rum-http-intake.logs.datadoghq.com` | `gray.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0a3b2d86676122d8d` | +| `rum.browser-intake-datadoghq.com` | `gray.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0a3b2d86676122d8d` | +| `data-obs-intake.datadoghq.com` | `lime.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ee865cd1c0a7ba32` | +| `trace.agent.datadoghq.com` | `lime.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ee865cd1c0a7ba32` | +| `network-devices.datadoghq.com` | `olive.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-05e3bfec4501e714d` | +| `*.api.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `*.synthetics.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `api.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `intake.synthetics.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `synthetics.datadoghq.com` | `orchid.intake.datadoghq.com` | `com.amazonaws.vpce.us-east-1.vpce-svc-07895350fd0109264` | +| `agent-http-intake.logs.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-025a56b9187ac1f63` | +| `http-intake.logs.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-0e36256cb6172439d` | +| `metrics.agent.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-09a8006e245d1e7b8` | +| `orchestrator.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ad5fb9e71f85fe99` | +| `process.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ed1f789ac6b0bde1` | +| `intake.profile.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-022ae36a7b2472029` | +| `dbm-metrics-intake.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-0ce70d55ec4af8501` | +| `config.datadoghq.com` | `---` | `com.amazonaws.vpce.us-east-1.vpce-svc-01f21309e507e3b1d` | + +{{% /site-region %}} + +{{% site-region region="ap2" %}} +## VPC Endpoint Service IDs + +AP2 uses a two-level DNS architecture for PrivateLink. Every customer-facing DNS record maps to a dedicated `color.intake.ap2.datadoghq.com` VPC endpoint address. Setting up a VPC endpoint for a given anchor address covers all customer-facing records that map to it. + +Use the following table to identify which VPC endpoints to set up for the Datadog features you use. More specific DNS records take precedence over wildcards—for example, `trace.agent.ap2.datadoghq.com` resolves to `lime.intake.ap2.datadoghq.com` even though `*.agent.ap2.datadoghq.com` points to `beige.intake.ap2.datadoghq.com`. + +| Name | Anchor | VPC Endpoint Service ID | +|---|---|---| +| `gcp-intake.logs.ap2.datadoghq.com` | `aqua.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-01b61a61d21fc7273` | +| `*.agent.ap2.datadoghq.com` | `beige.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06a30d6a016b746ff` | +| `agent.ap2.datadoghq.com` | `beige.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06a30d6a016b746ff` | +| `process.ap2.datadoghq.com` | `bisque.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0c26ca335d93a68b5` | +| `*.integrations.otlp.ap2.datadoghq.com` | `brown.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-04c61207a01a73496` | +| `opamp.ap2.datadoghq.com` | `brown.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-04c61207a01a73496` | +| `otlp.ap2.datadoghq.com` | `brown.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-04c61207a01a73496` | +| `agenthealth-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `awsmetrics-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `ci-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cicodescan-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cireport-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `citestcov-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `citestcycle-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cloudplatform-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `contimage-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `contlcycle-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cspm-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `cws-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `debugger-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `error-tracking-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `event-management-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `instrumentation-telemetry-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `intake.profile.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `kubeops-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `llmobs-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `ndm-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `ndmflow-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `netpath-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `ocimetrics-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `resources-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `sbom-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `sds-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `sentry-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `snmp-traps-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `softinv-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `webhook-intake.ap2.datadoghq.com` | `cyan.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0d936da0e6a30d3cd` | +| `agent-http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `aws-kinesis-http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `eventbridge-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `lambda-http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `obpipeline-intake.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `runtime-security-http-intake.logs.ap2.datadoghq.com` | `gold.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06460db30a7cfdace` | +| `live.logs.ap2.datadoghq.com` | `indigo.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0545109555aa68e7e` | +| `data-obs-intake.ap2.datadoghq.com` | `lime.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0f3e01f4180b2ae09` | +| `trace.agent.ap2.datadoghq.com` | `lime.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0f3e01f4180b2ae09` | +| `orchestrator.ap2.datadoghq.com` | `linen.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-031da3ffac78ef902` | +| `*.synthetics.ap2.datadoghq.com` | `orchid.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06ec78b291ce8020a` | +| `api.ap2.datadoghq.com` | `orchid.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06ec78b291ce8020a` | +| `quota.browser-intake-ap2-datadoghq.com` | `orchid.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06ec78b291ce8020a` | +| `synthetics.ap2.datadoghq.com` | `orchid.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-06ec78b291ce8020a` | +| `sourcemap-intake.ap2.datadoghq.com` | `plum.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-028e4348e80fa73f5` | +| `config.ap2.datadoghq.com` | `violet.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-01f8f80f4cb97bd10` | +| `dbm-metrics-intake.ap2.datadoghq.com` | `white.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-094469ee7a178f448` | +| `dbquery-intake.ap2.datadoghq.com` | `white.intake.ap2.datadoghq.com` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-094469ee7a178f448` | +{{% /site-region %}} + {{% /site-region %}} ## Verify that data is being sent using PrivateLink diff --git a/content/en/api/latest/_index.md b/content/en/api/latest/_index.md index 6963791ee67..96887c4bd75 100644 --- a/content/en/api/latest/_index.md +++ b/content/en/api/latest/_index.md @@ -23,7 +23,7 @@ algolia: tags: ['api'] --- -{{< h2 >}}API Reference{{< /h2 >}} +{{< h2-with-copy-btn >}}API Reference{{< /h2-with-copy-btn >}} The Datadog API is an HTTP REST API. The API uses resource-oriented URLs to call the API, uses status codes to indicate the success or failure of requests, returns JSON from all requests, and uses standard HTTP response codes. Use the Datadog API to access the Datadog platform programmatically. diff --git a/content/en/api/latest/rate-limits/_index.md b/content/en/api/latest/rate-limits/_index.md index e9c4e43ff7d..d89a0567892 100644 --- a/content/en/api/latest/rate-limits/_index.md +++ b/content/en/api/latest/rate-limits/_index.md @@ -3,7 +3,7 @@ title: Rate Limits type: api --- -{{< h2 >}}Rate Limits{{< /h2 >}} +{{< h2-with-copy-btn >}}Rate Limits{{< /h2-with-copy-btn >}} Many API endpoints are rate limited. Once you exceed a certain number of requests in a specific period, Datadog returns an error. diff --git a/content/en/api/latest/scopes/_index.md b/content/en/api/latest/scopes/_index.md index e1f137291a4..c3448390de5 100644 --- a/content/en/api/latest/scopes/_index.md +++ b/content/en/api/latest/scopes/_index.md @@ -3,7 +3,7 @@ title: Authorization Scopes type: api disable_sidebar: true --- -## Authorization scopes for OAuth clients +{{< h2-with-copy-btn >}}Authorization scopes for OAuth clients{{< /h2-with-copy-btn >}} Scopes are an authorization mechanism that allow you to limit and define the specific access applications have to an organization's Datadog data. When authorized to access data on behalf of a user or service account, applications can only access the information explicitly permitted by their assigned scopes. diff --git a/content/en/api/latest/using-the-api/_index.md b/content/en/api/latest/using-the-api/_index.md index 54df760694b..d901c331789 100644 --- a/content/en/api/latest/using-the-api/_index.md +++ b/content/en/api/latest/using-the-api/_index.md @@ -3,7 +3,7 @@ title: Using the API type: api --- -{{< h2 >}}Using the API{{< /h2 >}} +{{< h2-with-copy-btn >}}Using the API{{< /h2-with-copy-btn >}} Use the Datadog HTTP API to access the Datadog platform programmatically. You can use the API to send data to Datadog, build data visualizations, and manage your account. diff --git a/content/en/data_observability/jobs_monitoring/openlineage/_index.md b/content/en/data_observability/jobs_monitoring/openlineage/_index.md index 09872dab651..c54830bb710 100644 --- a/content/en/data_observability/jobs_monitoring/openlineage/_index.md +++ b/content/en/data_observability/jobs_monitoring/openlineage/_index.md @@ -2,19 +2,25 @@ title: Custom Jobs using OpenLineage description: "Monitor jobs from in-house tools, custom pipelines, and orchestrators that don't have native Datadog integrations." further_reading: - - link: '/data_observability/' - tag: 'Documentation' - text: 'Data Observability Overview' - - link: '/data_observability/jobs_monitoring/openlineage/datadog_agent_for_openlineage/' - tag: 'Documentation' - text: 'Set up Datadog Agent for OpenLineage Proxy' + - link: '/data_observability/' + tag: 'Documentation' + text: 'Data Observability Overview' + - link: '/data_observability/jobs_monitoring/openlineage/datadog_agent_for_openlineage/' + tag: 'Documentation' + text: 'Set up Datadog Agent for OpenLineage Proxy' ---
Custom jobs using OpenLineage is in Preview.
## Overview -Custom jobs use the [OpenLineage][1] standard to send job and lineage events to Datadog. Use custom jobs when you need to: +Custom jobs use the [OpenLineage][1] standard to send job and lineage events to Datadog. With custom jobs, you can: + +- Detect failing and long-running jobs +- Pinpoint and resolve the root cause of failed and long-running jobs +- Understand upstream dependencies and downstream data consumers with data lineage + +Use custom jobs when you need to: - Capture lineage from systems Datadog doesn't integrate with natively, such as in-house tools or custom ETL scripts - Emit lineage events for jobs or orchestrators where a native Datadog integration isn't available @@ -76,7 +82,6 @@ curl -X POST "https://data-obs-intake.datadoghq.com/api/v1/lineage" \ }' ``` - {{% /tab %}} {{% tab "OpenLineage Python client (HTTP transport)" %}} @@ -128,7 +133,6 @@ event = RunEvent( client.emit(event) ``` - {{% /tab %}} {{% tab "OpenLineage Python client (Datadog transport)" %}} @@ -195,11 +199,319 @@ export OPENLINEAGE__TRANSPORT__TYPE=datadog client = OpenLineageClient.from_environment() ``` +{{% /tab %}} +{{< /tabs >}} + +## Step 2: Send a `RUNNING` event (optional) + +**Note**: This step is optional. `RUNNING` events let you see a job's status before it finishes. If you only need to capture job completion status, skip to [Step 3](#step-3-send-a-complete-or-fail-event). + +While the job is in progress, send a `RUNNING` event to track it in Datadog's Jobs Monitoring. Use the same `runId` from the `START` event. + +{{< tabs >}} +{{% tab "Direct HTTP with curl" %}} + +```shell +curl -X POST "https://data-obs-intake.datadoghq.com/api/v1/lineage" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "eventTime": "2024-01-01T10:02:00Z", + "eventType": "RUNNING", + "run": { "runId": "" }, + "job": { + "namespace": "", + "name": "", + "facets": { + "jobType": { + "_producer": "", + "_schemaURL": "https://openlineage.io/spec/facets/2-0-3/JobTypeJobFacet.json", + "processingType": "BATCH", + "integration": "custom", + "jobType": "JOB" + } + } + }, + "producer": "" + }' +``` + +{{% /tab %}} + +{{% tab "OpenLineage Python client (HTTP transport)" %}} + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import job_type_job + +running_event = RunEvent( + eventType=RunState.RUNNING, + eventTime=datetime.utcnow().isoformat(), + run=Run(runId=""), # same runId as START + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(running_event) +``` + +{{% /tab %}} + +{{% tab "OpenLineage Python client (Datadog transport)" %}} + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import job_type_job + +running_event = RunEvent( + eventType=RunState.RUNNING, + eventTime=datetime.utcnow().isoformat(), + run=Run(runId=""), # same runId as START + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(running_event) +``` + +{{% /tab %}} +{{< /tabs >}} + +## Step 3: Send a `COMPLETE` or `FAIL` event + +When the job finishes, send a `COMPLETE` or `FAIL` event using the same `runId` from the `START` event. + +{{< tabs >}} +{{% tab "Direct HTTP with curl" %}} + +**Success** + +```shell +curl -X POST "https://data-obs-intake.datadoghq.com/api/v1/lineage" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "eventTime": "2024-01-01T10:05:00Z", + "eventType": "COMPLETE", + "run": { "runId": "" }, + "job": { + "namespace": "", + "name": "", + "facets": { + "jobType": { + "_producer": "", + "_schemaURL": "https://openlineage.io/spec/facets/2-0-3/JobTypeJobFacet.json", + "processingType": "BATCH", + "integration": "custom", + "jobType": "JOB" + } + } + }, + "producer": "" + }' +``` + +**Failure** + +```shell +curl -X POST "https://data-obs-intake.datadoghq.com/api/v1/lineage" \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{ + "eventTime": "2024-01-01T10:05:00Z", + "eventType": "FAIL", + "run": { + "runId": "", + "facets": { + "errorMessage": { + "_producer": "", + "_schemaURL": "https://openlineage.io/spec/facets/1-0-1/ErrorMessageRunFacet.json", + "message": "Job failed: division by zero", + "programmingLanguage": "Python", + "stackTrace": "Traceback (most recent call last):\n File \"job.py\", line 42, in run\n result = total / count\nZeroDivisionError: division by zero" + } + } + }, + "job": { + "namespace": "", + "name": "", + "facets": { + "jobType": { + "_producer": "", + "_schemaURL": "https://openlineage.io/spec/facets/2-0-3/JobTypeJobFacet.json", + "processingType": "BATCH", + "integration": "custom", + "jobType": "JOB" + } + } + }, + "producer": "" + }' +``` + +{{% /tab %}} + +{{% tab "OpenLineage Python client (HTTP transport)" %}} + +**Success** + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run + +complete_event = RunEvent( + eventType=RunState.COMPLETE, + eventTime=datetime.utcnow().isoformat(), + run=Run(runId=""), # same runId as START + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(complete_event) +``` + +**Failure** + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import error_message_run + +fail_event = RunEvent( + eventType=RunState.FAIL, + eventTime=datetime.utcnow().isoformat(), + run=Run( + runId="", # same runId as START + facets={ + "errorMessage": error_message_run.ErrorMessageRunFacet( + message="Job failed: division by zero", + programmingLanguage="Python", + stackTrace="Traceback (most recent call last):\n File \"job.py\", line 42, in run\n result = total / count\nZeroDivisionError: division by zero" + ) + } + ), + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(fail_event) +``` + +{{% /tab %}} + +{{% tab "OpenLineage Python client (Datadog transport)" %}} + +**Success** + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import job_type_job + +complete_event = RunEvent( + eventType=RunState.COMPLETE, + eventTime=datetime.utcnow().isoformat(), + run=Run(runId=""), # same runId as START + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(complete_event) +``` + +**Failure** + +```python +from datetime import datetime +from openlineage.client.event_v2 import RunEvent, RunState, Job, Run +from openlineage.client.facet_v2 import job_type_job, error_message_run + +fail_event = RunEvent( + eventType=RunState.FAIL, + eventTime=datetime.utcnow().isoformat(), + run=Run( + runId="", # same runId as START + facets={ + "errorMessage": error_message_run.ErrorMessageRunFacet( + message="Job failed: division by zero", + programmingLanguage="Python", + stackTrace="Traceback (most recent call last):\n File \"job.py\", line 42, in run\n result = total / count\nZeroDivisionError: division by zero" + ) + } + ), + job=Job( + namespace="", + name="", + facets={ + "jobType": job_type_job.JobTypeJobFacet( + processingType="BATCH", + integration="custom", + jobType="JOB" + ) + } + ), + producer="" +) + +client.emit(fail_event) +``` {{% /tab %}} {{< /tabs >}} -## Step 2: Verify in Datadog +## Step 4: Verify in Datadog After sending your events, check the following: @@ -212,16 +524,16 @@ To connect your custom job's lineage to datasets already tracked by Datadog's na Datadog resolves datasets into a hierarchy of account, database, schema, and table. If a name has fewer parts than expected (for example, `database.table` instead of `database.schema.table`), Datadog falls back to the nearest higher-order node in the lineage graph. -| Platform | Namespace | Name | -|---|---|---| -| BigQuery | `bigquery` | `{project}.{dataset}.{table}` | -| Snowflake | `snowflake://{org}-{account}` | `{database}.{schema}.{table}` | -| Redshift | `redshift://{aws_account_id}:{region}:{cluster}` | `{database}.{schema}.{table}` | -| PostgreSQL | `postgres://{host}:{port}` | `{database}.{schema}.{table}` | -| Databricks | `databricks://{workspace-url}` | `{database}.{schema}.{table}` | -| Trino | `trino://{host}:{port}` | `{catalog}.{schema}.{table}` | -| AWS Glue | `arn:aws:glue:{region}:{accountId}` | `{database}.{table}` | -| S3 | `s3://{bucket}` | `{path}` | +| Platform | Namespace | Name | +| ---------- | ------------------------------------------------ | ----------------------------- | +| BigQuery | `bigquery` | `{project}.{dataset}.{table}` | +| Snowflake | `snowflake://{org}-{account}` | `{database}.{schema}.{table}` | +| Redshift | `redshift://{aws_account_id}:{region}:{cluster}` | `{database}.{schema}.{table}` | +| PostgreSQL | `postgres://{host}:{port}` | `{database}.{schema}.{table}` | +| Databricks | `databricks://{workspace-url}` | `{database}.{schema}.{table}` | +| Trino | `trino://{host}:{port}` | `{catalog}.{schema}.{table}` | +| AWS Glue | `arn:aws:glue:{region}:{accountId}` | `{database}.{table}` | +| S3 | `s3://{bucket}` | `{path}` | For platforms not listed here, follow the [OpenLineage naming conventions][8]. @@ -256,17 +568,17 @@ The `jobType` job facet is **required**. It determines how Datadog classifies an Use `custom` for custom jobs. The values below are used by Datadog's native integrations. Using them for custom jobs may produce unexpected behavior. In particular, `SPARK` prevents span generation. -| Value | Platform | -|---|---| -| `custom` | Custom or unsupported platforms | -| `SPARK` | Apache Spark (native integration only; do not use for custom jobs) | -| `AIRFLOW` | Apache Airflow | -| `DBT` | dbt | -| `BIGQUERY` | Google BigQuery | -| `SNOWFLAKE` | Snowflake | -| `TRINO` | Trino | -| `ICEBERG` | Apache Iceberg | -| `TABLEAU` | Tableau | +| Value | Platform | +| ----------- | ------------------------------------------------------------------ | +| `custom` | Custom or unsupported platforms | +| `SPARK` | Apache Spark (native integration only; do not use for custom jobs) | +| `AIRFLOW` | Apache Airflow | +| `DBT` | dbt | +| `BIGQUERY` | Google BigQuery | +| `SNOWFLAKE` | Snowflake | +| `TRINO` | Trino | +| `ICEBERG` | Apache Iceberg | +| `TABLEAU` | Tableau | #### `processingType` values @@ -280,12 +592,12 @@ Common values include `JOB`, `TASK`, `DAG`, `MODEL`, `COMMAND`, and `QUERY`. ### Other supported facets -| Facet | What Datadog does | -|---|---| -| `parent` | Creates parent-child job hierarchy in the lineage graph | -| `errorMessage` | Generates error spans with `error.message` and `error.stack` tags | -| `tags` | Adds span tags to the run; `_dd.ol_service` value maps to the Datadog service name | -| `sql` | Parses and masks the SQL query; generates query events | +| Facet | What Datadog does | +| -------------- | ---------------------------------------------------------------------------------- | +| `parent` | Creates parent-child job hierarchy in the lineage graph | +| `errorMessage` | Generates error spans with `error.message` and `error.stack` tags | +| `tags` | Adds span tags to the run; `_dd.ol_service` value maps to the Datadog service name | +| `sql` | Parses and masks the SQL query; generates query events | ## Further reading diff --git a/content/en/real_user_monitoring/rum_without_limits/metrics.md b/content/en/real_user_monitoring/rum_without_limits/metrics.md index 4c98f598cb1..ca2e0b4b1b5 100644 --- a/content/en/real_user_monitoring/rum_without_limits/metrics.md +++ b/content/en/real_user_monitoring/rum_without_limits/metrics.md @@ -29,7 +29,8 @@ Datadog provides the below out-of-the-box metrics for a comprehensive overview o | Metric Name | Description | Dimensions | Platform | |-------------|-------------|------------|----------| -| `rum.measure.app.startup_time` | App startup time | Default, Percentiles breakdown | Mobile only | +| `rum.measure.app.startup_to_full_display` | Time to full display during application launch | Default, Percentiles breakdown | Mobile only | +| `rum.measure.app.startup_to_initial_display` | Time to initial display during application launch | Default, Percentiles breakdown | Mobile only | | `rum.measure.error` | Count of errors | Default, Is Crash, View Name | Mobile & Browser | | `rum.measure.error.anr` | Count of ANRs (an Android freeze) | Default, Is Crash, View Name | Mobile only | | `rum.measure.error.hang` | Count of hangs (an iOS freeze) | Default | Mobile only | @@ -84,4 +85,4 @@ Metrics can be managed through [APIs][3] or Datadog's dedicated [Terraform modul [1]: https://app.datadoghq.com/rum/performance-monitoring [2]: /real_user_monitoring/platform/generate_metrics/ [3]: /api/latest/rum-metrics/ -[4]: https://registry.terraform.io/providers/DataDog/datadog/3.60.0/docs/resources/rum_metric \ No newline at end of file +[4]: https://registry.terraform.io/providers/DataDog/datadog/3.60.0/docs/resources/rum_metric diff --git a/layouts/shortcodes/h2-with-copy-btn.html b/layouts/shortcodes/h2-with-copy-btn.html new file mode 100644 index 00000000000..55ff998797b --- /dev/null +++ b/layouts/shortcodes/h2-with-copy-btn.html @@ -0,0 +1,15 @@ +{{- /* + Page-title heading for hardcoded API Overview pages. Renders the same anchored +

as the h2 shortcode, paired with the page Copy button so the two sit + together — mirroring the layout of the generated API pages. + + Use once per page, for the top heading only: it emits the #page-copy-btn + element, so a second use would duplicate that id. +*/ -}} +{{ $anchorized := anchorize .Inner }} +
+

+ {{ .Inner }} +

+ {{ partial "api/api-copy-btn.html" .Page }} +