Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 26 additions & 16 deletions config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5868,16 +5868,21 @@ menu:
parent: observability_pipelines_processors
identifier: observability_pipelines_processors_split_array
weight: 317
- name: Tag Control
url: observability_pipelines/processors/tag_control
parent: observability_pipelines_processors
identifier: observability_pipelines_processors_tag_control
weight: 318
- name: Tags
url: observability_pipelines/processors/tags
parent: observability_pipelines_processors
identifier: observability_pipelines_processors_tags
weight: 318
weight: 319
- name: Throttle
url: observability_pipelines/processors/throttle
parent: observability_pipelines_processors
identifier: observability_pipelines_processors_throttle
weight: 319
weight: 320
- name: Destinations
url: observability_pipelines/destinations/
parent: observability_pipelines
Expand Down Expand Up @@ -5918,76 +5923,81 @@ menu:
parent: observability_pipelines_destinations
identifier: observability_pipelines_datadog_logs
weight: 407
- name: Datadog Metrics
url: observability_pipelines/destinations/datadog_metrics/
parent: observability_pipelines_destinations
identifier: observability_pipelines_datadog_metrics
weight: 408
- name: Elasticsearch
url: observability_pipelines/destinations/elasticsearch/
parent: observability_pipelines_destinations
identifier: observability_pipelines_elasticsearch
weight: 408
weight: 409
- name: Google Chronicle
url: observability_pipelines/destinations/google_chronicle
parent: observability_pipelines_destinations
identifier: observability_pipelines_google_chronicle
weight: 409
weight: 410
- name: Google Cloud Storage
identifier: observability_pipelines_google_cloud_storage
url: /observability_pipelines/destinations/google_cloud_storage/
parent: observability_pipelines_destinations
weight: 410
weight: 411
- name: Google Pub/Sub
identifier: observability_pipelines_google_pubsub
url: /observability_pipelines/destinations/google_pubsub/
parent: observability_pipelines_destinations
weight: 411
weight: 412
- name: HTTP Client
url: observability_pipelines/destinations/http_client/
parent: observability_pipelines_destinations
identifier: observability_pipelines_http_client
weight: 412
weight: 413
- name: Kafka
url: observability_pipelines/destinations/kafka/
parent: observability_pipelines_destinations
identifier: observability_pipelines_kafka
weight: 413
weight: 414
- name: Microsoft Sentinel
identifier: observability_pipelines_microsoft_sentinel
url: /observability_pipelines/destinations/microsoft_sentinel/
parent: observability_pipelines_destinations
weight: 414
weight: 415
- name: New Relic
identifier: observability_pipelines_new_relic
url: /observability_pipelines/destinations/new_relic/
parent: observability_pipelines_destinations
weight: 415
weight: 416
- name: OpenSearch
url: observability_pipelines/destinations/opensearch
parent: observability_pipelines_destinations
identifier: observability_pipelines_opensearch
weight: 416
weight: 417
- name: SentinelOne
url: observability_pipelines/destinations/sentinelone
parent: observability_pipelines_destinations
identifier: observability_pipelines_sentinelone
weight: 417
weight: 418
- name: Socket
url: observability_pipelines/destinations/socket
parent: observability_pipelines_destinations
identifier: observability_pipelines_socket
weight: 418
weight: 419
- name: Splunk HEC
url: observability_pipelines/destinations/splunk_hec
parent: observability_pipelines_destinations
identifier: observability_pipelines_splunk_hec
weight: 419
weight: 420
- name: Sumo Logic Hosted Collector
url: observability_pipelines/destinations/sumo_logic_hosted_collector
parent: observability_pipelines_destinations
identifier: observability_pipelines_sumo_logic_hosted_collector
weight: 420
weight: 421
- name: Syslog
url: observability_pipelines/destinations/syslog
parent: observability_pipelines_destinations
identifier: observability_pipelines_syslog
weight: 421
weight: 422
- name: Packs
url: observability_pipelines/packs/
parent: observability_pipelines
Expand Down
4 changes: 2 additions & 2 deletions content/en/observability_pipelines/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,11 +125,11 @@ Observability Pipelines includes prebuilt templates for common data routing and
{{% tab "Metrics" %}}

<div class="alert alert-info">
Metrics Volume and Cardinality Control is in Preview. Fill out the <a href="https://www.datadoghq.com/product-preview/metrics-ingestion-and-cardinality-control-in-observability-pipelines/">form</a> to request access.</div>
Metric Tag Governance is in Preview. Fill out the <a href="https://www.datadoghq.com/product-preview/metrics-ingestion-and-cardinality-control-in-observability-pipelines/">form</a> to request access.</div>

| Template | Description |
|----------|-------------|
| Metrics Volume and Cardinality Control | Manage the quality and volume of your metrics by keeping only the metrics you need, standardizing metrics tagging, and removing unwanted tags to prevent high cardinality. |
| Metric Tag Governance | Manage the quality and volume of your metrics by keeping only the metrics you need, standardizing metrics tagging, and removing unwanted tags to prevent high cardinality. |

{{% /tab %}}
{{< /tabs >}}
Expand Down
103 changes: 95 additions & 8 deletions content/en/observability_pipelines/configuration/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,19 +28,106 @@ Observability Pipelines lets you collect and process logs and metrics ({{< toolt

Build and deploy pipelines to collect, transform, and route your data using one of these methods:

- Pipeline UI
- [API][4]
- [Terraform][5]
- [Pipeline UI][4]
- [API][5]
- [Terraform][6]

See [Set Up Pipelines][6] for source, processor, and destination configuration details.
## Pipeline types

## Further reading
There are two types of pipelines:

{{< tabs >}}
{{% tab "Logs" %}}

Use one of the [logs templates][1] to create a log pipeline.

- Archive Logs
- Dual Ship Logs
- Generate Log-based Metrics
- Log Enrichment
- Log Volume Control
- Sensitive Data Redaction
- Split Logs

See [Set Up Pipelines][2] for more information on setting up a source, processors, and destinations.

[1]: /observability_pipelines/configuration/explore_templates/?tab=logs#templates
[2]: /observability_pipelines/configuration/set_up_pipelines/

{{% /tab %}}

{{% tab "Metrics" %}}

<div class="alert alert-info">
Metric Tag Governance is in Preview. Fill out the <a href="https://www.datadoghq.com/product-preview/metrics-ingestion-and-cardinality-control-in-observability-pipelines/">form</a> to request access.</div>

Use the [Metric Tag Governance][1] template to create a metrics pipeline.

See [Set Up Pipelines][2] for more information on setting up a source, processors and destination.

### Metrics data

Metrics sent to Observability Pipelines include the following:

- `name`: The metric name.
- `kind`: There are two kinds of metrics:
- `absolute` metrics: Represents the current value of a measurement at the time it is reported.
- `incremental` metrics: Represents the change in a measurement since the last reported value, which the system aggregates over time.
- `value`: The [metric type](#metric-types):
- `counter`
- `gauge`
- `distribution`
- `histogram`
- `timestamp`: The date and time the metric is created.
- `tags`: Includes tags such as `host`.

The `counter` metric type is the only `incremental` metric. `gauge`, `distribution`, and `histogram` metric types are `absolute` metrics.

An example of a metric:

```
{
"name":"datadog.agent.retry_queue_duration.bytes_per_sec",
"tags":{
"agent":"core",
"domain":"https://7-72-3-app.agent.datadoghq.com",
"host":"COMP-YGVQDJG75L",
"source_type_name":"System",
"env:prod"
},
"timestamp":"2025-11-28T13:03:09Z",
"kind":"absolute",
"gauge":{"value":454.1372767857143}
}
```

### Metric types

The available metric types:

| Metric type | Description | Example |
| ----------- | ----------- | ------- |
| COUNTER | Represents the total number of event occurrences in one time interval. This value can be reset to zero, but cannot be decreased. | You want to count the number of logs with `status:error`. |
| GAUGE | Represents a snapshot of events in one time interval. | You want to measure the latest CPU utilization per host for all logs in the production environment. |
| DISTRIBUTION | Represent the global statistical distribution of a set of values calculated across your entire distributed infrastructure in one time interval. | You want to measure the average time it takes for an API call to be made. |
| HISTOGRAM | Represents the statistical distribution of a set of values calculated in one time interval. | You want to measure response time distributions for a service or endpoint. |

See [Metric Types][3] for more information.

[1]: /observability_pipelines/configuration/explore_templates/?tab=metrics#metric-tag-governance
[2]: /observability_pipelines/configuration/set_up_pipelines/
[3]: /metrics/types/?tab=gauge#metric-types

{{% /tab %}}
{{< /tabs >}}

## Further reading

{{< partial name="whats-next/whats-next.html" >}}

[1]: /observability_pipelines/sources/
[2]: /observability_pipelines/processors/
[3]: /observability_pipelines/destinations/
[4]: /api/latest/observability-pipelines/#create-a-new-pipeline
[5]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs
[6]: /observability_pipelines/configuration/set_up_pipelines/
[4]: https://app.datadoghq.com/observability-pipelines
[5]: /api/latest/observability-pipelines/#create-a-new-pipeline
[6]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs
Original file line number Diff line number Diff line change
Expand Up @@ -65,14 +65,14 @@ When you have logs from different services and applications, you might need to s
{{% /tab %}}
{{% tab "Metrics" %}}

### Metrics Volume and Cardinality Control
### Metric Tag Governance

<div class="alert alert-info">
Metrics Volume and Cardinality Control is in Preview. Fill out the <a href="https://www.datadoghq.com/product-preview/metrics-ingestion-and-cardinality-control-in-observability-pipelines/">form</a> to request access.</div>
Metric Tag Governance is in Preview. Fill out the <a href="https://www.datadoghq.com/product-preview/metrics-ingestion-and-cardinality-control-in-observability-pipelines/">form</a> to request access.</div>

Metrics capture signals about your environment and offer insight into your system health, business workflows, and security activities. These metrics are sent from your various applications, network devices, and nodes, but the value of individual metrics can vary significantly.

To help you manage the quality and volume of your metrics, use the Metrics Volume and Cardinality Control template to process them in Observability Pipelines before sending them to your destinations. You can use processors to keep only the metrics you need, standardize metrics tagging, and remove unwanted tags to prevent high cardinality.
To help you manage the quality and volume of your metrics, use the Metric Tag Governance template to process them in Observability Pipelines before sending them to your destinations. You can use processors to keep only the metrics you need, standardize metrics tagging, and remove unwanted tags to prevent high cardinality.

{{% /tab %}}
{{< /tabs >}}
Expand Down
Loading
Loading