description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
Option 22
\nobject
The array-map processor transforms each element of a source array by applying\nsub-processors in order and collecting the results into a target array.\nResults can be written to a new array, to the source array (in-place), or to\nan existing target array. Sub-processors can read from $sourceElem.<field>\n(object element field), bare $sourceElem (primitive element), or any parent\nlog attribute path. Sub-processors write to $targetElem.<field> (object\noutput field) or bare $targetElem (primitive output).
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
preserve_source
\nboolean
When false and source != target, the source attribute is removed after\nprocessing. Cannot be false when source == target.
default: true
processors [required]
\n[ <oneOf>]
Sub-processors applied to each element. Allowed types: attribute-remapper,\nstring-builder-processor, arithmetic-processor, category-processor.
Option 1
\nobject
An attribute remapper sub-processor for use inside an array-map processor.\nUnlike the top-level attribute remapper, is_enabled, source_type, and\ntarget_type are not supported.
name
\nstring
Name of the sub-processor.
override_on_conflict
\nboolean
Override the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attribute paths.
target [required]
\nstring
Target attribute path.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 2
\nobject
An arithmetic sub-processor for use inside an array-map processor.\nUnlike the top-level arithmetic processor, is_enabled is not supported.
expression [required]
\nstring
Arithmetic operation to perform.
is_replace_missing
\nboolean
Replace missing attribute values with 0.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 3
\nobject
A string builder sub-processor for use inside an array-map processor.\nUnlike the top-level string builder processor, is_enabled is not supported.
is_replace_missing
\nboolean
Replace missing attribute values with an empty string.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
template [required]
\nstring
Formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 4
\nobject
A category sub-processor for use inside an array-map processor.\nUnlike the top-level category processor, is_enabled is not supported.
categories [required]
\n[object]
Array of filters to match against a log and the corresponding value to assign.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the category value.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
source [required]
\nstring
Attribute path of the source array. Elements are read-only via $sourceElem\ninside sub-processors.
target [required]
\nstring
Attribute path of the output array. Sub-processors write to $targetElem\n(or $targetElem.<field>) to build each output element.
type [required]
\nenum
Type of logs array-map processor. \nAllowed enum values: array-map-processor
default: array-map-processor
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
Option 22
\nobject
The array-map processor transforms each element of a source array by applying\nsub-processors in order and collecting the results into a target array.\nResults can be written to a new array, to the source array (in-place), or to\nan existing target array. Sub-processors can read from $sourceElem.<field>\n(object element field), bare $sourceElem (primitive element), or any parent\nlog attribute path. Sub-processors write to $targetElem.<field> (object\noutput field) or bare $targetElem (primitive output).
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
preserve_source
\nboolean
When false and source != target, the source attribute is removed after\nprocessing. Cannot be false when source == target.
default: true
processors [required]
\n[ <oneOf>]
Sub-processors applied to each element. Allowed types: attribute-remapper,\nstring-builder-processor, arithmetic-processor, category-processor.
Option 1
\nobject
An attribute remapper sub-processor for use inside an array-map processor.\nUnlike the top-level attribute remapper, is_enabled, source_type, and\ntarget_type are not supported.
name
\nstring
Name of the sub-processor.
override_on_conflict
\nboolean
Override the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attribute paths.
target [required]
\nstring
Target attribute path.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 2
\nobject
An arithmetic sub-processor for use inside an array-map processor.\nUnlike the top-level arithmetic processor, is_enabled is not supported.
expression [required]
\nstring
Arithmetic operation to perform.
is_replace_missing
\nboolean
Replace missing attribute values with 0.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 3
\nobject
A string builder sub-processor for use inside an array-map processor.\nUnlike the top-level string builder processor, is_enabled is not supported.
is_replace_missing
\nboolean
Replace missing attribute values with an empty string.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
template [required]
\nstring
Formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 4
\nobject
A category sub-processor for use inside an array-map processor.\nUnlike the top-level category processor, is_enabled is not supported.
categories [required]
\n[object]
Array of filters to match against a log and the corresponding value to assign.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the category value.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
source [required]
\nstring
Attribute path of the source array. Elements are read-only via $sourceElem\ninside sub-processors.
target [required]
\nstring
Attribute path of the output array. Sub-processors write to $targetElem\n(or $targetElem.<field>) to build each output element.
type [required]
\nenum
Type of logs array-map processor. \nAllowed enum values: array-map-processor
default: array-map-processor
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
Option 22
\nobject
The array-map processor transforms each element of a source array by applying\nsub-processors in order and collecting the results into a target array.\nResults can be written to a new array, to the source array (in-place), or to\nan existing target array. Sub-processors can read from $sourceElem.<field>\n(object element field), bare $sourceElem (primitive element), or any parent\nlog attribute path. Sub-processors write to $targetElem.<field> (object\noutput field) or bare $targetElem (primitive output).
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
preserve_source
\nboolean
When false and source != target, the source attribute is removed after\nprocessing. Cannot be false when source == target.
default: true
processors [required]
\n[ <oneOf>]
Sub-processors applied to each element. Allowed types: attribute-remapper,\nstring-builder-processor, arithmetic-processor, category-processor.
Option 1
\nobject
An attribute remapper sub-processor for use inside an array-map processor.\nUnlike the top-level attribute remapper, is_enabled, source_type, and\ntarget_type are not supported.
name
\nstring
Name of the sub-processor.
override_on_conflict
\nboolean
Override the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attribute paths.
target [required]
\nstring
Target attribute path.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 2
\nobject
An arithmetic sub-processor for use inside an array-map processor.\nUnlike the top-level arithmetic processor, is_enabled is not supported.
expression [required]
\nstring
Arithmetic operation to perform.
is_replace_missing
\nboolean
Replace missing attribute values with 0.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 3
\nobject
A string builder sub-processor for use inside an array-map processor.\nUnlike the top-level string builder processor, is_enabled is not supported.
is_replace_missing
\nboolean
Replace missing attribute values with an empty string.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
template [required]
\nstring
Formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 4
\nobject
A category sub-processor for use inside an array-map processor.\nUnlike the top-level category processor, is_enabled is not supported.
categories [required]
\n[object]
Array of filters to match against a log and the corresponding value to assign.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the category value.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
source [required]
\nstring
Attribute path of the source array. Elements are read-only via $sourceElem\ninside sub-processors.
target [required]
\nstring
Attribute path of the output array. Sub-processors write to $targetElem\n(or $targetElem.<field>) to build each output element.
type [required]
\nenum
Type of logs array-map processor. \nAllowed enum values: array-map-processor
default: array-map-processor
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
Option 22
\nobject
The array-map processor transforms each element of a source array by applying\nsub-processors in order and collecting the results into a target array.\nResults can be written to a new array, to the source array (in-place), or to\nan existing target array. Sub-processors can read from $sourceElem.<field>\n(object element field), bare $sourceElem (primitive element), or any parent\nlog attribute path. Sub-processors write to $targetElem.<field> (object\noutput field) or bare $targetElem (primitive output).
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
preserve_source
\nboolean
When false and source != target, the source attribute is removed after\nprocessing. Cannot be false when source == target.
default: true
processors [required]
\n[ <oneOf>]
Sub-processors applied to each element. Allowed types: attribute-remapper,\nstring-builder-processor, arithmetic-processor, category-processor.
Option 1
\nobject
An attribute remapper sub-processor for use inside an array-map processor.\nUnlike the top-level attribute remapper, is_enabled, source_type, and\ntarget_type are not supported.
name
\nstring
Name of the sub-processor.
override_on_conflict
\nboolean
Override the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attribute paths.
target [required]
\nstring
Target attribute path.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 2
\nobject
An arithmetic sub-processor for use inside an array-map processor.\nUnlike the top-level arithmetic processor, is_enabled is not supported.
expression [required]
\nstring
Arithmetic operation to perform.
is_replace_missing
\nboolean
Replace missing attribute values with 0.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 3
\nobject
A string builder sub-processor for use inside an array-map processor.\nUnlike the top-level string builder processor, is_enabled is not supported.
is_replace_missing
\nboolean
Replace missing attribute values with an empty string.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
template [required]
\nstring
Formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 4
\nobject
A category sub-processor for use inside an array-map processor.\nUnlike the top-level category processor, is_enabled is not supported.
categories [required]
\n[object]
Array of filters to match against a log and the corresponding value to assign.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the category value.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
source [required]
\nstring
Attribute path of the source array. Elements are read-only via $sourceElem\ninside sub-processors.
target [required]
\nstring
Attribute path of the output array. Sub-processors write to $targetElem\n(or $targetElem.<field>) to build each output element.
type [required]
\nenum
Type of logs array-map processor. \nAllowed enum values: array-map-processor
default: array-map-processor
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
Option 22
\nobject
The array-map processor transforms each element of a source array by applying\nsub-processors in order and collecting the results into a target array.\nResults can be written to a new array, to the source array (in-place), or to\nan existing target array. Sub-processors can read from $sourceElem.<field>\n(object element field), bare $sourceElem (primitive element), or any parent\nlog attribute path. Sub-processors write to $targetElem.<field> (object\noutput field) or bare $targetElem (primitive output).
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
preserve_source
\nboolean
When false and source != target, the source attribute is removed after\nprocessing. Cannot be false when source == target.
default: true
processors [required]
\n[ <oneOf>]
Sub-processors applied to each element. Allowed types: attribute-remapper,\nstring-builder-processor, arithmetic-processor, category-processor.
Option 1
\nobject
An attribute remapper sub-processor for use inside an array-map processor.\nUnlike the top-level attribute remapper, is_enabled, source_type, and\ntarget_type are not supported.
name
\nstring
Name of the sub-processor.
override_on_conflict
\nboolean
Override the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attribute paths.
target [required]
\nstring
Target attribute path.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 2
\nobject
An arithmetic sub-processor for use inside an array-map processor.\nUnlike the top-level arithmetic processor, is_enabled is not supported.
expression [required]
\nstring
Arithmetic operation to perform.
is_replace_missing
\nboolean
Replace missing attribute values with 0.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 3
\nobject
A string builder sub-processor for use inside an array-map processor.\nUnlike the top-level string builder processor, is_enabled is not supported.
is_replace_missing
\nboolean
Replace missing attribute values with an empty string.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
template [required]
\nstring
Formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 4
\nobject
A category sub-processor for use inside an array-map processor.\nUnlike the top-level category processor, is_enabled is not supported.
categories [required]
\n[object]
Array of filters to match against a log and the corresponding value to assign.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the category value.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
source [required]
\nstring
Attribute path of the source array. Elements are read-only via $sourceElem\ninside sub-processors.
target [required]
\nstring
Attribute path of the output array. Sub-processors write to $targetElem\n(or $targetElem.<field>) to build each output element.
type [required]
\nenum
Type of logs array-map processor. \nAllowed enum values: array-map-processor
default: array-map-processor
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
id
\nstring
ID of the pipeline.
is_enabled
\nboolean
Whether or not the pipeline is enabled.
is_read_only
\nboolean
Whether or not the pipeline can be edited.
name [required]
\nstring
Name of the pipeline.
processors
\n[ <oneOf>]
Ordered list of processors in this pipeline.
Option 1
\nobject
Create custom grok rules to parse the full message or a specific attribute of your raw event.\nFor more information, see the parsing section.
grok [required]
\nobject
Set of rules for the grok parser.
match_rules [required]
\nstring
List of match rules for the grok parser, separated by a new line.
support_rules
\nstring
List of support rules for the grok parser, separated by a new line.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
samples
\n[string]
List of sample logs to test this grok parser.
source [required]
\nstring
Name of the log attribute to parse.
default: message
type [required]
\nenum
Type of logs grok parser. \nAllowed enum values: grok-parser
default: grok-parser
Option 2
\nobject
As Datadog receives logs, it timestamps them using the value(s) from any of these default attributes.
\ntimestampdate_timestampTimestampeventTimepublished_dateIf your logs put their dates in an attribute not in this list,\n use the log date Remapper Processor to define their date attribute as the official log timestamp.\n The recognized date formats are ISO8601, UNIX (the milliseconds EPOCH format), and RFC3164.
Note: If your logs don’t contain any of the default attributes\n and you haven’t defined your own date attribute, Datadog timestamps\n the logs with the date it received them.
If multiple log date remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs date remapper. \nAllowed enum values: date-remapper
default: date-remapper
Option 3
\nobject
Use this Processor if you want to assign some attributes as the official status.
Each incoming status value is mapped as follows.
\nemerg or f (case-insensitive) map to emerg (0)a (case-insensitive) map to alert (1)c (case-insensitive) map to critical (2)err (case-insensitive) map to error (3)w (case-insensitive) map to warning (4)n (case-insensitive) map to notice (5)i (case-insensitive) map to info (6)d, trace or verbose (case-insensitive) map to debug (7)o or matching OK or Success (case-insensitive) map to OKinfo (6)Note: If multiple log status remapper processors can be applied to a given log,\n only the first one (according to the pipelines order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs status remapper. \nAllowed enum values: status-remapper
default: status-remapper
Option 4
\nobject
Use this processor if you want to assign one or more attributes as the official service.
Note: If multiple service remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
type [required]
\nenum
Type of logs service remapper. \nAllowed enum values: service-remapper
default: service-remapper
Option 5
\nobject
The message is a key attribute in Datadog.\nIt is displayed in the message column of the Log Explorer and you can do full string search on it.\nUse this Processor to define one or more attributes as the official log message.
Note: If multiple log message remapper processors can be applied to a given log,\nonly the first one (according to the pipeline order) is taken into account.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: msg
type [required]
\nenum
Type of logs message remapper. \nAllowed enum values: message-remapper
default: message-remapper
Option 6
\nobject
The remapper processor remaps any source attribute(s) or tag to another target attribute or tag.\nConstraints on the tag/attribute name are explained in the Tag Best Practice documentation.\nSome additional constraints are applied as : or , are not allowed in the target tag/attribute name.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
override_on_conflict
\nboolean
Override or not the target element if already set,
preserve_source
\nboolean
Remove or preserve the remapped source element.
source_type
\nstring
Defines if the sources are from log attribute or tag.
default: attribute
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Final attribute or tag name to remap the sources to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
target_type
\nstring
Defines if the final attribute or tag name is from log attribute or tag.
default: attribute
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 7
\nobject
This processor extracts query parameters and other important parameters from a URL.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
normalize_ending_slashes
\nboolean
Normalize the ending slashes or not.
sources [required]
\n[string]
Array of source attributes.
default: http.url
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.url_details
type [required]
\nenum
Type of logs URL parser. \nAllowed enum values: url-parser
default: url-parser
Option 8
\nobject
The User-Agent parser takes a User-Agent attribute and extracts the OS, browser, device, and other user data.\nIt recognizes major bots like the Google Bot, Yahoo Slurp, and Bing.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_encoded
\nboolean
Define if the source attribute is URL encoded or not.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: http.useragent
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: http.useragent_details
type [required]
\nenum
Type of logs User-Agent parser. \nAllowed enum values: user-agent-parser
default: user-agent-parser
Option 9
\nobject
Use the Category Processor to add a new attribute (without spaces or special characters in the new attribute name)\nto a log matching a provided search query. Use categories to create groups for an analytical view.\nFor example, URL groups, machine groups, environments, and response time buckets.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the target attribute which value is defined by the matching category.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
Option 10
\nobject
Use the Arithmetic Processor to add a new attribute (without spaces or special characters\nin the new attribute name) to a log with the result of the provided formula.\nThis enables you to remap different time attributes with different units into a single attribute,\nor to compute operations on attributes within the same log.
The formula can use parentheses and the basic arithmetic operators -, +, *, /.
By default, the calculation is skipped if an attribute is missing.\nSelect “Replace missing attribute by 0” to automatically populate\nmissing attribute values with 0 to ensure that the calculation is done.\nAn attribute is missing if it is not found in the log attributes,\nor if it cannot be converted to a number.
Notes:
\n- needs to be space split in the formula as it can also be contained in attribute names.0.1234567891,\nthe actual value stored for the attribute is 0.123456789.expression [required]
\nstring
Arithmetic operation between one or more log attributes.
is_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of expression by 0, false\nskip the operation if an attribute is missing.
name
\nstring
Name of the processor.
target [required]
\nstring
Name of the attribute that contains the result of the arithmetic operation.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 11
\nobject
Use the string builder processor to add a new attribute (without spaces or special characters)\nto a log with the result of the provided template.\nThis enables aggregation of different attributes or raw strings into a single attribute.
The template is defined by both raw text and blocks with the syntax %{attribute_path}.
Notes:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
is_replace_missing
\nboolean
If true, it replaces all missing attributes of template by an empty string.\nIf false (default), skips the operation for missing attributes.
name
\nstring
Name of the processor.
target [required]
\nstring
The name of the attribute that contains the result of the template.
template [required]
\nstring
A formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 12
\nobject
Nested Pipelines are pipelines within a pipeline. Use Nested Pipelines to split the processing into two steps.\nFor example, first use a high-level filtering such as team and then a second level of filtering based on the\nintegration, service, or any other tag or attribute.
A pipeline can contain Nested Pipelines and Processors whereas a Nested Pipeline can only contain Processors.
description
\nstring
A description of the pipeline.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
processors
\n[object]
Ordered list of processors in this pipeline.
tags
\n[string]
A list of tags associated with the pipeline.
type [required]
\nenum
Type of logs pipeline processor. \nAllowed enum values: pipeline
default: pipeline
Option 13
\nobject
The GeoIP parser takes an IP address attribute and extracts if available\nthe Continent, Country, Subdivision, and City information in the target attribute path.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources [required]
\n[string]
Array of source attributes.
default: network.client.ip
target [required]
\nstring
Name of the parent attribute that contains all the extracted details from the sources.
default: network.client.geoip
type [required]
\nenum
Type of GeoIP parser. \nAllowed enum values: geo-ip-parser
default: geo-ip-parser
Option 14
\nobject
Use the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in the processors mapping table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
default_lookup
\nstring
Value to set the target attribute if the source value is not found in the list.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_table [required]
\n[string]
Mapping table of values for the source attribute and their associated target attribute values,\nformatted as ["source_key1,target_value1", "source_key2,target_value2"]
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list\nor the default_lookup if not found in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 15
\nobject
Note: Reference Tables are in public beta.\nUse the Lookup Processor to define a mapping between a log attribute\nand a human readable value saved in a Reference Table.\nFor example, you can use the Lookup Processor to map an internal service ID\ninto a human readable service name. Alternatively, you could also use it to check\nif the MAC address that just attempted to connect to the production\nenvironment belongs to your list of stolen machines.
is_enabled
\nboolean
Whether or not the processor is enabled.
lookup_enrichment_table [required]
\nstring
Name of the Reference Table for the source attribute and their associated target attribute values.
name
\nstring
Name of the processor.
source [required]
\nstring
Source attribute used to perform the lookup.
target [required]
\nstring
Name of the attribute that contains the corresponding value in the mapping list.
type [required]
\nenum
Type of logs lookup processor. \nAllowed enum values: lookup-processor
default: lookup-processor
Option 16
\nobject
There are two ways to improve correlation between application traces and logs.
\nFollow the documentation on how to inject a trace ID in the application logs\n and by default log integrations take care of all the rest of the setup.
\nUse the Trace remapper processor to define a log attribute as its associated trace ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.trace_id
type [required]
\nenum
Type of logs trace remapper. \nAllowed enum values: trace-id-remapper
default: trace-id-remapper
Option 17
\nobject
There are two ways to define correlation between application spans and logs:
\nFollow the documentation on how to inject a span ID in the application logs.\n Log integrations automatically handle all remaining setup steps by default.
\nUse the span remapper processor to define a log attribute as its associated span ID.
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
sources
\n[string]
Array of source attributes.
default: dd.span_id
type [required]
\nenum
Type of logs span remapper. \nAllowed enum values: span-id-remapper
default: span-id-remapper
Option 18
\nobject
A processor for extracting, aggregating, or transforming values from JSON arrays within your logs.\nSupported operations are:
\nis_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
operation [required]
\n<oneOf>
Configuration of the array processor operation to perform.
Option 1
\nobject
Operation that appends a value to a target array attribute.
preserve_source
\nboolean
Remove or preserve the remapped source element.
default: true
source [required]
\nstring
Attribute path containing the value to append.
target [required]
\nstring
Attribute path of the array to append to.
type [required]
\nenum
Operation type. \nAllowed enum values: append
Option 2
\nobject
Operation that computes the length of a source array and stores the result in the target attribute.
source [required]
\nstring
Attribute path of the array to measure.
target [required]
\nstring
Attribute that receives the computed length.
type [required]
\nenum
Operation type. \nAllowed enum values: length
Option 3
\nobject
Operation that finds an object in a source array using a filter, and then extracts a specific value into the target attribute.
filter [required]
\nstring
Filter condition expressed as key:value used to find the matching element.
source [required]
\nstring
Attribute path of the array to search into.
target [required]
\nstring
Attribute that receives the extracted value.
type [required]
\nenum
Operation type. \nAllowed enum values: select
value_to_extract [required]
\nstring
Key of the value to extract from the matching element.
type [required]
\nenum
Type of logs array processor. \nAllowed enum values: array-processor
default: array-processor
Option 19
\nobject
The decoder processor decodes any source attribute containing a\nbase64/base16-encoded UTF-8/ASCII string back to its original value, storing the\nresult in a target attribute.
binary_to_text_encoding [required]
\nenum
The encoding used to represent the binary data. \nAllowed enum values: base64,base16
input_representation [required]
\nenum
The original representation of input string. \nAllowed enum values: utf_8,integer
is_enabled
\nboolean
Whether the processor is enabled.
name
\nstring
Name of the processor.
source [required]
\nstring
Name of the log attribute with the encoded data.
target [required]
\nstring
Name of the log attribute that contains the decoded data.
type [required]
\nenum
Type of logs decoder processor. \nAllowed enum values: decoder-processor
default: decoder-processor
Option 20
\nobject
A processor that has additional validations and checks for a given schema. Currently supported schema types include OCSF.
is_enabled
\nboolean
Whether or not the processor is enabled.
mappers [required]
\n[ <oneOf>]
The LogsSchemaProcessor mappers.
Option 1
\nobject
The schema remapper maps source log fields to their correct fields.
name [required]
\nstring
Name of the logs schema remapper.
override_on_conflict
\nboolean
Override or not the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attributes.
target [required]
\nstring
Target field to map log source field to.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs schema remapper. \nAllowed enum values: schema-remapper
Option 2
\nobject
Use the Schema Category Mapper to categorize log event into enum fields.\nIn the case of OCSF, they can be used to map sibling fields which are composed of an ID and a name.
Notes:
\ncategories [required]
\n[object]
Array of filters to match or not a log and their\ncorresponding name to assign a custom value to the log.
filter [required]
\nobject
Filter for logs.
query
\nstring
The filter query.
id [required]
\nint64
ID to inject into the category.
name [required]
\nstring
Value to assign to target schema field.
fallback
\nobject
Used to override hardcoded category values with a value pulled from a source attribute on the log.
sources
\nobject
Fallback sources used to populate value of field.
<any-key>
\n[string]
values
\nobject
Values that define when the fallback is used.
<any-key>
\nstring
name [required]
\nstring
Name of the logs schema category mapper.
targets [required]
\nobject
Name of the target attributes which value is defined by the matching category.
id
\nstring
ID of the field to map log attributes to.
name
\nstring
Name of the field to map log attributes to.
type [required]
\nenum
Type of logs schema category mapper. \nAllowed enum values: schema-category-mapper
name [required]
\nstring
Name of the processor.
schema [required]
\nobject
Configuration of the schema data to use.
class_name [required]
\nstring
Class name of the schema to use.
class_uid [required]
\nint64
Class UID of the schema to use.
profiles
\n[string]
Optional list of profiles to modify the schema.
schema_type [required]
\nstring
Type of schema to use.
version [required]
\nstring
Version of the schema to use.
type [required]
\nenum
Type of logs schema processor. \nAllowed enum values: schema-processor
default: schema-processor
Option 21
\nobject
Use this processor to remove an attribute from a log during processing.\nThe processor strips the specified attribute from the log event, which is useful\nwhen the attribute contains sensitive data or is no longer needed downstream.
attribute_to_exclude [required]
\nstring
Name of the log attribute to remove from the log event.
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
type [required]
\nenum
Type of logs exclude attribute processor. \nAllowed enum values: exclude-attribute
default: exclude-attribute
Option 22
\nobject
The array-map processor transforms each element of a source array by applying\nsub-processors in order and collecting the results into a target array.\nResults can be written to a new array, to the source array (in-place), or to\nan existing target array. Sub-processors can read from $sourceElem.<field>\n(object element field), bare $sourceElem (primitive element), or any parent\nlog attribute path. Sub-processors write to $targetElem.<field> (object\noutput field) or bare $targetElem (primitive output).
is_enabled
\nboolean
Whether or not the processor is enabled.
name
\nstring
Name of the processor.
preserve_source
\nboolean
When false and source != target, the source attribute is removed after\nprocessing. Cannot be false when source == target.
default: true
processors [required]
\n[ <oneOf>]
Sub-processors applied to each element. Allowed types: attribute-remapper,\nstring-builder-processor, arithmetic-processor, category-processor.
Option 1
\nobject
An attribute remapper sub-processor for use inside an array-map processor.\nUnlike the top-level attribute remapper, is_enabled, source_type, and\ntarget_type are not supported.
name
\nstring
Name of the sub-processor.
override_on_conflict
\nboolean
Override the target element if already set.
preserve_source
\nboolean
Remove or preserve the remapped source element.
sources [required]
\n[string]
Array of source attribute paths.
target [required]
\nstring
Target attribute path.
target_format
\nenum
If the target_type of the remapper is attribute, try to cast the value to a new specific type.\nIf the cast is not possible, the original type is kept. string, integer, or double are the possible types.\nIf the target_type is tag, this parameter may not be specified. \nAllowed enum values: auto,string,integer,double
type [required]
\nenum
Type of logs attribute remapper. \nAllowed enum values: attribute-remapper
default: attribute-remapper
Option 2
\nobject
An arithmetic sub-processor for use inside an array-map processor.\nUnlike the top-level arithmetic processor, is_enabled is not supported.
expression [required]
\nstring
Arithmetic operation to perform.
is_replace_missing
\nboolean
Replace missing attribute values with 0.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
type [required]
\nenum
Type of logs arithmetic processor. \nAllowed enum values: arithmetic-processor
default: arithmetic-processor
Option 3
\nobject
A string builder sub-processor for use inside an array-map processor.\nUnlike the top-level string builder processor, is_enabled is not supported.
is_replace_missing
\nboolean
Replace missing attribute values with an empty string.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the result.
template [required]
\nstring
Formula with one or more attributes and raw text.
type [required]
\nenum
Type of logs string builder processor. \nAllowed enum values: string-builder-processor
default: string-builder-processor
Option 4
\nobject
A category sub-processor for use inside an array-map processor.\nUnlike the top-level category processor, is_enabled is not supported.
categories [required]
\n[object]
Array of filters to match against a log and the corresponding value to assign.
filter
\nobject
Filter for logs.
query
\nstring
The filter query.
name
\nstring
Value to assign to the target attribute.
name
\nstring
Name of the sub-processor.
target [required]
\nstring
Target attribute path for the category value.
type [required]
\nenum
Type of logs category processor. \nAllowed enum values: category-processor
default: category-processor
source [required]
\nstring
Attribute path of the source array. Elements are read-only via $sourceElem\ninside sub-processors.
target [required]
\nstring
Attribute path of the output array. Sub-processors write to $targetElem\n(or $targetElem.<field>) to build each output element.
type [required]
\nenum
Type of logs array-map processor. \nAllowed enum values: array-map-processor
default: array-map-processor
tags
\n[string]
A list of tags associated with the pipeline.
type
\nstring
Type of pipeline.
You can set a default namespace for your current context to avoid having to type -n <NAMESPACE_NAME> with every command:
kubectl config set-context --current --namespace=<NAMESPACE_NAME>The storageClass (sc) used in the example file below is gp3, which is not installed by default and is not the default sc for EKS. To create the gp3 storage class, follow the instructions in this AWS guide. If you do not want to set gp3 as the default (and migrate from gp2), set storageclass.kubernetes.io/is-default-class: "false".
Datadog recommends gp3 storage volumes for BYOC Logs to provide the IOPS and throughput flexibility to support higher indexing rates.
+indexer.podSize to medium in datadog-values.yaml and run the helm upgrade --install command again.<HOST>:<PORT> que se utiliza para redirigir mediante proxy las métricas también se utilizará para redirigir los logs.
-Si utilizas el transporte TCP, consulta la sección Proxy TCP para logs.
-<HOST>:<PORT> que se utiliza para redirigir mediante proxy las métricas NO se utilizará para redirigir los logs. Consulta la página Proxy para logs.
-flare el subcomando no es compatible con su sitio de Datadog seleccionado ({{< region-param key="dd_datacenter" >}}). Usa envío manual de flares en su lugar.