diff --git a/explore-analyze/toc.yml b/explore-analyze/toc.yml index 2167ca2afa..15633cc75a 100644 --- a/explore-analyze/toc.yml +++ b/explore-analyze/toc.yml @@ -410,7 +410,9 @@ toc: - file: workflows/triggers/alert-triggers.md - file: workflows/steps.md - file: workflows/data.md - - file: workflows/create-workflows.md + children: + - file: workflows/data/templating.md + - file: workflows/create-workflows.md - file: workflows/monitor-troubleshoot.md - file: workflows/manage-workflows.md - hidden: workflows/use-cases.md diff --git a/explore-analyze/workflows/Untitled b/explore-analyze/workflows/Untitled new file mode 100644 index 0000000000..6b9e06225d --- /dev/null +++ b/explore-analyze/workflows/Untitled @@ -0,0 +1 @@ + to enrich a new \ No newline at end of file diff --git a/explore-analyze/workflows/data.md b/explore-analyze/workflows/data.md index 37ea5188cb..83e7b6cb90 100644 --- a/explore-analyze/workflows/data.md +++ b/explore-analyze/workflows/data.md @@ -2,7 +2,198 @@ applies_to: stack: preview 9.3 serverless: preview -description: Learn how data is processed and transformed in Elastic workflows. +description: Learn how data flows through workflows, use dynamic templating, and handle errors gracefully. --- -# Data \ No newline at end of file +# Data and error handling [workflows-data] + +A key feature of workflows is the ability to pass data between steps and handle failures gracefully. This page explains the mechanisms for controlling data flow and building resilient, fault-tolerant automations. + +## Data flow [workflows-data-flow] + +Every step in a workflow produces an output. By default, this output is added to a global `steps` object in the workflow's context, making it available to all subsequent steps. + +### Access step outputs [workflows-access-outputs] + +Use the following syntax to access the output of a specific step: + +```text +steps..output +``` + +You can also access error information from a step: + +```text +steps..error +``` + +### Example: Chain steps to move output data [workflows-chain-steps-example] + +This example demonstrates a common pattern: searching for data in one step and using the results in a later step. In this case, the workflow searches for a specific user's full name, then uses it to create a new security case. + +```yaml +name: Create case for a specific user +steps: + - name: find_user_by_id + type: elasticsearch.search + with: + index: "my-user-index" + query: + term: + user.id: "u-123" + + - name: create_case_for_user + type: kibana.createCaseDefaultSpace + with: + title: "Investigate user u-123" + description: "A case has been opened for user {{steps.find_user_by_id.output.hits.hits[0]._source.user.fullName}}." + tags: ["user-investigation"] + connector: + id: "none" + name: "none" + type: ".none" +``` + +In this example: + +1. The `find_user_by_id` step searches an index for a document. +2. The `create_case_for_user` step uses the output of the first step to enrich a new [{{elastic-sec}} case](../../solutions/security/investigate/cases.md). +3. The `description` field accesses `steps.find_user_by_id.output.hits.hits[0]._source.user.fullName` to dynamically include the user's full name in the case description. + +## Error handling [workflows-error-handling] + +By default, if any step in a workflow fails, the entire workflow execution stops immediately. You can override this behavior using the `on-failure` block, which supports retry logic, fallback steps, and continuation options. + +### Configuration levels [workflows-on-failure-levels] + +You can configure `on-failure` at two levels: + +**Step-level** — applies to a specific step: + +```yaml +steps: + - name: api-call + type: http + on-failure: + retry: + max-attempts: 3 + delay: "5s" +``` + +**Workflow-level** (configured under `settings`) - applies to all steps as the default error handling behavior: + +```yaml +settings: + on-failure: + retry: + max-attempts: 2 + delay: "1s" +steps: + - name: api-call + type: http +``` + +:::{note} +Step-level `on-failure` configuration always overrides workflow-level settings. +::: + +### Retry [workflows-on-failure-retry] + +Retries the failed step a configurable number of times, with an optional delay between attempts. + +```yaml +on-failure: + retry: + max-attempts: 3 # Required, minimum 1 (for example, "1", "2", "5") + delay: "5s" # Optional, duration format (for example, "5s", "1m", "2h") +``` + +The workflow fails when all retries are exhausted. + +### Fallback [workflows-on-failure-fallback] + +Executes alternative steps after the primary step fails and all retries are exhausted. In the following example, when the `delete_critical_document` step fails, the workflow executes two additional steps: one sends a Slack notification to devops-alerts using `{{workflow.name}}`, while the other logs the error details from the failed step using `{{steps.delete_critical_document.error}}`. + +```yaml +on-failure: + fallback: + - name: notify_on_failure + type: slack + connector-id: "devops-alerts" + with: + message: "Failed to delete document in workflow '{{workflow.name}}'" + - name: log_failure + type: console + with: + message: "Document deletion failed, error: {{steps.delete_critical_document.error}}" +``` + +Within fallback steps, access error information from the failed primary step using `steps..error`. + +### Continue [workflows-on-failure-continue] + +Continues workflow execution even if a step fails. The failure is recorded, but does not interrupt the workflow. + +```yaml +on-failure: + continue: true +``` + +### Combining options [workflows-on-failure-combining] + +You can combine multiple failure-handling options. They are processed in this order: retry → fallback → continue. + +In the following example: +1. The step retries up to 2 times with a 1-second delay. +2. If all retries fail, the fallback steps execute. +3. The workflow continues regardless of the outcome. + +```yaml +- name: create_ticket + type: jira + connector-id: "my-jira-project" + with: + projectKey: "PROJ" + summary: "New issue from workflow" + on-failure: + retry: + max-attempts: 2 + delay: "1s" + fallback: + - name: notify_jira_failure + type: slack + connector-id: "devops-alerts" + with: + message: "Warning: Failed to create ticket. Continuing workflow." + continue: true +``` + +### Restrictions [workflows-on-failure-restrictions] + +- Flow-control steps (`if`, `foreach`) cannot have workflow-level `on-failure` configurations. +- Fallback steps execute only after all retries have been exhausted. +- When combined, failure-handling options are processed in this order: retry → fallback → continue. + +## Dynamic values with templating [workflows-dynamic-values] + +To inject dynamic values into your workflow steps, use the templating engine. The templating engine uses the [Liquid templating language](https://liquidjs.com/) and allows you to: + +- **Reference step outputs**: Access data from previous steps using `steps..output`. +- **Use constants**: Reference workflow-level constants with `consts.`. +- **Apply filters**: Transform values with filters like `upcase`, `downcase`, and `date`. +- **Add conditional logic**: Use `if`/`else` statements for dynamic content. +- **Loop through data**: Iterate over arrays with `for` loops. + +For complete syntax details and examples, refer to [Templating engine](./data/templating.md). + +## Quick reference [workflows-data-quick-reference] + +By combining data flow, templating, and robust error handling, you can build complex, reliable automations that react to dynamic conditions and recover from unexpected failures. + +| Action | Syntax | Description | +|---------|--------|-------------| +| Step output | `steps..output` | Access the result of a previous step. | +| Step error | `steps..error` | Access error details from a failed step. | +| Retry on failure | `on-failure.retry` | Retry a failed step with optional delay. | +| Fallback steps | `on-failure.fallback` | Define recovery actions when a step fails. | +| Continue on failure | `on-failure.continue: true` | Allow the workflow to proceed after a failure. | \ No newline at end of file diff --git a/explore-analyze/workflows/data/templating.md b/explore-analyze/workflows/data/templating.md new file mode 100644 index 0000000000..ba5e3778ac --- /dev/null +++ b/explore-analyze/workflows/data/templating.md @@ -0,0 +1,297 @@ +--- +applies_to: + stack: preview 9.3 + serverless: preview +description: Learn how to use the Liquid templating engine to create dynamic workflows. +--- + +# Templating engine [workflows-templating] + +The workflow templating engine enables dynamic, type-safe template rendering using the [Liquid templating language](https://liquidjs.com/). It allows you to inject variables, apply transformations, and control data flow throughout your workflows. + +## Syntax overview [workflows-template-syntax] + +The templating engine supports several syntax patterns for different use cases: + +| Syntax | Purpose | Example | +|--------|---------|---------| +| Double curly braces | Insert values as strings | `"Hello, {{name}}"` | +| Dollar-sign prefix | Preserve data types (arrays, objects, numbers) | `${{myArray}}` | +| Percent tags | Control flow (conditionals, loops) | `{%if active%}...{%endif%}` | +| Raw tags | Output literal curly braces | `{%raw%}{{}}{%endraw%}` | + +### String interpolation [workflows-string-interpolation] + +Use double curly braces for basic string interpolation. Variables and expressions inside the braces are evaluated and rendered as strings. + +```yaml +message: "Hello {{user.name}}!" # Result: "Hello Alice" +url: "https://api.example.com/users/{{user.id}}" # Result: "https://api.example.com/users/12" +``` + +### Type-preserving expressions [workflows-type-preserving] + +Use the dollar-sign prefix (`${{ }}`) when you need to preserve the original data type (array, object, number, boolean). + +```yaml +# String syntax - converts to string +tags: "{{inputs.tags}}" # Result: "[\"admin\", \"user\"]" (string) + +# Type-preserving syntax - keeps original type +tags: "${{inputs.tags}}" # Result: ["admin", "user"] (actual array) +``` + +:::{important} +The type-preserving syntax must occupy the entire string value. You cannot mix it with other text. + +✅ **Valid:** + +```yaml +tags: "${{inputs.tags}}" +``` + +❌ **Invalid:** + +```yaml +message: "Tags are: ${{inputs.tags}}" +``` +::: + +| Feature | String syntax | Type-preserving syntax | +|---------|---------------|------------------------| +| Output type | Always string | Preserves original type | +| Arrays | Stringified | Actual array | +| Objects | Stringified | Actual object | +| Booleans | `"true"` / `"false"` | `true` / `false` | +| Numbers | `"123"` | `123` | + +### Control flow [workflows-control-flow] + +Liquid tags are control flow constructs that use the `{% %}` syntax. Unlike output expressions, tags execute logic without directly rendering a value. + +**Conditionals:** + +```yaml +message: | + {% if user.role == 'admin' %} + Welcome, administrator! + {% else %} + Welcome, user! + {% endif %} +``` + +**Loops:** + +```yaml +message: | + {% for item in items %} + - {{item.name}} + {% endfor %} +``` + +### Escaping template syntax [workflows-escaping] + +Use raw tags to output literal curly brace characters without rendering them: + +```yaml +value: "{%raw%}{{_ingest.timestamp}}{%endraw%}" # Result: "{{_ingest.timestamp}}" +``` + +## Working with data [workflows-working-with-data] + +This section covers common patterns for accessing and transforming data in your workflows. + +### Reference inputs [workflows-ref-inputs] + +Reference input parameters defined in the workflow using `{{inputs.}}`. Inputs are defined at the workflow level and can be provided when the workflow is triggered manually. + +```yaml +inputs: + - name: environment + type: string + required: true + default: "staging" + - name: batchSize + type: number + default: 100 + +triggers: + - type: manual + +steps: + - name: log_config + type: console + with: + message: | + Running with: + - Environment: {{inputs.environment}} + - Batch Size: {{inputs.batchSize}} +``` + +### Reference outputs [workflows-ref-step-outputs] + +Access output data from previous steps using `{{steps..output}}`: + +```yaml +steps: + - name: search_users + type: elasticsearch.search + with: + index: "users" + query: + term: + status: "active" + + - name: send_notification + type: slack + connector-id: "my-slack" + with: + message: "Found {{steps.search_users.output.hits.total.value}} active users" +``` + +### Reference constants [workflows-ref-constants] + +Reference workflow-level constants using `{{consts.}}`. Constants are defined at the workflow level and can be referenced when the workflow is triggered. + +```yaml +consts: + indexName: "my-index" + environment: "production" + +steps: + - name: search_data + type: elasticsearch.search + with: + index: "{{consts.indexName}}" + query: + match: + env: "{{consts.environment}}" +``` + +### Apply filters [workflows-apply-filters] + +Transform values using filters with the pipe `|` character: + +```yaml +message: | + User: {{user.name | upcase}} + Email: {{user.email | downcase}} + Created: {{user.created_at | date: "%Y-%m-%d"}} +``` + +### Preserve array and object types [workflows-preserve-types] + +When passing arrays or objects between steps, use the type-preserving syntax (`${{ }}`) to avoid stringification: + +```yaml +steps: + - name: get_tags + type: elasticsearch.search + with: + index: "config" + query: + term: + type: "tags" + + - name: create_document + type: elasticsearch.index + with: + index: "reports" + document: + # Preserves the array type, doesn't stringify it + tags: "${{steps.get_tags.output.hits.hits[0]._source.tags}}" +``` + +:::{important} +The type-preserving syntax must occupy the entire string value. You cannot mix it with other text. + +✅ **Valid:** + +```yaml +tags: "${{inputs.tags}}" +``` + +❌ **Invalid:** + +```yaml +message: "Tags are: ${{inputs.tags}}" +``` +::: + +### Use conditionals for dynamic content [workflows-conditionals-example] + +Add logic to customize output based on data: + +```yaml +steps: + - name: send_message + type: slack + connector-id: "alerts" + with: + message: | + {% if steps.search.output.hits.total.value > 100 %} + ⚠️ HIGH ALERT: {{steps.search.output.hits.total.value}} events detected! + {% else %} + ✅ Normal: {{steps.search.output.hits.total.value}} events detected. + {% endif %} +``` + +### Loop through results [workflows-loops-example] + +Iterate over arrays to process multiple items: + +```yaml +steps: + - name: summarize_results + type: console + with: + message: | + Found users: + {% for hit in steps.search_users.output.hits.hits %} + - {{hit._source.name}} ({{hit._source.email}}) + {% endfor %} +``` + +## Template rendering behavior [workflows-template-rendering] + +The engine renders templates recursively through all data structures, processing nested objects and arrays. + +**Input:** + +```yaml +message: "Hello {{user.name}}" +config: + url: "{{api.url}}" +tags: ["{{tag1}}", "{{tag2}}"] +``` + +**Rendered output:** + +```yaml +message: "Hello Alice" +config: + url: "https://api.example.com" +tags: ["admin", "user"] +``` + +### Type handling [workflows-type-handling] + +| Type | Behavior | +|------|----------| +| Strings | Processed as templates: variables are interpolated, and filters are applied | +| Numbers, Booleans, Null | Returned as-is | +| Arrays | Each element is processed recursively | +| Objects | Each property value is processed recursively (keys are not processed) | + +### Null and undefined handling [workflows-null-handling] + +| Case | Behavior | +|------|----------| +| Null values | Returned as-is | +| Undefined variables | Returned as empty string in string syntax and as `undefined` in type-preserving syntax | +| Missing context properties | Treated as undefined | + +## Learn more + +- [Liquid templating language](https://shopify.github.io/liquid/) +- [LiquidJS documentation](https://liquidjs.com/) \ No newline at end of file