[axonius][ticket] Add Axonius Ticket datastream#16604
[axonius][ticket] Add Axonius Ticket datastream#16604ShourieG merged 10 commits intoelastic:feature/axonius-0.1.0from
Conversation
muskan-agarwal26
changed the title
andrewkroh
added
documentation
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
|
Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution! |
…ntegrations into datastream-ticket
ShourieG
added
the
Team:Security-Service Integrations
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
ShourieG
left a comment
ShourieG
left a comment
There was a problem hiding this comment.
🤖 AI-Generated Review | Elastic Integration PR Review Bot
⚠️ This is an automated review generated by an AI assistant. Please verify all suggestions before applying changes. This review does not represent a human reviewer's opinion.
PR Review | elastic/integrations #16604
Field Mapping
Data Stream: ticket (package: axonius)
File: packages/axonius/data_stream/ticket/fields/fields.yml
Issue 1: All custom fields missing required descriptions
Severity: 🟠 High
Location: packages/axonius/data_stream/ticket/fields/fields.yml line 7
Problem: Every custom field definition is missing a required description property. All 41 leaf fields under axonius.ticket.* and axonius.ticket.event.* lack descriptions, which are mandatory for custom fields.
Recommendation:
- name: adapter_list_length
type: long
description: |
The number of adapters associated with this ticket asset.
- name: status
type: keyword
description: |
The current status of the ticket (e.g., open, closed, in_progress).Issue 2: Field description typed as keyword — may need text type
Severity: 🔵 Low
Location: packages/axonius/data_stream/ticket/fields/fields.yml line 27
Problem: axonius.ticket.description is typed as keyword. Ticket descriptions are typically free-form text; keyword prevents full-text search and truncates values exceeding ignore_above (default 1024 chars).
Recommendation:
- name: description
type: text
description: |
Human-readable description of the ticket.Issue 3: Field summary typed as keyword — may need text type
Severity: 🔵 Low
Location: packages/axonius/data_stream/ticket/fields/fields.yml line 55
Problem: axonius.ticket.summary is typed as keyword. Ticket summaries are typically free-form text and benefit from full-text search indexing.
Recommendation:
- name: summary
type: text
description: |
Brief summary or title of the ticket.File: packages/axonius/data_stream/ticket/fields/is-transform-source-true.yml
Issue 4: Custom field not under vendor namespace (labels.is_transform_source)
Severity: 🟠 High
Location: packages/axonius/data_stream/ticket/fields/is-transform-source-true.yml line 1
Problem: The field labels.is_transform_source is not nested under the axonius vendor namespace. Custom fields must be placed under the package vendor namespace to comply with integration requirements.
Recommendation:
# Rename to axonius.ticket.is_transform_source to comply with vendor namespace requirements
- name: axonius
type: group
fields:
- name: ticket
type: group
fields:
- name: is_transform_source
type: constant_keyword
value: 'true'
description: ...💡 Suggestions
- The
axonius.ticket.eventsub-group shadows the ECSevent.*namespace conceptually — consider renaming toaxonius.ticket.adapter_eventor similar to avoid potential confusion with ECSevent.*fields. axonius.ticket.adaptersis typed askeyword— if this field holds multiple adapter names (an array), this is fine, but consider adding adescriptionto clarify whether it's a single value or a list.axonius.ticket.tenant_numberis typed askeyword— if this is always numeric,longmay be more appropriate; if it contains alphanumeric tenant identifiers,keywordis correct.
Pipeline
Data Stream: ticket (package: axonius)
File: packages/axonius/data_stream/ticket/elasticsearch/ingest_pipeline/default.yml
Issue 1: Script processor missing on_failure handler
Severity: 🟡 Medium
Location: packages/axonius/data_stream/ticket/elasticsearch/ingest_pipeline/default.yml line 60
Problem: The script processor at lines 60–75 (renaming event.data.* fields) has no on_failure handler. Script processors are high-risk (runtime errors, unexpected data shapes) and should have an on_failure block to append to error.message.
Recommendation:
- script:
tag: script_rename_event_data_fields
lang: painless
description: Renames all event.data.* fields to root level.
source: |-
# ... existing source ...
on_failure:
- append:
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'Issue 2: set event.end copies potentially unparsed date string without if guard
Severity: 🟡 Medium
Location: packages/axonius/data_stream/ticket/elasticsearch/ingest_pipeline/default.yml line 115
Problem: The set processor copying axonius.ticket.closed to event.end has no if guard. If axonius.ticket.closed was not successfully parsed by the preceding date processor (e.g., the date parse failed and the field still holds a raw string), the raw string value would be copied into event.end, which is a date type field, potentially causing indexing failures.
Recommendation:
- set:
field: event.end
tag: set_event_end_from_ticket_event_data_closed
copy_from: axonius.ticket.closed
ignore_empty_value: true
if: ctx.axonius?.ticket?.closed != nullIssue 3: set event.created copies potentially unparsed date string without if guard
Severity: 🟡 Medium
Location: packages/axonius/data_stream/ticket/elasticsearch/ingest_pipeline/default.yml line 133
Problem: Same concern as Issue 2 — the set processor for event.created from axonius.ticket.created has no if guard. If the date parse failed, a raw string could be written to event.created.
Recommendation:
- set:
field: event.created
tag: set_event_created_from_ticket_event_data_created
copy_from: axonius.ticket.created
ignore_empty_value: true
if: ctx.axonius?.ticket?.created != nullIssue 4: fetch_time and first_fetch_time not removed after date parsing
Severity: 🔵 Low
Location: packages/axonius/data_stream/ticket/elasticsearch/ingest_pipeline/default.yml line 220
Problem: axonius.ticket.fetch_time and axonius.ticket.first_fetch_time are parsed by date processors but are not included in the cleanup remove list. These parsed date fields remain in the document as duplicates under axonius.ticket.* alongside their ECS-mapped counterparts.
Recommendation:
- remove:
field:
- axonius.ticket.event.accurate_for_datetime
- axonius.ticket.accurate_for_datetime
- axonius.ticket.closed
- axonius.ticket.created
- axonius.ticket.description
- axonius.ticket.fetch_time
- axonius.ticket.first_fetch_time
tag: remove_custom_duplicate_fields
ignore_missing: true
if: ctx.tags == null || !ctx.tags.contains('preserve_duplicate_custom_fields')Issue 5: Defensive error-state setters placed after null-cleanup script
Severity: 🔵 Low
Location: packages/axonius/data_stream/ticket/elasticsearch/ingest_pipeline/default.yml line 256
Problem: The end-of-pipeline set event.kind: pipeline_error and append preserve_original_event processors are placed after the script_to_drop_null_values cleanup script. If error.message was set by a processor-level on_failure, the cleanup script could potentially remove it if it evaluates to an empty string or null.
Recommendation:
# Move these two processors BEFORE script_to_drop_null_values
- set:
field: event.kind
tag: set_pipeline_error_into_event_kind
value: pipeline_error
if: ctx.error?.message != null
- append:
field: tags
value: preserve_original_event
allow_duplicates: false
if: ctx.error?.message != null
# Then run the null cleanup script
- script:
tag: script_to_drop_null_values
...💡 Suggestions
event.categoryandevent.typeare not set. For ticket/case data,event.category: ['configuration']or['iam']andevent.type: ['info']or['change']could be appropriate to improve ECS compliance for SIEM use cases.@timestampfallback: Ifaxonius.ticket.updatedis absent, consider falling back toaxonius.ticket.createdas@timestampto avoid defaulting to ingest time for events that have a meaningful creation timestamp.axonius.ticket.updatedis copied to@timestampbut is not in the cleanupremovelist — it remains as a duplicate underaxonius.ticket.updated. Consider adding it to the remove list.
Input Configuration
Data Stream: ticket (package: axonius)
File: packages/axonius/data_stream/ticket/agent/stream/cel.yml.hbs
Issue 1: Error branch returns events as object instead of array
Severity: 🔴 Critical
Location: packages/axonius/data_stream/ticket/agent/stream/cel.yml.hbs line 102
Problem: The non-200 error branch returns "events": {"error": {...}} (a plain map). The CEL input framework requires events to be a list of event objects. A plain object will cause a runtime type error or silently drop all error events, meaning error events are never published.
Recommendation:
"events": [
{
"error": {
"code": string(resp.StatusCode),
"id": string(resp.Status),
"message": "POST:" + state.url.trim_right("/") + "/api/v2/assets/" + string(state.worklist.asset_type_list[0]) + (
(size(resp.Body) != 0) ?
string(resp.Body)
:
string(resp.Status) + " (" + string(resp.StatusCode) + ")"
),
},
},
],Issue 2: Missing has() guard on assets.specific_data before map()
Severity: 🟡 Medium
Location: packages/axonius/data_stream/ticket/agent/stream/cel.yml.hbs line 71
Problem: assets.specific_data.map(d, ...) accesses specific_data without a has() guard. If any asset object in body.assets lacks this field, the entire batch will fail with a runtime no-such-key error.
Recommendation:
assets.?specific_data.orValue([]).map(d,
{
"message": { ... }.encode_json(),
}
)
Issue 3: CEL program needs minor celfmt formatting fixes
Severity: 🔵 Low
Location: packages/axonius/data_stream/ticket/agent/stream/cel.yml.hbs line 66
Problem: Two ternary conditions (resp.StatusCode == 200 and size(resp.Body) != 0) are not wrapped in parentheses, which is the canonical celfmt style.
Recommendation:
- ).do_request().as(resp, resp.StatusCode == 200 ?
+ ).do_request().as(resp, (resp.StatusCode == 200) ?
...
- size(resp.Body) != 0 ?
+ (size(resp.Body) != 0) ?Issue 4: No HTTP 429 (rate-limit) handling
Severity: 🔵 Low
Location: packages/axonius/data_stream/ticket/agent/stream/cel.yml.hbs line 100
Problem: All non-200 responses fall into the same error branch. If the Axonius API enforces rate limits, the integration will emit an error event and wait until the next scheduled interval rather than backing off gracefully.
Recommendation:
resp.StatusCode == 429 ?
{
"events": [],
"want_more": false,
}
:
{
"events": [{"error": {...}}],
"want_more": false,
}
Transform
Package: axonius
File: packages/axonius/elasticsearch/transform/latest_ticket/transform.yml
Issue 1: Transform frequency below 1m minimum (30s)
Severity: 🟡 Medium
Location: packages/axonius/elasticsearch/transform/latest_ticket/transform.yml line 17
Problem: The frequency is set to 30s, which is below the recommended minimum of 1m. Sub-minute transform frequencies cause excessive and unnecessary load on the Elasticsearch cluster.
Recommendation:
frequency: 5mIssue 2: retention_policy uses event.ingested instead of @timestamp
Severity: 🟡 Medium
Location: packages/axonius/elasticsearch/transform/latest_ticket/transform.yml line 30
Problem: The retention_policy.time.field is set to event.ingested rather than @timestamp. Retention policies should use @timestamp to expire documents based on actual event time, not ingestion time. Using event.ingested can cause documents to be retained or expired based on pipeline processing time rather than the business-meaningful event timestamp.
Recommendation:
retention_policy:
time:
field: '@timestamp'
max_age: 24hIssue 3: retention_policy max_age of 24h may be too short for long-lived tickets
Severity: 🟡 Medium
Location: packages/axonius/elasticsearch/transform/latest_ticket/transform.yml line 31
Problem: A latest transform is meant to hold the current state of each ticket. If a ticket is not updated within 24 hours, its record will be purged from the destination index, causing data loss for long-lived tickets that may be open for days or weeks.
Recommendation:
retention_policy:
time:
field: '@timestamp'
max_age: 2160h # 90 daysFile: packages/axonius/elasticsearch/transform/latest_ticket/fields/fields.yml
Issue 4: normalize: [array] on adapters/adapter_categories — verify aggregation type in transform.yml
Severity: 🔵 Low
Location: packages/axonius/elasticsearch/transform/latest_ticket/fields/fields.yml line 1
Problem: normalize: [array] is declared on axonius.ticket.adapters and axonius.ticket.event.adapter_categories. While this attribute is valid in transform field definitions and the Axonius domain model supports multi-valued output, the correctness depends on the aggregation type used in transform.yml. If top_metrics or last_value (single value) is used instead of scripted_metric, normalize: [array] should be removed.
Recommendation:
Confirm the pivot aggregation for axonius.ticket.adapters and axonius.ticket.event.adapter_categories produces array output (e.g., scripted_metric collecting all values). If top_metrics or last_value is used, remove normalize: [array].
File: packages/axonius/elasticsearch/transform/latest_ticket/fields/ecs.yml
Issue 5: ECS fields may need explicit external: ecs for pre-8.19 stack compatibility
Severity: 🔵 Low
Location: packages/axonius/elasticsearch/transform/latest_ticket/fields/ecs.yml line 1
Problem: For stacks prior to 8.19/9.1, the ecs@mappings component template does not automatically apply to transform destination indices — ECS fields must be explicitly declared with external: ecs in the destination field definitions. Without this, ECS field mappings may be absent on older stacks, causing mapping conflicts or missing type information.
Recommendation:
# Example: explicit ECS declaration for pre-8.19 compatibility
- name: "@timestamp"
external: ecs
- name: event.kind
external: ecs
- name: host.name
external: ecsVerify the kibana.version constraint in manifest.yml. If the minimum supported version is below 8.19/9.1, ensure all ECS fields in ecs.yml include external: ecs.
Summary
| Severity | Count |
|---|---|
| 🔴 Critical | 1 |
| 🟠 High | 3 |
| 🟡 Medium | 8 |
| 🔵 Low | 8 |
Total Actionable Items: 20
andrewkroh
removed
the
New Integration
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
|
Please add a description attribut to all custom fields. |
@ShourieG |
Yes please add some sort of description that makes sense, you can use an llm to best infer a description that makes sense. |
> 1. Added description to all custom fields. > 2. Changed retention period to 90d and field to' @timestamp'.
andrewkroh
removed
dashboard
6 participants
Proposed commit message
The release includes ticket data stream and associated dashboard.
Axonius fields are mapped to their corresponding ECS fields where possible.
Test samples were derived from live data samples, which were subsequently
sanitized.
Checklist
changelog.ymlfile.How to test this PR locally
To test the axonius package:
Related issues
Screenshots