Skip to content

Add documentation for event externalisation #3014

Open
alexane-bougeardbebin-sekoia wants to merge 3 commits into
mainfrom
event_eternalisation
Open

Add documentation for event externalisation #3014
alexane-bougeardbebin-sekoia wants to merge 3 commits into
mainfrom
event_eternalisation

Conversation

@alexane-bougeardbebin-sekoia

@alexane-bougeardbebin-sekoia alexane-bougeardbebin-sekoia commented May 27, 2026

Copy link
Copy Markdown
Contributor

@@ -0,0 +1,97 @@
# Event Externalisation

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexane-bougeardbebin-sekoia I think you put externalisation instead of eternalisation almost everwhere ?

@alexane-bougeardbebin-sekoia

Copy link
Copy Markdown
Contributor Author

@mathieuguarino-ctrl does it answer your need?

@rdettai-sk is this accurate? I gathered info from dust, but I can't make sure if everything is (still) up to date and accurate.

| **2. Event externalisation** | The relevant events are fetched, permanently indexed, and attached to the alert |

!!! note
Because these two steps run in parallel, there can be a **short latency** between when the alert appears in your queue and when all associated events are fully visible within it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

several minutes I think

@github-actions

github-actions Bot commented May 27, 2026

Copy link
Copy Markdown

Newest code from alexane-bougeardbebin-sekoia has been published to preview environment

🚀 Latest deployment was built on 2026-05-29 11:22:31 (b3749b99690cd1e1febe8b0ff8d7a573949426cf).

Comment on lines +39 to +58
## Understanding Timestamps

Understanding timestamp fields is key to accurate investigation and timeline reconstruction:

| Field | Meaning |
|---|---|
| **Alert date** | Date of the **first trigger** — it does not update even if new events enrich the alert later |
| **`timestamp`** | When the event was **generated by the source system** (in UTC) |
| **`event.created`** | When **Sekoia.io acknowledged/ingested** the event |
| **`event.start`** | When the monitored activity **began** |
| **`event.end`** | When the monitored activity **ended** |

The general ordering rule is:

```
event.start ≤ event.end ≤ event.created
```

!!! info
Events must be sent to Sekoia.io in UTC. If an event is sent in local time (e.g. UTC+2) with a 09:00 timestamp, it will be stored as 09:00 UTC.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is common to all events, I don't think it should be in this page.

Updated terminology from 'externalisation' to 'eternalisation' and revised related content for clarity and accuracy.
Comment on lines +81 to +82
!!! warning
Do not rely solely on the alert date for timeline reconstruction. Always use the `event.start` and `timestamp` fields for accurate sequencing.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another warning :

events linked to anomaly rules are not eternalized (there is no event tab in an Anomaly Alert)

Clarified information on eternalisation of events and its implications for alert visibility and investigation. Fixed formatting issues and added warnings regarding missing events.
@alexane-bougeardbebin-sekoia

Copy link
Copy Markdown
Contributor Author

@rdettai-sk @mathieuguarino-ctrl I'm starting the article again to make sure I cover only eternalisation. I am a bit embarrassed because if it sure helps users understand what this is and the cause of some issues, it doesn't really help them understand :

  • the benefits
  • how they can act
  • It gives a rather poor image of this process, don't you think? But I am missing infos to make it more actionnable/positive

@mathieuguarino-ctrl

mathieuguarino-ctrl commented May 29, 2026

Copy link
Copy Markdown
Contributor

Hello @alexane-bougeardbebin-sekoia

  • Eternalisation can be compared with Hot and Cold storage, which have specific retention values
  • How they can act : nothing to do on this part once an alert is created, it is eternalised, but some clients don't have the info that these events are here forever
  • Maybe we can link it to retention doc

Regarding this :

Events may be missing from an Alert

If eternalisation partially fails, an alert may be published with fewer events than expected, or none at all. This can make investigation difficult, as the supporting evidence is incomplete.

Eternalisation should (and must) not fail (except with incidents/bug?) I don't think this is something we want written.

@rdettai-sk

Copy link
Copy Markdown

it doesn't really help them understand :
the benefits
how they can act
It gives a rather poor image of this process, don't you think? But I am missing infos to make it more actionnable/positive

I can give you my understanding that too biased by the technical side of things, but I think it's interesting for you nevertheless. AFAIK, what we want functionally is just to have some events attached to alerts. Eternalization is a technical implementation detail that serves several purposes:

  • avoid loosing the events attached to alerts when their retention expires: retention policies just delete big chunks of data, it is not simple to tell them to delete everything but event x, y and z.
  • fast retrieval of events for an alert: technically, with the data volumes we have it is not straightforward to cherry pick an arbitrary number of events and attach them to an id (here the alert id)
  • (I might be forgetting reasons...)

But I insist, these are implementation details, I don't think we need or want to expose it to users. For instance when working on Exalog, we considered removing eternalization entirely. Users wouldn't have cared, as long as they have their events, their alerts and some events attached to alerts. We didn't because it is still technically the best approach.

I don't know how exactly the decision was made to expose eternals explicitly as a SOL datasource. I think again that it was the simplest technical solution to give users access to both to their main event stream and the events attached to alerts. But purely functionally, I guess the ideal approach would have been to have just one events datasource from which you can magically get events that are attached to alerts or not. This is what happens on the "Events" page.

I think you should involve someone from product (maybe @Sengthay ?), because I might not have the full picture on the functional side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants