|
| 1 | +--- |
| 2 | +name: span-timeline-events |
| 3 | +description: Use when adding, modifying, or debugging OTel span timeline events in the trace view. Covers event structure, ClickHouse storage constraints, rendering in SpanTimeline component, admin visibility, and the step-by-step process for adding new events. |
| 4 | +allowed-tools: Read, Write, Edit, Glob, Grep, Bash |
| 5 | +--- |
| 6 | + |
| 7 | +# Span Timeline Events |
| 8 | + |
| 9 | +The trace view's right panel shows a timeline of events for the selected span. These are OTel span events rendered by `app/utils/timelineSpanEvents.ts` and the `SpanTimeline` component. |
| 10 | + |
| 11 | +## How They Work |
| 12 | + |
| 13 | +1. **Span events** in OTel are attached to a parent span. In ClickHouse, they're stored as separate rows with `kind: "SPAN_EVENT"` sharing the parent span's `span_id`. The `#mergeRecordsIntoSpanDetail` method reassembles them into the span's `events` array at query time. |
| 14 | +2. The timeline only renders events whose `name` starts with `trigger.dev/` - all others are silently filtered out. |
| 15 | +3. The **display name** comes from `properties.event` (not the span event name), mapped through `getFriendlyNameForEvent()`. |
| 16 | +4. Events are shown on the **span they belong to** - events on one span don't appear in another span's timeline. |
| 17 | + |
| 18 | +## ClickHouse Storage Constraint |
| 19 | + |
| 20 | +When events are written to ClickHouse, `spanEventsToTaskEventV1Input()` filters out events whose `start_time` is not greater than the parent span's `startTime`. Events at or before the span start are silently dropped. This means span events must have timestamps strictly after the span's own `startTimeUnixNano`. |
| 21 | + |
| 22 | +## Timeline Rendering (SpanTimeline component) |
| 23 | + |
| 24 | +The `SpanTimeline` component in `app/components/run/RunTimeline.tsx` renders: |
| 25 | + |
| 26 | +1. **Events** (thin 1px line with hollow dots) - all events from `createTimelineSpanEventsFromSpanEvents()` |
| 27 | +2. **"Started"** marker (thick cap) - at the span's `startTime` |
| 28 | +3. **Duration bar** (thick 7px line) - from "Started" to "Finished" |
| 29 | +4. **"Finished"** marker (thick cap) - at `startTime + duration` |
| 30 | + |
| 31 | +The thin line before "Started" only appears when there are events with timestamps between the span start and the first child span. For the Attempt span this works well (Dequeued -> Pod scheduled -> Launched -> etc. all happen before execution starts). Events all get `lineVariant: "light"` (thin) while the execution bar gets `variant: "normal"` (thick). |
| 32 | + |
| 33 | +## Trace View Sort Order |
| 34 | + |
| 35 | +Sibling spans (same parent) are sorted by `start_time ASC` from the ClickHouse query. The `createTreeFromFlatItems` function preserves this order. Event timestamps don't affect sort order - only the span's own `start_time`. |
| 36 | + |
| 37 | +## Event Structure |
| 38 | + |
| 39 | +```typescript |
| 40 | +// OTel span event format |
| 41 | +{ |
| 42 | + name: "trigger.dev/run", // Must start with "trigger.dev/" to render |
| 43 | + timeUnixNano: "1711200000000000000", |
| 44 | + attributes: [ |
| 45 | + { key: "event", value: { stringValue: "dequeue" } }, // The actual event type |
| 46 | + { key: "duration", value: { intValue: 150 } }, // Optional: duration in ms |
| 47 | + ] |
| 48 | +} |
| 49 | +``` |
| 50 | + |
| 51 | +## Admin-Only Events |
| 52 | + |
| 53 | +`getAdminOnlyForEvent()` controls visibility. Events default to **admin-only** (`true`). |
| 54 | + |
| 55 | +| Event | Admin-only | Friendly name | |
| 56 | +|-------|-----------|---------------| |
| 57 | +| `dequeue` | No | Dequeued | |
| 58 | +| `fork` | No | Launched | |
| 59 | +| `import` | No (if no fork event) | Importing task file | |
| 60 | +| `create_attempt` | Yes | Attempt created | |
| 61 | +| `lazy_payload` | Yes | Lazy attempt initialized | |
| 62 | +| `pod_scheduled` | Yes | Pod scheduled | |
| 63 | +| (default) | Yes | (raw event name) | |
| 64 | + |
| 65 | +## Adding New Timeline Events |
| 66 | + |
| 67 | +1. Add OTLP span event with `name: "trigger.dev/<scope>"` and `properties.event: "<type>"` |
| 68 | +2. Event timestamp must be strictly after the parent span's `startTimeUnixNano` (ClickHouse drops earlier events) |
| 69 | +3. Add friendly name in `getFriendlyNameForEvent()` in `app/utils/timelineSpanEvents.ts` |
| 70 | +4. Set admin visibility in `getAdminOnlyForEvent()` |
| 71 | +5. Optionally add help text in `getHelpTextForEvent()` |
| 72 | + |
| 73 | +## Key Files |
| 74 | + |
| 75 | +- `app/utils/timelineSpanEvents.ts` - filtering, naming, admin logic |
| 76 | +- `app/components/run/RunTimeline.tsx` - `SpanTimeline` component (thin line + thick bar rendering) |
| 77 | +- `app/presenters/v3/SpanPresenter.server.ts` - loads span data including events |
| 78 | +- `app/v3/eventRepository/clickhouseEventRepository.server.ts` - `spanEventsToTaskEventV1Input()` (storage filter), `#mergeRecordsIntoSpanDetail` (reassembly) |
0 commit comments