feat: emit compute OTel spans (provision, restore, snapshot) in run traces

Supervisor emits OTel spans for compute lifecycle events so they appear
in the run's trace view with per-stage timing breakdowns.

Spans:
- compute.provision: emitted after gateway create returns, includes
  gateway/agent/fcrun timing and cache indicators from _timing response
- compute.restore: emitted after gateway restore returns (supervisor-side
  timing only, gateway restore timing not yet surfaced)
- compute.snapshot: emitted from snapshot callback handler using
  duration_ms from the agent, trace context from in-memory map (best-effort,
  does not survive restarts - TRI-7992)

Implementation:
- Hand-rolled OTLP JSON client (otlpTrace.ts) - builds ExportTraceServiceRequest
  payload and fire-and-forget POSTs to TRIGGER_API_URL/otel
- Trace context (traceparent) from DequeuedMessage links spans to run trace
- Resource attributes (ctx.environment.id, ctx.run.id, etc.) link to the
  correct run in the trace view
- COMPUTE_TRACE_SPANS_ENABLED env var (default true) to disable in prod
- Span start time offset by -1ms to ensure stable sort order before attempt

Also adds .claude/rules/span-timeline-events.md documenting how the trace
view timeline events system works (trigger.dev/ prefix, admin visibility,
ClickHouse SPAN_EVENT storage, start_time filter constraint).

Author: nicktrn

.claude/rules/span-timeline-events.md

@@ -0,0 +1,72 @@
+# Span Timeline Events
+
+The trace view's right panel shows a timeline of events for the selected span. These are OTel span events rendered by `app/utils/timelineSpanEvents.ts` and the `SpanTimeline` component.
+
+## How They Work
+
+1. **Span events** in OTel are attached to a parent span. In ClickHouse, they're stored as separate rows with `kind: "SPAN_EVENT"` sharing the parent span's `span_id`. The `#mergeRecordsIntoSpanDetail` method reassembles them into the span's `events` array at query time.
+2. The timeline only renders events whose `name` starts with `trigger.dev/` - all others are silently filtered out.
+3. The **display name** comes from `properties.event` (not the span event name), mapped through `getFriendlyNameForEvent()`.
+4. Events are shown on the **span they belong to** - events on one span don't appear in another span's timeline.
+
+## ClickHouse Storage Constraint
+
+When events are written to ClickHouse, `spanEventsToTaskEventV1Input()` filters out events whose `start_time` is not greater than the parent span's `startTime`. Events at or before the span start are silently dropped. This means span events must have timestamps strictly after the span's own `startTimeUnixNano`.
+
+## Timeline Rendering (SpanTimeline component)
+
+The `SpanTimeline` component in `app/components/run/RunTimeline.tsx` renders:
+
+1. **Events** (thin 1px line with hollow dots) - all events from `createTimelineSpanEventsFromSpanEvents()`
+2. **"Started"** marker (thick cap) - at the span's `startTime`
+3. **Duration bar** (thick 7px line) - from "Started" to "Finished"
+4. **"Finished"** marker (thick cap) - at `startTime + duration`
+
+The thin line before "Started" only appears when there are events with timestamps between the span start and the first child span. For the Attempt span this works well (Dequeued → Pod scheduled → Launched → etc. all happen before execution starts). Events all get `lineVariant: "light"` (thin) while the execution bar gets `variant: "normal"` (thick).
+
+## Trace View Sort Order
+
+Sibling spans (same parent) are sorted by `start_time ASC` from the ClickHouse query. The `createTreeFromFlatItems` function preserves this order. Event timestamps don't affect sort order - only the span's own `start_time`.
+
+## Event Structure
+
+```typescript
+// OTel span event format
+{
+  name: "trigger.dev/run",        // Must start with "trigger.dev/" to render
+  timeUnixNano: "1711200000000000000",
+  attributes: [
+    { key: "event", value: { stringValue: "dequeue" } },  // The actual event type
+    { key: "duration", value: { intValue: 150 } },         // Optional: duration in ms
+  ]
+}
+```
+
+## Admin-Only Events
+
+`getAdminOnlyForEvent()` controls visibility. Events default to **admin-only** (`true`).
+
+| Event | Admin-only | Friendly name |
+|-------|-----------|---------------|
+| `dequeue` | No | Dequeued |
+| `fork` | No | Launched |
+| `import` | No (if no fork event) | Importing task file |
+| `create_attempt` | Yes | Attempt created |
+| `lazy_payload` | Yes | Lazy attempt initialized |
+| `pod_scheduled` | Yes | Pod scheduled |
+| (default) | Yes | (raw event name) |
+
+## Adding New Timeline Events
+
+1. Add OTLP span event with `name: "trigger.dev/<scope>"` and `properties.event: "<type>"`
+2. Event timestamp must be strictly after the parent span's `startTimeUnixNano` (ClickHouse drops earlier events)
+3. Add friendly name in `getFriendlyNameForEvent()` in `app/utils/timelineSpanEvents.ts`
+4. Set admin visibility in `getAdminOnlyForEvent()`
+5. Optionally add help text in `getHelpTextForEvent()`
+
+## Key Files
+
+- `app/utils/timelineSpanEvents.ts` - filtering, naming, admin logic
+- `app/components/run/RunTimeline.tsx` - `SpanTimeline` component (thin line + thick bar rendering)
+- `app/presenters/v3/SpanPresenter.server.ts` - loads span data including events
+- `app/v3/eventRepository/clickhouseEventRepository.server.ts` - `spanEventsToTaskEventV1Input()` (storage filter), `#mergeRecordsIntoSpanDetail` (reassembly)

apps/supervisor/src/env.ts

@@ -83,6 +83,7 @@ const Env = z
     COMPUTE_GATEWAY_AUTH_TOKEN: z.string().optional(),
     COMPUTE_GATEWAY_TIMEOUT_MS: z.coerce.number().int().default(30_000),
     COMPUTE_SNAPSHOTS_ENABLED: BoolEnv.default(false),
+    COMPUTE_TRACE_SPANS_ENABLED: BoolEnv.default(true),
     COMPUTE_SNAPSHOT_DELAY_MS: z.coerce.number().int().min(0).max(60_000).default(5_000),
 
     // Kubernetes settings

apps/supervisor/src/index.ts

@@ -236,6 +236,11 @@ class ManagedSupervisor {
               runFriendlyId: message.run.friendlyId,
               snapshotFriendlyId: message.snapshot.friendlyId,
               machine: message.run.machine,
+              traceContext: message.run.traceContext,
+              envId: message.environment.id,
+              orgId: message.organization.id,
+              projectId: message.project.id,
+              dequeuedAt: message.dequeuedAt,
             });
 
             if (didRestore) {
@@ -288,6 +293,24 @@ class ManagedSupervisor {
         return;
       }
 
+      if (env.COMPUTE_TRACE_SPANS_ENABLED) {
+        const traceparent =
+          message.run.traceContext &&
+          "traceparent" in message.run.traceContext &&
+          typeof message.run.traceContext.traceparent === "string"
+            ? message.run.traceContext.traceparent
+            : undefined;
+
+        if (traceparent) {
+          this.workloadServer.registerRunTraceContext(message.run.friendlyId, {
+            traceparent,
+            envId: message.environment.id,
+            orgId: message.organization.id,
+            projectId: message.project.id,
+          });
+        }
+      }
+
       try {
         if (!message.deployment.friendlyId) {
           // mostly a type guard, deployments always exists for deployed environments
@@ -315,6 +338,7 @@ class ManagedSupervisor {
           snapshotId: message.snapshot.id,
           snapshotFriendlyId: message.snapshot.friendlyId,
           placementTags: message.placementTags,
+          traceContext: message.run.traceContext,
         });
 
         // Disabled for now

apps/supervisor/src/otlpTrace.test.ts

@@ -0,0 +1,114 @@
+import { describe, it, expect } from "vitest";
+import { buildOtlpTracePayload } from "./otlpTrace.js";
+
+describe("buildOtlpTracePayload", () => {
+  it("builds valid OTLP JSON with timing attributes", () => {
+    const payload = buildOtlpTracePayload({
+      traceId: "abcd1234abcd1234abcd1234abcd1234",
+      parentSpanId: "1234567890abcdef",
+      spanName: "compute.provision",
+      startTimeMs: 1000,
+      endTimeMs: 1250,
+      resourceAttributes: {
+        "ctx.environment.id": "env_123",
+        "ctx.organization.id": "org_456",
+        "ctx.project.id": "proj_789",
+        "ctx.run.id": "run_abc",
+      },
+      spanAttributes: {
+        "compute.total_ms": 250,
+        "compute.gateway.schedule_ms": 1,
+        "compute.cache.image_cached": true,
+      },
+    });
+
+    expect(payload.resourceSpans).toHaveLength(1);
+
+    const resourceSpan = payload.resourceSpans[0]!;
+
+    // $trigger=true so the webapp accepts it
+    const triggerAttr = resourceSpan.resource.attributes.find((a) => a.key === "$trigger");
+    expect(triggerAttr).toEqual({ key: "$trigger", value: { boolValue: true } });
+
+    // Resource attributes
+    const envAttr = resourceSpan.resource.attributes.find(
+      (a) => a.key === "ctx.environment.id"
+    );
+    expect(envAttr).toEqual({
+      key: "ctx.environment.id",
+      value: { stringValue: "env_123" },
+    });
+
+    // Span basics
+    const span = resourceSpan.scopeSpans[0]!.spans[0]!;
+    expect(span.name).toBe("compute.provision");
+    expect(span.traceId).toBe("abcd1234abcd1234abcd1234abcd1234");
+    expect(span.parentSpanId).toBe("1234567890abcdef");
+
+    // Integer attribute
+    const totalMs = span.attributes.find((a) => a.key === "compute.total_ms");
+    expect(totalMs).toEqual({ key: "compute.total_ms", value: { intValue: 250 } });
+
+    // Boolean attribute
+    const cached = span.attributes.find((a) => a.key === "compute.cache.image_cached");
+    expect(cached).toEqual({ key: "compute.cache.image_cached", value: { boolValue: true } });
+  });
+
+  it("generates a valid 16-char hex span ID", () => {
+    const payload = buildOtlpTracePayload({
+      traceId: "abcd1234abcd1234abcd1234abcd1234",
+      spanName: "test",
+      startTimeMs: 1000,
+      endTimeMs: 1001,
+      resourceAttributes: {},
+      spanAttributes: {},
+    });
+
+    const span = payload.resourceSpans[0]!.scopeSpans[0]!.spans[0]!;
+    expect(span.spanId).toMatch(/^[0-9a-f]{16}$/);
+  });
+
+  it("converts timestamps to nanoseconds", () => {
+    const payload = buildOtlpTracePayload({
+      traceId: "abcd1234abcd1234abcd1234abcd1234",
+      spanName: "test",
+      startTimeMs: 1000,
+      endTimeMs: 1250,
+      resourceAttributes: {},
+      spanAttributes: {},
+    });
+
+    const span = payload.resourceSpans[0]!.scopeSpans[0]!.spans[0]!;
+    expect(span.startTimeUnixNano).toBe("1000000000");
+    expect(span.endTimeUnixNano).toBe("1250000000");
+  });
+
+  it("omits parentSpanId when not provided", () => {
+    const payload = buildOtlpTracePayload({
+      traceId: "abcd1234abcd1234abcd1234abcd1234",
+      spanName: "test",
+      startTimeMs: 1000,
+      endTimeMs: 1001,
+      resourceAttributes: {},
+      spanAttributes: {},
+    });
+
+    const span = payload.resourceSpans[0]!.scopeSpans[0]!.spans[0]!;
+    expect(span.parentSpanId).toBeUndefined();
+  });
+
+  it("handles double values for non-integer numbers", () => {
+    const payload = buildOtlpTracePayload({
+      traceId: "abcd1234abcd1234abcd1234abcd1234",
+      spanName: "test",
+      startTimeMs: 1000,
+      endTimeMs: 1001,
+      resourceAttributes: {},
+      spanAttributes: { "compute.cpu": 0.25 },
+    });
+
+    const span = payload.resourceSpans[0]!.scopeSpans[0]!.spans[0]!;
+    const cpu = span.attributes.find((a) => a.key === "compute.cpu");
+    expect(cpu).toEqual({ key: "compute.cpu", value: { doubleValue: 0.25 } });
+  });
+});

apps/supervisor/src/otlpTrace.ts

@@ -0,0 +1,83 @@
+import { randomBytes } from "crypto";
+import { SimpleStructuredLogger } from "@trigger.dev/core/v3/utils/structuredLogger";
+
+const logger = new SimpleStructuredLogger("otlp-trace");
+
+export interface OtlpTraceOptions {
+  traceId: string;
+  parentSpanId?: string;
+  spanName: string;
+  startTimeMs: number;
+  endTimeMs: number;
+  resourceAttributes: Record<string, string | number | boolean>;
+  spanAttributes: Record<string, string | number | boolean>;
+}
+
+/** Build an OTLP JSON ExportTraceServiceRequest payload */
+export function buildOtlpTracePayload(opts: OtlpTraceOptions) {
+  const spanId = randomBytes(8).toString("hex");
+
+  return {
+    resourceSpans: [
+      {
+        resource: {
+          attributes: [
+            { key: "$trigger", value: { boolValue: true } },
+            ...toOtlpAttributes(opts.resourceAttributes),
+          ],
+        },
+        scopeSpans: [
+          {
+            scope: { name: "supervisor.compute" },
+            spans: [
+              {
+                traceId: opts.traceId,
+                spanId,
+                parentSpanId: opts.parentSpanId,
+                name: opts.spanName,
+                kind: 3, // SPAN_KIND_CLIENT
+                startTimeUnixNano: String(opts.startTimeMs * 1_000_000),
+                endTimeUnixNano: String(opts.endTimeMs * 1_000_000),
+                attributes: toOtlpAttributes(opts.spanAttributes),
+                status: { code: 1 }, // STATUS_CODE_OK
+              },
+            ],
+          },
+        ],
+      },
+    ],
+  };
+}
+
+/** Fire-and-forget: send an OTLP trace payload to the collector */
+export function sendOtlpTrace(
+  endpoint: string,
+  payload: ReturnType<typeof buildOtlpTracePayload>
+) {
+  fetch(`${endpoint}/v1/traces`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify(payload),
+    signal: AbortSignal.timeout(5_000),
+  }).catch((err) => {
+    logger.warn("failed to send compute provision span", {
+      error: err instanceof Error ? err.message : String(err),
+    });
+  });
+}
+
+function toOtlpAttributes(
+  attrs: Record<string, string | number | boolean>
+): Array<{ key: string; value: Record<string, unknown> }> {
+  return Object.entries(attrs).map(([key, value]) => ({
+    key,
+    value: toOtlpValue(value),
+  }));
+}
+
+function toOtlpValue(value: string | number | boolean): Record<string, unknown> {
+  if (typeof value === "string") return { stringValue: value };
+  if (typeof value === "boolean") return { boolValue: value };
+  if (Number.isInteger(value)) return { intValue: value };
+  return { doubleValue: value };
+}