refactor: logstore & log api

[alexluong]

LogStore Refactor & Multi-Backend Implementation

Summary

Refactors the LogStore interface from Event-focused to DeliveryEvent-focused design, simplifying the API and enabling ClickHouse support.

Interface Changes

Before

ListEvent(ctx, ListEventRequest) (ListEventResponse, error)
RetrieveEvent(ctx, tenantID, eventID string) (*Event, error)
RetrieveEventByDestination(ctx, tenantID, destinationID, eventID string) (*Event, error)
ListDelivery(ctx, ListDeliveryRequest) ([]*Delivery, error)
InsertManyDeliveryEvent(ctx, []*DeliveryEvent) error

After

ListDeliveryEvent(ctx, ListDeliveryEventRequest) (ListDeliveryEventResponse, error)
RetrieveEvent(ctx, RetrieveEventRequest) (*Event, error)
InsertManyDeliveryEvent(ctx, []*DeliveryEvent) error

Change	Rationale
ListEvent → ListDeliveryEvent	Primary entity is now DeliveryEvent, not Event
Removed ListDelivery	Deliveries are now returned with ListDeliveryEvent
Removed RetrieveEventByDestination	Consolidated into RetrieveEvent with RetrieveEventRequest
Re-enabled ClickHouse config	CH driver now fully implemented

ClickHouse Implementation

Schema

CREATE TABLE IF NOT EXISTS event_log (
    -- Event fields
    event_id String,
    tenant_id String,
    destination_id String,
    topic String,
    eligible_for_retry Bool,
    event_time DateTime64(3),
    metadata String,      -- JSON serialized
    data String,          -- JSON serialized

    -- Delivery fields
    delivery_id String,
    delivery_event_id String,
    status String,        -- 'success', 'failed'
    delivery_time DateTime64(3),
    code String,
    response_data String, -- JSON serialized

    INDEX idx_event_id event_id TYPE bloom_filter GRANULARITY 4,
    INDEX idx_topic topic TYPE bloom_filter GRANULARITY 4,
    INDEX idx_status status TYPE set(100) GRANULARITY 4
) ENGINE = ReplacingMergeTree
PARTITION BY toYYYYMMDD(delivery_time)
ORDER BY (tenant_id, destination_id, delivery_time, event_id, delivery_id);

Decision	Rationale
Single denormalized table	Each row = one delivery attempt with embedded event data. No JOINs needed
DateTime64(3)	Millisecond precision for cursor pagination
Daily partitions on delivery_time	Efficient pruning for time-range queries
ORDER BY	Optimized for typical query: tenant → destination → time range
ReplacingMergeTree	Handles duplicate inserts gracefully

Operations

InsertManyDeliveryEvent

INSERT INTO event_log (
  event_id, tenant_id, destination_id, topic, eligible_for_retry, event_time, metadata, data,
  delivery_id, delivery_event_id, status, delivery_time, code, response_data
)

Uses batch insert for efficient bulk writes.

ListDeliveryEvent

SELECT
    event_id, tenant_id, destination_id, topic, eligible_for_retry, event_time, metadata, data,
    delivery_id, delivery_event_id, status, delivery_time, code, response_data
FROM event_log
WHERE tenant_id = ?
    [AND destination_id IN (?)]
    [AND event_id = ?]
    [AND status = ?]
    [AND topic IN (?)]
    [AND event_time >= ? AND event_time <= ?]
    [AND delivery_time >= ? AND delivery_time <= ?]
    [AND cursor_condition]
ORDER BY delivery_time DESC, delivery_id DESC
LIMIT N+1

• Supports sorting by delivery_time (default) or event_time
• Cursor pagination with multi-column comparison for tie-breaking
• LIMIT N+1 to detect if more pages exist

RetrieveEvent

SELECT event_id, tenant_id, destination_id, topic, eligible_for_retry, event_time, metadata, data
FROM event_log
WHERE tenant_id = ? AND event_id = ?
    [AND destination_id = ?]
LIMIT 1

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
outpost-docs	Ready	Preview	Comment	Dec 9, 2025 10:29am
outpost-website	Ready	Preview	Comment	Dec 9, 2025 10:29am