RFC: Add flexible event querying primitives

[evansenter]

Summary

Ad-hoc session analysis (like calculating compaction rates) currently requires writing Python scripts against raw JSONL. The API lacks flexible querying primitives that would let LLMs perform arbitrary analysis via MCP tools.

Problem / Motivation

When analyzing compaction frequency, we couldn't use MCP tools. We had to:

1. Write Python to iterate raw JSONL files
2. Track state across entries (last timestamp, event counts between markers)
3. Compute custom aggregations (turns per compaction, rate per hour)

This pattern repeats for any analysis not pre-built into the API. The friction prevents exploratory analysis.

Context

• Discovered during: Compaction rate analysis (RFC RFC: Track compaction events with inferred timestamps #55)
• Relevant files: queries.py (current query patterns), server.py (MCP tools)
• Related issues/PRs: RFC: Track compaction events with inferred timestamps #55 (compaction tracking - specific case of this general problem)

Proposed Solution

1. Entry Type Filtering

Currently get_session_events() filters by tool. Add entry_type filter:

def get_session_events(..., entry_type: str | None = None):
    # entry_type in ('user', 'assistant', 'tool_use', 'tool_result', 'summary')

2. Event Context Windows

Get N events before/after a specific event (useful for understanding what happened around compaction, errors, etc.):

def get_event_context(
    event_uuid: str,
    before: int = 5,
    after: int = 5
) -> dict:
    """Return events surrounding a specific event."""
    return {
        "target": Event,
        "before": [Event, ...],
        "after": [Event, ...]
    }

3. Custom Grouping/Bucketing

Allow grouping by arbitrary time buckets:

def get_event_counts(
    days: int = 7,
    bucket: str = "hour",  # "minute", "hour", "day", "session"
    entry_type: str | None = None,
    tool: str | None = None
) -> list[dict]:
    """Count events by time bucket with optional filters."""
    return [{"bucket": "2026-01-06T14:00", "count": 42}, ...]

4. Marker-Based Aggregation

Count events between markers (e.g., user turns between compactions):

def get_events_between_markers(
    marker_type: str = "summary",  # entry_type that defines boundaries
    count_type: str = "user",      # entry_type to count
    days: int = 7
) -> dict:
    """Count events of one type between occurrences of another type."""
    return {
        "marker_count": 50,
        "counted_events": 3500,
        "avg_per_marker": 70.0,
        "median_per_marker": 65.0,
        "distribution": [{"marker_index": 1, "count": 42}, ...]
    }

5. Raw Event Iteration (Optional)

For truly ad-hoc analysis, expose raw event stream with pagination:

def iterate_events(
    session_id: str | None = None,
    entry_types: list[str] | None = None,
    cursor: str | None = None,
    limit: int = 100
) -> dict:
    """Low-level event iteration for custom analysis."""
    return {
        "events": [...],
        "next_cursor": str | None
    }

Assumptions

Assumption	Confidence	Impact if Wrong
LLMs can compose primitives into complex queries	High	May need higher-level convenience functions
Entry types are stable set	Medium	May need to handle unknown types gracefully
Time bucketing covers common use cases	High	Could add custom bucket expressions later

Open Questions

1. Should iterate_events return full Event objects or just essential fields (to reduce payload)?
2. Should marker-based aggregation support multiple marker types (e.g., "between compaction OR session start")?
3. Performance concern: context windows require efficient index on uuid - worth adding?

Actionable Requirements

#	Requirement	Owner	Blocked By
1	Add entry_type filter to get_session_events()	Claude	#55
2	Implement get_event_context() query + MCP tool	Claude	#55
3	Implement get_event_counts() with bucketing	Claude	#55
4	Implement get_events_between_markers()	Claude	#55
5	(Optional) Implement iterate_events()	Claude	#55
6	Update guide.md with new primitives	Claude	1-4

Test Requirements

• Unit: Each new query function with various filter combinations
• Integration: Compose primitives to answer "user turns per compaction" via MCP only
• Edge cases:
  • Empty result sets
  • Single-event sessions
  • Markers at session boundaries

Implementation Checklist

• Add entry_type filter (low-hanging fruit)
• Add get_event_context()
• Add get_event_counts() with bucketing
• Add get_events_between_markers()
• Self-play test: replicate compaction analysis using only MCP tools
• Update guide.md

[evansenter]

Correction: type: \"user\" includes tool results

During analysis, discovered that JSONL entries with "type": "user" include both:

1. Actual human messages (no toolUseResult field)
2. Tool results returned to Claude (has toolUseResult field)

This inflated our "user turns per compaction" estimate from ~70 to the correct ~9 actual user turns.

Implication for RFC

The entry_type filter proposed in this RFC should distinguish these:

# Current entry_types in our schema
entry_type in ('user', 'assistant', 'tool_use', 'tool_result', 'summary')

But raw JSONL uses:

• type: "user" for both human messages AND tool results
• type: "assistant" for Claude responses (which contain tool_use blocks)

Proposed: Add a higher-fidelity filter or computed field:

def get_session_events(..., 
    entry_type: str | None = None,      # Raw JSONL type
    is_human_message: bool | None = None # Computed: type=user AND no toolUseResult
):

Or normalize during ingestion:

• type: "user" + no toolUseResult → entry_type: "human_message"
• type: "user" + has toolUseResult → entry_type: "tool_result"

This affects Requirement #1 (entry_type filtering) and should be addressed before implementation.

[evansenter]

Additional Data: File Re-read Patterns

Observed Data

Sessions with 27-36 compactions show "anchor files" read 20-50 times:

Session	File	Times Read
f583f206	gemicro-developer/src/lib.rs	49x
e83981b9	gemicro-critique/src/lib.rs	36x
63948eb2	session_analytics/storage.py	34x
90990fe3	event_bus/server.py	34x

Insight: ~1-2 re-reads per compaction for critical files. These are "context anchors" that Claude needs to re-establish after compaction.

---

Technique Used

-- Files read 3+ times in a session (re-read candidates)
SELECT session_id, file_path, COUNT(*) as read_count
FROM events
WHERE tool_name = 'Read' AND file_path IS NOT NULL
GROUP BY session_id, file_path
HAVING read_count >= 3
ORDER BY read_count DESC

Limitation: Can't currently correlate with compaction timing (need #55 first). Once compaction events have timestamps, can compute:

• Files read before vs after each compaction
• Re-read rate per compaction
• "Lost context" proxy: files re-read within N minutes of compaction

---

Proposed APIs

1. File read frequency with re-read detection

def get_file_read_stats(
    days: int = 7,
    session_id: str | None = None,
    min_reads: int = 1
) -> dict:
    """File read patterns including re-reads."""
    return {
        "files": [
            {
                "file_path": str,
                "total_reads": int,
                "sessions": int,  # How many sessions read this file
                "avg_reads_per_session": float
            }
        ],
        "anchor_files": [...]  # Files with reads > 2 * session_count (re-read heavy)
    }

2. Post-compaction re-reads (requires #55)

def get_post_compaction_reads(
    days: int = 7,
    window_minutes: int = 10  # How soon after compaction
) -> dict:
    """Files re-read shortly after compaction (context recovery signal)."""
    return {
        "compactions_analyzed": int,
        "avg_files_reread": float,
        "common_rereads": [
            {"file_path": str, "reread_count": int, "pct_of_compactions": float}
        ]
    }

3. Session file timeline (enables custom analysis)

def get_file_timeline(
    session_id: str,
    include_compactions: bool = True
) -> dict:
    """Chronological file access with compaction markers."""
    return {
        "events": [
            {"timestamp": str, "type": "read"|"edit"|"write"|"compaction", "file_path": str | None}
        ]
    }

This gives the LLM raw temporal data to identify patterns like "re-read storm after compaction" or "file X always read after file Y".

[evansenter]

Now unblocked - RFC #55 completed and closed.