refactor: dashboard routes, SQLite-first architecture, auto-update

.github/workflows/ci.yml

@@ -28,6 +28,9 @@ jobs:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
       - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
       - run: bun install
+      - name: Dashboard typecheck
+        working-directory: apps/local-dashboard
+        run: bunx tsc --noEmit
       - run: bun run build:dashboard
 
   test:
@@ -38,4 +41,4 @@ jobs:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
       - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
       - run: bun install
-      - run: bun test --coverage
+      - run: bun test tests/ packages/telemetry-contract/ --coverage

AGENTS.md

@@ -36,6 +36,7 @@ selftune/
 │   ├── dashboard.ts         # Dashboard command entry point
 │   ├── dashboard-server.ts  # Bun.serve API + SPA server
 │   ├── dashboard-contract.ts # Shared dashboard payload types
+│   ├── export.ts             # SQLite → JSONL export command
 │   ├── types.ts             # Shared interfaces
 │   ├── constants.ts         # Log paths, known tools, skip prefixes
 │   ├── utils/               # Shared utilities
@@ -58,8 +59,14 @@ selftune/
 │   │   ├── codex-rollout.ts # Batch Codex ingestor (experimental)
 │   │   ├── opencode-ingest.ts # OpenCode SQLite/JSON adapter (experimental)
 │   │   └── openclaw-ingest.ts # OpenClaw session importer (experimental)
+│   ├── routes/              # HTTP route handlers (extracted from dashboard-server)
 │   ├── repair/              # Rebuild repaired skill-usage overlays
-│   ├── localdb/             # SQLite materialization + overview/report queries
+│   ├── localdb/             # SQLite schema, direct-write, queries, materialization
+│   │   ├── db.ts            # Database lifecycle + singleton
+│   │   ├── direct-write.ts  # Fail-open insert functions for all tables
+│   │   ├── queries.ts       # Read queries for dashboard + CLI consumers
+│   │   ├── schema.ts        # Table DDL + indexes
+│   │   └── materialize.ts   # JSONL → SQLite rebuild (startup/backfill only)
 │   ├── cron/                # Optional OpenClaw-specific scheduler adapter
 │   ├── memory/              # Evolution memory persistence
 │   ├── eval/                # False negative detection, eval set generation
@@ -138,6 +145,8 @@ See ARCHITECTURE.md for domain map, module layering, and dependency rules.
 | Skill Definition | skill/SKILL.md | Current |
 | Design Docs | docs/design-docs/index.md | Current |
 | Core Beliefs | docs/design-docs/core-beliefs.md | Current |
+| Live Dashboard SSE | docs/design-docs/live-dashboard-sse.md | Current |
+| SQLite-First Migration | docs/design-docs/sqlite-first-migration.md | Current |
 | Product Specs | docs/product-specs/index.md | Current |
 | Active Plans | docs/exec-plans/active/ | Current |
 | Completed Plans | docs/exec-plans/completed/ | Current |
@@ -161,7 +170,7 @@ This prevents stale docs and broken contracts.
 |------------------|---------------|
 | CLI commands in `index.ts` (add/rename/remove) | `skill/SKILL.md` Quick Reference + Workflow Routing table, `README.md` Commands table, `AGENTS.md` project tree |
 | CLI flags on any command | The command's `skill/Workflows/*.md` doc (flags table + examples) |
-| JSONL log schema or new log file | `constants.ts`, `types.ts`, `skill/references/logs.md`, `localdb/schema.ts` + `materialize.ts`, `ARCHITECTURE.md` data architecture |
+| JSONL log schema or new log file | `constants.ts`, `types.ts`, `skill/references/logs.md`, `localdb/schema.ts` + `materialize.ts` + `direct-write.ts` + `queries.ts`, `ARCHITECTURE.md` data architecture |
 | Dashboard contract (`dashboard-contract.ts`) | `apps/local-dashboard/src/types.ts`, dashboard components that consume the changed fields |
 | Hook behavior (`hooks/*.ts`) | `skill/Workflows/Initialize.md` hook table, `skill/settings_snippet.json` |
 | Orchestrate behavior | `skill/Workflows/Orchestrate.md`, `ARCHITECTURE.md` operating modes |

ARCHITECTURE.md

@@ -28,30 +28,34 @@ flowchart LR
   Agent -. hook hints .-> Hooks[Claude hooks]
 
   Sources --> Sync[selftune sync]
-  Hooks --> Logs[Append-only JSONL logs]
+  Hooks --> SQLite[(SQLite — primary store)]
+  Hooks --> Logs[Append-only JSONL audit trail]
+  Sync --> SQLite
   Sync --> Logs
   Sync --> Repaired[Repaired skill-usage overlay]
 
-  Logs --> Eval[Eval + grading]
+  SQLite --> Eval[Eval + grading]
   Repaired --> Eval
   Eval --> Orchestrate[selftune orchestrate]
   Orchestrate --> Evolution[Evolve / deploy / audit]
   Orchestrate --> Monitoring[Watch / rollback]
 
-  Logs --> LocalDB[SQLite materialization]
-  Repaired --> LocalDB
-  Evolution --> LocalDB
-  Monitoring --> LocalDB
+  Evolution --> SQLite
+  Monitoring --> SQLite
 
-  LocalDB --> API[dashboard-server v2 API]
-  API --> SPA[apps/local-dashboard]
+  Logs -. startup backfill .-> Materializer[Materializer — one-time rebuild]
+  Materializer --> SQLite
+
+  SQLite --> API[dashboard-server v2 API]
+  SQLite -. WAL watch .-> API
+  API -. SSE push .-> SPA[apps/local-dashboard]
   API --> CLI[status / last / badge]
 ```
 
 ## Operating Rules
 
 - **Source-truth first.** Transcripts, rollouts, and session stores are authoritative. Hooks are low-latency hints.
-- **Shared local evidence.** Downstream modules communicate through shared JSONL logs, repaired overlays, audit logs, and SQLite materialization.
+- **Shared local evidence.** Downstream modules communicate through SQLite (primary operational store), append-only JSONL audit trails, and repaired overlays.
 - **Autonomy with safeguards.** Low-risk description evolution can deploy automatically, but validation, watch, and rollback remain mandatory.
 - **Local-first product surfaces.** `status`, `last`, and the dashboard read from local evidence, not external services.
 - **Generic scheduling first.** `selftune cron setup` is the main automation path (auto-detects platform). `selftune schedule` is a backward-compatible alias.
@@ -72,7 +76,7 @@ flowchart LR
 | Orchestrator | `cli/selftune/orchestrate.ts` | Autonomy-first sync -> candidate selection -> evolve -> watch loop | B |
 | Monitoring | `cli/selftune/monitoring/` | Post-deploy regression detection and rollback triggers | B |
 | Local DB | `cli/selftune/localdb/` | SQLite materialization and payload-oriented queries | B |
-| Dashboard | `cli/selftune/dashboard.ts`, `cli/selftune/dashboard-server.ts`, `apps/local-dashboard/` | Local SPA shell, v2 API, overview/report/status UI | B |
+| Dashboard | `cli/selftune/dashboard.ts`, `cli/selftune/dashboard-server.ts`, `apps/local-dashboard/` | Local SPA shell, v2 API with SSE live updates, overview/report/status UI | B |
 | Observability CLI | `cli/selftune/status.ts`, `cli/selftune/last.ts`, `cli/selftune/badge/` | Fast local readouts of health, recent activity, and badge state | B |
 | Contribute | `cli/selftune/contribute/` | Opt-in anonymized export for community signal pooling | C |
 | Skill | `skill/` | Agent-facing routing table, workflows, and references | B |
@@ -160,33 +164,40 @@ don't need agent intelligence or user interaction.
 
 ## Data Architecture
 
-All data flows through append-only JSONL files. SQLite is a read-only
-materialized view used only by the dashboard.
+SQLite is the operational database for all reads. Hooks and sync write
+directly to SQLite via `localdb/direct-write.ts`. JSONL files are retained
+as an append-only audit trail and can be used to rebuild SQLite on demand.
 
 ```text
-Source of Truth: JSONL files (~/.claude/*.jsonl)
+Primary Store: SQLite (~/.selftune/selftune.db)
+├── Hooks write directly via localdb/direct-write.ts (primary write path)
+├── Sync writes directly via localdb/direct-write.ts
+├── All reads (orchestrate, evolve, grade, status, dashboard) query SQLite
+└── WAL-mode watch powers SSE live updates
+
+Audit Trail: JSONL files (~/.claude/*.jsonl)
 ├── session_telemetry_log.jsonl    Session telemetry records
-├── skill_usage_log.jsonl          Skill trigger/miss records
+├── skill_usage_log.jsonl          Skill trigger/miss records (deprecated; consolidated into skill_invocations SQLite table)
 ├── all_queries_log.jsonl          User prompt log
 ├── evolution_audit_log.jsonl      Evolution decisions + evidence
 ├── orchestrate_runs.jsonl         Orchestrate run reports
 └── canonical_telemetry_log.jsonl  Normalized cross-platform records
 
-Core Loop: reads JSONL directly
-├── orchestrate.ts  → readJsonl(TELEMETRY_LOG)
-├── evolve.ts       → readJsonl(EVOLUTION_AUDIT_LOG)
-├── grade.ts        → readJsonl(TELEMETRY_LOG)
-└── status.ts       → readJsonl(TELEMETRY_LOG + SKILL_LOG + QUERY_LOG)
+Core Loop: reads SQLite
+├── orchestrate.ts  → db.query("SELECT ... FROM sessions ...")
+├── evolve.ts       → db.query("SELECT ... FROM evolution_audit ...")
+├── grade.ts        → db.query("SELECT ... FROM sessions ...")
+└── status.ts       → db.query("SELECT ... FROM sessions, skill_usage, queries ...")
 
-Materialized View: SQLite (~/.selftune/selftune.db)
-├── materialize.ts reads ALL JSONL → inserts into SQLite tables
-└── dashboard-server.ts reads SQLite for fast API queries
+Rebuild Paths:
+├── materialize.ts  — runs once on startup for historical JSONL backfill
+└── selftune export — generates JSONL from SQLite on demand
 ```
 
-The core loop (orchestrate, evolve, grade, status) reads JSONL directly.
-SQLite is only used by the dashboard for fast queries over large datasets.
-This design keeps the core loop simple (no database dependency) while giving
-the dashboard fast aggregation.
+Hooks and sync write to both SQLite (primary) and JSONL (audit trail) in
+parallel. All reads go through SQLite. The materializer runs once on startup
+to backfill any historical JSONL data not yet in the database. `selftune export`
+can regenerate JSONL from SQLite when needed for portability or debugging.
 
 ## Repository Shape
 
@@ -198,14 +209,15 @@ cli/selftune/
 ├── orchestrate.ts        Main autonomous loop
 ├── schedule.ts           Generic scheduler install/preview
 ├── dashboard.ts          Dashboard command entry point
-├── dashboard-server.ts   Bun.serve API + SPA shell
+├── dashboard-server.ts   Bun.serve API + SPA shell + SSE live updates
 ├── dashboard-contract.ts Shared overview/report/run-report payload types
 ├── constants.ts          Paths and log file constants
 ├── types.ts              Shared TypeScript interfaces
 ├── utils/                JSONL, transcript, logging, schema, agent-call helpers
 ├── hooks/                Claude-specific hints, activation, enforcement
 ├── ingestors/            Claude/Codex/OpenCode/OpenClaw adapters
 ├── repair/               Rebuild repaired skill-usage overlay
+├── routes/               HTTP route handlers (extracted from dashboard-server)
 ├── eval/                 False-negative detection and eval generation
 ├── grading/              Session grading
 ├── evolution/            Propose / validate / deploy / rollback
@@ -219,7 +231,7 @@ cli/selftune/
 apps/local-dashboard/
 ├── src/pages/            Overview, per-skill report, and system status routes
 ├── src/components/       Dashboard components
-├── src/hooks/            Data-fetch hooks against the v2 API
+├── src/hooks/            Data-fetch hooks + SSE live update hook
 └── src/types.ts          Frontend types from dashboard-contract.ts
 
 skill/
@@ -348,14 +360,14 @@ marked consumed so they don't affect subsequent runs.
 | Artifact | Writer | Reader |
 |----------|--------|--------|
 | `~/.claude/session_telemetry_log.jsonl` | Hooks, ingestors, sync | Eval, grading, status, localdb |
-| `~/.claude/skill_usage_log.jsonl` | Hooks | Eval, repair, status |
-| `~/.claude/skill_usage_repaired.jsonl` | Sync / repair | Eval, status, localdb |
+| `~/.claude/skill_usage_log.jsonl` | Hooks | Eval, repair, status (deprecated — consolidated into `skill_invocations` table in SQLite) |
+| `~/.claude/skill_usage_repaired.jsonl` | Sync / repair | Eval, status, localdb (deprecated — consolidated into `skill_invocations` table in SQLite) |
 | `~/.claude/all_queries_log.jsonl` | Hooks, ingestors, sync | Eval, status, localdb |
 | `~/.claude/evolution_audit_log.jsonl` | Evolution | Monitoring, status, localdb |
 | `~/.claude/orchestrate_runs.jsonl` | Orchestrator | LocalDB, dashboard |
 | `~/.claude/improvement_signals.jsonl` | Hooks (prompt-log) | session-stop hook, orchestrator |
 | `~/.claude/.orchestrate.lock` | Orchestrator | session-stop hook (staleness check) |
-| `~/.selftune/*.sqlite` | LocalDB materializer | Dashboard server |
+| `~/.selftune/*.sqlite` | Hooks (direct-write), sync, materializer (backfill) | All reads: orchestrate, evolve, grade, status, dashboard |
 
 ## The Evaluation Model
 
@@ -387,3 +399,5 @@ marked consumed so they don't affect subsequent runs.
 - [docs/integration-guide.md](docs/integration-guide.md)
 - [docs/design-docs/evolution-pipeline.md](docs/design-docs/evolution-pipeline.md)
 - [docs/design-docs/monitoring-pipeline.md](docs/design-docs/monitoring-pipeline.md)
+- [docs/design-docs/live-dashboard-sse.md](docs/design-docs/live-dashboard-sse.md)
+- [docs/design-docs/sqlite-first-migration.md](docs/design-docs/sqlite-first-migration.md)

CLAUDE.md

@@ -0,0 +1,3 @@
+# Project Configuration
+
+@AGENTS.md

CONTRIBUTING.md

@@ -100,6 +100,57 @@ bun run lint:fix
 
 selftune intentionally has zero runtime dependencies. All functionality uses Bun built-ins. Do not add `dependencies` to `package.json`.
 
+## Local Data Management
+
+selftune's data pipeline: **hooks write directly to SQLite via `localdb/direct-write.ts`**. JSONL serves as an append-only audit trail for debugging and the contribute workflow. The materializer runs once on dashboard startup to backfill historical data. `selftune export` generates JSONL from SQLite on demand. The SQLite DB at `~/.selftune/selftune.db` is the operational database.
+
+### Rebuilding the Dashboard Database
+
+When developing locally (especially after schema changes), the SQLite database can become incompatible. To rebuild:
+
+```bash
+rm ~/.selftune/selftune.db
+selftune sync --force
+```
+
+`--force` ignores per-source markers and rescans all JSONL logs from scratch. The next `selftune dashboard` will serve fresh data.
+
+### Linking Local Source for Testing
+
+The globally installed `selftune` runs from npm, not your working tree. To test local changes end-to-end (hooks, materialization, dashboard):
+
+```bash
+npm link                     # global selftune → your source tree
+# ... test ...
+npm install -g selftune@latest  # revert to published version
+```
+
+While linked, hooks in `~/.claude/settings.json` point through the symlink to your local code — changes take effect immediately.
+
+### Schema Change Checklist
+
+When modifying JSONL log schemas or adding new fields, update all of these to keep the pipeline consistent:
+
+| File | What to update |
+|------|---------------|
+| `cli/selftune/types.ts` | Add/modify the TypeScript interface |
+| `cli/selftune/constants.ts` | Add log path constant if new file |
+| `cli/selftune/localdb/schema.ts` | Add column to SQLite schema |
+| `cli/selftune/localdb/materialize.ts` | Map JSONL field → SQLite column |
+| `cli/selftune/normalization.ts` | Update canonical derivation if applicable |
+| `cli/selftune/dashboard-contract.ts` | Expose field to dashboard API |
+| `apps/local-dashboard/src/` | Consume field in UI components |
+| `skill/references/logs.md` | Document the field for agents |
+
+### Common Data Issues
+
+| Symptom | Fix |
+|---------|-----|
+| Dashboard shows stale data | `selftune sync --force` |
+| SQLite schema mismatch after code change | `rm ~/.selftune/selftune.db && selftune sync --force` (materializer rebuilds from JSONL) |
+| Missing invocations after hook changes | Verify `~/.claude/settings.json` matchers, then `selftune doctor` |
+| Need to backfill from transcripts | `selftune ingest claude --force` |
+
 ## Questions?
 
 Open a [discussion](https://github.com/selftune-dev/selftune/discussions) or file an [issue](https://github.com/selftune-dev/selftune/issues).

Makefile

@@ -1,4 +1,4 @@
-.PHONY: all clean lint test test-fast test-slow check sandbox sandbox-llm sandbox-shell sandbox-openclaw sandbox-openclaw-keep sandbox-openclaw-clean clean-branches
+.PHONY: all clean lint test test-fast test-slow check typecheck-dashboard sandbox sandbox-llm sandbox-shell sandbox-openclaw sandbox-openclaw-keep sandbox-openclaw-clean clean-branches
 
 all: check
 
@@ -51,4 +51,7 @@ clean-branches:
 	@echo "Branch cleanup complete."
 	@git branch | wc -l | xargs -I{} echo "{} branches remaining"
 
-check: lint test sandbox
+typecheck-dashboard:
+	cd apps/local-dashboard && bunx tsc --noEmit
+
+check: lint typecheck-dashboard test sandbox