v1.2.0: dashboard performance overhaul + security hardening

[dmichael-fastly]

Summary

• Performance: cold and warm dashboard loads drop from seconds to sub-second on large services; sustained concurrent load no longer wedges the backend. Read-path I/O is structurally cut by a per-service DuckDB connection pool, a per-minute time-series rollup bundle, size-capped bin-packing local compaction (daily + weekly tiers), composite admin-page endpoints, and a frontend pre-warm + hover-prefetch pattern that makes navigation feel instant.
• Reliability: multi-worker login loop fixed; DuckDB pool/cron lock conflict resolved; iceberg s3fs proxy hook always registers; rollup current-hour merge restored; usage-log reconcile changed to UPSERT; telemetry backstop middleware added.
• Security: closed a cross-tenant ContextVar leak in the s3fs proxy hook (now via a ThreadPoolExecutor.submit monkeypatch); centralised path-param service-scope checks; removed a secret-in-URL leak on downloads; added strict input validation across the destructive-op surface; tightened CSRF gates.
• Version bumped to 1.2.0 across pyproject.toml, frontend/package.json, frontend/openapi.json, and backend/main.py.

See CHANGELOG.md for the full entry.

Highlights

Performance — structural

• Per-minute time-series rollup bundle and per-day rollup compaction.
• Size-capped bin-packing local compaction (default 256 MB) — preserves DuckDB scan parallelism on multi-month services.
• DuckDB pool tuning knobs (DUCKDB_POOL_CONN_MEMORY_LIMIT, DUCKDB_POOL_CONN_THREADS); view-binding moved outside the pool lock.
• Composite endpoints: POST /api/scoring/dashboard, GET /api/scoring/analytics, GET /api/scoring/config, expanded GET /api/network-health, new POST /api/origin/aggregates. Per-card endpoints stay mounted for back-compat.
• Parquet ingest sort key → (timestamp, ip) (~2× sessions speedup).
• ingested_files.file_date column + (source_name, file_date) index for log-accounting fast path.
• Iceberg buffer-file tombstoning; optimize_table adds union_by_name + retry-on-CAS-conflict.
• Bootstrap stale-while-revalidate dir-stats; views folded into the response.

Frontend

• starlette-compress replaces GZipMiddleware (br / zstd / gzip).
• Keep-alive on Next.js http/undici global agents.
• Pre-warm + lazy-mount for plotly + maplibre-gl + world.geojson; hover-prefetch sidebar links; per-insight skeleton cards.
• Modulepreload for the plotly chunk via a build-time-generated preload manifest.
• Shared useNowMs interval; MapLibre style-data listener replaces a 100 ms setTimeout poll.

Security

• ContextVar propagation patch on ThreadPoolExecutor.submit eliminates the prior endpoint-keyed proxy registry. Documented in MONKEYPATCHES.md §6.
• Centralised session-scope check on every scoped route; signed short-lived bearer on the download redirect.
• Strict validation (length caps, character allowlists, falco static analysis) before any VCL ships on the destructive-op surface.
• Cross-tenant cache-key audit — two missing service_id entries closed.

Documentation

• AGENTS.md — Key Systems entries for the DuckDB connection pool, the hourly Top-N rollup pipeline, and the response telemetry middleware; local-compaction section updated for the bin-packing tiers.
• MONKEYPATCHES.md — documents the new ThreadPoolExecutor.submit patch.

Test plan

• make ci green locally (lint + format + mypy + pytest + vcl-test + verify-deps + typecheck-frontend + test-frontend + osv + secret-scan)
• Cold dashboard load on a large service is sub-second
• Concurrent load (~20 in-flight queries) — backend stays responsive, no wedge
• Login flow with multiple uvicorn workers succeeds on first POST
• Cross-tenant smoke test — verify no leaked state across services after concurrent reads
• Modulepreload manifest is read at request time on a fresh container build
• Manually exercise the new composite endpoints (scoring dashboard / analytics / config, network-health shielding, origin aggregates)

🤖 Generated with Claude Code