v0.6.0: background indexing architecture and worker runtime groundwork

[emremy]

Summary

This PR introduces the v0.6.0 background indexing architecture for ColQL.

The release focuses on lifecycle correctness, worker runtime groundwork, stale-result safety, diagnostics, packaging stability, and large-table rebuild infrastructure.

This is primarily an architecture and runtime-hardening release rather than a pure performance release.

---

Highlights

Background indexing architecture

Added internal background rebuild infrastructure for:

• equality indexes
• sorted indexes

including:

• lifecycle state machine
• generation/epoch tracking
• stale-result discard
• atomic apply/swap semantics
• fallback query correctness
• worker-safe typed/encoded rebuild output

Lifecycle states now include:

• fresh
• dirty
• queued
• rebuilding
• failed

---

Worker runtime groundwork

Added:

• bounded worker pool
• chunk task queue
• real worker_threads runtime integration
• deterministic fake executor for tests
• worker packaging/runtime smoke tests
• ESM/CJS worker resolution validation

Worker rebuild paths currently remain internal/runtime-only.

Public query APIs remain synchronous.

---

Important behavior notes

Background indexing is not presented as a universal speedup.

Current benchmarks show:

• small/medium datasets may still be faster with synchronous rebuilds
• worker overhead can dominate at lower row counts
• the main value of the architecture is rebuild isolation and lifecycle correctness for large dirty-index rebuilds

---

Serialization changes

Clarified/documented:

• serialized indexes are unsupported
• worker jobs/lifecycle state are not serialized
• restored tables contain data only
• indexes must be recreated after restore

---

Diagnostics

Added unstable/internal diagnostics surfaces for:

• lifecycle state
• fallback counts
• stale discard counts
• rebuild mode
• background rebuild status

explain() remains non-executing.

---

Benchmarks

Added benchmark coverage for:

• background equality rebuild
• background sorted rebuild
• worker runtime overhead
• stale-result discard
• rebuild apply timing
• sync vs mock-background vs real-worker rebuild paths

CodSpeed coverage was expanded accordingly.

---

Packaging/build cleanup

• removed hashed chunk-* dist artifacts
• stabilized build output layout
• verified worker artifacts in npm tarball
• validated ESM/CJS worker startup from built package output

---

Validation

Passed:

• npm test
• npm run test:types
• npm run build
• npm run test:worker-runtime
• npm run bench:codspeed
• background indexing benchmarks
• worker runtime benchmarks
• npm pack dry-run

---

Known caveats

• query APIs are still synchronous
• automatic background scheduling is not enabled for normal user query paths yet
• worker rebuild paths are internal/runtime infrastructure in this release
• the existing non-blocking tsup CJS import.meta warning remains documented

[codspeed-hq]

Merging this PR will not alter performance

✅ 20 untouched benchmarks
🆕 5 new benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
🆕	background/equality/dictionary-code-encode-merge/10k	N/A	7.5 ms	N/A
🆕	background/equality/numeric-encode-merge/10k	N/A	21 ms	N/A
🆕	background/sorted/numeric-encode-merge/10k	N/A	50.4 ms	N/A
🆕	index/requery/after-sync-fallback-rebuild/10k	N/A	401.5 µs	N/A
🆕	index/sync-fallback-rebuild/after-indexed-column-mutation/10k	N/A	18.3 ms	N/A

---

_{Comparing v0.6.0 (c10dc55) with main (6f5b911)}