Track: _bm25/ package boundary — decomposition vs monolith decision

[jaytoone]

Tracking issue for the _bm25/ package layout decision raised in #1.

Context

hang-in decomposed bm25-memory.py into an _bm25/ subpackage in tunaCtx. PRs #3/#4/#5 were drafted against that layout but ported into the upstream monolith in 7f9d5a9 as an interim step.

Decision to make

• Option A (monolith): Keep everything in bm25-memory.py, port all hang-in fixes inline. Simple, no install changes.
• Option B (decomposition): Accept the _bm25/ package layout, update install/plugin machinery, gain test isolation and module boundaries.

Current stance

Monolith path for now (Option A). Decomposition is the right long-term architecture but adds blast radius in the same cycle as functional fixes.

Reopening conditions

• CTX ships a stable v0.4.x with all functional fixes merged
• Test suite (80 + 26 golden) is merged and CI is green
• Then: evaluate _bm25/ as a clean refactor cycle

cc @hang-in

[hang-in]

Acknowledged — the reopening conditions you've outlined match how I'd want to scope it too:

• v0.4.x stable with the functional fixes merged (7f9d5a9 + the eval-tokenizer unification once PR refactor(eval): unify 3 eval-pipeline tokenizers via monolith bm25-memory.tokenize (PR-3 re-author) #7 lands)
• Test suite (80 + 26 golden) merged and CI green
• Then evaluate _bm25/ as a clean refactor cycle, not bundled with anything else

I'll keep the fork's _bm25/ layout warm in the meantime so when the cycle opens we can compare against a working reference (DAG with no cycles, per-module unit tests already written). When you're ready to revisit, happy to open a draft PR with the package extraction as a single reviewable diff against the post-v0.4.x monolith.

No action needed on this issue from my side until those conditions land. Subscribed for updates.