Skip to content

Wire a model-text suggestion source into the suggestion engine #148

Description

@imnasnainaec

Follow-up to #94 — resolving the first model gap (the model text) the way the follow-up comment points to: extend the suggestion engine (#131), not bake a mode picker into project creation.

Additionally driven by a hard requirement: we want to fully support importing PT9 interlinearizations, and PT9's core glossing workflow (Options 1 & 3) uses a major-language project as the model text. So a major-language Scripture project must be a first-class model source here — not deferred.

Design principles this follows

  • Simplify project creation — attaching a model text is deferred, never a creation-time commitment.
  • Generalize the interlinearizer — a project isn't locked to one kind of suggestion source. The model can be a sibling interlinear project or a major-language Scripture project, attached, changed, or dropped whenever the analyst wants.
  • First-contact discoverable — the model surfaces intuitively at the point suggestions matter, and isn't worried about before then.

Reframe

#94's original framing tied the model text to a 4-way "Create new" mode chosen at project creation (PT9 Options 1 & 3). This issue takes the opposite approach:

Two kinds of model source (both in scope)

The #131 engine reuses approved glosses keyed by normalized surface form within the current draft (suggestion-engine.ts, selectPoolIndex). A model source extends that pool. Two kinds, matched two different ways:

  • Sibling interlinear project — surface-form reuse. Its approved TokenAnalysis glosses merge into the pool and match the source by surface form. Produces hits when it shares vocabulary with the source (same source text, a prior edition, a shared "master gloss" project). A direct extension of Add engine to generate token gloss suggestions from previous glosses #131. Also the natural landing spot for imported PT9 glosses (see below).
  • Major-language Scripture project — alignment-based (PT9's model text). A plain major-language translation (e.g. an English Bible) whose words are the gloss vocabulary. It is in a different language from the source, so surface-form matching does not apply. Suggestions come from source↔model word alignment: source word → aligned model word(s) → suggested gloss. Required for full PT9 interlinearization support (Options 1 & 3).

PT9 import interplay

The PT9 XML parser already exists (src/parsers/pt9/interlinearXmlParser.ts, schema) but is not wired into any import feature yet. Two things fall out for this issue:

  • When a PT9 interlinearization is imported, its clusters land as approved TokenAnalysis glosses. Those immediately enter the surface-form pool, so repeat occurrences of an already-glossed source word get suggestions from Add engine to generate token gloss suggestions from previous glosses #131's existing mechanism — no alignment needed for the already-glossed vocabulary.
  • The major-language model project is then what supplies suggestions for source words the import never glossed, via alignment. So the two mechanisms compose: imported glosses cover what PT9 already did; the model covers the rest.

The end-to-end PT9 XML → InterlinearProject import pipeline (mapping clusters/lexemes/glosses to analyses, resolving gloss text, linking the model + output projects, an import command/UI) is broader than this issue — tracked in #150. This issue covers only the model-text/suggestion side.

Model additions

  • InterlinearProject.modelProjectId?: string — the Platform.Bible project used as the suggestion source. May reference either a sibling interlinear project (→ surface-form reuse) or a plain major-language Scripture project (→ alignment). The engine chooses the resolution strategy by whether an interlinear analysis exists for the referenced project.
  • DraftProject.modelProjectId?: string — mirror it on the draft (exactly as targetProjectId is mirrored) so the model is live while editing, before Save As persists it.
  • On Save As, copy modelProjectId from draft → new InterlinearProject. On Open, seed it from the project → draft.

Engine / store additions

  • Surface-form tier (sibling interlinear model): load the model's TextAnalysis once (via interlinearizer.getProject), build its pool with buildPoolIndex, and seed it as a static modelPoolIndex that does not recompute on the draft's own edits. deriveTokenSuggestion gains a model tier: the draft's approved bucket owns the suggested slot; model entries the draft lacks become additional candidates.
  • Alignment tier (major-language model): a source↔model word-alignment step maps each source token to model word(s), whose text becomes the suggested gloss. Alignment can be seeded from imported PT9 cluster→gloss data and/or computed (eflomal / fast_align), and could be persisted via AlignmentLink. This is the larger part of the work.
  • Model-derived payloads carry a distinct producer (e.g. "model:{id}") so SuggestionDropdown can badge "from model" and they're never confused with local suggestions.
  • AnalysisStoreProvider gains a modelPoolIndex (or modelAnalysis) prop, threaded from the loader.

Settings additions

  • Add interlinearizer.useModelSuggestions: boolean to ProjectSettingTypes — a per-project view toggle to include/suppress model suggestions (parallels the existing showSuggestions demo flag, and lets users mute the model without detaching it). The which-project pointer stays on the model (above); only the on/off toggle is a setting.

Key technical wrinkle

buildPoolIndex documents that keying on surface form alone (ignoring writing system) is "correct for v1: the pool is a single source project whose word tokens share one writing system." That assumption:

  • Holds for a sibling interlinear model only when its writing system matches the source; otherwise the key must extend to (writingSystem, surfaceForm).
  • Is moot for a major-language model — cross-language surface forms never match, which is exactly why that path uses alignment rather than the pool key.

Either way the model's gloss language must overlap the project's analysisLanguages for gloss?.[analysisLanguage] to resolve.

Out of scope

Open questions

  • One field or two: a single modelProjectId for both model kinds (resolved by inspection), or separate modelInterlinearProjectId / modelTextProjectId?
  • Alignment approach for the major-language model: reuse imported PT9 cluster→gloss mappings, compute alignment (eflomal / fast_align), or both? Persist via AlignmentLink?
  • Explicit pointer vs. implicit shared pool — for the sibling-interlinear kind, should the pool implicitly widen to include all of the user's approved glosses on the same source instead of an explicit pointer? Explicit is recommended (matches PT9's mental model, gives control).
  • Should this go to user-questions.md for review outside the dev team, given it decides suggestion UX? (per AGENTS.md UX-decisions guidance)

Size: L — surface-form reuse alone is M; the alignment-based major-language model (required for PT9 parity) is the larger part.
Priority: P2 for the suggestion enhancement; the major-language model path is a prerequisite for full PT9 import parity.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions