RFC: manifest-driven module architecture to stop per-model interface churn

[msluszniak]

RFC: manifest-driven module architecture to stop per-model interface churn

Why now

We have 17 modules across CV and NLP. Every new model touches ~20 files across 7 trees, and the rate at which model-specific quirks leak into shared family interfaces is going up. Three recent PRs illustrate the trend:

• #1100 (pose estimation) — added ~28 files across src/modules/, src/hooks/, src/types/, src/constants/, src/index.ts, native models, RnExecutorchInstaller.cpp, JsiConversions.h, docs, and the example app.
• #1104 (privacy filter) — same pattern, 22 files, +1702/-0.
• #1187 (BlazeFace) — the trigger for this RFC. To add one face-detector with one extra parameter, the native ObjectDetection arg list grew from 7 to 8, dragging BaseInstanceSegmentation.{h,cpp} and every integration test along with it — even though instance segmentation has nothing to do with BlazeFace.

This isn't a per-PR problem. The architecture makes it the path of least resistance.

What's actually wrong

Three compounding issues:

1. Per-model quirks live in family interfaces. ObjectDetectionOptions.inputSize is accepted for RF-DETR and SSDLite but ignored — the type lies. LLM capabilities are baked into load args. Instance-segmentation applyNMS is in the constructor. Every quirk surfaces in the shared type.
2. JSI bridge is positional. Each generate call unrolls to generateFromString(input, threshold, iouThreshold, classIndices, methodName, ...). Adding a field churns C++ signatures, every test fixture, and JsiConversions.h (currently 702 lines).
3. Model definitions are tripled. Each model is defined three times — in modelRegistry.ts, modelUrls.ts, and an in-module ModelConfigs const. No single source of truth.

What good looks like

A new model should be:

• Additive. Adding it never modifies a shared interface.
• Locally discoverable. Everything about the model lives in one folder.
• Type-enforced. Quirks belong to that model's options type, not the family's.
• Quirk-confined. There is nowhere in the architecture for a model-specific field to live except in that model's own definition.

Proposed direction

Five pieces, each independently mergeable.

1. Declarative model manifest

One TS object per model, the single source of truth:

export const YOLO26N = defineModel({
  family: 'object-detection',
  sources: { model: '...yolo26n.pte' },
  pre:  { kind: 'noop' },
  post: { kind: 'yolo-detect', labels: CocoLabelYolo },
  variants: { inputSize: [384, 512, 640] as const },
  defaults: { detectionThreshold: 0.5, iouThreshold: 0.5 },
});

export const RF_DETR_NANO = defineModel({
  family: 'object-detection',
  sources: { model: '...rf-detr-nano.pte' },
  pre:  { kind: 'imagenet-normalize' },
  post: { kind: 'rf-detr', labels: CocoLabel },
  // no `variants` → inputSize is not in this model's options type
  defaults: { detectionThreshold: 0.7, iouThreshold: 0.55 },
});

export const BLAZEFACE = defineModel({
  family: 'object-detection',
  sources: { model: '...blazeface.pte' },
  pre:  { kind: 'centered-crop', mean: BLAZEFACE_MEAN, std: BLAZEFACE_STD },
  post: { kind: 'blazeface', anchors: BLAZEFACE_ANCHORS },  // ← quirk lives HERE
  defaults: { detectionThreshold: 0.6 },
});

The quirk lives in the model file. No shared interface grows.

2. Generic family module, options derived from the manifest

class FamilyModule<F extends Family, M extends ModelOf<F>>
  extends Module<OutputOf<F, M>>
{
  static async load<M>(model: M, onProgress?): Promise<FamilyModule<…, M>>;
  forward(input: InputOf<F>, options?: OptionsOf<F, M>): Promise<OutputOf<F, M>>;
  runOnFrame?: WorkletOf<F, M>;
}

OptionsOf<F, M> is computed from the manifest. If M declares variants: { inputSize: […] }, the type includes inputSize?: 384 | 512 | 640. If it doesn't, the field doesn't exist. The compiler enforces "no quirky options for models that don't support them."

3. Uniform JSI bridge

Replace 17 global.load<X> globals + N positional-arg generate* methods with two operations:

global.rne.load(family: string, sources: object, config: object) → ModuleHandle
handle.invoke(method: string, payload: object) → Promise<object>
handle.unload()

All payloads are jsi::Object with named fields. Each module has typed Config/Payload structs plus a parseConfig(jsi::Object) helper. Adding a field = touch one struct + the parser. No positional signatures, ever.

4. Co-located feature folders

src/features/
  object-detection/
    family.ts        ← generic FamilyModule, options & output types
    hook.ts          ← const useObjectDetection = makeUseFamily('object-detection')
    models/
      yolo26.ts
      ssdlite.ts
      rf-detr.ts
      blazeface.ts
    postprocessors/
    index.ts         ← barrel
  pose-estimation/  …
  llm/  …

common/rnexecutorch/features/
  object_detection/
    Family.{h,cpp}             ← generic family runner
    postprocessors/
      YoloPostprocess.{h,cpp}
      RfDetrPostprocess.{h,cpp}
      BlazefacePostprocess.{h,cpp}   ← BlazeFace's anchors live here
    Registry.cpp               ← static self-registration

A new model becomes: one TS file in models/, optionally one C++ postprocessor pair, optionally a docs page. Zero touches to src/index.ts (barrel re-exports the folder), RnExecutorchInstaller.cpp (static registration), JsiConversions.h (generic payload), or other models.

5. Composites for multi-stage pipelines

export const OCR_ENGLISH = defineComposite({
  family: 'ocr',
  stages: { detect: CRAFT_ENGLISH, recognize: CRNN_ENGLISH },
});

The composite family runner wires stage outputs to stage inputs per a declarative schema. New pipeline architectures (ControlNet, multi-decoder LLMs) become new compositions, not new modules.

What this fixes

Problem today	Fixed by
Options that apply to ~60% of a family lie in types	(1) + (2) — manifest-derived OptionsOf<M>
Model-level config baked into constructors (NMS, capabilities, VAD)	(1) — manifest fields, parsed once on load
Pipeline components hardcoded per-model	(5) — defineComposite
Native positional args → 20-file churn	(3) — jsi::Object payload
Label definitions scattered across 3 files	(1) + (4) — single source in the model manifest
251-line src/index.ts growing per model	(4) — export * from './features/<x>'
forward_${inputSize} string-template dispatch	(1) + (2) — variants is declarative
Inconsistent options pattern (bag vs positional vs none)	(2) — single OptionsOf<F, M> shape
Hook layer is ~20 nearly-identical files	(4) — makeUseFamily(family), ~1 line each

Migration plan (not a big-bang rewrite)

1. Land (3) as an internal primitive. Add rne.load / invoke next to the existing globals. No user-facing change.
2. Migrate one family end-to-end as a vertical proof. Object detection — we have the BlazeFace pain in front of us. Build src/features/object-detection/ alongside the existing module; the old export becomes a deprecated shim that delegates.
3. Convert families in churn-rate order — detection → pose → segmentation (they share the forward_${size} smell), then NLP, then audio + OCR (composites).
4. Delete central registries (RnExecutorchInstaller.cpp switch, per-family globals) once all families are migrated.
5. Flip src/index.ts to barrel-of-features when no consumer-facing import path needs to change.

Each step is independently shippable. Reversible up to step 4.

Known risks

• TS type ergonomics. OptionsOf<F, M> needs careful conditional types. Mitigate with satisfies ModelDefinition<F> so errors surface at definition, not at call sites.
• Static self-registration in C++ can drop postprocessors under aggressive DCE. Needs explicit KEEP rules or per-feature anchor TUs. Verify on Hermes + iOS release builds early.
• JSI object parsing overhead. Tens of µs per call — negligible vs inference, but measure on STT/VAD streaming where call frequency is high.
• Composites are the softest spot. OCR is 2 stages, T2I is 4 with a scheduler loop, TTS has a phonemizer that calls back into another model. May need escape hatches. Plan to revisit after migrating single-stage families.
• Manifest can't capture everything. LLM context strategies, TTS streaming callbacks are behavioral, not data. Those stay as family-level concerns — but they're the genuinely shared cases, not the per-model leaks.

What I'd like input on

1. Does the diagnosis match what others are feeling? Or are there pain points this misses?
2. Is the manifest-derived options story the right lever? Alternatives considered: pure options-bag refactor (cheaper, doesn't stop the leak), per-model subclass per quirk (proliferation risk).
3. Migration order — does ObjectDetection make sense as the vertical proof, or is a smaller family (Classification?) a better first step to validate the design without too many moving parts?
4. Risks I'm not seeing — particularly around the C++ static-registration story and how this interacts with the typedoc API-reference generation.
5. Scope discipline — should (3) (the JSI bridge) land independently as an internal cleanup before any of (1)/(2)/(4) are taken on?

[msluszniak]

cc: @barhanc maybe this will not decrease number of modified / changed files exactly, but will definitely improve flow, so we don't change common base files anymore and adding new models will be easier and healthier for the lib itself.