Design: Multi-source to single-target harmonization

[matthewhorridge]

Design: Support multi-source to single-target harmonization rules.

Background

• Some harmonizations require combining multiple source data elements into a single target element.
• Example: one-hot encoded source columns -> single enum target (e.g., REDCap coding).
• Other examples: split fields (first/last) to a canonical name, separate date/time to a combined timestamp, or multiple flags to a consolidated category.

Goals

• Define a rule model that can reference multiple source elements.
• Specify serialization schema changes and backward compatibility expectations.
• Clarify how rule lookup and application should work in the pipeline (e.g., list-of-sources mapping).
• Define how replay logging should represent multi-source transformations.

Open questions

• Should a new rule type be introduced (e.g., "multi_source") or extend existing rule schema?
• How to express source ordering and missing-value handling?
• How to support one-hot -> enum mapping explicitly (e.g., reduce + mapping)?
• How should validation work when sources are missing?

Deliverables (design only)

• Proposed schema with examples
• Changes needed in RuleRegistry lookup and harmonize pipeline
• Replay log implications
• Test strategy

[matthewhorridge]

Here is a concise plan and design discussion for multi-source → single-target rules:

Option A vs Option B

• Option A (recommended): introduce a new multi-source rule type with explicit "sources" array.
  Example:

{
  "sources": ["flag_a", "flag_b", "flag_c"],
  "target": "visit_type",
  "operations": [
    {
      "operation": "reduce",
      "reduction": "one-hot"
    },
    {
      "operation": "enum_to_enum",
      "mapping": {
        "0": "baseline",
        "1": "follow_up",
        "2": "screening"
      }
    }
  ]
}

Pros: explicit schema, preserves backward compatibility, supports source ordering.
Cons: new rule type handling in registry/pipeline and replay logs.

• Option B: extend existing schema by allowing "source" to be string or list.
  Example:

      {
        "source": ["flag_a", "flag_b"],
        "target": "visit_type",
        "...": "..."
      }

  Pros: fewer schema fields.
  Cons: polymorphic source increases ambiguity and risks subtle bugs.

Recommendation: Option A for clarity and compatibility.

Combining per-source transforms then reducing
Three design patterns:

1. Explicit per-source ops + combine stage:
   {sources:[...],per_source_operations:{...},combine:[...]}.
2. Introduce a map_each primitive that applies nested ops to each value, then reduce.
3. Implicit approach: rely on support_iterable so list inputs are processed element-wise, then reduce.
   Recommendation: Option A (explicit) or Option B (map_each) for clarity.

API impact (harmonize command)

• Current "pairs" is insufficient for multi-source.
• Options:
  A) Add a new "multi_source" list alongside "pairs" (backward compatible but two lists).
  B) Replace/alias with "mappings" list that allows either {source,target} or {sources,target}.
  C) Select by rule IDs (requires stable IDs in rules).
  Recommendation: introduce "mappings" and keep "pairs" as a deprecated alias.

Open questions

• How to define source ordering (critical for one-hot).
• Missing/None source handling and validation behavior.
• Replay log shape for multi-source rules.

[matthewhorridge]

Implementation Plan

Adopting Option A from the earlier design discussion: a new MultiSourceHarmonizationRule class with an explicit sources array. Full backward compatibility is preserved.

Schema

Multi-source rules live under a reserved "multi_source" top-level array in the rules JSON file:

{
  "height_in": {
    "height_cm": { "source": "height_in", "target": "height_cm", "operations": [...] }
  },
  "multi_source": [
    {
      "sources": ["flag_baseline", "flag_followup", "flag_screening"],
      "target": "visit_type",
      "operations": [
        { "operation": "reduce", "reduction": "one-hot" },
        { "operation": "enum_to_enum", "mapping": {"0": "baseline", "1": "follow_up", "2": "screening"} }
      ]
    }
  ]
}

Source ordering is determined by the sources array. The operations chain receives the values as an ordered list — existing primitives reduce and enum_to_enum cover the one-hot → enum use case without needing new primitives.

Files to change

File	Change
harmonization_rule.py	Extract _deserialize_operations() helper (shared with new class)
multi_source_rule.py (new)	MultiSourceHarmonizationRule with serialize, transform, from_serialization
rule_registry.py	_multi_source_rules list; add_multi_source_rule, query_multi_source, list_multi_source_mappings; updated load/save/clean
harmonize.py	Optional multi_source_mappings parameter; df.apply(axis=1) to gather multi-column values
cli.py	_select_multi_source_mappings, _filter_missing_multi; wired into main()
__init__.py	Export new class
tests/test_multi_source_rule.py (new)	Unit + integration tests
tests/test_cli.py	Multi-source CLI cases

Open questions resolved

• Source ordering: explicit, determined by the sources array position
• Missing sources: follows existing --on-missing semantics — a multi-source mapping is skipped if any source column is absent
• Replay log: Event.to_log() is unchanged; rule.serialize() emits "sources" instead of "source", so the log shape naturally reflects the multi-source structure

Out of scope for this issue

Heterogeneous column types where one source provides a value and another provides its unit (e.g., height_magnitude + height_unit). That pattern requires a new primitive (e.g., apply_unit_from_data) and will be tracked separately.

[matthewhorridge]

Implementation depends on the architectural refactor tracked in #102 (unify rule model + replace RuleRegistry with RuleSet). That work should land first.