Add cross-table lookup support for join-based transformations

[amc-corey-cox]

Summary

• Extends AliasedClass with source_key, lookup_key, and join_on fields to specify cross-table join keys
• Creates DuckDB-backed LookupIndex (src/linkml_map/utils/lookup_index.py) for fast keyed row lookups from CSV/TSV files
• Fixes _eval_set in eval_utils.py to accept {obj.attr} syntax (e.g., {demographics.age_at_exam}) with null propagation
• Wires cross-table resolution into Bindings via _resolve_join(), returning DynObj wrappers for attribute access
• Adds get_path() to DataLoader for resolving table names to file paths
• Creates transform_spec() engine (src/linkml_map/transformer/engine.py) that iterates class_derivation blocks, registers secondary tables, streams rows through map_object, and cleans up joins

Example YAML syntax

class_derivations:
  MeasurementObservation:
    populated_from: lab_results
    joins:
      demographics:
        join_on: participant_id          # shorthand: same column name in both tables
      # or explicit:
      # demographics:
      #   source_key: participant_id
      #   lookup_key: subject_id
    slot_derivations:
      analyte_value:
        populated_from: result_value
      age_at_observation:
        expr: '{demographics.age_at_exam} * 365'

Design notes

• Uses join_on instead of on because YAML 1.1 parses bare on as boolean True
• DuckDB loads all columns as VARCHAR (all_varchar=true) to avoid type coercion surprises
• Existing data-driven CLI path (_transform_iterator, map-data) is not touched — the new spec-driven path is added alongside it
• SQL injection is prevented by validating all identifiers against [a-zA-Z_][a-zA-Z0-9_]*

Test plan

• 8 unit tests for LookupIndex (register, lookup, missing row, drop, CSV, identifier validation, varchar coercion)
• 4 eval_utils tests for {obj.attr} curly-brace attribute access with null propagation
• 6 integration tests for cross-table lookups (join_on shorthand, explicit keys, null propagation, arithmetic expressions, multiple joins, missing key error)
• Full test suite passes (439 passed, 4 skipped, 0 failures)

Closes #134

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds spec-driven cross-table lookup (join) capabilities to the transformation engine, enabling {joined_table.column} expressions and join resolution via DuckDB-backed indexing.

Changes:

• Introduces a DuckDB-backed LookupIndex and a new transform_spec() engine that registers/drops join tables and streams primary rows through map_object.
• Extends join specifications (AliasedClass) with source_key, lookup_key, and join_on, and wires join resolution into Bindings.
• Updates expression evaluation to allow {obj.attr} null-propagating syntax, with new unit + integration tests.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tests/test_utils/test_lookup_index.py	Adds unit coverage for LookupIndex (register/lookup/drop/validation/varchar coercion).
tests/test_utils/test_eval_utils.py	Adds tests for {obj.attr} curly-brace attribute access + null propagation.
tests/test_transformer/test_cross_table_lookup.py	End-to-end tests for join-based lookups via the new spec-driven engine.
src/linkml_map/utils/lookup_index.py	Implements the DuckDB in-memory lookup/index used for joins.
src/linkml_map/utils/eval_utils.py	Extends _eval_set to allow {obj.attr} in null-propagation braces.
src/linkml_map/transformer/object_transformer.py	Adds join specs to Bindings and resolves joined rows to DynObj.
src/linkml_map/transformer/engine.py	New transform_spec() iterator that registers join tables and streams transformations.
src/linkml_map/loaders/data_loaders.py	Adds get_path() helper for resolving table identifiers to file paths.
src/linkml_map/datamodel/transformer_model.yaml	Documents/defines new join key fields on AliasedClass.
src/linkml_map/datamodel/transformer_model.py	Generated model updates for the new AliasedClass join key fields.

[turbomam]

Hey Corey — nice work on the cross-table joins. The architecture is clean: LookupIndex / DynObj / Bindings separation is well-layered, and I appreciate that the existing data-driven CLI path is untouched.

I put together two PRs that target your branch:

PR #142 — 11 passing edge-case tests

Covers gaps I noticed in test coverage:

• Duplicate key behavior — documents the LIMIT 1 first-match semantics (the test doesn't assert which duplicate wins, just that a row comes back)
• Empty secondary tables — headers-only TSV files register and query cleanly
• LookupIndex.close() lifecycle — close clears state, post-close ops raise, double-close is safe
• Engine no-joins regression — transform_spec with no joins: block (the common case has no test coverage through the engine path)
• Mixed derivations — joins and non-joins coexist in one spec

All 11 pass on cross-table-lookup. Should be a clean merge.

PR #144 + Issue #143 — resource cleanup gap (3 failing tests)

transform_spec() creates a LookupIndex on line 45 but never calls close(), so the DuckDB connection leaks. Also, LookupIndex doesn't support the with statement.

The fix is small — __enter__/__exit__ on LookupIndex + a top-level try/finally in transform_spec. Happy to implement if you'd like, or feel free to take it.

One other thing to consider

The lookup_row docstring says "Return the first row" but doesn't mention what "first" means when there are duplicate keys. Might be worth either:

• Documenting "arbitrary row for non-unique keys" in the docstring
• Or adding a warn_on_duplicates option

Not blocking — just flagging for your consideration.