populated_from_descendants: iterate source instances regardless of containment path

[amc-corey-cox]

Motivation

Some transformations need to lift instances of a given source class out of a deep tree into a flat target collection, regardless of where in the source hierarchy they appear. Today there's no syntax for this — populated_from is scoped to the immediate parent slot's range, so authors must either enumerate every containment path or pre-flatten the source in Python before invoking linkml-map.

Concrete driver: schema-automator's EML importer (linkml/schema-automator#208). EML's <enumeratedDomain> elements appear nested under dataset.dataTable[*].attributeList.attribute[*].measurementScale.nominal.nonNumericDomain (and similarly under ordinal). Building a flat enums map at the target schema's root requires walking all those nested locations. The same pattern will recur for future schema-as-source importers (XSD, JSON-Schema).

Proposed direction

Add a derivation key that matches descendants of a given source class rather than slot-scoped immediate children:

class_derivations:
  EnumDefinitions:
    populated_from_descendants: EnumeratedDomain   # all instances under tree_root
    target_class: enum_definition
    slot_derivations: { ... }

Or as a slot-derivation modifier:

slot_derivations:
  enums:
    populated_from: EnumeratedDomain
    descendants_of: $tree_root

Execution / memory

Streaming-compatible: a single pass can materialize matched descendants directly into the target collection without retaining ancestors. No new memory pressure beyond what the target collection itself requires.

For trans-specs that combine descendant iteration with cross-references (#237), the same opt-in DuckDB-backed scratch store proposed there serves both — collect descendants into a temp table in pass 1, query in pass 2. This unifies the execution-mode story: linkml-map has a streaming default and a single opt-in "scratch-store, two-pass" mode that unlocks descendant iteration, cross-references, and their combination.

Open questions

• Match semantics. All descendants, or first-match-per-branch? (Lean: all.)
• Order. Document-order is the natural default for tree-walks; guarantee it or leave implementation-defined?
• Surface choice. Explicit key (populated_from_descendants) is unambiguous. A path-expression form (populated_from: "**.EnumeratedDomain") is more compact but adds parser surface. Lean toward the explicit key first; paths can follow if real patterns demand it.

References / contrasts

• Proposal: parent-context class derivations for flat-to-nested mappings #216 — parent-context class derivations (one source → one nested target; opposite direction)
• Support multi-source class derivations via sources, slot-level sources, and _source_class binding #193 — multi-source class derivations via sources (enumerated source-class list; no tree walk)
• Implement multi-row aggregation, group-by, and collection membership (in) operator #188 — multi-row aggregation (row reduction, not class emission)
• implement pivot operation, for unmelting generic measurement/EAV data to wide tables #103 — PivotOperation (tabular reshape, orthogonal)
• Implicit cross-table join resolution for nested and regular class derivations #212 — implicit cross-table join (dot-path foundation this builds on)
• Cross-class reference resolution between derived artifacts #237 — cross-class references; shares the scratch-store mechanism
• Multi-artifact emission: one source instance into multiple target containers #239 — multi-artifact emission; composable with descendant iteration

[github-actions]

Summary & Context

What this issue proposes

A new populated_from_descendants derivation key (or equivalent mechanism) that allows a class derivation to collect all instances of a given source class from anywhere in a deep object tree, regardless of containment path, and emit them into a flat target collection. This is the "deep-to-flat lifting" problem — today the only workarounds are enumerating every nesting path explicitly in the trans-spec, or pre-flattening in Python before invoking linkml-map.

Concrete driver: The EML importer in schema-automator (schema-automator#208) needs to collect <enumeratedDomain> elements that can appear at dataset.dataTable[*].attributeList.attribute[*].measurementScale.nominal.nonNumericDomain (and under ordinal). Every nesting level multiplies the paths to enumerate. The same deep-tree problem recurs for any schema-as-source importer (XSD, JSON Schema).

---

Proposed syntax (two candidates)

Option A — explicit class-derivation key:

class_derivations:
  EnumDefinitions:
    populated_from_descendants: EnumeratedDomain   # match all instances anywhere in tree
    target_class: enum_definition
    slot_derivations: { ... }

Option B — slot-derivation modifier:

slot_derivations:
  enums:
    populated_from: EnumeratedDomain
    descendants_of: $tree_root

The issue leans toward Option A as the starting point (unambiguous, explicit), with a possible path-expression form (populated_from: "**.EnumeratedDomain") as a future compact alternative.

---

Execution model

• Streaming-compatible: a single tree walk can materialize matched descendants directly into the target collection with no ancestor retention.
• Two-pass mode (opt-in): when this feature is combined with cross-references (see the companion cross-refs issue, not yet filed as of 2026-05-14), the same DuckDB-backed scratch store proposed for cross-ref handling unifies both — collect descendants into a temp table in pass 1, query in pass 2. This keeps linkml-map's default single-pass streaming behavior intact.

---

Open questions to resolve

Question	Lean / notes
Match semantics	All descendants vs. first-match-per-branch → lean all
Ordering	Document-order is natural for tree walks; leave implementation-defined or guarantee it?
Surface choice	Explicit key first; path-expression form later if real patterns demand

---

Related issues — status snapshot

Issue	Title	Status	Relationship
#216	Parent-context class derivations (populated_from_parent)	Open	Inverse direction — flat source → nested target (structural expansion). Together with #236 these form a "lifting/lowering" pair covering both structural directions.
#193	Multi-source class derivations via sources	Open	Different: enumerated list of source classes with no tree walk; 94.7% reduction in repetition for BDC trans-specs
#188	Multi-row aggregation, group-by, in operator	Open	Row-reduction/aggregation within joined tables; orthogonal to tree walking but shares the DuckDB execution-mode story
#212	Implicit cross-table join resolution	Closed ✓	Dot-path foundation that descendant path resolution builds on
#103	PivotOperation	Closed ✓	Tabular reshape, fully orthogonal

Note: The companion "cross-references issue" mentioned in the execution section was flagged as "to be filed alongside this one" — check whether it has been filed since 2026-05-14.

---

Key architectural insight

This issue and #216 together address the structural asymmetry gap in linkml-map:

• Proposal: parent-context class derivations for flat-to-nested mappings #216 (populated_from_parent): target has more nesting than source → synthesize nested target objects from a flat source context
• populated_from_descendants: iterate source instances regardless of containment path #236 (populated_from_descendants): source has more nesting than target → lift deep source instances into a flat target collection

Neither direction is handled cleanly today. Both are recurring patterns in schema-as-source importers (EML, XSD, JSON Schema), and both are being driven by the same concrete use case in schema-automator.

---

Implementation note

The sources field already exists in the schema on ClassDerivation, SlotDerivation, EnumDerivation, and PermissibleValueDerivation (per #193) — a new populated_from_descendants key would need to be added to the transformer model YAML and plumbed through ObjectTransformer.map_object. Per CLAUDE.md: check for # EXTRACT: markers in map_object before touching that section.