populated_from dot-paths don't traverse inlined nested data (only FK joins)

[amc-corey-cox]

Motivation

populated_from dot-paths (#212) walk FK joins via ObjectIndex. For sources where the hierarchy is inlined (XML, JSON Schema, OWL, EML — anything tree-shaped that isn't relational), the same syntax doesn't reach: the FK-resolution path engages, finds no FK to resolve, warns Cross-class lookup requires object_index, and returns None.

Concrete driver: schema-automator's EML importer (linkml/schema-automator#208, branch eml-importer). Today it works around this by using three single-level class_derivations plus a Python loop that iterates dataset.dataTable[] and attributeList.attribute[] and invokes the transform per source instance. Future XSD / JSON-Schema importers will hit the same shape.

Proposed direction

When the source schema declares a slot with inlined: true (or inlined_as_list: true) and range: <SomeClass>, treat dot-path traversal as a structural walk through the inlined object/list rather than an FK lookup. Per-slot decision: identifier-shaped scalar → ObjectIndex (current); dict / list of dicts → walk directly (new).

With this in place, class_derivation chaining works for inlined sources symmetrically with FK ones:

class_derivations:
  SchemaDefinition:
    populated_from: EMLDocument
    slot_derivations:
      title:
        populated_from: dataset.title          # scalar, deep inline
      classes:
        populated_from: dataset.dataTable      # multivalued inline;
                                                # ClassDefinition auto-fires
                                                # per item
  ClassDefinition:
    populated_from: DataTable
    ...

No new memory cost — the source tree is already loaded; no ObjectIndex build needed for inlined paths.

Open questions

• Detection: prefer declarative (inlined: true from source schema). Runtime shape detection is a fallback.
• Diagnostics on missing path segments should name the failing segment, not bottom out as a silent None.

References / contrasts

• Implicit cross-table join resolution for nested and regular class derivations #212 — FK-join dot-paths (foundation this extends)
• populated_from_descendants: iterate source instances regardless of containment path #236 — descendant fan-in (different operation: "any descendant of X anywhere" vs this one's "known structural path through inline")
• Proposal: parent-context class derivations for flat-to-nested mappings #216 — parent-context derivations (adjacent shape, distinct)
• schema-automator EML importer (EML (Ecological Metadata Language) importer schema-automator#208) — concrete consumer currently working around this in Python

[github-actions]

Summary (auto-generated)

What this issue is about

populated_from dot-path traversal (e.g., dataset.title, dataset.dataTable) currently only works when crossing class boundaries via FK joins resolved through ObjectIndex. For tree-shaped / inlined source data (XML, JSON Schema, OWL, EML — anything where child objects are embedded in the parent rather than referenced by identifier), the dot-path hits the FK-resolution path, finds no FK to follow, emits a Cross-class lookup requires object_index warning, and returns None.

This is a structural gap in the traversal engine: the mapping DSL can express the right shape but the executor silently drops the value.

---

Concrete driver

schema-automator EML importer (schema-automator#208) — currently working around this by writing three separate single-level class_derivations and a Python loop that manually iterates dataset.dataTable[] and attributeList.attribute[], invoking the transformer per source instance. The desired declarative form:

class_derivations:
  SchemaDefinition:
    populated_from: EMLDocument
    slot_derivations:
      title:
        populated_from: dataset.title          # scalar deep inline — blocked today
      classes:
        populated_from: dataset.dataTable      # multivalued inline — blocked today
  ClassDefinition:
    populated_from: DataTable
    ...

The same workaround will be needed for future XSD / JSON-Schema importers unless this is fixed.

---

Proposed fix

When traversing a dot-path segment and the current slot's range class has inlined: true (or inlined_as_list: true) in the source schema, walk into the embedded object/list directly rather than routing through ObjectIndex. Decision per segment:

Segment value shape	Action
Identifier-shaped scalar	ObjectIndex lookup (current behavior)
Dict / list of dicts (inlined)	Structural walk (new behavior)

No additional memory cost — the source tree is already materialized; no ObjectIndex build needed for inlined paths.

---

Relationship to adjacent issues

Issue	Status	Relation
#212 — FK-join dot-paths	Closed (2026-05-08)	Foundation this extends. Covers implicit cross-table join resolution via ObjectIndex.
#236 — populated_from_descendants	Open	Different operation: "find all instances of class X anywhere in the tree" (unordered depth-first). This issue is "follow a known structural path". Same EML driver; complementary features.
#216 — parent-context class derivations	Open	Adjacent but distinct: synthesizing a nested target object from a flat source parent. Orthogonal axis.
schema-automator#208 — EML importer	Open	Primary consumer. Currently the workaround Python loop should collapse to pure declarative trans-spec once both this issue and #236 are resolved.

Key distinction from #236: This issue is about traversing a declared, structurally explicit path (a.b.c) through embedded objects. Issue #236 is about finding all instances of a given class anywhere in the tree without knowing the containment path in advance (**.EnumeratedDomain). Both are needed for the EML importer; they compose rather than compete.

---

Open questions (from issue)

1. Detection strategy: prefer declarative (inlined: true from source schema) with runtime shape detection (is the value a dict/list?) as a fallback. Worth deciding whether schema-absent fallback is on by default or opt-in.
2. Diagnostics: missing path segments should name the failing segment explicitly, not silently return None. This is a quality-of-life fix independent of the inlined-traversal feature.

---

Implementation sketch

The change is localized to the dot-path resolution logic in ObjectTransformer (likely the section that currently routes to ObjectIndex). The per-segment branch needs to check whether the current object is already a dict/list (inlined) before attempting FK resolution. Schema-level confirmation (inlined: true) can be layered on top for the strict/declarative detection mode.