Cross-class reference resolution between derived artifacts

[amc-corey-cox]

Motivation

When two derived artifacts must share a name to form a reference — typical case: a slot_definition.range pointing at an enum_definition.name where both were derived from the same source instance — there's no mechanism today to declare that pairing. slot() (133a9e2) addresses intra-class binding but doesn't reach across derivations.

Authors today work around this by writing the same naming expression in both derivations and trusting them to stay consistent. That's fragile and makes the cross-reference invisible to the planner: it can't detect that the two artifacts must agree, can't validate the round-trip, can't optimize execution.

Concrete driver: schema-automator's EML importer (linkml/schema-automator#208), where an EML attribute with an enumeratedDomain derives both a slot (in the parent class's attributes) and an enum (in the schema-level enums map), and the slot's range must equal the enum's name. Same pattern for future XSD/JSON-Schema importers.

Proposed direction

Allow a derivation to publish a named binding scoped to its source instance, and another derivation to consume it:

class_derivations:
  AttributeToEnum:
    populated_from: Attribute
    target_class: enum_definition
    publishes: { enum_name_for: self }
    slot_derivations:
      name: { expr: "<naming expression>" }

  AttributeToSlot:
    populated_from: Attribute
    target_class: slot_definition
    slot_derivations:
      range: { expr: "ref(enum_name_for=self)" }

The binding key (enum_name_for) plus the source-instance identifier (self) form a lookup; the runtime guarantees the consumer sees what the producer published.

Execution model — opt-in scratch store

Cross-references require persistence between derivations, which conflicts with streaming and with the deliberate earlier decision to remove implicit memoization from the runtime. The proposal is opt-in two-pass execution backed by a DuckDB scratch store:

• Pass 1: stream the source. Producer derivations write bindings into a DuckDB temp table (source_id, binding_key, value). Artifacts that don't depend on cross-refs are emitted normally.
• Pass 2: consumer derivations resolve ref() calls against the scratch table and emit dependent artifacts.

Properties:

• Opt-in per trans-spec. Specs that declare no publishes/ref pairs stay single-pass and streamable. The memory cost is paid only by specs that ask for it.
• Deterministic by construction. Read-before-write can't happen — all writes in pass 1 complete before any reads in pass 2. Missing-binding errors surface as clear diagnostics ("no binding enum_name_for=Attribute_42") rather than silent nulls.
• Aligned with project direction. linkml-map is already investing in DuckDB (SQL compiler, engine work). The scratch store fits idiomatically; the same infrastructure also serves descendant iteration (populated_from_descendants: iterate source instances regardless of containment path #236).
• Optimizable. When the dependency graph is tractable — producer and consumer in the same derivation, or unambiguously ordered — the planner can choose single-pass topo-sort instead. Two-pass is the safe default.

Reversibility

Cross-references make structural pairing explicit and therefore reversible by the inverse engine: given a slot with range: foo and an enum with name: foo, the inverse derivation reconstructs the source Attribute. Reversibility of the binding key expression depends on its own invertibility (e.g., slugify is one-way and breaks round-trips; equality / FK-style joins are fine). This inherits linkml-map's existing reversibility-where-lossless principle without adding new rules.

Open questions

• Source-key scope. Restrict bindings to source-instance identity (must be hashable, drawn from identifier slots), or allow arbitrary computed keys? Lean toward source-instance identity first — keeps the scratch table small and the inverse mechanical.
• Multiple producers, one consumer. Should a binding key be allowed to be published by more than one derivation? Probably error unless explicitly declared multi-valued.
• Surface syntax. publishes / ref is the strawman; alternatives (e.g., explicit bindings: section at the spec level) might be cleaner.

References / contrasts

• slot() (133a9e2) — intra-class binding; doesn't reach across derivations
• dictionary_key / cast_collection_as: MultiValuedDict — in-collection keying, different problem
• populated_from_descendants: iterate source instances regardless of containment path #236 — descendant iteration; shares the scratch-store mechanism
• Multi-artifact emission: one source instance into multiple target containers #239 — multi-artifact emission; depends on cross-refs for reversibility

[github-actions]

Issue Summary

What problem does this solve?

Today, when one source instance must produce two separate derived artifacts that need to reference each other by name — e.g., a slot_definition whose range must equal the name of a co-derived enum_definition — there is no declarative way to express that pairing in linkml-map. Authors work around this by duplicating the naming expression in both derivations and relying on them staying consistent. That is:

• Fragile: any divergence produces a broken reference with no error.
• Opaque to the planner: the runtime can't see the constraint, can't validate round-trips, and can't optimize execution order.

Concrete driver

schema-automator #208 — the EML (Ecological Metadata Language) importer. An EML <attribute> with an <enumeratedDomain> must simultaneously produce:

1. A slot_definition in the parent class's attributes map.
2. An enum_definition in the schema-level enums map.

...and the slot's range must equal the enum's name. The same cross-derivation naming problem recurs for XSD and JSON-Schema importers.

Proposed solution: publishes / ref binding

A producer derivation declares that it publishes a named value keyed by source instance:

class_derivations:
  AttributeToEnum:
    populated_from: Attribute
    target_class: enum_definition
    publishes: { enum_name_for: self }       # ← produce a binding
    slot_derivations:
      name: { expr: "<naming expression>" }

  AttributeToSlot:
    populated_from: Attribute
    target_class: slot_definition
    slot_derivations:
      range: { expr: "ref(enum_name_for=self)" }  # ← consume that binding

The runtime guarantees the consumer sees exactly what the producer wrote.

Execution model: opt-in DuckDB scratch store (two-pass)

Cross-derivation references require persistence between derivations, which conflicts with linkml-map's deliberate streaming-default design. The proposal opts in when needed:

• Pass 1 (source stream): Producer derivations write (source_id, binding_key, value) tuples into a DuckDB temp table. Independent artifacts are emitted normally.
• Pass 2: Consumer derivations resolve ref() lookups against the scratch table and emit dependent artifacts.

Key properties:

• Opt-in per trans-spec — specs with no publishes/ref remain single-pass and streamable.
• Deterministic — all writes complete before any reads; missing bindings surface as clear diagnostics, not silent nulls.
• Aligned with project direction — linkml-map is already investing in DuckDB (SQL compiler). The scratch store fits idiomatically and unifies the execution story with #236 (descendant iteration).
• Optimizable — when dependency order is clear, the planner can choose single-pass topo-sort instead.

Reversibility

Making the cross-reference explicit also makes it invertible: given slot.range = "foo" and enum.name = "foo", the inverse derivation can reconstruct the source Attribute. Reversibility depends on the binding key expression's own invertibility (equality/FK-style keys are fine; slugify is one-way and breaks round-trips).

---

Related issues

Issue	Status	Relationship
#236 — populated_from_descendants	Open	Filed together; shares the same DuckDB scratch-store execution model and the same concrete driver (EML importer). Orthogonal feature — descendant iteration vs. cross-derivation naming.
schema-automator #208 — EML importer	Open	Primary concrete driver for both #236 and #237.
#212 — Implicit cross-table join	Closed ✓	Solved the related but distinct problem of resolving values across tables within a single derivation via shared key columns.
#211 — Silent nulls from cross-table nested derivations	Open	Different manifestation of cross-derivation data access; no join declared → silently null values.
#217 — Sparse implicit-join semantics	Open	Follow-up design issue on no-match behavior in implicit joins; deferred from PR #214.
#213 — Expr eval silently returns None for unknown variables	Closed ✓	Root mechanism behind #211; now raises instead of silently returning null.

---

Open design questions

1. Source-key scope — Restrict bindings to source-instance identity (hashable identifier slots), or allow arbitrary computed keys? Current lean: source-instance identity first (keeps the scratch table small and the inverse mechanical).
2. Multiple producers, one consumer — Should a binding key be publishable by more than one derivation? Current lean: error unless explicitly declared multi-valued.
3. Surface syntax — publishes / ref is the strawman. A top-level bindings: section at the spec level may be cleaner. Needs design discussion.

---

Status

Open, no implementation started. Requires:

• Trans-spec schema extension (publishes, ref())
• Planner logic to detect publishes/ref pairs and select execution mode
• DuckDB scratch-store pass-1/pass-2 runner (infra shared with populated_from_descendants: iterate source instances regardless of containment path #236)
• Validator static checks (detect ref() with no matching publishes)
• Inverse-engine extension to reconstruct source from paired artifacts