RFC 006: Promote unnest/explode to core Substrait lowering

[dannymeijer]

Area

Specification (RFCs)

Summary

InQL RFC 002 classifies EXPLODE/unnest as a gap capability: no stable logical Rel exists in core Substrait, so implementations must use a documented extension relation with a registered URI (see docs/language/reference/substrait_operator_catalog.md — Gap profiles: Unnest / explode).

RFC 006 is opened in Blocked status to record the intent: when core Substrait adopts a portable unnest Rel that InQL can pin to, this capability should be reclassified from gap to core in the operator catalog and the extension encoding requirement retired.

Motivation

The current gap encoding for unnest adds extension URI maintenance burden and limits portability. Once upstream Substrait standardizes a logical unnest Rel, InQL should adopt it promptly so authors get core-portable EXPLODE semantics without a registered extension. Recording this now keeps the intent visible and gives the operator catalog a clear promotion path.

Proposal sketch

When a conforming portable unnest Rel lands in a Substrait revision InQL pins to:

1. Promote the unnest/explode entry in docs/language/reference/substrait_operator_catalog.md from gap to core.
2. Retire the extension encoding requirement, with a release-note entry per the revision and extension policy.
3. Update Incan compiler lowering to emit the core Rel instead of the extension relation.
4. Open RFC 006 as Planned and implement.

No InQL surface syntax changes are required — EXPLODE in query {} and explode on DataSet[T] remain as-is; only the emitted Substrait changes.

Alternatives considered

• Keep the extension encoding permanently — rejected; once a portable Rel exists, staying on an extension degrades portability for no benefit.
• Add the core Rel as an option alongside the extension — rejected; two encodings for the same operation create consumer ambiguity. The extension path should be cleanly retired.

Impact / compatibility

• Serialized Substrait plans containing the extension encoding for unnest will need to be re-emitted after the toolchain upgrade. This is a plan-level breaking change, documented in release notes per the revision and extension policy.
• No InQL surface syntax changes; no author migration required.
• Incan compiler lowering must be updated (Incan repo).

Implementation notes (optional)

• Track upstream Substrait for a stable unnest Rel proposal. A good signal is a merged spec PR in the substrait-io/substrait repository.
• Touch points: docs/language/reference/substrait_operator_catalog.md, Incan compiler lowering for EXPLODE/explode, release notes.

Checklist

• I checked for an existing RFC or issue covering this.