feat(ir): Add Submit IR node — print task launches as pl.submit(...)#1544
feat(ir): Add Submit IR node — print task launches as pl.submit(...)#1544Hzfengsy wants to merge 10 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR introduces ChangesSubmit IR node introduction
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request introduces a new first-class IR node kind Submit to represent asynchronous task launches (pl.submit(...) inside pl.manual_scope), separating them structurally from plain synchronous Call nodes. The changes span the C++ IR definitions, AST parser, Python printer, SSA conversion, dead code elimination, Python bindings, and type stubs, alongside comprehensive unit and integration tests. The review feedback highlights two key improvements: addressing a contradiction in the Python printer where a null current_program_ would trigger an abort instead of falling back to a safe printing mode, and replacing a raw assert in the AST parser with a user-friendly ParserSyntaxError when parsing annotated assignments of pl.submit.
There was a problem hiding this comment.
Pull request overview
This PR introduces a first-class Submit IR expression kind to represent pl.submit(...) task launches distinctly from plain synchronous Call, improving IR dumps and enabling submit-aware analyses/passes while preserving the existing late-pipeline/codegen Call-based behavior via a Submit → Call lowering in DeriveCallDirections.
Changes:
- Add
ir.Submitacross core IR (newObjectKind, kind traits, reflection descriptors) and wire it through visitors/mutators, structural equality/hash, DCE, SSA, and arithmetic analyzers. - Update the Python DSL parser and IR Python printer to round-trip
pl.submit(...), including the printer’s single-LHS tuple-typed form. - Add/adjust unit and integration tests and update IR documentation + introduce a
.clauderule for pass authors to handleSubmitalongsideCall.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/ut/language/parser/test_task_id_dsl.py | Update test helpers/assertions to treat Submit as call-like and use submit.deps. |
| tests/ut/language/parser/test_manual_scope_parsing.py | Broaden call collection to include Submit; add coverage for single-LHS submit binding; switch dep assertions to submit.deps. |
| tests/ut/ir/transforms/test_submit_passes.py | New tests validating SSA + round-trip behavior on Submit. |
| tests/ut/ir/transforms/test_submit_end_to_end.py | New integration tests ensuring mid-pipeline dumps show pl.submit(...). |
| tests/ut/ir/transforms/test_flatten_call_expr_pass.py | Ensure flattening preserves Submit.deps when rewriting submit args. |
| tests/ut/ir/printing/test_submit_printer.py | New tests for printing and structural equality/hash behavior of Submit. |
| src/ir/transforms/visitor.cpp | Add IRVisitor traversal of Submit args, deps, and certain var-typed attrs. |
| src/ir/transforms/utils/dead_code_elimination.cpp | Treat Submit as side-effecting for DCE; ensure expressions containing Submit are preserved. |
| src/ir/transforms/structural_hash.cpp | Add structural hashing dispatch for Submit. |
| src/ir/transforms/structural_equal.cpp | Add structural equality dispatch for Submit. |
| src/ir/transforms/python_printer.cpp | Print Submit as pl.submit(self.<kernel>, ..., deps=[...]) and collect GlobalVars through Submit. |
| src/ir/transforms/mutator.cpp | Add IRMutator support to rewrite Submit (args/deps/type and selected attrs). |
| src/ir/transforms/derive_call_directions_pass.cpp | Lower Submit → Call in DeriveCallDirections via SubmitToCallView. |
| src/ir/transforms/convert_to_ssa_pass.cpp | Ensure SSA conversion also rewrites Submit return type appropriately. |
| src/ir/arith/rewrite_simplify.h | Add Submit visitor hook to rewrite simplifier. |
| src/ir/arith/rewrite_simplify.cpp | Treat Submit as a leaf for rewrite simplifier. |
| src/ir/arith/modular_set.cpp | Treat Submit conservatively in modular set analyzer. |
| src/ir/arith/int_set.cpp | Treat Submit conservatively in int set analyzer. |
| src/ir/arith/const_int_bound.cpp | Treat Submit conservatively in const-int bound analyzer. |
| src/ir/arith/canonical_simplify.h | Add Submit visitor hook to canonical simplifier. |
| src/ir/arith/canonical_simplify.cpp | Treat Submit as a leaf for canonical simplifier. |
| src/codegen/orchestration/orchestration_codegen.cpp | Funnel Submit through existing codegen via SubmitToCallView. |
| python/pypto/pypto_core/ir.pyi | Add Submit stub API (op/args/deps/kwargs/attrs/arg_directions). |
| python/pypto/language/parser/ast_parser.py | Parse pl.submit(...) into ir.Submit; accept single-LHS printed form; move deps out of attrs into deps_. |
| python/bindings/modules/ir.cpp | Bind Submit into Python IR module with kwargs/attrs/arg_directions accessors. |
| include/pypto/ir/transforms/base/visitor.h | Add Submit dispatch in base visitor interface. |
| include/pypto/ir/transforms/base/mutator.h | Add Submit dispatch in base mutator interface. |
| include/pypto/ir/transforms/base/functor.h | Add Submit dispatch in ExprFunctor + dispatch table. |
| include/pypto/ir/kind_traits.h | Register Submit kind trait and include it under Expr kinds. |
| include/pypto/ir/expr.h | Define Submit IR node and the SubmitToCallView adapter. |
| include/pypto/ir/core.h | Add ObjectKind::Submit. |
| docs/en/dev/ir/01-hierarchy.md | Document Submit in hierarchy and add Submit vs Call section. |
| .claude/rules/pass-submit-awareness.md | New rule documenting how passes must handle Submit vs Call. |
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (1)
tests/ut/ir/transforms/test_submit_passes.py (1)
92-110: ⚡ Quick winMake the deps-renaming assertion match the test name.
test_ssa_renames_submit_args_and_depscurrently verifies arg SSA rewrite only. Either add a deps rewrite assertion (by forcing a deps var rewrite) or rename the test to avoid over-claiming coverage.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/ut/ir/transforms/test_submit_passes.py` around lines 92 - 110, The test name claims both args and deps are SSA-renamed but only asserts args; either update the test to assert deps were renamed or rename the test to match current behavior. To fix in test_ssa_renames_submit_args_and_deps, either (A) ensure _build_program_with_submit produces a deps Var that gets rewritten and add an assertion like checking submit_after.deps[i] is an ir.Var and is not caller_after.params[j] (use submit_after.deps and caller_after.params to locate symbols), or (B) rename the test to something like test_ssa_renames_submit_args to reflect it only verifies args; keep references to _build_program_with_submit and passes.convert_to_ssa to locate the setup and transformation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/en/dev/ir/01-hierarchy.md`:
- Around line 156-182: Update the RuntimeScope/dep-encoding paragraph to match
the Submit-first flow: state that the parser emits Submit nodes with a
first-class deps_ field (Submit.deps_) and that DeriveCallDirections lowers
Submit by folding deps_ into Call.attrs["manual_dep_edges"], so later passes and
runtime see Call with attrs["manual_dep_edges"]. Mention Submit, Submit.deps_,
DeriveCallDirections and Call.attrs["manual_dep_edges"] explicitly so the doc
consistently describes Submit-first encoding folded to Call during lowering.
In `@include/pypto/ir/expr.h`:
- Around line 905-923: SubmitToCallView currently copies submit->attrs_
unchanged which preserves any existing kAttrManualDepEdges when submit->deps_ is
empty and it silently drops non-Var-like entries in submit->deps_; update
SubmitToCallView to (1) remove/clear any existing kAttrManualDepEdges from attrs
when submit->deps_ is empty (so synthesized Call never inherits stale manual-dep
metadata) and (2) validate entries in submit->deps_ instead of quietly skipping
non-Var-like objects: iterate submit->deps_ and if an entry is null or its kind
is neither ObjectKind::Var nor ObjectKind::IterArg, surface an error (throw or
LOG and return) rather than ignoring it; continue to convert valid Var/IterArg
entries to VarPtr and call WithManualDepEdgesAttr(attrs, dep_vars) when deps are
present. Ensure you reference and modify the existing symbols SubmitToCallView,
attrs, submit->deps_, kAttrManualDepEdges, WithManualDepEdgesAttr, and Call
construction so the new behavior prevents stale attrs and dropped deps.
In `@python/pypto/language/parser/ast_parser.py`:
- Around line 987-994: The fast-path in the AST parser currently short-circuits
annotated-assignment validation for single-LHS pl.submit calls by calling
_parse_submit_single_lhs and returning; instead, modify _parse_submit_single_lhs
(or create a new helper) to return the constructed ir.Submit node and then let
the normal annotated-assignment binding/override_type validation flow handle it
so annotations are validated; update the call site (the block using _is_pl_call
and _parse_submit_single_lhs) to receive the ir.Submit result and proceed
through the same annotated-assignment/override_type path used for other RHS
expressions (the same logic used elsewhere in the function and mirrored in the
other occurrence around lines ~1363-1395).
In `@src/codegen/orchestration/orchestration_codegen.cpp`:
- Around line 892-899: The synthetic Call created by SubmitToCallView loses the
original Submit* identity so tuple-key lookups (used by
GenerateSubmitReturnAliases) miss entries; preserve the original node identity
by propagating the Submit pointer into the Call view or by recording a mapping
from the synthetic Call to the original Submit. Concretely, update
SubmitToCallView to attach the original Submit* (e.g., a member like
original_submit_ or a side-map from Call* -> Submit*) and change the tuple-key
lookup path (the code that inspects assign->value_ and later lookup logic used
by GenerateSubmitReturnAliases) to consult that original_submit_ mapping when
present so alias bookkeeping uses the Submit* key instead of the synthetic
Call*.
In `@src/ir/transforms/derive_call_directions_pass.cpp`:
- Around line 409-422: The Submit→Call lowering in VisitExpr_(const SubmitPtr&
op) loses original expression identity so first_writer_roots_ (which is keyed by
Call*) can't find Submit-originated first-writer roots, causing Out args to be
misclassified as InOut; fix this by preserving the original Submit identity
before calling VisitExpr_(view_call) — e.g., capture the original submit pointer
(op or submit.get()) and register a temporary mapping from the synthesized Call
pointer (view_call.get()) back to the original submit identity in
first_writer_roots_ (or adjust first_writer_roots_ to accept an alternative key
sourced from Submit) so the direction derivation code (VisitExpr_,
SubmitToCallView, first_writer_roots_) can look up the correct first-writer
context and keep Out arguments as first-writers.
In `@src/ir/transforms/python_printer.cpp`:
- Around line 827-869: The VisitExpr_ for Submit currently hard-fails when
current_program_ or As<GlobalVar> is missing; change it to detect whether the
callee is a GlobalVar inside a Program and only use the "self.<kernel>" form in
that case, otherwise fall back to a generic callee emission so
Print(Function)/Print(Expr) won't crash: in IRPythonPrinter::VisitExpr_(const
SubmitPtr& op) compute gvar = As<GlobalVar>(op->op_) and if (gvar &&
current_program_) emit ".submit(self.<kernel_name>...)" as now, else emit
".submit(" then VisitExpr(op->op_) (or otherwise print the op->op_ generically)
followed by the same args, deps and attrs logic; remove or replace the
INTERNAL_CHECK_SPAN that enforces current_program_ so the fallback path is used
instead of hard-failing.
In `@tests/ut/ir/transforms/test_submit_end_to_end.py`:
- Line 59: Rename the unused unpacked variables from pl.submit(self.stage1, x,
out) — currently named scratch and tid — to underscore-prefixed names (e.g.,
_scratch and _tid or just _) to silence RUF059; locate the call to pl.submit in
test_submit_end_to_end.py (the line invoking pl.submit(self.stage1, x, out)) and
change the left-hand unpacking identifiers accordingly so the test behavior
remains identical while avoiding lint warnings.
In `@tests/ut/ir/transforms/test_submit_passes.py`:
- Around line 12-15: Update the module docstring in the test_submit_passes.py
test module: replace or remove the stale sentence saying “parser does not yet
emit Submit” so it correctly reflects that the parser now emits Submit (e.g.,
state that tests construct Submit-bearing IR to verify passes handle
parser-emitted Submit) and ensure the docstring mentions why the tests exist
(preserve Submit structural shape) rather than claiming the parser can’t emit
it.
---
Nitpick comments:
In `@tests/ut/ir/transforms/test_submit_passes.py`:
- Around line 92-110: The test name claims both args and deps are SSA-renamed
but only asserts args; either update the test to assert deps were renamed or
rename the test to match current behavior. To fix in
test_ssa_renames_submit_args_and_deps, either (A) ensure
_build_program_with_submit produces a deps Var that gets rewritten and add an
assertion like checking submit_after.deps[i] is an ir.Var and is not
caller_after.params[j] (use submit_after.deps and caller_after.params to locate
symbols), or (B) rename the test to something like test_ssa_renames_submit_args
to reflect it only verifies args; keep references to _build_program_with_submit
and passes.convert_to_ssa to locate the setup and transformation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 1ae4a749-82b8-42a2-941a-b378f45cbe58
📒 Files selected for processing (33)
.claude/rules/pass-submit-awareness.mddocs/en/dev/ir/01-hierarchy.mdinclude/pypto/ir/core.hinclude/pypto/ir/expr.hinclude/pypto/ir/kind_traits.hinclude/pypto/ir/transforms/base/functor.hinclude/pypto/ir/transforms/base/mutator.hinclude/pypto/ir/transforms/base/visitor.hpython/bindings/modules/ir.cpppython/pypto/language/parser/ast_parser.pypython/pypto/pypto_core/ir.pyisrc/codegen/orchestration/orchestration_codegen.cppsrc/ir/arith/canonical_simplify.cppsrc/ir/arith/canonical_simplify.hsrc/ir/arith/const_int_bound.cppsrc/ir/arith/int_set.cppsrc/ir/arith/modular_set.cppsrc/ir/arith/rewrite_simplify.cppsrc/ir/arith/rewrite_simplify.hsrc/ir/transforms/convert_to_ssa_pass.cppsrc/ir/transforms/derive_call_directions_pass.cppsrc/ir/transforms/mutator.cppsrc/ir/transforms/python_printer.cppsrc/ir/transforms/structural_equal.cppsrc/ir/transforms/structural_hash.cppsrc/ir/transforms/utils/dead_code_elimination.cppsrc/ir/transforms/visitor.cpptests/ut/ir/printing/test_submit_printer.pytests/ut/ir/transforms/test_flatten_call_expr_pass.pytests/ut/ir/transforms/test_submit_end_to_end.pytests/ut/ir/transforms/test_submit_passes.pytests/ut/language/parser/test_manual_scope_parsing.pytests/ut/language/parser/test_task_id_dsl.py
- python_printer Submit: actually implement the documented fallback for the no-Program path instead of hard-failing on INTERNAL_CHECK_SPAN. ``self.<kernel>`` inside a Program; bare ``<kernel>`` (or the raw op name for non-GlobalVar callees) when invoked standalone. (gemini, copilot — python_printer.cpp:838) - ast_parser annotated assignment: replace the raw ``assert isinstance`` with a ``ParserSyntaxError`` carrying source-location context. (gemini — ast_parser.py:994) - SubmitToCallView: enforce the documented Var-like invariant on ``deps_`` entries. Null or non-Var/IterArg deps now throw ``pypto::TypeError`` at the view boundary instead of being silently dropped — losing a dep edge silently corrupts downstream codegen. (copilot — expr.h:918) - ExprContainsCall → ExprContainsCallLike (and CallFinder → CallLikeFinder): the predicate now returns true for Submit too; renaming keeps the API self-describing. (copilot — dead_code_elimination.cpp:346) - test_submit_printer test rename: ``test_submit_structural_equal_ignores_deps_order_independence`` → ``test_submit_structural_equal_is_deps_order_sensitive``. The old name contradicted the assertions, which check that swapped deps DO compare unequal. (copilot — test_submit_printer.py:143) - test_submit_passes docstring: drop the outdated ``parser does not yet emit Submit (Phase 3)`` line — the parser flip is part of this PR. (copilot — test_submit_passes.py:16) 4520 ir/parser/codegen tests pass.
- docs/en/dev/ir/01-hierarchy.md: align the older RuntimeScopeStmt paragraph with the Submit-first narrative — parser emits Submit with populated deps_; DeriveCallDirections lowers to Call.attrs ["manual_dep_edges"] for codegen. (coderabbit — 01-hierarchy.md:182) - SubmitToCallView: strip any pre-existing kAttrManualDepEdges from submit->attrs_ before adding the deps_-derived attr. Otherwise a stale attr entry would either duplicate the deps (when deps_ is populated) or survive as a stand-in (when deps_ is empty), making Submit::deps_ no longer the single source of truth. (coderabbit — expr.h:934) - test_submit_end_to_end: rename ``scratch, tid`` to ``_scratch, _tid`` so RUF059 stops flagging the deliberately-unused tuple unpack. (coderabbit — test_submit_end_to_end.py:59) Deferred Class D items (architectural — left open for human judgement on hw-native-sys#1544 as separate follow-ups): - ast_parser.py annotated-assignment Submit bypass — CodeRabbit asks to route through the normal override_type validation. Refactoring the helper to return an Expr that re-enters the annotated path touches the existing _parse_submit_assignment plumbing; deferred. - orchestration_codegen SubmitToCallView tuple-key loss — codegen is reached only post-DeriveCallDirections, so the IR has Call, not Submit; the defensive Submit branch is unreachable in the current pipeline. Worth keeping the defensive view but the tuple-key identity fix is unnecessary today. - derive_call_directions first-writer context for Submit — the first_writer_roots_ map is keyed by Call*; preserving Submit identity through the rewrite is a heavier change that affects the prior-writer collector. Deferred. 38 ir/printing/transforms tests still pass.
Introduces Submit as a first-class sibling of Call for representing pl.submit(...) task launches inside pl.manual_scope blocks. Submit has a typed deps_ field instead of stuffing TaskId vars into Call::attrs_[manual_dep_edges], so passes that walk uses see the cross-task dependencies through the normal use-def chain. Purely additive — no code emits Submit yet, no behaviour changes. Adds: - ObjectKind::Submit, KindTrait<Submit>, and Submit in KindTrait<Expr> - Submit class in expr.h mirroring Call's field layout plus deps_ - ExprFunctor::VisitExpr_(SubmitPtr) pure-virtual + dispatch - IRVisitor / IRMutator concrete defaults (walk / copy-on-write rewrite args + deps + arg-direction overrides) - Submit stubs in all five direct ExprFunctor subclasses under src/ir/arith/ - Python binding (nb::class_, def_prop_ro for kwargs/attrs/ arg_directions) and matching type stub in ir.pyi - .claude/rules/pass-submit-awareness.md describing the rule for pass authors to handle Submit alongside Call (paired with ir-kind-traits.md) The parser/printer/per-pass migration (deps_ as SSA-renamed uses, pl.submit print form, structural-equal/codegen consumers reading Submit::deps_ instead of the attrs key) lands in a follow-up commit. All 3469 IR/pass tests still pass on this branch.
Adds IRPythonPrinter::VisitExpr_(SubmitPtr) that emits the source-level ``pl.submit(self.<kernel>, arg1, arg2, ..., deps=[t1, t2])`` form for Submit nodes inside a Program context, matching the DSL syntax users write. Also surfaces ``attrs["arg_directions"]`` the same way Call does so post-DeriveCallDirections IR round-trips. The GlobalVarCollector helper used by the printer's topological sort now also walks Submit::op_ — cross-function deps through pl.submit are captured for ordering. Adds EQUAL_DISPATCH(Submit) in structural_equal.cpp and HASH_DISPATCH(Submit) in structural_hash.cpp. Since Submit's GetFieldDescriptors covers op/args/deps/attrs/kwargs, the existing reflection-based comparison and hashing work without further wiring; the dispatch entries just opt Submit into them. Tests (tests/ut/ir/printing/test_submit_printer.py, 6 cases): - pl.submit syntax round-trips inside a Program - deps=[...] kwarg renders correctly (bound params, multi-dep) - zero-arg Submit + deps still emits cleanly (no dangling comma) - structural_equal: Submit == itself; Submit != Call with same op/args; deps order matters 2129 IR tests pass (2123 prior + 6 new), no regressions. No code yet emits Submit, so this commit is still purely additive at the pipeline level.
Two scaffolding pieces let downstream Phase 3 / pass migrations flip the parser to emit Submit without earlier passes mis-handling it. DCE: - CallFinder also flags Submit so an expression containing a Submit is conservatively preserved. - IsSideEffectOp returns true for any AssignStmt / EvalStmt whose value is a Submit — a task launch is intrinsically side-effecting regardless of the kernel body. ConvertToSSA: - ExprSubstituter (the per-Call inner mutator) gains a Submit override that calls IRMutator's default (which already SSA-renames args_ and deps_ since both are reflection-walked fields) and then substitutes the return type the same way Call does. Submit does not carry manual_dep_edges in attrs, so the SubstCallAttrs branch is skipped. Tests (tests/ut/ir/transforms/test_submit_passes.py, 3 cases): - ConvertToSSA preserves Submit-ness across the pass. - Args / deps Var references on Submit get SSA-renamed (verifies the IRMutator base really does walk both fields). - Post-SSA program still prints as pl.submit(self.kernel, ...). The remaining per-pass migration (DeriveCallDirections, MaterializeTensorStrides, OptimizeOrchTensors, orchestration_codegen) bundles with Phase 3's parser flip — those passes will start seeing Submit only once the parser produces it. Until then this commit keeps the pipeline buildable and DCE/SSA correct on hand-built Submit IR. 3463 IR tests pass (3460 prior + 3 new), no regressions.
…llDirections
The DSL parser path for ``pl.submit(self.kernel, *args, deps=[...])`` now
builds an ``ir.Submit`` node with the typed ``deps_`` field instead of
an augmented ``ir.Call`` carrying ``attrs[manual_dep_edges]`` and a
TASK_ID-tail return type. Dumps captured before DeriveCallDirections
(roughly passes 0–33, including the user's original case after
InferTileMemorySpace at pass 18) now print the source-level
``pl.submit(self.kernel, arg1, arg2, deps=[t1, t2])`` form, making
task submission visually distinct from a plain function call.
To keep the existing Call-shaped codegen + late passes working
unchanged, DeriveCallDirections (pass 34) materializes the Submit →
Call view via the new ``SubmitToCallView`` adapter in expr.h:
``Submit::deps_`` is folded back into ``attrs[manual_dep_edges]`` on
the synthesised Call, and the standard direction-derivation logic
runs on the view. The result Call is what propagates downstream;
codegen, MaterializeStrides, OptimizeOrchTensors, etc. see Call as
before. The orchestration codegen AssignStmt visitor also funnels
any stray Submit through the same view as a defensive safety net.
Tests updated for the new IR shape:
- tests/ut/ir/transforms/test_flatten_call_expr_pass.py — the
``test_manual_dep_edges_survive_arg_flatten`` regression now checks
``submit.deps`` instead of ``call.attrs[manual_dep_edges]``, and
uses ``VerificationLevel.NONE`` because the single-LHS Submit IR
shape does not yet match the parser's required ``out, tid = ...``
2-tuple unpacking syntax for pl.submit.
- tests/ut/language/parser/test_task_id_dsl.py and
tests/ut/language/parser/test_manual_scope_parsing.py — the
``_calls_in`` / ``_all_calls`` helpers collect both Call and Submit
RHS values, and direct ``call.attrs.get("manual_dep_edges")`` reads
on kernel-call submits become ``submit.deps``. ``pl.at`` ScopeStmt
deps still use the attr (Submit replaces only the Call-side dep
encoding).
Full ``tests/ut/`` (excluding the pre-existing simpler-runtime-sync
failures in ``tests/ut/runtime``): 5040 passed, 0 failed.
…it dumps Documents Submit alongside Call in docs/en/dev/ir/01-hierarchy.md: - New row in the expression-nodes table. - New `Submit vs Call` section listing the semantic differences (sync vs async, attrs[manual_dep_edges] on ScopeStmt vs Submit::deps_, return-type augmentation, Python syntax) and noting that DeriveCallDirections is the lowering point. - Cross-link to `.claude/rules/pass-submit-awareness.md` so future pass authors see the dispatch rule alongside the IR docs. Adds tests/ut/ir/transforms/test_submit_end_to_end.py (2 cases): - `test_submit_visible_in_mid_pipeline_dump` exercises the user's original complaint — a mid-pipeline dump (post-Simplify, the same region as the InferTileMemorySpace dump that prompted this work) must show ``pl.submit(self.stage1, ...)`` and not the bare ``= self.stage1(...)`` legacy form. - `test_submit_with_deps_visible_in_mid_pipeline_dump` verifies the ``deps=[...]`` kwarg surfaces from the typed Submit::deps_ field. Both tests pass; full ``tests/ut/`` (excluding pre-existing simpler-runtime-sync failures) remains green at 5042 passing.
…ttrs
Closes the two known issues logged at the end of the Submit IR migration.
Parser (Issue 1): The printer for ``Submit`` emits ``res: pl.Tuple[...,
TASK_ID] = pl.submit(self.kernel, ...)`` when an ``AssignStmt``'s LHS is a
single Tuple-typed Var. The parser previously rejected this form,
requiring strict ``out, tid = pl.submit(...)`` unpacking, which broke
the default ``VerificationLevel.BASIC`` round-trip instrument for any
pass that produced a Submit on a single LHS. New
``_parse_submit_single_lhs`` handler accepts both the bare
``result = pl.submit(...)`` and the printer-emitted annotated
``res: pl.Tuple[..., TASK_ID] = pl.submit(...)`` form, binding the
whole flat ``Tuple{*<kernel results>, TaskId}`` to one Var (no
projection statements). The two-element unpacking form continues to
go through ``_parse_submit_assignment``. A bare ``pl.submit(...)``
expression outside any assignment is still rejected.
Visitor (Issue 2): ``IRVisitor::VisitExpr_(SubmitPtr)`` previously
walked only ``args_`` and ``deps_``. The matching ``IRMutator`` walks
and rewrites Vars under ``Submit::attrs_[arg_direction_overrides_vars]``
too, so visitor/mutator disagreed about whether those Vars are live —
a silent leak if any future pass or test fixture sets that key on a
Submit. The visitor now mirrors the mutator and walks those Vars,
keeping unused-var / def-use / SSA-liveness analyses consistent.
Test updates:
- ``test_submit_single_target_is_rejected`` -> renamed to
``test_submit_single_target_binds_full_tuple`` and inverted; it now
asserts the single-LHS form parses to a single-Var bind of the flat
tuple, with the trailing TASK_ID element preserved.
- ``test_submit_passes.py`` / ``test_submit_end_to_end.py`` /
``test_manual_dep_edges_survive_arg_flatten`` no longer set
``VerificationLevel.NONE`` — default ``BASIC`` verification (with
the RoundtripInstrument) now passes on Submit-bearing IR. New
``test_submit_single_lhs_form_round_trips`` regression guards
against the parser fix.
Removes the two corresponding entries from ``KNOWN_ISSUES.md``.
Full ``tests/ut/`` (excluding pre-existing simpler-runtime-sync
failures): 5043 passing.
- python_printer Submit: actually implement the documented fallback for the no-Program path instead of hard-failing on INTERNAL_CHECK_SPAN. ``self.<kernel>`` inside a Program; bare ``<kernel>`` (or the raw op name for non-GlobalVar callees) when invoked standalone. (gemini, copilot — python_printer.cpp:838) - ast_parser annotated assignment: replace the raw ``assert isinstance`` with a ``ParserSyntaxError`` carrying source-location context. (gemini — ast_parser.py:994) - SubmitToCallView: enforce the documented Var-like invariant on ``deps_`` entries. Null or non-Var/IterArg deps now throw ``pypto::TypeError`` at the view boundary instead of being silently dropped — losing a dep edge silently corrupts downstream codegen. (copilot — expr.h:918) - ExprContainsCall → ExprContainsCallLike (and CallFinder → CallLikeFinder): the predicate now returns true for Submit too; renaming keeps the API self-describing. (copilot — dead_code_elimination.cpp:346) - test_submit_printer test rename: ``test_submit_structural_equal_ignores_deps_order_independence`` → ``test_submit_structural_equal_is_deps_order_sensitive``. The old name contradicted the assertions, which check that swapped deps DO compare unequal. (copilot — test_submit_printer.py:143) - test_submit_passes docstring: drop the outdated ``parser does not yet emit Submit (Phase 3)`` line — the parser flip is part of this PR. (copilot — test_submit_passes.py:16) 4520 ir/parser/codegen tests pass.
- docs/en/dev/ir/01-hierarchy.md: align the older RuntimeScopeStmt paragraph with the Submit-first narrative — parser emits Submit with populated deps_; DeriveCallDirections lowers to Call.attrs ["manual_dep_edges"] for codegen. (coderabbit — 01-hierarchy.md:182) - SubmitToCallView: strip any pre-existing kAttrManualDepEdges from submit->attrs_ before adding the deps_-derived attr. Otherwise a stale attr entry would either duplicate the deps (when deps_ is populated) or survive as a stand-in (when deps_ is empty), making Submit::deps_ no longer the single source of truth. (coderabbit — expr.h:934) - test_submit_end_to_end: rename ``scratch, tid`` to ``_scratch, _tid`` so RUF059 stops flagging the deliberately-unused tuple unpack. (coderabbit — test_submit_end_to_end.py:59) Deferred Class D items (architectural — left open for human judgement on hw-native-sys#1544 as separate follow-ups): - ast_parser.py annotated-assignment Submit bypass — CodeRabbit asks to route through the normal override_type validation. Refactoring the helper to return an Expr that re-enters the annotated path touches the existing _parse_submit_assignment plumbing; deferred. - orchestration_codegen SubmitToCallView tuple-key loss — codegen is reached only post-DeriveCallDirections, so the IR has Call, not Submit; the defensive Submit branch is unreachable in the current pipeline. Worth keeping the defensive view but the tuple-key identity fix is unnecessary today. - derive_call_directions first-writer context for Submit — the first_writer_roots_ map is keyed by Call*; preserving Submit identity through the rewrite is a heavier change that affects the prior-writer collector. Deferred. 38 ir/printing/transforms tests still pass.
d5f23e1 to
210cce9
Compare
…xtend Submit to pl.at outliner Three correctness/architectural fixes for the Submit migration: 1. derive_call_directions: first-writer context for Submit (CodeRabbit `derive_call_directions_pass.cpp:422`). ``first_writer_roots_`` was keyed by ``const Call*``; when the Submit path lowered to a synthetic Call via ``SubmitToCallView``, the synthetic Call's pointer never matched what ``FirstWriterCollector::AnalyzeCall`` registered. The collector now also handles Submit (via SubmitToCallView for the args walk) and registers under the original Submit's pointer; the map's key type is widened from ``const Call*`` to ``const Expr*``. The mutator's Call path extracts an identity_key helper (``DeriveCallArgDirections``); the Submit override calls it with the original Submit's pointer so Out args get correct first-writer classification. 2. ast_parser: annotated-assignment Submit no longer bypasses override_type validation (CodeRabbit ``ast_parser.py:999``). ``_parse_submit_single_lhs`` now wraps a new ``_build_submit_single_lhs_expr`` helper that returns the Submit expression without binding. ``parse_annotated_assignment`` uses that helper to produce ``value_expr`` and then falls through to the standard annotation-consistency / override_type validation flow that every other RHS expression goes through. 3. OutlineIncoreScopes / OutlineClusterScopes / OutlineHierarchyScopes now emit ``ir.Submit`` (instead of an augmented ``ir.Call``) for ``with pl.at(...) as tid:`` scopes. ``scope_outline_utils.h`` builds the synthesised expression as ``Submit`` when ``scope_task_id_var`` is set, pulling ``Submit::deps_`` from the scope's ``kAttrManualDepEdges`` attr — so ``pl.at(..., deps=[t]) as tid`` prints as ``pl.submit(self.<outlined>, ..., deps=[t])`` in mid- pipeline dumps, consistent with explicit ``pl.submit(...)``. The no-TaskId path (plain ``pl.at(...):``) continues to emit a regular Call. DeriveCallDirections lowers both forms uniformly to Call for the late pipeline. 4541 ir/parser/codegen tests pass post-rebase onto upstream/main. The pl.at-as-tid lowering is verified by reproducing the user's example: ``with pl.at(level=pl.Level.CORE_GROUP, name_hint='s1') as s1_tid:`` now appears as ``ret__tmp_v0: pl.Tuple[..., TASK_ID] = pl.submit(self.s1, ...)`` in the post-OutlineIncoreScopes dump.
…sys#1544) ``pl.submit(*args, **kwargs) -> NoReturn`` in scope.py was mislabeled. The body does raise unconditionally (it's a parser construct, never called directly), but the SURFACE TYPE that user code sees after the parser intercepts the call is a 2-tuple ``(out, tid)`` (or ``((a, b), tid)`` for multi-output kernels). ``NoReturn`` is reserved for functions that never return normally — under that annotation, type checkers would reject every downstream ``out, tid = pl.submit(...)`` unpack site as dead code. Switch to ``-> Any`` so the parser-interception story stays correct for the user. The body keeps the existing ``raise RuntimeError(...)`` guard so a misuse outside ``@pl.function`` still fails loudly at runtime. Docstring updated to call this out, and to mention the single-LHS ``res = pl.submit(...)`` form added earlier in this PR. 723 parser tests pass.
Summary
Adds
Submitas a first-class IR kind sibling toCall, so apl.submit(self.kernel, ..., deps=[t])task launch is visually distinct from a plain function call in IR dumps. Mid-pipeline dumps (the user-facing reading point —passes_dump/19_after_InferTileMemorySpace.pyand friends) now render task submissions aspl.submit(self.stage1, ..., deps=[a_tid])instead of the bareself.stage1(...)form that conflated them with synchronous calls.The migration:
Submitclass) —op_,args_, typeddeps_field (replacesCall::attrs[manual_dep_edges]on the call side),attrs_,kwargs_.ObjectKind::Submit,KindTrait<Submit>,ExprFunctor::VisitExpr_(SubmitPtr). Concrete defaults inIRVisitor/IRMutatorwalk + rewrite args/deps/attrs.pl.submit(...)now emitsir.Submitdirectly. Both the unpacked formout, tid = pl.submit(...)and the printer's round-trip-friendly single-LHS formres: pl.Tuple[..., TASK_ID] = pl.submit(...)are accepted.IRPythonPrinter::VisitExpr_(SubmitPtr)emitspl.submit(self.<kernel>, args, deps=[...]).structural_equal/structural_hashgetSubmitdispatch via the reflection-based comparator usingGetFieldDescriptors.DeriveCallDirections(pass 34) materializesSubmit → Callvia the newSubmitToCallViewadapter:Submit::deps_is folded back intoattrs[manual_dep_edges]on the synthesised Call, the standard direction-derivation runs, and the result Call replaces the Submit in the IR. Codegen and every pass numbered ≥34 are unchanged — they still operate on Call exactly as before. Submit only lives for dumps 0–33, which is the user-relevant window.AssignStmtwhose value is a Submit is preserved (task launches are side-effecting); SSA renames Submit'sargsand typeddeps_field through theIRMutatordefault plus a small override that also remaps the return type.A new
.claude/rules/pass-submit-awareness.mdrule pairs withir-kind-traits.mdand warns pass authors to dispatch on Submit alongside Call.Files
C++ headers / impls (12):
include/pypto/ir/{core.h, expr.h, kind_traits.h},include/pypto/ir/transforms/base/{functor.h, visitor.h, mutator.h},src/ir/transforms/{visitor.cpp, mutator.cpp, python_printer.cpp, structural_equal.cpp, structural_hash.cpp, derive_call_directions_pass.cpp, convert_to_ssa_pass.cpp, utils/dead_code_elimination.cpp},src/ir/arith/{int_set, const_int_bound, modular_set, canonical_simplify, rewrite_simplify}.{cpp,h},src/codegen/orchestration/orchestration_codegen.cppPython bindings / type stubs / parser (3):
python/bindings/modules/ir.cpp,python/pypto/pypto_core/ir.pyi,python/pypto/language/parser/ast_parser.pyTests (5 files, 12 new test cases):
tests/ut/ir/printing/test_submit_printer.py,tests/ut/ir/transforms/test_submit_passes.py,tests/ut/ir/transforms/test_submit_end_to_end.py, plus updates totests/ut/ir/transforms/test_flatten_call_expr_pass.pyandtests/ut/language/parser/{test_manual_scope_parsing.py, test_task_id_dsl.py}so the test helpers (_calls_in,_all_calls) collect bothCallandSubmitRHS values and dep-probing assertions usesubmit.depsinstead ofcall.attrs[manual_dep_edges].Docs (2):
docs/en/dev/ir/01-hierarchy.md(Submit row + newSubmit vs Callsection);.claude/rules/pass-submit-awareness.md(new rule).Test plan
tests/ut/cases pass (excluding pre-existing simpler-runtime-sync failures intests/ut/runtime— unrelated, on baseline)test_submit_printer.py— 6 cases: pl.submit syntax inside Program,deps=[...]kwarg, zero-arg + deps, Submit ≠ Call structurally, deps order matterstest_submit_passes.py— 4 cases: SSA preserves Submit kind, SSA renamesargsanddeps_, post-SSA still printspl.submit, single-LHS round-trip with defaultVerificationLevel.BASICtest_submit_end_to_end.py— 2 cases:pl.submitvisible in mid-pipeline dump (the user's original complaint),deps=[...]kwarg surfaces from typedSubmit::deps_VerificationLevel.BASICRoundtripInstrument passes — print→re-parse→structural_equal works for Submit-bearing IRTestManualScopeCodegen) still pass — codegen sees Call after the DeriveCallDirections loweringclang-format/cpplint/ruff/pyright/markdownlintpre-commit hooks all green on each commit🤖 Generated with Claude Code