Skip to content

feat(ir): Add Submit IR node — print task launches as pl.submit(...)#1544

Open
Hzfengsy wants to merge 10 commits into
hw-native-sys:mainfrom
Hzfengsy:worktree-submit-print
Open

feat(ir): Add Submit IR node — print task launches as pl.submit(...)#1544
Hzfengsy wants to merge 10 commits into
hw-native-sys:mainfrom
Hzfengsy:worktree-submit-print

Conversation

@Hzfengsy
Copy link
Copy Markdown
Member

Summary

Adds Submit as a first-class IR kind sibling to Call, so a pl.submit(self.kernel, ..., deps=[t]) task launch is visually distinct from a plain function call in IR dumps. Mid-pipeline dumps (the user-facing reading point — passes_dump/19_after_InferTileMemorySpace.py and friends) now render task submissions as pl.submit(self.stage1, ..., deps=[a_tid]) instead of the bare self.stage1(...) form that conflated them with synchronous calls.

The migration:

  • IR (Submit class)op_, args_, typed deps_ field (replaces Call::attrs[manual_dep_edges] on the call side), attrs_, kwargs_. ObjectKind::Submit, KindTrait<Submit>, ExprFunctor::VisitExpr_(SubmitPtr). Concrete defaults in IRVisitor / IRMutator walk + rewrite args/deps/attrs.
  • Parserpl.submit(...) now emits ir.Submit directly. Both the unpacked form out, tid = pl.submit(...) and the printer's round-trip-friendly single-LHS form res: pl.Tuple[..., TASK_ID] = pl.submit(...) are accepted.
  • PrinterIRPythonPrinter::VisitExpr_(SubmitPtr) emits pl.submit(self.<kernel>, args, deps=[...]). structural_equal / structural_hash get Submit dispatch via the reflection-based comparator using GetFieldDescriptors.
  • LoweringDeriveCallDirections (pass 34) materializes Submit → Call via the new SubmitToCallView adapter: Submit::deps_ is folded back into attrs[manual_dep_edges] on the synthesised Call, the standard direction-derivation runs, and the result Call replaces the Submit in the IR. Codegen and every pass numbered ≥34 are unchanged — they still operate on Call exactly as before. Submit only lives for dumps 0–33, which is the user-relevant window.
  • DCE / SSA — Submit-aware: an AssignStmt whose value is a Submit is preserved (task launches are side-effecting); SSA renames Submit's args and typed deps_ field through the IRMutator default plus a small override that also remaps the return type.

A new .claude/rules/pass-submit-awareness.md rule pairs with ir-kind-traits.md and warns pass authors to dispatch on Submit alongside Call.

Files

C++ headers / impls (12): include/pypto/ir/{core.h, expr.h, kind_traits.h}, include/pypto/ir/transforms/base/{functor.h, visitor.h, mutator.h}, src/ir/transforms/{visitor.cpp, mutator.cpp, python_printer.cpp, structural_equal.cpp, structural_hash.cpp, derive_call_directions_pass.cpp, convert_to_ssa_pass.cpp, utils/dead_code_elimination.cpp}, src/ir/arith/{int_set, const_int_bound, modular_set, canonical_simplify, rewrite_simplify}.{cpp,h}, src/codegen/orchestration/orchestration_codegen.cpp

Python bindings / type stubs / parser (3): python/bindings/modules/ir.cpp, python/pypto/pypto_core/ir.pyi, python/pypto/language/parser/ast_parser.py

Tests (5 files, 12 new test cases): tests/ut/ir/printing/test_submit_printer.py, tests/ut/ir/transforms/test_submit_passes.py, tests/ut/ir/transforms/test_submit_end_to_end.py, plus updates to tests/ut/ir/transforms/test_flatten_call_expr_pass.py and tests/ut/language/parser/{test_manual_scope_parsing.py, test_task_id_dsl.py} so the test helpers (_calls_in, _all_calls) collect both Call and Submit RHS values and dep-probing assertions use submit.deps instead of call.attrs[manual_dep_edges].

Docs (2): docs/en/dev/ir/01-hierarchy.md (Submit row + new Submit vs Call section); .claude/rules/pass-submit-awareness.md (new rule).

Test plan

  • All 5043 tests/ut/ cases pass (excluding pre-existing simpler-runtime-sync failures in tests/ut/runtime — unrelated, on baseline)
  • New per-feature tests:
    • test_submit_printer.py — 6 cases: pl.submit syntax inside Program, deps=[...] kwarg, zero-arg + deps, Submit ≠ Call structurally, deps order matters
    • test_submit_passes.py — 4 cases: SSA preserves Submit kind, SSA renames args and deps_, post-SSA still prints pl.submit, single-LHS round-trip with default VerificationLevel.BASIC
    • test_submit_end_to_end.py — 2 cases: pl.submit visible in mid-pipeline dump (the user's original complaint), deps=[...] kwarg surfaces from typed Submit::deps_
  • Default VerificationLevel.BASIC RoundtripInstrument passes — print→re-parse→structural_equal works for Submit-bearing IR
  • Existing orchestration codegen tests (TestManualScopeCodegen) still pass — codegen sees Call after the DeriveCallDirections lowering
  • clang-format / cpplint / ruff / pyright / markdownlint pre-commit hooks all green on each commit

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings May 26, 2026 15:12
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 26, 2026

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR introduces Submit as a first-class IR expression kind representing async task launches from pl.submit(...) within pl.manual_scope. The change adds a new IR node with explicit deps_ field for cross-task dependencies, integrates it across visitor/mutator infrastructure, updates the parser to emit Submit nodes directly, implements pass handlers throughout the compiler pipeline, and provides comprehensive test coverage.

Changes

Submit IR node introduction

Layer / File(s) Summary
IR node kind definition and core types
include/pypto/ir/core.h, include/pypto/ir/expr.h, include/pypto/ir/kind_traits.h
ObjectKind gains Submit; new Submit expression stores op, args, explicit deps_, kwargs/attrs; KindTrait<Submit> added and KindTrait<Expr>::count updated; SubmitToCallView adapter synthesizes a Call view with manual_dep_edges for compatibility.
Visitor and functor dispatch infrastructure
include/pypto/ir/transforms/base/functor.h, include/pypto/ir/transforms/base/visitor.h, include/pypto/ir/transforms/base/mutator.h, src/ir/arith/canonical_simplify.h
Added VisitExpr_(const SubmitPtr&) hooks and dispatch entries so ExprFunctor/IRVisitor/IRMutator and related simplifiers dispatch Submit to dedicated handlers.
AST parser Submit handling
python/pypto/language/parser/ast_parser.py
Parser now recognizes single-LHS pl.submit(...) forms and constructs ir.Submit directly (augmented return type including TASK_ID, carries deps_list, strips manual_dep_edges from attrs); _parse_submit_single_lhs helper added.
Transformation pass Submit handlers
src/ir/transforms/visitor.cpp, src/ir/transforms/mutator.cpp, src/ir/transforms/convert_to_ssa_pass.cpp, src/ir/transforms/derive_call_directions_pass.cpp, src/ir/transforms/utils/dead_code_elimination.cpp, src/codegen/orchestration/orchestration_codegen.cpp, src/ir/transforms/structural_equal.cpp, src/ir/transforms/structural_hash.cpp, src/ir/arith/*
Multiple passes implement VisitExpr_(SubmitPtr): visitor walks args/deps/var-attrs; mutator remaps args/deps/type and arg-direction override vars; SSA substitutes types and preserves node kind; derive-call-directions converts Submit to Call-view for direction derivation; DCE treats Submit as side-effecting and adds CallLikeFinder; arithmetic analyses conservatively return unconstrained results; orchestration codegen funnels Submit through call-shaped codepath; structural equal/hash include Submit.
Python API bindings and type stubs
python/bindings/modules/ir.cpp, python/pypto/pypto_core/ir.pyi
Expose ir.Submit to Python (nanobind) with constructors accepting op, args, deps, type/span and overload with kwargs/attrs; read-only properties kwargs, attrs, arg_directions; type stubs updated.
Python printer and function dependency collection
src/ir/transforms/python_printer.cpp
Printer prints pl.submit(self.<kernel>, <args>, deps=[...], attrs={"arg_directions": [...]}) when round-trippable; GlobalVarCollector collects submit callees for topological analysis.
Developer guidance for pass authors
.claude/rules/pass-submit-awareness.md, docs/en/dev/ir/01-hierarchy.md
Adds documentation explaining Submit vs Call: semantics, return-type TASK_ID augmentation, deps_ as SSA uses, required pass handling (visit both kinds, preserve Submit, normalize TASK_ID when comparing return types), and verifier guidance.
Test coverage for Submit IR node and passes
tests/ut/ir/printing/test_submit_printer.py, tests/ut/ir/transforms/test_submit_passes.py, tests/ut/ir/transforms/test_submit_end_to_end.py, tests/ut/ir/transforms/test_flatten_call_expr_pass.py, tests/ut/language/parser/test_manual_scope_parsing.py, tests/ut/language/parser/test_task_id_dsl.py
Adds printer/unit/integration tests validating Submit printing, deps kwarg emission, structural equality/hash behavior, SSA preservation and renaming of args/deps, mid-pipeline visibility, parser round-trips, and updates parser tests to read deps from *.deps instead of attrs["manual_dep_edges"].

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

  • hw-native-sys/pypto#1516: Modifies pl.submit/pl.at parsing and deps handling via ir.Submit and deps_ plumbing in the same parser and test files.

Suggested reviewers

  • lyfne123

Poem

🐰 Hops through the IR with Submit's new grace,
Async tasks launched in their rightful place,
Deps flow like carrots, structured and true,
Visitors and mutators now know what to do—
A sibling to Call, forever to stay! 🥕✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 38.13% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'feat(ir): Add Submit IR node — print task launches as pl.submit(...)' directly and clearly describes the primary change: introducing a new Submit IR kind and its printing behavior.
Description check ✅ Passed The description is comprehensive and directly related to the changeset, explaining the Submit IR kind, parser/printer integration, lowering strategy, and test coverage.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new first-class IR node kind Submit to represent asynchronous task launches (pl.submit(...) inside pl.manual_scope), separating them structurally from plain synchronous Call nodes. The changes span the C++ IR definitions, AST parser, Python printer, SSA conversion, dead code elimination, Python bindings, and type stubs, alongside comprehensive unit and integration tests. The review feedback highlights two key improvements: addressing a contradiction in the Python printer where a null current_program_ would trigger an abort instead of falling back to a safe printing mode, and replacing a raw assert in the AST parser with a user-friendly ParserSyntaxError when parsing annotated assignments of pl.submit.

Comment thread src/ir/transforms/python_printer.cpp Outdated
Comment thread python/pypto/language/parser/ast_parser.py Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a first-class Submit IR expression kind to represent pl.submit(...) task launches distinctly from plain synchronous Call, improving IR dumps and enabling submit-aware analyses/passes while preserving the existing late-pipeline/codegen Call-based behavior via a Submit → Call lowering in DeriveCallDirections.

Changes:

  • Add ir.Submit across core IR (new ObjectKind, kind traits, reflection descriptors) and wire it through visitors/mutators, structural equality/hash, DCE, SSA, and arithmetic analyzers.
  • Update the Python DSL parser and IR Python printer to round-trip pl.submit(...), including the printer’s single-LHS tuple-typed form.
  • Add/adjust unit and integration tests and update IR documentation + introduce a .claude rule for pass authors to handle Submit alongside Call.

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tests/ut/language/parser/test_task_id_dsl.py Update test helpers/assertions to treat Submit as call-like and use submit.deps.
tests/ut/language/parser/test_manual_scope_parsing.py Broaden call collection to include Submit; add coverage for single-LHS submit binding; switch dep assertions to submit.deps.
tests/ut/ir/transforms/test_submit_passes.py New tests validating SSA + round-trip behavior on Submit.
tests/ut/ir/transforms/test_submit_end_to_end.py New integration tests ensuring mid-pipeline dumps show pl.submit(...).
tests/ut/ir/transforms/test_flatten_call_expr_pass.py Ensure flattening preserves Submit.deps when rewriting submit args.
tests/ut/ir/printing/test_submit_printer.py New tests for printing and structural equality/hash behavior of Submit.
src/ir/transforms/visitor.cpp Add IRVisitor traversal of Submit args, deps, and certain var-typed attrs.
src/ir/transforms/utils/dead_code_elimination.cpp Treat Submit as side-effecting for DCE; ensure expressions containing Submit are preserved.
src/ir/transforms/structural_hash.cpp Add structural hashing dispatch for Submit.
src/ir/transforms/structural_equal.cpp Add structural equality dispatch for Submit.
src/ir/transforms/python_printer.cpp Print Submit as pl.submit(self.<kernel>, ..., deps=[...]) and collect GlobalVars through Submit.
src/ir/transforms/mutator.cpp Add IRMutator support to rewrite Submit (args/deps/type and selected attrs).
src/ir/transforms/derive_call_directions_pass.cpp Lower Submit → Call in DeriveCallDirections via SubmitToCallView.
src/ir/transforms/convert_to_ssa_pass.cpp Ensure SSA conversion also rewrites Submit return type appropriately.
src/ir/arith/rewrite_simplify.h Add Submit visitor hook to rewrite simplifier.
src/ir/arith/rewrite_simplify.cpp Treat Submit as a leaf for rewrite simplifier.
src/ir/arith/modular_set.cpp Treat Submit conservatively in modular set analyzer.
src/ir/arith/int_set.cpp Treat Submit conservatively in int set analyzer.
src/ir/arith/const_int_bound.cpp Treat Submit conservatively in const-int bound analyzer.
src/ir/arith/canonical_simplify.h Add Submit visitor hook to canonical simplifier.
src/ir/arith/canonical_simplify.cpp Treat Submit as a leaf for canonical simplifier.
src/codegen/orchestration/orchestration_codegen.cpp Funnel Submit through existing codegen via SubmitToCallView.
python/pypto/pypto_core/ir.pyi Add Submit stub API (op/args/deps/kwargs/attrs/arg_directions).
python/pypto/language/parser/ast_parser.py Parse pl.submit(...) into ir.Submit; accept single-LHS printed form; move deps out of attrs into deps_.
python/bindings/modules/ir.cpp Bind Submit into Python IR module with kwargs/attrs/arg_directions accessors.
include/pypto/ir/transforms/base/visitor.h Add Submit dispatch in base visitor interface.
include/pypto/ir/transforms/base/mutator.h Add Submit dispatch in base mutator interface.
include/pypto/ir/transforms/base/functor.h Add Submit dispatch in ExprFunctor + dispatch table.
include/pypto/ir/kind_traits.h Register Submit kind trait and include it under Expr kinds.
include/pypto/ir/expr.h Define Submit IR node and the SubmitToCallView adapter.
include/pypto/ir/core.h Add ObjectKind::Submit.
docs/en/dev/ir/01-hierarchy.md Document Submit in hierarchy and add Submit vs Call section.
.claude/rules/pass-submit-awareness.md New rule documenting how passes must handle Submit vs Call.

Comment thread include/pypto/ir/expr.h Outdated
Comment thread src/ir/transforms/python_printer.cpp Outdated
Comment thread tests/ut/ir/transforms/test_submit_passes.py
Comment thread tests/ut/ir/printing/test_submit_printer.py Outdated
Comment thread src/ir/transforms/utils/dead_code_elimination.cpp Outdated
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🧹 Nitpick comments (1)
tests/ut/ir/transforms/test_submit_passes.py (1)

92-110: ⚡ Quick win

Make the deps-renaming assertion match the test name.

test_ssa_renames_submit_args_and_deps currently verifies arg SSA rewrite only. Either add a deps rewrite assertion (by forcing a deps var rewrite) or rename the test to avoid over-claiming coverage.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/ir/transforms/test_submit_passes.py` around lines 92 - 110, The test
name claims both args and deps are SSA-renamed but only asserts args; either
update the test to assert deps were renamed or rename the test to match current
behavior. To fix in test_ssa_renames_submit_args_and_deps, either (A) ensure
_build_program_with_submit produces a deps Var that gets rewritten and add an
assertion like checking submit_after.deps[i] is an ir.Var and is not
caller_after.params[j] (use submit_after.deps and caller_after.params to locate
symbols), or (B) rename the test to something like test_ssa_renames_submit_args
to reflect it only verifies args; keep references to _build_program_with_submit
and passes.convert_to_ssa to locate the setup and transformation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/en/dev/ir/01-hierarchy.md`:
- Around line 156-182: Update the RuntimeScope/dep-encoding paragraph to match
the Submit-first flow: state that the parser emits Submit nodes with a
first-class deps_ field (Submit.deps_) and that DeriveCallDirections lowers
Submit by folding deps_ into Call.attrs["manual_dep_edges"], so later passes and
runtime see Call with attrs["manual_dep_edges"]. Mention Submit, Submit.deps_,
DeriveCallDirections and Call.attrs["manual_dep_edges"] explicitly so the doc
consistently describes Submit-first encoding folded to Call during lowering.

In `@include/pypto/ir/expr.h`:
- Around line 905-923: SubmitToCallView currently copies submit->attrs_
unchanged which preserves any existing kAttrManualDepEdges when submit->deps_ is
empty and it silently drops non-Var-like entries in submit->deps_; update
SubmitToCallView to (1) remove/clear any existing kAttrManualDepEdges from attrs
when submit->deps_ is empty (so synthesized Call never inherits stale manual-dep
metadata) and (2) validate entries in submit->deps_ instead of quietly skipping
non-Var-like objects: iterate submit->deps_ and if an entry is null or its kind
is neither ObjectKind::Var nor ObjectKind::IterArg, surface an error (throw or
LOG and return) rather than ignoring it; continue to convert valid Var/IterArg
entries to VarPtr and call WithManualDepEdgesAttr(attrs, dep_vars) when deps are
present. Ensure you reference and modify the existing symbols SubmitToCallView,
attrs, submit->deps_, kAttrManualDepEdges, WithManualDepEdgesAttr, and Call
construction so the new behavior prevents stale attrs and dropped deps.

In `@python/pypto/language/parser/ast_parser.py`:
- Around line 987-994: The fast-path in the AST parser currently short-circuits
annotated-assignment validation for single-LHS pl.submit calls by calling
_parse_submit_single_lhs and returning; instead, modify _parse_submit_single_lhs
(or create a new helper) to return the constructed ir.Submit node and then let
the normal annotated-assignment binding/override_type validation flow handle it
so annotations are validated; update the call site (the block using _is_pl_call
and _parse_submit_single_lhs) to receive the ir.Submit result and proceed
through the same annotated-assignment/override_type path used for other RHS
expressions (the same logic used elsewhere in the function and mirrored in the
other occurrence around lines ~1363-1395).

In `@src/codegen/orchestration/orchestration_codegen.cpp`:
- Around line 892-899: The synthetic Call created by SubmitToCallView loses the
original Submit* identity so tuple-key lookups (used by
GenerateSubmitReturnAliases) miss entries; preserve the original node identity
by propagating the Submit pointer into the Call view or by recording a mapping
from the synthetic Call to the original Submit. Concretely, update
SubmitToCallView to attach the original Submit* (e.g., a member like
original_submit_ or a side-map from Call* -> Submit*) and change the tuple-key
lookup path (the code that inspects assign->value_ and later lookup logic used
by GenerateSubmitReturnAliases) to consult that original_submit_ mapping when
present so alias bookkeeping uses the Submit* key instead of the synthetic
Call*.

In `@src/ir/transforms/derive_call_directions_pass.cpp`:
- Around line 409-422: The Submit→Call lowering in VisitExpr_(const SubmitPtr&
op) loses original expression identity so first_writer_roots_ (which is keyed by
Call*) can't find Submit-originated first-writer roots, causing Out args to be
misclassified as InOut; fix this by preserving the original Submit identity
before calling VisitExpr_(view_call) — e.g., capture the original submit pointer
(op or submit.get()) and register a temporary mapping from the synthesized Call
pointer (view_call.get()) back to the original submit identity in
first_writer_roots_ (or adjust first_writer_roots_ to accept an alternative key
sourced from Submit) so the direction derivation code (VisitExpr_,
SubmitToCallView, first_writer_roots_) can look up the correct first-writer
context and keep Out arguments as first-writers.

In `@src/ir/transforms/python_printer.cpp`:
- Around line 827-869: The VisitExpr_ for Submit currently hard-fails when
current_program_ or As<GlobalVar> is missing; change it to detect whether the
callee is a GlobalVar inside a Program and only use the "self.<kernel>" form in
that case, otherwise fall back to a generic callee emission so
Print(Function)/Print(Expr) won't crash: in IRPythonPrinter::VisitExpr_(const
SubmitPtr& op) compute gvar = As<GlobalVar>(op->op_) and if (gvar &&
current_program_) emit ".submit(self.<kernel_name>...)" as now, else emit
".submit(" then VisitExpr(op->op_) (or otherwise print the op->op_ generically)
followed by the same args, deps and attrs logic; remove or replace the
INTERNAL_CHECK_SPAN that enforces current_program_ so the fallback path is used
instead of hard-failing.

In `@tests/ut/ir/transforms/test_submit_end_to_end.py`:
- Line 59: Rename the unused unpacked variables from pl.submit(self.stage1, x,
out) — currently named scratch and tid — to underscore-prefixed names (e.g.,
_scratch and _tid or just _) to silence RUF059; locate the call to pl.submit in
test_submit_end_to_end.py (the line invoking pl.submit(self.stage1, x, out)) and
change the left-hand unpacking identifiers accordingly so the test behavior
remains identical while avoiding lint warnings.

In `@tests/ut/ir/transforms/test_submit_passes.py`:
- Around line 12-15: Update the module docstring in the test_submit_passes.py
test module: replace or remove the stale sentence saying “parser does not yet
emit Submit” so it correctly reflects that the parser now emits Submit (e.g.,
state that tests construct Submit-bearing IR to verify passes handle
parser-emitted Submit) and ensure the docstring mentions why the tests exist
(preserve Submit structural shape) rather than claiming the parser can’t emit
it.

---

Nitpick comments:
In `@tests/ut/ir/transforms/test_submit_passes.py`:
- Around line 92-110: The test name claims both args and deps are SSA-renamed
but only asserts args; either update the test to assert deps were renamed or
rename the test to match current behavior. To fix in
test_ssa_renames_submit_args_and_deps, either (A) ensure
_build_program_with_submit produces a deps Var that gets rewritten and add an
assertion like checking submit_after.deps[i] is an ir.Var and is not
caller_after.params[j] (use submit_after.deps and caller_after.params to locate
symbols), or (B) rename the test to something like test_ssa_renames_submit_args
to reflect it only verifies args; keep references to _build_program_with_submit
and passes.convert_to_ssa to locate the setup and transformation.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 1ae4a749-82b8-42a2-941a-b378f45cbe58

📥 Commits

Reviewing files that changed from the base of the PR and between 1bb34f4 and 0877440.

📒 Files selected for processing (33)
  • .claude/rules/pass-submit-awareness.md
  • docs/en/dev/ir/01-hierarchy.md
  • include/pypto/ir/core.h
  • include/pypto/ir/expr.h
  • include/pypto/ir/kind_traits.h
  • include/pypto/ir/transforms/base/functor.h
  • include/pypto/ir/transforms/base/mutator.h
  • include/pypto/ir/transforms/base/visitor.h
  • python/bindings/modules/ir.cpp
  • python/pypto/language/parser/ast_parser.py
  • python/pypto/pypto_core/ir.pyi
  • src/codegen/orchestration/orchestration_codegen.cpp
  • src/ir/arith/canonical_simplify.cpp
  • src/ir/arith/canonical_simplify.h
  • src/ir/arith/const_int_bound.cpp
  • src/ir/arith/int_set.cpp
  • src/ir/arith/modular_set.cpp
  • src/ir/arith/rewrite_simplify.cpp
  • src/ir/arith/rewrite_simplify.h
  • src/ir/transforms/convert_to_ssa_pass.cpp
  • src/ir/transforms/derive_call_directions_pass.cpp
  • src/ir/transforms/mutator.cpp
  • src/ir/transforms/python_printer.cpp
  • src/ir/transforms/structural_equal.cpp
  • src/ir/transforms/structural_hash.cpp
  • src/ir/transforms/utils/dead_code_elimination.cpp
  • src/ir/transforms/visitor.cpp
  • tests/ut/ir/printing/test_submit_printer.py
  • tests/ut/ir/transforms/test_flatten_call_expr_pass.py
  • tests/ut/ir/transforms/test_submit_end_to_end.py
  • tests/ut/ir/transforms/test_submit_passes.py
  • tests/ut/language/parser/test_manual_scope_parsing.py
  • tests/ut/language/parser/test_task_id_dsl.py

Comment thread docs/en/dev/ir/01-hierarchy.md
Comment thread include/pypto/ir/expr.h
Comment thread python/pypto/language/parser/ast_parser.py Outdated
Comment thread src/codegen/orchestration/orchestration_codegen.cpp
Comment thread src/ir/transforms/derive_call_directions_pass.cpp
Comment thread src/ir/transforms/python_printer.cpp
Comment thread tests/ut/ir/transforms/test_submit_end_to_end.py Outdated
Comment thread tests/ut/ir/transforms/test_submit_passes.py Outdated
Hzfengsy added a commit to Hzfengsy/pypto that referenced this pull request May 26, 2026
- python_printer Submit: actually implement the documented fallback for
  the no-Program path instead of hard-failing on INTERNAL_CHECK_SPAN.
  ``self.<kernel>`` inside a Program; bare ``<kernel>`` (or the raw op
  name for non-GlobalVar callees) when invoked standalone. (gemini,
  copilot — python_printer.cpp:838)
- ast_parser annotated assignment: replace the raw ``assert isinstance``
  with a ``ParserSyntaxError`` carrying source-location context.
  (gemini — ast_parser.py:994)
- SubmitToCallView: enforce the documented Var-like invariant on
  ``deps_`` entries. Null or non-Var/IterArg deps now throw
  ``pypto::TypeError`` at the view boundary instead of being silently
  dropped — losing a dep edge silently corrupts downstream codegen.
  (copilot — expr.h:918)
- ExprContainsCall → ExprContainsCallLike (and CallFinder →
  CallLikeFinder): the predicate now returns true for Submit too;
  renaming keeps the API self-describing. (copilot —
  dead_code_elimination.cpp:346)
- test_submit_printer test rename:
  ``test_submit_structural_equal_ignores_deps_order_independence`` →
  ``test_submit_structural_equal_is_deps_order_sensitive``. The old name
  contradicted the assertions, which check that swapped deps DO compare
  unequal. (copilot — test_submit_printer.py:143)
- test_submit_passes docstring: drop the outdated ``parser does not yet
  emit Submit (Phase 3)`` line — the parser flip is part of this PR.
  (copilot — test_submit_passes.py:16)

4520 ir/parser/codegen tests pass.
Hzfengsy added a commit to Hzfengsy/pypto that referenced this pull request May 26, 2026
- docs/en/dev/ir/01-hierarchy.md: align the older RuntimeScopeStmt
  paragraph with the Submit-first narrative — parser emits Submit with
  populated deps_; DeriveCallDirections lowers to Call.attrs
  ["manual_dep_edges"] for codegen. (coderabbit — 01-hierarchy.md:182)
- SubmitToCallView: strip any pre-existing kAttrManualDepEdges from
  submit->attrs_ before adding the deps_-derived attr. Otherwise a
  stale attr entry would either duplicate the deps (when deps_ is
  populated) or survive as a stand-in (when deps_ is empty), making
  Submit::deps_ no longer the single source of truth. (coderabbit —
  expr.h:934)
- test_submit_end_to_end: rename ``scratch, tid`` to ``_scratch, _tid``
  so RUF059 stops flagging the deliberately-unused tuple unpack.
  (coderabbit — test_submit_end_to_end.py:59)

Deferred Class D items (architectural — left open for human judgement
on hw-native-sys#1544 as separate follow-ups):
- ast_parser.py annotated-assignment Submit bypass — CodeRabbit asks
  to route through the normal override_type validation. Refactoring
  the helper to return an Expr that re-enters the annotated path
  touches the existing _parse_submit_assignment plumbing; deferred.
- orchestration_codegen SubmitToCallView tuple-key loss — codegen is
  reached only post-DeriveCallDirections, so the IR has Call, not
  Submit; the defensive Submit branch is unreachable in the current
  pipeline. Worth keeping the defensive view but the tuple-key
  identity fix is unnecessary today.
- derive_call_directions first-writer context for Submit — the
  first_writer_roots_ map is keyed by Call*; preserving Submit
  identity through the rewrite is a heavier change that affects the
  prior-writer collector. Deferred.

38 ir/printing/transforms tests still pass.
Hzfengsy added 8 commits May 27, 2026 00:00
Introduces Submit as a first-class sibling of Call for representing
pl.submit(...) task launches inside pl.manual_scope blocks. Submit has
a typed deps_ field instead of stuffing TaskId vars into
Call::attrs_[manual_dep_edges], so passes that walk uses see the
cross-task dependencies through the normal use-def chain.

Purely additive — no code emits Submit yet, no behaviour changes.
Adds:
- ObjectKind::Submit, KindTrait<Submit>, and Submit in KindTrait<Expr>
- Submit class in expr.h mirroring Call's field layout plus deps_
- ExprFunctor::VisitExpr_(SubmitPtr) pure-virtual + dispatch
- IRVisitor / IRMutator concrete defaults (walk / copy-on-write
  rewrite args + deps + arg-direction overrides)
- Submit stubs in all five direct ExprFunctor subclasses under
  src/ir/arith/
- Python binding (nb::class_, def_prop_ro for kwargs/attrs/
  arg_directions) and matching type stub in ir.pyi
- .claude/rules/pass-submit-awareness.md describing the rule for
  pass authors to handle Submit alongside Call (paired with
  ir-kind-traits.md)

The parser/printer/per-pass migration (deps_ as SSA-renamed uses,
pl.submit print form, structural-equal/codegen consumers reading
Submit::deps_ instead of the attrs key) lands in a follow-up commit.

All 3469 IR/pass tests still pass on this branch.
Adds IRPythonPrinter::VisitExpr_(SubmitPtr) that emits the source-level
``pl.submit(self.<kernel>, arg1, arg2, ..., deps=[t1, t2])`` form for
Submit nodes inside a Program context, matching the DSL syntax users
write. Also surfaces ``attrs["arg_directions"]`` the same way Call does
so post-DeriveCallDirections IR round-trips.

The GlobalVarCollector helper used by the printer's topological sort
now also walks Submit::op_ — cross-function deps through pl.submit are
captured for ordering.

Adds EQUAL_DISPATCH(Submit) in structural_equal.cpp and
HASH_DISPATCH(Submit) in structural_hash.cpp. Since Submit's
GetFieldDescriptors covers op/args/deps/attrs/kwargs, the existing
reflection-based comparison and hashing work without further wiring;
the dispatch entries just opt Submit into them.

Tests (tests/ut/ir/printing/test_submit_printer.py, 6 cases):
- pl.submit syntax round-trips inside a Program
- deps=[...] kwarg renders correctly (bound params, multi-dep)
- zero-arg Submit + deps still emits cleanly (no dangling comma)
- structural_equal: Submit == itself; Submit != Call with same op/args;
  deps order matters

2129 IR tests pass (2123 prior + 6 new), no regressions. No code yet
emits Submit, so this commit is still purely additive at the pipeline
level.
Two scaffolding pieces let downstream Phase 3 / pass migrations flip
the parser to emit Submit without earlier passes mis-handling it.

DCE:
- CallFinder also flags Submit so an expression containing a Submit
  is conservatively preserved.
- IsSideEffectOp returns true for any AssignStmt / EvalStmt whose
  value is a Submit — a task launch is intrinsically side-effecting
  regardless of the kernel body.

ConvertToSSA:
- ExprSubstituter (the per-Call inner mutator) gains a Submit
  override that calls IRMutator's default (which already SSA-renames
  args_ and deps_ since both are reflection-walked fields) and then
  substitutes the return type the same way Call does. Submit does
  not carry manual_dep_edges in attrs, so the SubstCallAttrs branch
  is skipped.

Tests (tests/ut/ir/transforms/test_submit_passes.py, 3 cases):
- ConvertToSSA preserves Submit-ness across the pass.
- Args / deps Var references on Submit get SSA-renamed (verifies
  the IRMutator base really does walk both fields).
- Post-SSA program still prints as pl.submit(self.kernel, ...).

The remaining per-pass migration (DeriveCallDirections,
MaterializeTensorStrides, OptimizeOrchTensors, orchestration_codegen)
bundles with Phase 3's parser flip — those passes will start seeing
Submit only once the parser produces it. Until then this commit
keeps the pipeline buildable and DCE/SSA correct on hand-built
Submit IR.

3463 IR tests pass (3460 prior + 3 new), no regressions.
…llDirections

The DSL parser path for ``pl.submit(self.kernel, *args, deps=[...])`` now
builds an ``ir.Submit`` node with the typed ``deps_`` field instead of
an augmented ``ir.Call`` carrying ``attrs[manual_dep_edges]`` and a
TASK_ID-tail return type. Dumps captured before DeriveCallDirections
(roughly passes 0–33, including the user's original case after
InferTileMemorySpace at pass 18) now print the source-level
``pl.submit(self.kernel, arg1, arg2, deps=[t1, t2])`` form, making
task submission visually distinct from a plain function call.

To keep the existing Call-shaped codegen + late passes working
unchanged, DeriveCallDirections (pass 34) materializes the Submit →
Call view via the new ``SubmitToCallView`` adapter in expr.h:
``Submit::deps_`` is folded back into ``attrs[manual_dep_edges]`` on
the synthesised Call, and the standard direction-derivation logic
runs on the view. The result Call is what propagates downstream;
codegen, MaterializeStrides, OptimizeOrchTensors, etc. see Call as
before. The orchestration codegen AssignStmt visitor also funnels
any stray Submit through the same view as a defensive safety net.

Tests updated for the new IR shape:
- tests/ut/ir/transforms/test_flatten_call_expr_pass.py — the
  ``test_manual_dep_edges_survive_arg_flatten`` regression now checks
  ``submit.deps`` instead of ``call.attrs[manual_dep_edges]``, and
  uses ``VerificationLevel.NONE`` because the single-LHS Submit IR
  shape does not yet match the parser's required ``out, tid = ...``
  2-tuple unpacking syntax for pl.submit.
- tests/ut/language/parser/test_task_id_dsl.py and
  tests/ut/language/parser/test_manual_scope_parsing.py — the
  ``_calls_in`` / ``_all_calls`` helpers collect both Call and Submit
  RHS values, and direct ``call.attrs.get("manual_dep_edges")`` reads
  on kernel-call submits become ``submit.deps``. ``pl.at`` ScopeStmt
  deps still use the attr (Submit replaces only the Call-side dep
  encoding).

Full ``tests/ut/`` (excluding the pre-existing simpler-runtime-sync
failures in ``tests/ut/runtime``): 5040 passed, 0 failed.
…it dumps

Documents Submit alongside Call in docs/en/dev/ir/01-hierarchy.md:
- New row in the expression-nodes table.
- New `Submit vs Call` section listing the semantic differences (sync
  vs async, attrs[manual_dep_edges] on ScopeStmt vs Submit::deps_,
  return-type augmentation, Python syntax) and noting that
  DeriveCallDirections is the lowering point.
- Cross-link to `.claude/rules/pass-submit-awareness.md` so future
  pass authors see the dispatch rule alongside the IR docs.

Adds tests/ut/ir/transforms/test_submit_end_to_end.py (2 cases):
- `test_submit_visible_in_mid_pipeline_dump` exercises the user's
  original complaint — a mid-pipeline dump (post-Simplify, the same
  region as the InferTileMemorySpace dump that prompted this work)
  must show ``pl.submit(self.stage1, ...)`` and not the bare
  ``= self.stage1(...)`` legacy form.
- `test_submit_with_deps_visible_in_mid_pipeline_dump` verifies the
  ``deps=[...]`` kwarg surfaces from the typed Submit::deps_ field.

Both tests pass; full ``tests/ut/`` (excluding pre-existing
simpler-runtime-sync failures) remains green at 5042 passing.
…ttrs

Closes the two known issues logged at the end of the Submit IR migration.

Parser (Issue 1): The printer for ``Submit`` emits ``res: pl.Tuple[...,
TASK_ID] = pl.submit(self.kernel, ...)`` when an ``AssignStmt``'s LHS is a
single Tuple-typed Var. The parser previously rejected this form,
requiring strict ``out, tid = pl.submit(...)`` unpacking, which broke
the default ``VerificationLevel.BASIC`` round-trip instrument for any
pass that produced a Submit on a single LHS. New
``_parse_submit_single_lhs`` handler accepts both the bare
``result = pl.submit(...)`` and the printer-emitted annotated
``res: pl.Tuple[..., TASK_ID] = pl.submit(...)`` form, binding the
whole flat ``Tuple{*<kernel results>, TaskId}`` to one Var (no
projection statements). The two-element unpacking form continues to
go through ``_parse_submit_assignment``. A bare ``pl.submit(...)``
expression outside any assignment is still rejected.

Visitor (Issue 2): ``IRVisitor::VisitExpr_(SubmitPtr)`` previously
walked only ``args_`` and ``deps_``. The matching ``IRMutator`` walks
and rewrites Vars under ``Submit::attrs_[arg_direction_overrides_vars]``
too, so visitor/mutator disagreed about whether those Vars are live —
a silent leak if any future pass or test fixture sets that key on a
Submit. The visitor now mirrors the mutator and walks those Vars,
keeping unused-var / def-use / SSA-liveness analyses consistent.

Test updates:
- ``test_submit_single_target_is_rejected`` -> renamed to
  ``test_submit_single_target_binds_full_tuple`` and inverted; it now
  asserts the single-LHS form parses to a single-Var bind of the flat
  tuple, with the trailing TASK_ID element preserved.
- ``test_submit_passes.py`` / ``test_submit_end_to_end.py`` /
  ``test_manual_dep_edges_survive_arg_flatten`` no longer set
  ``VerificationLevel.NONE`` — default ``BASIC`` verification (with
  the RoundtripInstrument) now passes on Submit-bearing IR. New
  ``test_submit_single_lhs_form_round_trips`` regression guards
  against the parser fix.

Removes the two corresponding entries from ``KNOWN_ISSUES.md``.
Full ``tests/ut/`` (excluding pre-existing simpler-runtime-sync
failures): 5043 passing.
- python_printer Submit: actually implement the documented fallback for
  the no-Program path instead of hard-failing on INTERNAL_CHECK_SPAN.
  ``self.<kernel>`` inside a Program; bare ``<kernel>`` (or the raw op
  name for non-GlobalVar callees) when invoked standalone. (gemini,
  copilot — python_printer.cpp:838)
- ast_parser annotated assignment: replace the raw ``assert isinstance``
  with a ``ParserSyntaxError`` carrying source-location context.
  (gemini — ast_parser.py:994)
- SubmitToCallView: enforce the documented Var-like invariant on
  ``deps_`` entries. Null or non-Var/IterArg deps now throw
  ``pypto::TypeError`` at the view boundary instead of being silently
  dropped — losing a dep edge silently corrupts downstream codegen.
  (copilot — expr.h:918)
- ExprContainsCall → ExprContainsCallLike (and CallFinder →
  CallLikeFinder): the predicate now returns true for Submit too;
  renaming keeps the API self-describing. (copilot —
  dead_code_elimination.cpp:346)
- test_submit_printer test rename:
  ``test_submit_structural_equal_ignores_deps_order_independence`` →
  ``test_submit_structural_equal_is_deps_order_sensitive``. The old name
  contradicted the assertions, which check that swapped deps DO compare
  unequal. (copilot — test_submit_printer.py:143)
- test_submit_passes docstring: drop the outdated ``parser does not yet
  emit Submit (Phase 3)`` line — the parser flip is part of this PR.
  (copilot — test_submit_passes.py:16)

4520 ir/parser/codegen tests pass.
- docs/en/dev/ir/01-hierarchy.md: align the older RuntimeScopeStmt
  paragraph with the Submit-first narrative — parser emits Submit with
  populated deps_; DeriveCallDirections lowers to Call.attrs
  ["manual_dep_edges"] for codegen. (coderabbit — 01-hierarchy.md:182)
- SubmitToCallView: strip any pre-existing kAttrManualDepEdges from
  submit->attrs_ before adding the deps_-derived attr. Otherwise a
  stale attr entry would either duplicate the deps (when deps_ is
  populated) or survive as a stand-in (when deps_ is empty), making
  Submit::deps_ no longer the single source of truth. (coderabbit —
  expr.h:934)
- test_submit_end_to_end: rename ``scratch, tid`` to ``_scratch, _tid``
  so RUF059 stops flagging the deliberately-unused tuple unpack.
  (coderabbit — test_submit_end_to_end.py:59)

Deferred Class D items (architectural — left open for human judgement
on hw-native-sys#1544 as separate follow-ups):
- ast_parser.py annotated-assignment Submit bypass — CodeRabbit asks
  to route through the normal override_type validation. Refactoring
  the helper to return an Expr that re-enters the annotated path
  touches the existing _parse_submit_assignment plumbing; deferred.
- orchestration_codegen SubmitToCallView tuple-key loss — codegen is
  reached only post-DeriveCallDirections, so the IR has Call, not
  Submit; the defensive Submit branch is unreachable in the current
  pipeline. Worth keeping the defensive view but the tuple-key
  identity fix is unnecessary today.
- derive_call_directions first-writer context for Submit — the
  first_writer_roots_ map is keyed by Call*; preserving Submit
  identity through the rewrite is a heavier change that affects the
  prior-writer collector. Deferred.

38 ir/printing/transforms tests still pass.
@Hzfengsy Hzfengsy force-pushed the worktree-submit-print branch from d5f23e1 to 210cce9 Compare May 26, 2026 16:06
Hzfengsy added 2 commits May 27, 2026 00:23
…xtend Submit to pl.at outliner

Three correctness/architectural fixes for the Submit migration:

1. derive_call_directions: first-writer context for Submit (CodeRabbit
   `derive_call_directions_pass.cpp:422`). ``first_writer_roots_`` was
   keyed by ``const Call*``; when the Submit path lowered to a synthetic
   Call via ``SubmitToCallView``, the synthetic Call's pointer never
   matched what ``FirstWriterCollector::AnalyzeCall`` registered. The
   collector now also handles Submit (via SubmitToCallView for the args
   walk) and registers under the original Submit's pointer; the map's
   key type is widened from ``const Call*`` to ``const Expr*``. The
   mutator's Call path extracts an identity_key helper
   (``DeriveCallArgDirections``); the Submit override calls it with the
   original Submit's pointer so Out args get correct first-writer
   classification.

2. ast_parser: annotated-assignment Submit no longer bypasses
   override_type validation (CodeRabbit ``ast_parser.py:999``).
   ``_parse_submit_single_lhs`` now wraps a new
   ``_build_submit_single_lhs_expr`` helper that returns the Submit
   expression without binding. ``parse_annotated_assignment`` uses that
   helper to produce ``value_expr`` and then falls through to the
   standard annotation-consistency / override_type validation flow that
   every other RHS expression goes through.

3. OutlineIncoreScopes / OutlineClusterScopes / OutlineHierarchyScopes
   now emit ``ir.Submit`` (instead of an augmented ``ir.Call``) for
   ``with pl.at(...) as tid:`` scopes. ``scope_outline_utils.h`` builds
   the synthesised expression as ``Submit`` when ``scope_task_id_var``
   is set, pulling ``Submit::deps_`` from the scope's
   ``kAttrManualDepEdges`` attr — so ``pl.at(..., deps=[t]) as tid``
   prints as ``pl.submit(self.<outlined>, ..., deps=[t])`` in mid-
   pipeline dumps, consistent with explicit ``pl.submit(...)``. The
   no-TaskId path (plain ``pl.at(...):``) continues to emit a regular
   Call. DeriveCallDirections lowers both forms uniformly to Call for
   the late pipeline.

4541 ir/parser/codegen tests pass post-rebase onto upstream/main.
The pl.at-as-tid lowering is verified by reproducing the user's
example: ``with pl.at(level=pl.Level.CORE_GROUP, name_hint='s1') as
s1_tid:`` now appears as
``ret__tmp_v0: pl.Tuple[..., TASK_ID] = pl.submit(self.s1, ...)``
in the post-OutlineIncoreScopes dump.
…sys#1544)

``pl.submit(*args, **kwargs) -> NoReturn`` in scope.py was mislabeled.
The body does raise unconditionally (it's a parser construct, never
called directly), but the SURFACE TYPE that user code sees after the
parser intercepts the call is a 2-tuple ``(out, tid)`` (or
``((a, b), tid)`` for multi-output kernels). ``NoReturn`` is reserved
for functions that never return normally — under that annotation,
type checkers would reject every downstream ``out, tid = pl.submit(...)``
unpack site as dead code.

Switch to ``-> Any`` so the parser-interception story stays correct for
the user. The body keeps the existing ``raise RuntimeError(...)`` guard
so a misuse outside ``@pl.function`` still fails loudly at runtime.
Docstring updated to call this out, and to mention the single-LHS
``res = pl.submit(...)`` form added earlier in this PR.

723 parser tests pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants