Skip to content

fix(frontend): avoid stack overflow on long type-alias chains - #1074

Open
hiyouwhatsapp wants to merge 3 commits into
wavefnd:masterfrom
hiyouwhatsapp:fix/iterative-type-alias-chains-1068
Open

hiyouwhatsapp wants to merge 3 commits into
wavefnd:masterfrom
hiyouwhatsapp:fix/iterative-type-alias-chains-1068

Conversation

@hiyouwhatsapp

Copy link
Copy Markdown
Contributor

Fixes #1068

Long forward type-alias chains could recurse through semantic validation and type canonicalization until the compiler ran out of stack

I switched the alias checks and canonicalization to walk flat chains iteratively, and the typed HIR pass now reuses already resolved aliases

Added tests for 16,384 aliases, forward and reverse declarations, pointer aliases and cycles still giving a normal error

Haven't run the full compiler suite locally, CI still needs to check it

Use iterative semantic cycle checks and type resolution, then reuse flattened aliases in typed HIR

Cover long chains, declaration order, pointer aliases and cycle diagnostics

Signed-off-by: hiyouwhatsapp <89997383+hiyouwhatsapp@users.noreply.github.com>
Signed-off-by: hiyouwhatsapp <89997383+hiyouwhatsapp@users.noreply.github.com>

@LunaStev LunaStev left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 4ebe02325bd187aea27a03b5da0bbce531e77eac on Linux amd64 with Rust 1.89.

The flat-chain fix works in the checks I ran: a 16,384-declaration forward chain lowered successfully, and the two new HIR integration tests plus all three focused alias unit tests passed. However, shallow declarations joined through pointer aliases still exhaust the process stack.

Reproduce the remaining acyclic-graph abort

In a disposable checkout of this PR, create front/hir/examples/review_alias_chain.rs:

use hir::TypedProgram;
use lexer::Lexer;
use parser::parse_syntax_with_spans;

fn main() {
    let count = 8192;
    let mut source = String::new();
    for index in 0..count {
        let next = if index + 1 == count {
            "i32".to_string()
        } else {
            format!("A{}", index + 1)
        };
        source.push_str(&format!("type A{index} = ptr<{next}>;\n"));
    }
    source.push_str("fun main() {}\n");
    let tokens = Lexer::new(&source).tokenize().unwrap();
    let syntax = parse_syntax_with_spans(&tokens).unwrap();
    println!("{}", TypedProgram::lower(syntax).is_ok());
}
cargo +1.89.0 build --locked --jobs 2 -p hir --example review_alias_chain
(ulimit -c 0; timeout 20s target/debug/examples/review_alias_chain)

The equivalent HIR harness on this commit exited via SIGABRT with:

thread 'main' has overflowed its stack
fatal runtime error: stack overflow, aborting

Every declaration has only one pointer wrapper; this is not an individual source type exceeding the syntactic nesting limit. A 2,048-declaration pointer-chain control completed. The observed threshold is host-dependent, not a proposed language limit.

Requested fix boundary

The new iterative Struct walk still calls canonical_type_inner on a wrapped target, and the pointer/future/array branches recursively enter it again. HIR compression also intentionally handles only flat aliases. Please make traversal of the remaining legal acyclic alias graph stack-safe, including wrapped targets and any recursive canonicalization/clone/drop paths reached by lowering. I have not isolated which recursive frame is the first to overflow, so this is an end-to-end reproduction plus the remaining recursive paths, not a claimed exact crash backtrace.

This is a residual gap in the robustness scope linked by Fixes #1068, not a claim that the PR newly introduced the existing wrapped-chain failure. Do not increase the OS stack or add a new rejection limit without a separate language-policy decision. Preserve cycle diagnostics and type identities.

Add a bounded subprocess regression for a long shallow-declaration/wrapped-alias graph, in both declaration orders, alongside the passing flat-chain controls. Ensure the check/build path and supported code-generation use remain covered; the reproduction above exercises the frontend/HIR only.

Formatting

The changed files also fail the repository's rustfmt check. Reproduce with cargo +1.89.0 fmt --all -- --check. Format only the touched Rust files before resubmission.

Recheck — 2026-11-04

The PR is still at the exact reviewed head 4ebe023. The wrapped-alias paths and formatting differences remain unchanged. The prior bounded pointer-chain reproduction still applies; no additional stack-abort test or full-suite rerun was needed to establish that the reviewed code has not changed.

target.clone()
}
},
_ => self.canonical_type_inner(target, seen),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] The iterative walk ends at a wrapped target and re-enters the recursive canonicalizer here. A chain of 8,192 shallow declarations type A0 = ptr<A1>; ... still aborts during frontend/HIR lowering with stack overflow, while the flat 16,384-alias control passes. The review body provides the complete bounded reproduction. Please cover wrapped alias graphs before treating #1068 as completed; this is a remaining failure, not an alleged new regression.

Comment thread front/hir/tests/alias_chains.rs Resolved
@LunaStev

LunaStev commented Nov 5, 2026

Copy link
Copy Markdown
Member

Remaining changes before merge — reviewed head 4ebe02325bd187aea27a03b5da0bbce531e77eac.

Please complete the wrapped-alias robustness scope before merging:

  • The linked review's 8,192-declaration acyclic pointer-alias graph still aborts with stack overflow. Every source declaration has one wrapper; this is not a deeply nested individual type.
  • Make the remaining wrapped-target traversal and reached canonicalization/clone/drop paths stack-safe. Preserve cycle diagnostics, type identity and the existing language limits; do not increase the OS stack to conceal the failure.
  • Add a bounded subprocess regression for wrapped chains in both declaration orders, alongside the passing flat-chain controls.
  • Apply the pinned formatter to the touched files and make the full formatting check pass.
cargo +1.89.0 test --locked --jobs 2 -p hir --test alias_chains
cargo +1.89.0 fmt --all -- --check

Also run the wrapped-chain reproduction from the linked review. Its observed threshold is host-dependent, not a proposed new language limit.

Detailed findings and reproduction: review.

@LunaStev LunaStev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked head bec2d3034daaf9c83bfba558c853bc2e17ecd291 against the previously reviewed 4ebe0232.

Formatting is now corrected and exact-head CI 42059818859 passed all 50 jobs. However, the new commit only changes formatting. It does not address the wrapped-alias abort described in the existing review and bounded reproduction.

The recursive transition in canonical_type_inner for wrapped targets remains, and HIR still compresses only flat aliases. The 8,192-declaration pointer-alias reproduction therefore remains an outstanding request; no new crash run was needed to establish that these source paths are unchanged. Passing flat-chain tests and CI do not cover this graph.

Please complete the existing request: make legal acyclic wrapped-alias lowering stack-safe, preserve cycle diagnostics/type identity/existing language limits, and add bounded subprocess coverage in both declaration orders. Do not increase the OS stack or introduce a new language rejection limit to hide the failure. The earlier review distinguishes this residual robustness gap from a newly introduced regression.

The formatting thread can be resolved; the wrapped-alias thread must remain open.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prevent compiler stack overflow on long acyclic forward type-alias chains

2 participants