yolo26l on inf2/trn1 (NCv2): numerical error in C2PSA Attention.forward + LegalizeSundaMacro internal error at bs>=4

[jaggiK]

Bug report — AWS Neuron / NeuronCore-v2

Affected SDK: neuronx-cc 2.21.33363.0 (inf2 DLAMI 2.8) and neuronx-cc 2.24.8799.0 (trn1 DLAMI 2.9)
Affected silicon: NeuronCore-v2 — confirmed reproducible on inf2.xlarge and trn1.2xlarge. Not affecting NeuronCore-v1 (inf1.xlarge compiles and runs the same model correctly).

---

TL;DR

We have two distinct compiler bugs on NeuronCore-v2 affecting Ultralytics' YOLO26 (the successor model to YOLOv8). Together they make yolo26l unusable for production on inf2/trn1 even though the silicon clearly has the throughput. We need help with:

1. Bug A — silent numerical error: Attention.forward (used in YOLO26's C2PSA block) is lowered to a NEFF that produces mathematically wrong outputs. Match rate against CPU FP32 reference drops from 96% (correct, on inf1) to 44.7% on inf2/trn1 — the model silently misses ~55% of high-confidence detections at production thresholds. Reproducible across BF16 / FP16 / FP32 / --auto-cast matmult / --model-type=generic / --model-type=unet-inference / --optlevel 1-3 / explicit re-write of the attention math with einsum.

2. Bug B — internal compiler error at batch ≥ 4 in the YOLO26 backbone: compiling layers 0-9 of yolo26l at batch ≥ 4 reproducibly fails with [TEN404] LegalizeSundaMacro: Incorrect IR / RuntimeError: neuronx-cc failed with 70. Affects every flag combination tested. bs=3 compiles cleanly; bs=4, 8, 16 all fail.

These two bugs interact: Bug A forces us to bypass C2PSA (run it on CPU as a hybrid), and Bug B prevents us from amortizing the resulting two-call Neuron pipeline at higher batch sizes. The combined impact is that we hit 158 FPS hybrid throughput on inf2.xlarge with CPU C2PSA (or 199 FPS with the C2PSA offloaded to a remote GPU sidecar) — instead of the ~480 FPS the silicon can demonstrably achieve. We've separately measured the broken-accuracy full-yolo26l at bs=8 hitting 479 FPS in pure-Neuron mode; the throughput is there, only the compiler is wrong.

---

What we ran

• Model: Ultralytics YOLO26-large, official weights from ultralytics-pytorch 8.4.47.
• Instances: inf2.xlarge (i-0757abb324da395b0) and trn1.2xlarge (i-0a077315608c00165, separately verified) in us-west-2.
• Framework: torch_neuronx 2.9.0.2.13 / neuronx-cc 2.21-2.24 / PyTorch 2.8 / Ubuntu 24.04 DLAMI.
• Compile recipe: Jim Burtoft's contrib script (jimburtoft/neuronx-distributed-inference) — sets detect.end2end = False, calls model.fuse(), m.export = True, then torch_neuronx.trace(model, dummy, compiler_args=…).
• Validation: 51-image production CCTV dataset (person_dataset), conf>=0.25 threshold, IoU>=0.5 box matching, same-class detections vs CPU FP32 reference. Match rate is the fraction of CPU detections that have a corresponding Neuron detection.

Reproducer scripts and minimal-repro artifacts in our public-internal repo aegissystems/hw-cost_optimization. Specific repro paths:

• inf1_vs_inf2/yolo26/scripts/compile_yolo26_no_psa.py — diagnostic that proves Bug A
• inf1_vs_inf2/yolo26/scripts/test_rewritten_attention.py — proves Bug A is op-pattern-specific in lowering, not in user-PyTorch code
• inf1_vs_inf2/yolo26/scripts/compile_yolo26_hybrid_bs.py — minimal repro of Bug B (parameterized batch size)

---

Reference baseline — inf1.xlarge (NeuronCore-v1) compiles and runs YOLO26L correctly

The same YOLO26-large weights, the same input shape (384×640), and the equivalent compile recipe work correctly on NeuronCore-v1. This is the strongest evidence that the issues below are NCv2-side compiler bugs, not model bugs or recipe bugs.

Metric	inf1.xlarge (NCv1)	inf2.xlarge (NCv2)
Compile success at production batch sizes	✅ bs=1 with --neuroncore-pipeline-cores 4	❌ bs ≥ 4 fails (Bug B)
Match rate vs CPU FP32 (conf≥0.25)	95.7–99.3% ✅	44.7% ❌ (Bug A)
Mean IoU on matched detections	0.96	0.93
Score delta (Neuron − CPU FP32)	−0.002 (essentially zero)	+0.038 (systematic bias)
Peak FPS (production config)	286 FPS @ DP×4, 2 in-flight	158 FPS (CPU c2psa hybrid) / 199 FPS (GPU sidecar hybrid)
Compile recipe	prepare_yolo26(weights, dtype=torch.float32) + torch_neuron.trace(model, [dummy], compiler_args=["--neuroncore-pipeline-cores", "4"])	same prepare_yolo26 + torch_neuronx.trace (every flag combo broken)
Spot price (us-west-2)	$0.034/hr	$0.143/hr
$/M images at production accuracy	$0.033/M	$0.21–0.25/M (hybrid only — direct compile not deployable)

This is our current production deployment for yolo26l: inf1.xlarge with the JB recipe at 384×640, 286 FPS at >95% match rate, validated on a production CCTV dataset. The inf1 path is also tested at 640×640 (with --neuroncore-pipeline-cores 4 to fit, since one v1 NeuronCore alone exhausts SB at that resolution) and produces 99.3% match rate at 56.7 FPS.

The ask: if NCv1 can lower yolo26's Attention.forward correctly and at any batch size, NCv2 should be able to as well. We are essentially asking AWS to bring NCv2's lowering of this attention pattern to functional parity with NCv1's.

---

Bug A — C2PSA / Attention numerical lowering error

Symptom

A YOLO26-large compiled with Ultralytics' standard recipe and any reasonable flag combination produces a NEFF that:

• emits ~55% fewer high-confidence detections than the CPU FP32 reference at conf>=0.25
• when matched, the matched detections have IoU 0.93-0.95 with CPU (geometry close)
• but ~60 of CPU's 141 detections are completely missing from the Neuron output at any score, not score-shifted

Reproducibility (numbers)

Same prepare_yolo26 model, same dataset, same decoder both sides:

Variant	Match rate vs CPU FP32	Mean IoU (matched)	Score delta
--optlevel 3 --auto-cast all --model-type unet-inference (BF16)	44.68%	0.93	+0.038
--auto-cast none (full FP32)	44.68%	0.95	+0.039
--auto-cast matmult --auto-cast-type bf16 (BF16 matmul, FP32 softmax)	44.68%	0.95	+0.039
--auto-cast all --auto-cast-type fp16	44.68%	0.95	+0.039
--model-type=generic BF16 (640×640)	42.14%	0.92	+0.048
--optlevel 1 --auto-cast all --auto-cast-type bf16	44.68%	0.95	+0.039
trn1.2xlarge with current SDK (neuronx-cc 2.24, fresh DLAMI)	44.68%	0.93	+0.040
inf1.xlarge native (NeuronCore-v1, same model)	95.74%	0.96	−0.002

A confidence-threshold sweep on the BF16 unet-inference artifact rules out simple score-shift:

NRN conf  cpu_dets  nrn_dets  matched  match_rate
0.25      141       78        63       0.4468
0.20      141       90        65       0.4610
0.10      141       147       73       0.5177
0.05      141       266       81       0.5745     ← still 57% even at extremely permissive threshold

Even at conf=0.05 with 185 unmatched Neuron detections (266 total NRN − 81 matched), only 81/141 CPU detections find a Neuron equivalent. The missing detections genuinely don't appear at any coordinate in Neuron output — not just below threshold.

Isolation — confirmed it's the attention block

We narrowed the bug to the C2PSA module (layer 10 of yolo26l) by surgically replacing c2psa.m = nn.Identity() (skipping the 2 PSABlocks inside C2PSA), then comparing no-PSA Neuron against no-PSA CPU FP32 (same architecture both sides, isolating the lowering bug from architectural detection loss):

=== no-PSA CPU FP32 vs no-PSA Neuron BF16 ===
images:                51
CPU FP32 detections:   82
Neuron BF16 detections:84
Match rate IoU>=0.5:   0.9634       <-- correct! (vs 0.4468 with PSA)
Mean IoU (matched):    0.9902
Score delta (mean):    +0.0127

Conclusion: removing the 2 PSABlocks from C2PSA fixes the numerics. The bug is in NCv2's lowering of the attention pattern inside Attention.forward.

Further isolation — even C2PSA alone is broken

Compile just model.model[10] (the C2PSA module) with bs=8 random input, compare numerically against the same module on CPU:

Compiler args	abs error	relative error
--auto-cast none --optlevel 1	5.86	3.75 (output is ~3.75× off)
--auto-cast all --auto-cast-type bf16 --optlevel 1	5.87	3.76
--auto-cast all --auto-cast-type fp16 --optlevel 1	5.87	3.76
Single PSABlock alone, BF16 + optlevel 3	11.33	2.48
Custom Attention.forward rewritten with einsum + manual softmax	5.78	3.64

Identical error magnitude across precisions and optlevels — indicates a deterministic compiler bug in the lowering path, not a precision/scheduling issue. Rewriting the attention math with different ops (einsum vs @) doesn't help — the compiler pattern-matches the QK^T·softmax·V structure regardless.

What we believe is happening

Looking at the YOLO26 attention forward:

B, C, H, W = x.shape
N = H * W
qkv = self.qkv(x)
q, k, v = qkv.view(B, num_heads, key_dim*2 + head_dim, N).split(
    [key_dim, key_dim, head_dim], dim=2
)
attn = (q.transpose(-2, -1) @ k) * self.scale     # [B, H, N, N]
attn = attn.softmax(dim=-1)
x = (v @ attn.transpose(-2, -1)).view(B, C, H, W) + self.pe(v.reshape(B, C, H, W))
return self.proj(x)

At 384×640 input, the spatial dim at layer 10 is 12×20 = N=240, num_heads=4, key_dim=32, head_dim=64 (verified from model.model[10].m[0].attn on a fresh load). Small attention. CPU produces correct output. NCv1 (inf1) produces correct output. NCv2 produces output with relative error ~3.8 — strongly suggesting an op or graph-rewrite pass that's only on the v2 path is corrupting the result. Likely candidates (you would know better than us):

• The combination of transpose+@+reshape on the QKV split path
• The position encoding self.pe(v.reshape(B, C, H, W)) add interacting with the attention output
• A fused-attention pattern detector that rewrites this into a primitive that handles different shapes correctly elsewhere but mishandles this specific (B=1-8, H=8, N=240) shape

Help we need (Bug A)

1. Confirm/identify the broken op-pattern: which lowering pass on NCv2 transforms this attention pattern, and what's it doing differently from the v1 path that handles it correctly?
2. Provide a workaround flag if available (e.g., a way to disable a specific fusion or pattern rewrite for the attention pattern).
3. Estimate of timeline for a fix in neuronx-cc. We have a real production deadline this affects.

---

Bug B — LegalizeSundaMacro compile error in YOLO26 backbone at batch ≥ 4

Symptom

Compiling layers 0-9 of yolo26l (the convolutional backbone, no attention) at batch size ≥ 4 reproducibly fails:

2026-05-08T01:46:38Z [TEN404] (_add.1188) Internal tensorizer error:
LegalizeSundaMacro:Incorrect IR by
<class 'neuronxcc.starfish.penguin.targets.sunda.passes.LegalizeSundaMacro.LegalizeSundaMacro'>

Please open a support ticket at https://github.com/aws-neuron/aws-neuron-sdk/issues/new.
RuntimeError: neuronx-cc failed with 70

The error message asks us to open a ticket — that's literally why this report exists.

Reproducibility (numbers)

Layers 0-9 are: 5× Conv + 4× C3k2 + SPPF — all standard ultralytics modules, no attention, no exotic ops. (Verified layout: indices 0,1,3,5,7 = Conv; 2,4,6,8 = C3k2; 9 = SPPF.)

Batch	Compile result
bs=1	✅ compiles
bs=2 (FP32, optlevel 2)	✅ compiles
bs=3 (BF16, optlevel 3, unet-inference)	✅ compiles
bs=4	❌ LegalizeSundaMacro
bs=8	❌ LegalizeSundaMacro
bs=16	❌ LegalizeSundaMacro

Tested every flag combination at bs=4:

• --auto-cast all --auto-cast-type bf16 --optlevel 3 --model-type unet-inference --enable-fast-loading-neuron-binaries
• --auto-cast all --auto-cast-type bf16 --optlevel 2
• --auto-cast all --auto-cast-type bf16 --optlevel 1
• --auto-cast all --auto-cast-type fp16 --optlevel 3
• --auto-cast none --optlevel 2
• --auto-cast none --optlevel 1

All fail with the same LegalizeSundaMacro error.

Splitting the backbone in half doesn't help: we tried PRE_A = layers 0-4 and PRE_B = layers 5-9 separately at bs=8. Both halves trip the same bug.

Wrapping vs not wrapping doesn't help: tried nn.ModuleList-wrapped and direct model.model[:10] traversal — same error.

The bug is specific to ops in layers 0-9. The post-C2PSA portion (layers 11-23 — neck + Detect head) compiles cleanly at bs=8, 16, and 32 with the same flag set.

Why this matters

We use the no-attention-on-Neuron workaround for Bug A: compile a PRE module (layers 0-9) and a POST module (layers 11-23) separately, run C2PSA on CPU between them. The pipeline is bottlenecked by per-call Neuron call overhead because we can't go above bs=3 on the PRE leg. Effective throughput is 158 FPS with CPU C2PSA, or 199 FPS when C2PSA is offloaded to a remote T4 GPU over TCP (which moves the bottleneck off CPU but doesn't break the bs=3 NC ceiling).

If Bug B were fixed (PRE compiled at bs=8), the same hybrid pipeline projects to ~480 FPS at full accuracy — the silicon ceiling. We've measured the broken full-yolo26l at bs=8 at 479 FPS (reproduction), so we know the chip can do the work; it's the compiler that's gating us.

Help we need (Bug B)

1. Identify what op or shape pattern in the yolo26 backbone triggers LegalizeSundaMacro at bs≥4 — likely a specific Conv/C3k2 layer interaction with the SPPF block.
2. Workaround flag, if any — even a --disable-pass LegalizeSundaMacro style escape hatch would unblock us.
3. Fix in a future SDK release.

---

What we tried (and why none of it worked around the bugs)

Approach	Result
Every compile flag combo on full yolo26l	All produce 42-47% match rate (Bug A)
Compile C2PSA in isolation, every precision/optlevel	abs error 5.8-11, relative 2.5-3.8
Rewrite attention with einsum + explicit softmax	Same broken numerics
Confidence threshold sweep (0.25 → 0.05)	Confirms detections genuinely missing, not score-shifted
Drop second PSABlock (1 PSA instead of 2)	Compiles, but loses 24% of detections architecturally
Compile pre+post hybrid at bs=4, 8, 16 with every flag combo	LegalizeSundaMacro (Bug B)
Split PRE into halves (layers 0-4 + 5-9) at bs=8	Same LegalizeSundaMacro
Compile pre via no-PSA wrapper at bs=8	Same LegalizeSundaMacro
torch_neuronx.set_neuron_cores to pin pre→NC0, post→NC1	142 FPS (worse than set_multicore)
torch_neuronx.DataParallel instead of set_multicore	Hangs / NRT timeouts at high concurrency
Multi-process inf2 client (NEURON_RT_VISIBLE_CORES per process)	Neuron RT refuses 2nd init: Could not be initialized
GPU sidecar over TCP (offload C2PSA to remote T4)	199 FPS — improved by 26% over CPU c2psa, but bottleneck moved to Neuron silicon
GPU sidecar (offload everything post-C2PSA to GPU)	91 FPS — yolo26's FPN skip tensors (15 MB at bs=3 FP16) don't ship efficiently

---

Production impact

We ship object detection on 100k+ CCTV cameras. We cannot deploy yolo26l on inf2.xlarge because the broken model silently drops ~55% of detections — a safety risk we won't ship. Until Bug A is fixed, our yolo26l fleet is on inf1.xlarge (NCv1, accurate at 96% match, 286 FPS, $0.033/M spot). That's working, but:

1. inf1 is an older Inferentia generation; long-term we expect to consolidate on Inferentia2 and our roadmap assumes that path.
2. Other workloads in our fleet (yolov8x weapon detection, 220 FPS at $0.180/M spot on inf2) are already on inf2. Maintaining two Neuron generations adds operational complexity.
3. We would prefer a single inf2-based fleet running both models. The blocker is yolo26l.

We are happy to provide:

• Detailed minimal reproducer artifacts (already in the GitHub repo, scripts ready to run on a fresh inf2.xlarge).
• Direct collaboration: our ML team can debug end-to-end with whoever owns the NCv2 lowering. We have time-budget for this.
• Test signals on candidate fixes — if you publish a SDK candidate, we can validate it against our 51-image production dataset within hours.

Contact

• Primary contact: Jagadish Mahendran (jagadish.mahendran@aegissystems.ai)
• Repository (read-only): aegissystems/hw-cost_optimization
• Detailed writeups in repo:
  • YOLO26_INF2_BLOCKER.md — top-level summary
  • inf1_vs_inf2/yolo26/ACCURACY_NCV2.md — Bug A isolation
  • inf1_vs_inf2/yolo26/PERF_400FPS_INVESTIGATION.md — Bug B + performance investigation
  • inf1_vs_inf2/yolo26/GPU_SIDECAR_INVESTIGATION.md — sidecar workaround experiment

We would appreciate a confirmation that these bugs are reproducible on AWS's side and an estimated timeline for fixes. Thanks.

[github-actions]

Hi @jaggiK, Thank you for filing the issue! We will take a look and get back to you.

[aws-reutermj]

Hi Jagadish,

Thank you for the detailed bug report. To help the compiler team reproduce and debug these issues, we need HLO artifacts and compiler flags from your compilations.

I see you mentioned reproducer scripts in aegissystems/hw-cost_optimization, but the compiler team doesn't have access to that repository. Could you please generate and share the following artifacts?

What we need:

For Bug A (C2PSA numerical error):

Since you're using torch_neuronx.trace() on inf2/trn1, please add the compiler_workdir argument to your trace call:

torch_neuronx.trace(model, example, compiler_workdir='./bug_a_artifacts')

This will save compiler artifacts (including graph.hlo and .neff if compilation succeeds) to ./bug_a_artifacts/.

For input capture, please convert your input tensors to numpy arrays and save them:

import numpy as np
np.save('input_tensor.npy', example.cpu().numpy())

For Bug B (LegalizeSundaMacro at batch ≥4):

Same approach for the failing backbone compilation:

torch_neuronx.trace(backbone_layers_0_9, dummy_input_bs4, compiler_workdir='./bug_b_artifacts')

Since this compilation fails, the workdir will contain the HLO and compiler logs showing where LegalizeSundaMacro failed.

How to share:

Please upload the entire compiler_workdir folders (both ./bug_a_artifacts/ and ./bug_b_artifacts/) to an S3 bucket and grant read access to AWS account 621547421844, then share the S3 paths here. Alternatively, you can attach them directly to this ticket if the files aren't too large.

We also need:

• Full compiler logs (stdout/stderr from the trace calls)
• The exact compiler_args list you passed to torch_neuronx.trace()
• Confirmation of your SDK versions (you mentioned neuronx-cc 2.21.33363.0 and 2.24.8799.0 — which one produced which artifacts?)

Once we have these artifacts, the compiler team can reproduce locally and investigate both the numerical lowering issue and the LegalizeSundaMacro error.

[jaggiK]

Hi @aws-reutermj — thanks for the quick response. Compiler artifacts captured and uploaded as requested.

S3 location

Bucket: s3://aegis-neuron-bug-1323-20260513/ (us-west-2)
Cross-account read access granted to AWS account 621547421844 (s3:GetObject + s3:ListBucket via bucket policy).

s3://aegis-neuron-bug-1323-20260513/
├── all_logs.txt                                # full stdout/stderr from all 3 repro runs
├── bug_a_full_workdir/                         # 168 MB
│   ├── compiler_args.json
│   ├── command.txt                             # exact neuronx-cc invocation
│   ├── model/graph.hlo                         # HLO captured pre-compilation
│   ├── graph.neff                              # successful NEFF
│   ├── input_tensor.npy                        # FP32 dummy input, shape (1,3,384,640)
│   ├── cpu_reference_output.npy                # CPU FP32 ground-truth output
│   ├── neuron_output.npy                       # Neuron BF16 output
│   ├── numerical_error.json                    # abs_err / rel_err summary
│   └── yolo26l_inf2_384x640_bf16_unet.pt       # compiled torchscript artifact
├── bug_a_c2psa_workdir/                        # 47 MB
│   ├── cpu_reference_output.npy
│   ├── input_tensor.npy                        # shape (8,512,12,20)
│   ├── summary.json                            # rel/abs errors across 3 variants
│   ├── autocast-none-o1/{command.txt,compiler_args.json,model/graph.hlo,graph.neff,neuron_output.npy}
│   ├── bf16-o1/...
│   └── fp16-o1/...
└── bug_b_workdir/                              # 317 MB (includes the 1 passing artifact + 4 failed HLO captures)
    ├── summary.json
    ├── bs3-bf16-o3-unet-WORKS/{...,graph.neff}  # control variant — compiles cleanly
    ├── bs4-bf16-o3-unet/{...,model/graph.hlo,error.txt}
    ├── bs4-bf16-o2/...
    ├── bs4-fp32-o2/...
    └── bs8-bf16-o3-unet/...

All variants captured via torch_neuronx.trace(..., compiler_workdir=<dir>) as you described — model/graph.hlo + command.txt + compiler_args.json are inside every variant subdir.

SDK version (single environment, single SDK)

All artifacts in this upload were produced on a fresh inf2.xlarge spot instance with:

• AMI: ami-07e8761a28189a68d — Deep Learning AMI Neuron PyTorch 2.8 (Ubuntu 22.04) 20251029
• neuronx-cc 2.21.33363.0+82129205
• torch-neuronx 2.21.x (per the DLAMI defaults)
• PyTorch 2.8 venv at /opt/aws_neuronx_venv_pytorch_2_8/

We had previously also reproduced Bug A on trn1.2xlarge with neuronx-cc 2.24.8799.0+6f62ff7c — identical numeric error magnitude (44.68% match rate, abs_err ~5.93). I can capture those trn1 artifacts to the same S3 bucket if you want both SDK versions; let me know.

Bug A — observed numerics

Full yolo26l, bs=1, 384×640, --auto-cast all --auto-cast-type bf16 --optlevel 3 --model-type unet-inference --enable-fast-loading-neuron-binaries

{
  "abs_err": 236.0,
  "rel_err": 0.36419753086419754
}

Output shape: [1, 84, 5040]. Compares Neuron BF16 forward against the same prepare_yolo26(... dtype=bfloat16) model evaluated on CPU (single Python process, same input tensor). The 0.36 relative error propagates through the Detect head and causes the 44.7% production-decoder match-rate drop documented in the original issue.

C2PSA module alone, bs=8, three precisions

All three variants produce identical magnitude errors — strongly indicates a deterministic lowering bug in the attention path, not a precision/scheduling issue:

{
  "autocast-none-o1": {"compile_s": 7.72, "abs_err": 5.91, "rel_err": 3.64},
  "bf16-o1":          {"compile_s": 7.32, "abs_err": 5.93, "rel_err": 3.64},
  "fp16-o1":          {"compile_s": 7.54, "abs_err": 5.93, "rel_err": 3.64}
}

compiler_args per variant:

• autocast-none-o1: ["--auto-cast", "none", "--optlevel", "1"]
• bf16-o1: ["--auto-cast", "all", "--auto-cast-type", "bf16", "--optlevel", "1"]
• fp16-o1: ["--auto-cast", "all", "--auto-cast-type", "fp16", "--optlevel", "1"]

The C2PSA module is model.model[10] after prepare_yolo26(weights, dtype=torch.float32) — see repro_bug_a_c2psa.py (uploaded with the artifacts under all_logs.txt; can also share script files separately if useful). The relative error of ~3.6 means the Neuron output is off by a factor of 3.6× over the CPU reference in the worst element — clearly outside any rounding-noise band.

Bug B — LegalizeSundaMacro reproducibility

All variants compile layers 0-9 of yolo26l (the convolutional backbone, no attention) on the same SDK and same hardware:

{
  "bs3-bf16-o3-unet-WORKS":  {"status": "PASS", "compile_s": 29.30},
  "bs4-bf16-o3-unet":        {"status": "FAIL", "compile_s":  9.00, "error": "neuronx-cc failed with 70"},
  "bs4-bf16-o2":             {"status": "FAIL", "compile_s":  8.95, "error": "neuronx-cc failed with 70"},
  "bs4-fp32-o2":             {"status": "FAIL", "compile_s":  8.63, "error": "neuronx-cc failed with 70"},
  "bs8-bf16-o3-unet":        {"status": "FAIL", "compile_s":  9.67, "error": "neuronx-cc failed with 70"}
}

The failing variants all emit the same error:

[TEN404] (_add.1188) Internal tensorizer error: LegalizeSundaMacro:Incorrect IR
  by <class 'neuronxcc.starfish.penguin.targets.sunda.passes.LegalizeSundaMacro.LegalizeSundaMacro'>
RuntimeError: neuronx-cc failed with 70

Full TEN404 lines are in all_logs.txt. Each failing variant has its graph.hlo saved (compiler workdir captures it before the failed pass), so you should be able to reproduce locally with the same HLO + compiler_args.json.

Note the control variant bs3-bf16-o3-unet-WORKS (identical flags, batch size = 3 instead of 4) compiles cleanly in 29.3 s — same backbone, same flags, just one fewer batch dimension element. The error is specific to the bs≥4 lowering of this op pattern.

Summary of what to look at first

1. For Bug B (likely simpler): bug_b_workdir/bs4-bf16-o3-unet/model/graph.hlo vs bug_b_workdir/bs3-bf16-o3-unet-WORKS/model/graph.hlo. Same source model + flags, only the batch dimension differs. The LegalizeSundaMacro pass fires on _add.1188 (per the TEN404 message).

2. For Bug A (subtler, numerical): bug_a_c2psa_workdir/bf16-o1/model/graph.hlo is the smallest reproducer — just the C2PSA module (model.model[10]) at bs=8. Compare neuron_output.npy vs cpu_reference_output.npy element-wise.

Happy to:

• Capture the trn1 artifacts (neuronx-cc 2.24.x parallel set) on request
• Run any modified compiler_args you'd like tested
• Validate any candidate fix against our 51-image production dataset within hours

Thanks again — let me know what else you need.

[jaggiK]

@aws-reutermj added you to the repo

[jaggiK]

Verification update: .split() → slicing patch works at bs=1 only, fails at bs≥2

Hi @aws-reutermj — verified the patch end-to-end. Great news for bs=1, but we've found that the same patch produces broken output at any batch size ≥ 2. Sharing the data so your team can investigate the follow-on case before closing the bug.

bs=1 (the case you verified): ✅ FIXED

Applied the slicing replacement to Attention.forward exactly as described:

# BEFORE
q, k, v = qkv.view(B, num_heads, key_dim*2 + head_dim, N).split(
    [key_dim, key_dim, head_dim], dim=2
)
# AFTER
qkv_view = qkv.view(B, num_heads, key_dim*2 + head_dim, N)
q = qkv_view[:, :, :key_dim, :]
k = qkv_view[:, :, key_dim:key_dim*2, :]
v = qkv_view[:, :, key_dim*2:, :]

Recompiled yolo26l at bs=1 with original JB flags (--auto-cast all --auto-cast-type bf16 --optlevel 3 --model-type unet-inference --enable-fast-loading-neuron-binaries) on the same neuronx-cc 2.21.33363.0 inf2.xlarge:

Metric	Before patch	After patch	inf1 (NCv1) reference
Match rate vs CPU FP32	44.68%	95.74%	95.74%
Cosine similarity	~0.45	0.999983	—
Score delta (mean)	+0.038	−0.0013	−0.002
Score delta (median)	+0.029	−0.0004	—
Mean IoU (matched)	0.93	0.9453	0.9553

Production-decoder match rate matches our inf1 baseline exactly. Score delta essentially zero. Bug A as reported is resolved at bs=1.

Follow-on: same patch at bs≥2 is broken

We then recompiled the patched model at higher batches to see if we could amortize per-call overhead. All bs≥2 compilations succeed but produce mathematically wrong output across all batch positions. Sanity check: trace at bs=N with random torch.rand(N, 3, 384, 640, dtype=torch.bfloat16) input, then run inference on the same tensor and read max class score per batch element.

NRN max class scores per batch element (conf threshold for prod decoder = 0.25):
  bs=1 patched: [0.83]                              ✅ matches CPU (0.83)
  bs=2 patched: [0.014, 0.003]                      ❌
  bs=3 patched: [0.0007, 0.001, 0.014]              ❌
  bs=4 patched: [0.010, 0.003, 0.002, 0.0006]       ❌
  bs=8 patched: [0.169, 0.006, 0.003, 0.001,
                 0.005, 0.014, 0.002, 0.001]        ❌

CPU reference (same input, same patched code path) produces equal max scores per batch element when input is duplicated, and ~0.83 max score on real images — confirming the patch math is correct on CPU, the bug is on the Neuron lowering side for bs≥2.

We also ran the production validator (51-image dataset, conf≥0.25, IoU≥0.5) against the bs=8 patched artifact: 0.71% match rate (1 of 141 CPU detections matched). Versus bs=1 patched: 95.74%.

Reproducer

The bs=2/3/4/8 NEFFs + HLOs + compiler_args.json + input/output tensors are being uploaded to the same S3 bucket alongside the bs=1 artifacts:

s3://aegis-neuron-bug-1323-20260513/bug_a_patched/
├── bs1_splitfix/   ← works (sanity check artifact for reference)
├── bs2_splitfix/   ← broken
├── bs3_splitfix/   ← broken
├── bs4_splitfix/   ← broken
└── bs8_splitfix/   ← broken

Same cross-account read access to AWS account 621547421844. Will confirm upload in a follow-up comment when complete.

Why this matters for our deployment

• inf2 patched at bs=1: 269 FPS, $0.148/M spot at 95.74% match
• inf1 native (current production): 286 FPS, $0.033/M spot at 95.74% match

The patch makes inf2 deployable but the per-image cost is ~4.5× our current inf1 deployment. We'd need bs≥4 working to make the migration economically justifiable. With bs=8 working at the same accuracy, projected inf2 throughput is ~480 FPS at ~$0.073/M — finally beating inf1.

Ask

1. Can you investigate the bs≥2 case as a follow-on? Same compile recipe, same patch, only the batch dimension differs.
2. Is the bs≥2 issue likely the same root cause (a sibling .split()-like pattern in the lowered IR that the slicing patch doesn't cover), or a separately addressed bug?
3. Should we expect a single SDK fix to address both, or is the bs≥2 path independent?

Thanks again — happy to validate any candidate fix on the same 51-image production dataset within hours.

— Jagadish

[aws-reutermj]

Thanks for providing this. Working with the team still to evaluate the issue and will pass this on to them

[aws-reutermj]

For bug B, we have been unable to reproduce your issue internally. If you give us write access to that bucket, we can provide the investigation and logs for it.