[Bug] Mode B (kernelType=AICPU_CUSTOM): cust_aicpu_sd subprocess cache stale on AICore HBM writes → AICPU handshake deadlock (507018)

[hw-native-sys-bot]

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer

Description

PR #537 migrates the AICPU dispatcher from CANN 6.x RuntimeAicpuKernelLaunchExWithArgs (path A) to CANN 7.0+ rtsBinaryLoadFromFile + rtsLaunchCpuKernel (path B). The motivation is to enable a single host process binding both host_build_graph and tensormap_and_ringbuffer runtimes — path A is blocked by a process-wide one-shot firstCreatSo_ latch inside CANN preinstalled libaicpu_processer.so::BackendServerHandleManager::SaveSoFile, which makes loading a second runtime's inner SO a silent no-op.

In path B, the JSON descriptor uses opKernelLib=AICPUKernel + userDefined="True", which CANN routes to KERNEL_TYPE_AICPU_CUSTOM (4) (cann/runtime/src/runtime/core/src/kernel/program_common.cc). Per ae_so_manager.cc::GetSoPath, KERNEL_TYPE_AICPU_CUSTOM is the only path that searches /home/CustAiCpuUser/cust_aicpu_<dev>_<vf>_<pid>/ (where our uploaded SO actually lands); all other types search /usr/lib64/aicpu_kernels/... which is root-owned and unwritable from a user process. A gate at ae_so_manager.cc:514 (IsCustAicpuSd()) also enforces that KERNEL_TYPE_AICPU_CUSTOM MUST execute inside the aicpu_custom_scheduler subprocess.

Everything routes through correctly: CANN dispatches our Dyn* exports to the cust subprocess, our libsimpler_aicpu_<runtime>.so is dlopen'd, three phases (Null/Init/Run) all reach our code, and SchedulerContext::handshake_all_cores step 1 writes complete to all 9 cores' Handshake slots in shared HBM. AICPU writes are visible to host (verified via aclrtMemcpy DEVICE_TO_HOST readback). AICore stream dispatches, runs past its phase 1, writes aicore_regs_ready=1 back to HBM (also confirmed via host D2H).

The bug: the cust AICPU's L1 cache holds a stale 0 for the aicore_regs_ready field, even though HBM and host both see 1. SchedulerContext::handshake_all_cores step 2 spin loop never observes the update:

// src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp
while (hank->aicore_regs_ready == 0) {}   // ← cust AICPU stuck here, HBM has 1

After 2 s the host aclrtSynchronizeStreamWithTimeout(stream_aicpu_) reports ACL_ERROR_RT_AICPU_EXCEPTION (507018). Mode A (path A) does not exhibit the bug because the main aicpu_scheduler shares a cache coherency domain with AICore; the cust subprocess gets bound (SetAffinity) to a different AICPU cluster whose L1 is not snooped by AICore HBM writes.

User-space workarounds attempted (all fail):

Attempt	Result
volatile uint32_t field qualifier	No effect — prevents register reuse, not L1 cache
__atomic_load_n(..., __ATOMIC_ACQUIRE) (→ ldar)	No effect — only an ordering instruction, still reads L1
dc civac (clean + invalidate) in spin loop	Worse — same cache line co-hosts AICPU-written aicpu_ready/task and AICore-written aicore_regs_ready; civac writes back AICPU's dirty stale view, clobbering AICore's HBM writes
dc ivac (invalidate-only) in spin loop	Silently NOP'd from EL0 (SCTLR_EL1.UCI=0 in kernel)

Steps to Reproduce

1. Apply the PR Feat: AICPU launch via dispatcher bootstrap and per-task rtsLaunchCpuKernel #537 mode B refactor (single SO libsimpler_aicpu_<runtime>.so with merged outer dispatcher + inner runtime kernels; JSON opKernelLib=AICPUKernel + userDefined=True for all three phases — Null/Init/Run; orch SO candidate_dirs[] adds /home/CustAiCpuUser as first entry)
2. Run the simplest a2a3 onboard example:

   python3 -m venv --system-site-packages .venv
   source .venv/bin/activate
   pip install --no-build-isolation -e .
   python examples/a2a3/tensormap_and_ringbuffer/vector_example/test_vector_example.py -p a2a3

3. Test fails with timeout. Optional D2H diagnostic in device_runner.cpp::run after the timeout shows HBM has the right values:

   if (rc == ACL_ERROR_RT_STREAM_SYNC_TIMEOUT || rc == 507018) {
       Handshake h0 = {};
       aclrtMemcpy(&h0, sizeof(h0),
                   reinterpret_cast<const uint8_t*>(kernel_args_.args.runtime_args) + offsetof(Runtime, workers),
                   sizeof(h0), ACL_MEMCPY_DEVICE_TO_HOST);
       LOG_ERROR("workers[0] readback: aicpu_ready=%u task=0x%lx aicore_regs_ready=%u",
                 h0.aicpu_ready, (uint64_t)h0.task, h0.aicore_regs_ready);
   }

   Output: workers[0] readback: aicpu_ready=1 task=0x... aicore_regs_ready=1 — both handshake bits set in HBM, both visible to host, but cust AICPU never sees the AICore-written aicore_regs_ready=1.

Expected Behavior

tensormap_and_ringbuffer vector example passes end-to-end with kernelType=AICPU_CUSTOM (4) routing. Multi-runtime in a single host process becomes possible (one ChipWorker process binds both host_build_graph and tensormap_and_ringbuffer), which is the entire motivation for path B over path A.

Actual Behavior

=== Runtime: tensormap_and_ringbuffer  Level: 2 ===
  TestVectorExample::default ... [ERROR] run: [device_runner.cpp:797] aclrtSynchronizeStreamWithTimeout (AICPU) failed: 507018
FAILED: run_prepared failed with code 507018

Device log shows our dispatcher and inner kernel running cleanly in the cust subprocess; the deadlock is purely the AICPU-side read-side cache miss on AICore-written HBM data. The bug is not in user-space code and cannot be fixed there: the four standard AArch64 cache-bypass primitives all fail as documented above.

Possible fix directions

(Listed by where the change has to live — none can be done purely in this repo's user-space.)

#	Where	Change
A	CANN device kernel / driver	Enable EL0 dc ivac (set SCTLR_EL1.UCI=1 for the aicpu_custom_scheduler process). Smallest change, user-side spin loops can then explicitly invalidate.
B	CANN runtime / driver	Allocate Handshake HBM (runtime->workers) with non-cacheable / write-through attribute when called from aicpu_custom_scheduler context. Slight per-access HBM latency cost.
C	CANN cust scheduler	Pin aicpu_custom_scheduler worker threads to the same AICPU cluster as AICore's snoop domain (today aicpusd_worker.cpp::SetAffinity binds them to a separate cpuId=0).
D	simpler runtime (this repo)	Split Handshake so AICPU-written and AICore-written fields live on disjoint cache lines, then make AICPU spin loops dc civac only the AICore-written line — but EL0 invalidate semantics still apply, so this only works in combination with A or B. Alternatively, replace the spin-wait protocol with a device event/notify primitive that bypasses shared-memory polling entirely (substantial runtime refactor).

A/B/C are CANN-side; D is user-side but on its own is insufficient. We'd appreciate guidance from the CANN team on whether A or B is feasible for the cust subprocess in a near-term release.

Git Commit ID

ec7363a2eb6fbed4d71f848e1532dbcef7adc6c8

CANN Version

26.0.rc1 (V100R001C10SPC001B257)

Driver Version

26.0.rc1 (ascendhal 7.35.23)

Host Platform

Linux (aarch64)

Additional Context

Related open issues (different surface, may share root once A/B/C lands):

• rtStreamSynchronize (AICPU) fails with error 507018 in TensorMap and RingBuffer runtime #84 — 507018 in tensormap_and_ringbuffer (different reproducer)
• [sim] dcci lacks cache-line atomicity in simulation mode — handshake store reordering can cause deadlock #266 — cache coherency in handshake on sim
• [Bug] Handshake failure on Ascend910B3 during core initialization #480 — handshake failure on 910B3
• Multi-cid dispatch broken: two distinct ChipCallables on one chip child either fire wrong kernel or stream-timeout #759 — stream timeout on multi-cid dispatch

Concrete D2H diagnostic + per-iteration markers + cust subprocess routing analysis lived in the long debug session that produced this issue; the architecture diagram and CANN source pointers are in .docs/ISSUE-mode-b-cache-coherency.md of the PR #537 worktree. CANN open-source references:

• cann/runtime/src/runtime/core/src/kernel/program_common.cc — opKernelLib → kernelType table
• cann/runtime/src/aicpu_sched/aicpu_processer/ae_so_manager.cc — GetSoPath, LoadSo cust-vs-inner routing and the IsCustAicpuSd gate
• cann/runtime/src/aicpu_sched/aicpu_schedule/core/aicpusd_worker.cpp — SetAffinity binding worker threads to specific AICPU cores
• cann/runtime/src/aicpu_sched/aicpu_schedule/core/aicpusd_cust_so_manager.cpp — cust SO upload to /home/CustAiCpuUser/cust_aicpu_*/

[hw-native-sys-bot]

Investigation update — full attempt matrix and working solution

Continued investigation reframes the original diagnosis. The 507018 symptom is real, but the
root cause is not AICPU-side L1 staleness on AICore HBM writes. The cust_aicpu_sd
subprocess (kernelType = AICPU_CUSTOM (4) or CUSTOM_KFC (6)) is architecturally
sandboxed from AICore — it cannot reach the AICore notify/mailbox primitives that the
handshake protocol depends on, regardless of cache attributes on the shared HBM struct.

Stepping back, the host-side knobs that determine routing are three:

• cpuKernelMode ∈ {0, 1, 2}: JSON-only registration / JSON+SO upload / LoadFromData
• userDefined ∈ {True, False}: cust subprocess vs main aicpu_scheduler
• opKernelLib ∈ {AICPUKernel, CUSTAICPUKernel, KFCKernel, ...}: opaque label hashed
  by aicpu::GetAicpuKernelType into one of the four kernelType enum values

CANN's four AICPU kernelType values (pkg_inc/runtime/runtime/kernel.h):

kernelType	Enum	Process	Dispatcher ABI	SO discovery path	Can talk to AICore
2	KERNEL_TYPE_AICPU	main aicpu_scheduler	plain	/usr/lib64/aicpu_kernels/<vfId>/aicpu_kernels_device/	yes
4	KERNEL_TYPE_AICPU_CUSTOM	cust_aicpu_sd subprocess	plain	/home/CustAiCpuUser/cust_aicpu_*/	no
5	KERNEL_TYPE_AICPU_KFC	main aicpu_scheduler	KFC (framework core)	same as type 2	yes
6	KERNEL_TYPE_CUSTOM_KFC	cust_aicpu_sd subprocess	KFC	same as type 4	no

(Path strings confirmed in libtsdclient.so::tsd::TsdPathMgr::BuildKernelSoPath and
tsd::TsdPathMgr::BuildCustAiCpuUserPath; the cust flag is selected by JSON userDefined
and stored in libaicpu_engine_common.so::_cust_aicpu_flag.)

Full attempt matrix

#	Approach	opKernelLib	userDefined	cpuKernelMode	API	Effective kernelType	Result	Why
1	Original PR #537 mode B	AICPUKernel	True	1	rtsBinaryLoadFromFile + rtsLaunchCpuKernel	4 (AICPU_CUSTOM)	❌ 507018	Cust subprocess cannot reach AICore (originally diagnosed as L1 cache staleness; actual mechanism is the sandbox cutting off the AICore notify path)
2	Same as #1, with npu-smi set -t custom-op-secverify-mode=0	AICPUKernel	True	1	same	4	❌ 507018	npu-smi flag only disables signature verification on unsigned tar.gz — doesn't change routing
3	Mode B, switch to main scheduler with runtime upload	AICPUKernel	False	1	same	2 (AICPU)	❌ 507018 (kernel never runs)	cpuKernelMode=1 uploads SO to /home/CustAiCpuUser/..., but userDefined=False routes to main scheduler which looks in /usr/lib64/aicpu_kernels/<vfId>/aicpu_kernels_device/ — upload lands in the wrong namespace
4	Direction 1 (current working)	AICPUKernel	False	0	same	2 (AICPU)	✅ PASS	Pre-deploy aicpu_simpler.tar.gz to \$ASCEND_HOME_PATH/opp/built-in/op_impl/aicpu/kernel/ + ini whitelist entry. Driver extracts to /usr/lib64/aicpu_kernels/<vfId>/aicpu_kernels_device/ at process startup. userDefined=False routes to main scheduler which finds the pre-deployed SO. Multi-runtime works because the dispatcher caches per-runtime inner SOs by DeviceArgs.aicpu_so_bin (distinct rtMalloc'd device addresses per ChipWorker → no global one-shot latch).
5	LoadFromData / bisheng-style	(no JSON)	(no JSON)	2	rtsBinaryLoadFromData + rtsRegisterCpuFunc	? (no JSON to read)	❌ 507018 without magic; ❌ 107000 with magic	CANN validates the binary has RT_DEV_BINARY_MAGIC_ELF_AICPU = 0x41415243 ("AARC") baked into a specific offset. Only bisheng-compiled [aicpu] SOs have it — plain g++ -shared doesn't produce that magic.
6	LoadFromData with declared magic	(no JSON)	(no JSON)	2 + RT_LOAD_BINARY_OPT_MAGIC = 0x41415243	same	--	❌ 107000 at rtsBinaryLoadFromData	Confirms CANN reads magic from the binary itself, not from the option; the option is for cross-check

Cache-bypass diagnostics from the original report (volatile, __atomic_load_n /
ldar, dc civac, dc ivac) are dropped in retrospect — they were chasing a misdiagnosed
mechanism and all failed because the actual blocker is the cust subprocess's process-level
isolation, not cache attributes.

What we landed on

642ac727 — "Feat:
Direction 1 aicpu dispatcher — stable SO + per-runtime kernel upload". Configuration:

• opKernelLib = AICPUKernel, userDefined = False, cpuKernelMode = 0 →
  kernelType = 2 → main aicpu_scheduler
• One stable libsimpler_aicpu_dispatcher.so (no per-runtime suffix, no inner kernel
  sources). Dispatcher dlopens per-runtime inner SOs at runtime from
  DeviceArgs.aicpu_so_bin (host-uploaded bytes), keyed by device address — distinct
  ChipWorkers get distinct cache entries, removing the single-process multi-runtime block.

One-time host setup needed once per machine:

sudo cp build/lib/a2a3/onboard/aicpu_simpler.tar.gz \
  \$ASCEND_HOME_PATH/opp/built-in/op_impl/aicpu/kernel/

sudo tee -a \$ASCEND_HOME_PATH/aarch64-linux/conf/ascend_package_load.ini <<'INI'

name:aicpu_simpler.tar.gz
install_path:2
optional:true
package_path:opp/built-in/op_impl/aicpu/kernel
load_as_per_soc:false
INI

sudo npu-smi set -t custom-op-secverify-mode=0   # unsigned tar.gz

End-to-end examples/a2a3/tensormap_and_ringbuffer/vector_example and the host_build_graph
counterpart both pass; multi-runtime in a single host process is unblocked.

Suggested CANN-side improvements that would let us drop the manual setup

Independent of whether the original A/B/C fix directions for the cust-subprocess case are
pursued, these would simplify the deployment story for users in our position:

1. A cpuKernelMode that uploads at runtime but lands in the main scheduler's
   /usr/lib64/aicpu_kernels/<vfId>/aicpu_kernels_device/ namespace (instead of the cust
   /home/CustAiCpuUser/ namespace). Today there is no such mode — runtime upload implies
   cust subprocess routing.
2. Or: document transformer_tile_fwk_aicpu_kernel.tar.gz (referenced inside
   libtsdclient.so and routed to aicpu_kernels_device/tile_fwk_machine/) as a public
   extension point for tile-framework AICPU kernels, with a stable contract for what goes
   in that tar.gz.

[hw-native-sys-bot]

补充实验 #7 — CUSTKFCKernel + cpuKernelMode=1

pypto 的参考实现 (framework/src/machine/runtime/load_aicpu_op.cpp::CustomAiCpuSoLoad) 用
opKernelLib="CUSTKFCKernel" + userDefined="False" + cpuKernelMode=1 处理"运行时上传的
自定义 op"，希望复用这条路绕开 tar.gz 预部署。

实测分支：pr-537-custkfc-experiment（从 pr-537-work 拉出，仅改 load_aicpu_op.cpp）。

	配置	结果
#7	opKernelLib=CUSTKFCKernel（三相全部）userDefined=False，cpuKernelMode=1	❌ AICore 超时 507046

device plog 关键行：

ConstructAICpuSqeForDavinciTask: kernel_type=6   ← Null/Init/Run 三相都被识别为 type 6
Force cpy args ...                                ← runtime upload 成功，无 "open so failed"
AICore kernel execute failed, task timeout
errorStr: timeout or trap error                   ← AICore 在等 AICPU 的握手信号

含义：CUSTKFCKernel 在 CANN 9.0 里仍然映射到 KERNEL_TYPE_CUSTOM_KFC = 6，还是走
cust_aicpu_sd 子进程。"CUST" 不是上传机制的标记，而是 kernelType 路由的标记，与
userDefined 字段无关。SO 上传成功、AICPU 入口跑起来了，但 cust 子进程依旧无法跟
AICore 完成握手 —— 跟 #1 (opKernelLib=AICPUKernel + userDefined=True) 失败模式一致。

pypto 用 CUSTKFCKernel 跑得通的 control flow ops 大概率不依赖跟 AICore 的实时握手
（它们是 framework 上层的控制逻辑，不是 AICPU↔AICore 同步原语的使用方）。

更新后的结论矩阵

opKernelLib	进程	SO 来源	能跟 AICore 通信
AICPUKernel (type 2)	主 aicpu_scheduler	tar.gz 预部署到 /usr/lib64/aicpu_kernels/<vfId>/aicpu_kernels_device/	✅
CUSTAICPUKernel (type 4)	cust_aicpu_sd 子进程	API 上传到 /home/CustAiCpuUser/	❌
KFCKernel (type 5)	主 aicpu_scheduler	同 type 2	✅
CUSTKFCKernel (type 6)	cust_aicpu_sd 子进程	API 上传到 /home/CustAiCpuUser/	❌

所有"运行时通过 API 上传 SO"的路径（type 4 / type 6）都强制路由到 cust 子进程。
要在主调度器跑，必须经过 tar.gz 预部署机制（type 2 / type 5）。

Direction 1 (AICPUKernel + userDefined=False + cpuKernelMode=0 + tar.gz) 依然是
唯一同时满足"单进程多 runtime"和"AICore 握手"两条要求的路径。

给 CANN 团队的建议（更具体）

CANN 9.0 的路由表把"运行时上传"和"主调度器进程"做成了互斥两个维度。如果能放开这个
限制（例如新增 KERNEL_TYPE_AICPU_KFC 的 cpuKernelMode=1 路径，让 SO 上传到
主调度器的命名空间），框架侧需要 tar.gz 预部署的 ops 都能省掉。具体落点：

• 在 aicpu::GetAicpuKernelType 的 opKernelLib→kernelType 映射里增加一项：
  opKernelLib="KFCKernel" + cpuKernelMode=1 → kernelType=5 + 主调度器
  upload path（与 type 6 上传到 cust 命名空间区分）。
• 或：保留现有的 userDefined 字段做进程选择，但允许 userDefined="False" + cpuKernelMode=1 时 SO 上传到主调度器命名空间（目前这个组合下 SO 仍上传到
  /home/CustAiCpuUser/，主调度器找不到）。

[hw-native-sys-bot]

Resolution: zero-deploy bootstrap via libaicpu_extend_kernels.so

The "Direction 1" tar.gz approach worked but required three manual sudo
steps per host (cp tar to opp, edit ascend_package_load.ini, npu-smi set
custom-op-secverify-mode=0). Final design replaces all of that with an
in-process bootstrap. Single commit 40eb9c7d ("Feat: zero-deploy AICPU
dispatcher — bootstrap via libaicpu_extend_kernels") replaces the prior
Direction 1 commit on pr-537-work.

Key insight

The dispatcher SO runs in the main aicpu_scheduler's sched thread
(HwHiAiUser), which has write permission on
/usr/lib64/aicpu_kernels/0/aicpu_kernels_device/. So the dispatcher
can deploy itself into the main scheduler's preinstall path — no
host-side packaging or sudo needed.

Mechanism (verified end-to-end in device log)

1. Host LoadAicpuOp::BootstrapDispatcher issues one Mode A call:

   rtAicpuKernelLaunchExWithArgs(
     kernelType  = KERNEL_TYPE_AICPU_KFC,
     opName      = "AST_DYN_AICPU",
     soName      = "libaicpu_extend_kernels.so",  // CANN-preinstalled
     kernelName  = "DynTileFwkKernelServerInit",
     args.cfgdata.aicpu_so_bin = device-uploaded dispatcher bytes)
   

2. libaicpu_extend_kernels.so::SaveSoFile + SetTileFwkKernelMap + ExecuteFunc forward into our dispatcher's DynTileFwkBackendKernelServerInit.

3. Our Init detects "I'm being uploaded with myself as payload" by SO size
   (RTLD_DEEPBIND makes pointer-identity self-checks unreliable across
   independent dlopen instances), then copies the bytes into the main
   scheduler's preinstall path:
   /usr/lib64/aicpu_kernels/0/aicpu_kernels_device/libsimpler_aicpu_dispatcher.so.

4. Subsequent Mode B rtsBinaryLoadFromFile with
   kernelSo="libsimpler_aicpu_dispatcher.so" resolves the dispatcher
   from the same preinstall path. Three exports map to device_runner's
   three phases:

   • DynTileFwkBackendKernelServerNull — preload inner SO (no invoke)
   • DynTileFwkBackendKernelServerInit — invoke inner Init
   • DynTileFwkBackendKernelServer — invoke inner Server

5. Dispatcher caches inner SO handles by aicpu_so_bin device address —
   distinct ChipWorker instances get distinct rtMalloc'd addresses,
   hence independent cache entries, sidestepping
   libaicpu_extend_kernels's firstCreatSo_ one-shot latch.

Necessary dispatcher exports

libaicpu_extend_kernels::SetTileFwkKernelMap dlsyms a fixed set of
symbols on the uploaded SO and dlcloses if any is missing:
StaticTileFwkBackendKernelServer, DynTileFwkBackendKernelServerInit,
DynTileFwkBackendKernelServer. We export all three (Static is a stub)
plus one extra DynTileFwkBackendKernelServerNull used only by Mode B's
Null phase.

One subtle bug found along the way

device_runner's three-phase pattern is Null → Init → Run with the
contract "Null preloads, Init invokes inner Init, Run invokes inner
Server". An earlier draft mapped Null to Server (Exec), which made Run
execute before Init — leaving the kernel in an inconsistent state and
triggering a 50 s spin-wait timeout (looked like the AICore handshake
issue this issue originally tracked). Dedicated DynTileFwkBackendKernelServerNull
entry fixes that cleanly.

Test result

tensormap_and_ringbuffer/vector_example on a2a3 onboard: 8/8 PASSED
end-to-end with zero host-side deploy (no tar.gz, no sudo, no
npu-smi set).

Closing

The original L1-cache-staleness diagnosis in this issue was incorrect —
the cust subprocess path is architecturally cut off from AICore notify
primitives, not blocked by cache attributes. The actual fix is to route
through the main aicpu_scheduler (this commit). All other attempts
documented earlier in this issue (CUSTKFCKernel, LoadFromData, cust
subprocess) confirmed the only viable path is the main scheduler — but
without requiring tar.gz pre-deployment.

Closing this issue.

[hw-native-sys-bot]

Resolved by pr-537-work commit 40eb9c7d (zero-deploy bootstrap via libaicpu_extend_kernels). See the comment above for the final architecture.