Inf2 ERROR NRT:nrt_allocate_neuron_cores

[pcayen]

Describe the bug

We are running inf2 nodes in our cluster.
Ocasionally, a service will be scheduled on the node instance but will fail with the errors added below.
We are unsure what causes this bug.
Any help / guidance on this matter would be greatly appreciated.

Model Name

Custom, built in-house.
tinyvec_neuron_dynamic_batch_torch_2_9_1_16000.pt

Describe the workload type

Inference workload.

resources:
  limits:
    aws.amazon.com/neuroncore: '1'
    cpu: '1'
    memory: 1500Mi
  requests:
    aws.amazon.com/neuroncore: '1'
    cpu: 500m
    memory: 750Mi

Instance Type

Karpenter snippet:

requirements:
  - key: node.kubernetes.io/instance-type
    operator: In
    values:
      - inf2.xlarge

Release version

libneuronxla 2.2.16408.0+50c26cbd
torch 2.9.1+cpu
torch-neuronx 2.9.0.2.13.26312+8e870898
torch-xla 2.9.0
transformers 4.57.6

Reproduction Steps

uncertain

We use Keda to scale our services based on work-queue loads.
Once the queue is empty, we scale back down to zero.
Once the pod is removed from the inf2 node, we suspect that something goes wrong during teardown
and the core isn't released properly.
More work comes in, a new model is scheduled and then the new pod fails repeatedly with the errors below.

Our backend team has tried manually killing pods to attempt to replicate with no luck.

Regression Issue

• Select this option if this issue appears to be a regression.

Possible Solution

N/A

Logs/Context/Additional Information

2026-May-19 15:44:37.069500 1:18 ERROR NRT:nrt_infodump NRT version: 2.31.24.0 (0b044f4ce917b633a70eb3d0bc460f34ac3da620)
2026-May-19 15:44:37.071215 1:18 ERROR NRT:nrt_infodump Embedded FW version: unknown (unknown)
2026-May-19 15:44:37.072632 1:18 ERROR NRT:nrt_infodump CCOM not loaded
2026-May-19 15:44:37.073773 1:18 ERROR NRT:nrt_infodump NCFW version: 2.31.1.0 (cf13a49f86829014f0575b6d50c112ddf68b53c0)
2026-May-19 15:44:37.076914 1:18 ERROR NRT:nrt_infodump Cluster ID: N/A
2026-May-19 15:44:37.078007 1:18 ERROR NRT:nrt_infodump Kernel: Linux 6.12.83-113.160.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Fri May 8 17:56:26 UTC 2026
2026-May-19 15:44:37.081644 1:18 ERROR NRT:nrt_infodump Driver version: 2.27.4.0
2026-May-19 15:44:37.083090 1:18 ERROR NRT:nrt_infodump Failure: NRT_FAILURE in nrt_init()
2026-May-19 15:44:37.084445 1:18 ERROR NRT:nrt_infodump Visible cores: 1
2026-May-19 15:44:37.085612 1:18 ERROR NRT:nrt_infodump Environment:
2026-May-19 15:44:37.086743 1:18 ERROR NRT:nrt_infodump NEURON_LOGICAL_NC_CONFIG=1
2026-May-19 15:44:37.089503 1:18 ERROR NRT:nrt_infodump NEURON_LIBRARY_PATH=/home/botni/.local/lib/python3.11/site-packages/libneuronxla/libneuronpjrt.so
2026-May-19 15:44:37.091666 1:18 ERROR NRT:nrt_infodump NEURON_RT_ROOT_COMM_ID=localhost:62182
2026-May-19 15:44:37.093119 1:18 ERROR NRT:nrt_infodump NEURON_RT_NUMERICAL_ERRORS_VERBOSITY=none
2026-May-19 15:44:37.472620 1:18 ERROR NRT:nrt_allocate_neuron_cores Logical Neuron Core(s) not available - Requested:lnc1-lnc1 Available:0 Logical Core size:1
2026-May-19 15:44:37.997567 1:18 ERROR NRT:nrt_allocate_neuron_cores Logical Neuron Core(s) not available - Requested:lnc1-lnc1 Available:0 Logical Core size:1

[github-actions]

Hi @pcayen, Thank you for filing the issue! We will take a look and get back to you.

[geetasg]

@pcayen can you please share device plugin logs?

[mehvaibh]

Hi @pcayen,

Thanks for the detailed report. We're investigating this. From the logs, the core is stuck at the kernel driver level and since the new pod fails repeatedly, this isn't a transient timing issue but a persistent state that doesn't self-recover.

One thing that would help us reproduce: you mentioned manual killing doesn't trigger this, but KEDA teardown does. A few questions:

---

1. About the failure

• What eventually fixes it? Does the node need to be drained/terminated, does it recover on its own after some time, or does something else unblock it?
• Roughly how long is the gap between KEDA scaling to zero and the next pod starting? Seconds, minutes, or hours?
• Does Karpenter keep the inf2 node alive during the idle period, or does it terminate and launch a fresh node for the next scale-up?

---

2. About your setup

• Are you running the Neuron scheduler extension (k8s-neuron-scheduler) in your cluster?
• Is there anything else running on the inf2 node that uses Neuron devices? (health checks, monitoring agents, sidecars, etc.)

---

3. Capture at failure time

When the error happens, before the node gets recycled:

FAILING_POD=
NODE=$(kubectl get pod $FAILING_POD -o jsonpath='{.spec.nodeName}')

Any other pods on this node?
kubectl get pods -A --field-selector spec.nodeName=$NODE -o wide

Get a shell on the node:
kubectl debug node/$NODE -it --image=amazonlinux:2023 -- bash

Shows which cores are in use and by which PID:
chroot /host /opt/aws/neuron/bin/neuron-ls

---

4. Device plugin logs
   As asked above too