Sandbox Not Available Anymore After Machine Restart

[boagerard]

Description

It is the second day it happens to me. I can not connect to my sandbox. Here is the result for the three following commands:

nemoclaw gege status
openshell sandbox list
openshell status

  Sandbox: gege
    Model:    nvidia/nemotron-3-super-120b-a12b
    Provider: nvidia-nim
    GPU:      no
    Policies: npm, pypi, telegram, docker, huggingface
    NIM:      not running

No sandboxes found.
Server Status

  Gateway: openshell
  Server: https://127.0.0.1:8080/
  Status: Connected
  Version: 0.0.10

I have to onboard again.

Do I miss something or is it the correct behavior?

Thank you in advance !

Reproduction Steps

1. Start the VM
2. nemoclaw onboard => sandbox name is gege
3. Use it
4. End of the day: openshell gateway destroy
5. Shutdown the VM
6. Start the VM
7. openshell gateway start
8. nemoclaw gege connect

=> status: NotFound, message: "sandbox not found"

Environment

• OS: Ubunutu 24.04 inside VMware Workstation Pro 25H2 for Windows
• Node: 24.14
• Docker: 29.3
• Nemoclaw: the command nemoclaw --version not working

Logs

Checklist

• I confirmed this bug is reproducible
• I searched existing issues and this is not a duplicate

[mak1jk]

Root Cause Analysis

The issue is in OpenShell's deploy/docker/cluster-entrypoint.sh, not NemoClaw itself.

What happens on container restart:

1. Container starts → entrypoint generates a new random SSH_HANDSHAKE_SECRET from /dev/urandom
2. Secret is injected into the HelmChart → gateway pod (openshell-0) picks up the new secret
3. But the Sandbox CRD's pod spec retains the previous secret in its OPENSHELL_SSH_HANDSHAKE_SECRET env var
4. Gateway sends NSSH1 handshake with new secret, sandbox rejects it → handshake verification failed
5. All SSH connections fail → sandbox inaccessible

Confirmed by comparing env vars after restart:

openshell-0:   OPENSHELL_SSH_HANDSHAKE_SECRET=<new-value>
my-assistant:  OPENSHELL_SSH_HANDSHAKE_SECRET=<old-value>  ← mismatch!

Workaround (until upstream fix)

After restarting the container, sync the secrets manually:

# Get the gateway's current secret
GW_SECRET=$(docker exec <container> kubectl exec -n openshell openshell-0 -- \
  printenv OPENSHELL_SSH_HANDSHAKE_SECRET)

# Update the Sandbox CRD to match
docker exec <container> kubectl get sandbox <name> -n openshell -o json \
  | sed "s/OPENSHELL_SSH_HANDSHAKE_SECRET\",\"value\":\"[^\"]*\"/OPENSHELL_SSH_HANDSHAKE_SECRET\",\"value\":\"${GW_SECRET}\"/" \
  | docker exec -i <container> kubectl apply -f -

# Force pod recreation
docker exec <container> kubectl delete pod <name> -n openshell

# Wait ~30s for pod Ready, then reconnect
openshell forward start --background 18789 <name>

Upstream Fix

Submitted to OpenShell: Vouch Request + branch

The fix persists the generated secret to /var/lib/rancher/k3s/server/ssh-secret.dat and reuses it on subsequent starts, ensuring the gateway always injects the same secret that the Sandbox CRD already has.

[drew]

Please track this issue on the OpenShell side here, NVIDIA/OpenShell#488.

[Jo-ar]

Hi @mak1jk thanks for this. I tried the above solution. It does sync the secrets and enabling for the nemoclaw sandboxname connect to run. However since you have forced pod recreation, your original sandbox state gets destroyed as well. Have you verified original state of sandbox has persisted? Thanks heaps

[mak1jk]

@boagerard Good catch — you're right. The manual workaround (sync secret + kubectl delete pod) does destroy sandbox state because the sandbox pod uses ephemeral storage (no PVC for /sandbox/). After recreation, the workspace is re-initialized from scratch.

Important distinction:

• The workaround (manual secret sync + pod deletion) → state is lost. This is a temporary measure only.
• The upstream fix (entrypoint patch) → the secret persists across restarts, so the pod never needs to be recreated. Gateway and sandbox secrets match from the start. State is preserved.

If you need the workaround today, you could mitigate state loss by:

1. openshell sandbox download <name> /sandbox/ /tmp/backup/ before deleting the pod
2. Delete and wait for pod recreation
3. openshell sandbox upload <name> /tmp/backup/ /sandbox/ to restore

But the proper solution is the entrypoint fix — no pod recreation needed at all.

[mak1jk]

Hi @mak1jk thanks for this. I tried the above solution. It does sync the secrets and enabling for the nemoclaw sandboxname connect to run. However since you have forced pod recreation, your original sandbox state gets destroyed as well. Have you verified original state of sandbox has persisted? Thanks heaps

The best solution is the one of Drew but at the moment even with that I'm not able to make it run on Wsl

[hulynn]

@jacobtomlinson @ericksoa
Still reproducible as of v0.1.0 / OpenShell 0.0.16. After a host reboot, nemoclaw list shows "No sandboxes registered" and a full re-onboard is required. This is a significant usability issue — sandbox persistence across reboots is essential for any production or long-running use case. Related: #888, #859, #1086. Would really appreciate this being prioritized.

[Koneisto]

For anyone hitting this: the workspace lives on the pod's overlay filesystem, not a PVC. Every pod restart (reboot, crash, gateway restart) resets it to image defaults.

I documented an automated workaround using a cron watchdog that detects wipes and restores from backup:

1. Nightly backup — SSH+tar from sandbox to host disk, maintains a latest-good symlink (only updated when backup has real content, so a backup taken during broken SSH doesn't overwrite a good one)
2. Watchdog (cron every minute) — writes a sentinel file (/sandbox/.openclaw-data/.nemoclaw-restored) after successful restore. Sentinel is on the overlay so it disappears on wipe. Watchdog detects missing sentinel → restores workspace + skills + cron from latest-good via tar-over-SSH → resets sessions (systemSent:true caches stale system prompt) → writes new sentinel
3. Session reset is critical — without clearing /sandbox/.openclaw-data/agents/main/sessions/ after restore, the agent ignores the restored SOUL.md/USER.md

Full documentation including the sentinel pattern, backup script, and watchdog template: https://github.com/Koneisto/nemoclaw-skill (see references/workspace-backup.md "Automated Recovery" section)