Swamp S3 Datastore: Stale Locks Cause Indefinite Hangs

[kief]

Another one. My clanker diagnosed and wrote up the issue. All I know is my stuff is broke.

Swamp version: 20260325.134747.0
Environment: GitHub Actions ubuntu-latest, AWS eu-west-1

Summary

When using SWAMP_DATASTORE=s3:<bucket>/<prefix>, Swamp model method calls hang indefinitely due to stale lock files in S3 that are never released or expired. The ttlMs field in the lock payload is ignored — once a lock file exists in S3, subsequent operations that need to acquire the same lock block forever.

Reproduction

Setup

• S3 bucket with versioning enabled (required per Swamp docs for conditional writes)
• SWAMP_DATASTORE=s3:claw-hosting-swamp-state/swamp-state
• 10 models run sequentially via swamp model method run <name> create --json
• Models have CEL cross-references (e.g. route-table references vpc, subnet, igw)

Steps to reproduce

1. Run a sequence of swamp model method run calls against models with S3 datastore
2. After 3-6 successful model operations, the next operation hangs indefinitely
3. Inspecting S3 reveals .lock files that were never deleted after the previous operation completed

Observed behaviour

This was reproduced 3 times across different CI runs:

Occurrence 1 — Create sequence, hang on model 4 of 10:

Running create on bc01-vpc        # ~12s, succeeded
Running create on bc01-igw        # ~8s, succeeded
Running create on bc01-subnet     # ~8s, succeeded
Running create on bc01-route-table  # HUNG — no output for 30+ minutes

S3 state at time of hang:

swamp-state/.datastore.lock                              (197 bytes, never released)
swamp-state/data/@claw/subnet/12c6b12c-.../.lock         (197 bytes, never released)

Lock file contents:

{
  "holder": "runner@runnervm46oaq",
  "hostname": "runnervm46oaq",
  "pid": 3652,
  "acquiredAt": "2026-03-25T15:31:31.513Z",
  "ttlMs": 30000,
  "nonce": "55f05fa4-07ef-4f9e-9b2c-30f4934f2daf"
}

The ttlMs: 30000 (30 seconds) was set but the lock was still blocking operations 30+ minutes later.

Occurrence 2 — Create sequence, all 10 models succeeded:

After manually deleting the stale locks from occurrence 1 and clearing all S3 state, the full 10-model create sequence completed without hanging. This confirms the locks from a previous run were the cause, not a fundamental issue with the model operations.

Occurrence 3 — Destroy sequence, hang on model 8 of 10:

Running delete on bc01-instance       # succeeded
Running delete on bc01-key-pair       # failed (unrelated schema error)
Running delete on bc01-state          # succeeded
Running delete on bc01-sg             # succeeded
Running delete on bc01-instance-profile  # failed (unrelated CEL error)
Running delete on bc01-instance-role  # succeeded
Running delete on bc01-route-table    # succeeded
Running delete on bc01-subnet         # HUNG

S3 state at time of hang:

swamp-state/.datastore.lock                                    (197 bytes)
swamp-state/data/@claw/route-table/bc51e40d-.../.lock          (197 bytes)

Same pattern: global .datastore.lock and a per-model .lock left behind from a previous operation, blocking the current one.

Expected behaviour

1. Lock files should be deleted after the operation that acquired them completes (success or failure)
2. If a lock file exists but its ttlMs has expired (based on acquiredAt), subsequent operations should treat it as stale and acquire a new lock
3. If the process that held the lock has exited (different PID/hostname), the lock should be considered abandoned

Workaround

Manually delete stale lock files from S3 before running operations:

aws s3 rm s3://<bucket>/<prefix>/.datastore.lock
aws s3 rm s3://<bucket>/<prefix>/ --recursive --exclude "*" --include "*/.lock"

This unblocks the next operation, but the locks will accumulate again after a few operations.

Impact

This makes the S3 datastore unusable for sequential multi-model operations in CI pipelines. Each swamp model method run call is a separate process, and locks from process N are not released before process N+1 runs, eventually causing a hang.

The issue does NOT occur with the local .swamp/ datastore, only with SWAMP_DATASTORE=s3:....

Environment details

• Swamp version: 20260325.134747.0 (from .swamp.yaml swampVersion field)
• S3 bucket: versioning enabled, eu-west-1
• IAM permissions: full S3 access to the bucket (GetObject, PutObject, DeleteObject, ListBucket, GetBucketVersioning, ListBucketVersions)
• Each model method run is a separate OS process (called from bash via swamp model method run <name> <method> --json)
• Typically 3-8 operations succeed before the hang occurs

[github-actions]

Hi @kief, thanks for opening this issue!

We appreciate you taking the time to report it. A maintainer will review it as soon as possible.

In the meantime, please make sure you've included all relevant details (steps to reproduce, expected vs actual behaviour, swamp version, etc.) to help us triage it quickly.

[stack72]

🔍 Triage started — fetching issue context

[stack72]

📋 Classified as bug (high) — S3Lock.acquire() has an infinite loop bug: when a stale lock is detected, the code deletes the lock and 'continue's to retry, but this bypasses the maxWaitMs timeout check at the bottom of the loop. If deleteObject fails silently (all errors caught), the loop retries stale detection → failed delete → continue indefinitely, never reaching the timeout. Additionally, acquireModelLocks() has a global lock polling loop with no timeout at all. These two bugs explain why operations hang for 30+ minutes instead of timing out after 60 seconds.

[stack72]

📝 Implementation plan generated (v1) — Fix two infinite-loop bugs in S3 distributed locking that cause indefinite hangs when stale locks exist. The primary bug is in S3Lock.acquire() where the 'continue' after stale lock detection bypasses the maxWaitMs timeout check, creating an infinite loop when deleteObject fails silently. The secondary bug is in acquireModelLocks() where the global lock polling loop has no timeout at all. The fix restructures the acquire loop to always check timeout, adds a timeout to the global lock poll, and adds logging to release() for visibility into lock cleanup failures.

[stack72]

Approved Implementation Plan (v1)

Summary: Fix two infinite-loop bugs in S3 distributed locking that cause indefinite hangs when stale locks exist. The primary bug is in S3Lock.acquire() where the 'continue' after stale lock detection bypasses the maxWaitMs timeout check, creating an infinite loop when deleteObject fails silently. The secondary bug is in acquireModelLocks() where the global lock polling loop has no timeout at all. The fix restructures the acquire loop to always check timeout, adds a timeout to the global lock poll, and adds logging to release() for visibility into lock cleanup failures.

DDD Analysis

Affected domain: datastore/distributed_lock (DistributedLock interface, LockTimeoutError). The S3Lock is an infrastructure implementation of the domain DistributedLock interface. The fix is entirely within the infrastructure layer (s3_lock.ts) and the CLI adapter layer (repo_context.ts). No domain model changes needed.

Implementation Steps

1. Move the timeout check before the continue in the stale lock detection branch of S3Lock.acquire(). When a stale lock is detected and deleted, the continue skips the maxWaitMs timeout check, creating an infinite loop if deleteObject consistently fails. Restructure the loop so the timeout is checked on every iteration regardless of which branch executes. (risk: Changing loop control flow could affect lock acquisition timing — must preserve immediate retry when deleteObject succeeds)
2. The while(true) loop in acquireModelLocks() that polls for the global lock to be released has no timeout. Add a maxWaitMs-based timeout so the polling loop throws LockTimeoutError instead of hanging indefinitely. (risk: Need to import LockTimeoutError and choose appropriate timeout value (2x TTL = 60s suggested))
3. The release() method catches all errors silently. Add a logger.warn() call when deleteObject fails so operators can see why locks accumulate. (risk: Need to add logger import to s3_lock.ts which changes module dependency footprint)
4. Add a test to s3_lock_test.ts that simulates a stale lock where deleteObject always fails. Verify that acquire() throws LockTimeoutError within maxWaitMs instead of hanging. (risk: Mock needs to simulate persistent delete failure with stale lastModified — slightly more complex mock setup)

Files

• src/infrastructure/persistence/s3_lock.ts
• src/cli/repo_context.ts
• src/infrastructure/persistence/s3_lock_test.ts

Testing Strategy

1. Unit test: mock S3Client where deleteObject always throws, headObject returns stale lastModified — verify acquire() throws LockTimeoutError within maxWaitMs. 2. Unit test: verify existing stale lock force-acquisition still works when deleteObject succeeds. 3. Manual verification: run sequential model method runs against S3 datastore, kill a process mid-operation, verify next operation recovers within timeout.

Potential Challenges

• The S3Lock.acquire() loop restructuring must preserve the existing behavior: when deleteObject succeeds, the retry should still happen immediately without waiting retryIntervalMs.
• S3Lock currently has no logger dependency — adding logging to release() requires importing the logger infrastructure.
• The acquireModelLocks timeout needs to be long enough to accommodate legitimate global lock holds but short enough to detect stale locks.
• Testing the stale lock delete failure scenario requires a mock S3Client where deleteObject throws but getObject and headObject work normally.

Approved via swamp @si/issue-lifecycle