Network notifiers fail to start, causing node to shut down

[reinkrul]

Lots of network chatter (new transactions from many nodes) in combination with slower disk I/O can cause a Nuts node to fail to start:

• network.connectToKnownNodes() causes syncing of the network, which can many transactions from many nodes
• this leads to lots of write transactions, which block read transactions
• then the network notifiers are Run() in Start(), but they fail due to lock time-outs
• then the node stops because it couldn't start the network module

"Could not start the server" error="unable to start Network: failed to start notifiers: unable to obtain BBolt read lock: database error: context deadline exceeded

Mitigations:

• don't let Start() return an error when it can't start a notifier; do it in a goroutine instead, retrying if it failed
• limit the number of parallel TransactionList queries

[reinkrul]

Root cause analysis

The startup failure has two contributing causes that compound each other on Azure ACI with an SMB (Azure Files) volume mount.

1. Reads blocked by writes (causes the startup failure)

The go-stoabs BBolt wrapper adds a Go-level sync.RWMutex on top of BBolt's own internal locking. Go's sync.RWMutex has writer-preference: once a writer is waiting, all new readers are blocked. This does not reflect how BBolt works internally — BBolt uses two independent locks so reads never interact with the write lock and proceed freely while a write is in progress.

The effect: a single slow write transaction blocks all concurrent reads at the Go level. During startup, connectToKnownNodes() triggers many incoming transactions, causing continuous write activity. Network notifiers try to start (needing a read transaction) and time out after 1 second, causing the node to shut down.

Fix: go-stoabs#146 — replace the sync.RWMutex with a plain sync.Mutex that only locks write transactions. Read transactions acquire no Go-level lock and go straight to BBolt's native handling.

2. Writes slow due to fdatasync on SMB (causes high write latency)

BBolt performs 2× fdatasync per write transaction commit (once after writing data pages, once after writing the meta page). On local disk this is sub-millisecond. On Azure Files via SMB, each fdatasync is a synchronous flush over the network — at the observed 55 tx/s with 2× fdatasync per commit, each flush takes on the order of single-digit milliseconds, far above local disk but consistent with synchronous network I/O over SMB.

This caps throughput and makes each write hold its lock far longer than it would on local disk, which amplifies the reader starvation problem above.

Fix: go-stoabs#4 — WithSyncInterval opens BBolt with NoSync = true and runs a background goroutine that calls db.Sync() at most once per configured interval. Final flush is guaranteed on Close().

---

Performance test results

Three builds were tested on Azure ACI with an SMB volume mount, syncing from the network on startup.

Build	Transactions	Avg tx/s	Min tx/s	Max tx/s
Baseline v5.4.28	24,494	55 tx/s	26	90
Lock removal only (go-stoabs#146)	18,829	55 tx/s	0	81
Lock removal + fdatasync interval (go-stoabs#4)	53,006	212 tx/s	0	411

Conclusions

Lock removal alone (go-stoabs#146) does not improve throughput — same 55 tx/s average. Removing the read lock eliminates reader starvation but doesn't touch the write path. The bottleneck is fdatasync latency on the SMB mount, not lock contention.

Lock removal + fdatasync interval (go-stoabs#4) delivers a ~4× improvement (55 → 212 tx/s average, up to 411 tx/s peak), by batching fdatasync calls instead of flushing after every write transaction.

Both PRs are needed, but for different reasons:

• go-stoabs#146 fixes the startup failure described in this issue: readers no longer time out waiting behind a slow writer, so notifiers can start successfully even while the node is syncing.
• go-stoabs#4 fixes the underlying throughput problem on SMB mounts.