chaser_header: store headers immediately to stop IBD tree OOM (stopgap)

[echennells]

Problem

During initial block download from genesis, chaser_organize::organize() caches any header that is not yet storable into the in-memory tree_ (cache()), which is explicitly unbounded:

// TODO: guard cache against memory exhaustion (DoS).
tree_.emplace(block->get_hash(), block);

chaser_header::is_storable is true only at a checkpoint height, at the milestone block, or when the chain is is_current && is_hard. A header is held in tree_ until the chain reaches the next storable point, at which the whole branch flushes. So the tree accumulates the main-chain header sequence between storable points.

On testnet3 those points are sparse — one checkpoint at height 546 and the milestone at 2,500,000 — leaving a ~2.5M-block gap with nothing storable. The entire main chain across that gap accumulates in tree_.

Repro: testnet3, from genesis, 30 GB box — RSS climbs 0 → ~28 GB (~2.3M cached headers) in ~4 min and the process is OOM-killed before it even reaches the 2.5M milestone that would flush it. (Mainnet syncs fine stock — its denser storable points and shorter chain keep the peak within RAM.)

Change (stopgap)

Make chaser_header::is_storable return true, so every PoW-validated header is itself a storable point and flushes immediately — the tree never accumulates. Mirrors chaser_block::is_storable, which already returns true. Validated: a from-genesis testnet3 sync holds anonymous RSS flat (~16 MB) where it previously OOM'd.

Honest scope — stopgap, not the complete fix

This bounds RAM but trades it for disk: with every header immediately storable, a peer feeding a long low-work chain gets it committed to the store instead of held in (bounded) RAM. It swaps an unbounded-RAM vector for an unbounded-disk one.

The complete fix is to bound the IBD accumulation itself — evoskuil's // TODO: guard cache against memory exhaustion (DoS). A naive in-organize size-cap does not work: protocol_header_in_31800::handle_organize drops the peer on any non-trivial error, and the chain must accumulate the whole gap to reach the next storable point, so a cap smaller than the gap stalls. Candidate real fixes (a design decision): depth/work-based incremental storability, denser storable points, or header-download backpressure.

Note: the separate chaser_organize.ipp:180 checkpoint-purge TODO concerns sybil forks below checkpoints — a different problem that does not address this accumulation OOM.

🤖 Generated with Claude Code

[evoskuil]

This is not a viable resolution to deferred work on a well-known DoS vector. Moving this to persistent storage just makes that worse.

FWIW The OOM aspect is just lousy OS behavior - the OS is not out of memory - it has practically unlimited virtual memory. The speed at which it is used is entirely under OS control. Nothing worse than having to work around OS bugs. Interestingly the resolution proposed above is to move it to disk, which the OS should already be doing.

I don't see it as wise to take on this stopgap. There is useful work that could be done here to collapse the accumulation of potentially weak headers. That is not an OS issue, as there is no limit to the number of weak headers that we might accumulate, which can overrun the disk (and would still do so under this proposal).