Bloatnet - lessons learned

[wmitsuda]

This issue is to capture lessons learned during the creation of the bloatnet snapshotter.

I'm going to describe everything I experienced as part of it and some ideas that occurred. We can use this issue to discuss and propose concrete actionable items as subissues.

Disclaimer: focus so far was to unblock bloatnet snapshotter construction, so I investigated most things enough to find fast workarounds and save references/pprofs for future analysis, so some of my conclusions below may be missing entire context.

Non-cacheable p2p data during initial sync

Scenario: we are syncing an existing chain, no snapshotter exists, we are doing it for the first time.

What happens: during the first cycle, it downloads headers/bodies backwards from the tip, then it starts executing it.

What happened on bloatnet: we were 4 months behind the tip, there were over 4 bootnodes only, all slow. The backward download process by itself was very slow and not persisted, which means if after a few hours downloading it, even after it finished downloading and started executing it, if need to restart Erigon, you lose all progress, including everything you downloaded via devp2p and took hours.

Why that is important: I quite often had to restart Erigon in order to adjust some setting bc bloatnet was an unexplored territory, every time I used to lose progress. That should be a very common case if you are bootstrapping a shadowfork and you are very behind.

Workaround: NOT use an external CL, I manually applied FCU using the internal_triggerFCU RPC I did specially to make my life easier.

Possible fixes: (1) cache headers/blocks, do not throw it away on stop. (2) Align downloader batches with --sync.block.loop.limit batches.

More "prune-friendly" data structure for domains

I saw prunes taking hours like:

[INFO] [12-18|04:11:02.285] [snapshots] PruneSmallBatches finished   took=10h48m46.601212982s stat="commitment| kv: 80.55M from steps 1854-1854; storage| kv: 165.09M from steps 1854-1854"

logs are not totally clear, I think pruning is missing a ticker (didn't investigate the code deeper), but during execution you can notice very long pauses between execution batches, which I understand to be either pruning or commit (or both).

Current domain pruning requires full table scan bc the step is inside values, maybe some specialized data model separating latest state from "sorted by step" "keys/value for archival" would reduce required I/O and disk fragmentation at the expense of more I/O for writing latest state.

The step size reduction idea sounded like a general workaround for it, but apparently didn't help in my separated experiment before - on my local machine DB kept increasing.

Reclaimable space not shrinking below a certain plateau

The DB inflation may make sense, but it stopping shrinking after a certain point sounds strange, or maybe there is a reason I hadn't discovered yet.

Issue: #18421

Memory hungry commitment saving (not computation!)

Quite often during sync I've noticed OOMs. I've tracked them down, see pprof below:

I'm going to only attach one partial screenshot, but I've archived several mem pprofs in different moments.

The culprit seems to be the bytes.Clone inside BranchEncoder, but that's where the allocation is done, actually the references are passed down and held in TemporalMemBatch.

Basically, bloatnet generates a huge amount of changes in commitment, those changes are batched in-mem uncompressed and that OOMs.

Workaround: by default --sync.block.loop.limit is 5000, for bloatnet I had to decrease it to 1024 (by experimenting with different values), effectively telling Erigon to "batch less work before commiting to DB" at the expense of computing commitment more often (which is not good bc that's is pretty slow, so bigger batch == better). But for a 64GB machine, that was necessary.

What does it mean: it means Erigon, on non chaintip, is vulnerable to OOMs if the amount of work defined by --sync.block.loop.limit surpasses machine mem. But that's not intuitive, bc the inmem representation depends on lots of internals.

And by non chaintip I mean not only sync from scratch, but any case which you node loses chaintip, be it bc you stopped it for a while, machine become slow bc it swapped for a long time, etc.

Side note: I remember myself experiencing OOMs on regular mainnet in the past, on non chaintip, after resuming stopped nodes, on machines with 32GB. I suspect we've found a major bottleneck here, bc on chaintip Erigon should have no problems with 32GB or even less I guess. Solving this one should allow us to lower our minimum mem specs.

I think that case can be unfolded into more specific topics:

--sync.block.loop.limit as the unit of work

That's not a much stable unit to denominate work, bc block size is different among chains; even on mainnet, a hardfork can bring more data into a block and that translates into more work -> more mem required

I think one future direction could be for us to denominate those batch sizes in some unit that better represents the amount of work generated by those batches, for example, gas used. That way Erigon could "auto-tune" itself to machine specs. Asking users to guess --sync.block.loop.limit is not ideal.

Find a solution for holding uncompressed references in TemporalMemBatch

Specialized in-mem compressed format? Don't know, but it feels awkward that we need to hold everything in-mem when 99% of time the tx will commit successfully. It feels we can do more to improve that situation.

[AskAlexSharov]

We already have --batchSize param which limiting RAM usage by execution. And it also predicting overhead of commitment. Maybe prediction is not enough conservative - then let's just make it so:

func (sd *SharedDomains) SizeEstimate() uint64 {
	return sd.mem.SizeEstimate()
}


func (sd *TemporalMemBatch) SizeEstimate() uint64 {
	sd.latestStateLock.RLock()
	defer sd.latestStateLock.RUnlock()

	// multiply 2: to cover data-structures overhead (and keep accounting cheap)
	// and muliply 2 more: for Commitment calculation when batch is full
	return uint64(sd.metrics.CachePutSize) * 4
}

On bloatnet node you increased batchSize: "2g". Default value is 512mb

Your pic shows: 4.5g used by updates and 10gb used by commitment. So, let's change uint64(sd.metrics.CachePutSize) * 4 to 8 or 16

[AskAlexSharov]

it feels awkward that we need to hold everything in-mem we don't need to hold anything in mem. Just allow to hold something - for batch blocks execution and batch commitment calc. But we can exec blocks and 1-by-1 - no problem.

[AskAlexSharov]

in-mem compressed format - Mark added unique.Handle[common.Address] use in main - it's in-mem deduplication.

[AskAlexSharov]

That way Erigon could "auto-tune" itself to machine specs. - kind-of No. Because Erigon must use as little RAM as possible - and leave as much RAM for PageCache as possible. So, machine size doesn't matter much - for Erigon's configuration - because Erigon already does auto-tune by "using all available RAM (if user increase it) for PageCache" - and will get faster this way

[AskAlexSharov]

--sync.block.loop.limit=5000 flag is not for RAM management (it's not in GB's). It's mostly to "allow Erigon commit batch of block to DB - and it will allow for Erigon to produce new files, prune old data, etc...". It's not perfect - but it also used by Caplin to "download only 5K blocks and then throw them to Erigon" instead of "download ALL blocks and then throw them to Erigon" (increase Caplin's temporary db, etc...).

What happened on bloatnet: we were 4 months behind the tip, there were over 4 bootnodes only, all slow. The backward download process by itself was very slow and not persisted, which means if after a few hours downloading it, - this is reason why Caplin preserving --sync.block.loop.limit=5000. But bloatnet doesn't use Caplin and I don't know how to configure Lighthouse to not download backwards 100% of blocks - I guess it's possible.

[AskAlexSharov]

About More "prune-friendly" data structure for domains: yes, but it also will make reads slower (more shards to check on each read).

On this picture:

• prune did prune almost 100% of this table - means "full scan" will happen even if change table schema (need remove everything)
• On right side of picture - it's less than 1 Step in DB. And CommitmentVals is 160Gb. Means "add step prefix to key" will not help (all this data is 1 step).

So, i guess only stepSize reduction can help here.

Ilya working on Prune last couple months - for now he decided to keep "full-scan" in Domains: #17891 -> #18201

[wmitsuda]

On bloatnet node you increased batchSize: "2g". Default value is 512mb

I actually think this setting came from the default server template or I copied from another place, bc I was not aware or the meaning of this param (and probably didn't change it bc of it).

[wmitsuda]

it feels awkward that we need to hold everything in-mem we don't need to hold anything in mem. Just allow to hold something - for batch blocks execution and batch commitment calc. But we can exec blocks and 1-by-1 - no problem.

are you sure? see the pproof image, it goes from BranchEncoder all way down to TemporalMemBatch. I did some instrumentation before the holidays to check the allocation was being held inside TemporalMemBatch.

but now the chain is on tip, I can try to recheck on my machine (which is far behind the tip)

[wmitsuda]

--sync.block.loop.limit=5000 flag is not for RAM management (it's not in GB's). It's mostly to "allow Erigon commit batch of block to DB - and it will allow for Erigon to produce new files, prune old data, etc...". It's not perfect - but it also used by Caplin to "download only 5K blocks and then throw them to Erigon" instead of "download ALL blocks and then throw them to Erigon" (increase Caplin's temporary db, etc...).

it may not be, but it was the best solution I could figure without changing the code to constrain mem usage and avoid OOMs, by figuring the max amount of blocks that would generate commitment changes to DB without OOMing.

[wmitsuda]

What happened on bloatnet: we were 4 months behind the tip, there were over 4 bootnodes only, all slow. The backward download process by itself was very slow and not persisted, which means if after a few hours downloading it, - this is reason why Caplin preserving --sync.block.loop.limit=5000. But bloatnet doesn't use Caplin and I don't know how to configure Lighthouse to not download backwards 100% of blocks - I guess it's possible.

I think we should be able to use Caplin, but that was not my focus at this moment, goal was to sync the chain. The thing with external CL is that it triggers FCU with chaintip hash, I think that is how protocol works, erigon just understands it should fill up everything up to FCU hash.

[wmitsuda]

So, i guess only stepSize reduction can help here.

there is on crazy idea I was thinking about, but didn't give much thought: something about an append-only journal file for domain archival, per domain/per step. So, prune would be backwards traversal to flatten that file into 1 step file + delete journal file. But there may be flaws in that solution probably.

[AskAlexSharov]

“are you sure? see the pproof image, it goes from BranchEncoder“ - i mean: —batchSize=10gb and —batchSize=32mb both will work well-enough.

[AskAlexSharov]

“append-only journal file” - this is very interesting idea which flying around last 4years. But i didn’t see much “how to do it” - likely it will require tons of indices on top of append-only files: like “key -> offset”. Not sure if it will be enough or not.

As i know: Default SSTable size of PebbleDB is 4 MB. it’s analog of stepSize - and PebbleDB does layers - each layer is 10X bigger than prev one (to reduce amount of layers)

[sudeepdino008]

--sync.block.loop.limit=5000 flag is not for RAM management (it's not in GB's). It's mostly to "allow Erigon commit batch of block to DB - and it will allow for Erigon to produce new files, prune old data, etc...". It's not perfect - but it also used by Caplin to "download only 5K blocks and then throw them to Erigon" instead of "download ALL blocks and then throw them to Erigon" (increase Caplin's temporary db, etc...).

it may not be, but it was the best solution I could figure without changing the code to constrain mem usage and avoid OOMs, by figuring the max amount of blocks that would generate commitment changes to DB without OOMing.

#16856

[taratorio]

@sudeepdino008 #16856 is not related to this. That is Caplin. Bloatnet is using externalcl so it is purely related to the EL backward downloader. We can persist the downloaded headers in a scratch DB to solve this problem and get rid of it once all these headers have been inserted and executed. It is solvable, we just need to find time for it and tbh it is low priority.