Endless Indexing

[Krokochik]

I awaited for indexing complete, got a message about compacting finished, the folder with electrs now uses 800 gb, but i still keep getting messages about indexing, the http server is unavailable. I restarted electrs 16 hours ago (after compaction finished and i started get indexing messages), but it seems electrs is not synced.

[murbot]

@Krokochik, did you get anywhere with this? I'm in exactly the same boat, and it's so slow. The system io, cpu and memory are all very low (close to 0), so there don't appear to be any bottlenecks.

[Krokochik]

@Krokochik, did you get anywhere with this? I'm in exactly the same boat, and it's so slow. The system io, cpu and memory are all very low (close to 0), so there don't appear to be any bottlenecks.

Sadly, the only thing I came up is to use electrs instead of esplora 'cause esplora was indexing over 3 weeks and even didn't complete for that period. Please let me know if you get smth

[murbot]

@Krokochik, did you get anywhere with this? I'm in exactly the same boat, and it's so slow. The system io, cpu and memory are all very low (close to 0), so there don't appear to be any bottlenecks.

Sadly, the only thing I came up is to use electrs instead of esplora 'cause esplora was indexing over 3 weeks and even didn't complete for that period. Please let me know if you get smth

Ok, thanks for letting me know. I've exhausted all my resources at this point, so I think I'll abort and switch to romanz/electrs as well. What a shame.

[shesek]

There are two indexing stages, one for adding transactions and for one indexing history, each followed by rocksdb compaction.

The logs should show "adding transactions from N blocks", then compaction, then "indexing history from N blocks", then another final compaction.

I restarted electrs (after compaction finished and i started get indexing messages)

It seems that you finished the first stage but not the second before restarting, and that the "indexing" messages are normal and just means that it did not finish indexing yet.

[shesek]

(As an aside, for personal use you probably should enable --lightmode, which reduces storage requirements and makes the first stage significantly faster)

[Krokochik]

There are two indexing stages, one for adding transactions and for one indexing history, each followed by rocksdb compaction.

The logs should show "adding transactions from N blocks", then compaction, then "indexing history from N blocks", then another final compaction.

I restarted electrs (after compaction finished and i started get indexing messages)

It seems that you finished the first stage but not the second before restarting, and that the "indexing" messages are normal and just means that it did not finish indexing yet.

Yes, "indexing" messages are completely normal, but what is not normal is that the second indexing stage lasts over 3 weeks on powerfuly machine with ssd. About --lightmode: as you say, the first index stage completed so I have no need to accelerate it. I also need esplora not for personal, but public use

[Krokochik]

Also as you can see CPU usage time is significantly lower than executing time, so I'm confident this is an issue

[Krokochik]

Also now I switched to original romanz/electrs and that indexing finished in about 9 hours

[shesek]

Looking at your log messages it appears that its indexing nearly 600 blocks every 10 minutes, or nearly 1 block per second, which seems reasonable. Did it slow down significantly when it reached more recent (and heavier) blocks?

lasts over 3 weeks ... CPU usage time is significantly lower than executing time

It's difficult to tell why without more information. One possibility is that it doesn't saturate CPU due to a bottleneck with your SSD.

Also now I switched to original romanz/electrs and that indexing finished in about 9 hours

romanz/electrs has different design trade-offs and indeed uses a much more lightweight index. If it’s meeting your needs, it might be a good idea to stick with it.

[Krokochik]

As I could remember, it was indexing 1 blk*.dat file every 10 mins, and it seems to be absurd since bitcoin has 4523 block files, i.e. indexing should take approximately 45230min=31day. The ssd has speed about 7400mb/s. Ideally I need esplora index designed for public use, but I can't wait so much

[Krokochik]

Now I checked and yeah, on the 2nd stage it is indexing 1 blk*.dat file per ~10 minutes (on another machine now). This issue is critical

[pokrovskyy]

Same experience. Currently at history sync, its beyond slow. Currently projecting around 1 week sync. SSD is high-speed datacenter-grade, CPU barely touched, RAM at 10%. I would assume this should maybe get parallelized somehow because currently it does not seem hit any viable bottlenecks. Will keep it running for a week and see where it gets me. Would settle for original esplora but this one has some useful endpoints I need + HTTP API, so would still give it a try! But indexing should be significantly improved IMO

[Krokochik]

I'd say you have no chance to make index just in a week, but anyway please give us know what happen

[pokrovskyy]

I have re-done the math this morning and timed .blk file processing average (on the history phase). So far I'm getting 75s per one .blk file. All .blk files are about the same size, so I suppose it does not matter that average Bitcoin block gets bigger over time (just less blocks per .blk file). With 4k files left should be in fact 3.5 days. I'll try to not forget and update you here how it goes.

On bottleneck tracking:

• during 95% of the 75s interval there are no writes with avg 25mb/s reads (waaay below my SSD capacity)
• then there is quick table dump to history RocksDB store
• CPU utilization is under 20%, memory at 11%

IDK what's the deal here, but it sure seems it can use some optimization, it seems the bottleneck is somewhere in the logic and not in the resources

[pokrovskyy]

Well, although indexing continues at projected pace (at 30% now), I will be aborting the sync. Ended up switching to https://github.com/cculianu/Fulcrum - took around 6 hrs to fully sync (160GB final index) and it works blazing fast (comparing to original https://github.com/romanz/electrs). Also put together a simple Nodejs HTTP API wrapper as a replacement for Explora's HTTP API, so no reason to continue with Esplora's backend sync (for my particular use-case)