fix(store/state): perform potentially blocking RocksDB access in a blocking context

[kkovaacs]

This PR moves runtime access to RocksDB-backed store state structures onto Tokio’s blocking path via block_in_place().

• Added blocking helpers for State::inner and State::forest lock access.
• Routed account/nullifier tree reads, tree mutation computation, tree mutation application, and account state forest reads/writes through those helpers.
• Kept storage_map_key_cache updates on the normal async path since they do not touch the RocksDB-backed forest. Likewise, blockchain operations are entirely in-memory, so those too are left in async context.

Using block_in_place() is required because using spawn_blocking() would require the closure to be 'static, which in turn would require having an additional Arc wrapper around inner and forest.

Closes #1969

[Mirko-von-Leipzig]

Looks fine ito code, I'm just a bit concerned that we're using block_in_place? Should these be spawn_blocking?

[Mirko-von-Leipzig]

Should we be using block_on? I guess we're sort of saved because we know this will only happen once, and always sequentially?

[kkovaacs]

An alternative would be using a oneshot channel for explicit synchronization on the database commit -- that would avoid this ugly block_on() call here.

[Mirko-von-Leipzig]

I think we can live with these as is until we get proper write isolation.

[kkovaacs]

Should these be spawn_blocking?

I've considered that -- but it gets ugly because that way we need an additional Arc wrapper around the inner state and the forest. I think using block_in_place() could work for these and I wouldn't put much more effort into this before cleaning up our write path (#1901).

[sergerad]

I'm wondering if we should consider implementing a solution closer to the actual RocksDB code. Something like this (perhaps our own impl) https://crates.io/crates/async-rocksdb?

[kkovaacs]

I'm wondering if we should consider implementing a solution closer to the actual RocksDB code. Something like this (perhaps our own impl) https://crates.io/crates/async-rocksdb?

The problem is not only the RocksDB layer though... Both LargeSmt and LargeSmtForest are using rayon extensively to do some computations/updates concurrently. So ultimately we would have to implement an async wrapper around LargeSmt / LargeSmtForest to make all this work correctly.

[Mirko-von-Leipzig]

The other thing to consider is that we're actually not doing any async at all in any part of the store. We're using async wrappers for everything but in reality all the actual work is sync.

We've sort of done the wrong thing by adding async wrappers and I'd like to go the opposite direction. As in, apply_block (and most queries) has SQLite (sync), RocksDb (sync), file IO (sync) and hashing/verification (sync). At the moment we're patching these all to have separate async interfaces and hiding the blocking inside. But really we shouldn't have any of these be async.