[Fix] Pause BFT block advancement during sync

[kaimast]

This PR aims to fix #3911 and #3636. The PR also contains changes from three smaller PRs (#4065, #4138, and #4139) that can be reviewed and merged separately.

Motivation

Recently, we added CheckBlockError as a machine-readable response to failures in block checks. The proposed changes use this new error type to differentiate between invalid blocks, and cases where the ledger has already advanced, and the proposed block is contained in the ledger already.
The snarkVM changes also ensure that calls to the ledger are atomic, but provides no transactional guarantees across multiple invocations of Ledger functions.

BFT has a lock field that aims to protect, block advancement. This lock already existed before this change, but was not accessible from outside the struct. A problem with this setup is that check_block_content and advance_to_next_block are dependent operations. Generally, one assumes that if the former succeeds, the latter will also. However, without additional locking, BFT can call advance_to_block between Sync's calls of check_block_content and advance_to_next_block. In this case, Sync second call will fail.

Consistent usage of PendingBlocks

Before this PR, pending blocks were only used during DAG sync. This PR changes it to always use pending blocks during sync, preventing a validator from syncing a block that contains a leader certificate that was skipped by the majority of the network. It also means that the two sync modes BFT are now almost identical, except that "fast" sync does not update the DAG until it has caught up with the network. This makes the other changes (next section) a lot easier to implement.

This README contains a longer write-up on how sync and block creation works in BFT. Please verify that it makes sense, and let me know if you notice inconsistencies between the write-up and the code.

Improved Coordination between Sync and Primary

The implementation implements an approached based on optimistic concurrency control to address this. Sync builds a chain of PendingBlocks without grabbing the BFT lock (otherwise, Sync would hold the lock for long periods of time).
Once a chain of pending blocks has reached the availability threshold, it will grab the lock, check their contents for correctness, and advance the ledger. In this step, check_block_content will verify the height of the block again and, if the height is below the current ledger height, Sync will abort and retry.

The main change for BFT is that update_dag does not grab the lock anymore, but callers are required to grab the lock before calling update_dag. There is a new method BFT::pause_for_block_sync that Sync calls before trying to commit blocks.
I removed the ALLOW_LEDGER_ACCESS parameter from update_dag to remove complexity from this function. This flag was only set to false for some tests, and does not seem to be needed any more even in this case, as the LedgerService abstraction allows to simply ignore writes. I would like to eventually have separate update_dag functions for sync and BFT, but this PR already reduces the complexity of the function noticeably.

Testing

[x] Pass CI
[x] Successfully run prerelease tests
[x] Successfully run prerelease tests again
[x] Run benchmarks and ensured there are no performance regressions

Notes

• Sync on this branch is about three times as fast as on staging. I assume this is mostly due to the improvements in snarkVM and unrelated to the changes in this branch.

[vicsn]

Pausing this for now, the latest stress-test run prerelease-v4.4.20 had a halt. First tackling #4047

[kaimast]

ProvableHQ/snarkVM#3070 needs to be merged first.

[kaimast]

I ran another set of stress tests on top of the two commits from today. Still passes.

Grafana Link

[kaimast]

I addressed all comments and resolved conflicts with mainnet.

[kaimast]

I added the changes from #4179 to this PR, which should make merging easier.

[raychu86]

Logic looks sound to me. Please run proper stress tests before merging

[vicsn]

Stresstest prerelease-v4.5.29 succeeded