Big Block Issue Optimization #2599
Description
Description
Background:
Currently, if a large block is encountered, the SDK may fail to index properly.
For example, Kusama: 25845986.
It behaves differently under HTTP and WS, and here’s the information we know so far:
- When using WS, it may cause an out-of-memory (OOM) error because the block is too large (at this point, we cannot do anything, and the process will be blocked), potentially leading to a 1006 connection closed error.
- When using HTTP, it may fail due to too many concurrent requests, resulting in many HTTP request failures.
Issues:
- How to detect large blocks?
- WS cannot retrieve large block results properly, causing OOM.
- HTTP cannot handle too many concurrent requests for large blocks, leading to timeouts and connection closures.
Optimizations:
- Implement an RPC response size detection mechanism. If the response exceeds the set size, record this block in the metadata, indicating that the current block height is a large block, which may warrant increasing the timeout.
- Add logs during the processing of large blocks to notify that the process is still running normally.