Currently when loading a compressed solution file, it takes very long because zstd doesn't support multi-threaded decompression. Only compression.
One way around this would be to instead serialize into multiple files instead of one large file. Particularly the Vec[u8] data from storage1 and node_arena which can then be deserialized in parallel into a single vector in memory.
It looks like pzstd uses a similar strategy.
I can submit a PR if the library author is interested. I have something working now which looks promising:
- loading a 6GB solution compressed with zstd level 3: 10s
- loading the same solution split into file chunks compressed with zstd level 3, decompressed in parallel: 3.9s
Currently when loading a compressed solution file, it takes very long because zstd doesn't support multi-threaded decompression. Only compression.
One way around this would be to instead serialize into multiple files instead of one large file. Particularly the
Vec[u8]data fromstorage1andnode_arenawhich can then be deserialized in parallel into a single vector in memory.It looks like pzstd uses a similar strategy.
I can submit a PR if the library author is interested. I have something working now which looks promising: