Memory improvements#161
Open
inqrphl wants to merge 23 commits into
Open
Conversation
added 14 commits
April 29, 2026 13:11
mainly done for the improvements to garbage collector https://go.dev/doc/go1.26#new-garbage-collector
… initializing concurrently
add a modified library with capacity limits for testing peer keeps an *simdjson.ParsedJson pointer add a pointer to peer in request, gets reused during initialization NewResultSet is renamed to NewResultSetRjson add NewResultSetSimdjson function, uses simdjson instead of Rjson add some tests for simdjson parsing
tracks the table initialization metrics for all 11 tables, merges them as well
it is a sum of many requests and their stats. it does not have a single request. leaving the request as is would be misleading
useful to switch between rjson and simdjson libraries without recompiling.
added 8 commits
May 7, 2026 15:04
now tracks the ssh and sshd as well, alongside interface speeds
fix compilation errors in tests
update compress package
inqrphl
commented
May 7, 2026
Author
inqrphl
left a comment
inqrphl
left a comment
There was a problem hiding this comment.
Looked through non-code changes
Author
|
I can rename the MaxParallelPeerInitializations to MaxParallelBackendInitializations to differentiate it better from MaxParallelPeerConnections if needed |
JsonParsingLibrary is renamed to JSONParsingLibrary
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improvements for faster startup, smaller peak memory consumption and tracing
add option: MaxParallelPeerInitializations
Additional semaphore used during the peer initialization. Controls how many peers can do the initial table synchronization at the same time.
During startup with an lmd connected to many backends, the bandwidth is shared between all of them. In a bandwidth limited scenario, this leads to lmd trying to download all of them at once and parse them, due to low bandwidth. It is kept in a state where its doing a lot of things at once, possibly increasing peak memory consumption.
It is better to limit the backends being processed, so that their tables can be downloaded, parsed and put into in-memory rows quickly. This semaphore is only used during initialization
There is already another config option called MaxParallelPeerConnections, but that is peer specific. It controls how many connections that peer can make.
add option: JsonParsingLibrary
can be used to switch between rsync and simdjson-go .
Unsure if this will ultimately help at the end, the idea was that simdjson-go can help with its AVX2/AVX512 optimizations.
This may speed up parsing, or improve memory consumption, especially on server processors with AVX512 support / with multiple AVX512 units per core. Could not test it comprehensively, as my laptop does not support it.
The toggle is left there for further testing.
add struct: ObjectInitializationMetadata
used for keeping track of all table initializations of a peer. Than print out more detailed times for total fetch time, parse time, lock time, prep time, total rows, size, download speed etc.
Download speed may especially be important.
Add tests that directly parse livestatus query results.
Mainly used to check simdjson-go for now