SQLite locking improvements for concurrent RPC#144
Open
Conversation
Remove manual locking of SQLite databases (which was missing some locks), and just open each connection with SQLITE_OPEN_FULLMUTEX to enable SQLite-internal full locking. This solves potential locking issues, while it may increase lock contention. That will be fixed in a follow-up change.
This refactors the snapshotting logic. Instead of having a fixed two-layer hierarchy (SQLiteStorage parent, SQLiteDatabase child snapshot), all snapshot handling now lives on SQLiteDatabase itself, and it supports (in theory) a tree of snapshots. We do not make use of this yet, but it will allow us in the future to have snapshots (copies) of existing snapshots, to be used as thread-local snapshots for RPC processing.
domob1812
force-pushed
the
sqlite-locking
branch
from
6c1ecee to
fcc8c41
Compare
Use the SQLite snapshot extension, which allows us to get "exact copies" of existing WAL read snapshots, for SQLiteGame. In particular, we can use this to give each RPC request its own, private database connection, which can then be used without lock contention or locking issues with any other RPCs running in parallel.
domob1812
force-pushed
the
sqlite-locking
branch
from
fcc8c41 to
c13eb99
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The existing logic had some issues when many concurrent RPC calls were using the same read-only SQLite database snapshot for their logic (lock contention for some operations, and missing locks / race conditions for some others). This set of changes provides a comprehensive fix:
FULLMUTEXmode for all SQLite databases (SQLite handles locking correctly) instead of our own custom locking logic. This prevents actual locking bugs and race conditions.Both changes together should make the code more robust and stable on one hand, and also more performant on the other.