ScheduledMerges: migrate union level late (when used in a new merge)#822
Draft
ScheduledMerges: migrate union level late (when used in a new merge)#822
Conversation
mheinzel
commented
Mar 3, 2026
Comment on lines
+1232
to
+1235
| -- This case is an optimisation over the one below. We only do a | ||
| -- single batch of lookups, but there's an extra allocation to | ||
| -- construct the list of runs. | ||
| pure (lookupBatch k (Just wb) (runs ++ [r])) |
Collaborator
Author
There was a problem hiding this comment.
This optimisation is probably worth it in the real implementation, but we probably don't need it in the prototype. I'll temporarily leave it here while we discuss the possible approaches, but will remove it before merging this PR. A short comment hinting at the possible optimisation should be sufficient.
83f94db to
9e43f0e
Compare
This separates the cases clearly and also matches the structure of the implementation more closely.
45c4485 to
eaa28b6
Compare
9e43f0e to
b7bc5c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on top of #814.
When creating the union of multiple tables, we create a special union level containing a merging tree. Once the tree has been completed, i.e. merged into a single run, we want to get rid of the union level by migrating the run to the regular levels. The main motivation is that we want it to become part of a last level merge, so compaction can occur. Last level merges are especially useful for compaction, since they can drop all
deleteentries.This PR demonstrates one of two approaches for migration. Since the main motivation is to allow for creating last level merges, we delay migration until the exact point where the completed union can become part of a merge. This avoids weakening the invariants of the regular levels, which would often be necessary to migrate the union earlier.
I still plan to add some tracing once we decided which approach for migrating the union we want to take.