PHOENIX-7931 Coalesce per-batch mutations into a single record#2540
Open
tkhurana wants to merge 1 commit into
Open
PHOENIX-7931 Coalesce per-batch mutations into a single record#2540tkhurana wants to merge 1 commit into
tkhurana wants to merge 1 commit into
Conversation
Switch the replication log framing to a cell-oriented record (List<Cell> + record-level attributes) and coalesce IndexRegionObserver.replicateMutations to emit one record per (data table, batch) and one record per index target table per batch, instead of one record per Mutation. The consumer reconstructs Put/Delete mutations on the row+type boundary via MutationCellGrouper, so coprocessor-merged cells (local index, conditional TTL, ON DUPLICATE KEY UPDATE) survive the framing change. Drops the Mutation overload from LogFile.Writer (production no longer needs it) and rewrites the test layer to assert against cell lists.
tkhurana
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PHOENIX-7931: Coalesce per-batch replication appends into a single record
Targets the
PHOENIX-7562-feature-newfeature branch (not master).What changed
On the write path, a server-side batch's mutations for a given table are now coalesced into a single replication log record carrying a flat cell stream, rather than one record per mutation. On the replay path,
MutationCellGrouperreconstructs Put/Delete mutations from that cell stream on the row + put-vs-delete boundary, mirroring the algorithm HBase'sReplicationSinkuses for WALEdits.This cuts ring-buffer pressure substantially (≈5× fewer Disruptor events at high batch sizes) without changing the fsync floor — coalescing already minimized inner syncs. The on-disk record format changes accordingly (per-cell row/family/qualifier/timestamp/type/value, with a per-record table name + commitId + attributes header).
Key files
IndexRegionObserver— per-(table, batch) cell aggregation before a singleappendMutationCellGrouper(new) — replay-side cell-stream → mutation regroupingLogFileCodec/LogFileRecord/LogFile— new coalesced record formatReplicationLogGroup/ReplicationLog— cell-oriented append APITesting
MutationCellGrouperTest(10 cases): empty input, single Put/Delete, contiguous same-row/type coalescing, row-change splits, Put→Delete same-row type-boundary split, three-row Put/Delete/Put, adjacent DeleteColumn/DeleteFamily subtype split, non-consecutive same-row partitioning, and cell preservation.testAppendAndSyncSingleBatchRecordCountIT pins the coalesced record-count contract with cross-cluster cell-level equality.testSyncMetricsEmittedunit test — deterministic replacement for the flaky IT metrics assertion (injects an fsync delay to clear ms-truncation; reads metrics only after a graceful close that joins the consumer).LogFileCodecTestround-trip coverage for the new format (multi-family ordering, empty-record rejection, all cell types).testReplicationSyncPathSimulator(opt-in) for sync-coalescing measurement under contention.Verification
phoenix-core-servercompiles clean;MutationCellGrouperTest,LogFileCodecTest, and the replication unit tests pass.mvn spotless:checkpasses on bothphoenix-core-serverandphoenix-core.