[WIP][protocol][common][vpj] Activate StoreMetaValue v44 and add VPJ dual-write gating predicate#2821
Draft
sixpluszero wants to merge 1 commit into
Draft
Conversation
…write gating predicate Removes the StoreMetaValue v43 versionOverride so storageMode is readable in Java; AdminOperation v98 stays pinned (admin-tool wiring lands in a follow-up). Adds the StorageMode enum (INTERNAL/DUAL_WRITE/EXTERNAL), exposes getStorageMode/setStorageMode on the Version interface and its two implementations, and wires the field into ReadOnlyStore.convertVersion() so it round-trips through serialization. Adds push.job.external.storage.writer.class and a static helper ExternalStorageWriteUtils.isDualWriteToExternalStorageFromVpjEnabled that returns true iff the config is non-empty AND the target version's storageMode is DUAL_WRITE. The writer SPI and BufferedDualWriter integration are intentionally deferred to a follow-up PR; nothing calls this predicate yet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Statement
PR #2814 staged
storageModeandexternalStorageReadModeonStoreMetaValue v44/AdminOperation v99, but pinned both viaversionOverridesso neither field is readable from Java today. The Venice Batch-Only Store Dual-Write design (separate doc) needs VPJ to consult the per-versionstorageModeat job-setup time and decide whether to engage the dual-write code path. Until v44 is active and there is a small SPI-style config knob on the VPJ side, no follow-up wiring can land.This PR is the minimal scaffolding: activate v44, expose
storageModethrough the JavaVersionmodel, define a VPJ config key for the external-storage writer impl, and add a single static predicate that combines the two halves of the gate. The actual writer interface andAbstractVeniceWriterwrapper are deliberately deferred to the follow-up PR — this change is gating-only and has no runtime effect until that wiring lands.Solution
StoreMetaValue v44is now active (AdminOperation v99stays pinned)build.gradleremoves theStoreMetaValue v43override.AdminOperation v98stays pinned — admin-tool / controller wiring for the store-level setter is a separate follow-up. Activating v44 makes three new fields readable in Java:targetRegionPromotedandexternalStorageReadModestay dormant (no Java reads them yet); onlystorageModeis consumed by this PR.Java
StorageModeenumcom.linkedin.venice.meta.StorageMode—INTERNAL(0),DUAL_WRITE(1),EXTERNAL(2). ImplementsVeniceEnumValuewith the strict-throwvalueOf(int)pattern, mirroringCompressionStrategy.Version.getStorageMode()/setStorageMode()Added to the
Versioninterface and implemented onVersionImpl(delegates tostoreVersion.storageMode) andReadOnlyStore.ReadOnlyVersion(setter throws, matching the existingsetCompressionStrategypattern).ReadOnlyStore.convertVersion()also writes the value back intoStoreVersionso the field round-trips through Store ↔ StoreVersion serialization.VPJ config key
push.job.external.storage.writer.class— fully-qualified class name of the external-storage writer impl. Defined inVenicePushJobConstants. Presence of a non-empty value is the VPJ-side half of the gating predicate. The class itself will be loaded reflectively by the follow-up PR; this PR only checks the config string is non-empty.Gating predicate
ExternalStorageWriteUtils.isDualWriteToExternalStorageFromVpjEnabled(VeniceProperties, Version)returnstrueiff:version.getStorageMode() == StorageMode.DUAL_WRITE.Null-safe on both arguments. Nothing calls this helper yet — the follow-up PR will plug it into the Spark-mode
createBasicVeniceWriter()path.Code changes
push.job.external.storage.writer.class(default empty/unset — gating predicate staysfalse). Plus the per-versionstorageModefield whose default isINTERNAL(0). With both at defaults today's behavior is unchanged.Concurrency-Specific Checks
N/A — this PR only adds an enum, an immutable static predicate, and accessors on existing model classes. No new threads, locks, or shared state.
How was this PR tested?
StorageModeTestextendsVeniceEnumValueTest<StorageMode>(mirrorsCompressionStrategyTest): verifies the int↔enum mapping is stable and out-of-bound values throw.ExternalStorageWriteUtilsTest: 8 cases covering both halves set, writer-class missing / empty, modeINTERNAL/EXTERNAL/ default, nullVersion, nullVeniceProperties.storageModedefaults to0(INTERNAL), so existing stores' behavior is unchanged.targetRegionPromoted(false) andexternalStorageReadMode(0,VENICE_ONLY) also default-zero; no Java code reads them yet../gradlew compileJava compileTestJava— clean across all modules; no downstream breakage from the v44 unpin or theVersioninterface extension../gradlew :internal:venice-common:test --tests 'com.linkedin.venice.meta.StorageModeTest'— passes../gradlew :clients:venice-push-job:test --tests 'com.linkedin.venice.vpj.ExternalStorageWriteUtilsTest'— 8/8 pass.Version/Store serialization tests — all pass:TestVersion,ReadOnlyStoreTest,TestZKStore,VersionJsonSerializerTest,ReadOnlyViewStoreTest,StoreVersionInfoTest,NameRepositoryTest.Does this PR introduce any user-facing or breaking changes?
StoreMetaValue v44exposes three new fields whose defaults preserve current behavior.