[server][dvc] Fail-fast on blob-transfer PartitionState/StoreVersionState schema-version mismatch#2811
Open
jingy-li wants to merge 5 commits into
Open
[server][dvc] Fail-fast on blob-transfer PartitionState/StoreVersionState schema-version mismatch#2811jingy-li wants to merge 5 commits into
jingy-li wants to merge 5 commits into
Conversation
The previous commit added a fast-fail check on the metadata response, but the server's response order is files first, metadata last — so a client that catches the mismatch at the metadata stage has already paid for the entire file transfer. This commit moves the primary check to the request side, modeled on the existing snapshot-table-format check next to it. Client `prepareRequest` now stamps the two schema-version headers on the GET. Server `channelRead0` calls a new `BlobTransferUtils.compareRequestedSchemaVersionsAgainstLocal(...)` right after the table-format validation. On mismatch it returns 400 BAD_REQUEST with an `X-Blob-Transfer-Schema-Mismatch: true` marker header and echoes its own protocol versions in the response, BEFORE any file work begins. Client `P2PFileTransferClientHandler.channelRead0` recognizes the marker on a non-OK response and throws the typed `VeniceBlobTransferIncompatibleSchemaException` with full peer-vs-local context (peer = server's local versions echoed in the rejection). Backward compat preserved: a request without the new headers (older client) is not rejected — the server falls through to the existing flow. The response-side check from the prior commit stays as the safety net for the inverse case (older server that does not yet validate requests will still send full metadata; new client catches at metadata-parse time instead of waiting for the receive timeout). Tests cover: server rejects mismatched request with 400 + marker + no file responses; server doesn't reject requests without the headers (backward compat); client builds the typed exception from the rejection response with correct peer/local versions populated. Existing integration tests in TestNettyP2PBlobTransferManager (real client↔server flow) all pass unchanged because the new client always advertises versions that match the new server. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> [blob-transfer] add fast-fail schema version check on metadata response The P2P blob-transfer client deserialized the peer's PartitionState bytes inside P2PMetadataTransferHandler without a SchemaReader, so any peer that serialized PartitionState with a protocol version higher than this binary knows triggered VeniceMessageException at the Netty pipeline tail after the body had been fully transferred. The transfer future was never failed explicitly, so the replica had to wait for blobReceiveTimeoutInMin before falling back to Kafka bootstrap. We saw this on ltx1-app12860.stg with PartitionState v21 during the rolling deploy that introduced PR linkedin#2707. Server now stamps two new headers on the metadata response: X-Blob-Transfer-Partition-State-Schema-Version X-Blob-Transfer-Store-Version-State-Schema-Version each carrying the local AvroProtocolDefinition.X.getCurrentProtocolVersion(). Client compares peer's value to its own current version at HTTP header-parse time, before any body is consumed, and throws the new VeniceBlobTransferIncompatibleSchemaException on mismatch. Throwing from channelRead0 flows through the existing exceptionCaught -> completeExceptionally -> ctx.close() path, so the per-host transfer future fails immediately and the orchestrator can pick the next peer (or fall back to Kafka) without waiting on the receive timeout. Policy is exact equality. Blob transfer is the fast path; if binaries are not in lock-step we'd rather skip P2P and let Kafka handle bootstrap than rely on cross-version metadata promotion. Skew is bounded to rolling-deploy windows. Backward-compat is preserved during rollout: a missing header passes through (peer not yet upgraded), and a malformed/out-of-range header logs a warning and passes through (parse bug must not crash the channel). The existing deserialize-time exception in InternalAvroSpecificSerializer remains as the safety net for the truly incompatible case, so no regression vs. today. Tests cover: header stamped on metadata response only (not on file responses), known-version pass-through, mismatched PartitionState fails fast, mismatched StoreVersionState fails fast, single-header-missing pass-through, and malformed/out-of-range pass-through. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VeniceBlobTransferIncompatibleSchemaException Covers the constructor, all getters, message formatting, and both branches of render() (known version and VERSION_UNKNOWN sentinel). Bumps venice-client-common diff branch coverage from 91.6% to 92.74%.
testIsVeniceException compared a VeniceBlobTransferIncompatibleSchemaException against VeniceException via instanceof. Since the class extends VeniceException, the relationship is enforced at compile time and SpotBugs flags the runtime check as vacuous (BC_VACUOUS_INSTANCEOF). The remaining three tests still cover the constructor, all getters, both branches of render(), and the VERSION_UNKNOWN sentinel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Statement
P2P blob transfer ships RocksDB snapshot files plus a BlobTransferPartitionMetadata payload that embeds Avro-serialized
PartitionState (offset record) and StoreVersionState. Today the two peers do not negotiate the protocol versions they used to serialize that metadata, the receiver only discovers a mismatch when it tries to deserialize the body, which is:
This is the fast path during rolling deploys, where peer-vs-local binary skew on PartitionState / StoreVersionState versions is exactly when this misfires.
Solution
Both sides (client/server) advertise their compiled-in AvroProtocolDefinition.{PARTITION_STATE,STORE_VERSION_STATE}.getCurrentProtocolVersion() and check at the earliest possible point:
Policy is exact equality (not "peer ≤ local"): blob transfer is the fast path; on skew we'd rather step aside to Kafka bootstrap than rely on cross-version Avro promotion of partition metadata.
Rolling-deploy compatibility is intentional:
Code changes
Concurrency-Specific Checks
Both reviewer and PR author to verify
synchronized,RWLock) are used where needed.ConcurrentHashMap,CopyOnWriteArrayList).How was this PR tested?
Does this PR introduce any user-facing or breaking changes?