fix: encode null as varint(-1) in VarIntBytes for correct Kafka tombstones#217
Merged
ShogunPanda merged 4 commits intoplatformatic:mainfrom Feb 20, 2026
Merged
Conversation
Contributor
|
Can you please sign the DCO? |
Contributor
|
Also, can you please fix CI? |
Contributor
Author
|
Sorry @ShogunPanda! Fixed the CI and signed the DCO. |
tburch
force-pushed
the
fix/tombstone-null-encoding
branch
from
e58118e to
0c740da
Compare
…tones Writer.appendVarIntBytes() was encoding null values as varint(0) (empty byte array) instead of varint(-1) (null) per the Kafka protocol spec. Reader.readVarIntBytes() had the symmetric bug, converting -1 back to empty buffer, masking the issue in pure @platformatic/kafka environments. This broke tombstone semantics for any non-@platformatic/kafka consumer: - Log compaction retains empty byte arrays instead of deleting keys - Kafka Connect AvroConverter throws BufferUnderflowException on 0-byte values - Any consumer expecting null for key deletion sees empty bytes instead Fixes platformatic#216 Signed-off-by: Tristan Burch <tristan@day.ai>
The protocol layer now correctly reads/writes null values, but the KafkaRecord TypeScript types still declared Buffer-only. This updates the types to Buffer | null and adds null coalescing in the consumer deserializer calls so null values from tombstone records are safely handled. Signed-off-by: Tristan Burch <tristan@day.ai>
tburch
force-pushed
the
fix/tombstone-null-encoding
branch
from
0c740da to
a70cd28
Compare
Contributor
|
@tburch I'm afraid you missed one entry. Once you fix it I'll approve and merge. |
…ltering The KafkaRecord key type is `Buffer | null`, but `#filterUncommittedMessages` accessed `batch.records[0].key.readInt16BE(2)` without a null check, causing TS2531. Use optional chaining to handle the case where key is null. Signed-off-by: Tristan Burch <tristan@day.ai>
Contributor
Author
|
@ShogunPanda pushed a fix that I think will take care of it. |
Contributor
|
@tburch Unfortunately it is still failing. |
readVarIntBytes() now returns Buffer | null, so all test and playground code that accesses record key/value/headers needs null-aware handling. Signed-off-by: Tristan Burch <tristan@day.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #216
Writer.appendVarIntBytes()encodes null values asvarint(0)(empty byte array) instead ofvarint(-1)(null) per the Kafka protocol specification.Reader.readVarIntBytes()has the symmetric bug — it converts-1back to an empty buffer, masking the issue when both sides use@platformatic/kafka.This breaks tombstone semantics for any non-
@platformatic/kafkaconsumer:AvroConverterthrowsBufferUnderflowExceptionon 0-byte valuesChanges
src/protocol/writer.ts—appendVarIntBytes(null)now writesvarint(-1)instead ofvarint(0)src/protocol/reader.ts—readVarIntBytes()now returnsnullwhen it readsvarint(-1), instead of converting to empty buffer. Return type updated toBuffer | null.Tests updated to match corrected behavior.