[kafka_deserializers] New plugin pack: msgpack format + payload compression codecs#3001
Open
piochelepiotr wants to merge 3 commits intoDataDog:masterfrom
Open
[kafka_deserializers] New plugin pack: msgpack format + payload compression codecs#3001piochelepiotr wants to merge 3 commits intoDataDog:masterfrom
piochelepiotr wants to merge 3 commits intoDataDog:masterfrom
Conversation
Plugin pack for the kafka_actions check. Does not register a runtime
check: contributes additional capabilities to kafka_actions via the
'datadog_kafka_actions.formats' and 'datadog_kafka_actions.compressions'
entry-point groups introduced in datadog-kafka-actions 2.7.0.
Adds:
- 'msgpack' format handler
- 'gzip', 'zlib', 'snappy', 'lz4', 'lz4_dd_hdr', 'zstd' compression codecs
The 'lz4_dd_hdr' codec covers the DataDog/golz4 framing (4-byte LE
uncompressed-size header + raw LZ4 block) used by xray-converter; it is
not interchangeable with the standard LZ4 frame format ('lz4').
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Use 2026-present in license headers to match repo standard - Drop hard pyproject dep on datadog-kafka-actions (unresolvable on PyPI until the core plugin API ships; both wheels coexist in the agent's embedded Python at runtime) - Add _compat module with stub fallback for FormatHandler / CompressionCodec so unit tests run in environments where kafka_actions is not installed; the real classes are imported when available Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add CODEOWNERS entry - Shorten manifest/pyproject descriptions and simplify README to avoid ddev validate readmes shell-quote tripping - Use importlib (not 'from datadog_checks.kafka_actions import ...') to load the host plugin base classes at runtime, satisfying ddev validate imports' constraint that integrations-extras packages not statically reference other integrations' namespaces Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a new package
kafka_deserializersto integrations-extras. It does not register a runtime check; it is a plugin wheel that contributes additional capabilities to thekafka_actionscheck via the entry-point groups introduced in integrations-core PR DataDog/integrations-core#23650 (datadog-kafka-actions>=2.7.0).Once the wheel is installed into the agent's embedded Python,
kafka_actionsdiscovers the new capabilities automatically.msgpackgzip,zlib,snappy,lz4,lz4_dd_hdr,zstdThe
lz4_dd_hdrcodec covers the DataDog/golz4 framing (4-byte little-endian uncompressed-size header followed by raw LZ4 block bytes) used bycloud-integrations/aws/xray-converter. It is not interchangeable with the standard LZ4 frame format — uselz4for that.Motivation
Producers across
dd-goanddd-sourceship Kafka payloads in formats and codecs that the corekafka_actionscheck does not currently support — most prominently MessagePack values, gzip-compressed JSON, zstd-compressed payloads, and the xray-converter LZ4 framing. Each of these depends on a third-party Python wheel (msgpack,python-snappy,lz4,zstandard) that has no business living inside the core check.Putting these in a separate plugin wheel keeps core dependency-light and lets us iterate on format/codec coverage on its own release cadence.
Configuration
The new config keys live on
kafka_actions(introduced in PR #23650). Oncedatadog-kafka-deserializersis installed:Depends on
datadog-kafka-actions>=2.7.0.Review checklist
🤖 Generated with Claude Code