feat(kafka-logger): add configurable api_version for Apache Kafka 4.x support#13040
Open
macdoor wants to merge 14 commits intoapache:masterfrom
Open
feat(kafka-logger): add configurable api_version for Apache Kafka 4.x support#13040macdoor wants to merge 14 commits intoapache:masterfrom
macdoor wants to merge 14 commits intoapache:masterfrom
Conversation
… support - Set default api_version to 2 (Kafka 4.x drops magic0/magic1) - Make api_version configurable (0, 1, 2) for Kafka < 0.10.0.0 compatibility - Add api_version to broker_config when creating producer fix apache#12984 Made-with: Cursor
dosubot
bot
added
size:M
Baoyuantop
reviewed
Feb 27, 2026
added 3 commits
February 27, 2026 16:03
- Revert CHANGELOG.md (maintainers update it during release) - Set api_version default to 0 for backward compatibility - Use conf.api_version directly without fallback Made-with: Cursor
Avoid passing api_version=nil to producer config when not specified, let lua-resty-kafka use its default (0) for backward compatibility. Made-with: Cursor
… line length (100) Made-with: Cursor
Baoyuantop
reviewed
Feb 28, 2026
Baoyuantop
reviewed
Feb 28, 2026
Made-with: Cursor
dosubot
bot
added
size:L
added 3 commits
February 28, 2026 13:13
Made-with: Cursor
…hema guarantee Made-with: Cursor
… numbering
- Switch Kafka 4.x image from apache/kafka to bitnamilegacy/kafka:4.0.0
- Add single-node replication factor settings for KRaft
- Simplify topic creation retry loop to {1..20}
- Run reindex to fix kafka-logger.t test numbering for lint
Made-with: Cursor
macdoor
force-pushed
the
feat/kafka-logger-api-version
branch
from
e98db71 to
1812f51
Compare
The bitnamilegacy/kafka:4.0.0 KRaft broker was not starting because KAFKA_ENABLE_KRAFT and KAFKA_KRAFT_CLUSTER_ID were missing. This caused the topic creation retry loop (20 × 60s timeout) to block all other service init for ~22 minutes, making the CI exceed its time limit. - Add KAFKA_ENABLE_KRAFT, KAFKA_KRAFT_CLUSTER_ID, ALLOW_PLAINTEXT_LISTENER - Move Kafka 4.x topic creation to end of init script - Reduce retry count from 20 to 10 Made-with: Cursor
Baoyuantop
reviewed
Mar 4, 2026
added 3 commits
March 4, 2026 18:10
…default - Switch Kafka 4.x test broker to official apache/kafka:4.0.0 image with KRaft configuration based on upstream docker examples - Call kafka-topics.sh via PATH inside the container for Kafka 4.x topic creation - Set api_version schema default to 1 and update comments/docs so that the default matches lua-resty-kafka while still documenting that 2 is required for Kafka 4.x Made-with: Cursor
Made-with: Cursor
Made-with: Cursor
Baoyuantop
previously approved these changes
Mar 6, 2026
There was a problem hiding this comment.
Pull request overview
Adds an api_version configuration option to the kafka-logger plugin to support newer Kafka Produce API versions (notably Kafka 4.x), and updates CI/test infrastructure and docs accordingly.
Changes:
- Add
api_versionto the kafka-logger plugin schema and pass it through to the lua-resty-kafka producer config. - Extend plugin tests to validate
api_versionschema and add an integration test targeting Kafka 4.x withapi_version=2. - Update docs (EN/ZH) and CI docker-compose/init scripts to provision a Kafka 4.x broker for verification.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
apisix/plugins/kafka-logger.lua |
Introduces api_version schema field and forwards it into broker_config when creating the producer. |
t/plugin/kafka-logger.t |
Adds schema validation tests for api_version and an integration path exercising Kafka 4.x with api_version=2. |
docs/en/latest/plugins/kafka-logger.md |
Documents the new api_version option and its intended usage for Kafka 4.x. |
docs/zh/latest/plugins/kafka-logger.md |
Same as EN docs, in Chinese. |
ci/pod/docker-compose.plugin.yml |
Adds a Kafka 4.x service used by plugin CI. |
ci/init-plugin-test-service.sh |
Adds topic creation/verification steps for the Kafka 4.x service. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…r --list (Copilot review) Made-with: Cursor
The 3-listener setup with Docker hostname in CONTROLLER_QUORUM_VOTERS caused DNS resolution race during early KRaft bootstrap, leaving the broker stuck at "available brokers: 0". - Reduce to 2 listeners (PLAINTEXT + CONTROLLER) - Use localhost for controller quorum voters (always resolvable) - Use localhost:9092 for topic creation inside the container - Add docker logs output for debugging broker startup failures Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
api_versionis 1 (same as lua-resty-kafka). Setapi_versionto 2 for Apache Kafka 4.x (Kafka 4.x drops magic0/magic1).api_versionconfigurable (0, 1, 2) for compatibility with different Kafka versions.api_versiontobroker_configwhen creating the producer.Description
Which issue(s) this PR fixes:
Fixes #12984
Fixes #11811
Checklist