Skip to content

KAFKA-20410: Add DLQ configuration parameters for Share Groups (KIP-1191)#21979

Open
sjhajharia wants to merge 2 commits intoapache:trunkfrom
sjhajharia:add-dlq-configs
Open

KAFKA-20410: Add DLQ configuration parameters for Share Groups (KIP-1191)#21979
sjhajharia wants to merge 2 commits intoapache:trunkfrom
sjhajharia:add-dlq-configs

Conversation

@sjhajharia
Copy link
Copy Markdown
Collaborator

@sjhajharia sjhajharia commented Apr 7, 2026

Summary

This PR adds the configuration foundation for Share Groups Dead-Letter
Queue (DLQ) functionality as specified in KIP-1191: Dead-letter queues
for share
groups
.

Changes

New Configurations

Cluster-level configs (ShareGroupConfig.java):

  1. errors.deadletterqueue.auto.create.topics.enable (default: false)
    • Enables automatic creation of DLQ topics
  2. errors.deadletterqueue.topic.name.prefix (default: "dlq.")
    • Required prefix for DLQ topic names when auto-create is enabled

Group-level configs (ShareGroupConfig.java):

  1. errors.deadletterqueue.topic.name (default: "")
    • Specifies the DLQ topic name for a share group
    • Empty string means DLQ disabled for that group
  2. errors.deadletterqueue.copy.record.enable (default: false)
    • When true: Copy full original record to DLQ
    • When false: Copy only context metadata (offset, delivery count,
      reason)

Topic-level config (TopicConfig.java):

  1. errors.deadletterqueue.group.enable (default: not set)
    • Marks a topic as eligible for use as a DLQ

Validation Logic

  • DLQ topic names must not start with __ (reserved for internal
    topics)
  • When errors.deadletterqueue.auto.create.topics.enable=true, the
    topic name must start with the configured prefix
  • When auto-create is disabled, topic name can be any valid Kafka topic
    name (no prefix requirement)
  • Blank topic name ("") is valid and indicates DLQ is disabled

Tests

Added 7 comprehensive test cases in ShareGroupConfigTest:

  • testDLQConfigDefaults() - Verify default values
  • testDLQConfigCustomValues() - Validate custom configurations
  • testDLQTopicNameCannotStartWithDoubleUnderscore() - Reject __
    prefix
  • testDLQTopicNameMustMatchPrefixWhenAutoCreateEnabled() - Enforce
    prefix with auto-create
  • testDLQTopicNamePrefixNotRequiredWhenAutoCreateDisabled() - Allow
    any name without auto-create
  • testDLQBlankTopicNameIsValid() - Accept blank as "DLQ disabled"
  • testDLQValidTopicNameWithAutoCreate() - Validate correct
    configuration

Reviewers: David Jacot david.jacot@gmail.com

@sjhajharia sjhajharia marked this pull request as ready for review April 7, 2026 11:28
@sjhajharia sjhajharia changed the title MINOR: Add DLQ configuration parameters for Share Groups (KIP-1191) KAFKA-20410: Add DLQ configuration parameters for Share Groups (KIP-1191) Apr 7, 2026
@sjhajharia sjhajharia added the KIP-932 Queues for Kafka label Apr 7, 2026
@github-actions github-actions bot added the core Kafka Broker label Apr 7, 2026
/** Dead Letter Queue Configurations (KIP-1191) **/

// Cluster-level DLQ configs
public static final String ERRORS_DEADLETTERQUEUE_AUTO_CREATE_TOPICS_ENABLE_CONFIG = "errors.deadletterqueue.auto.create.topics.enable";
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All configs starts with the same prefix. Why don’t we follow the same pattern for those?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are intentionally not including the group.share. prefix on these configs to keep them generic enough so that in the future, we can re-use them as and when needed and maintain compatibility still.

@github-actions github-actions bot removed the triage PRs from the community label Apr 8, 2026
@apoorvmittal10 apoorvmittal10 self-requested a review April 8, 2026 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants