Skip to content

perf: optimize DrainBatchesForOneNode O(n) re-enqueue scan on leader migration #577

@thomhurst

Description

@thomhurst

Context

PR #576 added a re-enqueue scan in DrainBatchesForOneNode (RecordAccumulator.cs) to handle the case where a partition leader migrates between Ready() and Drain(). When GetPartitionsForNode(nodeId) returns empty, all non-empty partition deques are re-enqueued to _readyPartitions.

Concern

The O(n) scan over _partitionDeques runs inside the send loop. While the partitions.Count == 0 condition is expected to be transient (leader migration), it can also fire for:

  • Newly-created topics where metadata hasn't populated yet
  • Stale metadata after broker disconnection

For producers routing to many topics/partitions, this linear scan could add latency.

Possible improvements

  1. Narrow the condition: distinguish genuine leader migrations from stale metadata (e.g., check if the node was recently known to own partitions)
  2. Track which partitions Ready() consumed per node: only re-enqueue those specific partitions instead of scanning all deques
  3. Cap the scan: limit re-enqueue to partitions that were recently sealed (within the last N seconds)

Current behavior

The scan is harmless correctness-wise — duplicate enqueues are documented as safe, and the scan only fires on the rare partitions.Count == 0 path. This is a performance optimization, not a bug fix.

Introduced in PR #576, commit 09979d2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium priorityperformancePerformance improvements

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions