Skip to content

SentinelChannel: add backward-compatible fanout for mixed-version clusters#2457

Open
TheCodingLand wants to merge 4 commits intocelery:mainfrom
TheCodingLand:fix/sentinel-fanout-compat
Open

SentinelChannel: add backward-compatible fanout for mixed-version clusters#2457
TheCodingLand wants to merge 4 commits intocelery:mainfrom
TheCodingLand:fix/sentinel-fanout-compat

Conversation

@TheCodingLand
Copy link

@TheCodingLand TheCodingLand commented Feb 7, 2026

Commit 06a3b23 fixed SentinelChannel to format keyprefix_fanout with the actual database number (e.g. /{db}./0.), aligning sentinel with the base Channel behavior.

However, this changed the PUB/SUB topic that sentinel workers use for fanout (broadcast) messages, which means workers running kombu < 5.4.0 and >= 5.4.0 publish and subscribe to different Redis channels and can no longer exchange control commands (ping, shutdown, rate_limit, etc.) during rolling upgrades.

This commit adds a sentinel_fanout_compat transport option (default False) that, when explicitly enabled:

  • Publishes fanout messages to both the new formatted topic (/0.exchange) and the legacy literal topic (/{db}.exchange)
  • Subscribes to both topic patterns
  • Deduplicates messages received on both topics using a bounded hash set

This allows mixed-version clusters to communicate during rolling upgrades. Users performing a rolling upgrade should set sentinel_fanout_compat=True in their transport_options until all workers run kombu >= 5.4.0, then remove it.

app = Celery('myapp', broker='sentinel://sentinel:26379/0',
             broker_transport_options={
                 'master_name': 'mymaster',
                 'sentinel_fanout_compat': True,  # enable during rolling upgrade
             })

Closes #2152

@TheCodingLand TheCodingLand force-pushed the fix/sentinel-fanout-compat branch from 943395e to 8f53bff Compare February 7, 2026 20:02
@auvipy auvipy requested review from auvipy and Copilot and removed request for Copilot February 8, 2026 12:45
@codecov
Copy link

codecov bot commented Feb 8, 2026

Codecov Report

❌ Patch coverage is 82.41758% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.12%. Comparing base (92006e7) to head (2230ac9).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
kombu/transport/redis.py 82.41% 5 Missing and 11 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2457      +/-   ##
==========================================
+ Coverage   81.11%   81.12%   +0.01%     
==========================================
  Files          77       77              
  Lines        9738     9828      +90     
  Branches     1098     1119      +21     
==========================================
+ Hits         7899     7973      +74     
- Misses       1631     1636       +5     
- Partials      208      219      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in Redis Sentinel fanout compatibility mode so mixed kombu versions (pre/post 5.4.0 topic formatting change) can still exchange broadcast/control commands during rolling upgrades.

Changes:

  • Introduces sentinel_fanout_compat transport option on SentinelChannel to dual-publish and dual-subscribe to both formatted (/0.exchange) and legacy (/{db}.exchange) topics.
  • Implements deduplication logic to avoid double-delivery when listening on both topics.
  • Adds unit tests covering default behavior, dual publish/subscribe, unsubscribe, and dedup behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
kombu/transport/redis.py Adds sentinel_fanout_compat option and SentinelChannel logic for dual topic fanout + dedup.
t/unit/transport/test_redis.py Adds tests validating compat option behavior, topic selection, and dedup.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@TheCodingLand TheCodingLand force-pushed the fix/sentinel-fanout-compat branch from 8f53bff to 0ea47cd Compare February 8, 2026 13:26
@TheCodingLand
Copy link
Author

I updated the PR based on review feedback

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +1629 to +1639
# -- deduplicate using delivery_tag
tag = message.get('properties', {}).get(
'delivery_tag')
if tag is None:
# Fallback for non-standard messages: use
# hash of the raw serialised bytes.
raw = payload['data']
if isinstance(raw, str):
raw = raw.encode()
tag = hash(raw)
if tag in self._seen_fanout_payloads:
Copy link

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deduplication keys are currently just delivery_tag (or hash(raw) fallback). If two different fanout exchanges ever reuse the same delivery_tag (or if the hash fallback collides / same payload bytes are published on two exchanges), messages could be incorrectly dropped. Consider namespacing the dedup key by exchange/channel (e.g., (exchange, delivery_tag)), and for the fallback prefer a stable digest of the raw bytes plus channel instead of Python’s salted hash().

Copilot uses AI. Check for mistakes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TheCodingLand can you please cross check the suggestions? and share your findings

Comment on lines +1557 to +1563
super()._put_fanout(exchange, message, routing_key, **kwargs)
if self._compat_enabled:
legacy_topic = self._get_legacy_publish_topic(
exchange, routing_key)
with self.conn_or_acquire() as client:
client.publish(legacy_topic, dumps(message))

Copy link

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When compat is enabled, _put_fanout publishes via super()._put_fanout(...) and then opens a second conn_or_acquire() context to publish the legacy topic. This results in two separate client acquisitions per fanout publish. Consider acquiring a client once and publishing to both topics on the same connection to reduce overhead.

Copilot uses AI. Check for mistakes.
auvipy and others added 3 commits February 10, 2026 14:06
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, please fix the merge conflict carefully

Copy link
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please, fix the merge conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kombu version >= 5.4.0rc seem to break celery control commands from celery with versions <5.4.0rc

3 participants