Skip to content

🛠 Fix: Properly retrieve exceptions after graceful task cancellation#1991

Closed
gigaverse-oz wants to merge 4 commits into
livekit:mainfrom
gigaverse-oz:fix-task-cancellation
Closed

🛠 Fix: Properly retrieve exceptions after graceful task cancellation#1991
gigaverse-oz wants to merge 4 commits into
livekit:mainfrom
gigaverse-oz:fix-task-cancellation

Conversation

@gigaverse-oz
Copy link
Copy Markdown

Summary

This PR improves the gracefully_cancel (now called cancel_and_wait) function by ensuring that exceptions from cancelled or completed tasks are explicitly retrieved. This prevents asyncio from logging warnings like:

_GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=CancelledError()>

Root Cause

Previously, gracefully_cancel:

  • Cancelled tasks
  • Waited for their completion via internal waiters
  • Did not retrieve .exception() from the tasks

When asyncio.gather() is used, it returns a _GatheringFuture, which wraps multiple tasks. If the gathered task is cancelled or raises an exception and that exception is never accessed, asyncio logs a warning — even if the tasks are awaited indirectly.

Fix

After all futures are awaited, we now iterate over each future:

  • Call fut.exception() to mark the exception as retrieved

Example

Previously:

await gracefully_cancel(*tasks)
# Logs: _GatheringFuture exception was never retrieved

Now:

await gracefully_cancel(*tasks)
# No warning – all exceptions are properly retrieved

Impact

  • Silences unwanted _GatheringFuture warnings in normal cancellation flows
  • Improves robustness and clarity during shutdown
  • No behavioral change to existing application logic

Gather all the tasks output when cancelled. Even asyncio.CancelledError exceptions.
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Apr 14, 2025

CLA assistant check
All committers have signed the CLA.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 14, 2025

❌ Invalid Changeset Format Detected

One or more changeset files in this PR have an invalid format. Please ensure they adhere to:

  • Start with --- and include a closing --- on its own line.
  • Each package line must be in the format:
    "package-name": patch|minor|major
  • No duplicate package entries allowed.
  • A non-empty change description must follow the front matter.

Error details:
.github/next-release/changeset-5ca20d29.md: Failed to read file from git branch 'pr_head'.

@davidzhao davidzhao requested a review from theomonnom April 14, 2025 16:20
@theomonnom
Copy link
Copy Markdown
Member

Hey, is there a way to easily reproduce this, which plugin were you using? I wonder if it might be more effective to check where we're actually using asyncio.gather.

@gigaverse-oz
Copy link
Copy Markdown
Author

Hi @theomonnom,

I used deepgram, stt. This is the place that have the issue:

You create 3 tasks, have only try-finally and use asyncio.wait with asyncio.FIRST_COMPLETED.
When I close the STT, the tasks get asyncio.TaskCancelled exception that is not catched (you exit the "wait" when the first one raises).
You "clean" the function by calling await utils.aio.gracefully_cancel(*tasks, wait_reconnect_task)
Simple solution will be also to do asyncio.gethar(*tasks) in the finally, but I assumed the issue might repeat in more places/plugins.

I'd be happy with either solutions as long it can be soon. It litters our logs :-(

@theomonnom theomonnom closed this May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants