Skip to content

[None][fix] Guard CUDA event elapsed_time in perf_metrics_manager to prevent executor crash #12868

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yifjiang wants to merge 4 commits into
base: main
Choose a base branch
from
+22 −8
Open

[None][fix] Guard CUDA event elapsed_time in perf_metrics_manager to prevent executor crash #12868

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
77db9e3
fix: guard CUDA event elapsed_time in perf_metrics_manager to prevent…
yifjiang Apr 13, 2026
9bd2416
fix: add logger.warning when CUDA event elapsed_time fails
yifjiang Apr 13, 2026
ea1fbd0
Merge branch 'main' into fix/perf-metrics-cuda-event-crash
pcastonguay Apr 14, 2026
4eaecc8
Merge branch 'main' into fix/perf-metrics-cuda-event-crash
pcastonguay Apr 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 22 additions & 8 deletions tensorrt_llm/_torch/pyexecutor/perf_metrics_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

import torch

from tensorrt_llm.logger import logger
from tensorrt_llm.serve.responses_utils import get_steady_clock_now_in_seconds

from .llm_request import PerfTimingInfo
Expand Down Expand Up @@ -167,14 +168,27 @@ def compute_batch_gpu_times(self, requests):
perf.gpu_forward_end_event.synchronize()
if perf.gpu_sample_end_event and not perf.gpu_sample_end_event.query():
perf.gpu_sample_end_event.synchronize()
batch_gpu_forward_time = perf.gpu_forward_start_event.elapsed_time(
perf.gpu_forward_end_event
)
batch_gpu_sample_time = (
perf.gpu_forward_end_event.elapsed_time(perf.gpu_sample_end_event)
if perf.gpu_sample_end_event
else 0.0
)
try:
batch_gpu_forward_time = perf.gpu_forward_start_event.elapsed_time(
perf.gpu_forward_end_event
)
batch_gpu_sample_time = (
perf.gpu_forward_end_event.elapsed_time(perf.gpu_sample_end_event)
if perf.gpu_sample_end_event
else 0.0
)
except RuntimeError as e:
# CUDA event timing can fail if events were not recorded
# on the current stream. Skip metrics for this batch rather
# than crashing the executor thread.
logger.warning(
"Failed to compute GPU event elapsed_time: %s. "
"Setting batch GPU times to 0.0. This may indicate "
"an issue with the forward pass or stream synchronization.",
e,
)
batch_gpu_forward_time = 0.0
batch_gpu_sample_time = 0.0

target["gpu_forward_time"] = batch_gpu_forward_time
target["gpu_sample_time"] = batch_gpu_sample_time
Expand Down
Loading