[BugFix] Fix process_response_dict to support async in serving_completion #5758

kxz2002 · 2025-12-25T02:49:15Z

Motivation

The process_response_dict function in serving_completion.py does not support asynchronous calls, which causes issues when using EB5's processor.

💡 If this PR is a Cherry Pick, the PR title needs to follow the format by adding the [Cherry-Pick] label at the very beginning and appending the original PR ID at the end. For example, [Cherry-Pick][CI] Add check trigger and logic(#5191)

💡 如若此PR是Cherry Pick，PR标题需遵循格式，在最开始加上[Cherry-Pick]标签，以及最后面加上原PR ID，例如[Cherry-Pick][CI] Add check trigger and logic(#5191)

Modifications

Add a check to determine whether the process_response_dict function is an asynchronous call, enabling support for asynchronous invocations.

Usage or Command

no change in user command.

Accuracy Tests

no need

Checklist

Add at least a tag in the PR title.
- Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
- You can add new tags based on the PR content, but the semantics must be clear.
Format your code, run pre-commit before commit.
Add unit tests. Please write the reason in this PR if no unit tests.
Provide accuracy results.
If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

paddle-bot · 2025-12-25T02:49:21Z

Thanks for your contribution!

codecov-commenter · 2025-12-25T04:18:00Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@01c18f3). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #5758   +/-   ##
==========================================
  Coverage           ?   65.51%           
==========================================
  Files              ?      337           
  Lines              ?    43082           
  Branches           ?     6639           
==========================================
  Hits               ?    28225           
  Misses             ?    12754           
  Partials           ?     2103

Flag	Coverage Δ
GPU	`65.51% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR fixes a bug where the process_response_dict function in serving_completion.py did not support asynchronous data processors. The fix adds runtime checks to determine if the function is a coroutine and appropriately awaits it if needed.

Adds inspect.iscoroutinefunction() checks to detect async processors
Handles both sync and async code paths for process_response_dict calls
Enables compatibility with EB5's processor which requires async support

Copilot · 2025-12-25T11:31:33Z

fastdeploy/entrypoints/openai/serving_completion.py

+                    if inspect.iscoroutinefunction(self.engine_client.data_processor.process_response_dict):
+                        await self.engine_client.data_processor.process_response_dict(
+                            res, stream=True, include_stop_str_in_output=request.include_stop_str_in_output
+                        )
+                    else:
+                        self.engine_client.data_processor.process_response_dict(
+                            res, stream=True, include_stop_str_in_output=request.include_stop_str_in_output
+                        )


The inspect.iscoroutinefunction check is performed on every iteration inside the loop, which can impact performance. Consider caching this check result in the __init__ method of the class as an instance variable (e.g., self._is_process_response_dict_async), and then use the cached value here. This would eliminate the repeated introspection overhead during request processing.

Copilot · 2025-12-25T11:31:33Z

fastdeploy/entrypoints/openai/serving_completion.py

+                    if inspect.iscoroutinefunction(self.engine_client.data_processor.process_response_dict):
+                        await self.engine_client.data_processor.process_response_dict(
+                            data, stream=False, include_stop_str_in_output=request.include_stop_str_in_output
+                        )
+                    else:
+                        self.engine_client.data_processor.process_response_dict(
+                            data, stream=False, include_stop_str_in_output=request.include_stop_str_in_output
+                        )


This code block is duplicated across two locations in the file (lines 318-325 and lines 513-520). Consider extracting this logic into a helper method (e.g., async def _call_process_response_dict) to improve maintainability and ensure consistency if the logic needs to change in the future.

Copilot · 2025-12-25T11:31:33Z

fastdeploy/entrypoints/openai/serving_completion.py

+                    if inspect.iscoroutinefunction(self.engine_client.data_processor.process_response_dict):
+                        await self.engine_client.data_processor.process_response_dict(
+                            data, stream=False, include_stop_str_in_output=request.include_stop_str_in_output
+                        )
+                    else:
+                        self.engine_client.data_processor.process_response_dict(
+                            data, stream=False, include_stop_str_in_output=request.include_stop_str_in_output
+                        )


The new async path for process_response_dict is not covered by tests. Since this file has comprehensive test coverage in tests/entrypoints/openai/test_serving_completion.py, consider adding test cases that verify the async code path works correctly. This would involve mocking process_response_dict as an async function using AsyncMock and verifying it gets called with await.

Copilot · 2025-12-25T11:31:34Z

fastdeploy/entrypoints/openai/serving_completion.py

+                    if inspect.iscoroutinefunction(self.engine_client.data_processor.process_response_dict):
+                        await self.engine_client.data_processor.process_response_dict(
+                            data, stream=False, include_stop_str_in_output=request.include_stop_str_in_output
+                        )
+                    else:
+                        self.engine_client.data_processor.process_response_dict(
+                            data, stream=False, include_stop_str_in_output=request.include_stop_str_in_output
+                        )


The inspect.iscoroutinefunction check is performed on every iteration inside the loop, which can impact performance. Consider caching this check result in the __init__ method of the class as an instance variable (e.g., self._is_process_response_dict_async), and then use the cached value here. This would eliminate the repeated introspection overhead during request processing.

LiqinruiG

LGTM

LiqinruiG

LGTM

…tion (PaddlePaddle#5758) * support process_response_dict async initial commit * fixbug * add unit test * optimize

…tion (#5758) (#5802) * support process_response_dict async initial commit * fixbug * add unit test * optimize

kxz2002 added 2 commits December 25, 2025 10:10

support process_response_dict async initial commit

6dad5be

fixbug

3203134

kxz2002 temporarily deployed to Metax_ci December 25, 2025 02:49 — with GitHub Actions Inactive

paddle-bot bot added the contributor External developers label Dec 25, 2025

Jiang-Jia-Jun requested a review from Copilot December 25, 2025 11:28

Copilot started reviewing on behalf of Jiang-Jia-Jun December 25, 2025 11:28 View session

Copilot AI reviewed Dec 25, 2025

View reviewed changes

add unit test

d1c5e09

kxz2002 had a problem deploying to Metax_ci December 26, 2025 05:05 — with GitHub Actions Error

Merge branch 'develop' into bugfix/async_completion

8a640f5

kxz2002 temporarily deployed to Metax_ci December 26, 2025 05:06 — with GitHub Actions Inactive

LiqinruiG previously approved these changes Dec 26, 2025

View reviewed changes

optimize

60932b8

kxz2002 dismissed LiqinruiG’s stale review via 60932b8 December 26, 2025 07:51

kxz2002 temporarily deployed to Metax_ci December 26, 2025 07:51 — with GitHub Actions Inactive

LiqinruiG approved these changes Dec 26, 2025

View reviewed changes

LiqinruiG merged commit cad2932 into PaddlePaddle:develop Dec 26, 2025
18 of 20 checks passed

kxz2002 added a commit to kxz2002/FastDeploy that referenced this pull request Dec 26, 2025

[BugFix] Fix process_response_dict to support async in serving_comple…

f6deaa5

…tion (PaddlePaddle#5758) * support process_response_dict async initial commit * fixbug * add unit test * optimize

kxz2002 mentioned this pull request Dec 26, 2025

[Cherry-Pick] Fix process_response_dict to support async in serving_completion (#5758) #5802

Merged

5 tasks

LiqinruiG pushed a commit that referenced this pull request Dec 29, 2025

[BugFix] Fix process_response_dict to support async in serving_comple…

df775c2

…tion (#5758) (#5802) * support process_response_dict async initial commit * fixbug * add unit test * optimize

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[BugFix] Fix process_response_dict to support async in serving_completion #5758

[BugFix] Fix process_response_dict to support async in serving_completion #5758

kxz2002 commented Dec 25, 2025 •

edited

Loading

Uh oh!

paddle-bot bot commented Dec 25, 2025

Uh oh!

codecov-commenter commented Dec 25, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 25, 2025

Uh oh!

Copilot AI Dec 25, 2025

Uh oh!

Copilot AI Dec 25, 2025

Uh oh!

Copilot AI Dec 25, 2025

Uh oh!

LiqinruiG left a comment

Uh oh!

LiqinruiG left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[BugFix] Fix process_response_dict to support async in serving_completion #5758

[BugFix] Fix process_response_dict to support async in serving_completion #5758

Conversation

kxz2002 commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Usage or Command

Accuracy Tests

Checklist

Uh oh!

paddle-bot bot commented Dec 25, 2025

Uh oh!

codecov-commenter commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

Uh oh!

LiqinruiG left a comment

Choose a reason for hiding this comment

Uh oh!

LiqinruiG left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kxz2002 commented Dec 25, 2025 •

edited

Loading

codecov-commenter commented Dec 25, 2025 •

edited

Loading