WIP : ADD: Batch request (Non-streaming) for OpenAI Plugin#66
WIP : ADD: Batch request (Non-streaming) for OpenAI Plugin#66thameem-abbas wants to merge 7 commits intoopenshift-psap:mainfrom
Conversation
thameem-abbas
changed the title
|
@sjmonson @dagrayvid @npalaska |
|
Thanks @thameem-abbas 👍 Initial tests with changes from this PR work well. I'll add more in the afternoon. |
There was a problem hiding this comment.
I still don't like this approach. It duplicates too much code; fine for a one-off, but to merge it would be preferable to have all request methods take a list of queries and return a list of Results (see later comment).
|
|
||
| return result | ||
|
|
||
| def request_batch_http(self, queries, user_id, test_end_time: float = 0): |
There was a problem hiding this comment.
Changing the interface of request_func() based on calling args is bad practice. See above comment.
| # if timeout passes, queue.Empty will be thrown | ||
| # User should continue to poll for inputs |
| if plugin_type == "openai_plugin": | ||
| plugin = openai_plugin.OpenAIPlugin( | ||
| config.get("plugin_options") | ||
| config.get("plugin_options"), batch_size |
There was a problem hiding this comment.
Again, this should be handled in a way the the plugin does not need to know the batch size in advance. However, if you must... just set config["plugin_options"] = batch_size to avoid changing the interface.
Agreeing on change of condition. Co-authored-by: Samuel Monson <smonson@irbash.net>
Don't know what was running through my head when I wrote that. Thanks Co-authored-by: Samuel Monson <smonson@irbash.net>
Fix redundant annotation Co-authored-by: Samuel Monson <smonson@irbash.net>
thameem-abbas
changed the title
|
Has this worked been picked back up/re-reviewed? |
|
@rgreenberg1 This hasn't been picked up for a while now. It's out of sync with main by quite a bit. Can prioritize this if there is a need for this to land in llm-load-test. |
5 participants
Adds non-streaming batch request support in OpenAI plugin.