Fix finish_reason stop when tracking usage#2641
Conversation
|
With this change docker-agent can connect to a docker-agent that serves an agent 
|
![docker-agent[bot]](https://avatars.githubusercontent.com/in/2787757?s=60&v=4){kind=small}
There was a problem hiding this comment.
Assessment: 🟡 NEEDS ATTENTION
This PR correctly implements the happy-path for stream_options.include_usage: the usage chunk is emitted after the stop chunk and before [DONE], Choices serializes as "choices":[] (no omitempty), and the OpenAPI schema is updated. Two medium-severity interoperability concerns with the OpenAI streaming protocol were found.
rumpl
force-pushed
the
fix-openai-usage-event
branch
2 times, most recently
from
4d04f1a to
ef58ef7
Compare
| @@ -251,9 +251,6 @@ func runAgentLoop(ctx context.Context, rt runtime.Runtime, sess *session.Session | |||
| // returning nil when nothing is known so we can omit the field entirely | |||
There was a problem hiding this comment.
so was it intentional, or should we remove/adjust this part of the comment?
|
|
||
| if req.Stream { | ||
| err := s.streamChatCompletion(c, rt, sess, model) | ||
| err := s.streamChatCompletion(c, rt, sess, model, req.StreamOptions != nil && req.StreamOptions.IncludeUsage) |
There was a problem hiding this comment.
i gotta say she's a beauty :) can we maybe make a var? it's not entirely sure what it is at a glance
There was a problem hiding this comment.
I'll just change it so that StreamOptions is not a pointer
rumpl
force-pushed
the
fix-openai-usage-event
branch
from
ef58ef7 to
64ef8a4
Compare
The server now correctly sends usage at the end of the stream the same way openai's API does. Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
rumpl
force-pushed
the
fix-openai-usage-event
branch
from
64ef8a4 to
5daf796
Compare
3 participants
The server now correctly sends usage at the end of the stream the same way openai's API does.