[Feature Request] Per request sampling params

[qihqi]

Currently sampling params such as temperature are set as commandline flags in when the server starts.

It would be nice for each request to pass in the sampling params instead.

[kiratp]

Beyond sampling parameters, the following would be very helpful

1. Prompt token counts: Makes it easier to potentially trim the next request
2. logprobs - Extremely useful for scenarios like LLM-as-judge or similar
3. seed - getting deterministic responses is pretty useful during development and for certain use cases for end user applicaitons

[qihqi]

HI @kiratp few questions:

On 1. prompt_token_counts would be the same behavior as per-request max_output_len?
On 2> logprobs is a boolean arg on input to signify to return the logprobs in the response protocol buffer?
On 3> This would be a global seed as command line argument to start the server; as seed itself is a global in torch?

[kiratp]

1. Yes. idea would be to get the actual token counts for prompt and completion (something like this: https://platform.openai.com/docs/api-reference/making-requests)
2. Yes
3. That is fine