[BUG] Compatibility Issue Between max_tokens and max_completion_tokens

[TamiTakamiya]

Bug Description

Output of OLS version

Tested with road-core/service main branch at ed4ad18

Describe the bug

When road-core/service is configured with our self-hosted Granite 3.1 model on OpenShift AI,

llm_providers:
  - name: my_rhoai_g31
    type: rhoai_vllm
    url: https://granite3-1-8b-wisdom-model-staging.apps.stage2-west.v2dz.p1.openshiftapps.com/v1
    credentials_path: /home/ttakamiy/secrets/granite31-8b-token.txt
    models:
      - name: granite3-1-8b
        context_window_size: 128000

Gradio UI shows following error as the response to the "Hello" message:

Sorry, an error occurred: {"detail":{"response":"[{'type': 'extra_forbidden', 'loc': ('body', 'max_completion_tokens'), 'msg': 'Extra inputs are not permitted', 'input': 4096}]","cause":"Error code: 400 - {'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'max_completion_tokens'), 'msg': 'Extra inputs are not permitted', 'input': 4096}]", 'type': 'BadRequestError', 'param': None, 'code': 400}"}}

This seems to be the same issue as this open issue on langchain.

To Reproduce

Steps to reproduce the behavior:

1. Configure road-core/service to use the Granite 3.1 model on OpenShift AI. See the description section above. Aslo enable debug for Gradio UI and disable authentication.
2. Run road-core/service
3. Open Gradio UI at http://localhost:8080/ui
4. Send "Hello" to LLM --> The error occurs.

Expected behavior

Granite 3.1 model should reply with a greeting message.

Screenshots or output

Additional context

Ansible Lightspeed team implemented a workaround suggested in the langchain issue with ansible#99 .

[asamal4]

the problem is -> with latest langchain, the parameter name gets changed for openai internally and for vllm based provider we are using openai method from langchain. if specific vllm version is not handling, then we will get this error.

There are some potential solutions.

• workaround by overriding langchain's openai method.
• use langchain's vllm method (I haven't tested, not sure if it gets updated very frequently and if supports all features)

But using the latest vllm is much easier than doing above.. Probably provider can update the version.
In OLS, we had same issue with RHELAI and we are going to recommend use latest RHELAI.
cc: @tisnik