MLX support

[TomLucidor]

From a user in https://github.com/cubist38/mlx-openai-server I am looking to see which tools might be able to turn MLX OpenAI-compatibles into something like Claude Code Router without eating into extra RAM YishenTu/claudian#66 (comment)

[TomLucidor]

Further elaboration (and also easier to dig around) nguyenphutrong/quotio#243

[vishalveerareddy123]

Hi @TomLucidor
I think it should be supported OOB
with the following setting

1. Install & Start MLX Server

Install

pip install mlx-openai-server

Start with a model (example: Qwen)

mlx-openai-server launch \
--model-path mlx-community/Qwen2.5-Coder-7B-Instruct-4bit \
--model-type lm \
--quantize 4

Server runs at http://localhost:8000/v1

2. Configure Lynkr

Option A: Use as OpenAI provider
export MODEL_PROVIDER=openai
export OPENAI_ENDPOINT=http://localhost:8000/v1/chat/completions
export OPENAI_API_KEY=not-needed
export OPENAI_MODEL=mlx-community/Qwen2.5-Coder-7B-Instruct-4bit

Option B: Use as LM Studio provider (similar API)
export MODEL_PROVIDER=lmstudio
export LMSTUDIO_ENDPOINT=http://localhost:8000
export LMSTUDIO_MODEL=default

3. Start Lynkr

npm start

[vishalveerareddy123]

Let me know if this doesn't work

[TomLucidor]

As a secondary backup https://github.com/waybarrios/vllm-mlx

[vishalveerareddy123]

@TomLucidor anything openai compatible should work in a similar fashion

[TomLucidor]

@vishalveerareddy123 issues with testing due to another blocker #34