This issue was also reported by someone else here: openvinotoolkit/openvino#33665
With the current implementation it is not possible to use the GPU for other workloads while OVMS is running, as it will run out of free vRAM. Other LLM servers approach this by setting a user-defined idle timeout on a model. If no API requests are made to query a specific model for a specific amount of time, that model gets removed from memory. When a new request comes in for the model, it automatically gets loaded into memory again. It would be nice if OVMS had a similar feature.
An explanation of how llama.cpp does it can be found at https://github.com/ggml-org/llama.cpp/tree/master/tools/server#sleeping-on-idle
This issue was also reported by someone else here: openvinotoolkit/openvino#33665
With the current implementation it is not possible to use the GPU for other workloads while OVMS is running, as it will run out of free vRAM. Other LLM servers approach this by setting a user-defined idle timeout on a model. If no API requests are made to query a specific model for a specific amount of time, that model gets removed from memory. When a new request comes in for the model, it automatically gets loaded into memory again. It would be nice if OVMS had a similar feature.
An explanation of how llama.cpp does it can be found at https://github.com/ggml-org/llama.cpp/tree/master/tools/server#sleeping-on-idle