4*V100

[liziming5353]

请问4张16G的V100显卡可以部署o-4.5的推理不

[airlsyn]

请问4张16G的V100显卡可以部署o-4.5的推理不

Yes, 4x V100 (16GB each) can run MiniCPM-o 4.5 inference. Here are two recommended approaches:

Option 1: GGUF Quantized Model + llama.cpp-omni

For setup instructions, see the Quick Start section in the llama.cpp-omni README. Below is a summary of VRAM usage — refer to the llama.cpp-omni README for full details:

Configuration	LLM Quantization	Model Size	VRAM Estimate	Single V100 16GB
Full Omni	F16	~18 GB	~20 GB	Won't fit, needs multi-GPU
Full Omni	Q8_0	~11 GB	~13 GB	Runs, but tight
Full Omni	Q4_K_M	~8 GB	~9 GB	Runs comfortably
Vision Only	Q8_0	~9 GB	~10 GB	Runs fine
Audio Only	Q8_0	~10 GB	~12 GB	Runs fine

Option 2: Multi-GPU FP16 Inference with Transformers

Follow the Inference with Transformers section in the MiniCPM-o README. Simply specify device_map when loading the model, e.g.:

model = AutoModel.from_pretrained(
    "openbmb/MiniCPM-o-4_5",
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="auto",
)

Note: V100 GPUs do not support BF16. Make sure to set torch_dtype to torch.float16 instead of torch.bfloat16 as shown in the official examples.

[liziming5353]

我使用device_map="auto"的时候遇到了“RuntimeError: Expected all tensors to be on the same device, but got src is on cuda:3, different from other tensors on cuda:0 (when checking argument in method wrapper_CUDA_scatter__src)”的问题，似乎是modeling_minicpmo.py里没有做相应适配

[airlsyn]

我使用device_map="auto"的时候遇到了“RuntimeError: Expected all tensors to be on the same device, but got src is on cuda:3, different from other tensors on cuda:0 (when checking argument in method wrapper_CUDA_scatter__src)”的问题，似乎是modeling_minicpmo.py里没有做相应适配

Could you share your input?