GPU memory

[sonnefred]

Hi, thanks for sharing your work, and I'd like to try the model. Is there any demand for the GPU memory when running inference? Thanks.

[xiyichen]

Same question. It seems to run out of memory on a 45 GB GPU.

[Abyssaledge]

Thanks for using our work. We are taking efforts to support low-RAM inference mode. Please stay tuned.

[YuxueYang1204]

Hi, thanks for your interest in NeoVerse!

The default mode requires ~47 GB allocated on GPU (torch.cuda.memory_allocated) with a peak of ~74 GB (torch.cuda.max_memory_allocated). We've added a --low_vram flag that reduces peak memory to ~38 GB by offloading models to CPU and only loading them to GPU when needed.

See the updated README for details. Hope this helps!