RTX 5070 Ti Laptop 12GB 跑 Qwen3.8-27B：NInfer + KVMem、256K 逻辑上下文、RAM KV、MTP、快速缓存续接与 dsh 本地 Agent / 搜索 / 浏览器。含源码、Windows 部署及实测。
NInfer for Windows and around 16GB VRAM: RTX 5070 Ti / 5080 / 5090, Qwen3.8-27B GSQ-RCO Q3, CUDA 13 Native engine, tray manager, model conversion and measured setup guides.
High-performance single-GPU inference for selected model checkpoints and GPUs.
Qwen3.8-27B on RTX 5090 Laptop 24GB — TWIN-TURBO NVFP4 (native llama.cpp) vs Coder390 Q3 LynnStyle (.ninfer + NInfer engine in WSL2/Docker): 86 tok/s decode, 262K context, C4 aggregate 228 tok/s. Every number a controlled measurement.
monitor for a local LLM runtime (llama-swap, NInfer, Strata, more) -- metrics ported to berth.
Independent Windows distribution of ninfer-all with RTX 3000, 4000 and 5000 builds, runtime contracts and full upstream attribution.
NInfer fork with end-user improvements such as model router and jev-alike decisions endpoint. Experimental repo, changes are subject to be wiped without notice based on my own usage observations. Packaged with nix.
Wire the 1CatAI Split-D D256 FlashAttention kernel (fishlikeX/sm70-attn, MIT) into NInfer on Tesla V100 sm_70: +36-41% prefill, TTFT -3min, decode unchanged. Measured data + integration guide. Published by the user with AI assistance.
Single-GPU NInfer + KVMem integration: multi-lane long contexts, DFlash2 + vision, native prefix reuse and reproducible measurements.
笔记本 RTX 5070 Ti Laptop 12G（12,227 MiB / 140 W TGP）· Bonsai-2-27B 三元量化 · MTP vs DFlash2 同上下文 A/B。Laptop GPU only — NOT the desktop 5070 Ti (16 GB / ~300 W); numbers are not comparable across the two.
Converter toolchain behind the NInfer builds of the Ternary Bonsai 2 family. Two routes: v2 containers for Swift-Bonsai-2 27B, v3 containers for Ternary-Bonsai-2 2B/4B/9B. Byte-for-byte ternary packing, no dequantisation or requantisation.
KVMem + ActKV long-context NInfer for Tesla V100 (sm_70) on Windows: host-archived KV working set, Swift-1.5 27B + MTP, native GGUF loading, OpenAI/Anthropic-compatible serving
一个端点托管两个本地大模型：按需加载、三协议兼容、鉴权限速与 token 用量统计
1508 tok/s on a single RTX 5090 Laptop: 8 lanes x 16 tok/round, Qwen3.8-27B NVFP4, with raw engine logs
NInfer sm_120a 引擎包 · 8 GB RTX 5060 Laptop 部署实践：集显档启动器 / 控制面板 / 桌面应用 / 实测记录
To associate your repository with the ninfer topic, visit your repo's landing page and select "manage topics."