Xolotl

DeepSeek-V4.1-Flash - all 552 billion parameters - on one gaming PC, at 27 tok/s.
Plus GLM-5.3-Flash, Qwen3.8-Flash-Next and Qwen3.8-27B: one engine, one OpenAI / Anthropic-compatible server.
Built on Strata and heavily extended · Linux · NVIDIA · MIT

A Strata fork. Xolotl keeps Strata's engine, server and kernels at its core (merged at 0.1.44.1) and adds ~71,000 lines on top: the DeepSeek-V4.1 and V4-Flash engines, the GLM-5.3-Flash engine and the Qwen3.8-27B dense port, each with its own quantisation and expert-tiering work.

Starred this as Strata-DSV4.1? You are in the right place. Strata-DSV4.1 grew into Xolotl: the same DeepSeek-V4.1 engine (faster now), plus GLM-5.3-Flash and the Qwen3.8 models, in one tree. Old links and clones redirect here; nothing DeepSeek-V4.1 was removed.

Latest

Full notes for every item: docs/xolotl/CHANGELOG.md.

• 7 Nov - DeepSeek-V4.1 fast mode (opt-in): 41 tok/s decode on real agent tasks, against 32.6 normally (+26 %, the Gauntlet on the 4090), solving the same tasks (8/12 in both modes); short34 top-1 86.2 % vs 87.0 %. A 2-bit (Q2_K) copy of the biggest experts in a sidecar file. Fast mode
• 7 Nov - Xolotl Direct (Linux, experimental, opt-in): NVMe straight into VRAM on a GeForce card. Experts read from the drive skip system RAM: one expert 11 -> 3.1 ms, zero RAM-bus traffic, the same output. With the layer-ahead VRAM stash: +3 % decode on DeepSeek-V4.1. Needs Resizable BAR and two small kernel modules. Setup
• 7 Nov - DeepSeek-V4.1 loads in 44 s instead of 96 s, same output. The converted dense weights are cached on disk after the first load, and the RAM arena is pinned while it fills. On by default. Every switch now also takes an XOLOTL_ name (XOLOTL_ARENA_PIN_AFTER=0 = STRATA_ARENA_PIN_AFTER=0; the old names keep working). Details
• 6 Nov - +8 % decode on every RTX 40 card: QFN 45 GB 139 -> 150 tok/s on the 4090, same output. A tuned set of bit-exact speed switches for sm_89, on by default (STRATA_SM89_PERF=0 turns it off). Details
• 6 Nov - DeepSeek-V4.1: +11 % long-prompt prefill, +2.7 % decode, same output bit for bit. The next prompt chunk's Engram reads start a chunk early, and the shared expert runs while the routed experts load. Both on by default, ideas from Basalt. Details
• 6 Nov - +13-17 % prefill on IQ1_M layers (QFN 45 GB: up to 3,890 tok/s at 16K on the 4090), and closer to the Q8_0 parent than before. On by default (STRATA_MMQ_IQ1M=0 turns it off), port of Basalt. Details
• 4 Nov - faster short requests with the disk prefix cache (#115): 9.0 s -> 0.32 s for a short request after a 47K-token prompt, opt-in. Also the MiniMax replay fix (#116), the Basalt ports and RTX 5090 --kv-q8 numbers (#114). Details
• 3 Nov - LFM3 as the ears and voice of QFN on one 4090 (scripts/xol-qfn-voice.sh). Details
• 2 Nov - LFM3-Omni-2B, natively: text, pictures, audio and video in, text and speech out, ~300 tok/s. docs/LFM3.md
• 31 Oct - Qwen3.8-Flash-Next XQ 45 GB: Swift 1.5, uncensored, 199 tok/s decode and 3,204 tok/s prefill at 16K on the 4090, Gauntlet v2 6.68. Details
• 30 Oct - Qwen 4, day-one prep: what's known and the plan. docs/QWEN4.md
• 25 Oct - DeepSeek-V4.1 V4, the best DeepSeek build: 86.9% top-1 agreement with the full model, 162/164 HumanEval. Run it with --rank v4/rank-1017-v4.bin. Details
• October - community work:
  • grammar and a disk prefix cache (#99, #105-#113)
  • serving fixes (#96-#104)
  • nine PRs (#86-#95)
  • ISTA-DASLab and Unsloth GGUFs
  • ZFS ARC (#83)
  • deep sleep (#75)
  • Qwen3.6 serving (#76-#80)
  • GLM long-conversation fix (#46)
  • batch slots, the conversation pool, multi-GPU and more

Fast mode (DeepSeek-V4.1)

DeepSeek-V4.1's decode speed is set by how many expert bytes cross the RAM bus and the NVMe drive per token. Fast mode keeps a 2-bit (Q2_K) copy of the two biggest expert formats (the MXFP4 and Q3_K experts, about half of those bytes) in a sidecar file and reads that instead. The model file is not changed, and turning it off is unsetting one variable.

On real agent work it decodes at 41.1 tok/s against 32.6 in normal mode (+26 %), and solves the same tasks. Both numbers are decode during the Gauntlet's coding tasks below (speculative drafts land more often on code); on a fixed 8K-prompt probe it is 33.0 vs 26.1 tok/s.

RTX 4090, V4, 8K prompt	decode	short34 top-1	short34 NLL
normal	26.1 tok/s	87.01 %	0.512
fast	33.0 tok/s (+27 %)	86.16 %	0.540

_{Decode here = the 8K-prompt probe; short34 = agreement with the full model's next token over our 34-text set.}

build-xolotl/ds41_generate -m V41-XQ-1017-v4.gguf --build-shadow /nvme/V4.fast.x41s   # once, ~30 min CPU, ~40 GB
XOLOTL_SHADOW=$(cat /nvme/V4.fast.x41s.list)  <your usual serve command>

Agentic coding, the Gauntlet v3-small (6 hard tasks graded by hidden tests, 10 turns, thinking on, served settings, 2 runs each):

	solved	score	decode during the tasks
normal	8 / 12	4.99	32.6 tok/s
fast	8 / 12	4.99	41.1 tok/s (+26 %)

The same tasks solved and missed in both modes (each missed 04 once; 03 stays half-done in every run).

It is lossy, so it is opt-in: on the Gauntlet it cost nothing measurable, but short34 shows it moves a few tokens in a hundred, so keep the normal mode where every token counts. Everything else (switches, the importance matrix, what was tried): docs/XOLOTL_FAST.md. The idea is compress-then-correct from PROMETHEVS (Ryan Flavell).

What one PC can do

One RTX 4090 (24 GB) · Ryzen 7 5700X · 90 GB DDR4 · one NVMe drive (110 GB RAM since 1 Nov; the numbers below are from the 90 GB runs unless marked). Every number is measured on that machine, through the real server, while it solved real tasks (21 HumanEval problems, 10 tool-call tasks, 3 agentic bug-fix loops), greedy, thinking on with a 32K-token thinking budget.

Model	Size	Prefill	Decode	HumanEval	Tool calls	Agentic	Context
DeepSeek-V4.1-Flash	552B MoE	180-320 tok/s	26.8 tok/s	21/21	10/10	3/3	512K
DeepSeek-V4.1-Flash-1017 V4 (XQ)	552B MoE	289-409 tok/s	25.7 tok/s	21/21	10/10	3/3	512K
GLM-5.3-Flash	320B MoE	290-375 tok/s	25.3 tok/s	19/21	10/10	3/3	512K
Qwen3.8-Flash-Next	125B MoE	3,000-3,430 tok/s	127.8 tok/s	21/21	10/10	3/3	200K
Qwen3.8-27B (dense)	27B	1,240-1,390 tok/s	98.5 tok/s	21/21	10/10	3/3	200K
LFM3-Omni-2B (XQ-6.0, omni)	2B	26,000-32,800 tok/s	~300 tok/s	13/21	9/10	0/3	32K

_{Decode = generation speed during those tasks. Prefill = a fresh 6K-17K-token prompt through the server (tools/throughput.py; Qwen3.8-Flash-Next: 3,430 / 3,005 tok/s at 6.5K / 16.7K, Qwen3.8-27B: 1,385 / 1,244 tok/s, DeepSeek-V4.1: 178 / 268 tok/s at 6.1K / 16.5K - its prompt batch is sized from free VRAM, ~6-9K tokens per sweep on a 24 GB card with the speculative-decode draft loaded). DeepSeek-V4.1 with the speculative-decode draft off the card during long prompts (the default since 2026-10-17): 187 / 317 tok/s at 6.4K / 16.7K. GLM-5.3-Flash: soft prune + --route-bias 0.25 (tools/glm/serve/strata-glm-maya-abl-512k-bias.json), prefill 301 / 374 tok/s at 6.1K / 15.6K and 368-376 on fresh 30K / 44K prompts. DeepSeek-V4.1-Flash-1017 V4 (25 Oct): dense half Q8_0, rank-1017-v4.bin + --tier-batch-pcie 0.25, prefill 289-297 / 395-409 tok/s at 6.1K / 16.5K; full HumanEval (164) 162/164. How close the DeepSeek quant gets to DeepSeek's own API on the same tasks: docs/xolotl/BENCHMARKS.md. LFM3-Omni-2B (2 Nov, xolotl-lfm3, alone on the card): a non-reasoning model, so thinking off; prefill 32,796 / 26,049 tok/s at 5.3K / 14K, decode 292-310 tok/s; it also hears, sees and speaks (speech out at 12x real time) - see docs/LFM3.md. Its agentic 0/3 is the 2B model calling a tool the agent does not have.}

Speed-ups since those runs (same 4090 box; before/after on the same prompt and settings; full log in docs/xolotl/BENCHMARKS.md):

Date	Change	Model	Prefill before	Prefill after	Decode
7 Nov	Fast mode: 2-bit (Q2_K) copy of the biggest experts (opt-in, lossy: short34 top-1 87.0 -> 86.2 %)	DeepSeek-V4.1-Flash-1017 V4, agent tasks / 8K prompt			32.6 -> 41.1 tok/s on agent tasks (+26 %); 26.1 -> 33.0 on the 8K probe
6 Nov	RTX 40 (sm_89) speed switches (default on)	Qwen3.8-Flash-Next XQ 45 GB, 16K prompt	4,054 tok/s	4,069 tok/s (same)	139 -> 150 tok/s (+8 %)
6 Nov	Engram read-ahead + late shared expert (default on)	DeepSeek-V4.1-Flash-1017 V4, 26.6K prompt / 8K decode	456 tok/s	506 tok/s (+11 %)	25.7 -> 26.5 tok/s (+2.7 %)
6 Nov	IQ1_M experts on MMQ for prompts (default on)	Qwen3.8-Flash-Next XQ 45 GB, 16K prompt	3,239-3,430 tok/s	3,796-3,890 tok/s (+13-17 %)	within noise

Smaller cards (12-16 GB). The engine fits the work to the card, and the experts that don't fit live in RAM and on the SSD, so system RAM matters as much as VRAM. Community runs:

Card	RAM	Model	Prefill	Decode	Run
RTX 3060 12 GB	64 GB	Qwen3.8-Flash-Next Q2_0	882 tok/s (4K), 981 (32K), 934 (128K)	39-45 tok/s	2026-10-09
RTX 4070 12 GB	32 GB	Qwen3.8-Flash-Next IQ2_XS		48 tok/s	2026-10-08 (Hcl192088)
RTX 3060 12 GB	46 GB	DeepSeek-V4.1-Flash	149 tok/s (8K)	8-10 tok/s	docs/DEEPSEEK41.md
RTX 5070 12 GB	64 GB	DeepSeek-V4.1-Flash	121-231 tok/s (4-8K)	4.7 tok/s	docs/DEEPSEEK41.md

More cards, AMD and Intel included: docs/COMMUNITY_BENCHMARKS.md.

What that means in practice:

• A 552B frontier model as a local coding agent. Point Claude Code, Codex, opencode or Open WebUI at it - tool calls, streaming, prefix reuse across agent turns, a thinking budget.
• No server farm. The weights (152 GiB for DeepSeek-V4.1) live across VRAM, pinned RAM and NVMe; the engine carries up the experts each token needs.
• Fast where it counts. The Qwen models run at 100-200 tok/s; an agent's small follow-up turn on DeepSeek-V4.1 costs ~8 s on a 6K-16K context, and the engine learns the conversation's experts between turns. When speed matters more than the last bit of quality, DeepSeek-V4.1's fast mode decodes at 41 tok/s on agent tasks (32.6 normally).

Quick start

Linux, an NVIDIA GPU with 24 GB, CUDA 12.x/13.x, and RAM for the arena (90 GB was used above).

git clone https://github.com/Indras-Mirror/Xolotl.git xolotl && cd xolotl
bash tools/xolotl/fetch-llama.sh   # llama.cpp at the pinned commit + tools/xolotl/ggml/xolotl-ggml.patch -> third_party/llama.cpp
cmake -S . -B build-xolotl -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DSTRATA_GGML_CUDA=ON \
  -DSTRATA_MMQ_KQUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=89 -DSTRATA_GGML_DIR=$PWD/third_party/llama.cpp -DSTRATA_DSV41_FQ_OP=ON \
  -DSTRATA_DSV41_HC_MIX=ON
ninja -C build-xolotl      # strata, strata-dense, glm_generate, ds41_generate, ds4_generate

Get a model:

• DeepSeek-V4.1-Flash: the 1017 builds of the October checkpoint (V4 recommended, with v4/rank-1017-v4.bin), or the earlier GSQ build (152 GiB, mixed ternary / 2-bit / 3-bit). Both cards have the download and Engram steps. Optional for V4: the fast-mode sidecar (~40 GB), built once with ds41_generate --build-shadow.
• Qwen3.8-Flash-Next: XQ 45 GB (the two .gguf shards plus pack-qfn45/).

Then serve it (edit the model paths in the DeepSeek and GLM configs first):

python3 tools/v41/serve/serve_ds41.py --engine strata --config tools/v41/serve/strata-ds41.json --port 8150           # DeepSeek-V4.1-Flash
XOLOTL_SHADOW=$(cat /nvme/V4.fast.x41s.list) python3 tools/v41/serve/serve_ds41.py ...                                # the same in fast mode (V4)
python3 tools/glm/serve/serve_glm.py  --engine strata --config tools/glm/serve/strata-glm-maya-abl-512k.json --port 8144  # GLM-5.3-Flash
./scripts/xol-qfn.sh --serve-only --model <shard 1 .gguf> --pack <pack-qfn45>                                       # Qwen3.8-Flash-Next (port 8099)
python3 -m serve.server --engine strata --config <your-27b.json> --port 8090                                          # Qwen3.8-27B (exe: build-xolotl/strata-dense)

Then point any OpenAI- or Anthropic-compatible client at http://127.0.0.1:<port>/v1.

Will it run on my machine? (DeepSeek-V4.1)

What I've measured is one setup: RTX 4090 (24 GB), 90 GB DDR4, Linux, NVMe. Beyond that:

• OS: Linux, or Windows 11 natively (MSVC + CUDA; build and serve steps in docs/WINDOWS.md). Everything measured here is on Linux.
• GPU: NVIDIA, compute capability 7.5 or newer (RTX 20/30/40/50). Build with your card's architecture: -DCMAKE_CUDA_ARCHITECTURES=86 for RTX 30, 89 for RTX 40, 120 for RTX 50 (use CUDA 13.x for RTX 50; 12.8 builds had prompt-path crashes). Tesla V100 (sm_70): experimental DS41-on-Buster path only — see BUSTER.md (STRATA_EXPERIMENTAL_SM60=ON, CUDA 12.x, no chunk-MMQ on Volta).
• VRAM: 24 GB is what I run. 16 GB should load but is untested, and prompts get processed in smaller batches, so prefill will be slower. Below 16 GB won't work.
• RAM: 90 GB here, with a 64 GB expert cache in RAM. Less RAM means a smaller cache, so more experts come off the NVMe each token and decode is slower. More RAM helps: around 256 GB holds every expert.
• Disks: about 152 GiB for the quant plus ~200 GB for the Engram tables, and ~40 GB more for the optional fast mode. All of it wants a fast NVMe on ext4 or NTFS. Not exFAT (Linux's exFAT driver serialises parallel reads; expert reads were 5x slower here) and not a hard drive (the Engram lookups are ~48 small random reads per token).

How it works, in one paragraph

A mixture-of-experts model only uses a few experts per token. Xolotl keeps every expert somewhere - the hottest in VRAM, tens of GiB in a pinned RAM arena the GPU reads by DMA, the rest on NVMe read at the drive's ceiling - and moves the ones each token's router asks for, a layer ahead. Each model adds its own tricks: DeepSeek-V4.1 gets a calibrated mixed-precision quantisation, speculative decoding with exact sampling, a bit-exact 4-bit KV cache for 512K context, an arena that adapts to the conversation, an opt-in fast mode that reads a 2-bit copy of its biggest experts, and Xolotl Direct, which lets the NVMe drive write experts straight into VRAM; GLM-5.3-Flash gets its abliteration LoRA on every path and soft pruning; the Qwen models get MTP speculative decoding. Read the full write-up.

Read more

How Xolotl works	the three memory tiers, per-model engines, the server, the agent-loop work
Benchmarks	every measured number, exactness checks, how to measure your own PC
Changelog	everything that landed, newest first, with the PR numbers and who did it
DeepSeek-V4.1-Flash engine	GSQ quantisation, DSpark, the fp4 KV cache, Engram
DeepSeek-V4.1 fast mode	the 2-bit expert sidecar: build, serve, switches, what it costs and what we tried
DeepSeek-V4.1 on Windows	native MSVC + CUDA build, configs, RTX 50 notes
GLM-5.3-Flash engine	KDA + MLA attention, the LoRA on routed experts, soft pruning
LFM3-Omni engine	pictures, audio, video and speech on a 2B model, park/unpark beside a big one
Qwen3.8 / upstream Strata	the Strata engine this grew from
Brand kit	logo, banner, palette

Credits

Built on Strata by Niko1221 and ggml / llama.cpp, with the Qwen3.8-27B dense port by perronemirko. Model weights by DeepSeek, Zhipu AI, Qwen and Liquid AI, with quants and fine-tunes by peasantsmith, ukisai, d0xin and gcsa-abliterix. Thanks to every contributor: NektoDron, Xiaomi2008, AlexeyMatskevich, paul123637-lab, kelheor, exye137, penink-WH, cpacsm75, jesdga95 (Basalt) and Ryan Flavell (PROMETHEVS). Who did what: CREDITS.md.

Xolotl is Quetzalcoatl's dark twin, who carries the dead up from the underworld - as the engine carries experts up from RAM and NVMe. Research code from one developer, measured on one PC; reports from other hardware welcome. MIT.

Support

If Xolotl's been useful to you and you want to help support what I do, you can shout me a coffee on Ko-fi. Every bit helps keep this going :)