EXLR8 is a quantization format for serving models at low bitrates with fewer compromises on the compute side.
On unified memory hardware, decode speed is bound by how many bytes of expert weights you read per token, so the format is engineered around one question: fewest bytes read, fewest surprises at load time.
Ancestral note: The codec itself is borrowed: each 16×16 weight tile is EXL3's trellis code (QTIP-style). This is a path through a 16-bit-state trellis where every weight contributes K new bits, chosen by Viterbi search.
What EXLR8 adds is:
- Codebook values are rounded to FP8 E4M3 inside the Viterbi search — decoded weights are exactly FP8, so prefill runs on FP8 tensor cores.
- Every routed expert is stored at three widths (K2/K3/K4); the mix is chosen per expert-half at load time by a manifest — no re-encode.
- Rotation is shared, not per-expert: one sign vector per layer on the hidden side, so the kernel rotates the token's activation once and reuses it across all 8 active experts.
- The layout is TP-native: a single sharding rule serves TP=3–7 with zero re-layout, and a
node can fetch only its own slices over HTTP range requests (about 75 GB per node at TP=4 under the
homemanifest, instead of the whole checkpoint).
Decode quality comes from calibrated rounding (GPTQ/LDLQ against real activation statistics).
A pure-PyTorch reference loader for the format is in loader/. Every file and tensor is listed in FORMAT.md.
The provider based Python fileset writer is in exlr8writer/. It connects to a caller's checkpoint or quantization workflow through a small layer data hook.
| release on Hugging Face | model |
|---|---|
| kindlingai/glm-5.3-exlr8-k2-k3-k4 | GLM-5.3 |
| kindlingai/glm-5.3-flash-exlr8-k2-k3-k4 | GLM-5.3-Flash |
| kindlingai/deepseek-v4.1-flash-exlr8-native | DeepSeek-V4.1-Flash: the official weights, byte for byte, in the exlr8 file layout |
gr8 serves all three on DGX Spark (GB10) boxes.
GLM-5.3 is a mixture-of-experts model. Most of its weights sit in the routed experts: each of the 78 main layers has 256 experts, and each token uses only a few of them. Each expert holds three matrices:
- gate: 2048 × 6144
- up: 2048 × 6144
- down: 6144 × 2048
The hidden size is 6144, and the expert’s inner size is 2048. A matrix only ever acts on a vector: y = W x.
In BF16 (16 bits per number) the model is about 1.5 TB. For four DGX Sparks we've got 512GB (4 × 128 GB), so the experts must drop to about 3.25 bits per weight.
Rounding hurts most when a few weights are huge outliers next to many small ones. A grid fine enough for the small ones can’t reach the outliers, and a grid wide enough for the outliers wastes its levels on the small ones.
Here is the fix: Take an orthogonal matrix Q (so QᵀQ = I) and note that W x = (W Q)(Qᵀ x).
So we can store W’ = W Q in place of W and rotate the input by Qᵀ at run time. The output is unchanged.
We choose Q = (random ±1 diagonal) × (Hadamard matrix). A Hadamard matrix has entries ±1/√n and orthogonal rows,
and a fast transform (like the FFT) applies it in n log n steps. Each entry of W’ is then a sum of many weights
with random signs. By the central limit theorem, the entries look like a bell curve with no outliers. We rotate
on both sides of the matrix, and we store the sign vectors with the model (the sg tensors).
We also keep scale vectors: one number per row, and one per block of 128 columns. That lets the coded numbers live on a fixed, unit-size bell curve.
Plain rounding would pick one of 2^K levels per weight, for example 8 levels at K = 3. EXLR8 uses the EXL3 / QTIP trellis
code instead, which packs better:
- Cut
W’into 16 × 16 tiles of 256 weights each. - Each tile gets one bitstream of 256·K bits.
- To decode, slide a 16-bit window along the stream, moving
Kbits each step. Each window position gives a 16-bit number. A cheap fixed function, a kind of hash, turns that number into a value that follows a bell curve. That value is the weight.
Neighbouring weights share most of their window, so you can’t set each weight freely. In return, the values the code can reach fill the space far better than a grid of independent levels. The encoder picks the best bitstream with the Viterbi algorithm, which finds the cheapest path through a graph of states, as in error-correcting codes.
The “-e4m3” part means each decoded value is rounded to an 8-bit float (FP8 E4M3). The GPU can then multiply in FP8 directly, and the encoder includes that rounding when it scores choices, so nothing is lost after encoding.
Without calibration, the encoder minimises plain weight error ‖W − Ŵ‖². That error doesn’t measure what matters, which is the
error in the layer’s output on real inputs:
E = Σ over inputs x of ‖(W − Ŵ) x‖²
Put the calibration inputs as columns of a matrix X, and let Δ = W − Ŵ. Then:
E = trace(Δ X Xᵀ Δᵀ) = trace(Δ H Δᵀ), with H = X Xᵀ
What H is: H is a symmetric n × n matrix: 6144 × 6144 for gate and up, 2048 × 2048 for down. It is called the Hessian because
it is the second derivative of E with respect to the weights.
What H tells you: its eigenvectors show which input directions real data actually uses. Large eigenvalues mark directions where
an error costs a lot. Small ones mark directions the data almost never visits, where an error is nearly free.
When H = I: E reduces to ‖Δ‖², which is the uncalibrated case.
To build H we ran the BF16 model one layer at a time over about 2 million calibration tokens. At every expert we added
x xᵀ for each token routed to it. A rarely used expert sees few tokens, so its H is a poor estimate, and the encoder
regularises it before use.
H gets rotated with the same Q as the weights, so it lives in the same coordinates.
LDLQ is the same method as GPTQ. It encodes the columns of W’ one block at a time (16 columns, one tile wide).
After each block, it changes the not-yet-encoded columns so they cancel the error just made, in the directions H says matter.
How H drives it: you factor H = L D Lᵀ, with L lower triangular and D diagonal. The entries of L say how much of
each finished column’s error to push into each later column.
Why it can work: a weight’s error can be undone by changing other weights that see correlated inputs.
Each expert matrix is split into 16 pieces (the "halves") of 128 inner channels × 6144, about 786k weights each. Each piece
is encoded at K = 2, 3 and 4 bits, and a manifest picks one width per piece.
- home: the main mix, averaging 3.25 bits on the experts.
- down1: one step down, about 2.25 bits.
- up1: one step up, about 4 bits.
- bit225, bit243, bit260, bit325, bit338: about 2.25 to 3.38 bits on the routed experts, spent where they lower the KL divergence against BF16 the most.
EXLR8 rotates each weight matrix so its numbers look like a bell curve. It codes them at 2-4 bits per weight with a trellis
code tuned for the GPU’s FP8. It uses H = XXᵀ from real text so the rounding errors accumulate in the less-important weights.
The first EXLR8 quant is GLM-5.3 from z.ai: kindlingai/glm-5.3-exlr8-k2-k3-k4.
GLM-5.3's MoE layers (3–77) have 256 routed experts, 8 active per token; hidden dim 6144, intermediate dim 2048. One expert is 37,748,736 weights (three 6144×2048/2048×6144 matrices: gate, up, down). A layer's 256 experts take 4.83 GB at K4 (10.87 GB for all three widths) — the design exists to make per-token reads out of that as small as the budget allows.
Each weight matrix w (in, out) is first put through an invertible transform on both sides. Each
side gets:
- a per-channel sign (
±1), - a scale (per-channel on the intermediate side; constant per 128-block on the hidden side),
- a fixed 128×128 Hadamard (
exllamav3's, scaled 1/√128) applied to each consecutive 128-channel block.
After scaling and rotating, the quantizer encodes the transformed matrix; the loader undoes the transform at decode. Reconstruction is:
w = R( diag(suh) · L(T) ) · diag(svh)
where T is the tile decoded from the trellis, L/R apply the 128-block Hadamard on the
input/output side, and suh/svh are the stored side vectors (sign × scale, folded together).
A global scale g per (projection, width) is folded into suh so decoded values come out at the
right magnitude.
The sharing trick. The hidden side's signs (signs.in, signs.out, 6144 values each) are
one vector per layer, shared by all 256 experts, the shared expert, and all three widths. The intermediate side keeps per-expert signs, but they
are folded into the stored per-channel vectors, so they live inside the weights and cost nothing at
runtime.
Why: with 8 experts active per token, per-expert hidden-side rotations would force the kernel to rotate the activation 8 times (or keep 8 rotated copies). With shared signs and block-constant scales, the activation is rotated once, and one A operand serves all 8 experts' matmuls.
A 16×16 tile is encoded jointly, not weight-by-weight: it is a path through a 16-bit-state
trellis, each weight contributing K new bits along the path. The Viterbi search picks the path
minimizing the tile's squared error, with per-step values drawn from the mcg codebook
(multiplier 0xCBAC1FED, a format constant). Joint encoding is the whole trick: the trellis's
state carries information between neighboring weights, so K bits per weight here buy more than K
independent bits would.
A tile of K-bit weights is 256 × K bits = 32·K bytes, packed exactly as exllamav3's
packs it: 16K little-endian int16 values in tensor-core order. Decoding is the same algorithm
as exllamav3 plus the E4M3 rounding below.
Every value the codebook produces is rounded to E4M3 (round-to-nearest-even, saturating) inside the Viterbi search. The search therefore only considers paths whose decoded weights are exactly representable in FP8, and prefill can feed decoded weights straight to FP8 tensor-core MMA.
The rounding is part of the format: a decoder that skips it, a stock EXL3 decoder included, reads measurably worse weights from the same files, so loaders must apply it.
Round-to-nearest ignores how weights are actually used. EXLR8 doesn't:
- For each routed expert e, calibration data gives
H_gu[e] = Σ x xᵀover tokens routed to e (x = the 6144-wide expert input, shared by gate and up), andH_down[e] = Σ a aᵀ(a =silu(gate x) ⊙ up x, 2048-wide). The calibration set is 977 rows / 2.0M tokens (+64 held-out rows). - The Hessian is pulled into the quantizer's rotated domain (
H' = S D Had H Had D S) and factorized in 16×16 blocks:H = L D Lᵀ. - LDLQ then walks the input rows from the last 16-row block to the first: each block is
quantized after adding the error already committed by the blocks behind it (fed back through
L), and its own residual error is fed forward. Error cancellation along the input dimension is what calibrated rounding buys.
Trellis codes don't nest (a 2-bit path is not a prefix of the 3-bit path), so each expert is encoded separately at K2, K3 and K4 — one file set per width. A layer's 256 experts cost 2.42 / 3.62 / 4.83 GB at K2/K3/K4; all three, all 75 layers, ≈ 819 GB on disk. That's the price of the next bullet.
At load time, a manifest picks the width per unit:
- gate/up: per half — a "half" is 128 of the expert's 2048 intermediate channels (the storage unit; halves pair into 256-channel fragments aligned to 64 KiB).
- down: per expert (LDLQ error feedback crosses halves along the input dimension — down's input is the intermediate — so its width must be uniform across the expert).
A deployment picks widths to fit its per-rank memory budget, using the per-half and per-expert error
tables recorded at encode time. The shipped home manifest (64 experts at K4, 192 at K3 ≈ 3.25
bits/weight) holds 3.93 GB per layer ≈ 75 GB per node at TP=4.
For TP = N, half h of expert e in layer L lives on rank:
rank = (16e + h + L) mod N
One rule works for every N from 3 to 7. At TP=4 each rank gets exactly 4 of every expert's 16 halves
(rank r owns h0, h0+4, h0+8, h0+12 with h0 = (r − L) mod 4). Kernels get a pointer table ptr[e][h]
instead of contiguous ranges, and per-half widths are fine because the table is re-read every launch.
| EXL3 | EXLR8 | |
|---|---|---|
| Code | trellis, K bits/weight, mcg codebook | same codec + E4M3 rounding inside the search |
| Codebook values | fp16 | exactly E4M3 (prefill on FP8 MMA) |
| Granularity | one width per tensor | three widths stored; per-half manifest at load |
| Rounding | LDLQ against calibration Hessians, per tensor | LDLQ against calibration Hessians, shared across widths |
| Rotation | per-tensor side vectors | shared per-layer hidden-side rotation → one activation rotate per token |
| Topology | format knows nothing about TP | one sharding rule for TP=3–7; range-fetchable slices |