GitHub - Quackster/tessera: Tessera: a fast LLM inference engine for AMD GPUs, built on ROCm with a portable Vulkan backend.

Status: Heavy development. The API changes without notice. Greedy text generation is stable on the 27B targets. DFlash2 and MTP speculation run end to end. The code is usable for development and for backend tests. It is not ready for production use.

Tessera is a C++20 LLM inference engine. It runs open weight models on GPU. It has two compute backends. Vulkan supports cross vendor GPUs. ROCm supports AMD GPUs. One binary uses one backend. The backend is set at configure time.

First class targets are Qwen 3.8 27B in GGUF and Qwen 3.8 in MXFP4, with and without DFlash2 speculative decoding.

Why AMD (ROCm) and Vulkan only? The project author tests with a 7900 XTX and two R9700 cards. There is no recent NVIDIA GPU in the lab. The last NVIDIA GPU used here was a 1080 Ti. Backend support follows hardware available for testing.

Features

Path	Contents	Use	Hugging Face
`~/models/Qwen3.8-27B-GGUF/`	`Qwen3.8-27B-UD-Q4_K_M.gguf`, `mmproj-BF16.gguf`, `MTP/mtp-Qwen3.8-27B-Q4_0.gguf`, `config.json`	Main GGUF target.	unsloth/Qwen3.8-27B-GGUF
`~/models/Qwen3.8-27B-MXFP4-MTPFP8/`	MXFP4 weights with FP8 MTP, converted output, see below	Main MXFP4 target.	amd/Qwen3.8-27B-Quark-AWQ-MXFP4
`~/models/Qwen3.8-27B-DFlash2-FP8/`	DFlash2 draft checkpoint	Draft layout tests.	tcclaviger/Qwen3.8-27B-DFlash2-FP8
`~/models/Ornith-1.5-35B-A3B-GGUF/`	`Ornith-1.5-35B-Q4_K_M.gguf`, `mmproj-Ornith-1.5-35B-BF16.gguf`	MoE GGUF target.	ornith-ai/Ornith-1.5-35B-A3B-GGUF
`~/models/Ornith-1.5-35B-A3B-MXFP4-GGUF/`	MoE MXFP4 GGUF weights	Not yet supported.	tsaipifong/Ornith-1.5-35B-A3B-MXFP4-GGUF
`~/models/Ornith-1.5-35B-A3B-BF16/`	bf16 safetensors	Not yet supported.	ornith-ai/Ornith-1.5-35B-A3B
`~/models/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/`	`IQ3_S/*-00001-of-00002.gguf` (compute shard plus n-gram shard), `mmproj-Qwen3.8-Flash-Next-BF16.gguf`	Flash-Next decode target. The expert cache is sized to the free device memory and the batched staging is pipelined (the staging ring); the 48 tok/s target and the intermittent non-finite fault are open.	ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

Details

| Area | State | Detail | | --- | --- | --- | | Public API | Done | `Engine`, `Backend`, `SpeculativeStrategy`. | | GGUF loader | Done | v2/v3 headers, bounds-checked, memory-mapped (not copied). | | MXFP4 loader | Done | Tensor map, weight upload. FP8 and MXFP4 GEMM verified. | | Backends | Done | Vulkan and ROCm init, buffers, copy, sync. Batched host upload. | | Parallelism | Done | One engine and model per GPU behind a pool with least loaded routing. Tensor, pipeline, context, expert, speculative split and hybrid modes match single device output. | | Kernel launch | Done | Vulkan records a whole decode step into one command buffer. ROCm runs one blocking stream with pinned staging. | | DFlash2 | Done | Draft forward built and running. 4.4 tokens/step on 27B MXFP4. | | CLI | Done | One flag set for `run`, `serve` and `calibrate`. Load, summary, streaming, sampling, serve, KV-cache flags, speculate, images, thinking flags, calibrate with an optional point limit and concurrent benchmark. `serve` honors every layout: replicas, `--pipeline`, `--tp`, `--cp`, `--spec-split` and `--hybrid`. | | Generation | Done | Greedy decode, chunked prefill, thinking budget, stop tokens, configurable KV cache. | | KV cache | Done | fp32/fp16/int8/4-bit/FP8 E4M3 on both backends. Row-parallel quant kernel. Prefill appends a whole row range with one cast or quantize pass per buffer. | | Sampling | Done | Seeded sampling with Qwen 27B defaults. `serve` starts from the launcher flags, prints the effective values at boot, and each request overrides temperature, top_p, top_k, min_p, the penalties and the seed in the body; a value outside the accepted range is rejected with 422. | | GEMM | Done | Q4_K to IQ, FP8/MXFP4, bf16, fp32. Tensor core paths for decode and verify. ROCm prefill runs fp16 WMMA tiles for the block formats (Q4_K to Q8_0); other backends fall through to the tiled kernel. | | Activation quant | Done | Per-token FP8 E4M3 quant-dequant. W4A8 opt-in. | | Attention and RoPE | Done | GQA, tiled attention, all KV variants. Split key range for single token. Quantized prefill decodes eight key codes per load and serves four query rows per workgroup. | | Weight upload | Done | Manifest to device buffers. One batched upload per load. | | Decode loop | Done | Single-token vanilla and hybrid GGUF loops. | | Hybrid SSM | Done | Load, kernels, both decode paths, MTP head done. The delta scan keeps each state column in registers across a prefill chunk. | | Speculative decode | Done | MTP and DFlash2 end to end. Output equals greedy. GGUF MTP reaches 31-37 tok/s on the 27B target (ROCm). DFlash2 about 60 vs 23 tok/s. | | MXFP4 path | Done | Parse, kernels, HF config, tokenizer. Matches GGUF greedy output. | | DFlash2 decode | Partial | Real draft end to end. 3.5 to 4.4 tokens/step. Two defects fixed. | | Baseline pinning | Done | Fixed-seed fixtures identical on Vulkan and ROCm. | | Tokenizer and chat | Done | Byte-level BPE, Jinja2-subset templates, stop tokens. | | Serving | Partial | HTTP/1.1 server, sessions, web UI, streamed reasoning, tool calls. Tool turns stream live: reasoning, answer text and each completed tool call go out as the model resolves them, buffered and streamed responses split identically. Sampling defaults come from the launcher flags; a turn without a body override uses them, `temperature 0` decodes greedy, and a request without a seed draws a fresh one. Plain turns run without a drafter; a turn that uses one drops a greedy-only strategy when it samples and says so in the log. Capability harness (`tools/harness/probe_harness.py`) drives task steering, long-horizon context and session memory over the API on both backends. | | Vision | Partial | CLIP encoder, merger, image injection. Prefill speed work remains. | | Architecture modules | Done | One module per model family. | | MoE, MLP, norms | Done | Norms and MLP kernels done. Ornith-1.5 MoE decodes on both backends (84 tok/s ROCm, 88 Vulkan). | | Calibration | Done | Measures speed settings per machine and keeps the fastest. Covers split target and cap (ROCm), prefill chunk, attention split, and the draft block with a strategy attached. A parallel layout (`--tp`, `--cp`, `--pipeline`) runs both MTP (`--speculate`) and DFlash2 (`--draft`), sweeps the knobs it honors, and stores the chosen settings with the measured decode and prefill rates. `--concurrent` benchmarks 2, 3, and 4 sessions and the largest count that fits the context. Sizes and stores the layer count from free device memory, so an MoE model larger than VRAM loads paged, and `run` and `serve` apply it. `--calibrate-limit N` caps a run at N measurements. Writes JSON keyed by backend, device, model, context, KV type, strategy, and layout. |

Chat UI

Lightweight, dark-mode AI chat UI with a simple sidebar for API keys and conversation management. It supports streamed model responses, collapsible thinking output, retry controls, token limits, and rendering rich content like inline SVGs directly in the chat. Accessible via the IP:PORT serve command.

Here's an example of an SVG generation:

Two R9700s with Qwen3.8-Flash-Next MTP

Two AMD Radeon AI PRO R9700 cards run the Qwen3.8-Flash-Next IQ3_S target with MTP speculation on a layer split. This configuration works without Resizable BAR.

Without Resizable BAR the BAR window is small. Tessera keeps part of each stage in system RAM and pages the rest on demand. --gpu-layers 24 sizes the resident part so the last stage also fits the output head, the token embedding, and the MTP head. --pipeline-split 24,24 puts 24 trunk blocks on each card.

./cmake-build-rocm/tessera-cli run \
  --model ~/models/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf \
  --mtp ~/models/Qwen3.8-Flash-Next-MTP \
  --speculate --pipeline --pipeline-split 24,24 --gpus 0,1 \
  --context 262144 --kv-q8 --gpu-layers 24

The run prefills the whole prompt, streams each output token, and stops at the declared stop token. On this machine MTP decode runs about 12 tokens per second. The MTP head needs no draft checkpoint. --speculate drafts from the head in --mtp.

One R9700 with Qwen3.8-27B MTP and DFlash2

One AMD Radeon AI PRO R9700 runs the Qwen3.8-27B target at the full 262144 token context with an int8 KV cache. The GGUF target uses MTP speculation from its own head. The MXFP4 target uses the DFlash2 draft checkpoint.

Example launch command with MTP speculation on the GGUF target:

./cmake-build-rocm/tessera-cli serve \
  --model ~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --gpus 0 \
  --draft-block 2 \
  --speculate --context 262144 --kv-q8 \
  --host 0.0.0.0 --port 8080

Example launch command with the DFlash2 draft on the MXFP4 target:

./cmake-build-rocm/tessera-cli serve \
  --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --gpus 0 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8 \
  --draft-block 2 \
  --context 262144 --kv-q8 --host 0.0.0.0 --port 8080

Architecture

Core code is backend agnostic. All device work goes through the Backend interface. Only src/backends/vulkan/ includes Vulkan headers. Only src/backends/rocm/ includes HIP and ROCm headers. No vendor type crosses the backend boundary.

Models are data plus one module per architecture. Each architecture owns its config parse, weight map and layer assembly behind the Architecture interface (today src/models/qwen3_5/). It drives generic kernels for GEMM, attention, MLP, MoE, norms, RoPE, activations, and embeddings. Generic code has no per model branches.

Speculative decoding plugs in behind the strategy interface. It is off by default. DFlash2 is the current strategy. The non speculative path is the reference baseline. It stays stable within per backend tolerance.

Layout:

include/tessera/ holds the public API.
src/core/ holds the engine, the model, and the weight loaders.
src/models/ holds one architecture module per model family.
src/backends/vulkan/ holds the Vulkan backend.
src/backends/rocm/ holds the ROCm backend.
src/spec/ holds speculative decoding strategies.
src/serve/ holds the HTTP layer.
tools/cli/ holds the thin CLI.
tests/ holds the single GoogleTest binary.
docs/PROGRESS.md tracks progress.

See AGENTS.md for architecture rules and for hard rules.

Model support

Format	State	Detail
GGUF	Done	Parses v2 and v3, single files and `-00001-of-00005` splits. Supports Q4_K, Q3_K, Q5_K, Q6_K, Q8_0, Q2_0, IQ4_NL, IQ4_XS, IQ3_S, IQ3_XXS and IQ2_S. Checks bounds.
GSQ-RCO	Partial	Qwen3.8-Flash-Next GSQ-RCO loads and decodes: split shards merge, config parses, and the 51B n-gram table streams rows from SSD. The QSA indexer, hyper-connection mixer, PLE gather and GDN output-norm kernels run on both backends. Paged weights are held in system RAM.
Flash-Next MTP head	Done	The GSQ-RCO GGUF ships no MTP head. `tessera-cli fetch-mtp` reads the BF16 checkpoint's safetensors headers over HTTP and downloads only the 31 `mtp.*` tensors, resumable and checked against the pinned SHA256.
MXFP4 safetensors	Done	Parses the map and packs the weights. config.json drives the decode config.
NVFP4 MoE	Done	GGUF mixture-of-experts (Ornith-1.5) loads and decodes on Vulkan and ROCm.
DFlash2 FP8 draft	Done	The draft loads and runs the block. Speculation accepts 4.4 tokens per step on the 27B MXFP4 target.

Model weights live outside the repo. Each variant uses one flat directory. Model paths are runtime configuration. Code and tests never hard code model paths.

MXFP4 Qwen checkpoint preparation

The MXFP4 target needs a converted checkpoint. The base download ships the MTP head in bf16. Tessera serves the FP8 MTP variant. The converter is fp8_mtp.py. It rewrites the eight MTP projections to FP8. It updates config.json. It copies the tokenizer files.

The raw source is fp8_mtp.py.

# Download the base MXFP4 checkpoint.
hf download amd/Qwen3.8-27B-Quark-AWQ-MXFP4 --local-dir ~/models/Qwen3.8-27B-Quark-AWQ-MXFP4

# Fetch the converter. It needs torch on the host. CPU torch is enough. No GPU is needed.
# pip install torch --index-url https://download.pytorch.org/whl/cpu
curl -fsSL "https://codeberg.org/ggz14/radiance-vllm-mxfp4/raw/commit/dba9defefb2de7f914fe9cb45ffdf49989c923d6/fp8_mtp.py" \
  -o /tmp/fp8_mtp.py

# Convert the checkpoint. This takes about 15 minutes. It writes about 19 GB.
python3 /tmp/fp8_mtp.py ~/models/Qwen3.8-27B-Quark-AWQ-MXFP4 ~/models/Qwen3.8-27B-MXFP4-MTPFP8

The script prints one relative error per MTP tensor. Good rows read about 0.02 to 0.03. A row near 0.116 means that layer was not converted. The output directory is the --model path for tessera.

CLI examples:

# GGUF target, greedy decode.
./cmake-build-vulkan/tessera-cli run --model ~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf

# GGUF target with MTP speculation. The MTP head drafts from the target, so it needs no draft path.
./cmake-build-vulkan/tessera-cli run --model ~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf --speculate

# MXFP4 target with the DFlash2 draft checkpoint.
./cmake-build-vulkan/tessera-cli run --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8

# MXFP4 target and DFlash2 draft over the HTTP server.
./cmake-build-vulkan/tessera-cli serve --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8 --port 8080

The CLI loads the model and uploads weights. It attaches the draft strategy when --draft or --speculate is set. With --max-completion-tokens N it runs N greedy decode steps and streams the text to stdout. --speculate drafts with the MTP head. --draft drafts with the DFlash2 checkpoint. The DFlash2 draft is tested against the MXFP4 target. --mmproj plus --image prepend image tokens.

Calibration

Calibration measures the engine settings that depend on the machine. It keeps the fastest value. It changes only speed. It never changes the produced tokens. Run it once on each machine and model, before the first run or serve.

The calibrate subcommand loads the model one time and sweeps the candidate values for each setting. Without a limit it runs to completion. It writes a JSON file and prints a report with the sweep points, the chosen settings, and an example run command. The example commands run at the memory-bound max context.

Before the load it sizes --gpu-layers from the free device memory. A model that fits stays a full upload. An MoE model larger than VRAM loads paged, so calibration works for it. The chosen layer count is stored in the entry and applied by run and serve, unless an explicit --gpu-layers overrides it.

# Sweep on ROCm with the DFlash2 draft and write the result.
./cmake-build-rocm/tessera-cli calibrate \
  --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8 \
  --calibration ~/.config/tessera/calibration.json

The calibrate flags:

--model <path> model to measure. Required.
--draft <dir> DFlash2 draft checkpoint. It enables the draft block and draft context sweeps.
--speculate MTP drafting from the target's own nextn head. It enables the draft block sweep.
--calibration <file> file to read and write. The environment variable TESSERA_CALIBRATION supplies it when the flag is absent. One of the two is required.
--context <n> context length in the hardware key. Default 32768.
--gpus <a,b> one device index or a comma-separated list. Default 0, the first device. A single index (for example --gpus 0) runs the single-device sweep.
--kv-f16, --kv-q8, --kv-q4, --kv-fp8 KV cache type. Default int8 (kv8).
--prompts <file> fixed prompts, one per non-empty line. Default three fixed prompts.
--calibrate-limit <n> measure at most n sweep points in total, in sweep order. Each point is one engine run. A cut setting keeps its fastest measured value. A skipped setting keeps its default.
--concurrent also runs the concurrent-session benchmark. It measures 2, 3, and 4 sessions together and reports the total and per-session tok/s, plus the largest count that fits the requested context. The path is non-speculative.
--gpus <a,b> with two or more devices and exactly one of --tp, --cp, or --pipeline calibrates a parallel layout: it stores the memory-bound max context and the fastest speed settings the layout honors. --draft and --speculate are single-device only.

It sweeps the speed settings for the current backend and draft setup. A skipped setting keeps its default. It stops a setting early when three candidates in a row are slower than the best value seen so far.

run and serve apply a saved entry with --calibration <file> or TESSERA_CALIBRATION. An explicit flag always wins. Only calibrate writes the file. Create the parent directory before you run calibrate.

# Serve with the calibrated values for this machine and model.
./cmake-build-rocm/tessera-cli serve \
  --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8 \
  --calibration ~/.config/tessera/calibration.json \
  --port 8080

The file is JSON keyed by backend, device, model, context, KV type, and strategy. A non matching entry is ignored.

Usage

Simple text prompt with the CLI:

./cmake-build-vulkan/tessera-cli run \
  --model ~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --prompt-text "Explain gravity in one sentence." \
  --max-completion-tokens 64

--max-completion-tokens is optional. Without it the run fills the remaining context. Output tokens stream to stdout.

The CLI applies the model chat template by default. It uses the raw text with --no-chat. String prompts need a model with a tokenizer. GGUF carries it in metadata. The MXFP4 loader parses tokenizer.json. String prompts fail on a model without a tokenizer, such as a bare DFlash2 draft directory.

C++ API with a text prompt:

#include "tessera/engine.hpp"

auto engine = tessera::Engine::Create({});
if (!engine) return 1;
auto model = (*engine)->LoadModel(
    tessera::ModelOptions{"~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf"});
if (!model) return 1;

const tessera::Tokenizer* tokenizer = (*model)->GetTokenizer();
std::string text = "Explain gravity in one sentence.";
if (auto rendered = (*model)->ChatPrompt(text)) text = *rendered;
auto ids = tokenizer->Encode(text);

tessera::GenerateOptions options;
options.max_completion_tokens = 64;  // Optional. Zero fills the remaining context.
options.prompt_tokens = *ids;
auto generated = (*engine)->Generate(**model, options);
if (!generated) return 1;
auto text = tokenizer->Decode(*generated);

HTTP API with the server:

# GGUF target.
./cmake-build-vulkan/tessera-cli serve \
  --model ~/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --port 8080

# MXFP4 target with the DFlash2 draft. MTP speculation uses --speculate instead.
./cmake-build-vulkan/tessera-cli serve \
  --model ~/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  --draft ~/models/Qwen3.8-27B-DFlash2-FP8 \
  --port 8080

curl -X POST http://127.0.0.1:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain gravity in one sentence.", "max_completion_tokens": 64}'
curl http://127.0.0.1:8080/health

max_completion_tokens is optional. Without it the server fills the remaining context. Speculation is set when the server starts and applies to every served turn.

With --api-key set, requests need Authorization: Bearer <key>. GET /health and GET /metrics stay public.

Docker

Build Tessera from source or pull a prebuilt image, then run it in Docker with your model weights mounted read-only and the required GPU devices passed through

Build from source and run

The repository ships a Dockerfile and docker/entrypoint.sh. One image uses one backend. The entrypoint runs serve and reads every flag from the environment, so a container needs no command line. Model weights stay outside the image and are mounted read only.

Build the ROCm image:

docker build -t tessera:rocm .

Build the Vulkan image:

docker build -t tessera:vulkan --build-arg TESSERA_BACKEND=vulkan .

The ROCm image uses rocm/dev-ubuntu-24.04:7.2. Keep it at the host driver version. Change it with --build-arg ROCM_IMAGE=.... A card other than the R9700 needs its own ISA: --build-arg GPU_ARCH=gfx1100.

The build uses gcc-15, from the Ubuntu toolchain test PPA. gcc-13 and gcc-14 internal-compiler-error on this code. CMake disables C++ module scanning, because the code uses no modules. The build runs two compile jobs, because the project documents compiler crashes under heavy parallel load. Raise the jobs on a large machine with --build-arg BUILD_JOBS=8.

Run the server. Mount the models directory and pass the GPUs through.

docker run --rm -it \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --group-add render \
  --ipc=host --security-opt seccomp=unconfined \
  -v ~/models:/models:ro -p 8080:8080 \
  -e TESSERA_MODEL=/models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf \
  -e TESSERA_GPUS=0 \
  -e TESSERA_SPECULATE=1 -e TESSERA_DRAFT_BLOCK=2 \
  -e TESSERA_CONTEXT=262144 -e TESSERA_KV=q8 \
  tessera:rocm

A Vulkan container needs only --device=/dev/dri. It does not need /dev/kfd.

Pull a prebuilt image

Prebuilt images are published to the GitHub Container Registry:

docker pull ghcr.io/quackster/tessera:rocm    # AMD ROCm, also tagged latest
docker pull ghcr.io/quackster/tessera:vulkan  # Vulkan

Run a pulled image with your own settings. The pulled image name replaces tessera:rocm in the run command above. Pass the -e flags for the settings you want. The entrypoint reads the environment on every start, so new -e values change the settings for that run and need no rebuild.

docker run --rm -it \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --group-add render \
  --ipc=host --security-opt seccomp=unconfined \
  -v ~/models:/models:ro -p 8080:8080 \
  -e TESSERA_MODEL=/models/Qwen3.8-27B-MXFP4-MTPFP8 \
  -e TESSERA_DRAFT=/models/Qwen3.8-27B-DFlash2-FP8 \
  -e TESSERA_GPUS=0 -e TESSERA_DRAFT_BLOCK=2 \
  -e TESSERA_CONTEXT=262144 -e TESSERA_KV=q8 \
  ghcr.io/quackster/tessera:rocm

An image whose environment is already baked in is edited by overriding the same variables at run time, or by rebuilding with new defaults. The environment section below lists every variable.

Docker environment

Each TESSERA_* variable becomes one serve flag. An unset variable keeps the CLI default. The container binds 0.0.0.0 and port 8080 by default. The list below applies to both the built and the pulled image.

Variable	Flag	Default
`TESSERA_MODEL`	`--model`	required
`TESSERA_DRAFT`	`--draft`	none
`TESSERA_MTP`	`--mtp`	none
`TESSERA_GPUS`	`--gpus`	0
`TESSERA_CONTEXT`	`--context`	32768
`TESSERA_KV`	`--kv-f16`, `--kv-q8`, `--kv-q4`, `--kv-fp8`	fp32
`TESSERA_KV_OFFLOAD`	`--kv-offload`	off
`TESSERA_GPU_LAYERS`	`--gpu-layers`	0
`TESSERA_SPECULATE`	`--speculate`	off
`TESSERA_DRAFT_BLOCK`	`--draft-block`	0
`TESSERA_PIPELINE`	`--pipeline`	off
`TESSERA_PIPELINE_SPLIT`	`--pipeline-split`	none
`TESSERA_TP`	`--tp`	off
`TESSERA_CP`	`--cp`	off
`TESSERA_SPEC_SPLIT`	`--spec-split`	off
`TESSERA_HYBRID`	`--hybrid`	off
`TESSERA_HOST`	`--host`	0.0.0.0
`TESSERA_PORT`	`--port`	8080
`TESSERA_API_KEY`	`--api-key`	none
`TESSERA_CALIBRATION`	`--calibration`	none
`TESSERA_ARGS`	raw extra flags	none

Set a boolean variable to 1, true, yes, or on to add its flag. TESSERA_ARGS holds raw flags for a setting with no variable. TESSERA_COMMAND selects the subcommand and defaults to serve.

The launch examples above map to the environment as follows.

# Flash-Next, MTP, two GPU pipeline.
-e TESSERA_MODEL=/models/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf \
-e TESSERA_MTP=/models/Qwen3.8-Flash-Next-MTP -e TESSERA_GPUS=0,1 \
-e TESSERA_SPECULATE=1 -e TESSERA_PIPELINE=1 -e TESSERA_PIPELINE_SPLIT=24,24 \
-e TESSERA_CONTEXT=262144 -e TESSERA_KV=q8 -e TESSERA_GPU_LAYERS=24

# MXFP4 target with the DFlash2 draft.
-e TESSERA_MODEL=/models/Qwen3.8-27B-MXFP4-MTPFP8 \
-e TESSERA_DRAFT=/models/Qwen3.8-27B-DFlash2-FP8 \
-e TESSERA_GPUS=0 -e TESSERA_DRAFT_BLOCK=2 -e TESSERA_CONTEXT=262144 -e TESSERA_KV=q8

Command-line arguments

The first argument selects the mode: run decodes a prompt, serve starts the HTTP server, calibrate sweeps the machine, fetch-mtp fetches an MTP head, and --list-gpus lists the devices and exits. Every mode accepts the same flag set, so one parser covers every command. A flag a command cannot honor is reported as ignored and the run continues; a conflicting combination of layout flags is an error. The table gives every flag, the modes that honor it, and its default.

Argument	Scope	Default	Meaning
`--model <path>`	both	required	A `.gguf` file or an MXFP4 model directory.
`--draft <dir>`	both	none	DFlash2 draft checkpoint directory.
`--context <n>`	both	32768	Maximum context length.
`--gpu-layers <n\|all>`	both	0	Layers kept in VRAM. The rest page from system RAM. 0 uploads everything. `help` prints counts without loading. With `--calibration`, `run` and `serve` take a saved layer count unless this flag overrides it.
`--draft-block <n>`	both	0	Draft block tokens. 0 uses the checkpoint block.
`--prefill-chunk <n>`	both	0	Prefill tokens per forward. 0 selects 512.
`--split-target <n>`	both	0	fp8 MXFP4 split-K target (ROCm only). 0 uses 320.
`--split-cap <n>`	both	0	fp8 MXFP4 split-K cap (ROCm only). 0 uses 4.
`--attention-split <n>`	both	0	Flash-decoding chunks at long context. 0 uses 16.
`--draft-context <n>`	both	0	DFlash2 draft context rows. 0 keeps the checkpoint window.
`--quiet`	both	off	Suppress progress and info logs.
`--prompt-text <str>`	run	none	Text prompt. It is tokenized, so the model needs a tokenizer.
`--mmproj <path>`	run	none	Vision projector (mmproj) GGUF.
`--image <path>`	run	none	Binary PPM image to prepend as tokens.
`--no-chat`	run	off	Do not apply the chat template.
`--max-completion-tokens <n>`	run	0	Completion tokens. 0 fills the remaining context.
`--max-thinking-tokens <n>`	run	0	Think-block budget per turn. 0 leaves thinking unlimited.
`--enable-thinking`	run, serve	on	Render the chat template with thinking enabled. Thinking is on by default. `serve` uses it as the default for requests that do not set one.
`--no-thinking`	run, serve	off	Render the chat template with thinking disabled. `serve` uses it as the request default.
`--reasoning-effort <level>`	run, serve	xhigh	Thinking difficulty for the chat template. Ignored with `--no-chat`. `serve` uses it as the request default.
`--speculate`	both	off	MTP speculative decode from the target head. Needs no checkpoint.
`--mtp <dir>`	both	unset	Directory of a separately fetched MTP head (from `tessera-cli fetch-mtp`) for a model whose GGUF ships none, such as the Flash-Next target.
`--sample`	run	off	Sample instead of greedy decode.
`--temperature <f>`	run	0.6	Sampling temperature.
`--top-p <f>`	run	0.95	Nucleus probability.
`--top-k <n>`	run	20	Keep the top n tokens.
`--min-p <f>`	run	0.0	Minimum probability.
`--presence-penalty <f>`	run	0.0	Presence penalty.
`--repetition-penalty <f>`	run	1.0	Repetition penalty.
`--seed <n>`	run	0	Sampling RNG seed.
`--kv-f16`	run	off	Store the KV cache in fp16.
`--kv-q8`	run	off	Store the KV cache in int8.
`--kv-q4`	run	off	Store the KV cache in 4-bit.
`--kv-fp8`	run	off	Store the KV cache in FP8 E4M3.
`--host <ip>`	serve	127.0.0.1	Bind address.
`--port <n>`	serve	8080	Bind port.
`--stats-interval <s>`	serve	5	Live tok/s cadence on streams and console. 0 disables.
`--no-auto-title`	serve	off	Keep the first user line as the chat title instead of asking the model.
`--api-key <k>`	serve	none	Accepted API key. Repeatable.
`--allow-origin <o>`	serve	none	CORS origin. Repeatable. `*` allows all origins.
`--calibration <file>`	run, serve, calibrate	env `TESSERA_CALIBRATION`	Calibration file. `run` and `serve` apply a saved entry for the current machine and model unless an explicit flag overrides it. The entry's layer count applies too. `calibrate` reads and writes it.
`--prompts <file>`	calibrate	three fixed prompts	Fixed prompts, one per non-empty line, for the calibration sweep.
`--calibrate-limit <n>`	calibrate	none	Measure at most n sweep points in total.
Parallelism			The flags below spread work across the `--gpus` devices.
`--gpus <a,b>`	run, serve, calibrate	0	One GPU index or a comma-separated list (for example `--gpus 0` or `--gpus 0,1`). `serve` runs one model replica per listed GPU, or one layout pool per group when a split flag is set. Before each replica loads, `serve` refuses it when its paged weights cannot fit host RAM with the replicas already loaded. `run` uses only the first device unless a split flag is set; `calibrate` runs the single-device sweep for one index and the parallel sweep for two or more.
`--pipeline`	run, serve	off	Split layers across the `--gpus` devices.
`--tp`	run, serve	off	Split attention, FFN and MoE experts across the `--gpus` devices.
`--cp`	run, serve	off	Split the key range across the `--gpus` devices.
`--spec-split`	run, serve	off	Run the MTP draft head on the second `--gpus` device. Needs `--speculate`.
`--hybrid`	run, serve	off	Data parallel groups of model parallel teams.
`--dp <n>`	run, serve	1	Group count for `--hybrid`.

The KV cache is fp32 when no --kv-* flag is set. The sampling defaults match the Qwen 3.8 27B defaults.

Environment variables:

TESSERA_API_KEY supplies --api-key when the flag is absent.
TESSERA_MXFP4_W4A8=1 enables the served MXFP4 target's W4A8 linear activation.
TESSERA_CALIBRATION supplies --calibration when the flag is absent.
TESSERA_MXFP4_SPLIT and TESSERA_MXFP4_SPLITCAP override the split-K target and cap as a diagnostic (they win over the request value).
TESSERA_TILED_MIN_ROWS, TESSERA_TILED_MIN_COLS and TESSERA_PREFILL_ATTN_PAIRS override the tiled-GEMM dispatch thresholds and the prefill attention work budget for measurement.

Build and test

Requirements:

CMake 3.30 or later.
A C++ compiler with C++23 support. The code stays C++20 style.
For Vulkan: Vulkan SDK and glslangValidator.
For ROCm: ROCm toolchain, rocminfo, and a supported AMD GPU.

Configure and build:

cmake -B cmake-build-vulkan -DTESSERA_BACKEND=vulkan
cmake --build cmake-build-vulkan -j

cmake -B cmake-build-rocm -DTESSERA_BACKEND=rocm
cmake --build cmake-build-rocm -j

CMake defaults to a Release build. Measure speed only on Release. Build serially. The gcc-15 toolchain crashes under parallel builds.

Run tests:

ctest --test-dir cmake-build-vulkan --output-on-failure
ctest --test-dir cmake-build-rocm --output-on-failure

All runtime coverage lives in one test binary. Add test functions to tests/*.cpp. Do not add new CMake test executables. Device tests skip cleanly when no device is present. Numerical checks use per backend tolerance.

Verified devices:

Device	Architecture	Memory	Status
AMD Radeon RX 7900 XTX	RDNA 3, Navi 31, gfx1100, 96 compute units	24 GB GDDR6	Verified on Vulkan (RADV NAVI31) and ROCm (gfx1100).
AMD Radeon AI PRO R9700	RDNA 4, gfx1201, 64 compute units	32 GB GDDR6	Verified on Vulkan (RADV) and ROCm (gfx1201).

State the backend used when reporting results.

Current progress

See PROGRESS.md.

License

Tessera uses the Apache License 2.0. See LICENSE.

Name		Name	Last commit message	Last commit date
docker		docker
docs		docs
include/tessera		include/tessera
src		src
tools		tools
.clang-format		.clang-format
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CMakeLists.txt		CMakeLists.txt
CREDITS.md		CREDITS.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
logo.svg		logo.svg

Navigation Menu

Repository navigation

About

Repository files navigation

Features

Chat UI

Two R9700s with Qwen3.8-Flash-Next MTP

One R9700 with Qwen3.8-27B MTP and DFlash2

Architecture

Model support

MXFP4 Qwen checkpoint preparation

Calibration

Usage

Docker

Docker environment

Command-line arguments

Build and test

Current progress

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Features

Chat UI

Two R9700s with Qwen3.8-Flash-Next MTP

One R9700 with Qwen3.8-27B MTP and DFlash2

Architecture

Model support

MXFP4 Qwen checkpoint preparation

Calibration

Usage

Docker

Docker environment

Command-line arguments

Build and test

Current progress

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages