v3.1.4

Full Changelog: v3.1.3...v3.1.4

v3.1.3

Full Changelog: v3.1.2...v3.1.3

v3.1.2

Full Changelog: v3.1.1...v3.1.2

v3.1.1

Full Changelog: v3.1.0...v3.1.1

v3.1.0-rc.1

Full Changelog: v3.0.0...v3.1.0-rc.1

v3.1.0

Full Changelog: v3.0.0...v3.1.0

v3.0.0

Highlights

Peak throughput: 618 MiB/s — up to 42x faster than other .NET tokenizers on multilingual/CJK text.

Performance improvements

• O(n log n) min-heap BPE merge — replaces O(n²) linear scan in FindParts, improving cold-path CJK/multilingual by ~13%
• Restored ASCII fast path — fixes a 35% regression on cached CJK/multilingual text introduced in v2.3.0
• Cache speedup: 5-13x on repeated multilingual/CJK text (was 4-9x)

CountTokens — zero allocation

Input	Tiktoken	Throughput	vs competitors
Hello, World! (13 B)	88 ns	141 MiB/s	1.9-3.6x
Multilingual (382 B)	1.1 us	339 MiB/s	4.7-13.6x
CJK-heavy (1,676 B)	2.6 us	618 MiB/s	14.3-42.3x
Python code (879 B)	5.5 us	153 MiB/s	1.8-4.0x
Multilingual long (4,312 B)	9.0 us	458 MiB/s	7.9-31.6x
Bitcoin whitepaper (19,884 B)	105.1 us	180 MiB/s	2.4-3.8x

New

• ColdPath benchmark project — dedicated benchmark for cache vs no-cache comparison
• SmokeTest project — local perf regression guard using machine-independent cache speedup ratios
• Aho-Corasick investigation — research document on alternative tokenization approaches

Cross-language context (Apple M4 Max, o200k_base)

Implementation	Encode Throughput	CountTokens Throughput
Tiktoken .NET (cached)	114-484 MiB/s	141-618 MiB/s
Tiktoken .NET (no cache)	44-145 MiB/s	47-155 MiB/s
tiktoken Rust v3	34-88 MiB/s	—
GitHub bpe Rust v0.3	33-64 MiB/s	29-66 MiB/s
OpenAI tiktoken Python 0.12	7-20 MiB/s	—