High-performance .NET BPE tokenizer — up to 618 MiB/s, competitive with Rust. Zero-allocation counting, multilingual cache, o200k/cl100k/r50k/p50k encodings + HuggingFace tokenizer.json support.
sdk ai csharp dotnet high-performance tokenizer openai zero-allocation bpe huggingface tiktoken gpt4o o200k-base cl100k-base
-
Updated
Apr 14, 2026 - C#