libtokenizers C/C++ bindings for Huggingface Tokenizers. Acknowledgements MLC tokenizers-cpp. We take inspiration from their work, and expand it to support more platforms & architectures.
