Skip to content

Commit aae1209

Browse files
authored
[Rust] Bump huggingface tokenizer to 0.21.1 (mlc-ai#73)
This PR bumps the huggignface tokenizer dependency to version 0.21.1 to address the tokenizer issue in some latest models with latest trained tokenizers.
1 parent 4fbe996 commit aae1209

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

rust/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,6 @@ crate-type = ["staticlib"]
88

99
[dependencies]
1010

11-
tokenizers = { version = "0.21.0", default-features = false, features = ["onig"] }
11+
tokenizers = { version = "0.21.1", default-features = false, features = ["onig"] }
1212
serde = { version = "1.0", features = [ "derive" ] }
1313
serde_json = "1.0"

0 commit comments

Comments
 (0)