Commit 528d695
authored
[CMake] Enable SentencePiece tokenizer by default (mlc-ai#47)
The SentencePiece tokenizer was disabled by default to
reduce the binary size of the built library, while it causes
some error when users expect to use the SentencePiece
tokenizer.
This PR enables it by default. And we will need to manually
disable it if we need to reduce its binary size.1 parent 5de6f65 commit 528d695
1 file changed
+1
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
91 | | - | |
| 91 | + | |
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
| |||
0 commit comments