Commit b5cd28b
committed
fix(merger): copy processor_config.json for custom processors
AutoProcessor.from_pretrained falls back to the base tokenizer for
custom processors (e.g. AeroRealtimeProcessor), so save_pretrained
does not emit processor_config.json. Copy it from the source
checkpoint if present.1 parent d1ac27a commit b5cd28b
1 file changed
Lines changed: 8 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
199 | 200 | | |
200 | 201 | | |
201 | 202 | | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
202 | 210 | | |
0 commit comments