Fix: respect -t/--tokenizer CLI flag over sibling tokenizer/ directory

[mario-sanz]

Fixes #609

Problem

In convert_checkpoint_to_hf.py, the -t/--tokenizer CLI flag is silently ignored when a sibling tokenizer/ directory exists next to the checkpoint. The code unconditionally enters the if tokenizer_path.exists() branch, so the user-provided tokenizer_id is never consulted.

This is a common scenario for SFT checkpoints, which are typically saved with an adjacent tokenizer/ directory.

Fix

Add a tokenizer_id is None guard to the condition on convert_checkpoint_to_hf.py, so the sibling tokenizer/ directory is only used as a fallback when no explicit -t override was provided.

Precedence after the fix:

1. Explicit -t/--tokenizer CLI argument (highest priority)
2. Sibling tokenizer/ directory next to the checkpoint
3. tokenizer_config.identifier from the checkpoint metadata

This matches the behavior already present in the hybrid converter (convert_checkpoint_to_hf_hybrid.py).

Tests

Added convert_checkpoint_to_hf_tokenizer_test.py with 5 test cases covering the full tokenizer selection logic:

• test_tokenizer_id_used_when_no_sibling_dir: -t is used when no sibling tokenizer/ directory exists.
• test_tokenizer_id_overrides_sibling_dir: -t takes precedence when a sibling tokenizer/ directory also exists (the core bug scenario). Before the fix, this test fails, but after the fix, it succeeds.
• test_sibling_dir_used_when_no_tokenizer_id: The sibling tokenizer/ directory is used as a fallback when -t is not provided.
• test_config_identifier_used_as_fallback: TokenizerConfig.identifier is used when neither -t nor a sibling directory is available.
• test_no_tokenizer_saved_when_nothing_available: Tokenizer saving is skipped entirely when no source is available.

[tyler-romero]

Hey! Thanks for the fix. This looks good to me but can you actually remove the test file? AI generated like 6 tests for just the if/else related to the tokenizer_id which is huge overkill.

[mario-sanz]

Hey! Thanks for the fix. This looks good to me but can you actually remove the test file? AI generated like 6 tests for just the if/else related to the tokenizer_id which is huge overkill.

Hi @tyler-romero - done!

Actually, I thought it would be a good idea to include tests for all possible cases (the 5 combinations of using the tokenizer with the -t flag and the tokenizer/ directory), but if you think it's not necessary, let's leave it as it is!

[mario-sanz]

@tyler-romero, could you take a look at this? Thanks!