[Tokenizer] Fix silent output truncation in encode/decode finalize

[jtuyls]

Summary

Return RESOURCE_EXHAUSTED instead of silently truncating when the output buffer is too small, matching the documented API contract in tokenizer.h.

Test results

HuggingFace smoketest (huggingface_smoketest.py)

1667/1667 tokenization comparisons pass across ~80 HuggingFace models (0 mismatches).

76 additional tests fail to load (not tokenization mismatches) — these are tiktoken models listed in the HF smoketest that aren't valid HuggingFace model identifiers. They're tested by the dedicated tiktoken smoketest instead. These tests are removed from the HF smoketest in #23830.

Tiktoken smoketest (tiktoken_smoketest.py)

72/76 tokenization comparisons pass across 4 tiktoken encodings (cl100k_base, o200k_base, r50k_base, p50k_base).

4 failures — identical between upstream and this PR (pre-existing, same as fix-tokenizer-added-tokens branch):

Encoding	Failing test	Cause
cl100k_base	special_token_endoftext	IREE matches <|endoftext|> as special token; tiktoken's encode_ordinary treats it as literal text
o200k_base	special_token_endoftext	Same
r50k_base	special_token_endoftext	Same
p50k_base	special_token_endoftext	Same

Root cause: the IREE tokenizer has no "encode ordinary" mode (equivalent to tiktoken's disallowed_special=()). Fixed in #23830.