Add raw frontier path for debug MTP chat reuse

[guelfoweb]

This PR adds a token-level raw frontier path for persistent MTP chat reuse, kept behind debug flags.

Problem:
Persistent MTP reuse across chat turns could corrupt output because the client and shim were using different frontier representations:

• visible assistant text retokenized by Python
• prompt-side/pre-generation boundary in the shim
• raw token-level frontier actually seen by the model

This could lead to corrupted follow-up output such as s,t,e,p... or empty-visible-content.

What changes:

• The shim exports raw emitted token ids and end-turn frontier token ids.
• The native client stores committed raw frontier tokens separately from visible chat frontier tokens.
• Debug-only chat follow-up reuse can use raw frontier + visible prefix matching.
• Missing raw frontier or prefix mismatch falls back to the existing safety path.
• Default behavior remains safety fallback for chat follow-ups.

Flags:

• Default: safety fallback remains active.

• Debug reuse:

  • ORBIT_MTP_CHAT_REUSE_RAW=1
  • ORBIT_MTP_CHAT_REUSE_DEBUG=1

Validation:

• PYTHONPATH=src python3 -m unittest tests.test_native_streaming tests.test_native_thinking tests.test_native_mtp_experimental tests.test_native_session_state tests.test_native_persistent_mtp -q PASS
• python3 -m compileall -q src tests scripts PASS
• git diff --check PASS

Smoke:
Default safety:

• follow-up chat uses safety fallback
• no corruption observed

Debug raw reuse:

• 4-turn chat passed

• follow-ups stayed on MTP

• reused tokens observed:

  • turn 2: 25
  • turn 3: 47
  • turn 4: 73

• no s,t,e,p...

• no empty-visible-content

• no control-token loop

Risk:
This does not promote multi-turn MTP reuse to default. It only makes the raw token-level path available behind debug flags. Default promotion should require broader multi-turn benchmark and stability validation.

Final decision:
Merge candidate as debug-only infrastructure.