fix: add surrogate pair sanitization to prevent JSON API errors#1205
fix: add surrogate pair sanitization to prevent JSON API errors#1205
Conversation
Adding CLAUDE.md with task information for AI processing. This file will be removed when the task is complete. Issue: #1204
Add sanitizeSurrogates() utility that removes lone/orphaned Unicode surrogates (U+D800-U+DFFF) from strings before they're used in JSON serialization or API requests. This prevents the "no low surrogate in string" error from Anthropic's API when tool outputs contain binary data interpreted as text. Applied in: - lib.mjs: exported sanitizeSurrogates() utility - claude.lib.mjs: sanitize prompts before passing to Claude CLI - interactive-mode.lib.mjs: sanitize strings in safeJsonStringify() Includes case study analysis in docs/case-studies/issue-1204/, reproducible experiment, and 22 unit tests. Fixes #1204 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
konard
changed the title
This reverts commit 42acf14.
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost estimation:
Now working session is ended, feel free to review and add any feedback on the solution draft. |
|
🤖 AI Work Session Started Starting automated work session at 2026-01-31T00:22:01.972Z The PR has been converted to draft mode while work is in progress. This comment marks the beginning of an AI work session. Please wait working session to finish, and provide your feedback. |
Add changeset file documenting the patch-level fix for issue #1204. This changeset will trigger version bump to 1.14.2 in the next release. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
✅ PR Ready for ReviewAll CI checks have passed and the PR is now ready for review. Summary of ChangesAdded changeset file (
Merged latest main branch:
All tests passing:
Next StepsThis PR can now be merged. Upon merge, the changeset workflow will:
🤖 This PR is ready for human review and merge |
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost estimation:
Now working session is ended, feel free to review and add any feedback on the solution draft. |
1 participant
Summary
Fixes #1204 — "The request body is not valid JSON: no low surrogate in string"
Root Cause
Claude Code CLI doesn't sanitize tool outputs containing lone/orphaned Unicode surrogates (U+D800-U+DFFF) before sending the JSON request body to the Anthropic API. This is a known upstream bug that remains unfixed as of January 2026.
When tool outputs contain binary data interpreted as text (e.g., compiled Rust output,
.affiles, mutation testing progress bars), byte sequences in the surrogate range produce lone surrogates that are invalid in JSON per RFC 8259. JavaScript'sJSON.stringify()accepts them but Anthropic's server-side parser rejects them.Changes
src/lib.mjs: Added exportedsanitizeSurrogates()utility that replaces lone surrogates with U+FFFD (Unicode replacement character)src/claude.lib.mjs: Sanitize prompts before passing to Claude CLI, preventing surrogate corruption in the initial requestsrc/interactive-mode.lib.mjs: AddedsanitizeSurrogates()tosafeJsonStringify()replacer function, ensuring all string values in GitHub PR comments are cleandocs/case-studies/issue-1204/: Full case study with timeline, root cause analysis, and upstream referencesexperiments/issue-1204-surrogate-reproduction.mjs: Reproducible demonstration of the problem and fixtests/test-surrogate-sanitization.mjs: 22 unit tests covering all edge casesLimitations
This fix sanitizes content that hive-mind controls (prompts, interactive mode comments). The core issue — tool output corruption within Claude Code's internal conversation history — requires an upstream fix in
anthropics/claude-code. We've commented on the upstream issue with our analysis and suggested fix.Test Plan
node tests/test-surrogate-sanitization.mjs)node tests/test-interactive-mode.mjs)node experiments/issue-1204-surrogate-reproduction.mjs)npm run lint)CLAUDE_5_HOUR_SESSION_THRESHOLDunrelated to changes)This PR was created by the AI issue solver