Skip to content

feat: split long input into chunks for translation#56

Merged
min0625 merged 1 commit into
mainfrom
feat/long-input-chunking
Jul 7, 2026
Merged

feat: split long input into chunks for translation#56
min0625 merged 1 commit into
mainfrom
feat/long-input-chunking

Conversation

@min0625

@min0625 min0625 commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Summary

  • Long input (over 2000 runes) is split at paragraph boundaries (falling back to line breaks, then whitespace) and translated chunk by chunk, so no single request can hit the model's output-token limit and come back truncated
  • Each chunk's original separator (paragraph gap, line break, single space, or nothing) is reprinted verbatim between chunk outputs, so internal whitespace survives exactly regardless of how the document was split
  • Rotation-mode language detection samples a single representative chunk instead of the whole document, so detection cost doesn't grow with document length; a neutral sample only short-circuits output for single-chunk documents
  • Marks "Batch translation mode" as done in the README roadmap

Test plan

🤖 Generated with Claude Code

Input over 2000 runes is split at paragraph boundaries (falling back to
line breaks, then whitespace) and translated chunk by chunk so a long
document can't hit the model's output-token limit and come back
truncated. Each chunk's original separator is reprinted verbatim so
whitespace survives exactly; rotation-mode detection samples a single
chunk instead of the whole document.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.44944% with 17 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
cmd/mint/main.go 85.52% 6 Missing and 5 partials ⚠️
cmd/mint/chunk.go 93.93% 3 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@min0625 min0625 merged commit b4dffdc into main Jul 7, 2026
6 checks passed
@min0625 min0625 deleted the feat/long-input-chunking branch July 7, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant