Open
Conversation
dejan-velimirovic-calendly
suggested changes
Feb 20, 2026
- Remove 30s waitUntil delay from per-batch task re-queue (was causing test timeouts since the original code had no such delay) - Add failedChunkData JSON field to batch collection so per-batch tasks can store chunk-level failure data independently - Aggregate failedChunkData from batch records in finalizeRunIfComplete() instead of relying on in-memory accumulation from the old single-task flow Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rker architecture Splits prepare-bulk-embedding into coordinator + per-collection workers. Each worker processes one page of one collection, queuing a continuation job before processing to ensure crash safety. Default batchLimit is 1000 when not explicitly set. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The second test was creating a separate Payload instance sharing the same DB and job queues, causing two crons to compete for jobs. This led to double-execution and mock state inconsistency (expected 4 to be 2). Now both tests use the single beforeAll instance with cleanup between. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Every test file that creates a Payload instance now calls payload.destroy() in afterAll (or try/finally for in-test instances). This stops background cron jobs from accumulating across tests, which was causing heap exhaustion in CI. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --max-old-space-size=8192 to test:int NODE_OPTIONS (cross-env was overriding the CI env var, so the heap limit never took effect) - Fix polling.spec.ts queueSpy assertions: coordinator/worker adds an extra queue call, so poll-or-complete-single-batch is now call 3 and 4 instead of 2 and 3 - Add extensive [vectorize-debug] console.log throughout task handlers (coordinator, worker, poll-single, finalize, streamAndBatchDocs) to diagnose any remaining CI hangs - Remove redundant NODE_OPTIONS from CI workflow (now in the script) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ncrementally Remove the backward-compatible fan-out task since the per-batch architecture hasn't been released yet. Refactor finalizeRunIfComplete to aggregate batch counts incrementally during pagination instead of collecting all batch objects into memory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bump version 0.5.4 → 0.5.5 - Add 0.5.5 entry to CHANGELOG.md (coordinator/worker, batchLimit, per-batch polling) - Document batchLimit in README CollectionVectorizeOption section - Remove all diagnostic console.log statements from bulkEmbedAll.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.