Skip to content

fix: chunk IDs in hydrateEntryBylines to avoid D1 SQL variable limit#242

Closed
JiayuuWang wants to merge 1 commit intoemdash-cms:mainfrom
JiayuuWang:contribot/fix-d1-byline-chunk-ids
Closed

fix: chunk IDs in hydrateEntryBylines to avoid D1 SQL variable limit#242
JiayuuWang wants to merge 1 commit intoemdash-cms:mainfrom
JiayuuWang:contribot/fix-d1-byline-chunk-ids

Conversation

@JiayuuWang
Copy link
Copy Markdown

Summary

Fixes #219.

Cloudflare D1 enforces a SQL bound-parameter limit lower than SQLite's default 999. When getEmDashCollection returns a large result set, hydrateEntryBylines passed all entry IDs in one call to getBylinesForEntries, which builds unbounded IN (?, ?, …) clauses across three code paths:

  1. BylineRepository.getContentBylinesManyWHERE cb.content_id IN (…)
  2. getAuthorIds fallback — WHERE id IN (…)
  3. BylineRepository.findByUserIdsWHERE user_id IN (…)

On collections with 50+ entries this triggers D1_ERROR: too many SQL variables, and byline data is silently dropped (caught by the existing try/catch).

Fix

Chunk the ID array in hydrateEntryBylines into batches of 50 before calling getBylinesForEntries, then merge the result maps. This is the single control point for all three downstream queries. A batch size of 50 stays well within D1's limit while keeping round-trips low.

const CHUNK_SIZE = 50;
type BylinesMap = Awaited<ReturnType<typeof getBylinesForEntries>>;
const bylinesMap: BylinesMap = new Map();
for (let i = 0; i < ids.length; i += CHUNK_SIZE) {
  const chunkMap = await getBylinesForEntries(type, ids.slice(i, i + CHUNK_SIZE));
  for (const [k, v] of chunkMap) {
    bylinesMap.set(k, v);
  }
}

Test plan

  • Deploy a collection with 60+ published entries to a Cloudflare Workers environment
  • Call getEmDashCollection('posts', { limit: 100 }) — should succeed with byline data hydrated
  • Verify no D1_ERROR: too many SQL variables errors in the Worker logs

🤖 Generated with Claude Code

Cloudflare D1 enforces a bound-parameter limit lower than SQLite's
default 999. When getEmDashCollection returns a large result set,
hydrateEntryBylines passes all entry IDs in a single call to
getBylinesForEntries, which issues unbounded IN (...) clauses
through getContentBylinesMany, getAuthorIds, and findByUserIds.

This causes D1_ERROR: too many SQL variables on collections with
50+ entries, and byline data is silently dropped.

Fix: chunk IDs into batches of 50 before calling getBylinesForEntries,
then merge the result maps. 50 is conservative enough to stay within
D1's limit while keeping round-trips low.

Fixes emdash-cms#219

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@changeset-bot
Copy link
Copy Markdown

changeset-bot bot commented Apr 4, 2026

⚠️ No Changeset found

Latest commit: d7208ec

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 4, 2026


Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

@ascorbic
Copy link
Copy Markdown
Collaborator

Thanks for the contribution. This has already been fixed in #223 so I'm closing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hydrateEntryBylines exceeds D1 SQL variable limit on large collections

2 participants