Skip to content

perf(import): convert JSON array rows lazily during import#1541

Merged
datlechin merged 1 commit into
mainfrom
fix/json-import-array-memory
Jun 1, 2026
Merged

perf(import): convert JSON array rows lazily during import#1541
datlechin merged 1 commit into
mainfrom
fix/json-import-array-memory

Conversation

@datlechin

Copy link
Copy Markdown
Member

Follow-up to #1536 (post-merge review).

JSONImportPlugin.performImport built the entire array import into a [[String: PluginCellValue]] buffer (parseRows did extractRows(...).map(convertRow)) before inserting any row. For a large .json array that's a full duplicate of the row collection held alongside the parsed object graph.

This converts each row to PluginCellValue lazily at insert time instead: parseRows now returns the raw [[String: Any]] and the loop calls convertRow per row. Peak memory drops by roughly that extra buffer. No behavior change.

Note: the deeper cost (the whole file + JSONSerialization object graph in memory) is inherent to the array path; NDJSON/JSONL already streams via url.lines. A streaming array parser would be a separate, larger change.

Tests unchanged (they exercise extractRows/convertRow/parseRow directly, none of which changed signature). Local full-suite validation was blocked by an unrelated DerivedData/SPM checkout issue on my machine; relying on CI's clean build here.

@datlechin datlechin merged commit 00ff7c5 into main Jun 1, 2026
2 of 3 checks passed
@datlechin datlechin deleted the fix/json-import-array-memory branch June 1, 2026 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant