Add mmap-io-backed append-only file, mmap-backed Index, and mmap-backed Partition with direct mapped-buffer reads#315

Draft

Copilot wants to merge 23 commits into

mainfrom

copilot/implement-mmap-storage-layer

Commits on May 14, 2026

Add append-only mapped file abstraction with tests and benchmark

Copilot
and
albe
authored
Add mapped file throughput benchmark script

Copilot
and
albe
authored
Use @riaskov/mmap-io for true file memory mapping

Copilot
and
albe
authored
Refactor AppendOnlyMmapedFile: move write-only methods to writable subclass, use mmap.PAGESIZE, remove dirty tracking

Copilot
and
albe
authored
Move pageSize to writable subclass, simplify writeBufferSize expression

Copilot
and
albe
authored
Add mmap-backed Index implementation beside existing Index (MmapReadableIndex, MmapWritableIndex, MmapReadOnlyIndex)

Copilot
and
albe
authored
Address code review: remove redundant isOpen guard in close(), drop eslint-disable comment

Copilot
and
albe
authored
planning: file.write(size) API + bench + caching

Copilot
and
albe
authored
Add write(size) API, entry caching, hasPendingData flush guard, and MmapIndex bench

Copilot
and
albe
authored
Split write(data) and reserve(size), add cache forward-trim, clear markerWasInvalid after recovery

Copilot
and
albe
authored
Refactor index benchmark to shared writer/readers with scaling steps

Copilot
and
albe
authored
Split benchmark read ranges across readers

Copilot
and
albe
authored
Implement mmap partition classes and benchmark updates

Copilot
and
albe
authored
Fix mmap partition exports and run storage benchmark

Copilot
and
albe
authored
Address review feedback and rerun storage benchmark

Copilot
and
albe
authored
Replace deprecated substr usage in mmap readable partition

Copilot
and
albe
authored
Fix mmap partition review issues and rerun validations

Copilot
and
albe
authored
Use mmap buffer directly in mmap partition read and write paths

Copilot
and
albe
authored
perf(mmap): defer marker write to flush, skip fdatasync when syncOnFlush=false, remove redundant mmap.sync

Copilot
and
albe
authored
perf(mmap): address review - clarify comments in flush()

Copilot
and
albe
authored
Add forked storage benchmark script

Copilot
and
albe
authored
Add forked multi-process storage benchmark

Copilot
and
albe
authored

Commits on May 22, 2026

Optimize mmap grow path to keep marker writes flush-driven

Copilot
and
albe
authored

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add mmap-io-backed append-only file, mmap-backed Index, and mmap-backed Partition with direct mapped-buffer reads#315

Add mmap-io-backed append-only file, mmap-backed Index, and mmap-backed Partition with direct mapped-buffer reads#315
Copilot wants to merge 23 commits into
mainfrom
copilot/implement-mmap-storage-layer

Commits on May 14, 2026

Add append-only mapped file abstraction with tests and benchmark

Add mapped file throughput benchmark script

Use @riaskov/mmap-io for true file memory mapping

Refactor AppendOnlyMmapedFile: move write-only methods to writable subclass, use mmap.PAGESIZE, remove dirty tracking

Move pageSize to writable subclass, simplify writeBufferSize expression

Add mmap-backed Index implementation beside existing Index (MmapReadableIndex, MmapWritableIndex, MmapReadOnlyIndex)

Address code review: remove redundant isOpen guard in close(), drop eslint-disable comment

planning: file.write(size) API + bench + caching

Add write(size) API, entry caching, hasPendingData flush guard, and MmapIndex bench

Split write(data) and reserve(size), add cache forward-trim, clear markerWasInvalid after recovery

Refactor index benchmark to shared writer/readers with scaling steps

Split benchmark read ranges across readers

Implement mmap partition classes and benchmark updates

Fix mmap partition exports and run storage benchmark

Address review feedback and rerun storage benchmark

Replace deprecated substr usage in mmap readable partition

Fix mmap partition review issues and rerun validations

Use mmap buffer directly in mmap partition read and write paths

perf(mmap): defer marker write to flush, skip fdatasync when syncOnFlush=false, remove redundant mmap.sync

perf(mmap): address review - clarify comments in flush()

Add forked storage benchmark script

Add forked multi-process storage benchmark

Commits on May 22, 2026

Optimize mmap grow path to keep marker writes flush-driven