Implement auto-repair by albe · Pull Request #107 · albe/node-event-storage

albe · 2020-05-31T19:00:25Z

Implements automatic recovery from unfinished commits and makes sure that indexes are not containing elements that were lost.
Also detects if the primary index has fallen behind. A repair for that case will be added after #254 is merged.

Related to #31

albe · 2020-06-28T14:51:14Z

+     * @private
+     */
+    checkTornWrite() {
+        let position = this.storage.length;


To fix torn writes (which should be a storage level concern) we need to infer the position from the actual documents in the storage rather than the information from the index. Hence the storage needs to read the last document of each partition and return the highest document header sequenceNumber.

albe · 2021-02-27T23:34:42Z

+        const indexes = [];
+
+        let leastConsistentEntry = this.storage.index.lastEntry;
+        this.storage.forEachSecondaryIndex(index => {


Note: Updating secondary indexes is problematic, because they can fall back behind the primary index. However, we do not know if an index has actually fallen behind or if it just didn't match any later documents. This leads to the worst case of one secondary index only matching the first document written, but no other, which would mean the whole storage needs to be reindexed.

albe · 2021-05-16T15:35:27Z

        };
        for (let event of events) {
-            const eventMetadata = Object.assign({ commitId, committedAt }, metadata, { commitVersion, streamVersion });
+            const eventMetadata = Object.assign({ commitId, committedAt }, metadata, { commitVersion, commitSize, streamVersion });


This is already implemented as of #159

albe · 2021-05-16T15:41:06Z

This needs to be rewritten to only deal with unfinished commits as the torn writes has already been covered by #155

albe · 2026-03-20T20:02:17Z

@copilot continue working on this. Take the previous comments into consideration and check why the tests are currently not finishing

Copilot · 2026-03-20T20:02:24Z

@albe I've opened a new pull request, #251, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/f62b2596-872d-4f02-aedf-f9da349f654b

Fix auto-repair: remove broken incomplete code causing infinite loop in checkUnfinishedCommits

coveralls · 2026-03-20T20:21:13Z

coverage: 97.958% (+0.1%) from 97.826%
when pulling 967ba2d on auto-repair
into 28d4e34 on main.

albe · 2026-03-20T22:01:52Z

@copilot add tests that check that a repair (torn write/commit) will not lose any previously committed data.

Copilot · 2026-03-20T22:01:58Z

@albe I've opened a new pull request, #252, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/312ca871-a86e-472c-ace8-51858ee39989

Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/06d32a8b-3e1a-4041-a3a3-4a81c46bd21a

Repair secondary indexes when opened after primary index truncation

albe · 2026-03-20T22:58:32Z

@copilot find uncovered lines in this change and improve test coverage, since coveralls is still reporting -0.3% overall coverage on this PR

Copilot · 2026-03-20T22:58:39Z

@albe I've opened a new pull request, #253, to work on those changes. Once the pull request is ready, I'll request review from you.

…remove dead createStorage, add stale-index test Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/3c271323-430d-4355-a4ff-66d289bf2e48

albe · 2026-03-21T15:04:32Z

        try {
            this.readFrom(after);
+            // copy all truncated documents to some delete log
+            this.branchOff('truncated-' + Date.now(), after);


@copilot actually, putting this into the try was wrong. It needs to be guaranteed that when the partition file is truncated, the truncated part is stored into a new file (which is not supposed to be read by the library, but for manual recovery attempts). This is important when recovery finds corrupted data. The event store needs to return to a consistent state, but the malformed data might still contain valuable information.

Copilot · 2026-03-21T15:04:39Z

@albe I've opened a new pull request, #259, to work on those changes. Once the pull request is ready, I'll request review from you.

…data Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/bd5b05a2-8387-4400-b7d5-aa21091a308e

Guarantee truncated data is saved before partition truncation, even when corrupted

albe · 2026-03-21T15:28:40Z

@copilot take a look at the failing test

Copilot · 2026-03-21T15:28:45Z

@albe I've opened a new pull request, #260, to work on those changes. Once the pull request is ready, I'll request review from you.

…y error Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/44a7f6dc-8449-4945-b8c7-edd9acbddaee

Fix bench-index: use per-call unique filenames to avoid consistency error

albe · 2026-03-21T15:44:29Z

@copilot there is still a failing test:

  1) EventStore
       preRead
         aborts the read when the hook throws:
     Uncaught Error: Not authorized to read
      at WritableStorage.<anonymous> (test/EventStore.spec.js:1290:27)
      at WritableStorage.emit (node:events:524:28)
      at WritableStorage.readFrom (src/Storage/ReadableStorage.js:87:173)
      at WritableStorage.read (src/Storage/ReadableStorage.js:94:545)
      at EventStore.checkUnfinishedCommits (src/EventStore.js:27:302)
      at /home/runner/work/node-event-storage/node-event-storage/src/EventStore.js:23:854
      at /home/runner/work/node-event-storage/node-event-storage/src/EventStore.js:36:507
      at FSReqCallback.oncomplete (node:fs:187:23)

Copilot · 2026-03-21T15:44:35Z

@albe I've opened a new pull request, #261, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/892aa17c-c679-47ec-8a06-df725d6f9a7c

Fix uncaught error when preRead hook throws during checkUnfinishedCommits

albe · 2026-03-22T10:19:07Z

@copilot there is one more thing to consider: is there some scenario in which the primary index can run short of the actual documents stored in the storage? If so, how can we detect this status efficiently at startup time? When we do detect this condition, with #254 we will then be able to also repair the primary index. But in this PR we just want to detect this situation if it's even realistic, not yet resolve it.

Copilot · 2026-03-22T10:19:13Z

@albe I've opened a new pull request, #264, to work on those changes. Once the pull request is ready, I'll request review from you.

…ent in checkTornWrites() Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/a683727c-251a-4224-9d79-a1a6f3a68323

…n convention Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/0a866da0-b3ca-485a-a159-2fe01e2a8443

…Header in checkTornWrite() Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/cd6d455a-7dd3-462e-a3be-61c6eccf8112

Detect primary index lagging behind partition data at startup

Start implementing auto-repair

2bb334e

albe added the enhancement label May 31, 2020

albe mentioned this pull request Jun 27, 2020

Project Status #29

Open

albe commented Jun 28, 2020

View reviewed changes

albe mentioned this pull request Jul 19, 2020

Split partitions into blocks/chunks #122

Open

albe mentioned this pull request Feb 9, 2021

Separate documents by a unique sequence and make partition backwards scannable #145

Merged

albe mentioned this pull request Feb 27, 2021

Automatically repair torn writes on opening #155

Merged

albe commented Feb 27, 2021

View reviewed changes

albe commented May 16, 2021

View reviewed changes

Merge branch 'main' into auto-repair

10bbc59

Initial plan

26945c3

Copilot AI mentioned this pull request Mar 20, 2026

Fix auto-repair: remove broken incomplete code causing infinite loop in checkUnfinishedCommits #251

Merged

Copilot AI and others added 2 commits March 20, 2026 20:09

Fix checkUnfinishedCommits - remove broken code causing infinite loop

75ddaf9

Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/f62b2596-872d-4f02-aedf-f9da349f654b

Merge pull request #251 from albe/copilot/sub-pr-107

b56a267

Fix auto-repair: remove broken incomplete code causing infinite loop in checkUnfinishedCommits

Initial plan

079cfa0

Copilot AI mentioned this pull request Mar 20, 2026

Repair secondary indexes when opened after primary index truncation #252

Merged

Copilot AI and others added 3 commits March 20, 2026 22:24

Add tests verifying repair does not lose previously committed data

0a6183f

Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/312ca871-a86e-472c-ace8-51858ee39989

Repair secondary indexes when opened after primary index truncation

e60f448

Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/06d32a8b-3e1a-4041-a3a3-4a81c46bd21a

Merge pull request #252 from albe/copilot/sub-pr-107

b79e1e8

Repair secondary indexes when opened after primary index truncation

Initial plan

209ea30

Copilot AI mentioned this pull request Mar 20, 2026

Fix stale-index repair bug and improve coverage in checkUnfinishedCommits #253

Merged

albe commented Mar 21, 2026

View reviewed changes

Initial plan

efee33e

Copilot AI mentioned this pull request Mar 21, 2026

Guarantee truncated data is saved before partition truncation, even when corrupted #259

Merged

Copilot AI and others added 2 commits March 21, 2026 15:05

Fix: guarantee branchOff runs before truncation, even with corrupted …

ebc6385

…data Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/bd5b05a2-8387-4400-b7d5-aa21091a308e

Merge pull request #259 from albe/copilot/sub-pr-107

53a59d5

Guarantee truncated data is saved before partition truncation, even when corrupted

Initial plan

6c02882

Copilot AI mentioned this pull request Mar 21, 2026

Fix bench-index: use per-call unique filenames to avoid consistency error #260

Merged

Copilot AI and others added 2 commits March 21, 2026 15:31

Fix bench-index.js: use unique filenames per call to avoid consistenc…

af1107d

…y error Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/44a7f6dc-8449-4945-b8c7-edd9acbddaee

Merge pull request #260 from albe/copilot/sub-pr-107

2608a02

Fix bench-index: use per-call unique filenames to avoid consistency error

Initial plan

03002e3

Copilot AI mentioned this pull request Mar 21, 2026

Fix uncaught error when preRead hook throws during checkUnfinishedCommits #261

Merged

Copilot AI and others added 2 commits March 21, 2026 15:51

Fix: checkUnfinishedCommits gracefully handles preRead hook errors

26150b5

Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/892aa17c-c679-47ec-8a06-df725d6f9a7c

Merge pull request #261 from albe/copilot/sub-pr-107

2c94d36

Fix uncaught error when preRead hook throws during checkUnfinishedCommits

albe changed the title ~~Start implementing auto-repair~~ Implement auto-repair Mar 21, 2026

albe mentioned this pull request Mar 21, 2026

Failure recovery #31

Closed

Initial plan

4fa67b6

Copilot AI mentioned this pull request Mar 22, 2026

Detect primary index lagging behind partition data at startup #264

Merged

Copilot AI and others added 4 commits March 22, 2026 10:39

Detect primary-index-lagging: add getLastSequenceNumber() and emit ev…

436b007

…ent in checkTornWrites() Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/a683727c-251a-4224-9d79-a1a6f3a68323

Merge getLastSequenceNumber() into checkTornWrite() with signed retur…

6d1b2a1

…n convention Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/0a866da0-b3ca-485a-a159-2fe01e2a8443

Deduplicate findDocumentPositionBefore/prepareReadBuffer/readDocument…

f6b934c

…Header in checkTornWrite() Co-authored-by: albe <4259532+albe@users.noreply.github.com> Agent-Logs-Url: https://github.com/albe/node-event-storage/sessions/cd6d455a-7dd3-462e-a3be-61c6eccf8112

Merge pull request #264 from albe/copilot/sub-pr-107

967ba2d

Detect primary index lagging behind partition data at startup

albe merged commit bf45844 into main Mar 22, 2026
10 checks passed

albe deleted the auto-repair branch March 22, 2026 11:40

Conversation

albe commented May 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

albe Jun 28, 2020

Choose a reason for hiding this comment

Uh oh!

albe Feb 27, 2021

Choose a reason for hiding this comment

Uh oh!

albe Feb 27, 2021

Choose a reason for hiding this comment

Uh oh!

albe May 16, 2021

Choose a reason for hiding this comment

Uh oh!

albe commented May 16, 2021

Uh oh!

albe commented Mar 20, 2026

Uh oh!

Copilot AI commented Mar 20, 2026

Uh oh!

coveralls commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

albe commented Mar 20, 2026

Uh oh!

Copilot AI commented Mar 20, 2026

Uh oh!

albe commented Mar 20, 2026

Uh oh!

Copilot AI commented Mar 20, 2026

Uh oh!

albe Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Mar 21, 2026

Uh oh!

albe commented Mar 21, 2026

Uh oh!

Copilot AI commented Mar 21, 2026

Uh oh!

albe commented Mar 21, 2026

Uh oh!

Copilot AI commented Mar 21, 2026

Uh oh!

albe commented Mar 22, 2026

Uh oh!

Copilot AI commented Mar 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

albe commented May 31, 2020 •

edited

Loading

coveralls commented Mar 20, 2026 •

edited

Loading