Storage does not persist reliably if used in an async application

[DarthHater]

Hello again!

We are still using this library, and we've noticed that after a LOT of puts, our app continues to progress forward, and we end up exiting before the async queue can be processed. Is there any suggested way of dealing with this? In similar KV stores, there's been a .close() that wraps things up or etc... but all I've seen is _persist which looks to be private, and I don't want to touch it as a result :)

[DarthHater]

  private doPersist(): Promise<void> {
    return new Promise((resolve, reject) => {
      db._persist((err: any) => {
        if (err) {
          reject(err);
        }
        resolve();
      });
    })
  }

This is what we ended up doing to kinda work around this, and added a await this.doPersist() to where we do our puts.

[DarthHater]

Ok back to say that did NOT work, we are running into all sorts of issues with this not persisting before our application exits.

[DarthHater]

If we call _persist ourselves, as I expected, we shoot ourselves in the foot quite well, it seems that in cases past the initial save that it will collide with the library doing it, itself. Any ideas on how to approach this?

[amativos]

I can't give any actionable advice right now but your raise a good point. I actually encountered similar reliability issues myself some time ago. I originally wrote this library for use in an embedded system, and we're still running it in production. At some point we noticed intermittent data loss issues that were likely due to something similar to what you're describing. I spent some time digging into it, but eventually decided to delegate storage to a more reliable solution.

I updated the internals using the better-sqlite3 library to make all data persist to SQLite. But I wanted to keep the simplicity without changing any of our user code, so I ended up writing additional code to essentially flatten incoming key/value pairs before storing them in a table. better-sqlite3 also gave us fast synchronous writes. It worked out nicely and we haven't seen any issues since.

Unfortunately I haven't gotten around to publishing these changes. I made the above updates directly in our codebase. I'd need to be careful when publishing since better-sqlite3 is a non-trivial dependency and can break existing builds. There may be a way to fix the current implementation but I'd need to spend more time digging into it.

[kozmoz]

Had a similar issue and wrote a function that waits until all work by the data store is done. I use it just before exiting the application:

2020-04-03: improved function

/**
 * Wait until the file storage is done processing all tasks.
 *
 * @returns {Promise<void|Error>} resolves when done, rejects on error
 */
function waitForFileStorage() {
    return new Promise(resolveFn => {

        // Prevent that new items will be added to the queue.
        exports.getDataStore().queue.push = () => {
            // NOP
        };

        // Wait until the data store work queue is empty.
        const length = exports.getDataStore().queue.length();
        if (length === 0) {
            resolveFn();
            return;
        }
        logger.info(`Persistence store still busy, wait... (${length})`);
        exports.getDataStore().queue.drain = resolveFn;
    });
};