Endless loop of i/o-hammering

[willosof]

Sometimes the process goes into an endless loop of writing between the filename, temp filename and backup filename. Crazy IO intensive, but luckily my application (in production!) still works, so crisis avoided.

Everything is idle in the application itself and on the rest over the server, but IO works like crazy:

$ uptime
 19:01:58 up 18 days,  3:19,  4 users,  load average: 1.87, 1.90, 1.97
$ vmstat -w 1
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
 r  b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st
 0  1            0      7107944      1410072      6929344    0    0     0     7    1    0   0   0 100   0   0
 0  1            0      7107944      1410072      6929344    0    0     0  3024  808 2869   1   0  92   7   0
 0  1            0      7106820      1410076      6929480    0    0     0  2980  809 2714   1   0  92   7   0
 0  1            0      7106704      1410076      6929480    0    0     0  2884  683 2715   1   0  93   7   0
 3  0            0      7106704      1410076      6929480    0    0     0  2844  645 2708   1   0  92   7   0

ls can't even see the storage.json anymore, it's more like

-rw-rw-r--    1 williamv williamv  70530 Apr 21 19:08 storage.json~
-rw-rw-r--    1 williamv williamv  70530 Apr 21 19:08 storage.json~~

and

-?????????    ? ?        ?             ?            ? storage.json

and

-?????????    ? ?        ?             ?            ? storage.json~
-?????????    ? ?        ?             ?            ? storage.json~~

It would be a very good idea to throttle actual saving to disk using lodash throttle or something like that with force save intervals :)