Utilize multithreading worker to compress wdb

[tanaykdb]

Modified compressfromtable function to leverage the worker multithreading capability

[tanaykdb]

started wdb with this config

localhost,{KDBBASEPORT}+5,wdb,wdb1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,1,,,${KDBCODE}/processes/wdb.q,1,,

created a dummy table tradetest with 1M rows in the wdb and did a savedown after ensuring no slave threads

n:2000000
tradetest:([] time:n?.z.t;sym:n?`AAPL`GOOG`MSFT`TSLA`AMZN;price:n?100f;size:n?1000i;side:n?`buy`sell)
count tradetest
2000000
tradetest
time         sym  price    size side
------------------------------------
02:38:58.781 AAPL 67.92309 25   buy
06:27:19.864 GOOG 78.06477 905  sell
02:17:00.720 GOOG 85.0319  525  sell
09:18:31.947 GOOG 59.80458 885  sell
09:10:47.616 AMZN 33.61348 966  sell
09:48:38.971 TSLA 38.00265 528  sell
03:08:29.925 AMZN 15.42322 551  buy
00:39:33.802 AAPL 59.50376 705  buy
04:00:45.124 AMZN 81.53233 685  buy
07:54:59.152 AMZN 0.705262 779  sell
09:43:15.759 TSLA 7.51429  304  buy
00:19:51.520 AAPL 64.30961 686  sell
05:35:42.188 AAPL 12.84528 721  sell
07:33:44.062 GOOG 97.48562 777  sell
04:51:41.097 MSFT 43.20784 427  sell
04:37:23.936 GOOG 66.70407 358  buy
08:50:22.131 MSFT 97.02195 49   buy
08:35:05.825 AAPL 24.88348 622  sell
..
:46655>
.cmp.compressfromtable
{[table]
    statstab::([] file:`$(); algo:`int$(); compressedLength:`long$();uncompressedLength:`long$());
    / Check if process is single threaded - if multi then compress in parallel then clean up after
    // Add metrics on any files due to be compressed to be used afterwards for comparison
    table:update compressionvaluepre:{(-21!x)`compressedLength}'[fullpath] from table;
    $[0= system"s";
        singlethreadcompress[table];
        $[0<system "s";multithreadcompress[table];{[x;y] compressfromtable select from x where table=y}[table;] peach distinct table`table]];
    / Update the stats tab table after the compression
    {statstabupdate[x`fullpath;x`calgo;x`currentsize;x`compressionvaluepre]} each table}
meta tradetest
c    | t f a
-----| -----
time | t
sym  | s
price| f
size | i
side | s
system "s"
0i
\t .u.end[.z.d]
144

Checked in hdb for data

tmathur@homer:~$ qcon :46653:admin:admin
tables[]
`heartbeat`logmsg`packets`quote`trade`tradetest
select count i by date from tradetest
date      | x
----------| -------
2026.03.30| 2000000
100#select from tradetest where date=.z.d
date       time         sym  price    size side
-----------------------------------------------
2026.03.30 00:00:00.014 AAPL 57.72587 628  sell
2026.03.30 00:00:00.063 AAPL 31.67625 614  sell
2026.03.30 00:00:00.093 AAPL 63.47609 481  sell
2026.03.30 00:00:00.167 AAPL 44.24649 39   sell
2026.03.30 00:00:00.181 AAPL 98.29237 146  sell
2026.03.30 00:00:00.284 AAPL 93.66673 963  buy
2026.03.30 00:00:00.331 AAPL 79.40124 749  buy
2026.03.30 00:00:00.390 AAPL 45.26111 629  buy
2026.03.30 00:00:00.473 AAPL 5.547214 164  buy
2026.03.30 00:00:00.546 AAPL 22.30016 290  sell
2026.03.30 00:00:00.708 AAPL 57.25001 581  sell
2026.03.30 00:00:00.883 AAPL 58.51943 982  buy
2026.03.30 00:00:00.946 AAPL 9.149578 256  buy
2026.03.30 00:00:00.967 AAPL 23.81385 250  buy
2026.03.30 00:00:00.991 AAPL 39.81918 260  buy
2026.03.30 00:00:01.210 AAPL 43.31621 247  buy
2026.03.30 00:00:01.272 AAPL 55.46391 757  sell
2026.03.30 00:00:01.522 AAPL 95.34102 848  sell
..
:46653>

now deleted all data from hdb for testing and restarted the stack

tmathur@homer:~/TorQ-Finance-Starter-Pack/appconfig$ cd
tmathur@homer:~$ bash bin/torq.sh stop all
14:17:19 | Shutting down discovery1...
14:17:19 | Shutting down stp1...
14:17:19 | Shutting down rdb1...
14:17:19 | Shutting down hdb1...
14:17:20 | Shutting down hdb2...
14:17:20 | Shutting down wdb1...
14:17:20 | Shutting down sort1...
14:17:20 | Shutting down gateway1...
14:17:21 | Shutting down housekeeping1...
14:17:21 | Shutting down feed1...
14:17:21 | sctp1 is not currently running
14:17:21 | Shutting down sortworker1...
14:17:21 | Shutting down sortworker2...
14:17:22 | Shutting down metrics1...
tmathur@homer:~$ cd datatemp/
tmathur@homer:~/datatemp$ cd hdb
tmathur@homer:~/datatemp/hdb$ ll
total 16
drwxr-xr-x 3 tmathur users 4096 Mar 30 14:05 ./
drwxr-xr-x 7 tmathur users 4096 Mar 25 14:20 ../
drwxr-xr-x 6 tmathur users 4096 Mar 30 14:03 2026.03.30/
-rw-r--r-- 1 tmathur users   90 Mar 30 09:43 sym
tmathur@homer:~/datatemp/hdb$ rm -rf 2026*
tmathur@homer:~/datatemp/hdb$ ll
total 12
drwxr-xr-x 2 tmathur users 4096 Mar 30 14:17 ./
drwxr-xr-x 7 tmathur users 4096 Mar 25 14:20 ../
-rw-r--r-- 1 tmathur users   90 Mar 30 09:43 sym
tmathur@homer:~/datatemp/hdb$ cd

Changed the configuration to use 2 slave threads

localhost,{KDBBASEPORT}+6,sort,sort1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,1,,,${KDBCODE}/processes/wdb.q,1,-s -2 -parentproctype wdb,

restarted the whole stack

tmathur@homer:~$ bash bin/torq.sh start all
14:17:50 | Starting discovery1...
14:17:50 | Starting stp1...
14:17:50 | Starting rdb1...
14:17:50 | Starting hdb1...
14:17:51 | Starting hdb2...
14:17:51 | Starting wdb1...
14:17:51 | Starting sort1...
14:17:51 | Starting gateway1...
14:17:52 | Starting housekeeping1...
14:17:52 | Starting feed1...
14:17:52 | Starting sctp1...
14:17:52 | Starting sortworker1...
14:17:53 | Starting sortworker2...
14:17:53 | Starting metrics1...

In the wdb, created the table tradetest again

tmathur@homer:~$ qcon :46655:admin:admin
tables[]
`heartbeat`logmsg`packets`quote`trade
n:2000000
tradetest:([] time:n?.z.t;sym:n?`AAPL`GOOG`MSFT`TSLA`AMZN;price:n?100f;size:n?1000i;side:n?`buy`sell)
tradetest
time         sym  price    size side
------------------------------------
11:07:39.380 MSFT 63.75437 958  buy
02:51:55.546 AMZN 63.47421 248  buy
00:12:49.377 TSLA 88.18232 734  buy
09:20:24.108 GOOG 38.67286 253  sell
09:57:01.108 AAPL 10.29056 411  buy
11:24:25.553 TSLA 52.32121 434  sell
00:19:46.001 GOOG 55.74642 680  buy
10:15:01.439 AMZN 63.98269 672  sell
01:53:17.511 TSLA 68.26658 728  buy
10:31:42.704 MSFT 72.80386 662  sell
06:03:28.576 GOOG 82.79578 482  buy
09:07:39.280 TSLA 15.86833 53   sell
04:36:21.456 GOOG 56.55567 796  sell
09:15:43.252 MSFT 97.24472 48   buy
03:56:57.019 GOOG 55.04991 647  buy
05:06:33.285 TSLA 66.2644  552  sell
00:29:05.666 AAPL 57.56012 709  buy
09:59:36.325 AAPL 43.59702 811  sell
..
system "s"
-2i
\t .u.end[.z.d]
72

verified data in the hdb after savedown and it matches.