Merge pull request #23 from AquaQAnalytics/schema-documentation-hw

Schema documentation hw

Author: hamza060

README.md

@@ -1,10 +1,10 @@
 # TorQ-TAQ
 
-The TorQ-TAQ Loader architecture is an extension to TorQ, which efficiently loads NYSE TAQ files using the streaming decompress algorithm with .Q.fpn  
+The TorQ-TAQ Loader architecture is an extension to TorQ, which efficiently loads NYSE TAQ files using the streaming decompress algorithm with .Q.fpn.
 
-# Quick Installisation
+# Quick Installation
 
-To download TorQ TAQ, get latest installation script and download it to the directory where you want your codebase to live
+To download TorQ-TAQ, get latest installation script and download it to the directory where you want your codebase to live.
 
 ````
 wget https://raw.githubusercontent.com/AquaQAnalytics/TorQ-TAQ/master/installlatest.sh
@@ -37,15 +37,15 @@ Once this is complete, the necessary TorQ packages will be installed
 To start TorQ TAQ, run the following command within your terminal:
 
 ````
-$ ./bin/torq.sh start all
+$ ./deploy/bin/torq.sh start all
 15:36:23 | Starting discovery1...
 15:36:23 | Starting gateway1...
 15:36:23 | Starting orchestrator1...
 15:36:23 | Starting taqloader1...
 15:36:24 | Starting taqloader2...
 15:36:24 | Starting qmerger1...
 
-$ ./bin/torq.sh summary
+$ ./deploy/bin/torq.sh summary
 TIME      |  PROCESS        |  STATUS  |  PID    |  PORT
 15:36:56  |  discovery1     |  up      |  21434  |  10610
 15:36:57  |  gateway1       |  up      |  21607  |  10611
@@ -66,10 +66,11 @@ Three files are currently supported: trade, quote, and national best bid offer (
 Depending on if the file recognised is a trade, quote or nbbo file, the way the data is saved behaves differently. 
 
 ## Trade/NBBO ##
-Trade/NBBO data is laoded to the temporary HDB - tempdb, located in `deploy/tempdb/final/YYYY.MM.DD/trade|nbbo/`
 
 **_NOTE:_** tempdb and hdb directories are created once a TAQ .gz file is loaded in - the paths to these directories are defined within [default.q](appconfig/settings/default.q)
 
+Trade/NBBO data is laoded to the temporary HDB - tempdb, located in `deploy/tempdb/final/YYYY.MM.DD/trade|nbbo/`
+
 By default, when Trade/NBBO data is loaded to the temporary HDB, it lives in the `deploy/tempdb/final/YYYY.MM.DD/` directory until all data from this day has been loaded. 
 
 ## Quote ##
@@ -83,11 +84,68 @@ When the quote split file has been loaded successfully, it is merged using the m
 When the trade, nbbo and all 26 quote split files have been successfully loaded and merged, the orchestrator then calls the merge process to call the function which moves all of the loaded and merged data to the final hdb in its relevant date partition.
 
 ## Support Functionality ##
+### Saving to HDB ###
 
-We have included a function called `manualmovetohdb` – This function can be called with arguments `[date;filetype]` in the orchestrator to manually move loaded data to the hdb. date is a date atom and filetype is a symbol or list of symbols (any of trade, quote, or nbbo). By default, data is only moved when all files have been successfully loaded and merged. However, this can be called to move the data at a different point in time.
+We have included the `manualmovetohdb` function, which allows you to manually move loaded data to the HDB by calling it with `[date;filetype]` arguments in the orchestrator. The date argument is a date atom and the filetype argument is a symbol or a list of symbols (trade, quote, or nbbo). By default, data is moved only after all files have been successfully loaded and merged. However, this function allows you to move the data at any point in time.
+### Changing Table Schema ###
+TorQ-TAQ is equipped to handle trade, quote, and national best bid offer (nbbo) files from the NYSE website, as previously noted. The functionality to customize the schema of these tables can be found in [taq.q](code/common/taq.q), enabling users to adjust the column names and datatypes to fit their requirements or to use a different format. This process involves modifying the dictionaries defined in maketaqparams. For example, the trade table schema from the NYSE format can be updated to load trade data from alternative sources that may have distinct columns or datatypes.
+
+Taking the original trade data parameters from `maketaqparams`
+````
+    tradeparams:defaults,(!) . flip (
+        (`headers;`ticktime`exch`sym`cond`size`price`stop`corr`sequence`tradeid`cts`trf`parttime);
+        (`types;"JSSSIFBIJICCJ");
+        (`tablename;`trade);
+        (`separator;enlist"|");
+        (`dbdir;hdbdir);             // this parameter is defined in the top level taqloader script
+        (`symdir;symdir);            // where we enumerate against
+        (`tempdb;tempdb);
+        (`dataprocessfunc;{[params;data] `sym`ticktime`exch`cond`size`price`stop`corr`sequence`cts`trf xcols delete from
+        (update sym:.Q.fu[{` sv `$" " vs string x}each;sym],ticktime:params[`date]+ timeconverter[ticktime],parttime:params[`date]+ timeconverter[parttime] from data) where null ticktime});
+        (`date;.z.d)
+    );
+````
+In this example, we would be interested to change the headers to only include the following ```` `ticktime`exch`sym`cond`size`price`parttime ````. 
 
+To achieve this, the following changes were made:
+````
+    tradeparams:defaults,(!) . flip (
+        (`headers;`ticktime`exch`sym`cond`size`price`parttime);
+        (`types;"JSSSIF      J");
+        (`tablename;`trade);
+        (`separator;enlist"|");
+        (`dbdir;hdbdir);             // this parameter is defined in the top level taqloader script
+        (`symdir;symdir);            // where we enumerate against
+        (`tempdb;tempdb);
+        (`dataprocessfunc;{[params;data] `sym`ticktime`exch`cond`size`price xcols delete from
+        (update sym:.Q.fu[{` sv `$" " vs string x}each;sym],ticktime:params[`date]+ timeconverter[ticktime],parttime:params[`date]+ timeconverter[parttime] from data) where null ticktime});
+        (`date;.z.d)
+    );
+````
+Now once trade data is decompressed and loaded, it will have the following schema:
+````
+q)meta trade
+c       | t f a
+--------| -----
+date    | d    
+sym     | s    
+ticktime| p    
+exch    | s    
+cond    | s    
+size    | i    
+price   | f    
+parttime| p 
+````
+To clarify, the files dropped in the filedrop directory are compared against the names listed in the runload function within the orchestrator.
+````
+    filetype: $[
+        file like "*TRADE*";`trade;
+        file like "*SPLITS*";`quote;
+        file like "*NBBO*";`nbbo;
+        [.lg.e[`fifoloader;errmsg:(string file)," is an unknown or unsupported file type"];'errmsg]];
+````
 ## Example ##
-Here we just want to load in the NYSE trade data for date partition 2022.10.03 and manually move it to the HDB
+Here we only want to load in NYSE trade data for date 2022.10.03 and manually move it to the HDB.
 
 1. Begin by downloading EQY_US_ALL_TRADE_20221003.gz from the NYSE website directly into your filedrop directory whilst your stacks are up
 
@@ -163,4 +221,4 @@ tempdb
 ````
 
 
->An overview blog [is here](https://www.aquaq.co.uk/q/torq-taq-a-nyse-taq-loader/), further documentation is in the docs [directory](docs/torqtaqtutorial.md). 
+>An overview blog [is here](https://www.aquaq.co.uk/q/torq-taq-a-nyse-taq-loader/), further documentation is in the [docs](docs/torqtaqtutorial.md) directory.