Console callbacks for TicTac AAE CLI commands

[hmmr]

This PR implements "Tictac runtime management and reporting" section of the TicTac AAE and NextGenRepl CLI commands proposal.

The riak_kv_console:titacaae_cmd/1 functions will be exposed as the following riak admin commands:

Tictac AAE runtime management and reporting

  riak admin tictacaae rebuild_schedule [-n NODE] [-p PARTITION] [RW RD]
  riak admin tictacaae storeheads [-n NODE] [-p PARTITION] [VALUE]
  riak admin tictacaae tokenbucket [-n NODE] [-p PARTITION] [VALUE]
  riak admin tictacaae rebuildtick|exchangetick|maxresults|rangeboost [-n NODE] [VAL]
  riak admin tictacaae rebuild-soon [-n NODE] [-p PARTITION] DELAY
  riak admin tictacaae rebuild-now [-n NODE] [-p PARTITION] DELAY
  riak admin tictacaae rebuildtreeworkers|rebuildstoreworkers|aaefoldworkers [-n NODE] POOLSIZE
  riak admin tictacaae treestatus [--format table|json] [--show STATES]

Notes:

• Commands rebuildtick, exchangetick, maxresults, rangeboost are simple set/show commands, reading and setting the corresponding app env vars in the riak_kv app, on local node by default or on other node if specified with option -n.

• rebuild-soon will schedule a tree rebuild, and rebuild-now will additionally send a poke to controller(s) associated with partitions or nodes as specified with options -n and -p.

• Commands storeheads, tokenbucket, rebuild_schedule will extract or inject the actual relevant values from/to the state of the running AAE controller processes.

• Commands rebuildtreeworkers, rebuildstoreworkers, aaefoldworkers will read/set the size of worker pools af1, af4 and be, correspondingly.

• The treestatus command will collect various items from running AAE contollers and produce a report like this:

$ ./dev/dev1/riak/bin/riak admin tictacaae treestatus --show=all
                                        Partition ID      Status      Last Rebuild Date     Next Rebuild Date   Controller PID  Key Store Status
----------------------------------------------------  ----------  ---------------------  --------------------  ---------------  ----------------
   1004782375664995756265033322492444576013453623296     unbuilt                  never   2025-03-21T19:14:21       <0.2780.0>            native
                                                   0     unbuilt                  never   2025-03-23T04:59:09       <0.2296.0>            native
   1073290264914881830555831049026020342559825461248     unbuilt                  never   2025-03-16T14:05:43       <0.2763.0>            native
   ...

AAE fold operations

  riak tictacaae fold list-buckets NVAL
  riak tictacaae fold find-keys BUCKET KEY_RANGE MODIFIED_RANGE sibling_count=COUNT|object_size=BYTES
  riak tictacaae fold find-keys BUCKET KEY_RANGE MODIFIED_RANGE sibling_count=COUNT|object_size=BYTES
  riak tictacaae fold find|count-tombstones KEY_RANGE SEGMENTS MODIFIED_RANGE
  riak tictacaae fold reap-tombstones KEY_RANGE SEGMENTS MODIFIED_RANGE CHANGE_METHOD
  riak tictacaae fold object-stats BUCKET KEY_RANGE MODIFIED_RANGE
  riak tictacaae fold erase-keys BUCKET KEY_RANGE SEGMENTS MODIFIED_RANGE CHANGE_METHOD
  riak tictacaae fold repair-keys BUCKET KEY_RANGE MODIFIED_RANGE

These will call the corresponding aae_fold operation and dump results in JSON format in a file named "aaefold-%o-results-%t.json", where "%o" will be substituted with the operation being performed, and "%t", with the current datetime string, or to a file explicitly specified with option -o.

Depends on these PRs: OpenRiak/kv_index_tictactree#4, OpenRiak/riak_core#20 and OpenRiak/poolboy#1.

Required by OpenRiak/riak#13.

Integration tests in https://github.com/OpenRiak/riak_test/pull/38.

[hmmr]

After converting it to use clique, this PR is great again and ready for review.

Note that OpenRiak/poolboy#1 and OpenRiak/kv_index_tictactree#4 as well as OpenRiak/riak_core#20 need to be dealt with first.

[hmmr]

Marking it WIP pending review of OpenRiak/poolboy#1 and OpenRiak/riak_core#20.

[hmmr]

This PR in clique is to address this dialyzer issue https://github.com/OpenRiak/riak_kv/pull/31/checks.

[martinsumner]

@hmmr when using the fold from the command line, and not specifying a file, in which directory is the file stored?

The code uses file:get_cwd/0. However, as this code is abstracted from the actual running of the command, is this the user's CWD, or the CWD of the script?

[martinsumner]

#93

[hmmr]

@hmmr when using the fold from the command line, and not specifying a file, in which directory is the file stored?

Results will be written into the CWD of the riak VM, as user riak. It's probably not ideal from the point of view of the user expecting it to be saved in their own CWD, which is why the command prints the full path of the results file:

hmmr@bmbp:~/X/titokyo/g/riak $ ./dev/dev1/riak/bin/riak admin tictacaae fold list-buckets nval=3
Results will be written to /home/hmmr/X/titokyo/g/riak-3.4.0/dev/dev1/riak/aaefold-list-buckets-results-2025-12-05T15:23:44.json
ok

I was thinking of passing the user CWD to the callback code, for it to save the results into it, but since the VM will be writing that file as riak:riak, it will likely fail to create it in a user dir which will almost certainly be mode 755 and not owned by riak user. Hence the message showing the full path, and an -o option for the user to instruct riak to write that file into, say, /tmp (which is usually 777).

[martinsumner]

For the documentation it might be best then that a note or warning is added, suggesting that the -o option should be used, will the full path of the output file specified. I will make a change.