Do old chunks need to be kept around?

[volker-fr]

I am trying to understand the need of keeping old chunks around. The use case for them seem to be limited and mostly around debugging or auditing (e.g. message got deleted?)

For auditing, having changes here handled by Slackdump itself to log changes would be more beneficial over having chunks of every old run stored indefinitely.

I understand the need to keep the latest successful completed information around. I am not sure if this even needs to be in the chunk format, too.

I am curious to learn more about the idea behind keeping all chunks around or what I am overseeing.

[rusq]

Hey @volker-fr , trying to understand what you mean by "old chunks" - is it from the consequent runs of slackdump archive -legacy?

[volker-fr]

@rusq if I run slackdump archive I get messageA in chunk id1

If I run slackdump resume I get the same messageA in chunk id2.

The MESSAGE table also gets on each resume duplicate entries for messageA. If we run slackdump resume daily, we end up with a lot of chunks and duplicate entries.

This is with latest Slackdump 4.1.0.

# after slackdump archive in a free slack (90 day history)
% ./slackdump-4.1.0 archive -o test-slackdump <CHANNEL_ID>
% sqlite3 test-slackdump/slackdump.sqlite 'select count(*) from CHUNK; select count(*) from MESSAGE;'
40
292

# Single resume
% ./slackdump-4.1.0 resume -lookback 100d test-slackdump
% sqlite3 test-slackdump/slackdump.sqlite 'select count(*) from CHUNK; select count(*) from MESSAGE;'
80
584

You can see how messages double

The message count & the chunk size grew. At the same time there has been no new message and both runs have been less as 1min apart). This is a single slack channel with only 90 days of history.

Therefore it is not clear if we need to keep all the chunks (and duplicate messages) around. The only difference between those duplicates is the CHUNK_ID, message id (incremental number) and LOAD_DTTM.

It is for sure faster to just append data, but with large slacks & recurring resumes the amount grows significantly on each run. In my case I run it daily for the last 3 months & weekly for the entire history.

[rusq]

That is correct, you can read the motivation for the lookback parameter in #584 , it's a solution to get the thread messages which were posted in historical threads up to lookback duration.

Ideally, as I mentioned in the discussion #633, resume should be smarter not to fetch or filter out the items which are unchanged.