Right now steps (at least 03.split) that write their output to $TMPDIR. As stated by @jelmervdl, this is probably because LUMI has not $SCRATCH set and files are written to a temporary dir in the node that is owned by the user and not the group. We should have something like this for LUMI that creates the temporary directory and changes ownerships.
Example of giashard doing it well but 03.split not:
$ ll output_wide15_filtered_sample3-shards/es/*/*
output_wide15_filtered_sample3-shards/es/0/1:
total 376M
-rw-r----- 1 zaragoza2 project_462000252 188M May 9 18:04 plain_text.gz
-rw-rw---- 1 zaragoza2 zaragoza2 187M May 10 16:40 sentences.gz
-rw-r----- 1 zaragoza2 project_462000252 12K May 9 18:04 source.gz
-rw-r----- 1 zaragoza2 project_462000252 655K May 9 18:04 url.gz
output_wide15_filtered_sample3-shards/es/1/1:
total 322M
-rw-r----- 1 zaragoza2 project_462000252 162M May 9 18:04 plain_text.gz
-rw-rw---- 1 zaragoza2 zaragoza2 160M May 10 16:39 sentences.gz
-rw-r----- 1 zaragoza2 project_462000252 11K May 9 18:04 source.gz
-rw-r----- 1 zaragoza2 project_462000252 661K May 9 18:04 url.gz
Right now steps (at least 03.split) that write their output to
$TMPDIR. As stated by @jelmervdl, this is probably because LUMI has not$SCRATCHset and files are written to a temporary dir in the node that is owned by the user and not the group. We should have something like this for LUMI that creates the temporary directory and changes ownerships.Example of giashard doing it well but 03.split not: