Skip to content

MID-6955: S3 actions using s5cmd#175

Open
dkaruga-piwikpro wants to merge 3 commits intomasterfrom
MID-6955-s3-actions-using-s5cmd
Open

MID-6955: S3 actions using s5cmd#175
dkaruga-piwikpro wants to merge 3 commits intomasterfrom
MID-6955-s3-actions-using-s5cmd

Conversation

@dkaruga-piwikpro
Copy link
Contributor

Previous attempt

We had already implemented s5cmd in PRs (#169, #170), but it was unstable because of too many workers sending requests at once.
It was reverted in #172 and #173.

Current attempt

We are still using fast s5cmd but I researched and tweaked the configuration. It should work as fast as before, but more stable than before.
Changes compared to the previous attempt:

  • --numworkers 50 previously we were using the default value of 256. It was as fast as 50, but very resource-intensive for the internal network and our proxy for S3.
  • --retry-count 30 previously we were using the default value of 10. It's just a safety measure in case we still have some issues with the network.

Tests

I checked this branch using tag-tests.
If you want to check it before merging, you need to use branch MID-6955-s3-actions-using-s5cmd-test.

Why is this a separate action?

This is a separate action rather than replacing our existing workflow because previous implementation was unstable.
We need to verify that this current implementation does not have any issues before making it the default.

@dkaruga-piwikpro dkaruga-piwikpro requested a review from a team March 3, 2026 07:06
@dkaruga-piwikpro dkaruga-piwikpro requested a review from a team as a code owner March 3, 2026 07:06
Copy link
Contributor

@mateusz-uzarowicz mateusz-uzarowicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested the new solution on PIT. The sample test size was small, but we still get the Allure report very quickly - even faster when using the AWS CLI.
The Tag tests will be the decisive factor, since there are many of them, but it looks promising (1m < 5m 🎉).
I’d also prefer to wait for results from other where report generation currently takes longer than the tests themselves

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants