Open
Conversation
Contributor
mateusz-uzarowicz
left a comment
There was a problem hiding this comment.
I tested the new solution on PIT. The sample test size was small, but we still get the Allure report very quickly - even faster when using the AWS CLI.
The Tag tests will be the decisive factor, since there are many of them, but it looks promising (1m < 5m 🎉).
I’d also prefer to wait for results from other where report generation currently takes longer than the tests themselves
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previous attempt
We had already implemented s5cmd in PRs (#169, #170), but it was unstable because of too many workers sending requests at once.
It was reverted in #172 and #173.
Current attempt
We are still using fast
s5cmdbut I researched and tweaked the configuration. It should work as fast as before, but more stable than before.Changes compared to the previous attempt:
--numworkers 50previously we were using the default value of 256. It was as fast as 50, but very resource-intensive for the internal network and our proxy for S3.--retry-count 30previously we were using the default value of 10. It's just a safety measure in case we still have some issues with the network.Tests
I checked this branch using tag-tests.
If you want to check it before merging, you need to use branch MID-6955-s3-actions-using-s5cmd-test.
Why is this a separate action?
This is a separate action rather than replacing our existing workflow because previous implementation was unstable.
We need to verify that this current implementation does not have any issues before making it the default.