[FLINK-39499][tests] Replace MinIO with SeaweedFS#28369
Open
Efrat19 wants to merge 6 commits into
Open
Conversation
…minio replacement Following the recent deprecation of minIO
…ink-filesystems Following the recent deprecation of minIO
Collaborator
Contributor
Author
|
@flinkbot run azure |
4 similar comments
Contributor
Author
|
@flinkbot run azure |
Contributor
Author
|
@flinkbot run azure |
Contributor
Author
|
@flinkbot run azure |
Contributor
Author
|
@flinkbot run azure |
Following the recent deprecation of minIO
Unlike MiniIO, seaweedFS isn't backed by filesystem, so we need to create the s3 manually, remove the mount, and read the number of lines using regular s3 apis (i.e not overwrites get_total_number_of_valid_lines implementation for local runs)
Contributor
Author
|
@flinkbot run azure |
Contributor
Author
|
@flinkbot run azure |
Unlike MinIO, SeaweedFS supports the s3 API so s3-reading tests can now be enabled, with a note that while hadoop_seaweedfs test is bidirectional, presto_seaweedfs_read only reads from s3, as RecoverableWriter isn't supported by presto, and local filesystem reading isn't supported by seaweedFS (only S3 API)
Efrat19
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
The minio repo was archived on 25.4.26. This PR aims to refactor our tests to use SeaweedFS instead of MinIO to address the recent MinIO deprecation and its lack of S3-read API support.
Brief change log
presto_seaweedfs_readandhadoop_seaweedfsVerifying this change
This change is covered by refactoring existing tests, such as:
org.apache.flink.fs.s3.common.SeaweedFsTestContainerTestorg.apache.flink.fs.s3.common.HAApplicationRunOnSeaweedFsS3StoreITCaseorg.apache.flink.fs.s3.common.HAJobRunOnSeaweedFsS3StoreITCaseorg.apache.flink.fs.s3.common.S5CmdOnSeaweedFsITCasepresto_seaweedfs_readhadoop_seaweedfsDoes this pull request potentially affect one of the following parts:
SeaweedFSand removesMiniO@Public(Evolving): noDocumentation
flink-filesystems/flink-s3-fs-nativedocs were updated to reflect the change from MinIO to SeaweedFS.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 4.8