Add fileNameGenerator to the constructor of IcebergInsertTableHandle#1571
Open
Add fileNameGenerator to the constructor of IcebergInsertTableHandle#1571
Conversation
Zouxxyy
changed the title
Zouxxyy
changed the title
Author
|
CC @PingLiuPing For a look, thanks |
Alchemy-item: (ID = 917) [OAP] Support struct schema evolution matching by name commit 1/1 - c816a39
Alchemy-item: (ID = 883) [OAP] [13620] Allow reading integers into smaller-range types commit 1/1 - 4cae2f5
… outer join Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 882) [OAP] [11771] Fix smj result mismatch issue commit 1/1 - ada7dd2
Alchemy-item: (ID = 954) [OAP] [14722] Fix memory leak caused by asynchronous prefetch commit 1/1 - aedf6b0
This reverts commit fd0682b. Alchemy-item: (ID = 983) Iceberg staging hub commit 2/18 - f7d02da765fd15bcbe20aa8c66c98393adb98b6c
…facebookincubator#15477)" This reverts commit 1895711. Alchemy-item: (ID = 983) Iceberg staging hub commit 4/18 - aee2a67daa46a851e0d439ff58d62d7f71aac670
…incubator#15443)" This reverts commit 51d4a94. Alchemy-item: (ID = 983) Iceberg staging hub commit 5/18 - 9ed400840cdb030e413a9a439870e4547b31eca0
The function toValues removes duplicated values from the vector and return them in a std::vector. It was used to build an InPredicate. It will be needed for building NOT IN filters for Iceberg equality delete read as well, therefore moving it from velox/functions/prestosql/InPred icate.cpp to velox/type/Filter.h. This commit also renames it to deDuplicateValues to make it easier to understand. Alchemy-item: (ID = 983) Iceberg staging hub commit 7/18 - 8ff8f5b5716231384ef10410ecd79b85d0474938
This commit introduces EqualityDeleteFileReader, which is used to read Iceberg splits with equality delete files. The equality delete files are read to construct domain filters or filter functions, which then would be evaluated in the base file readers. When there is only one equality delete field, and when that field is an Iceberg identifier field, i.e. non-floating point primitive types, the values would be converted to a list as a NOT IN domain filter, with the NULL treated separately. This domain filter would then be pushed to the ColumnReaders to filter our unwanted rows before they are read into Velox vectors. When the equality delete column is a nested column, e.g. a sub-column in a struct, or the key in a map, such column may not be in the base file ScanSpec. We need to add/remove these subfields to/from the SchemaWithId and ScanSpec recursively if they were not in the ScanSpec already. A test is also added for such case. If there are more than one equality delete field, or the field is not an Iceberg identifier field, the values would be converted to a typed expression in the conjunct of disconjunts form. This expression would be evaluated as the remaining filter function after the rows are read into the Velox vectors. Note that this only works for Presto now as the "neq" function is not registered by Spark. See https://github.com/ facebookincubator/issues/12667 Note that this commit only supports integral types. VARCHAR and VARBINARY need to be supported in future commits (see facebookincubator#12664). Co-authored-by: Naveen Kumar Mahadevuni <Naveen.Mahadevuni@ibm.com> # Conflicts: # velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp # Conflicts: # velox/dwio/common/ScanSpec.h # Conflicts: # velox/type/Filter.h Alchemy-item: (ID = 983) Iceberg staging hub commit 8/18 - 117b7f9740b46ed7559a49a0bb0de044c831f060
# Conflicts: # velox/connectors/hive/HiveConnectorUtil.cpp Alchemy-item: (ID = 983) Iceberg staging hub commit 9/18 - fd66e1e225804aef6f4e225449ff04170f2249c4
Co-authored-by: Chengcheng Jin <Chengcheng.Jin@ibm.com> Alchemy-item: (ID = 983) Iceberg staging hub commit 10/18 - d501ccc1e895f96b7b80375b343dda5683cf619e
Alchemy-item: (ID = 983) Iceberg staging hub commit 11/18 - 157015bfad057b507fe20f65209c9fdb7f72df19
# Conflicts: # velox/dwio/parquet/writer/Writer.cpp # velox/dwio/parquet/writer/Writer.h # velox/dwio/parquet/writer/arrow/ArrowSchema.cpp # velox/dwio/parquet/writer/arrow/ArrowSchema.h # velox/dwio/parquet/writer/arrow/Metadata.cpp Alchemy-item: (ID = 983) Iceberg staging hub commit 12/18 - 6d3be8c32de448c61469c6c37343f041186ab377
Alchemy-item: (ID = 983) Iceberg staging hub commit 13/18 - 306dd1c2c891b148d91831c0a386b46590a70863
Alchemy-item: (ID = 983) Iceberg staging hub commit 14/18 - 5a76f957ae7177a6797ad06d55593d50a7da2967
Alchemy-item: (ID = 983) Iceberg staging hub commit 15/18 - ea55d14cfad376405f3af95aebb4c9ad8972db88
Alchemy-item: (ID = 983) Iceberg staging hub commit 16/18 - 973a1e4b845cd8dcea1b2186dfa98774b1f28413
Alchemy-item: (ID = 983) Iceberg staging hub commit 17/18 - bfbd027b1bb9e77d0cdcf66c9c011181a8aa3415
Alchemy-item: (ID = 983) Iceberg staging hub commit 18/18 - 836264cb069107fa801099ba2d791c085ed863d1
Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 906) fix: Adding daily tests commit 1/2 - e2eb2c6
we can cache ccache on every build even on failure, since ibm/velox is always incremental build Alchemy-item: (ID = 906) fix: Adding daily tests commit 2/2 - 0899ddc
Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 956) fix: Remove website folder to bypass the security issues commit 1/1 - 42debeb62c2e4f7b44f4a50e4288ce3d82907853
This was referenced Apr 14, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expose fileNameGenerator in IcebergInsertTableHandle so that users can provide a custom fileNameGenerator when creating it—for example, to include partition ID, task ID, or other information in Gluten.