[DFT] feat(cudf): GPU Decimal (Part 2 of 3) (#16750)#1433
[DFT] feat(cudf): GPU Decimal (Part 2 of 3) (#16750)#1433prestodb-ci wants to merge 30 commits intooss-baselinefrom
Conversation
|
Test passed for commit 1777c7523, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/394/display/redirect for details |
Signed-off-by: Yuan <yuanzhou@apache.org> Set ccache maximum size to 1G Remove sed command from Gluten workflow Removed a sed command that replaces 'oap-project' with 'IBM' in the get-velox.sh script. Modify get-velox.sh to change 'ibm' to 'ibm-xxx' Update the get-velox.sh script to replace 'ibm' with 'ibm-xxx'. Update sed command to be case-insensitive Update gluten.yml fix iceberg unit test Signed-off-by: Yuan <yuanzhou@apache.org> Update gluten.yml Enable enhanced features in gluten build script Update cache keys for Gluten workflow
|
❌ Test commit 2a50cd131 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/2/display/redirect for details |
|
❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/4/display/redirect for details |
|
❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/5/display/redirect for details |
prestodb-ci
force-pushed
the
staging-1777c7523-pr
branch
from
28dabdb to
3026e31
Compare
|
❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/6/display/redirect for details |
|
❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/7/display/redirect for details |
|
❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/8/display/redirect for details |
FelixYBW
changed the title
|
❌ Test commit 0007f37c6 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2057/9/display/redirect for details |
prestodb-ci
force-pushed
the
staging-1777c7523-pr
branch
from
3026e31 to
72886ca
Compare
set restore key
prestodb-ci
changed the title
prestodb-ci
force-pushed
the
staging-1777c7523-pr
branch
from
72886ca to
4a8bc9b
Compare
|
❌ Test commit a017fca78 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/427/display/redirect for details |
|
❌ Test commit a236cf5b3 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/428/display/redirect for details |
|
❌ Test commit 0f21ff954 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/429/display/redirect for details |
|
❌ Test commit e01ca2187 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/430/display/redirect for details |
|
❌ Test commit 4f39eef54 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/431/display/redirect for details |
|
❌ Test commit eba05296b failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/437/display/redirect for details |
|
❌ Test commit bf9e9fd6d failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/440/display/redirect for details |
|
❌ Test commit 4ff28c6bf failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/449/display/redirect for details |
|
❌ Test commit bd5278006 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/1358/display/redirect for details |
|
❌ Test commit dc9e9f369 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/pipeline-rebase-ibm-velox/1364/display/redirect for details |
|
❌ DFT test for commit dc9e9f369 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2742/5/display/redirect for details |
|
❌ DFT test for commit dc9e9f369 failed, open https://ci.ibm.prestodb.dev/job/presto-performance/job/presto-performance/job/PR-2742/7/display/redirect for details |
…acebookincubator#17221) Summary: When the build side of a hash join has no data (e.g. anti-join with empty result), inputs_ is empty. With a debug build, the logging at `noMoreInput()` accesses `inputs_[0]` without bound checking, which could result in a `SIGSEGV`. This fixes Q16 (TPC-H) which uses NOT IN (anti-join) where the build side can be empty for some partitions. Pull Request resolved: facebookincubator#17221 Reviewed By: kagamiori Differential Revision: D101671299 Pulled By: peterenescu fbshipit-source-id: 0971d85671e67810789d8148ce3891e757faf4f7
…r#16037) Summary: Replaces previous PR facebookincubator#15014 that erroneously also included changes in some `experimental/cudf` files. This experimental feature is targeted at a multi-node setup of Prestissimo when running with the Cudf extension. It provides an exchange mechanism to transfer CudfVectors between GPU device memory without the need to first copy data to host memory. This mechanism is in addition to the HTTP-based exchange used in Prestissimo. The main goal is to exploit fast hardware interconnects that are available between GPUs. The cudf exchange builds on top of the existing experimental Cudf extension. The approach follows the same design as the HTTP based exchange. Its components operate on CudfVectors instead of SerializedPages and include: * CudfOutputQueueManager and CudfQueues instead of the OutputBufferManager and OutputBuffers * CudfExchangeClient and CudfExchangeSource instead of ExchangeClient and ExchangeSource/PrestoExchangeSource * CudfExchange and CudfExchangeQueue instead of Exchange and ExchangeQueue * CudfExchangeServer instead of Prestissimo's HTTP resource The code in this PR does not contain the necessary hooks needed to initialize Cudf exchange. Pull Request resolved: facebookincubator#16037 Reviewed By: pratikpugalia Differential Revision: D101650220 Pulled By: peterenescu fbshipit-source-id: 207b7964d0039f1885055ee4dae302a200480159
Summary: This is Part 2 of 3 of the GPU Decimal implementation. It adds decimal functions and some other tidying-up. More explanation and annotation to follow. Pull Request resolved: facebookincubator#16750 Reviewed By: kgpai Differential Revision: D101837619 Pulled By: kagamiori fbshipit-source-id: 4e7971aba5f3f5668d19bb2ff53571aa28b74d72
Alchemy-item: (ID = 1270) [OAP] Support struct schema evolution matching by name commit 1/1 - 4316ae5
…ter join Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 1227) [OAP] [11771] Fix smj result mismatch issue commit 1/1 - 987fd37
Alchemy-item: (ID = 1309) feat: Enable the hash join to accept a pre-built hash table for joining commit 1/1 - b1a8bb4
Alchemy-item: (ID = 1294) feat: Change SpillPartitionId::kMaxSpillLevel to 7 commit 1/1 - 7280b67
Co-authored-by: Li Zhou <unidevel@hotmail.com> Alchemy-item: (ID = 1153) Iceberg staging hub commit 1/6 - c5a69de3d1021073c13a99e1c7c6d6fcce355178 Support insert data into iceberg table. Add iceberg partition transforms. Co-authored-by: Chengcheng Jin <Chengcheng.Jin@ibm.com> Add NaN statistics to parquet writer. Collect Iceberg data file statistics in dwio. Integrate Iceberg data file statistics and adding unit test. Support write field_id to parquet metadata SchemaElement. Implement iceberg sort order Add clustered Iceberg writer mode. Fix parquet writer ut Add IcebergConnector Fix unittest error Resolve confict Resolve confict Fix test build issue Fix crash Alchemy-item: (ID = 1334) Iceberg core code commit 1/1 - 41bbbf4
Signed-off-by: Yuan <yuanzhou@apache.org> Alchemy-item: (ID = 906) fix: Adding daily tests commit 1/2 - e2eb2c6
we can cache ccache on every build even on failure, since ibm/velox is always incremental build Alchemy-item: (ID = 906) fix: Adding daily tests commit 2/2 - 0899ddc
Alchemy-item: (ID = 1281) Remove website folder commit 1/1 - 35730e5
This commit introduces `PartitionedVector` - a low-level execution abstraction that provides an in-place, partition-aware layout of a vector based on per-row partition IDs. 1. **In-place rearrangement**: Rearrange vector data in memory without creating multiple copies 2. **Buffer reuse**: Allow reuse of temporary buffers across multiple partitioning operations 3. **Minimal abstraction**: Similar to `DecodedVector`, focus on efficient execution rather than operator semantics 4. **Thread-unsafe by design**: Optimized for single-threaded execution contexts For more information please see #1703 Alchemy-item: (ID = 1150) Introducing PartitionedVector commit 1/1 - 960f41b Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 1/11 - 76dc41a
…dthValuesInPlace Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 3/11 - ff2e34b
Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 4/11 - 875c92c
PartitionedFlatVector::partition() and PartitionedRowVector::partition() called mutableRawNulls() unconditionally. mutableRawNulls() allocates a null buffer if one does not exist, causing mayHaveNulls() to return true for every vector after partitioning, even when the original had no nulls. Fix both sites to check rawNulls() first and only call mutableRawNulls() when a null buffer already exists. Add noNullBufferAllocatedForNullFreeFlat and noNullBufferAllocatedForNullFreeRow tests to PartitionedVectorTest to cover this case. # Conflicts: # velox/vector/PartitionedVector.cpp Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 5/11 - 281a365
Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 6/11 - 6519a8f
Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 7/11 - d8f34b4
Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 8/11 - 9eafc9d
This commit introduces PrestoIterativePartitioningSerializer, which buffers RowVectors across multiple append() calls, partitions rows in-place using PartitionedVector, and on flush() serializes each non-empty partition into a Presto wire-format IOBuf. The serializer has no dependency on velox_exec: it returns raw folly::IOBuf objects, leaving SerializedPage creation to the caller. Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 9/11 - 6f09ea9
This commit introduces OptimizedPartitionedOutput, a PartitionedOutput operator backed by PrestoIterativePartitioningSerializer. Enabled via query config key "optimized_repartitioning" (default off). LocalPlanner selects it over the standard PartitionedOutput when the flag is set. TODO: replicateNullsAndAny is not yet supported and raises a user error. Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 10/11 - c114147
…geBenchmark - Added normal vs optimized PartitionedOutput comparison by running each exchange case twice with kOptimizedPartitionedOutputEnabled=false/true. - Added per-mode benchmark names: - exchange<Case>_normalPartitionedOutput - exchange<Case>_optimizedPartitionedOutput in ExchangeBenchmark.cpp. - Refactored result printing into shared helpers and fixed output consistency in ExchangeBenchmark.cpp. Alchemy-item: (ID = 1327) Optimized PartitionedOutput staging hub commit 11/11 - 211901c
Signed-off-by: Hazmi <ialhazmim@gmail.com> Alchemy-item: (ID = 1203) Fix iceberg min max statistics for decimal type when encoded as int32 commit 1/1 - 0ac9930
Alchemy-item: (ID = 1335) Fix GCC 11.2 compilation issue commit 1/1 - b8d656e
Signed-off-by: Linsong Wang <linsong.wang@ibm.com>
15 participants
DFT rebase for branch staging-7ed90a53d-dft-pr with head 7ed90a5