Open
Conversation
ColinLeeo
force-pushed
the
write_with_parallel
branch
from
38b7336 to
d0cf8a4
Compare
ColinLeeo
force-pushed
the
write_with_parallel
branch
from
d0cf8a4 to
9207296
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change introduces thread pool-based write parallelization to the TsFile C++ write path.
Threading is controlled at compile time via the ENABLE_THREADS CMake option, with parallel_write_enabled_ and write_thread_count_ added to ConfigValue for runtime control.
The core approach: write_table() first sorts the Tablet by device (via the new Tablet::sort_by_device()) to make same-device rows contiguous, then precomputes page boundaries from page_writer_max_point_num_ and the current time page's existing point count, and finally dispatches the time column and all value columns into the thread pool in parallel — each column writes the same row segments and explicitly seals pages at the precomputed boundaries, preserving time/value page alignment required by the aligned model.
On the testing side, TsFileWriterTableTest has been converted to a gtest parameterized test, running both Serial
(parallel_write_enabled_=false) and Parallel (parallel_write_enabled_=true) suites — all 34 test cases pass under a single build configuration.