-
Notifications
You must be signed in to change notification settings - Fork 23
[TASK-191] Arrow serialization for decimal and temporal types #196
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TASK-191] Arrow serialization for decimal and temporal types #196
Conversation
|
@luoyuxia @leekeiabstraction PTAL 🙏 |
edf324a to
f7efa2f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Enables Apache Arrow serialization for Fluss DECIMAL and temporal types (DATE, TIME, TIMESTAMP, TIMESTAMP_LTZ) to match feature parity with the Java client.
Changes:
- Added Arrow schema/type mappings for decimal and temporal Fluss types and made schema conversion fallible (
Result). - Implemented Arrow appends for decimal/date/time/timestamps, including decimal rescaling/validation and timestamp unit handling.
- Updated write/batch/scanner codepaths and tests to propagate the new fallible APIs.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
crates/fluss/src/row/datum.rs |
Adds datum accessors/conversions and Arrow append support for decimal + temporal types. |
crates/fluss/src/record/arrow.rs |
Extends Arrow schema/type mapping and RecordBatch builder creation for decimal/temporal Arrow types; propagates errors instead of panicking. |
crates/fluss/src/client/write/batch.rs |
Makes Arrow write batch construction fallible to reflect schema/builder validation. |
crates/fluss/src/client/write/accumulator.rs |
Propagates new Result return from Arrow write batch creation. |
crates/fluss/src/client/table/scanner.rs |
Propagates fallible Arrow schema conversion when initializing scanners/read contexts. |
crates/fluss/src/client/table/log_fetch_buffer.rs |
Updates tests/helpers to handle fallible Arrow schema conversion and builder creation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
f7efa2f to
2f12672
Compare
2f12672 to
3bc9d15
Compare
leekeiabstraction
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TY for the PR! Left some comments
leekeiabstraction
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added further question
luoyuxia
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fresh-borzoni Thanks for the pr. LGTM overall, only lefr minor comments.
|
@leekeiabstraction @luoyuxia |
luoyuxia
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fresh-borzoni Thanks. LGTM!
closes #191
Enables Arrow serialization for decimal and temporal types, providing feature parity with the Java client.
DECIMAL(p, s)Decimal128(p, s)DATEDate32TIMETime32/Time64TIMESTAMPTimestamp(all precisions)TIMESTAMP_LTZTimestamp(all precisions)