Reject DecimalType with precision larger than 38#3593
Open
kratos0718 wants to merge 2 commits into
Open
Conversation
The Iceberg spec requires decimal precision to be 38 or less, but DecimalType(39, 0) (and parsing 'decimal(39, 0)') was accepted silently, producing a value not representable in decimal's fixed-byte storage. Validate precision <= 38 in DecimalType.__init__, matching the Java reference which raises for precision > 38. Closes apache#3583
DecimalType(100, 2) exceeds the spec's max precision of 38 and is now rejected by the new validation; use the maximum valid precision instead.
Member
|
There is an existing PR #3584 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Iceberg spec requires decimal precision to be 38 or less ("Scale is fixed, precision must be 38 or less"). Currently
DecimalType(39, 0)— and parsing the type string"decimal(39, 0)"— is accepted silently, producingDecimalType(precision=39, scale=0). Precision above 38 isn't representable in decimal's fixed-byte storage, so silently accepting it can corrupt downstream encoding.The Java reference implementation rejects this:
DecimalType.of(39, 0)raisesIllegalArgumentException: Decimals with precision larger than 38 are not supported: 39.Closes #3583.
Change
Validate
precision <= 38inDecimalType.__init__, raisingValueErrorwith the same message as the Java reference. This covers both direct construction and the string-parse path (_parse_decimal_typeconstructsDecimalType(precision, scale), which routes through__init__).Test
Added
test_decimal_type_rejects_precision_over_38:DecimalType(39, 0)raises, and the boundaryDecimalType(38, 0)remains valid.