Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Dec 12, 2025

Motivation

Schema.validate() silently accepts Decimal columns with mismatched precision or scale, allowing data with wrong decimal format to pass validation.

Changes

  • Modified Decimal.validate_dtype() to validate both precision and scale, not just the decimal type
    • Scale must always match the schema definition
    • Precision must match when explicitly specified (accepts any precision when precision=None)
  • Added parametrized tests for precision/scale validation and casting behavior
  • Updated existing test expectations to reflect corrected validation behavior

Before:

class T(dy.Schema):
    value = dy.Decimal(precision=38, scale=10)

# Silently accepts scale=3
df = T.validate(
    pl.DataFrame({"value": [Decimal("38.1")]})
    .with_columns(pl.col("value").cast(pl.Decimal(38, 3)))
)
# Result: Decimal[38,3] ❌

After:

# Without cast: raises SchemaError for mismatched scale
df = T.validate(...)  # SchemaError: got Decimal[38,3], expected Decimal[38,10]

# With cast: properly converts to schema's precision/scale
df = T.validate(..., cast=True)  # Result: Decimal[38,10] ✅
Original prompt

This section details on the original issue you should resolve

<issue_title>Decimals validation / cast ignores scale</issue_title>
<issue_description>Schema.validate() silently ignores the Decimal scale declared in the schema:

import polars as pl
import dataframely as dy
from decimal import Decimal

class T(dy.Schema):
    value = dy.Decimal(precision=38, scale=10)

df = T.validate(
    pl.DataFrame(
        {
            T.value.name: [Decimal("38.1")],
        }
    )
    .with_columns(
        T.value.col.cast(pl.Decimal(precision=38, scale=3))
    )
)
print(df)

This results in:

shape: (1, 1)
┌───────────────┐
│ value         │
│ ---           │
│ decimal[38,3] │
╞═══════════════╡
│ 38.100        │
└───────────────┘

Casting also ignores the scale defined in the schema, as the resulting column keeps the inputs scale:

class T(dy.Schema):
    value = dy.Decimal(precision=38, scale=10)

df = T.validate(
    pl.DataFrame(
        {
            T.value.name: [Decimal("38.1")],
        }
    ),
    cast=True
)
print(df)

Same result:

shape: (1, 1)
┌───────────────┐
│ value         │
│ ---           │
│ decimal[38,1] │
╞═══════════════╡
│ 38.1          │
└───────────────┘

Running with dataframely version 2.1.0</issue_description>

<agent_instructions>Fix this issue by adapting the dtype check for the Decimal column: it should pay attention to the precision and scale.</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 3 commits December 12, 2025 15:27
Co-authored-by: borchero <22455425+borchero@users.noreply.github.com>
Co-authored-by: borchero <22455425+borchero@users.noreply.github.com>
Co-authored-by: borchero <22455425+borchero@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix decimals validation to respect declared scale Fix Decimal column validation to enforce precision and scale Dec 12, 2025
Copilot AI requested a review from borchero December 12, 2025 15:36
@borchero borchero changed the title Fix Decimal column validation to enforce precision and scale fix: Properly check for decimal precision and scale Dec 12, 2025
@github-actions github-actions bot added the fix label Dec 12, 2025
@codecov
Copy link

codecov bot commented Dec 12, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (8cc54d9) to head (6dbfd08).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #241   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           53        53           
  Lines         3066      3066           
=========================================
  Hits          3066      3066           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@borchero borchero marked this pull request as ready for review December 12, 2025 17:37
@borchero borchero merged commit b813ac9 into main Dec 12, 2025
31 checks passed
@borchero borchero deleted the copilot/fix-decimal-validation-scale branch December 12, 2025 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decimals validation / cast ignores scale

3 participants