BigInt values from Parquet lose precision in display pipeline

[paddymul]

Summary

Parquet files can contain INT64/UINT64 values that exceed JavaScript's Number.MAX_SAFE_INTEGER (2^53 - 1 = 9007199254740991). These values survive Parquet decoding but get mangled before reaching AG Grid, resulting in potential precision loss or incorrect sorting/filtering.

AG Grid 35.1 now has native BigInt cell data type support, which we could leverage to handle this correctly end-to-end.

Current Pipeline Behavior

The data flows through these stages:

1. Parquet file — INT64/UINT64 types are preserved in the schema
2. Python read (read_utils.py, data_loading.py) — Polars reads as pl.Int64/pl.UInt64, pandas as Int64/UInt64 nullable types. No precision loss here.
3. Serialization (serialization_utils.py) — DataFrame serialized to Parquet bytes via fastparquet/PyArrow. INT64 type preserved in the Parquet binary.
4. Frontend decode (resolveDFData.ts) — hyparquet v1.8.2 decodes INT64 to BigInt64Array and UINT64 to BigUint64Array. Values are still correct at this point.
5. JSON conversion — toJsonSafe() converts BigInt values to strings via .toString(). This preserves the digits but changes the type.
6. AG Grid display — The IntegerDisplayerA displayer receives a string, not a number or BigInt. max_digits defaults to 12, which is insufficient for large int64 values (up to 19 digits).

Where precision breaks

• Values > 2^53 that get coerced to JS Number at any point silently lose precision (e.g. 9007199254740993 becomes 9007199254740992)
• The string conversion in toJsonSafe() preserves digits but means AG Grid treats the column as text — sorting is lexicographic, not numeric
• The max_digits: 12 cap in _dtype_to_displayer() (data_loading.py:244-254) may truncate display formatting for values with more than 12 digits

Relevant Code Paths

Python (type detection → serialization)

• buckaroo/server/data_loading.py:244-254 — _dtype_to_displayer() maps integer dtypes to "integer" displayer with max_digits: 12
• buckaroo/serialization_utils.py:176-215 — to_parquet() serialization
• buckaroo/pluggable_analysis_framework/polars_utils.py:54-57 — NUMERIC_POLARS_DTYPES includes pl.Int64, pl.UInt64

Frontend (decode → display)

• packages/buckaroo-js-core/src/components/DFViewerParts/resolveDFData.ts:52-69 — parseParquetRow() JSON-parses cells after Parquet decode
• packages/buckaroo-js-core/src/components/DFViewerParts/DFWhole.ts:29-33 — IntegerDisplayerA type definition
• packages/buckaroo-js-core/src/components/DFViewerParts/gridUtils.ts:44-61 — getCellRendererorFormatter() applies display formatting

Proposed Fix

1. Preserve BigInt through the decode pipeline

In resolveDFData.ts, avoid converting BigInt values to strings. Either pass them through as native BigInts or convert to Number only when safe (value <= Number.MAX_SAFE_INTEGER).

2. Use AG Grid 35.1 native BigInt support

For columns with INT64/UINT64 data containing values > 2^53, set cellDataType: 'bigint' on the ColDef. This gives us correct sorting, filtering, and editing for free.

3. Update the integer displayer

• Increase max_digits for int64 columns (up to 19 digits for signed, 20 for unsigned)
• Or add a dedicated "bigint" displayer type that handles large values

4. Update Python type mapping

In _dtype_to_displayer(), detect Int64/UInt64 columns and emit a displayer config that signals the frontend to use BigInt handling.

Integration Test Plan

A test should cover the full round-trip:

# Python side
import polars as pl

df = pl.DataFrame({
    "small_int": [1, 2, 3],
    "big_int": [9007199254740993, 9007199254740994, 9007199254740995],  # > 2^53
    "max_uint64": [0, 2**63 - 1, 2**64 - 1],  # unsigned edge cases
})
# Write to parquet, serialize through Buckaroo pipeline
# Assert values survive serialization

// Frontend side
// Decode parquet bytes with hyparquet
// Assert BigInt values are not truncated
// Assert typeof value is 'bigint', not 'number'
// Assert AG Grid column receives correct cellDataType

Edge cases to test

• Values exactly at Number.MAX_SAFE_INTEGER (2^53 - 1) — should work as normal numbers
• Values at Number.MAX_SAFE_INTEGER + 1 — must not silently round
• Int64 min/max: -9223372036854775808 to 9223372036854775807
• UInt64 max: 18446744073709551615
• Null/missing values in BigInt columns
• Mixed columns where some values fit in Number and some don't

Priority

Low — this only affects datasets with integer values > 2^53, which is uncommon in typical data analysis. But it's a correctness issue that could cause silent data corruption for affected users (e.g. database primary keys, blockchain data, large ID systems).

[paddymul]

Implementation Plan

Overview

Prevent precision loss for int64 values exceeding JavaScript's Number.MAX_SAFE_INTEGER (2^53 - 1).

Steps

1. Python: Detect large integers in serialization_utils.py:

   • During DataFrame serialization, check if any int64/uint64 column has values > 2^53
   • For those columns, convert values to strings before JSON serialization
   • Add a column metadata flag "is_bigint": true so the JS side knows these are numeric strings

2. JS: Format BigInt columns in the AG Grid cell renderer:

   • If column metadata has is_bigint: true, display the string value as-is (no numeric formatting)
   • Optionally use BigInt for sorting/filtering if AG Grid 35+ supports it
   • Ensure copy-to-clipboard preserves the full value

3. Alternative approach — if using Parquet/Arrow transport:

   • Arrow IPC natively supports int64; flechette can return BigInt values
   • This may be a stronger argument for Evaluate Arrow IPC + flechette as transport format #546 (Arrow IPC transport)

4. Tests:

   • Unit test: serialize a DataFrame with values > 2^53, verify they survive round-trip without precision loss
   • Playwright: render a table with large integers, verify displayed values match input

Files to modify

• buckaroo/serialization_utils.py — bigint detection and string conversion
• JS cell renderer / value getter — bigint-aware display
• Column metadata schema

@codex

[chatgpt-codex-connector]

Summary

• Added backend bigint detection for integer columns in viewer config generation. Columns with values outside JS safe integer range are now marked with is_bigint: true, and max_digits is widened to 19 (signed) / 20 (unsigned). buckaroo/server/data_loading.pyL257-L311
• Extended frontend integer displayer typing to support bigint metadata (is_bigint) and bigint row values in typed data rows. packages/buckaroo-js-core/src/components/DFViewerParts/DFWhole.tsL29-L34 packages/buckaroo-js-core/src/components/DFViewerParts/DFWhole.tsL198-L201
• Updated integer formatting so bigint-tagged string values are rendered as-is (no JS numeric coercion), preventing precision loss in display. packages/buckaroo-js-core/src/components/DFViewerParts/Displayer.tsL114-L132
• Added bigint-aware AG Grid comparator logic for integer columns tagged is_bigint, so sorting stays numeric even when values are transported as strings. packages/buckaroo-js-core/src/components/DFViewerParts/gridUtils.tsL44-L62
• Added/updated tests for bigint behavior:
  • Frontend test coverage for bigint formatting and sorting comparator behavior. packages/buckaroo-js-core/src/components/DFViewerParts/gridUtils.test.tsL450-L476
  • New Python unit tests for backend bigint flagging and null-handling behavior in config generation. tests/unit/server/test_data_loading_bigint.pyL1-L39

Testing

• ✅ cd packages/buckaroo-js-core && npm test -- --runTestsByPath src/components/DFViewerParts/gridUtils.test.ts --watch=false
• ⚠ pytest -q tests/unit/server/test_data_loading_bigint.py (fails in this environment because pandas is not installed)
• ⚠️ mcp__browser_tools__run_playwright_script attempt against http://localhost:4173 for screenshot capture (no app was serving on that port; ERR_EMPTY_RESPONSE)

I also committed the changes on the current branch and created a follow-up PR entry via the make_pr tool.

View task →

[paddymul]

Related: Open PR #549

PR #549 (refactor: switch transport from Parquet to Arrow IPC) would address this issue at the transport level — Arrow IPC natively supports int64, and flechette can return BigInt values in JavaScript.

If Arrow IPC is adopted, this issue may be resolved without special handling in the serialization layer. If Parquet transport is kept, the string-conversion approach in the plan above is needed.