Standardize on parquet, allow fallback to json

[paddymul]

Checks

• I have checked that this enhancement has not already been requested

How would you categorize this request. You can select multiple if not sure

Performance

Enhancement Description

in every place the table accepts json for tabular or summary data, it should also accept parquet. the python buckaroo widgets should default to sending parquet in these places, particularly for summary stats

Pseudo Code Implementation

n/a

Prior Art

n/a

[paddymul]

Implementation Plan

Overview

Accept both Parquet and JSON for tabular and summary data transport, with Parquet preferred and JSON as fallback.

Steps

1. JS: Content-type detection — wherever the JS side receives data:

   • Check if the payload is Parquet (magic bytes PAR1) or JSON (starts with { or [)
   • Route to appropriate decoder (hyparquet for Parquet, JSON.parse for JSON)

2. Python: Prefer Parquet — in serialization:

   • Try Parquet encoding first (requires fastparquet/pyarrow)
   • If import fails (e.g. pyodide, minimal install), fall back to JSON
   • Log which format was used for debugging

3. Summary stats — currently summary stats may only be JSON. Add Parquet support:

   • Serialize summary stats as a small Parquet table
   • Or keep as JSON (summary data is small, Parquet overhead not worth it)

4. Wire format negotiation — for server mode:

   • Client sends Accept: application/vnd.apache.parquet or Accept: application/json
   • Server responds with the preferred available format

5. Tests:

   • Unit test: verify JSON fallback works when fastparquet is not installed
   • Integration: verify both formats produce identical widget renders

Files to modify

• buckaroo/serialization_utils.py — format selection and fallback
• JS data loading code — content-type detection
• buckaroo/server/ — content negotiation

@codex

[chatgpt-codex-connector]

Summary

• Added a parquet-first dataframe transport helper df_to_payload() that returns {'format': 'parquet_b64', 'data': ...} when parquet serialization works, and falls back to JSON row data when parquet is unavailable/fails. It also logs which path was used. buckaroo/serialization_utils.pyL228-L239
• Switched widget/dataflow dataframe serialization paths to use parquet-first transport across pandas, dataflow, polars, and geopandas codepaths (_df_to_obj now uses df_to_payload). buckaroo/buckaroo_widget.pyL153-L155 buckaroo/buckaroo_widget.pyL421-L422 buckaroo/dataflow/dataflow.pyL425-L427 buckaroo/polars_buckaroo.pyL66-L72 buckaroo/geopandas_buckaroo.pyL59-L62
• Added server-side infinite-scroll fallback transport logic: _slice_to_transport() now prefers parquet responses and falls back to JSON inline payloads on parquet ImportError; responses now include a format field. buckaroo/server/data_loading.pyL129-L147 buckaroo/server/data_loading.pyL238-L238 buckaroo/server/data_loading.pyL350-L350
• Updated websocket/client handling so JSON-only infinite responses are emitted immediately (no binary frame required). packages/js/WebSocketModel.tsL32-L40
• Updated infinite-scroll JS decoding to detect Parquet via magic bytes (PAR1) and otherwise parse JSON bytes, plus graceful no-buffer handling. packages/buckaroo-js-core/src/components/BuckarooWidgetInfinite.tsxL24-L35 packages/buckaroo-js-core/src/components/BuckarooWidgetInfinite.tsxL106-L124
• Added unit tests for parquet-preferred and JSON-fallback behavior of df_to_payload(). tests/unit/serialization_utils_test.pyL154-L168

Testing

• ✅ python -m compileall buckaroo/serialization_utils.py buckaroo/buckaroo_widget.py buckaroo/dataflow/dataflow.py buckaroo/polars_buckaroo.py buckaroo/geopandas_buckaroo.py buckaroo/server/data_loading.py
• ⚠️ pytest -q tests/unit/serialization_utils_test.py tests/unit/server/test_server.py (environment missing pandas, so test collection fails)
• ⚠️ npm --prefix packages/js run -s build (workspace package resolution unavailable in this environment: buckaroo-js-core unresolved)

Committed on current branch and created a follow-up PR via make_pr with title:
“Follow-up: parquet-first transport with JSON fallback in widgets/server”.

View task →

[paddymul]

Related: Open PR #549

PR #549 (refactor: switch transport from Parquet to Arrow IPC) replaces the parquet transport with Arrow IPC. Once that merges, the "standardize on parquet" framing becomes moot — Arrow IPC would be the primary format with JSON as fallback.

This issue's scope may shift to: "ensure JSON fallback works everywhere Arrow IPC is used." See also #546 (Arrow IPC evaluation).