Evaluate Arrow IPC + flechette as transport format

[paddymul]

Motivation

Buckaroo currently uses Parquet as the binary transport between Python and JS:

• Python: fastparquet / pyarrow → Parquet bytes
• JS: hyparquet (~10 KB) decodes in browser

This works, but fastparquet only operates on pandas DataFrames and uses pandas internal APIs (pandas._libs.tslibs, pandas._libs.json). This is the #1 blocker for making pandas an optional dependency (#533-adjacent).

Switching the transport format to Arrow IPC with flechette as the JS decoder would:

1. Make the serialization layer backend-agnostic (pyarrow works with pa.Table directly — no pandas needed)
2. Drop fastparquet from core dependencies
3. Enable zero-copy typed arrays on the JS side
4. Potentially improve deserialization performance (7-11x faster row extraction vs apache-arrow JS)

Research summary

Full writeup: docs/flechette-arrow-ipc-research.md

flechette (@uwdata/flechette)

Built by the UW Interactive Data Lab (Heer et al — the D3/Vega/Mosaic group). Used by Mosaic, Arquero v7, and vega-loader-arrow in production.

	hyparquet (current)	flechette	apache-arrow JS
Size (min+gz)	10 KB	14 KB	43 KB
Dependencies	0	0	Multiple
Reads Parquet	Yes	No	No
Reads Arrow IPC	No	Yes	Yes
Zero-copy	No	Yes	Yes
Tree-shaking	Works	Works	Broken
Row extraction speed	Baseline	7-11x vs apache-arrow	Baseline

apache-arrow JS is not recommended — 3x the bundle, broken tree-shaking (open JIRA for years), and still can't read Parquet.

What the Python side looks like

pyarrow is already a core dependency. Both pandas and polars produce pyarrow Tables:

import pyarrow as pa
import pyarrow.ipc as ipc

# From pandas
table = pa.Table.from_pandas(df)
# From polars
table = polars_df.to_arrow()

# Serialize to IPC bytes
sink = pa.BufferOutputStream()
writer = ipc.new_stream(sink, table.schema)
writer.write_table(table)
writer.close()
raw_bytes = sink.getvalue().to_pybytes()

This replaces to_parquet() (fastparquet, pandas-only) with a backend-agnostic path.

What the JS side looks like

import { tableFromIPC } from '@uwdata/flechette';

// Infinite scroll (binary buffer from model.send)
const table = tableFromIPC(buffer);
const rows = table.toArray(); // [{col1: val, col2: val}, ...]

// Summary stats (base64 payload)
const table = tableFromIPC(b64ToArrayBuffer(payload.data));
const rows = table.toArray();

Mixed-type columns (summary stats)

Current approach: pre-JSON-encode each cell into a string column, then JSON.parse on JS side. Same strategy works identically with Arrow IPC.

Proposed plan

Phase 1 (minimal, unblocks optional pandas)

Swap Python serialization from fastparquet → pyarrow. Keep hyparquet on JS side. pyarrow-produced Parquet is readable by hyparquet. Zero JS changes.

Phase 2 (performance + clean architecture)

Switch transport from Parquet to Arrow IPC. Replace hyparquet with flechette (+4 KB bundle). Enables zero-copy typed arrays and faster deserialization.

Bundle impact

+4 KB gzipped (Phase 2 only). Phase 1 has zero JS impact.

Files affected

Python (both phases):

• buckaroo/serialization_utils.py — main serialization logic
• buckaroo/buckaroo_widget.py — calls to_parquet() for infinite scroll
• pyproject.toml — move fastparquet to [pandas] extra

JS (Phase 2 only):

• packages/buckaroo-js-core/src/components/DFViewerParts/resolveDFData.ts
• packages/js/widget.tsx — buffer handling in infinite scroll
• packages/buckaroo-js-core/package.json — swap hyparquet → flechette

References

• flechette GitHub
• flechette docs
• hyparquet GitHub
• Arrow IPC spec
• pyarrow IPC docs

[paddymul]

Implementation Plan

Overview

Evaluate replacing Parquet with Arrow IPC as the Python→JS transport format, using flechette as the JS decoder.

Steps

1. Benchmark current Parquet path — measure serialization time and payload size for representative DataFrames (100 rows, 10K rows, 100K rows) with various dtypes. Record: Python encode time, JS decode time, wire size.

2. Prototype Arrow IPC path:

   • Python: pyarrow.ipc.RecordBatchStreamWriter → bytes (works natively with both pandas and polars, no fastparquet dependency)
   • JS: flechette (~6KB gzipped) to decode Arrow IPC bytes
   • Measure same metrics as step 1

3. Compare:

   • Encode/decode speed
   • Wire size (Parquet is compressed, Arrow IPC is not by default — test with LZ4 compression)
   • Library size (hyparquet ~10KB vs flechette ~6KB)
   • Polars compatibility (Arrow IPC is native to polars, Parquet requires pandas conversion)

4. Decision criteria:

   • If Arrow IPC is faster end-to-end AND removes the fastparquet/pandas dependency for polars users → adopt
   • If comparable → keep Parquet (less risk)
   • Document findings in this issue

5. If adopting: implement behind a feature flag first, then make default in a follow-up release.

Files to modify (prototype)

• buckaroo/serialization_utils.py — Arrow IPC serialization path
• packages/buckaroo-js-core/ — flechette decoder integration
• Benchmark script in scripts/

@codex

[chatgpt-codex-connector]

Summary

• Added a prototype Arrow IPC serialization path in Python (to_arrow_ipc, to_arrow_ipc_b64) plus summary-payload support (sd_to_arrow_ipc_b64) so we can benchmark and A/B transport formats without changing defaults yet. buckaroo/serialization_utils.pyL230-L251 buckaroo/serialization_utils.pyL305-L328
• Extended JS payload typing and resolver logic to accept both parquet_b64 and arrow_ipc_b64, decoding Arrow IPC with @uwdata/flechette while preserving existing Parquet handling and cache behavior. packages/buckaroo-js-core/src/components/DFViewerParts/DFWhole.tsL205-L217 packages/buckaroo-js-core/src/components/DFViewerParts/resolveDFData.tsL1-L3 packages/buckaroo-js-core/src/components/DFViewerParts/resolveDFData.tsL19-L27 packages/buckaroo-js-core/src/components/DFViewerParts/resolveDFData.tsL131-L144 packages/buckaroo-js-core/src/components/DFViewerParts/resolveDFData.tsL190-L205 packages/buckaroo-js-core/src/components/DFViewerParts/resolveDFData.tsL225-L236
• Added benchmark tooling:
  • Python benchmark script for encode time + payload size and payload artifact generation across row scales and compression variants. scripts/benchmark_transport_formats.pyL1-L6 scripts/benchmark_transport_formats.pyL56-L80 scripts/benchmark_transport_formats.pyL82-L94
  • JS decode benchmark script to compare hyparquet vs flechette decode time on generated payloads. packages/buckaroo-js-core/scripts/benchmark-decode.mjsL1-L6 packages/buckaroo-js-core/scripts/benchmark-decode.mjsL13-L31 packages/buckaroo-js-core/scripts/benchmark-decode.mjsL46-L70
• Documented the end-to-end benchmark workflow in docs and added @uwdata/flechette as a core JS dependency for the prototype path. docs/flechette-arrow-ipc-benchmarking.mdL1-L35 packages/buckaroo-js-core/package.jsonL32-L42

Testing

• ✅ python -m compileall buckaroo/serialization_utils.py scripts/benchmark_transport_formats.py
• ❌ pnpm -C packages/buckaroo-js-core run build:tsc (fails because @uwdata/flechette is not installed in this environment yet)
• ⚠️ pnpm -C packages/buckaroo-js-core install (blocked by registry/network auth limitation: ERR_PNPM_FETCH_403)

Also completed requested workflow items:

• Committed changes on current branch (a522b82).
• Created a follow-up PR via make_pr titled “Prototype Arrow IPC transport decoding + benchmark harness.”

View task →

[paddymul]

Related: Open PR #549

PR #549 (refactor: switch transport from Parquet to Arrow IPC) implements this on a branch:

• Replaces to_parquet() with to_arrow_ipc() using pyarrow.ipc
• Replaces hyparquet with @uwdata/flechette for JS parsing
• Updates 22 files (buckaroo_widget, lazy_infinite, polars_buckaroo, serialization_utils, resolveDFData.ts, etc.)
• Drops fastparquet dependency

This PR needs review/merge to close this issue. Also directly impacts #479 (parquet standardization) and #502 (pyodide CI — fastparquet was the blocker).