draft: Thinking about zero-copy data exchange with Python (& Arrow)

[kylebarron]

Background:

I was exploring #26, exploring variable-length string types. But that's annoying because there's no zero-copy sharing between the Zarr string layout and the Numpy v2 string layout. So that would require a full memory copy, and, until PyO3/rust-numpy#505 is implemented, there's no save way to convert to/from numpy v2 in (safe, high level) Rust Numpy API.

However, the variable length string/bytes types in Zarr are exactly the same binary layout as the Arrow StringArray/BinaryArray layout. This got me thinking about adding Arrow as another serialization option. (It would expose contiguous Zarr arrays as a 1D array, plus an attribute on Data describing the shape)

So the general idea here is: the Data object should be fully generic. Faithful to Zarr, not preferring any one serialization format. It should support:

• copying data into numpy buffers
  • Exposing raw buffers (i.e. the ArrayBytes type) for the Python consumer to copy into whatever format they'd like
• buffer protocol exchange (for types supported by the buffer protocol)
• Arrow exchange (for types supported by Arrow)
• DLPack exchange (for types supported by DLPack)