Skip to content

feat: native zero-copy exchange for all Zarr memory layouts#31

Merged
kylebarron merged 20 commits into
mainfrom
kyle/dlpack-tensor-spike
Jun 19, 2026
Merged

feat: native zero-copy exchange for all Zarr memory layouts#31
kylebarron merged 20 commits into
mainfrom
kyle/dlpack-tensor-spike

Conversation

@kylebarron

@kylebarron kylebarron commented Jun 19, 2026

Copy link
Copy Markdown
Member

As explored before, such as in #29, we need to think carefully about our data layouts so they can be exchanged to Python with minimal overhead or copying.

This PR takes the approach of designing DecodedArray, a custom holder for ArrayBytes and which implements FromArrayBytes. This means DecodedArray can be used directly as a result type from operations like Array.retrieve_chunk.

The DecodedArray is actually an enum over four concrete types (essentially the ArrayBytes variants):

  • Tensor: primitive type
  • Variable: variable length strings and bytes
  • Masked tensor: Tensor + a validity mask
  • MaskedVariable: Variable + a validity mask

These each become Python types that the user can use isinstance to determine which they have.

Each of these types expose different options for zero copy conversion:

  • Tensor: buffer protocol, numpy, dlpack, arrow
  • Variable: arrow
  • MaskedTensor: numpy masked array, arrow
  • MaskedVariable: arrow

Closes #29

@kylebarron kylebarron changed the title Kyle/dlpack tensor spike feat: DecodedArray type, supporting native zero-copy exchange for all Zarr formats Jun 19, 2026
@kylebarron kylebarron marked this pull request as ready for review June 19, 2026 18:59
@kylebarron kylebarron changed the title feat: DecodedArray type, supporting native zero-copy exchange for all Zarr formats feat: DecodedArray type, supporting native zero-copy exchange for all Zarr memory layouts Jun 19, 2026
@kylebarron kylebarron changed the title feat: DecodedArray type, supporting native zero-copy exchange for all Zarr memory layouts feat: native zero-copy exchange for all Zarr memory layouts Jun 19, 2026
@kylebarron kylebarron enabled auto-merge (squash) June 19, 2026 19:08
@kylebarron kylebarron merged commit 0531307 into main Jun 19, 2026
6 checks passed
@kylebarron kylebarron deleted the kyle/dlpack-tensor-spike branch June 19, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant