Tracker: arrow-layers module

[ibgreen]

v9.3+

• [Feat] Update Arrow layers to support both RecordBatch and Table input #145
• tracker: Arrow Schema support loaders.gl#1443

v9.1

• Fork @ibgreen
• Docs
  • Integrate into website
  • Clean up / extend
• Fix examples
• Modernize examples
• Integrate examples into website

[charlieforward9]

To move this tracker along during 9.3...

What is the difference between this module and https://github.com/geoarrow/deck.gl-geoarrow, in writing.

@ibgreen mentioned different formats of passing the data in, but since this hasn't been very maintained, I'm curious if we should sunset it and include the geoarrow official layer like we did all the other external community layers.

Itd be nice to clear this off the backlog soon

@kylebarron

[xinaesthete]

To move this tracker along during 9.3...

What is the difference between this module and https://github.com/geoarrow/deck.gl-geoarrow, in writing.

@ibgreen mentioned different formats of passing the data in, but since this hasn't been very maintained

FWIW, I did get some very quick feedback from the author yesterday when I started trying to use this, and they immediately made a pass on some of the needed maintenance.

I would be very interested in a pointer to a good summary on this... actually, now may be a good time for me to jump onto Slack. I'm in the process of trying to implement proper support for parquet points and shapes in https://github.com/Taylor-CCB-Group/SpatialData.js

[ibgreen]

• On a high level, the long-term goal has always been to have the deck.gl ecosystem integrate Apache Arrow as a first class citizen, rather than offer a duplicated catalog of arrow layers.
• This means that deck layers/loaders.gl loaders etc will be Arrow native, luma will offer Arrow compatible GPGPU support etc.
• The arrow-layers module here in community has only been intended as a staging area to get some work done independently but this will eventually be integrated into deck etc.
• This arrow-ification effort got side tracked for a long time, but is now in full swing.
• I am mid-stream in the first step, which is re-building loaders.gl 5.0 to be arrow-first.
• @Pessimistress is working on GPGPU support in luma.gl
• etc

We would love to have speaking partners to help guide the new architecture into something that feels right for a broad group of users.

There is a channel #openvis-wg-binary-data on the OpenJS slack that would be a good place to get a discussion going.

[charlieforward9]

Thank you for the status update!

And as for the https://github.com/geoarrow/deck.gl-geoarrow, I'm understanding that the processes within these layers are what you are targeting as first class? Making this repo an "archive"? Or would implementations significantly differ ?

[kylebarron]

@geoarrow/deck.gl-geoarrow (in the process of renaming from @geoarrow/deck.gl-layers) is a module that duplicates most existing deck.gl layers to allow passing GeoArrow tables. This is much faster than providing JSON input and is a core part of what makes Lonboard so fast to render large data from Python. See "how it works".

• On a high level, the long-term goal has always been to have the deck.gl ecosystem integrate Apache Arrow as a first class citizen, rather than offer a duplicated catalog of arrow layers.

I'm 100% supportive of Arrow becoming a first-class citizen and I'd be happy to one-day deprecate @geoarrow/deck.gl-geoarrow.

But deck.gl-geoarrow is an essential stopgap that enables today fast GeoArrow rendering via deck.gl's binary data interface.

I'm currently revamping deck.gl-geoarrow with higher code quality, nicer documentation, etc. It'll be really useful for me to present deck.gl-geoarrow and deck.gl-raster as my (opinionated) workflow to support rendering large vector and raster data in deck.gl.

• This means that deck layers/loaders.gl loaders etc will be Arrow native, luma will offer Arrow compatible GPGPU support etc.

I don't really have the time to spend on integrating deck.gl-geoarrow upstream, but feel free to ping me on Arrow-related PRs into deck/luma and I'll try to review if I have time.