docs: document multi-worker DataLoader support for remote tables#260
docs: document multi-worker DataLoader support for remote tables#260mintlify[bot] wants to merge 4 commits into
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
prrao87
added
the
needs_new_release
|
cc @westonpace - I presume these docs make sense based on your latest additions. |
|
I pushed a follow-up commit to make the PyTorch examples match current LanceDB behavior. What I tested:
Why this change is necessary:
|
|
@westonpace could you please review this PyTorch docs update when possible? As I was testing the code that the agent wrote, some more recent changes that we merged recently caused this error to surface. TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'pyarrow.lib.ChunkedArray'>I've patched a fix that uses Also, are we missing anything in the documentation about how to handle these on |
westonpace
left a comment
westonpace
left a comment
There was a problem hiding this comment.
Some suggestions but nice examples
| significant performance penalty converting from Arrow, Lance's internal representation, to this default format. Use a | ||
| direct `Table` with `collate_fn` when you want Arrow-to-tensor conversion, or a `Permutation` when you want the default | ||
| PyTorch dict-of-tensors behavior. |
There was a problem hiding this comment.
Hmm, Permutation can also support direct Arrow-to-tensor conversion. It just isn't the default. This makes it sound like you'd have to use a Table.
|
|
||
| Set `num_workers > 0` to read from LanceDB in multiple PyTorch worker processes. LanceDB tables and `Permutation` objects are picklable, so each worker reopens the table after it starts. | ||
|
|
||
| Prefer the `spawn` start method when using multiple workers; LanceDB uses internal threads. See [the performance guide](/performance) for more multiprocessing guidance. |
There was a problem hiding this comment.
Actually forkserver is probably better than spawn (and will be the new python default). I'd say that should be our preference.
Once the streaming dataset is available my guidance would be to use forkserver and use num_workers=1 unless you can prove you have GIL contention in your trasform function.
2 participants
Summary
Document that remote LanceDB tables can now be used directly with multi-worker PyTorch
DataLoaders, and refresh guidance on the optionalwith_connection_factoryescape hatch.Changes
num_workers,spawn, andpersistent_workers.db://) tables working in worker processes out of the box.Context
Triggered by an upstream change that lets remote tables carry their connection state through pickling so they reopen correctly in PyTorch DataLoader workers, while keeping the connection factory available for custom credential loading.