Zero dimension vectors cause Lance to panic

[wjones127]

Not that this is useful, but it should return an error instead of panicking.

import pyarrow as pa
import lance
schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("vec", pa.list_(pa.float32(), 0)),  # Fixed size = 0
])
data = pa.table({
    "id": [1, 2, 3],
    "vec": [[], [], []],  # Empty arrays matching the 0 dimension
}, schema=schema)

ds = lance.write_dataset(data, "memory://")
ds.to_table()

thread 'lance_background_thread' panicked at /home/will/Documents/lance/rust/lance-encoding/src/data.rs:403:9:
attempt to divide by zero
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'lance_background_thread' panicked at /home/will/Documents/lance/rust/lance-table/src/utils/stream.rs:390:46:
called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(46), "attempt to divide by zero", ...)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/will/Documents/lance/python/python/lance/dataset.py", line 1073, in to_table
    ).to_table()
  File "/home/will/Documents/lance/python/python/lance/dataset.py", line 4631, in to_table
    return self.to_reader().read_all()
  File "pyarrow/ipc.pxi", line 762, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: External error: RuntimeError: Task was aborted

[Maxwell-Guo]

please assign this issue to me, would like to take a look . @wjones127

[Maxwell-Guo]

@wjones127 hi, I have some ways to deal with this issue:
1、add an assert before here
2、change the code to check_div

self.child.num_values().checked_div(self.dimension)

but for this way i want to know if there is a default value for num_value ,that is to say when dimension is 0 ,can we return the default value ?
looking forward to your reply.

[wjones127]

but for this way i want to know if there is a default value for num_value ,that is to say when dimension is 0 ,can we return the default value ?

I think if it's zero, the dividing by it should be None.

https://doc.rust-lang.org/std/primitive.u64.html#method.checked_div

For the scope of this, I would make your goal to have it error, and not panic.

But in the future we could think about making this work without an error.

[Maxwell-Guo]

@wjones127 I have run cargo test locally , and all test cases passed , here is the pr.

[karol-kedzia]

Heads up — this still pops in pylance 3.0.1 and 4.0.0, but the panic now fires on write rather than on to_table. So under data_storage_version="2.1" or "2.2" a fixed_size_list[0] schema is straight-up unwriteable. "stable" (= 2.0) still goes through fine.

Min repro that panics on write

import pyarrow as pa, lance, tempfile, os

schema = pa.schema([
    ("k", pa.string()),
    ("delta", pa.list_(pa.float32(), 0)),  # fixed_size_list, width 0
])
table = pa.table({"k": ["x"], "delta": [[]]}, schema=schema)

with tempfile.TemporaryDirectory() as td:
    lance.write_dataset(table, os.path.join(td, "x.lance"),
                        data_storage_version="2.1")  # or "2.2"

thread 'lance-cpu' panicked at .../lance-encoding/src/data.rs:399:9:
attempt to divide by zero
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: RecvError(())

Quick look at DictionaryDataBlock::decode_helper:

let estimated_size_bytes = self.dictionary.data_size()
    * (self.indices.num_values + self.dictionary.num_values() - 1)
    / self.dictionary.num_values();

There's an early return for indices.num_values == 0 right above, but no symmetric guard for dictionary.num_values() == 0, which is what a width-0 fixed-size-list ends up giving you. stable (= 2.0) doesn't go through this dictionary path so it's unaffected.

We're working around it on our side (rejecting 0-dim patterns up front, since a zero-dimension geometry doesn't really mean anything for our use case anyway), so no rush from us — just dropping a note in case it's useful that the panic regressed earlier into the pipeline. A typed error at the writer boundary instead of the panic would be nicer to bisect than chasing the stack into a background thread, but totally understand if it stays an edge case 🙂