Add `pandas` to main dependencies?

[JoeZiminski]

(AFAICT) pandas is required for sorting in kilosort as this line here as BasePhyKilosortSortingExtractor requires it.

As this should be quite a common use case (sorting with kilosort4 locally), should we add pandas to the core dependencies? I don't think it's particularly heavy (?)

[alejoe91]

I think also @chrishalcrow suggested it! I'm not against, even because in #4183 alot of metrics functionality is moved to core and uses pandas indeed!

[chrishalcrow]

@samuelgarcia suggested the opposite: try to move pandas as far away from core as possible. So another solution is to try and remove it from BasePhyKilosortSortingExtractor.

My argument in favor of it is that you need pandas installed to be able to open a sorting analyzer which has quality metrics as an extension (which is hopefully all of them!!!)

[JoeZiminski]

Do you know if it would be possible in the case of BasePhyKilosortSortingExtractor? It looks like it's reading the .tsv outputs from kilosort which I guess would require writing a custom csv/tsv writer. I also think using the sorting analyzer with quality metrics is a similarly convincing use case to add to main dependencies.

Do you know the reason for avoiding pandas? I never noticed it take particularly long to import, but we could benchmark.

[h-mayorquin]

I agree with @samuelgarcia on this one. Pandas is heavy and I don't particularly like the API.

I think that a lot of what can be accomplished with pandas can be accomplished with a structured numpy dtype.

That said, some of the quality metrics and comparison modules make heavy use of it and I think if more cases are shown where pandas would make things simpler across many points in the library I could change my mind.

Note that the standard library has a csv parser:
https://docs.python.org/3/library/csv.html

[chrishalcrow]

I'd be up for a project trying to reduce our dependency on it in the QM and comparison modules.

[JoeZiminski]

Thanks all, if pandas is heavy and there is benefit in removing it it could be worth taking a look at. I don't think it would be trivial necessarily looking at the pandas functionality used in BasePhyKilosortSortingExtractor and quality metrics, but could be worth a shot to see how hard it is.

I would suggest however that the workflow order is:

1. add pandas to core dependencies
2. work to remove pandas from BasePhyKilosortSortingExtractor and qualitry metrics
3. remove pandas from core dependencies

I would class quality metrics (and probably kilosort-sorting) as core uses of the library. At present I think the behaviour is unexpected for users and can be inconvenient, e.g. people might wait some time for a HPC compute node only to have it fail due to this issue.

[MarinManuel]

Hi,
I'm reviving this thread because I was investigating a weird error trying to use read_phy().

Traceback (most recent call last):
  File "test_SI.py", line 7, in <module>
    out = si.read_phy('phy/')
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "envs/SI104/lib/python3.12/site-packages/spikeinterface/extractors/phykilosortextractors.py", line 268, in __init__
    BasePhyKilosortSortingExtractor.__init__(
  File "envs/SI104/lib/python3.12/site-packages/spikeinterface/extractors/phykilosortextractors.py", line 175, in __init__
    unit_ids[i] = new_si_id
    ~~~~~~~~^^^
ValueError: assignment destination is read-only

After some digging around, I figured out that the culprit is pandas 3.0 in this environment. Downgrading to pandas 2.3.3 solves the issue. just FYI.