`SpikeGLXRawIO` event extraction from `XD` channels is memory inefficient

[h-mayorquin]

I am reading a nidq file of about 3 hours with reported size of 2.8 GiB. Reading with the following:

from pathlib import Path 


folder_path = Path("/home/heberto/data/just_nidq")


from neo.rawio.spikeglxrawio import SpikeGLXRawIO


rawio = SpikeGLXRawIO(dirname=folder_path)
rawio.parse_header()

Leads to an RSS allocation of 3.8 GiB.

Command line: /home/heberto/miniconda3/envs/work/bin/memray run --native --follow-fork dev.py
Start time: 2024-12-11 13:47:45.762000-06:00
End time: 2024-12-11 13:47:47.532000-06:00
Duration: 0:00:01.770000
Total number of allocations: 42613
Total number of frames seen: 3262
Peak memory usage: 3.9 GiB
Python allocator: pymalloc

The problem could be solved by delaying this parsing to _get_event_timestamps

python-neo/neo/rawio/spikeglxrawio.py

Lines 207 to 225 in 0ae6e76

	# No events
	event_channels = []
	# This is true only in case of 'nidq' stream
	for stream_name in stream_names:
	if "nidq" in stream_name:
	info = self.signals_info_dict[0, stream_name]
	if len(info["digital_channels"]) > 0:
	# add event channels
	for local_chan in info["digital_channels"]:
	chan_name = local_chan
	chan_id = f"{stream_name}#{chan_name}"
	event_channels.append((chan_name, chan_id, "event"))
	# add events_memmap
	data = np.memmap(info["bin_file"], dtype="int16", mode="r", offset=0, order="C")
	data = data.reshape(-1, info["num_chan"])
	# The digital word is usually the last channel, after all the individual analog channels
	extracted_word = data[:, len(info["analog_channels"])]
	self._events_memmap = np.unpackbits(extracted_word.astype(np.uint8)[:, None], axis=1)
	event_channels = np.array(event_channels, dtype=_event_channel_dtype)

[samuelgarcia]

Reading events while parsing header is a big mess in neo.rawio.
We should have option to not parse events for all reader to avoid this.
Very often it needs a big memmap and also sometimes a np.nonzero() which is terrible.

[bendichter]

@h-mayorquin can you provide the file that is causing the issue?

[h-mayorquin]

Yes, @yarikoptic publicly shared a file with the desired characteristics to test this here:

catalystneuro/neuroconv#1112 (comment)