Add support for VECTORIZED orientation in BrainVisionRawIO by Kkuntal990 · Pull Request #1797 · NeuralEnsemble/python-neo

Kkuntal990 · 2025-10-17T22:56:30Z

Description

This PR adds support for VECTORIZED data orientation in BrainVisionRawIO, which was previously causing a NeoReadWriteError.

Problem

BrainVision files can store data in two orientations:

MULTIPLEXED: ch1_s1, ch2_s1, ch3_s1, ch1_s2, ch2_s2, ... (interleaved)
VECTORIZED: all ch1 samples, then all ch2 samples, etc. (sequential)

Previously, Neo only supported MULTIPLEXED orientation, causing imports to fail on VECTORIZED files such as those in OpenNeuro dataset ds004621.

Solution

Modified the orientation check to accept both MULTIPLEXED and VECTORIZED
Implemented custom _get_analogsignal_chunk() method that:
- Uses the default parent implementation for MULTIPLEXED files (no behavior change)
- For VECTORIZED files, reads each channel's data from its sequential location in the binary file
Maintains full backward compatibility with existing MULTIPLEXED files

Testing

✅ Validation against MNE-Python: Tested on real-world VECTORIZED dataset (ds004621, 127 channels × 740,360 samples)

Correlation: 1.0
Maximum absolute difference: 0.0
Results match MNE-Python exactly

✅ Synthetic tests: Created test files in both orientations and verified correct data reading

✅ Backward compatibility: MULTIPLEXED files continue to work as before

Changes

neo/rawio/brainvisionrawio.py:
- Added self._data_orientation to track orientation type
- Implemented custom _get_analogsignal_chunk() for VECTORIZED reading
- Updated error message to reflect support for both orientations

Related Issue

Dataset: OpenNeuro ds004621
Github Issue

Fixes KAN-43: https://eeglab.atlassian.net/browse/KAN-43 Previously, BrainVisionRawIO only supported MULTIPLEXED data orientation (interleaved channels), causing a NeoReadWriteError when attempting to read files with VECTORIZED orientation (sequential channel data). Changes: - Modified data orientation check to accept both MULTIPLEXED and VECTORIZED - Added custom _get_analogsignal_chunk() method to handle VECTORIZED reading - For VECTORIZED files, reads each channel's data from its sequential location in the binary file - Maintains backward compatibility with MULTIPLEXED files (uses parent class implementation) Testing: - Validated against MNE-Python on real-world VECTORIZED dataset (ds004621) with 127 channels × 740,360 samples - results match exactly (correlation=1.0) - Tested both MULTIPLEXED and VECTORIZED orientations with synthetic data - All existing functionality preserved for MULTIPLEXED files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

zm711 · 2025-10-19T16:10:58Z

Thanks @Kkuntal990. We will try to review this soon. I've run the tests now.

h-mayorquin · 2025-10-20T19:11:38Z

Can you open an issue here in neo? I can't see the jira board.

Kkuntal990 · 2025-10-22T21:42:04Z

Can you open an issue here in neo? I can't see the jira board.

Done

samuelgarcia · 2025-10-23T09:51:43Z

Thanks a lot for adding the other data layout in brainvision.

2 points:

I am not sure to like the np.fromfile() implementation maybe we could also use memmap here.
In multiprocessing onlinux with mp_context=fork this lead to errors because processes share the same fid.
Also,maybe, we could move this logic extend in base the BaseRawIO._get_analogsignal_chunk() and we should add a time_axis=0/1 concept to describe the data layout (time, chans) or (chans, times).

- Replace np.fromfile() with np.memmap for multiprocessing compatibility - Minimize code changes in _get_analogsignal_chunk() - Remove file handle caching logic (memmap handles this) - All tests still pass with identical results

@samuelgarcia

Moved VECTORIZED orientation logic to BaseRawIO as suggested by @samuelgarcia: - Added time_axis parameter to buffer_description (0=MULTIPLEXED, 1=VECTORIZED) - Extended BaseRawIO._get_analogsignal_chunk() to handle time_axis=1 for raw buffers - Removed custom _get_analogsignal_chunk() override from BrainVisionRawIO - Fixed _get_signal_size() to correctly handle raw buffers with time_axis=1 Benefits: - Cleaner, more general solution applicable to other readers - Consistent with existing HDF5 time_axis pattern - Reduced code duplication - All tests pass with identical MNE-Python validation

When time_axis=1, shape should be (channels, time) not (time, channels). This makes raw binary handling consistent with HDF5: - time_axis=0: shape is (time, channels) - MULTIPLEXED - time_axis=1: shape is (channels, time) - VECTORIZED Changes: - BrainVisionRawIO: Set shape as (channels, time) when VECTORIZED - BaseRawIO: Use shape[time_axis] consistently for i_stop default - Removed duplicate time_axis retrieval All tests pass with identical MNE-Python validation.

Kkuntal990 · 2025-11-05T02:40:34Z

Latest Updates: Refactored to use time_axis pattern in BaseRawIO

Thanks @samuelgarcia for the suggestion! I've refactored the implementation to move the logic into BaseRawIO using the time_axis parameter, making it consistent with the existing HDF5 pattern.

Changes Made

1. Replaced np.fromfile() with np.memmap()

Avoids multiprocessing issues with shared file handles
More efficient and safer

2. Extended BaseRawIO with time_axis support for raw buffers

Added time_axis=1 handling in BaseRawIO._get_analogsignal_chunk()
Follows the same convention as HDF5:
- time_axis=0: shape is (time, channels) - MULTIPLEXED
- time_axis=1: shape is (channels, time) - VECTORIZED
Updated _get_signal_size() to use shape[time_axis] consistently

3. Simplified BrainVisionRawIO

Sets time_axis in buffer_description based on DataOrientation
Adjusts shape to match time_axis convention (for VECTORIZED: (channels, time))
No custom override needed - relies entirely on BaseRawIO

4. Fixed shape convention consistency

Initially had inconsistency where raw buffers used (time, channels) even with time_axis=1
Now fully consistent: time_axis=1 means shape is (channels, time) for both raw and HDF5
More general: Other readers can now use time_axis for similar layouts
Cleaner: Removed BrainVision-specific override logic
Consistent: Same pattern throughout BaseRawIO (raw & HDF5)

Please review when you can

Kkuntal990 · 2025-11-11T15:50:42Z

@samuelgarcia @h-mayorquin if possible could you please review the PR ?

Thank you for your time.

samuelgarcia · 2025-11-14T14:56:26Z

Hi @Kkuntal990.
Really sorry for the dealy.
I will try my best next week

Kkuntal990 · 2025-12-02T11:17:30Z

Hi @Kkuntal990. Really sorry for the dealy. I will try my best next week

Hi, could you please review if you get time

arnodelorme · 2025-12-10T00:29:05Z

Could we merge this branch because it is a blocker for us. Tx

samuelgarcia · 2025-12-10T20:18:48Z

Désolé. J'ai battu mon record de procratstination...

zm711 · 2026-01-01T16:32:07Z

@Kkuntal990,

once we fix our testing I'll be doing the next release (early January). Could you let me know if you want to be included in our author list? I would need the name you want to appear and any affiliations that should be included.

Kkuntal990 added 3 commits November 4, 2025 18:02

Simplify VECTORIZED implementation to use memmap

9fc58a0

- Replace np.fromfile() with np.memmap for multiprocessing compatibility - Minimize code changes in _get_analogsignal_chunk() - Remove file handle caching logic (memmap handles this) - All tests still pass with identical results

alejoe91 approved these changes Dec 10, 2025

View reviewed changes

samuelgarcia approved these changes Dec 10, 2025

View reviewed changes

samuelgarcia merged commit 8be2db7 into NeuralEnsemble:master Dec 10, 2025
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for VECTORIZED orientation in BrainVisionRawIO#1797

Add support for VECTORIZED orientation in BrainVisionRawIO#1797
samuelgarcia merged 4 commits intoNeuralEnsemble:masterfrom
Kkuntal990:BrainVisionIO-vectorized-fix

Kkuntal990 commented Oct 17, 2025 •

edited

Loading

Uh oh!

zm711 commented Oct 19, 2025

Uh oh!

h-mayorquin commented Oct 20, 2025

Uh oh!

Kkuntal990 commented Oct 22, 2025

Uh oh!

samuelgarcia commented Oct 23, 2025

Uh oh!

Kkuntal990 commented Nov 5, 2025 •

edited

Loading

Uh oh!

Kkuntal990 commented Nov 11, 2025

Uh oh!

samuelgarcia commented Nov 14, 2025

Uh oh!

Kkuntal990 commented Dec 2, 2025

Uh oh!

arnodelorme commented Dec 10, 2025

Uh oh!

samuelgarcia commented Dec 10, 2025

Uh oh!

Uh oh!

zm711 commented Jan 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

Kkuntal990 commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Problem

Solution

Testing

Changes

Related Issue

Uh oh!

zm711 commented Oct 19, 2025

Uh oh!

h-mayorquin commented Oct 20, 2025

Uh oh!

Kkuntal990 commented Oct 22, 2025

Uh oh!

samuelgarcia commented Oct 23, 2025

Uh oh!

Kkuntal990 commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Latest Updates: Refactored to use time_axis pattern in BaseRawIO

Changes Made

1. Replaced np.fromfile() with np.memmap()

2. Extended BaseRawIO with time_axis support for raw buffers

3. Simplified BrainVisionRawIO

4. Fixed shape convention consistency

Uh oh!

Kkuntal990 commented Nov 11, 2025

Uh oh!

samuelgarcia commented Nov 14, 2025

Uh oh!

Kkuntal990 commented Dec 2, 2025

Uh oh!

arnodelorme commented Dec 10, 2025

Uh oh!

samuelgarcia commented Dec 10, 2025

Uh oh!

Uh oh!

zm711 commented Jan 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Kkuntal990 commented Oct 17, 2025 •

edited

Loading

Kkuntal990 commented Nov 5, 2025 •

edited

Loading