Possible to allow for nested structure in Acquisition Container?

[arjunsridhar12345]

I have many raw hardware signals from different sources that need to be stored in the acquisition container. I would like to store them in nested modules, similar to what can be done in the processing container. Having this structure would make access much friendlier and make downstream analysis much more straightforward.

Is this something that could be added to the core schema, or is there an underlying design reason that acquisition cannot support nested structure?

I also found this extension, which seems to support this use case:
https://pypi.org/project/ndx-acquisition-module/

Is this extension considered stable and appropriate for organizing acquisition data, or would it make sense for this functionality to be supported directly in the acquisition container?

Tagging @bruno-f-cruz for any additional comments.

Sample example is shown below. Currently this is how the acqusition is structured where there are numerous signals and devices all flattened and having long names to distinguish between them:

• Behavior.HarpBehavior.AnalogData
• Behavior.HarpBehavior.AssemblyVersion
• Behavior.HarpBehavior.Camera0Frame
• Behavior.HarpBehavior.Camera0Frequency
• Behavior.HarpBehavior.Camera1Frame
• Behavior.HarpBehavior.Camera1Frequency
• Behavior.HarpBehavior.ClockConfiguration
• Behavior.HarpBehavior.CoreVersionHigh
• ...

Ideally something like below would be nice to have to organize the different streams by devices:

/acquisition
    /Behavior
        /HarpBehavior
            AnalogData
            AssemblyVersion
            Camera0Frame
            Camera0Frequency
            Camera1Frame
            Camera1Frequency
            ClockConfiguration
            CoreVersionHigh
            ...

[rly]

@arjunsridhar12345 Thanks for the proposal. The ndx-acquisition-module extension is not maintained by the core NWB team, but by community members.

@arjunsridhar12345 In your example, what other modules would you create besides "Behavior"? Is "HarpBehavior" an NWB extension object? It seems like it would be a custom group with attributes or datasets for version, frame, frequency, etc. Would some of those fields may be better placed on the device object?

=====

It seems like experiments are becoming increasingly multi-stream/multi-modal where this functionality would be useful in core NWB. @oruebel Would OpenEphys also benefit from this organization? Previously, most people created one or just a couple data objects in acquisition, so adding an extra layer of hierarchy to organize the data objects was not needed, and adding flexibility in the existing organization and access pattern was not worth it.

Now that we have multiple use cases for this identified by the community, I am OK with moving forward with adding optional support for this, such that:

• the old way of placing data objects not in an AcquisitionModule is still allowed but discouraged in the NWB Best Practices
• adding data objects into a module with specific names ("ecephys", "behavior", "ophys", etc.) is encouraged (see also Create default processing modules pynwb#1944), even if there is only one acquired data object.

Alternatively, we could add the option but not change the best practices.

[bruno-f-cruz]

In our case, the leaves of the hierarchical structure are always dataframes with a timestamp for index (but it would be great if this could be generalized). However, since those leaves are often grouped by device we would like to keep that association in "acquisition". Is there an alternative pattern that would currently support this?

[rly]

Is there an alternative pattern that would currently support this?

I still don't have a great sense of what the data are. Could you share an example NWB or other data file?

Are you looking for one level of nesting ("Behavior") in the original example, or two ("Behavior" and "HarpBehavior")?

As an aside: Can the dataframes be merged into a single dataframe per device? I ask because HDF5/PyNWB/MatNWB performance degrades in some situations, such as streaming read from S3, when there are too many objects (more than around 500) in the file.

[arjunsridhar12345]

@rly thanks for getting back.

Currently, all of these are raw data streams from different devices that are stored as a flat DynamicTable (from the dataframes Bruno mentioned) per stream from each device. Right now the table names follow the pattern Behavior.Device.Stream in acquisition to distinguish them — there might be more natural ways to organize this that I’m not aware of.

For example, HarpBehavior device has many streams, and then another device say, HarpManipulator would have raw data streams also.

The kind of nesting we’re thinking about I think is per device: similar to how processing works I believe. So this would be 2 levels basically before the actual data. We’d have a top-level “Behavior” module, and within that, sub-modules for each device, each containing the data for relevant streams. This would make the structure clearer without needing custom extensions. I think something like below is possible in processing:

processing/
└─ Behavior                  # Top-level ProcessingModule
   ├─ HarpBehavior           # Sub-module for device 1
   │   ├─ AnalogData         DynamicTable
   │   ...
   └─ HarpManipulator        # Sub-module for device 2
       ├─ DigitalInputState  # Dynamic Table
       ...
   ... # for as many devices as desired

I think accessing then is more straightforward then having to use the long name pattern that's currently there, and it follows a nice grouping pattern by device as well. Is there something currently that supports this?

I confirmed the above through a little dummy example:

As an aside: Can the dataframes be merged into a single dataframe per device? I ask because HDF5/PyNWB/MatNWB performance degrades in some situations, such as streaming read from S3, when there are too many objects (more than around 500) in the file.

I'm not sure how feasible this would be but yeah that could be possible if performance becomes very degraded.

Let me know if that makes sense, I'll try to see if I can share an NWB file if not

[rly]

Thanks for the clearer example! I think I understand.

Note that in the processing group, ProcessingModule cannot contain another ProcessingModule, so there is only one level of organization allowed. The NWB best practices recommend that processed data are organized into modules by data modality, e.g., "ecephys", "behavior", "ophys", not device, but users can break from that best practice if another organization is more intuitive.

Since the acquisition group typically contains raw data and we want to support acquisition systems writing directly to an NWB file, I think it would make sense to recommend one AcquisitionModule per device. Would it also make sense to allow nesting and add a higher-level module for data modality? Some acquisition systems record data from multiple modalities.

Option 1 - organized by modality, may be more intuitive for using the data:

behavior (AcquisitionModule)
---- system1 (AcquisitionModule)
-------- events (DynamicTable)
---- system2 (AcquisitionModule)
-------- video (ImageSeries)
ecephys (AcquisitionModule)
---- system1 (AcquisitionModule)
-------- AP data (ElectricalSeries)

Option 2 - organized by device, may be more intuitive for writing the data:

system1 (AcquisitionModule)
---- behavior (AcquisitionModule)
-------- events (DynamicTable)
---- ecephys (AcquisitionModule)
-------- AP data (ElectricalSeries)
system2 (AcquisitionModule)
---- behavior (AcquisitionModule)
-------- video (ImageSeries)

Option 3 - single level, one module per modality, consistent with processing module

behavior (AcquisitionModule)
---- events (DynamicTable)
---- video (ImageSeries)
ecephys (AcquisitionModule)
---- AP data (ElectricalSeries)

Option 4 - single level, one module per device

system1 (AcquisitionModule)
---- events (DynamicTable)
---- AP data (ElectricalSeries)
system2 (AcquisitionModule)
---- video (ImageSeries)

Option 5 - leave as is - flat

events (DynamicTable)
AP data (ElectricalSeries)
video (ImageSeries)

Also linking #259 which talks about option 4 and having AcquisitionModule link to a Device.

And linking NeurodataWithoutBorders/helpdesk#37 which talks about a use case for AcquisitionModule objects to group recorded stimulus data, behavior data, and recorded environment data, which is a little more like option 3.

[arjunsridhar12345]

Since the acquisition group typically contains raw data and we want to support acquisition systems writing directly to an NWB file, I think it would make sense to recommend one AcquisitionModule per device. Would it also make sense to allow nesting and add a higher-level module for data modality? Some acquisition systems record data from multiple modalities.

This makes sense to me I think. I think this would be useful given the data we interact with mainly is structured by modality. So it could somewhat mirror the acquisition system which would be intuitive for interacting with data as mentioned and looking at option 1