supporting video in NWB datasets

[bendichter]

As discussed in dandi/helpdesk#30, it would be useful to have a way to upload stand-alone video files to DANDI and have them be associated with NWB files.

The way NWB handles this is to support linking to external files. In the ImageSeries object, you have the option to specify an external_path of a video file. This file is usually in the same folder as the NWB file. Then, instead of reading the data internally, an API would know to read the data from the designated location. It seems that would be a bit tricky to implement on DANDI, because the files use web-based referencing instead of a local file system. Maybe the best approach would be to change the value in external_path to an s3 location. I don't think think could be done during dandi organize because that s3 location would not yet be known. @yarikoptic do you have any ideas for how we might implement this?

[yarikoptic]

Some bullet point thoughts

• I don't think we should have s3 dandiarchive location paths in the nwb file: overfits for the archive, and would not be easily "usable" locally if file is e.g. already downloaded.
  • within our extracted metadata - yes we should "link" them somehow (problem -- asset IDs are mutable, we don't want to change linkage with a minimal extracted metadata change, blob IDs are not really a semantic descriptor, but probably the best fit here)
• another problem is that external_path then needs to be adjusted by dandi organize (so no symlinks, etc modes; copy or modify data files)! That would end up "expensive" and lengthy etc.
• above aspect is not "unique" - we were already thinking about possible ways how metadata changes in .nwb could be "sidecar"-ed in some supplementary lightweight file/record.
• We can easily implement it at dandi (-cli, whatnot) level with e.g. a matching .json (I think it would end up being consistent with BIDS usage of .json) and add fields there to provide such linkage/overloads between files, and then incorporate that information within dandischema representation using semantic linkage annotation
• ideally there should be nwb-level solution: not just external_path for a specific "entry" , but establish chaining of the multiple (or at least two) .nwb files:
  • then dandi organize could populate those "next one" file with any desired overrides, without needing to modify the original file

[bendichter]

ideally there should be nwb-level solution: not just external_path for a specific "entry" , but establish chaining of the multiple (or at least two) .nwb files:

What do you mean by an nwb-level solution? You can already chain together two NWB files with external links, but that has all the same problems as you mention above. Are you talking about a separate text file that links two files together? Also, note that linking two nwb files together would not solve the current problem, because one of the files needs to be a standard video file (like .mp4)

[yarikoptic]

I thought about having 3 files:

• whatsoever_modalities.nwb - the original data etc, just renamed
• whatsoever.mp4 - video, just renamed
• whatsoever.nwb - small file we (dandi organize) creates which
  • chains into whatsoever_modalities.nwb (relative path)
  • has video ( whatsoever.mp4) referenced via external_path for that video modality data type
  • overrides (if possible) for any metadata (or data?) we need to tune

The point will be for users to load whatever.nwb to get most compete representation , or if desired, load individual files

[satra]

so for web based access, someone would need to check via the path api if these files exist? this seems to build in assumptions like bids into a naming scheme. can we also come up with something that allows some referencing just on the Web. should dandi server update metadata record to inject related resource links?

[yarikoptic]

If there was a way to specify multiple alternative external_path (pretty much like git annex does), then we could have added urls to point to end point to fetch blob by digest or Sha 256.
So may be just a matter of making those external references to lists within nwb?

[yarikoptic]

@bendichter Should we consider this issue closed by #841 which added support for organizeing video files into dandisets?

[bendichter]

yes!