Java wrapper for big DIA-MS Thermo files

[gsaxena888]

Hi,

I was wondering if there was any interest in creating a partial (or full) Java wrapper around the C++ SDK that works on linux? (By partial, I mean that ultimately, the only key fields I (and others) probably need is : m/z, intensity, rt, scan #, precursor m/z, precursor m/z tolerance, ms type (eg ms1 or ms2)).

(If there's some code that's partially written, we could probably take it from there...(We don't know C++ at all though, so we'd be just extending whatever was already written by following a pattern/recipe etc.)).

I think issue #83 discusses a prototype but I'm not sure if it's been updated for latest Thermo Raw parser etc or the extent to which it works well on linux and with huge DIA files etc.

-Gautam

[caetera]

Hi Gautam,

the modern version of RawFileReader (a cross-platform API provided by Thermo to access raw files) is .NET-based. ThermoRawFileParser is written in C# and thus can run natively on all platforms that have dotnet from Microsoft.
The older C-based API, aka MSFileReader, to the best of my knowledge, is not developed anymore - the latest version is from 2017. It still works, at least for some files, but, again, to the best of my knowledge, most active projects (Proteowizard, MSFragger, MaxQuant, etc) use the modern .NET API.

I do not have any experience in running .NET inside/through Java, and I am afraid it is far out of scope for TRFP.

The actual version of TRFP writes scans iteratively, i.e., it does not create a concise spectrum representation (as an object that can be serialized). There is, however, a stale branch from 2020 that implements basic spectral reading into spectrum objects. It lacks full support for precursors, but basic scan data (m/z, intensity, scan type, etc) is implemented. It should be possible to serialize these objects into something Java can read.

Finally, you might want to consider MSFragger's expertise. It is written in Java and it reads raw files on major platforms; thus, it should have some kind of wrapping implemented.

I hope it helps.

[gsaxena888]

Ok -- would you know if there is any plan to support other output formats that are more binary in nature, such as HDF5 or parquet? (I'm assuming it would be both faster to write to such a format, and faster to read etc.)

[caetera]

Parquet output is supported.

The format has been recently updated (see #192 and #188); the changes have been merged into the next release. Version 1.5.0-alpha has these changes implemented.

Is there an accepted standard for HDF5 output? If so, it would be easier to implement it if necessary.

[gsaxena888]

Great! I'll give that a try! @caetera Do you know (roughly) when it might be beta or GA? (I'm assuming it if it's alpha there are still some kinks to be worked out etc.)

[caetera]

There are a couple of features (still to be implemented) that are planned for the release, but the format for the parquet output won't change. I think it should be possible to get to "release candidate" (stable implementation, internally tested over a bunch of different files) in a couple of weeks, but the "official" release might take more time (no estimate).

[david-bouyssie]

If you want to read Thermo raw files using a C# -> Java bridge, you can try this Scala project:
https://github.com/mzdb/mzdb4s

The IO submodules were developed specifically for creating mzDB files so the API is rather limited, it just allow you to get an iterator of all MS and MS/MS spectra, not to perform random access to individual spectra.

If you want a better API you could try to create Java bindings for this @mobiusklein 's Rust crate:
https://github.com/mobiusklein/thermorawfilereader.rs/

BTW, @caetera do you have somewhere the specs of the created parquet file?

[caetera]

Hi @david-bouyssie

Thank you for helping with the Java.

The specification of mzparquet is vaguely described in #188 and https://github.com/lazear/mz_parquet; there is, however, an ongoing discussion to extend the schema (#193), and some changes (extra metadata) might be added after that. The minimal set of columns is, however, unlikely to change.