Could not find ROIs

[pengyu1608]

Hi André,

I have also came across problem when running the pipeline. I have downloaded the processed data from zenodo.

The directory structure is subfolder_per_sample. For example, in the processed/20200728_NL6699/tiffs directory, there are files:

20200728_NL6699-01_full_mask.tiff
20200728_NL6699-02_full_mask.tiff
20200728_NL6699-03_full_mask.tiff
20200728_NL6699-04_full_mask.tiff
20200728_NL6699-05_full_mask.tiff
20200728_NL6699-06_full_mask.tiff
20200728_NL6699-07_full_mask.tiff
20200728_NL6699-08_full_mask.tiff

And in the metadata directory, there is the file samples.csv, like this:

sample_name,wcmc_code,autopsy_code,acquisition_name,acquisition_date,acquisition_id,disease,organ,phenotypes,instrument,panel_annotation_file,panel_version,panel_file,observ
ations,mcd_file
20200728_NL6699,,S19-6699,20200728_NL6699,20200728,NL6699,Healthy,Lung,Healthy,fluidigm_hyperion,metadata/panel_markers.COVID19-2.csv,,,,data/20200728_NL6699/20200728_NL6699
.mcd

However, when I run the command: python -u src/clinical.py, it returns:

"Could not find ROIs for any of the samples. Either pass metadata with one row per ROI, or set processed_dir in order for ROIs to be discovered, and make sure select the right project stucture with subfolder_per_sample. "

Could you kindly explain to me how to fix it? Thanks.

Peng

[afrendeiro]

Hi Peng,

Each ROI object is determined based on the image stack, not based on the segmentation mask.

So inside each sample's tiffs directory you'd need to have the '_full.tiff' files which are the stacks. For this project since the files arent Ome-tiff, you also need the '_full.csv' which indicate the identity of the channels. The masks can be generated from the stacks and are not necessary to define the ROI structure.

Sorry this is perhaps not explicitly said but the file structure is following convention of other IMC tools. Let me know if this is clear.
Best wishes, André

[pengyu1608]

Thanks, André. Now I have '_full.tiff' in each sample's tiffs directory. But where can I find the '_full.csv' file? I have cloned the whole git directory but can not find them. Can you show me some example files and where to put them? Thanks. Peng From: André F. Rendeiro Date: 2022-03-23 19:11 To: ElementoLab/covid-imc CC: Peng Yu; Author Subject: Re: [ElementoLab/covid-imc] Could not find ROIs (Issue #8) Hi Peng, Each ROI object is determined based on the image stack, not based on the segmentation mask. So inside each sample's tiffs directory you'd need to have the '_full.tiff' files which are the stacks. For this project since the files arent Ome-tiff, you also need the '_full.csv' which indicate the identity of the channels. The masks can be generated from the stacks and are not necessary to define the ROI structure. Sorry this is perhaps not explicitly said but the file structure is following convention of other IMC tools. Let me know if this is clear. Best wishes, André — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[afrendeiro]

Oh so sorry I replied earlier without checking what format the files were that I distributed. In this case, there are MCD files in the repository with raw data (https://zenodo.org/record/4110560) so you'd need these.
From these you can run for example imc prepare to reconstitute the structure needed. Not sure if that's what you did to get the stack files already?

An example of the structure needed can be seen here: https://zenodo.org/record/5719188
This is how I thought the data from the COVID project were uploaded, but I think I didn't due to size restrictions in Zenodo.

On a larger note, what are you trying to achieve? To reproduce the whole project starting from the raw data, I provide a script to download all files from the project: https://github.com/ElementoLab/covid-imc/blob/main/src/_download_datasets.py (called by make download). These can be processed with imc process and then you'd be ready to run the various analysis scripts with make analysis.

However if you'd rather skip the processing and use the single-cell data for further research, I made available a h5ad file with all the information here: https://zenodo.org/record/4139443

[pengyu1608]

Actually I would like to reproduce the interaction analysis part. While running imc process, it reported: TypeError: save() got an unexpected keyword argument 'ome' I guess these files might use some new format, right? Message ID: ***@***.***>

[afrendeiro]

TypeError: save() got an unexpected keyword argument 'ome'

Actually, I just ran into the same issue today. I believe it's due to the pinned version of `tifffile`. Can you update it (`pip install tifffile -U`)? In case you're interested, the interaction analysis is also abstracted in this module: https://imc.readthedocs.io/en/latest/api.html#module-imc.ops.adjacency

…

On Thu, Mar 24, 2022 at 10:47 PM Peng Yu ***@***.***> wrote: Actually I would like to reproduce the interaction analysis part. While running imc process, it reported: TypeError: save() got an unexpected keyword argument 'ome' I guess these files might use some new format, right? Message ID: ***@***.***> — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPYGNTFZSXCQJZJXANE22LVBUSLHANCNFSM5RNADQBA> . You are receiving this because you commented.Message ID: ***@***.***>

[pengyu1608]

'pip install tifffile -U' does help. And 'tifffile==2020.6.3' should be removed from the requirements.txt. In addition, the deepcell module should be added. Message ID: ***@***.***>

[afrendeiro]

I'll do a release soon with the tifffile version. For `imc process` install with extras: pip install imc[extra] as here: https://github.com/ElementoLab/imc#quick-start

…

On Fri, Mar 25, 2022, 04:39 Peng Yu ***@***.***> wrote: 'pip install tifffile -U' does help. And 'tifffile==2020.6.3' should be removed from the requirements.txt. In addition, the deepcell module should be added. Message ID: ***@***.***> — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPYGNXEFZWU7KE7KDYDMGLVBV3SJANCNFSM5RNADQBA> . You are receiving this because you commented.Message ID: ***@***.***>

[pengyu1608]

Thanks, André. Now the cmd 'imc process' could be run. However, there is a new problem, the imc process just hang there for more than 24h: Finished segmentation step! Starting quantification step for 10 TIFF files: - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-01_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-02_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-03_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-04_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-05_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-06_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-07_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-08_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-09_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-10_full.tiff! <tifffile.TiffFile '20200609_ARDS_1921-01_full.tiff'> contains no pages Any thoughts? Message ID: ***@***.***>

[afrendeiro]

Hi Peng, You don't have to run the whole thing since you already have the segmentation masks. Run imc --help and imc process --help to see all options. In your case you only need to convert mcd to tiffs to get the stacks. So you can do --stop-step prepare and you can even probably turn off the ilastik options because those files are only needed for segmentation.

…

On Tue, Mar 29, 2022, 04:09 Peng Yu ***@***.***> wrote: Thanks, André. Now the cmd 'imc process' could be run. However, there is a new problem, the imc process just hang there for more than 24h: Finished segmentation step! Starting quantification step for 10 TIFF files: - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-01_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-02_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-03_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-04_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-05_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-06_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-07_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-08_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-09_full.tiff - processed/20200609_ARDS_1921/tiffs/20200609_ARDS_1921-10_full.tiff! <tifffile.TiffFile '20200609_ARDS_1921-01_full.tiff'> contains no pages Any thoughts? Message ID: ***@***.***> — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPYGNUH6DL4B3Z42P67AP3VCK3ERANCNFSM5RNADQBA> . You are receiving this because you commented.Message ID: ***@***.***>

[pengyu1608]

Okay, Thanks. I am moving on. I have run 'imc process --stop-step prepare' and the tiff files were successfully created. Then I try to run 'make analysis' and installed two other modules as requested: pip install pyarrow pip install fastparquet But a new file was reported missing maybe also because the size limit of Zenodo, Running analysis! # Analysis of first submission # # Global look at the clinical data [Fig1b] python -u src/clinical.py FileNotFoundError: [Errno 2] No such file or directory: 'metadata/clinical_annotation.pq' make: *** [analysis] Error 1 Message ID: ***@***.***>

[pengyu1608]

Dear André: I am not clear about the cell interaction analysis (the Fig 2 f,g in your COVID paper). Currently I have defined the cell types on each ROI and retrived the coordinates of each cell (x,y on the image). Can you briefly explain the logic and main steps of interaction calculation? For example, how to represent the distance of two cell clusters? As I am not familiar with Python. Maybe I could calculate with other tools. It will be best if you can supply some demo notebook for the part. Thanks. Peng Message ID: ***@***.***>