Analyzing Multiple Files Simultaneously with AMR++

[FarzanehRah]

Hi,

Is it possible to analyze multiple files simultaneously with AMR++? I have over 80 samples, I requested 30 CPUs per task and 18 GB of memory per CPU. After 6 days of processing, I’ve found that 80% of my data has already passed trimming. I noticed that the analysis is done one at a time for each step. Is there a way to analyze several samples simultaneously at each step?

Thank you!

[FarzanehRah]

After 6.5 days of running, AMR++ stopped with the error: samtools bam2fq: Error writing to FASTx files.: No such file or directory. When attempting to resume, Nextflow can't find the cache database. Nextflow needs to be executed in a shared file system that supports file locks. Alternatively, you can run it in a local directory and specify the shared work directory by using the -w command line option. java.io.IOException: Can't open cache DB: /lustre/scratch/usr/AMR/.nextflow/cache/5cb82aea-b7ec-40c1-aa8c-c3d28871ddb0/db :((

[susheelbhanu]

Hi, I have a similar question - although there seems to be an issue with running multiple samples.

My command is as follows:

nextflow run main_AMR++.nf --pipeline resistome --reads /prj/results/preprocessed/reads/*/*R{1,2}.fq --output <OUTPUT_DIR> --threads 48

However, it only seems to be running for the first sample and not all the samples. Should all the samples be in the same directory?

 NE X T F L O W   ~  version 24.04.4

Launching `main_AMR++.nf` [distracted_saha] DSL2 - revision: 8426bc09be

 A M R + +    N F   P I P E L I N E
 ===================================
 Reads being analyzed : /prj/results/preprocessed/reads/A30_2022/A30_2022_filtered.R1.fq
 With the pipeline    : resistome
 Output folder        : /prj//results/amr_plus_plus

===================================
Running the resistome subworkflow
===================================

executor >  local (5)
[c4/4aace2] FASTQ_RESISTOME_WF:bwa_align (A30_2022_filtered)              [100%] 1 of 1 ✔
[33/1047c4] FASTQ_RESISTOME_WF:runresistome (A30_2022_filtered)           [100%] 1 of 1 ✔
[b1/631261] FASTQ_RESISTOME_WF:resistomeresults (Make AMR count matrix)   [100%] 1 of 1 ✔
[09/400164] FASTQ_RESISTOME_WF:runrarefaction (A30_2022_filtered)         [100%] 1 of 1 ✔
[6a/6e6812] FASTQ_RESISTOME_WF:plotrarefaction (Plot rarefaction results) [100%] 1 of 1 ✔
Pipeline completed!
Started at  2024-08-07T04:49:33.140258966+01:00
Finished at 2024-08-07T04:55:12.852192730+01:00
Time elapsed: 5m 40s
Execution status: OK
Completed at: 07-Aug-2024 04:55:12
Duration    : 5m 40s
CPU hours   : 0.1
Succeeded   : 5

For example, see below the structure of my files versus those in the test dataset

 >>ls /prj/results/preprocessed/reads/*/*_filtered.R{1,2}.fq | head

/prj/results/preprocessed/reads/A30_2022/A30_2022_filtered.R1.fq
/prj/esults/preprocessed/reads/A30_2022/A30_2022_filtered.R2.fq
/prj/results/preprocessed/reads/A3_2022/A3_2022_filtered.R1.fq
/prj/results/preprocessed/reads/A3_2022/A3_2022_filtered.R2.fq
/prj/results/preprocessed/reads/A43_2021/A43_2021_filtered.R1.fq
/prj/results/preprocessed/reads/A43_2021/A43_2021_filtered.R2.fq
/prj/results/preprocessed/reads/A44_2021/A44_2021_filtered.R1.fq
/prj/results/preprocessed/reads/A44_2021/A44_2021_filtered.R2.fq


>>ls data/raw/*R{1,2}.fastq.gz | head                            
data/raw/S1_test_R1.fastq.gz
data/raw/S1_test_R2.fastq.gz
data/raw/S2_test_R1.fastq.gz
data/raw/S2_test_R2.fastq.gz
data/raw/S3_test_R1.fastq.gz
data/raw/S3_test_R2.fastq.gz

Thanks,
Susheel

[FarzanehRah]

Hi Susheel, yes, place all your fastq files in the same directory. The pipeline will automatically process each pair of files.

[susheelbhanu]

Thanks. It'll help to update the readme with these instructions.

Best!