COMBINE_ANNOTATIONS failed

[shaze]

Description of the bug

When annotating, my run failed with the following

ERROR ~ Error executing process > 'WRIGHTONLABCSU_DRAM:DRAM:ANNOTATE:COMBINE_ANNOTATIONS'

Caused by:
  Process `WRIGHTONLABCSU_DRAM:DRAM:ANNOTATE:COMBINE_ANNOTATIONS` terminated with an error exit status (1)


Command executed:

  # export constants for script
  export FASTA_COLUMN="input_fasta"

  # Create a log directory if it doesn't exist
  mkdir logs

  # Define the log file path
  touch logs/combine_annotations.log
  log_file="logs/combine_annotations.log"

  combine_annotations.py --annotations [4915_6, /projects/scott/DRAM/work/0b/1ac9e9c09b2de76e39b46c1fc11ed3/4915_6_formatted_sulfur_hits.csv, C178_17, /projects/scott/DRAM/work/1b/cd3fb78e917474dac397085e0a853e/C178_17_formatted_sulfur_hits.csv, C178_11, projects/scott/DRAM/work/df/fb766a666bf0d8eea8d16d77a4d603/C178_11_formatted_vog_hits.out, 4921_30, projects/scott/DRAM/work/77/49e86283e8ffb94fca13027539fd6f/4921_30_formatted_sulfur_hits.csv, 4918_21, /projects/scott/DRAM/work/2c/6f80a844cd6dd33c0f8d6fed5ff ........

When I investigated and tried to run manually I got messages like

/projects/scott/DRAM/work/0b/1ac9e9c09b2de76e39b46c1fc11ed3/4915_6_formatted_sulfur_hits.csv can't be found. The file is there on the system, but is not accessible in the container: By adding a -B /projects to the .command.run file I could get it to run to completion and also by adding

apptainer.runOptions = " -B /projects/Scott"

to the nextflow.config file

But perhaps the underlying problem is that the COMBINE_ANNOTATIONS process should have its file inputs as paths rather than vals so that the annotation files get staged in to the working directory

process COMBINE_ANNOTATIONS {
    label 'process_low'

    errorStrategy 'finish'

    conda "${moduleDir}/environment.yml"
    container "community.wave.seqera.io/library/python_pandas_hmmer_mmseqs2_pruned:4a55e4bf58e4a06b"

    input:
    val all_annotations
    val all_genes

System information

System information

• DRAM version, WrightonLabCSU/dram v2.0.0-beta11-g630b86e
• Nextflow version 25.04.4
• HPC
• Executor slurs
• Container engine: apptainer
• Rocky 9

Command used and terminal output

`nextflow run DRAM -r dev --rename --call --input_fasta //lion/qc-contigs-all/indivs --annotate --use_uniref --use_uniref --use_vogdb --use_sulfur -profile singularity --outdir out -resume`

Relevant files

No response

[madeline-scyphers]

Thank you for this, sorry it took me a couple weeks to look at these, I was out of the office.

You are probably right that it should be paths instead of vals. Since paths I believe Nextflow automatically will do a symlink with paths. Where vals are just treated as strings. I will get working on this after I get caught up on things and try and get a fix out soon!

[shaze]

Many thanks.

I also encountered another problem at the point DISTILL_SCRIPT runs

FileNotFoundError: [Errno 2] No such file or directory: /home/scott/.nextflow/assets/WrightonLabCSU/DRAM/assets/forms/distill_sheets/distill_carbon.tsv

The problem is that that the DISTILL_DIR that distill.py" uses, viz., /home/scott/nextflow/assets/WrightonLabCSU/DRAM/assets/` is not auto mounted. I added that in my config

[madeline-scyphers]

Huh, this is all very interesting. I haven't used Nextflow before, so I am learning as I go. But I believe I following the instructions in both Nextflow and nf-core. As well as our test versions we have used that we install directly from pulling also work. Thanks for flagging this. I will try and figure out what is wrong and figure out a better way to catch that.

[shaze]

The problem is more of a singularity issue than a Nextflow issue though the interaction complicates matters. The underlying problem is that the file system of the container is not the same as the host operating system.

In this case, you have in distill.py

DISTILL_DIR = Path(__file__).parent.parent

But this directory is not in the container's file system. (One reason why it may have worked in your test environment and not mine is that I am running the workflow from my /projects directory which is on a different file system to the /home/scott directory). I think you either have explicitly bind the project directory into the environment, or perhaps more cextflow-idiomatic would be to stage the distill_sheets directory into the DISTILL process

Anyway -- enough kibitzing

Most important, thank you for all this work on DRAM. We have found it immensely useful for our project.

[madeline-scyphers]

Right. Yes, the problem is that with singularity (and other containerization environments supported) that not everything is getting mounted for the container runtime. But what I meant was that Nextflow is supposed to take care of all of that in a system and environment agnostic way (mostly). So whether you say -profile singulary or apptainer or docker, or podman, etc. shouldn't matter, because NF takes care of the nitty gritty. Including, binding everything inside the bin directory for you, and I thought also everything in the assets directory. It seems like the latter is not true. So either move the distill sheets into the bin directory I think would solve it. Or pass them as CLI args to the script with them getting staged in the NF process.

[shaze]

I think that while this is correct: because NF takes care of the nitty gritty. Including, binding everything inside the bin directory for you, the problem here is that your distill.py directory refers to something in a cousin directory which is not bound. When I look at the .comman.run the code is this (editing out some things)

apptainer exec --no-home --pid -B /projects/scott/DRAM/work -B /home/scott/.nextflow/assets/WrightonLabCSU/DRAM/bin /home/scott/.singularity/community.wave.seqera.io-library-python_pandas_openpyxl_click-71fdc06a3fdfbdc1.img

In other words in the process this is accessible /home/scott/.nextflow/assets/WrightonLabCSU/DRAM/bin but /home/scott/.nextflow/assets/WrightonLabCSU/DRAM/bin/../../assets/forms/distill_sheets (the definition of DISTILL_DIR) is not.

[madeline-scyphers]

Yeah, exactly. I will get that fixed on the next release.

[madeline-scyphers]

THis should be fixed in the latest release, beta13