Test failing during the create_features step

[patriciatran]

Hi,

I am trying to run alphapulldown with snakemake on a remote machine that does not have permissions to access all files on the system. (This is actually related to this issue (KosinskiLab/AlphaPulldown#602 - thanks for answering us!)

I am running into this issue during the create_features step: "container creation failed: image driver mount failure: image driver squashfuse_ll instance exited with error: squashfuse_ll exited: fuse: device /dev/fuse not found.". The first step, download_uniprot worked fine. Do you have any suggestions on what we could try to resolve this please?

PS: I see two different repo AlphapulldownSnakemake and Alphapulldown -- what is the difference? They both seem to be using snakemake. And I hope I posted this issue in the correct location.

Thank you!

Patricia

~~

I'm attaching the relevant parts of the snakemake command printout here:

Software versions:

snakedeploy deploy-workflow \
  https://github.com/KosinskiLab/AlphaPulldownSnakemake \
  AlphaPulldownSnakemake \
  --tag 2.1.8

Snakemake
/opt/conda/envs/snake/bin/snakemake
9.19.0
Apptainer:
/opt/conda/envs/snake/bin/apptainer
apptainer version 1.4.5

command:

snakemake --profile config/profiles/desktop \
--cores 8 \
--shared-fs-usage none

Printout:

Activating singularity image /var/lib/condor/execute/slot1/dir_103000/scratch/71d53befa61c41a1b281a601757614f0.simg
INFO:    fuse2fs not found, will not be able to mount EXT3 filesystems
WARNING: While bind mounting '/alphafold3:/alphafold3': destination is already in the mount point list
WARNING: While bind mounting '/staging:/staging': destination is already in the mount point list
FATAL:   container creation failed: image driver mount failure: image driver squashfuse_ll instance exited with error: squashfuse_ll exited: fuse: device /dev/fuse not found. Kernel module not loaded?
RuleException:
CalledProcessError in file "https://raw.githubusercontent.com/KosinskiLab/AlphaPulldownSnakemake/2.1.8/workflow/Snakefile", line 238:
Command ' apptainer  exec --home '/var/lib/condor/execute/slot1/dir_103000/scratch/AlphaPulldownSnakemake' --nv --bind '/tmp/snakemakeq450bkgc/source-cache' /var/lib/condor/execute/slot1/dir_103000/scratch/71d53befa61c41a1b281a601757614f0.simg bash -c 'set -euo pipefail;  
        create_individual_features.py \
            --fasta_paths=/var/lib/condor/execute/slot1/dir_103000/scratch/AlphaPulldownSnakemake/temp_fold/data/P01579.fasta \
            --data_dir=/alphafold3 \
            --output_dir=/var/lib/condor/execute/slot1/dir_103000/scratch/AlphaPulldownSnakemake/temp_fold/features \
            --save_msa_files=False --use_precomputed_msas=True --max_template_date=2050-01-01 --compress_features=True --data_pipeline=alphafold3'' returned non-zero exit status 255.
  File "https://raw.githubusercontent.com/KosinskiLab/AlphaPulldownSnakemake/2.1.8/workflow/Snakefile", line 238, in __rule_create_features
[Wed Apr  8 10:00:06 2026]
Error in rule create_features:
    message: None
    jobid: 6
    input: /var/lib/condor/execute/slot1/dir_103000/scratch/AlphaPulldownSnakemake/temp_fold/data/P01579.fasta
    output: /var/lib/condor/execute/slot1/dir_103000/scratch/AlphaPulldownSnakemake/temp_fold/features/P01579_af3_input.json
    shell:
        
        create_individual_features.py \
            --fasta_paths=/var/lib/condor/execute/slot1/dir_103000/scratch/AlphaPulldownSnakemake/temp_fold/data/P01579.fasta \
            --data_dir=/alphafold3 \
            --output_dir=/var/lib/condor/execute/slot1/dir_103000/scratch/AlphaPulldownSnakemake/temp_fold/features \
            --save_msa_files=False --use_precomputed_msas=True --max_template_date=2050-01-01 --compress_features=True --data_pipeline=alphafold3
        
        (command exited with non-zero exit code)
Select jobs to execute...
Execute 1 jobs...

[DimaMolod]

Hi @patriciatran,

Thanks for the detailed report.

This looks like an Apptainer runtime issue rather than a problem in create_features itself. The job is failing while Apptainer tries to mount the container image:

fuse: device /dev/fuse not found

So create_individual_features.py likely never starts.

That also explains why download_uniprot works: it runs on the host, while create_features is the first step that actually enters the container.

A few things to try:

• Check with your admins whether /dev/fuse is available on the worker nodes.
• If FUSE mounts are not allowed on your system, try Apptainer with --unsquash or use a sandbox image instead of a .sif.
• If containers are not possible at all on that machine, the alternative would be to install AlphaPulldown directly in the environment and run without containerization.

About the two repositories:

• AlphaPulldown contains the core code and containers.
• AlphaPulldownSnakemake contains the Snakemake workflow that runs that code.

Since this failure happens at the workflow/container layer, AlphaPulldownSnakemake is probably the better repo for follow-up, so I am transferring the issue there.

[patriciatran]

Thanks for your response. I'm going to look into those options on my end. If I find a solution to this, I'll post it here in case it helps others.

[patriciatran]

Hi,

I have an update on my tests.
I ended up building a Docker image that has a snakemake installation and a unprivileged user set-up for apptainer.

I ran the snakemake workflow using, using the sandbox option was seemed promising:

snakemake --profile config/profiles/desktop \
	--cores 8 \
	--use-singularity \
	--singularity-args "--unsquash --pid --ipc -B $(pwd):$(pwd) --cwd $(pwd)"

As part of the snakemake log, the error occurs here (same module, but no more /dev/fuse issues now)

Activating singularity image /var/lib/condor/execute/slot3/dir_2660650/scratch/71d53befa61c41a1b281a601757614f0.simg
INFO:    Converting SIF file to temporary sandbox...
WARNING: While bind mounting '/alphafold3:/alphafold3': destination is already in the mount point list
WARNING: While bind mounting '/staging:/staging': destination is already in the mount point list
/usr/bin/bash: line 3: create_individual_features.py: command not found
INFO:    Cleaning up image...
RuleException:
CalledProcessError in file "https://raw.githubusercontent.com/KosinskiLab/AlphaPulldownSnakemake/2.1.8/workflow/Snakefile", line 238:
Command ' apptainer  exec --home '/var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake' --unsquash --pid --ipc -B /var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake:/var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake --cwd /var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake --bind '/var/lib/condor/execute/slot3/dir_2660650/scratch/.cache/snakemake/snakemake/source-cache' /var/lib/condor/execute/slot3/dir_2660650/scratch/71d53befa61c41a1b281a601757614f0.simg bash -c 'set -euo pipefail;  
        create_individual_features.py \
            --fasta_paths=/var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake/temp_fold/data/P01258.fasta \
            --data_dir=/alphafold3 \
            --output_dir=/var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake/temp_fold/features \
            --save_msa_files=False --use_precomputed_msas=True --max_template_date=2050-01-01 --compress_features=True --data_pipeline=alphafold3'' returned non-zero exit status 127.
  File "https://raw.githubusercontent.com/KosinskiLab/AlphaPulldownSnakemake/2.1.8/workflow/Snakefile", line 238, in __rule_create_features

I copied the command causing the error, and it worked properly and I was able to find create_individual_features.py without issue.

apptainer exec --home '/var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake' --unsquash --pid --ipc -B /var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake:/var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake --cwd /var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake --bind '/var/lib/condor/execute/slot3/dir_2660650/scratch/.cache/snakemake/snakemake/source-cache' /var/lib/condor/execute/slot3/dir_2660650/scratch/71d53befa61c41a1b281a601757614f0.simg bash -c 'set -euo pipefail;  create_individual_features.py --helpshort'

Expand output

$ apptainer exec --home '/var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake' --unsquash --pid --ipc -B /var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake:/var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake --cwd /var/lib/condor/execute/slot3/dir_2660650/scratch/AlphaPulldownSnakemake --bind '/var/lib/condor/execute/slot3/dir_2660650/scratch/.cache/snakemake/snakemake/source-cache' /var/lib/condor/execute/slot3/dir_2660650/scratch/71d53befa61c41a1b281a601757614f0.simg bash -c 'set -euo pipefail;  create_individual_features.py --helpshort'
INFO:    Converting SIF file to temporary sandbox...

Feature generator for AlphaFold 2 and AlphaFold 3, supporting classic Hmmer, MMseqs2, and truemultimer modes.


flags:

/opt/venv/bin/create_individual_features.py:
  --bfd_database_path: (no help available)
  --[no]compress_features: (no help available)
    (default: 'false')
  --data_dir: Path to directory of supporting data.
  --data_pipeline: <alphafold2|alphafold3>: Choose pipeline: alphafold2 or alphafold3
    (default: 'alphafold2')
  --db_preset: <full_dbs|reduced_dbs>: (no help available)
    (default: 'full_dbs')
  --description_file: (no help available)
  --fasta_paths: Paths to FASTA files, each containing a prediction target.
    (a comma separated list)
  --hb_allowance: (no help available)
    (default: '0.4')
    (a number)
  --hhblits_binary_path: (no help available)
  --hhsearch_binary_path: (no help available)
  --hmmalign_binary_path: (no help available)
    (default: '/hmmer/bin/hmmalign')
  --hmmbuild_binary_path: (no help available)
    (default: '/hmmer/bin/hmmbuild')
  --hmmsearch_binary_path: (no help available)
    (default: '/hmmer/bin/hmmsearch')
  --jackhmmer_binary_path: (no help available)
    (default: '/hmmer/bin/jackhmmer')
  --kalign_binary_path: (no help available)
  --max_template_date: Max template release date.
  --mgnify_database_path: (no help available)
  --[no]multiple_mmts: (no help available)
    (default: 'false')
  --new_uniclust_dir: (no help available)
  --nhmmer_binary_path: (no help available)
    (default: '/hmmer/bin/nhmmer')
  --obsolete_pdbs_path: (no help available)
  --output_dir: Path to output directory.
  --path_to_mmt: (no help available)
  --pdb70_database_path: (no help available)
  --pdb_seqres_database_path: (no help available)
  --plddt_threshold: (no help available)
    (default: '0.0')
    (a number)
  --[no]re_search_templates_mmseqs2: (no help available)
    (default: 'false')
  --[no]save_msa_files: (no help available)
    (default: 'false')
  --seq_index: (no help available)
    (an integer)
  --[no]skip_existing: (no help available)
    (default: 'false')
  --[no]skip_msa: (no help available)
    (default: 'false')
  --small_bfd_database_path: (no help available)
  --template_mmcif_dir: (no help available)
  --threshold_clashes: (no help available)
    (default: '1000.0')
    (a number)
  --uniprot_database_path: (no help available)
  --uniref30_database_path: (no help available)
  --uniref90_database_path: (no help available)
  --[no]use_hhsearch: (no help available)
    (default: 'false')
  --[no]use_mmseqs2: (no help available)
    (default: 'false')
  --[no]use_precomputed_msas: (no help available)
    (default: 'false')

Try --helpfull to get a list of all flags.
INFO:    Cleaning up image...

I also ran the same command with apptainer shell [command] to enter an interactive container, and it was able to find create_individual_features.py --helpshort without issue.

In summary, the "command not found" error only occurs when I'm running and trying to use it within an Apptainer image that was "initiated" by snakemake.

I am curious to know if you've seen an error like this before, where the apptainer container image technically works and runs properly on its own, but not when initiated by snakemake?

Thank you.

[DimaMolod]

Hi,
Apparently, the issue is that under --unsquash the container's own PATH (where /opt/venv/bin for the AF3 image and /opt/conda/bin for the AF2 image live) isn't propagated properly when launched through Snakemake, so create_individual_features.py can't be found, even though running the same apptainer exec command manually works.
Can you try this and let me know if it works?

snakemake --profile config/profiles/desktop \
  --cores 8 \
  --use-singularity \
  --singularity-args "--nv --unsquash --pid --ipc --env PREPEND_PATH=/opt/venv/bin:/opt/conda/bin -B $(pwd):$(pwd) --cwd $(pwd)"

The --env PREPEND_PATH=... prepends both possible install locations to the container's PATH, which should cover both the venv-based and conda-based images. If it works, I will add that args to the config/profile/desktop/config.yaml