DeepTrio v1.9.0/v1.10.0 PACBIO candidate_sweep crashes: candidates_writer.write AttributeError

[codespacebyaditya]

DeepTrio crashes during the candidate_sweep phase when running with --model_type PACBIO on PacBio HiFi trio data.
Every shard fails with:
AttributeError: '_GeneratorContextManager' object has no attribute 'write'. The bug was reproduced across two independent HPC clusters, on both CPU and GPU SIF images, and on both v1.9.0 and v1.10.0.
DeepTrio --model_type=WGS (Illumina) works correctly using the same v1.9.0 SIF- the bug is specific to the PACBIO model path that triggers candidate_sweep.

Setup:
Operating system: RHEL 8 (HPC compute nodes)
DeepVariant version: DeepTrio v1.9.0 and v1.10.0
Installation method: Singularity 3.10.0 (singularity pull docker://google/deepvariant:deeptrio-1.9.0 and docker://google/deepvariant:deeptrio-1.10.0; also tested GPU SIFs deeptrio-1.9.0-gpu and deeptrio-1.10.0-gpu)
Type of data: PacBio HiFi WGS trio (child + mother + father), haplotagged BAMs aligned to GRCh38 (hg38)

Command:
singularity exec
--bind /usr/lib/locale/
--bind /path/to/data:/path/to/data
deepvariant_deeptrio-1.9.0.sif
/opt/deepvariant/bin/deeptrio/run_deeptrio
--model_type PACBIO
--ref /path/to/hg38.fa
--reads_child /path/to/child.GRCh38.haplotagged.bam
--reads_parent1 /path/to/mother.GRCh38.haplotagged.bam
--reads_parent2 /path/to/father.GRCh38.haplotagged.bam
--output_vcf_child /output/child.vcf.gz
--output_vcf_parent1 /output/mother.vcf.gz
--output_vcf_parent2 /output/father.vcf.gz
--output_gvcf_child /output/child.g.vcf.gz
--output_gvcf_parent1 /output/mother.g.vcf.gz
--output_gvcf_parent2 /output/father.g.vcf.gz
--sample_name_child "CHILD"
--sample_name_parent1 "MOTHER"
--sample_name_parent2 "FATHER"
--num_shards 32
--intermediate_results_dir /output/tmp/intermediate_results

v1.9.0 traceback:
Traceback (most recent call last):
File ".../deeptrio/make_examples.py", line 447, in
app.run(main)
File ".../absl/app.py", line 312, in run
_run_main(main, args)
File ".../absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File ".../deeptrio/make_examples.py", line 437, in main
make_examples_core.make_examples_runner(options)
File ".../deepvariant/make_examples_core.py", line 3306, in make_examples_runner
candidates_writer.write(
AttributeError: '_GeneratorContextManager' object has no attribute 'write'

v1.10.0 traceback:
File ".../deepvariant/make_examples_core.py", line 3596, in make_examples_runner
candidates_writer.write(
AttributeError: '_GeneratorContextManager' object has no attribute 'write'

As a control, we confirmed that DeepTrio --model_type=WGS (Illumina) completes successfully using the same v1.9.0 CPU SIF on the same cluster, proving Singularity, the SIF, and the general DeepTrio pipeline work correctly.

Kindly let me know if there's something being done incorrectly on my end. Happy to share .err and .outs to help debug.

[lucasbrambrink]

Hi!

Thanks for raising this issue. I've tried to replicate the issue on GCP with running our PacBio DeepTrio case study on singularity, but this did not result in any errors raised.

Here's my worklog:

SSH into a fresh GCP instance

gcloud compute instances create "${USER}-deeptrio-gh" \
  --scopes "cloud-platform,https://www.googleapis.com/auth/gerritcodereview" \
  --image-family "ubuntu-2204-lts" \
  --image-project "ubuntu-os-cloud" \
  --machine-type "n2-standard-96" \
  --boot-disk-size "300" \
  --zone "us-west1-b"

gcloud compute ssh "${USER}-deeptrio-gh" \
  --zone "us-west1-b"

Install singularity

I installed singularity on the machine following instructions on https://sylabs.io/guides/3.7/admin-guide/installation.html#installation-on-linux

Run case study

I followed the doc here: https://github.com/google/deepvariant/blob/r1.10/docs/deeptrio-pacbio-case-study.md

Download reference

mkdir -p reference

FTPDIR=ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids

curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz | gunzip > reference/GRCh38_no_alt_analysis_set.fasta
curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai > reference/GRCh38_no_alt_analysis_set.fasta.fai

Download bam files

mkdir -p input
HTTPDIR=https://storage.googleapis.com/deepvariant/pacbio-case-study-testdata

curl ${HTTPDIR}/HG002.pfda_challenge.grch38.phased.chr20.bam > input/HG002.pfda_challenge.grch38.phased.chr20.bam
curl ${HTTPDIR}/HG002.pfda_challenge.grch38.phased.chr20.bam.bai > input/HG002.pfda_challenge.grch38.phased.chr20.bam.bai

curl ${HTTPDIR}/HG003.pfda_challenge.grch38.phased.chr20.bam > input/HG003.pfda_challenge.grch38.phased.chr20.bam
curl ${HTTPDIR}/HG003.pfda_challenge.grch38.phased.chr20.bam.bai > input/HG003.pfda_challenge.grch38.phased.chr20.bam.bai

curl ${HTTPDIR}/HG004.pfda_challenge.grch38.phased.chr20.bam > input/HG004.pfda_challenge.grch38.phased.chr20.bam
curl ${HTTPDIR}/HG004.pfda_challenge.grch38.phased.chr20.bam.bai > input/HG004.pfda_challenge.grch38.phased.chr20.bam.bai

Run DeepTrio

singularity pull docker://google/deepvariant:deeptrio-1.9.0

singularity exec \
  --bind "${PWD}/input":"/input" \
  --bind "${PWD}/output":"/output" \
  --bind "${PWD}/reference":"/reference" \
  deepvariant_deeptrio-1.9.0.sif \
  /opt/deepvariant/bin/deeptrio/run_deeptrio \
  --model_type PACBIO \
  --ref /reference/GRCh38_no_alt_analysis_set.fasta \
  --reads_child /input/HG002.pfda_challenge.grch38.phased.chr20.bam \
  --reads_parent1 /input/HG003.pfda_challenge.grch38.phased.chr20.bam \
  --reads_parent2 /input/HG004.pfda_challenge.grch38.phased.chr20.bam \
  --output_vcf_child /output/HG002.output.vcf.gz \
  --output_vcf_parent1 /output/HG003.output.vcf.gz \
  --output_vcf_parent2 /output/HG004.output.vcf.gz \
  --sample_name_child 'HG002' \
  --sample_name_parent1 'HG003' \
  --sample_name_parent2 'HG004' \
  --num_shards $(nproc) \
  --intermediate_results_dir /output/intermediate_results_dir \
  --output_gvcf_child /output/HG002.g.vcf.gz \
  --output_gvcf_parent1 /output/HG003.g.vcf.gz \
  --output_gvcf_parent2 /output/HG004.g.vcf.gz \
  --regions chr20:10,000,000-11,000,000

This led to a successful completion, without errors. This makes me think it might have to do with your specific HPC setup, but I'm not too familiar with that environment. You are welcome to try running the above case study commands on your HPC to make sure the error is reproducible on that data as well.

Finally, I can provide access to an updated docker image that might fix your issue—unfortunately I cannot confirm this since I can't reproduce the issue. I'll follow up via email!