Segmentation fault in make_examples when running DeepVariant 1.10.0 with PacBio HiFi BAM

[ChuanzhengWei]

Have you checked the FAQ? https://github.com/google/deepvariant/blob/r1.10/docs/FAQ.md:
Yes
Describe the issue:
I encountered a segmentation fault during the make_examples stage when running DeepVariant v1.10.0 with PacBio HiFi data.

The pipeline starts normally and several shards complete successfully, but one shard crashes with a segmentation fault in make_examples_core.py during writes_examples_in_region.

The crash happens mid-execution, not at startup.

Setup

• Operating system: Linux (HPC cluster)
• DeepVariant version: 1.10.0
• Installation method (Docker, built from source, etc.): Singularity container (deepvariant-1.10.0.sif)
• Type of data: (sequencing instrument, reference genome, anything special that is unlike the case studies?)
  PacBio HiFi reads
  Reads aligned with minimap2 (map-hifi preset ~50×)
  Plant genome reference (~700 Mb genome)
  Input BAM is sorted and indexed, and contig names match the reference genome.

Steps to reproduce:

• Command:

 prefix=test
threads=24

singularity exec \
  -B ${PWD}/${prefix}_input:/input \
  -B ${PWD}/${prefix}_output:/output \
  deepvariant-1.10.0.sif \
  /opt/deepvariant/bin/run_deepvariant \
  --sample_name=${prefix} \
  --model_type=PACBIO \
  --ref=/input/reference.fasta \
  --reads=/input/${prefix}.bam \
  --output_vcf=/output/${prefix}.dp.vcf.gz \
  --output_gvcf=/output/${prefix}.dp.gvcf.gz \
  --intermediate_results_dir=/output/deepvariant_tmp \
  --num_shards=${threads} \
  --logging_dir=/output/logs \
  --vcf_stats_report=true

• Error trace: (if applicable)

Fatal Python error: Segmentation fault

File ".../make_examples_core.py", line 1991 in writes_examples_in_region
File ".../make_examples_core.py", line 3651 in make_examples_runner
File ".../make_examples.py", line 229 in main

Any additional context:
I have tried modifying the num_shards parameter and also allocating more memory, but the run still fails.
I have attached the full runtime log: make_examples.log.

make_examples.log

[ChuanzhengWei]

The quick start example from runs successfully on my system:
[https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-quick-start.md]

I previously used DeepVariant version 1.6 about two years ago, and the same dataset ran successfully with that version.

[pichuan]

Hi @ChuanzhengWei,

Looking at your make_examples.log, it seems like it was --task 9 that crashed. To narrow down the issue, can you rerun:

singularity exec \
  -B ${PWD}/${prefix}_input:/input \
  -B ${PWD}/${prefix}_output:/output \
  deepvariant-1.10.0.sif \
  /opt/deepvariant/bin/make_examples \
  --mode calling \
  --ref /input/reference.fasta \
  --reads /input/test.bam \
  --examples '/output/deepvariant_tmp/make_examples.tfrecord@24.gz' \
  --checkpoint /opt/models/pacbio \
  --checkpoint_json /opt/models/pacbio/model.example_info.json \
  --output_local_read_phasing '/output/deepvariant_tmp/read-phasing_debug@24.tsv' \
  --output_phase_info \
  --call_small_model_examples \
  --gvcf '/output/deepvariant_tmp/gvcf.tfrecord@24.gz' \
  --sample_name test \
  --task 9 \
  --output_debug_info

This should be the same as what your command was running, but narrow it down to just the make_examples binary, and just --task 9. Adding --output_debug_info should enable this extra logging code:

    if options.output_debug_info:
      logging_with_options(
          options,
          'Processing %s:%d-%d'
          % (region.reference_name, region.start, region.end),
      )

Then, hopefully that helps narrow down to where it crashes.

If you're ok with extracting a small BAM snippets around that region, check whether that still reproduces your errors, and share the small BAM to us, it will be very useful for us to reproduce the issue and debug! (You can email to me at pichuan@google.com)

But, if you can't share the data for reproducibility, we can continue to discuss here and see if we can help you identify the issue.

[ChuanzhengWei]

Hi @pichuan ,

Based on the debug log, the crash occurs right after processing the following region:

Task 9/24: Processing Chr09:30775000-30800000
Task 9/24: Processing Chr09:31375000-31400000
Fatal Python error: Segmentation fault

I extracted a BAM snippet covering Chr09:31375000-31400000 and prepared a compressed archive (test.tar.gz) containing the BAM file, its index, and a screenshot of the read alignment in this region.

The archive has been sent to your email for debugging.

Thanks again for your help.

[kishwarshafin]

@ChuanzhengWei , we were able to reproduce this.

Did you try any of the intermediate versions of deepvariant with your data? Or you just upgraded from 1.6.0 to 1.10.0 recently? Just trying to see if you had this issue with previous versions.

[ChuanzhengWei]

@kishwarshafin , After encountering the failure with v1.10.0, I reran the same dataset using DeepVariant 1.6.0 (I had previously downloaded that container). The run completed successfully and did not produce any errors.

Therefore, this issue seems to occur with v1.10.0, while v1.6.0 works correctly for my dataset.

[pichuan]

Hi @ChuanzhengWei ,

We have identified the issue, and made a fix.
We're still discussing on how to address this issue with the release version, but right now I built a Docker version for you to test:

Please help us test this Docker: gcr.io/deepvariant-docker/deepvariant:gh1060

Here is how I tested on your data:

DOCKER="gcr.io/deepvariant-docker/deepvariant:gh1060"

docker run \
  -v /tmp/gh1060:/data \
  ${DOCKER} \
  bin/make_examples \
  --mode calling \
  --ref /data/Chr09.fasta \
  --reads /data/subset.bam \
  --examples '/data/make_examples.tfrecord.gz' \
  --checkpoint /opt/models/pacbio \
  --checkpoint_json /opt/models/pacbio/model.example_info.json \
  --trained_small_model_path '/opt/smallmodels/pacbio/model.keras' \
  --output_local_read_phasing '/data/read-phasing_debug.tsv' \
  --output_phase_info \
  --call_small_model_examples \
  --gvcf '/data/gvcf.tfrecord.gz' \
  --sample_name test \
  --regions "Chr09:31375001-31400000"

This worked for me:

I0318 02:53:09.841561 139707487002624 make_examples_core.py:384] Summary stats:
        2554 candidate variants found
        1804 examples written
         244 candidate variants phased
         956 small model examples called

And, the same command was failing with:

DOCKER="google/deepvariant:1.10.0"

---

In case you're curious:

The fix we have done is patching ssw.c with this change to avoid overflow:

--- a/src/ssw.c
+++ b/src/ssw.c
@@ -602,7 +602,8 @@
 	int32_t i, j, e, f, temp1, temp2, s = 16, s1 = 8, l, max = 0, len;
 	int64_t s2 = 1024;
 	char op, prev_op;
-	int32_t width, width_d, *h_b, *e_b, *h_c;
+	int32_t width, *h_b, *e_b, *h_c;
+	int64_t width_d;
 	int8_t *direction, *direction_line;
     len = refLen > readLen ? refLen : readLen;
 	cigar* result = (cigar*)malloc(sizeof(cigar));
@@ -620,7 +621,7 @@
 			e_b = (int32_t*)realloc(e_b, s1 * sizeof(int32_t));
 			h_c = (int32_t*)realloc(h_c, s1 * sizeof(int32_t));
 		}
-		while (width_d * readLen * 3 >= s2) {   // width_d*readLen* overflow before s2
+		while (width_d * (int64_t)readLen * 3 >= s2) {
 			++s2;
 			kroundup32(s2);
 			direction = (int8_t*)realloc(direction, s2 * sizeof(int8_t));
@@ -633,7 +634,7 @@
 			j = i + band_width; end = end < j ? end : j; // band end
 			edge = end + 1 < width - 1 ? end + 1 : width - 1;
 			f = h_b[0] = e_b[0] = h_b[edge] = e_b[edge] = h_c[0] = 0;
-			direction_line = direction + width_d * i * 3;
+			direction_line = direction + width_d * (int64_t)i * 3;
 
 			for (j = beg; LIKELY(j <= end); j ++) {
 				int32_t b, e1, f1, d, de, df, dh;

---

I will now assign this to @kishwarshafin so he can follow up later.

[ChuanzhengWei]

Hi, @pichuan , thank you very much for the fix.

I tested the Docker image gcr.io/deepvariant-docker/deepvariant:gh1060 on my dataset, and it completed successfully. I was able to obtain the output VCF without the previous segmentation fault.

I also noticed that the variant calls from v1.6 and this fixed build differ quite a lot on my dataset.I have attached summary statistics generated from the two versions for comparison (v1.6.dp.visual_report.html and v1.10.dp.visual_report.html). In this situation, which version would you recommend I trust more for downstream analysis?
v1.6.dp.visual_report.html
v1.10.dp.visual_report.html

In addition, could you please provide some guidance on the recommended workflow for multi-sample gVCF merging and filtering with DeepVariant outputs?

Thank you again for your help.

[pichuan]

Hi, @pichuan , thank you very much for the fix.

I tested the Docker image gcr.io/deepvariant-docker/deepvariant:gh1060 on my dataset, and it completed successfully. I was able to obtain the output VCF without the previous segmentation fault.

I also noticed that the variant calls from v1.6 and this fixed build differ quite a lot on my dataset.I have attached summary statistics generated from the two versions for comparison (v1.6.dp.visual_report.html and v1.10.dp.visual_report.html). In this situation, which version would you recommend I trust more for downstream analysis? v1.6.dp.visual_report.html v1.10.dp.visual_report.html

@ChuanzhengWei Can you run 1.10 with --disable_small_model and share the visual_report.html as well?

Given that you're on a non-human data, I'd be. curious to see how different the plot looks like without small model.

In addition, could you please provide some guidance on the recommended workflow for multi-sample gVCF merging and filtering with DeepVariant outputs?

Please see https://github.com/google/deepvariant/blob/r1.10/docs/trio-merge-case-study.md as an example. Even though the example is for trio, you can do this for multi-sample gVCF merging. A few recommendations:

• Use DeepVariant_unfiltered.
• To make the compute more manageable, split your data by chromosome first.

For your reference, I have actually run GLnexus on the 1000 Genomes dataset (more than 3000 samples), so it should be pretty scalable. Happy to help if you have more questions, and curious to hear about your research findings if you're willing to share over email :)

Thank you again for your help.

[ChuanzhengWei]

Thanks for the suggestion. I will rerun DeepVariant with --disable_small_model and share the resulting visual_report.html for comparison.

Thank you also for the guidance on multi-sample gVCF merging.

I appreciate your help and the quick fix. I will update the issue once I have the new results. ^_^