DeepVariant segmentation fault

[adthrasher]

Have you checked the FAQ? https://github.com/google/deepvariant/blob/r1.10/docs/FAQ.md:
Yes

Describe the issue:
I am encountering a segmentation fault when running DeepVariant. It seems to be consistent across samples. I've attached the relevant log segment below, but I can attach more if needed.

I'm running in LSF with Apptainer. I have allocated 62GB of RAM, and the process used a max of 36GB of RAM. I'm rewriting /tmp in the Apptainer call to a location with sufficient disk space.

reference_name: "KI270519.1"
start: 138042
 is [[0]], which is invalid.
I0305 21:06:32.871657 23456244236288 postprocess_variants.py:1603] VCF and gVCF creation took 3.5776108264923097 minutes.
I0305 21:08:13.537272 23456244236288 postprocess_variants.py:1603] VCF and gVCF creation took 5.192403384049734 minutes.
I0305 21:08:26.462935 23456244236288 postprocess_variants.py:1603] VCF and gVCF creation took 5.370254564285278 minutes.
I0305 21:09:24.510577 23456244236288 postprocess_variants.py:1603] VCF and gVCF creation took 6.329206657409668 minutes.
I0305 21:09:26.545674 23456244236288 postprocess_variants.py:1603] VCF and gVCF creation took 6.357302673657736 minutes.
I0305 21:10:13.330816 23456244236288 postprocess_variants.py:1603] VCF and gVCF creation took 7.099282324314117 minutes.
I0305 21:10:15.430971 23456244236288 postprocess_variants.py:1603] VCF and gVCF creation took 7.0951718171437586 minutes.
Checking the headers of 8 files.
[E::bcf_hdr_read] Input is not detected as bcf or vcf format
[E::naive_concat_check_headers] Failed to parse header: /tmp/tmp0z2qucay.gz

Fatal Python error: Segmentation fault

Current thread 0x00001555551b0000 (most recent call first):
  File "/tmp/Bazel.runfiles_gdi8y2hh/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1316 in _concat_vcf
  File "/tmp/Bazel.runfiles_gdi8y2hh/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1778 in main
  File "/tmp/Bazel.runfiles_gdi8y2hh/runfiles/absl_py/absl/app.py", line 258 in _run_main
  File "/tmp/Bazel.runfiles_gdi8y2hh/runfiles/absl_py/absl/app.py", line 312 in run
  File "/tmp/Bazel.runfiles_gdi8y2hh/runfiles/com_google_deepvariant/deepvariant/postprocess_variants.py", line 1804 in <module>

I can see that several of the temporary files are empty.

drwxrwx---+ 2 athrashe Domain Users      4096 Mar  5 20:56 __pycache__
-rwxrwx---+ 1 athrashe Domain Users         0 Mar  5 21:02 tmp0z2qucay.gz
-rwxrwx---+ 1 athrashe Domain Users  31747312 Mar  5 21:09 tmp1clnadfn.gz
-rwxrwx---+ 1 athrashe Domain Users         0 Mar  5 21:02 tmp2cgg_e61.gz
-rwxrwx---+ 1 athrashe Domain Users 163271675 Mar  5 21:10 tmp65evb3vd.gz
-rwxrwx---+ 1 athrashe Domain Users  24929580 Mar  5 21:08 tmp8rp4z179.gz
-rwxrwx---+ 1 athrashe Domain Users 107367872 Mar  5 21:08 tmp9bsbk_jj.gz
-rwxrwx---+ 1 athrashe Domain Users 150461173 Mar  5 21:09 tmp9lmu0tzi.gz
-rwxrwx---+ 1 athrashe Domain Users 148044961 Mar  5 21:09 tmp9uu0_ojx.gz
-rwxrwx---+ 1 athrashe Domain Users  35569344 Mar  5 21:10 tmpem20u1rr.gz
-rwxrwx---+ 1 athrashe Domain Users 176980341 Mar  5 21:10 tmplqj087hb.gz
-rwxrwx---+ 1 athrashe Domain Users  37789447 Mar  5 21:10 tmpp0qnv1he.gz
-rwxrwx---+ 1 athrashe Domain Users  58673361 Mar  5 21:06 tmppgmaga5i.gz
-rwxrwx---+ 1 athrashe Domain Users  31354921 Mar  5 21:09 tmppz804kk_.gz
-rwxrwx---+ 1 athrashe Domain Users 119155965 Mar  5 21:08 tmpt96z1ia0.gz
-rwxrwx---+ 1 athrashe Domain Users  12845221 Mar  5 21:06 tmpywd3qhng.gz
-rwxrwx---+ 1 athrashe Domain Users  24290463 Mar  5 21:08 tmpzt25eer1.gz

Setup

• Operating system:
• DeepVariant version: google/deepvariant:1.9.0-gpu
• Installation method (Docker, built from source, etc.): Docker via Apptainer
• Type of data: (sequencing instrument, reference genome, anything special that is unlike the case studies?) I am using the WGS data for HG002 from Genome in a Bottle. It is aligned against GRCh38.p14.genome.fa with minimap2

Steps to reproduce:

• Command:
• Error trace: (if applicable)

Does the quick start test work on your system?
Please test with https://github.com/google/deepvariant/blob/r0.10/docs/deepvariant-quick-start.md.
Is there any way to reproduce the issue by using the quick start?

Any additional context:

[kishwarshafin]

@adthrasher can you please share the command you are running and the entire log? Also, would you be able to upgrade to 1.10.0 and see if the error still happens?