Add Giraffe/DeepVariant to vg CI

[adamnovak]

Changelog Entry

To be copied to the draft changelog by merger:

• Giraffe/DeepVariant is now under CI test.

Description

This tests short-read Giraffe on Illumina reads on BRCA1, and long read Giraffe on HiFi reads on all of chr21, as part of the vg CI process. It adds an additional test-running CI job to each run; we might want to delete some of the existing CI tests for map and call to trim it down again.

This isn't a good enough test to really make sure we won't have accuracy regressions at full-genome scale. And before merging we should make sure that the accuracy values we're getting for the chr21 test are plausibly good enough, because I run out of home directory quota trying to run the test through in my vg clone on Mustard.

[adamnovak]

This can't pass CI because the Python in quay.io/vgteam/vg_ci_prebake is too old for current Toil.

If I rebuild the image, since I upgraded the base dind image, it includes a Docker that is too new to pull very old Docker images:

docker run --privileged -ti --rm quay.io/vgteam/vg_ci_prebake:latest docker pull quay.io/glennhickey/pigz:2.3.1
...
2.3.1: Pulling from glennhickey/pigz
Docker Image Format v1 and Docker Image manifest version 2, schema 1 support has been removed. Suggest the author of quay.io/glennhickey/pigz:2.3.1 to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/

So in order to upgrade the CI system base image I will need to go into toil-vg and find and replace all the images it wants to use (at least in what's used in vg CI) that are too old, probably finding replacements for some that are not available as current-format images. And then I'll need to update the CI base image.

I think this might involve breaking CI on all the other branches (unless I update in a tag, merge, replace the latest tag, change back to latest, and merge again). So I'm reluctant to do it immediately before curtailment.

I've squirreled away the current CI base image as quay.io/vgteam/vg_ci_prebake:working-2023 in case something goes wrong.

[adamnovak]

After merging this, I need to push up the Dockerfile change to vg_ci_prebake (just changing the Python dev package name there as well), let :latest build, and change the CI image back to :latest.