I recently ran cactus-pangenome on 7 input assemblies. Below is the end of the job's log. It reports an error about missing VCF indexes (see "Could not retrieve index file"); however, I think it also reports successful complete (see: "Finished toil run successfully"). Does this log indicate that toil failed, but then auto-retried something? Is there anything I should confirm in the output files? Thanks for any help.
[2026-01-15T13:16:39+0000] [MainThread] [W] [toil.lib.humanize] Deprecated toil method. Please use "toil.lib.conversions.bytes2human()" instead."
[2026-01-15T13:16:39+0000] [MainThread] [I] [toil-rt] 2026-01-15 13:16:39.836599: Successfully ran: "bcftools concat -O z --threads 1 0.clip.raw.vcf.gz 1.clip.raw.vcf.gz 2.clip.raw.vcf.gz 3.clip.raw.vcf.gz 4.clip.raw.vcf.gz 5.clip.raw.vcf.gz 6.clip.raw.vcf.gz 7.clip.raw.vcf.gz 8.clip.raw.vcf.gz 9.clip.raw.vcf.gz 10.clip.raw.vcf.gz 11.clip.raw.vcf.gz 12.clip.raw.vcf.gz 13.clip.raw.vcf.gz 14.clip.raw.vcf.gz 15.clip.raw.vcf.gz 16.clip.raw.vcf.gz 17.clip.raw.vcf.gz 18.clip.raw.vcf.gz 19.clip.raw.vcf.gz 20.clip.raw.vcf.gz 21.clip.raw.vcf.gz" in 0.0594 seconds and 9.2 Mi memory
[E::idx_find_and_load] Could not retrieve index file for '/mnt/scratch/work.4494565/toilwf-5245dd59ed275063b32b96f2d0a4b80c/dda6/job/tmp31yh29s7/clip.raw.vcf.gz'
[2026-01-15T13:16:39+0000] [MainThread] [I] [toil-rt] 2026-01-15 13:16:39.843775: Running the command: "tabix -p vcf /mnt/scratch/work.4494565/toilwf-5245dd59ed275063b32b96f2d0a4b80c/dda6/job/tmp31yh29s7/clip.raw.ploidy.vcf.gz"
[2026-01-15T13:16:39+0000] [MainThread] [W] [toil.lib.humanize] Deprecated toil method. Please use "toil.lib.conversions.bytes2human()" instead."
[2026-01-15T13:16:39+0000] [MainThread] [I] [toil-rt] 2026-01-15 13:16:39.852913: Successfully ran: "tabix -p vcf /mnt/scratch/work.4494565/toilwf-5245dd59ed275063b32b96f2d0a4b80c/dda6/job/tmp31yh29s7/clip.raw.ploidy.vcf.gz" in 0.0077 seconds and 2.2 Mi memory
[2026-01-15T13:16:39+0000] [MainThread] [C] [toil.worker] Worker crashed with traceback:
Traceback (most recent call last):
File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/worker.py", line 578, in workerScript
with job._executor(
File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
next(self.gen)
File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/job.py", line 3174, in _executor
fileStore.startCommit(jobState=True)
File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/fileStores/nonCachingFileStore.py", line 250, in startCommit
list(map(self.jobStore.delete_file, self.filesToDelete))
File "/home/cactus/cactus_env/lib/python3.10/site-packages/toil/jobStores/fileJobStore.py", line 649, in delete_file
os.remove(self._get_file_path_from_id(file_id))
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/scratch/work.4494565/js2/files/for-job/kind-deconstruct/instance-xlx22fxf/file-4da9d2be17ea4661ab620ca148132a30/rm-pg2.chr18.clip.raw.vcf.gz'
[2026-01-15T13:16:39+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host fa6a6f5bad5f
<=========
[2026-01-15T13:16:39+0000] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'vcf_cat' kind-vcf_cat/instance-o_v_olvo v3 with ID kind-vcf_cat/instance-o_v_olvo to 1
[2026-01-15T13:16:39+0000] [MainThread] [I] [toil.leader] Issued job 'vcf_cat' kind-vcf_cat/instance-o_v_olvo v4 with job batch system ID: 610 and disk: 456.7 Gi, memory: 28.5 Gi, cores: 1, accelerators: [], preemptible: False
[2026-01-15T13:16:50+0000] [MainThread] [I] [toil.leader] Issued job 'export_join_wrapper' kind-export_join_wrapper/instance-t3jd76ac v1 with job batch system ID: 611 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2026-01-15T13:17:51+0000] [MainThread] [I] [toil.leader] Finished toil run successfully.
[2026-01-15T13:17:51+0000] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server.
[2026-01-15T13:17:51+0000] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread.
[2026-01-15T13:17:51+0000] [MainThread] [I] [toil.lib.history] Workflow eaf1ec3e-5a59-4e95-84e1-963fb3660a1a stopped. Success: True
[2026-01-15T13:17:53+0000] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/mnt/scratch/work.4494565/js2)
[2026-01-15T13:17:53+0000] [MainThread] [I] [toil.statsAndLogging] cactus-pangenome has finished after 31084.874818839133 seconds
Hello,
I recently ran cactus-pangenome on 7 input assemblies. Below is the end of the job's log. It reports an error about missing VCF indexes (see "Could not retrieve index file"); however, I think it also reports successful complete (see: "Finished toil run successfully"). Does this log indicate that toil failed, but then auto-retried something? Is there anything I should confirm in the output files? Thanks for any help.