Skip to content

Improve configuration validation around autoscaling#5525

Merged
adamnovak merged 14 commits into
masterfrom
issues/5380-improve-option-validation
May 28, 2026
Merged

Improve configuration validation around autoscaling#5525
adamnovak merged 14 commits into
masterfrom
issues/5380-improve-option-validation

Conversation

@adamnovak

Copy link
Copy Markdown
Member

This will fix #5380 by preventing the workflow from starting up at all when given a provisioner and a batch system that shouldn't be used together.

It also revises some of the logging and error reporting to make the resulting message more human-readable.

I have several levels of errors now, some of which one shouldn't be able to reach unless one somehow sneaks past the previous levels. But I suppose it's possible someone is doing weird things in a Python workflow?

Changelog Entry

To be copied to the draft changelog by merger:

  • Toil will no longer throw away workflow output when finishing a workflow that managed to start with both in-workflow autoscaling and a non-scalable batch system.
  • Toil will now catch and prohibit starting a workflow with in-workflow autoscaling and a non-scalable batch system.
  • CWL and WDL interpreters now set up fancy logging much earlier in startup
  • Configuration inconsistency problems will now be reported as error messages rather than stack traces
  • Kubernetes batch system will no longer log a method instead of a job count
  • toil launch-cluster will no longer try and launch a cluster without a --keyPairName.

Reviewer Checklist

  • Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
    • If it is coming from an external repo, make sure to pull it in for CI with:
      contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
      
    • If there is no associated issue, create one.
  • Read through the code changes. Make sure that it doesn't have:
    • Addition of trailing whitespace.
    • New variable or member names in camelCase that want to be in snake_case.
    • New functions without type hints.
    • New functions or classes without informative docstrings.
    • Changes to semantics not reflected in the relevant docstrings.
    • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
    • New features without tests.
  • Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
  • Finish the review with an overall description of your opinion.

Merger Checklist

  • Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
  • Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
  • Merge with the Github "Squash and merge" feature.
    • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
  • Copy its recommended changelog entry to the Draft Changelog.
  • Append the issue number in parentheses to the changelog entry.

@adamnovak adamnovak merged commit 4c4568f into master May 28, 2026
3 checks passed
@adamnovak adamnovak deleted the issues/5380-improve-option-validation branch May 28, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No outputs in s3 bucket after toil-cwl-runner k8s run, with error non-scalable batch system

1 participant