Skip to content

fix: set nonzero exit code for SIGTERMException#21623

Merged
justusschock merged 6 commits intoLightning-AI:masterfrom
javierdejesusda:fix/19916-sigterm-exit-code
Mar 30, 2026
Merged

fix: set nonzero exit code for SIGTERMException#21623
justusschock merged 6 commits intoLightning-AI:masterfrom
javierdejesusda:fix/19916-sigterm-exit-code

Conversation

@javierdejesusda
Copy link
Copy Markdown
Contributor

@javierdejesusda javierdejesusda commented Mar 28, 2026

What does this PR do?

Fixes #19916

SIGTERMException inherits from SystemExit but was raised without arguments, so SystemExit.code defaulted to None — which Python converts to exit code 0. This made it impossible for wrapper scripts or job schedulers (e.g. SLURM) to distinguish a SIGTERM kill from a clean exit.

The fix overrides __init__ to pass 143 (128 + signal 15) to SystemExit, following the standard Unix exit code convention for signal-terminated processes.

Before submitting


📚 Documentation preview 📚: https://pytorch-lightning--21623.org.readthedocs.build/en/21623/

`SIGTERMException(SystemExit)` was raised without arguments, causing
`SystemExit.code` to default to `None`, which Python converts to exit
code 0. Scripts monitoring the training process exit code could not
distinguish a SIGTERM termination from a successful exit.

Override `__init__` to pass 143 (128 + SIGTERM signal number 15) to
`SystemExit`, following the standard Unix convention.

Fixes Lightning-AI#19916
@github-actions github-actions bot added the pl Generic label for PyTorch Lightning package label Mar 28, 2026
Comment thread src/lightning/pytorch/utilities/exceptions.py Outdated
Comment thread src/lightning/pytorch/CHANGELOG.md Outdated
Comment thread src/lightning/pytorch/utilities/exceptions.py Outdated
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79%. Comparing base (648ec13) to head (b4f0e88).
⚠️ Report is 4 commits behind head on master.
✅ All tests successful. No failed tests found.

❗ There is a different number of reports uploaded between BASE (648ec13) and HEAD (b4f0e88). Click for more details.

HEAD has 168 uploads less than BASE
Flag BASE (648ec13) HEAD (b4f0e88)
cpu 84 42
python 6 3
lightning_fabric 27 0
pytest 42 0
python3.12 24 12
lightning 30 15
python3.11 12 6
python3.13 18 9
python3.12.7 18 9
python3.10 6 3
Additional details and impacted files
@@            Coverage Diff            @@
##           master   #21623     +/-   ##
=========================================
- Coverage      87%      79%     -8%     
=========================================
  Files         270      267      -3     
  Lines       23895    23839     -56     
=========================================
- Hits        20675    18765   -1910     
- Misses       3220     5074   +1854     

@justusschock justusschock merged commit f4027f7 into Lightning-AI:master Mar 30, 2026
100 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pl Generic label for PyTorch Lightning package

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KeyboardInterrupt raises an exception which results in a zero exit code

3 participants