Skip to content

spindleRunBE() failure prevents log file generation #147

@rountree

Description

@rountree

A proposed ci.yml refactoring splits flux-ubuntu tests into four separate jobs and runs them in parallel on separate nodes. Three jobs cover SPINDLE_DEBUG=[1-3] and the fourth does not set SPINDLE_DEBUG.

The run in question succeeded for SPINDLE_DEBUG=[1,3] and the no-debugging option, but failed on SPINDLE_DEBUG=2 with the error message:

Run docker exec node-1 bash -c 'cd Spindle-build/testsuite && SPINDLE_DEBUG=2 flux alloc --nodes=${workers} ./runTests --nodes=${workers} --tasks-per-node=3'
  docker exec node-1 bash -c 'cd Spindle-build/testsuite && SPINDLE_DEBUG=2 flux alloc --nodes=${workers} ./runTests --nodes=${workers} --tasks-per-node=3'
  shell: /usr/bin/bash -e {0}
Running: ./run_driver --dependency --push
60.237s: flux-shell[0]: stderr: spindleRunBE failed!
Error: The action 'Run spindle-flux-ubuntu-debug2 testsuite' has timed out after 7 minutes.

At that point the CI script should extract the log files, but based on the ls the script performs, no files were generated. Looking at the likely code path, LOGGING_FINI is not called during any of the error paths, so the log files were likely not flushed to disk before spindle exited.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions