segmentation violation error at template matching step

[madelinerollins]

Hi - I am working through the PyTom tutorial but am having some issues with errors for the batch template matching section. I set up the job as described in the instructions, but receive this repeating segmentation violation error in the terminal:

I am running the GUI through an interactive job, so my jobs a ran locally and not using a queuing system. I'm also unsure if this is a hardware issue or something else. Do you have any suggestions on how to fix this error?

[McHaillet]

Hi madeline,
I dont immediately recognize your issue, could you provide some more info on the command that you ran?
Also, do you have a more extensive error message in the terminal than the image you currently show?

@sroet , does this look familiar to you?

Best, Marten

[sroet]

@sroet , does this look familiar to you?

Not really, only thing I can see from this image is the line:

The user requested CUDA support with the --mca mpi_cuda_support 1 flag but the library was not compiled with support.

But without more of a trace I have no clue where that error comes from (not even it if is our code or one of our dependencies). A quick google suggest a that openmpi might have been compiled without CUDA awareness

[madelinerollins]

Yes, of course - my apologies for not including more information in the original post.
I am at the Template Matching step (Section 6.1) in the tutorial. I would like to do template matching only for the first tomogram (000) in the practice dataset.

Once I select the tomograms, template, and mask for this step, this is the output I get in the terminal:
`multiple template matching job-sub

multiple template matching job-submissions
/beagle3/bkeenan/madeline/pytom_practice/tomo/03_Tomographic_Reconstruction/tomogram_000/reconstruction/WBP
/beagle3/bkeenan/madeline/pytom_practice/tomo/03_Tomographic_Reconstruction/tomogram_001/reconstruction/WBP
/beagle3/bkeenan/madeline/pytom_practice/tomo/03_Tomographic_Reconstruction/tomogram_002/reconstruction/WBP
/beagle3/bkeenan/madeline/pytom_practice/tomo/03_Tomographic_Reconstruction/tomogram_003/reconstruction/WBP
QLayout: Attempting to add QLayout "" to SimpleTable "", which already has a layout
Generating PB v03_BatchTemplateMatch_ExecutePushButton

Then I finish adding the parameters in the GUI. I am running the template matching on GPUs and using a GPU index of 0.

Once I press run, this is the complete error message that I get in the terminal:

starting 1 jobs on device 0
submitting threaded job
[0]
[1,0]<stderr>:--------------------------------------------------------------------------
[1,0]<stderr>:The user requested CUDA support with the --mca mpi_cuda_support 1 flag
[1,0]<stderr>:but the library was not compiled with any support.
[1,0]<stderr>:--------------------------------------------------------------------------
[1,0]<stderr>:[beagle3-0010:2608055] *** Process received signal ***
[1,0]<stderr>:[beagle3-0010:2608055] Signal: Segmentation fault (11)
[1,0]<stderr>:[beagle3-0010:2608055] Signal code: Address not mapped (1)
[1,0]<stderr>:[beagle3-0010:2608055] Failing at address: (nil)
[1,0]<stderr>:[beagle3-0010:2608055] [ 0] /lib64/libpthread.so.0(+0x12b20)[0x7f8fbd3e9b20]
[1,0]<stderr>:[beagle3-0010:2608055] *** End of error message ***
[1,0]<stderr>:/programs/x86_64-linux/pytom/1.0a/pytom/bin/pytom: line 21: 2608055 Segmentation fault      (core dumped) python3.8 -O $*
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[53403,1],0]
  Exit code:    139
--------------------------------------------------------------------------
[1,0]<stderr>:--------------------------------------------------------------------------
[1,0]<stderr>:The user requested CUDA support with the --mca mpi_cuda_support 1 flag
[1,0]<stderr>:but the library was not compiled with any support.
[1,0]<stderr>:--------------------------------------------------------------------------
[1,0]<stderr>:[beagle3-0010:2608226] *** Process received signal ***
[1,0]<stderr>:[beagle3-0010:2608226] Signal: Segmentation fault (11)
[1,0]<stderr>:[beagle3-0010:2608226] Signal code: Address not mapped (1)
[1,0]<stderr>:[beagle3-0010:2608226] Failing at address: (nil)
[1,0]<stderr>:[beagle3-0010:2608226] [ 0] [1,0]<stderr>:/lib64/libpthread.so.0(+0x12b20)[0x7f988e209b20]
[1,0]<stderr>:[beagle3-0010:2608226] *** End of error message ***
[1,0]<stderr>:/programs/x86_64-linux/pytom/1.0a/pytom/bin/pytom: line 21: 2608226 Segmentation fault      (core dumped) python3.8 -O $*
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[55142,1],0]
  Exit code:    139
--------------------------------------------------------------------------

Since I'm having this issue with the Batch Template matching, is this is in any way linked to the bug discussed is Issue #7 ?

[sroet]

@madelinerollins , do you mind running: ompi_info --parsable --all | grep mpi_built_with_cuda_support for me in the shell you normally start your pytom process from and post the results?
That should tell us if you are using an OpenMPI version that is CUDA aware

[madelinerollins]

@sroet Sure - I'm running PyTom through SBGrid, so I sourced SBGrid first:

source /programs/sbgrid.shrc

After that, I ran the ompi_info --parsable -all | grep mpi_built_with_cuda_support command. Here is the output:

mca:opal:base:param:opal_built_with_cuda_support:synonym:name:mpi_built_with_cuda_support
mca:mpi:base:param:mpi_built_with_cuda_support:value:false
mca:mpi:base:param:mpi_built_with_cuda_support:source:default
mca:mpi:base:param:mpi_built_with_cuda_support:status:read-only
mca:mpi:base:param:mpi_built_with_cuda_support:level:4
mca:mpi:base:param:mpi_built_with_cuda_support:help:Whether CUDA GPU buffer support is built into library or not
mca:mpi:base:param:mpi_built_with_cuda_support:enumerator:value:0:false
mca:mpi:base:param:mpi_built_with_cuda_support:enumerator:value:1:true
mca:mpi:base:param:mpi_built_with_cuda_support:deprecated:no
mca:mpi:base:param:mpi_built_with_cuda_support:type:bool
mca:mpi:base:param:mpi_built_with_cuda_support:synonym_of:name:opal_built_with_cuda_support
mca:mpi:base:param:mpi_built_with_cuda_support:disabled:false

[sroet]

Thanks!

The important line is
mca:mpi:base:param:mpi_built_with_cuda_support:value:false

This means that you are running on a system where the openmpi is not compiled for being CUDA aware.

Assuming this is a managed system where you did not install openmpi yourself, please ask your system administrator to provide an openmpi version that is CUDA-aware.

Otherwise please recompile openmpi with these options

[madelinerollins]

Thank you! I passed this information along to my system administrator and he first asked me to try running module load openmpi-cuda-aware which is built with CUDA-aware support. However, I still getmca:mpi:base:param:mpi_built_with_cuda_support:value:false after I source SBGrid.

He found that when we run source /programs/sbgrid.shrc, it loads up the mpiexec version under the sbgrid bundle:
/programs/x86_64-linux/system/sbgrid_bin/mpiexec which was built without cuda support.

Even we run module load openmpi-cuda-aware after this source command (to override the path to the mpiexec with cuda support), the pytom command /programs/x86_64-linux/pytom/1.0a/pytom-1.0a/pytom/bin/pytom has the line:
source /programs/x86_64-linux/pytom/1.0a/pytom-1.0a/pytom/bin/paths.sh which points mpiexec back to /programs/x86_64-linux/system/sbgrid_bin/mpiexec (without cuda support).

Would be possible to get the full pytom command with all of the arguments to add some lines to set --mca to not use CUDA support for the MPI runs? Or would it be possible to remove the --mca cuda support part when running pytom?

[sroet]

Hey @madelinerollins,

That is not part of the pytom command that we currently generate during compile time, that is just:

#!/usr/bin/env bash
cat {sys.prefix}/pytom_data/LICENSE.txt

FID=0
export PYTOM_GPU=-1

for a in $*
do
    if [ $FID -eq 1 ]; then
        FID=0
        export PYTOM_GPU="$a"
    fi

    if [ "$a" = '-g' ] || [ "$a" = '--gpuID' ]; then
        export PYTOM_GPU=1
        FID=1
    fi
done

{sys.executable} -O $*

Where sys.prefix and sys.executable are replaced by whatever python environment and python executable you used during installation.

However, it seems like SBGrid is installing an old PyTom version, as the paths.sh file has not been part of PyTom since Feb 8th. Please ask SBGrid to deploy the current PyTom version (1.1). As far as I am aware, we are not the ones maintaining the SBGrid version of our PyTom.