pyhermes module not found

[tospines]

Hello!
Python is having troubles to find the pyhermes module, even though I am importing pyhermes in the build folder, where the pyhermes.cpython-38-x86_64-linux-gnu.so is located. I have tried adding explicitly the build path to the PYTHONPATH variable and to the python sys.path(), but for both cases I get the same error:
>>> import pyhermes
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'pyhermes'

I am installing Hermes in a cluster of computers, within an environment where I have full control, and there was no problem in the installation process.

Any ideas on how to solve this?
Thanks!

[adundovi]

Hi @tospines, do you use virtualenv to install Hermes on Linux as described here?

[tospines]

Yes, I am totally following the same steps as described there. Curiously I do not get any error or strange warning in the installation step, but then, it does not find the pyhermes package.

[adundovi]

Here are the steps which should be checked for the Python binding to work:

1. During the cmake phase one should ensure that the cmake outputs the following (exact versions are not important):

• Found PythonInterp: ~/.virtualenvs/hermes/bin/python (found version "3.10.2")
• Found PythonLibs: /usr/lib64/libpython3.10.so (found version "3.10.2")
• Python: Found!
  -- Version: 3.10/310
  -- Executeable: ~/.virtualenvs/hermes/bin/python
  -- Include: /usr/include/python3.10
  -- Library: /usr/lib64/libpython3.10.so
  -- Site-package directory: ~/.virtualenvs/hermes/lib64/python3.10/site-packages

2. The build phase should finish successfully, of course, and it has to contain the following line:

• [100%] Linking CXX shared module pyhermes.cpython-310-x86_64-linux-gnu.so

3. Testing: with make test one can check all C++ parts of the code where all tests should pass, however, to test the Python binding one can run python interpreter within the build directory where pyhermes.cpython-310-x86_64-linux-gnu.so is present - and there one should be able to import pyhermes:

$ python
Python 3.10.2 (main, Jan 17 2022, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyhermes
>>> dir(pyhermes)
['BremsstrahlungIntegrator', 'CircularWindow', 'DarkMatterIntegrator',  (...), 'units']

4. If the previous step works, one can finally execute make install which will copy all compiled binaries and resource files to ~/.virtualenvs/hermes/ and one can use pyhermes module from any path as long as the given virtualenv is active.

This has been trying many times on various machines, so please double-check all of these and report back if you have some strange outputs or errors.

[tospines]

I have followed those steps and found a couple of differences with respect what you described:

• In the cmake step, I do not see any reference to "Found PythonInterp" or "Found PythonLibs",. This is what I got:

  CMAKE_PREFIX_PATH=$HERMES_DIR cmake -DCMAKE_INSTALL_PREFIX=$HERMES_DIR -DENABLE_TESTING=On ..
  -- Build Type: Release
  -- Use --as-needed linker flags!
  -- Use absolute RPATH /home/x_pedde/.virtualenvs/hermes/lib
  -- Downloading data file from heat.gssi.it ~ 130 MB
  -- Extracting data file
  -- FFTW3 with single precision (FFTW3F): Found!
  -- Include: /usr/include
  -- Library: /usr/lib64/libfftw3f.so
  -- found version string 3.49
  -- Found CFITSIO 3.49: /home/x_pedde/cfitsio/cfitsio-3.49/libcfitsio.so
  -- Hermes version: v1.0.1-29-g414f9bf 414f9bf refs/heads/master
  Units dimensionality check enabled.
  -- Python: Found!
  -- Version: 3.6/36
  -- Executeable: /home/x_pedde/.virtualenvs/hermes/bin/python
  -- Include: /software/sse/easybuild/prefix/software/Python/3.6.7-foss-2018a-nsc1/include/python3.6m
  -- Library: /usr/lib64/libpython3.6m.so
  -- Site-package directory: /home/x_pedde/.virtualenvs/hermes/lib/python3.6/site-packages
  -- pybind11 v2.6.3 dev1
  -- Doxygen and Sphinx not found: cannot build documentation
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /proj/dmsearches/users/x_pedde/Hermes/build

Then, after the make -j step, I got the "[100%] Linking CXX shared module pyhermes.cpython-310-x86_64-linux-gnu.so" message as you said. Unfortunately, once I run the make test I get an error at the end:
--The following tests FAILED:
26 - testPixelTimes (Failed)

I am attaching to this message the LastTest.log in case it can help.
LastTest.log
LastTestsFailed.log

Finally, I tried to import pyhermes but it did not work, as earlier.

Should I link the Python interpreter in the cmake step or something like this?

[adundovi]

26 - testPixelTimes (Failed)

OK, this test can fail as it only measures performance, i.e., pixel calculation times for various integrators to check for performance regressions when introducing new features or changing the code. These times are somehow manually picked by me to match some reasonably recent desktop hardware, but on laptops, these times could be inadequate.

Regarding your issue of loading pyhermes in Python - it looks to me that you have several Python versions installed on your system and with one version your pyhermes module is compiled, while you run the interpreter where you try to load it with the other one.

To understand which Python interpreter you run, please give me the output of: $ python -V. Have you activated a virtual environment before running cmake? You could manually specify the Python paths (PYTHON_INCLUDE_PATH and PYTHON_LIBRARIES) with -D flag in cmake.

[tospines]

Yes, you are right, there are many Python versions installed here. The one I have used for installing and, then, for trying to import pyhermes is Python 3.6.7 (this is the output of python -V).
As I am working in a cluster of computers I first activate an environment and, then, the Hermes environment. Once I am within the HERMES environment, I run the cmake.

[adundovi]

OK, but you mentioned in your first message a file named pyhermes.cpython-38-x86_64-linux-gnu.so from what I read that Python 3.8 (38) is found during the cmake phase and this will certainly not work with Python 3.6.7. So, either use Python 3.8 within your virtual environment or instruct cmake to find 3.6.7.

[tospines]

Yo were right, it seemed that the python version that I had changed when I entered in my environment. Solved now!
Thank you very much.

[fayez0]

Hello!

I am facing the same problem here. I've installed Hermes on Mac OS. I used a virtualenv to install and run it, but I still have the same problem. When I run cmak test, I get this error:

The following tests FAILED:
9 - testIntegrationMethods (Subprocess aborted)
Errors while running CTest
Output from these tests are in: /Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [test] Error 8
(base) fayezabu-ajamieh@Fayezs-MacBook-Pro build % --rerun-failed --output-on-failure
zsh: command not found: --rerun-failed

[adundovi]

Hi @fayez0,
please add the last log file (/Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build/Testing/Temporary/LastTest.log) to provide more details.
//Andrej

[fayez0]

Here it is
LastTest.log

[fayez0]

Any idea why I keep getting this error? It seems the code cannot find pyhermes

[fayez0]

Here is the portion of the test that has the error:

9/28 Testing: testIntegrationMethods
9/28 Test: testIntegrationMethods
Command: "/Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build/testIntegrationMethods"
Directory: /Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build
"testIntegrationMethods" start time: Apr 03 10:32 IST
Output:

dyld[20643]: symbol not found in flat namespace '_cblas_caxpy'

Test time = 0.01 sec

Test Failed.
"testIntegrationMethods" end time: Apr 03 10:32 IST
"testIntegrationMethods" time elapsed: 00:00:00

[carmeloevoli]

Could you try to compile it and test it without python first? Just to be sure that the C++ part works properly.
Use -DENABLE_PYTHON=OFF