Feature request: Python interface

[LecrisUT]

Description

There are a few issues that create friction between Python and Lmod:

• Python environments i.e. subprocess.Popen do not persist between commands
• Lmod functions are exported as functions instead of having sourceable interface
• Lmod interface is not guaranteed to be loaded e.g. in default github CI

These factors make it hard to chain python/shell commands with pushed/poped Lmod module environment

Proposal

Provide a simple Python interface that can call Lmod functions directly. A minimum interface could be:

• module list
• module load
• module unload/purge

The introspection commands could be direct calls to the Lmod interface with some basic output conversion. The main part is how the load/unload interface can look like. Here a Lmod class would carry an env dictionary with all the Lmod native environment operations like prepend_path chained accordingly. Then when requested, these can be evaluated through Lua or equivalent to have the plain python dictionary that can be fed through subprocess

Considerations

• There are a few python packages that can call lua scripts
• If two introspection commands module getenv and module getscripts (exporting the environment variable and executed scripts respectively for a given loaded environment) can be provided on the Lmod side, this would simplify writing this interface
• Supporting external scripts that source environment variables might not be easily supported
• swig supports making bindings for both languages. That could make such interfaces available to many more other languages as well

[LecrisUT]

I have looked a bit and I found env_modules_python.py. That's really nice, and I see there are already a great deal of setup implemented there. However, the interface could be improved and packaged directly to PyPI. One change to make it more accessible is to output a lmod json [commands] instead of raw python interface.

[ocaisa]

Maybe https://lmodule.readthedocs.io/en/latest/index.html is of interest to you (I've never used it, I just know of it's existence).

[LecrisUT]

Thanks for the link. I did a quick look at the implementation and I do not really like it. The interface goes through the shell script so it can still fail in the Github CI interface and it only provides with simple module load text commands that you would manually have to chain

I am investigating BaseShell.lua and I think I can make a PR with a simple json interface soon as a precursor to this

[mrcawood]

I'll discuss this with Robert, but feel free make a PR and thanks for your input

[LecrisUT]

Thanks, the first part is in #673, I'll need some feedback there of how it should be exported most generally. After that I could make the python interface management, to be included in this project or a one of the third-party ones like lmodule. Preferably it should be here in order to make downstream packaging easier.