From 8bc87ffc29877a175d4f6e6a10f9fb9608f73d1b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix-Antoine=20Fortin?= Date: Thu, 23 Mar 2023 16:54:34 -0400 Subject: [PATCH 1/8] Draft ideas for jupyter-lmod docs in Lmod --- docs/source/370_jupyter_lmod.rst | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 docs/source/370_jupyter_lmod.rst diff --git a/docs/source/370_jupyter_lmod.rst b/docs/source/370_jupyter_lmod.rst new file mode 100644 index 000000000..60996fedb --- /dev/null +++ b/docs/source/370_jupyter_lmod.rst @@ -0,0 +1,26 @@ +Jupyter and Lmod +================ + +- It is possible to use Lmod inside of Jupyter + - using the terminal command-line + - using jupyter-lmod + +- jupyter-lmod is a plugin that integrates inside Jupyter Notebook and JupyterLab UI +and provides Lmod functionalities graphically + - Filter available module through textfield + - Load and unload modules with button + - Display a module content + - Create and load collections + +- How does it work? jupyter-lmod uses lmod python interface to generate Python code +that modifies the environment variables of the Python process running Jupyter. All child +processes of Jupyter (including kernels) created after the module function calls, inherits +the environment variables as defined by Lmod's modules. + +- The plugin requires the export of the environment variable `LMOD_CMD` before the launch +of jupyter. If you are using JupyterHub, with some spawners (i.e.: LocalSpawner, SudoSpawner), +you will have to add `LMOD_CMD` to `c.Spawner.env_keep` in `jupyterhub_config.py` + +- For more information on how to install it, refer to jupyter-lmod github repo: +https://www.github.com/cmd-ntrf/jupyter-lmod + From fcdc83907a9309955c1763db936a30a7c44fbcff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix-Antoine=20Fortin?= Date: Fri, 24 Mar 2023 12:01:50 -0400 Subject: [PATCH 2/8] Improve jupyter-lmod doc draft --- docs/source/370_jupyter_lmod.rst | 54 +++++++++++++++++++++++--------- docs/source/index.rst | 1 + 2 files changed, 40 insertions(+), 15 deletions(-) diff --git a/docs/source/370_jupyter_lmod.rst b/docs/source/370_jupyter_lmod.rst index 60996fedb..2af7631dd 100644 --- a/docs/source/370_jupyter_lmod.rst +++ b/docs/source/370_jupyter_lmod.rst @@ -1,26 +1,50 @@ Jupyter and Lmod ================ -- It is possible to use Lmod inside of Jupyter - - using the terminal command-line - - using jupyter-lmod - -- jupyter-lmod is a plugin that integrates inside Jupyter Notebook and JupyterLab UI -and provides Lmod functionalities graphically - - Filter available module through textfield - - Load and unload modules with button +It is possible to use Lmod inside of Jupyter with the terminal or jupyter-lmod. + +jupyter-lmod is a frontend and server extension for Jupyter (notebook and jupyterlab). +The extension provides most of Lmod functionalities graphically: + - Browser and filter available modules + - Load and unload modules - Display a module content - Create and load collections -- How does it work? jupyter-lmod uses lmod python interface to generate Python code +Installation +~~~~~~~~~~~~ + +To install jupyter-lmod extension: :: + + pip install jupyterlmod + +This enables the extension for both Jupyter Notebook and JupyterLab. + +How does it work? +~~~~~~~~~~~~~~~~~ + +jupyter-lmod uses lmod python interface to generate Python code that modifies the environment variables of the Python process running Jupyter. All child -processes of Jupyter (including kernels) created after the module function calls, inherits -the environment variables as defined by Lmod's modules. +processes of Jupyter (including kernels) created after Lmod calls inherit +the environment variables as defined by the set of loaded modules. + +Lmod environment variable configuration for jupyter-lmod +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The plugin requires the environment variable **$LMOD_CMD** to be defined when Jupyter starts, +plus all environment variables required by Lmod to work properly (i.e.: **$MODULEPATH**). + +Some JupyterHub spawners do not setup the complete user's environment, leaving **$LMOD_CMD** +and **$MODULEPATH** undefined. To address this issue, the environment variables have to be +defined in **jupyterhub_config.py** using +`c.Spawner.environment `_. + +If the environment variables are defined before JupyterHub launch, you can instead add their +name only to `c.Spawner.env_keep `_ +instead. -- The plugin requires the export of the environment variable `LMOD_CMD` before the launch -of jupyter. If you are using JupyterHub, with some spawners (i.e.: LocalSpawner, SudoSpawner), -you will have to add `LMOD_CMD` to `c.Spawner.env_keep` in `jupyterhub_config.py` +References +~~~~~~~~~~ -- For more information on how to install it, refer to jupyter-lmod github repo: +For more information, refer to jupyter-lmod github repo: https://www.github.com/cmd-ntrf/jupyter-lmod diff --git a/docs/source/index.rst b/docs/source/index.rst index f99a5d75d..fd04e3e81 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -146,6 +146,7 @@ Advanced Topics 340_inherit 350_community 360_check_syntax + 370_jupyter_lmod Internal Structure of Lmod ^^^^^^^^^^^^^^^^^^^^^^^^^^ From e933d07b162afffa5ed9cf9686a8f00e6acb30de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix-Antoine=20Fortin?= Date: Fri, 24 Mar 2023 12:03:04 -0400 Subject: [PATCH 3/8] Typo --- docs/source/370_jupyter_lmod.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/370_jupyter_lmod.rst b/docs/source/370_jupyter_lmod.rst index 2af7631dd..ca2924a50 100644 --- a/docs/source/370_jupyter_lmod.rst +++ b/docs/source/370_jupyter_lmod.rst @@ -38,7 +38,7 @@ and **$MODULEPATH** undefined. To address this issue, the environment variables defined in **jupyterhub_config.py** using `c.Spawner.environment `_. -If the environment variables are defined before JupyterHub launch, you can instead add their +If the environment variables are defined before JupyterHub launch, you can add their name only to `c.Spawner.env_keep `_ instead. From 7a21881c161b56f9ba1229cf913f3cf04ddb49b2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix-Antoine=20Fortin?= Date: Fri, 24 Mar 2023 14:36:29 -0400 Subject: [PATCH 4/8] Add clarification on PYTHONPATH --- docs/source/370_jupyter_lmod.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/source/370_jupyter_lmod.rst b/docs/source/370_jupyter_lmod.rst index ca2924a50..8e01f7ccb 100644 --- a/docs/source/370_jupyter_lmod.rst +++ b/docs/source/370_jupyter_lmod.rst @@ -27,6 +27,9 @@ that modifies the environment variables of the Python process running Jupyter. A processes of Jupyter (including kernels) created after Lmod calls inherit the environment variables as defined by the set of loaded modules. +In concrete terms, if a module modifies **$PYTHONPATH** and it is loaded after +a Python kernel was spawned, the kernel will not be aware of the change. + Lmod environment variable configuration for jupyter-lmod ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From c1381e083d0157b0e1f1e8ff2bb661b077bebb82 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix-Antoine=20Fortin?= Date: Mon, 27 Mar 2023 10:31:32 -0400 Subject: [PATCH 5/8] Update docs --- docs/source/370_jupyter_lmod.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/source/370_jupyter_lmod.rst b/docs/source/370_jupyter_lmod.rst index 8e01f7ccb..8fec02aba 100644 --- a/docs/source/370_jupyter_lmod.rst +++ b/docs/source/370_jupyter_lmod.rst @@ -27,8 +27,11 @@ that modifies the environment variables of the Python process running Jupyter. A processes of Jupyter (including kernels) created after Lmod calls inherit the environment variables as defined by the set of loaded modules. -In concrete terms, if a module modifies **$PYTHONPATH** and it is loaded after -a Python kernel was spawned, the kernel will not be aware of the change. +In concrete terms, kernels in use by active notebooks will not be aware of changes +made to the environment (e.g. **$PATH**, **$PYTHONPATH**, **$LD_LIBRARY_PATH**) by modules +loaded after they were spawned. Depending on the Jupyter version, a manual kernel restart +or a complete kernel shutdown/start cycle is needed to make an active notebook aware of changes +to the environment variables. Lmod environment variable configuration for jupyter-lmod ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 50efcf3a451de45707db10cab21ff3278001a620 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix-Antoine=20Fortin?= Date: Mon, 27 Mar 2023 16:16:16 -0400 Subject: [PATCH 6/8] Add doc on dynamic launcher buttons --- docs/source/370_jupyter_lmod.rst | 70 ++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/docs/source/370_jupyter_lmod.rst b/docs/source/370_jupyter_lmod.rst index 8fec02aba..300d6901a 100644 --- a/docs/source/370_jupyter_lmod.rst +++ b/docs/source/370_jupyter_lmod.rst @@ -48,6 +48,76 @@ If the environment variables are defined before JupyterHub launch, you can add t name only to `c.Spawner.env_keep `_ instead. +Controlling launcher items with jupyter-lmod +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +jupyter-lmod can add a kernel or a server proxy to the Jupyter launcher +following module loading if the module matches some specifications. + +kernels ++++++++ + +If a module adds one or more absolute paths to **$JUPYTER_PATH**, Jupyter's kernel spec manager +will be notified of the changes. If the added paths contains kernel spec files (e.g: **kernel.json**), +the kernels will be added to the available choices in Jupyter interface. + +The following example demonstrates this feature with Julia. Here, we have a **ijulia-kernel/1.8** module +that prepends a path to **$JUPYTER_PATH**. :: + + whatis("Description: IJulia kernelspec for Julia 1.8") + whatis("Homepage: https://julialang.org/") + whatis("URL: https://julialang.org/") + conflict("ijulia-kernel") + depends_on("julia/1.8.1") + depends_on("ijulia/.1.24.0") + prepend_path("JUPYTER_PATH","/software/Core/ijulia-kernel/1.8/share/jupyter") + + +In **/software/Core/ijulia-kernel/1.8/share/jupyter/kernels/julia-1.8**, there are four files :: + + kernel.json logo-32x32.png logo-64x64.png logo-svg.svg + +and **kernel.json** looks like this :: + + { + "display_name": "Julia 1.8.1", + "argv": [ + "/software/Core/julia/1.8.1/bin/julia", + "-i", + "--color=yes", + "/software/Core/ijulia/1.24.0/packages/IJulia/6TIq1/src/kernel.jl", + "{connection_file}" + ], + "language": "julia", + "env": {}, + "interrupt_mode": "signal" + } + +The folders and file under **share/jupyter/kernels** are typically generated by a kernelspec installer. + + +server proxies +++++++++++++++ + +Jupyter Server Proxy is an extension for Jupyter and JupyterLab that let user run arbitrary +external processes (such as RStudio, Shiny Server, Syncthing, PostgreSQL, Code Server, etc) +alongside their notebook server and provide authenticated web access to them using a path +like /rstudio next to others like /lab. It also provides button in the launcher to launch +these external processes. + +When jupyter-lmod and jupyter-server-proxy are both installed, it is possible for the launcher +button of an external process to appear only when its corresponding module is loaded. +In order to do so, the frontend extension of jupyter-server-proxy must be deactivated as +the control of the launcher will be entrusted to jupyter-lmod. :: + + jupyter nbextension disable --py jupyter_server_proxy --sys-prefix + jupyter labextension disable @jupyterlab/server-proxy + jupyter labextension disable jupyterlab-server-proxy + +Then, the name of each `server proxy ` +must match the name of module. When a module with a name corresponding to a server +proxy name is loaded, jupyter-lmod will add a button to the launcher to start that process. + References ~~~~~~~~~~ From 3b11c1ccb7f73829b3ba5fc8f6692fc95ee70ba9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix-Antoine=20Fortin?= Date: Wed, 29 Mar 2023 13:29:44 -0400 Subject: [PATCH 7/8] Fix typos --- docs/source/370_jupyter_lmod.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/source/370_jupyter_lmod.rst b/docs/source/370_jupyter_lmod.rst index 300d6901a..d0bc95f26 100644 --- a/docs/source/370_jupyter_lmod.rst +++ b/docs/source/370_jupyter_lmod.rst @@ -4,7 +4,7 @@ Jupyter and Lmod It is possible to use Lmod inside of Jupyter with the terminal or jupyter-lmod. jupyter-lmod is a frontend and server extension for Jupyter (notebook and jupyterlab). -The extension provides most of Lmod functionalities graphically: +The extension graphically provides most of Lmod's functions: - Browser and filter available modules - Load and unload modules - Display a module content @@ -45,7 +45,7 @@ defined in **jupyterhub_config.py** using `c.Spawner.environment `_. If the environment variables are defined before JupyterHub launch, you can add their -name only to `c.Spawner.env_keep `_ +names only to `c.Spawner.env_keep `_ instead. Controlling launcher items with jupyter-lmod @@ -58,10 +58,10 @@ kernels +++++++ If a module adds one or more absolute paths to **$JUPYTER_PATH**, Jupyter's kernel spec manager -will be notified of the changes. If the added paths contains kernel spec files (e.g: **kernel.json**), +will be notified of the changes. If the added paths contain kernel spec files (e.g: **kernel.json**), the kernels will be added to the available choices in Jupyter interface. -The following example demonstrates this feature with Julia. Here, we have a **ijulia-kernel/1.8** module +The following example demonstrates this feature with Julia. Here, we have an **ijulia-kernel/1.8** module that prepends a path to **$JUPYTER_PATH**. :: whatis("Description: IJulia kernelspec for Julia 1.8") @@ -93,16 +93,16 @@ and **kernel.json** looks like this :: "interrupt_mode": "signal" } -The folders and file under **share/jupyter/kernels** are typically generated by a kernelspec installer. +The folders and files under **share/jupyter/kernels** are typically generated by a kernelspec installer. server proxies ++++++++++++++ -Jupyter Server Proxy is an extension for Jupyter and JupyterLab that let user run arbitrary +Jupyter Server Proxy is an extension for Jupyter and JupyterLab that let users run arbitrary external processes (such as RStudio, Shiny Server, Syncthing, PostgreSQL, Code Server, etc) alongside their notebook server and provide authenticated web access to them using a path -like /rstudio next to others like /lab. It also provides button in the launcher to launch +like /rstudio next to others like /lab. It also provides buttons in the launcher to launch these external processes. When jupyter-lmod and jupyter-server-proxy are both installed, it is possible for the launcher From 339cc86fcda79bb19096a3c28f252e1a9577bd66 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?F=C3=A9lix-Antoine=20Fortin?= Date: Wed, 29 Mar 2023 13:53:36 -0400 Subject: [PATCH 8/8] Update docs --- docs/source/370_jupyter_lmod.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/source/370_jupyter_lmod.rst b/docs/source/370_jupyter_lmod.rst index d0bc95f26..7dc95ef62 100644 --- a/docs/source/370_jupyter_lmod.rst +++ b/docs/source/370_jupyter_lmod.rst @@ -10,6 +10,10 @@ The extension graphically provides most of Lmod's functions: - Display a module content - Create and load collections +The extension is located in the 'Softwares' tab of the main interface in Jupyter Notebook, +while in JupyterLab, it can be accessed as a sidepanel by clicking on the Lmod logo icon +located in the sidebar. + Installation ~~~~~~~~~~~~