$SINGULARITY_ENVIRONMENT, --nv, and random cleanup

Author: GodloveD

_posts/recipes/2017-05-09-hostlibs-gpus-and-mpi.md

@@ -4,6 +4,11 @@ category: recipes
 permalink: tutorial-gpu-drivers-open-mpi-mtls
 ---
 
+**Note: _Much of the GPU portion of this tutorial is deprecated by the `--nv`
+option that automatically binds host system driver libraries into your 
+container at runtime.  See [the `exec` command](/docs-exec#a-gpu-example) for 
+an example_**
+
 Singularity does a fantastic job  of isolating you from the host so you don't
 have to muck about with `LD_LIBRARY_PATH`, you just get exactly the library
 versions you want. However, in some situations you need to use library

pages/docs/contributing/contributing-docs.md

@@ -3,12 +3,14 @@ title: Contributing to Documentation
 sidebar: main_sidebar
 permalink: contributing-docs
 folder: releases
+toc: false
 ---
 
 We (like almost all open source software providers) have a documentation dillemma... We tend to focus on the code features and functionality before working on documentation. And there is very good reason for this, we want to share the love so nobody feels left out!
 
 The following documentation page assumes one is running on OS X, but if you are not, you should be able to easily transpose the necessary commands to your operating system of choice.
 
+{% include toc.html %}
 
 ## Setting Up Your Development Environment
 
@@ -113,8 +115,10 @@ asciinema rec -w 1 demo-asciicast.js
 # To play
 asciinema play demo-asciicast.js
 ```
+Make sure to resize your terminal to 25 rows x 115 columns or less before you 
+begin your recording.  Otherwise it will not display properly on the webpage.
 
-Once you've generated an asciicast, you should drop the file (e.g., some `demo-asciicast.js`) into the `assets/asciicast` folder, and then include the following in the page or post:
+Once you've generated an asciicast, you should drop the file (e.g., `demo-asciicast.js`) into the `assets/asciicast` folder, and then include the following in the page or post:
 
 ```bash
 {{ "{% include asciicast.html source='demo-asciicast.js' title='How to make demos' author='email@domain.com'" }}%}

pages/docs/overview/faq.md

@@ -213,7 +213,10 @@ See the Singularity on HPC page for more details.
 
 ### Does Singularity support containers that require GPUs?
 
-Yes, Singularity has been tested to run some test and diagnostic code from within a container without modification. There are however potential issues that can come into play when using GPUs, for instance there are version API compatibilities between kernel and user land which will have to be considered.
+Yes. Many users run GPU dependant code within Singularity containers.  The
+experimental `--nv` option allows you to leverage host GPUs without installing 
+system level drivers into your container. See [the `exec` command](/docs-exec#a-gpu-example) for
+an example.
 
 ## Container portability
 

pages/docs/user-docs/docs-bootstrap-image.md

@@ -3,9 +3,9 @@ title: Getting Started with Bootstrap
 sidebar: user_docs
 permalink: bootstrap-image
 folder: docs
+toc: false
 ---
 
-## Bootstrapping a Container
 Bootstrapping is the process where we install an operating system and then configure it appropriately for a specified need. To do this we use a bootstrap definition file (a text file called `Singularity`) which is a recipe of how to specifically build the container. Here we will overview the sections, best practices, and a quick example.
 
 {% include toc.html %}
@@ -142,7 +142,25 @@ Version 2.0
 ```
 
 #### %environment
-Akin to labels, you can add pairs of `VARIABLE` and `VALUE` under environment to be sourced when the container is used as variables in the environment. The entire section is written to a file that gets sourced, so you should generally use the same conventions that you might use in a `bashrc` or `profile`. See <a href="/docs-environment-metadata">Environment and Metadata</a> for more information about these two sections.
+You can add environment variables to be sourced when the container is used in the `%environment` section. The entire section is written to a file that gets sourced, so you should generally use the same conventions that you might use in a `bashrc` or `profile`. 
+
+```
+%environment
+    export VADER=badguy
+    export LUKE=goodguy
+    export SOLO=someguy
+```
+
+You can also add environment variables to your container in the `%post` section (see below) using the following syntax:
+
+```
+%post
+    echo 'export JAWA_SEZ=wutini' >>$SINGULARITY_ENVIRONMENT
+```
+
+Variables added to your container using the `$SINGULARITY_ENVIRONMENT` syntax take precedence over those added in the `%environment` section. 
+
+See <a href="/docs-environment-metadata">Environment and Metadata</a> for more information about the `%labels` and `%environment` sections.
 
 
 #### %post

pages/docs/user-docs/docs-bootstrap.md

@@ -6,9 +6,90 @@ toc: false
 folder: docs
 ---
 
+The process of *bootstrapping* a Singularity container is equivalent to describing a recipe for the container creation. There are several recipe formats that Singularity supports, but only the primary format of version 2.3 will be documented here. If you want a general overview with examples, see <a href="/bootstrap-image">Bootstrapping an Image</a>. The detailed options for each of the header and sections are provided here.
+
 {% include toc.html %}
 
-The process of *bootstrapping* a Singularity container is equivalent to describing a recipe for the container creation. There are several recipe formats that Singularity supports, but only the primary format of version 2.3 will be documented here. If you want a general overview with examples, see <a href="/bootstrap-image">Bootstrapping an Image</a>. The detailed options for each of the header and sections are provided here.
+## Usage
+
+```
+USAGE: singularity [...] bootstrap <container path> <definition file>
+
+The bootstrap command is useful for creating a new bootstrap or modifying an
+existing one using a definition file that describes how to build the
+container. The quickest, easiest Bootstrap is dumping DOcker layers into
+a container:
+
+    BootStrap: docker
+    From: library/ubuntu:14.04.1
+
+For more documentation on definition file recipes, look below.
+
+note: This command must be executed as root.
+
+
+
+CREATE OPTIONS:
+    -T|--notest     Bootstrap without running tests in %test section
+    -f|--force      Force a rebootstrap of an OS (note: this does not delete
+                    what is currently in the image, just causes the core to
+                    be reinstalled)
+    -s|--section    Only run a given section within the bootstrap (setup,
+                    post, files, environment, test, labels, none)
+       --nobase     Skip the generation of the base OS
+
+
+DEFFILE BASEOS EXAMPLES:
+
+    Docker:
+        BootStrap: docker
+        From: tensorflow/tensorflow:latest
+        IncludeCmd: yes #Include the CMD/ENTRYPOINT as the Singularity Runscript
+
+    YUM/RHEL:
+        BootStrap: yum
+        OSVersion: 7
+        MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
+        Include: yum #This is important to include inside the base container
+
+    Debian/Ubuntu:
+        BootStrap: debootstrap
+        OSVersion: trusty
+        MirrorURL: http://us.archive.ubuntu.com/ubuntu/
+
+
+DEFFILE SECTION EXAMPLES:
+
+    %setup
+        echo "This is a scriptlet that will be executed on the host, as root, after"
+        echo "the container has been bootstrapped. To install things into the container"
+        echo "reference the file system location with \$SINGULARITY_BUILDROOT"
+
+    %post
+        echo "This scriptlet section will be executed from within the container after"
+        echo "the bootstrap/base has been created and setup"
+
+    %test
+        echo "Define any test commands that should be executed after container has been"
+        echo "built. This scriptlet will be executed from within the running container"
+        echo "as the root user. Pay attention to the exit/return value of this scriptlet"
+        echo "as any non-zero exit code will be assumed as failure"
+        exit 0
+
+    %labels
+        HELLO MOTO
+        KEY VALUE
+
+    %files
+        /path/on/host/file.txt /path/on/container/file.txt 
+        relative_file.txt /path/on/container/relative_file.txt
+
+
+EXAMPLES:
+
+    $ singularity create /tmp/Debian.img
+    $ sudo singularity bootstrap /tmp/Debian.img debian.def
+```
 
 ## The header fields:
 
@@ -48,7 +129,7 @@ The Docker bootstrap module will create a core operating system image based on a
 Once the `Bootstrap` module has completed, the sections are identified and utilized if present. The following sections are supported in the bootstrap definition, and integrated during the bootstrap process:
 
 
-#### %setup
+### %setup
 This section blob is a Bourne shell scriptlet which will be executed on the host outside the container during bootstrap. The path to the container is accessible from within the running scriptlet environment via the variable `$SINGULARITY_ROOTFS`. For example, consider the following scriptlet:
 
 ```
@@ -65,8 +146,7 @@ As we investigate this example scriptlet, you will first see this is the outside
 
 *note: Any uncaught command errors that occur within the scriptlet will cause the entire build process to halt!*
 
-
-#### %post
+### %post
 Similar to the `%setup` section, this section will be executed once during bootstrapping, but this scriptlet will be run from inside the container. This is where you should put additional installation commands, downloads, and configuration into your containers. Here is an example to consider:
 
 ```
@@ -94,8 +174,26 @@ The above example runs inside the container, so in this case we will first insta
 
 *another note: This is not a good example of a reproducible definition because it is pulling Open MPI from a moving target. A better example, would be to pull a static released version, but this serves as a good example of building a `%post` scriptlet.*
 
+### %environment 
+Beginning with Singularity v2.3 you can set up your environment using the `%environment` section of the definition file.  For example, if you wanted to add a directory to your search path, you could do so like this.
+
+```
+%environment
+    export PATH=/opt/good/stuff:$PATH
+```
+
+In older versions of Singularity you could set up your container's environment by adding text to a file called `/environment`.  This method is now deprecated.  If you need to add environment variables to your container during the `%post` section (perhaps because you will not know the values of some variables until some installation steps have completed).  You can do so with the following syntax:
+
+```
+%post
+    echo 'export PATH=/opt/good/stuff:$PATH' >>$SINGULARITY_ENVIRONMENT
+```
+
+Text in the `%environment` section will be appended to a file called `/.singularity.d/env/90-environment.sh`.  Text redirected to the `$SINGULARITY_ENVIRONMENT` variable will added to a file called `/.singularity.d/env/91-environment.sh`.  At runtime, scripts in `/.singularity/env` are sourced in order.
+
+This means that variables in `$SINGULARITY_ENVIRONMENT` take precedence over those added via `%environment`.
 
-#### %runscript
+### %runscript
 The `%runscript` is another scriptlet, but it does not get executed during bootstrapping. Instead it gets persisted within the container to a file called `/singularity` which is the execution driver when the container image is ***run*** (either via the `singularity run` command or via executing the container directly).
 
 When the `%runscript` is executed, all options are passed along to the executing script at runtime, this means that you can (and should) manage argument processing from within your runscript. Here is an example of how to do that:
@@ -108,7 +206,7 @@ When the `%runscript` is executed, all options are passed along to the executing
 
 In this particular runscript, the arguments are printed as a single string (`$*`) and then they are passed to `/usr/bin/python` via a quoted array (`$@`) which ensures that all of the arguments are properly parsed by the executed command. The `exec` command causes the given command to replace the current entry in the process table with the one that is to be called. This makes it so the runscript shell process ceases to exist, and the only process running inside this container is the called Python command.
 
-#### %test
+### %test
 You may choose to add a `%test` section to your definition file. This section will be run at the very end of the boostrapping process and will give you a chance to validate the container during the bootstrap process. If you are building on Singularity Hub, [it is a good practice](https://github.com/singularityhub/singularityhub.github.io/wiki/Generate-Images#add-tests) to have this test section so you can be sure that your container works as expected. A non-zero status code indicates that one or more of your tests did not pass. You can also execute this scriptlet through the container itself, such that you can always test the validity of the container itself as you transport it to different hosts. Extending on the above Open MPI `%post`, consider this example:
 
 ```

pages/docs/user-docs/docs-environment-metadata.md

@@ -1,15 +1,16 @@
 ---
-title: Changing Existing Containers
+title: Environment and Metadata
 sidebar: user_docs
 permalink: docs-environment-metadata
 folder: docs
+toc: false
 ---
 
-## Container Metadata
+Singularity containers support environment variables and labels that can be added by user during the bootstrap process.
 
-Singularity containers have two level of metadata - environment variables, and labels from the user and bootstrap process.
+{% include toc.html %}
 
-### Environment
+## Environment
 
 If you are importing a Docker container, the environment will be imported as well. If you want to define custom environment variables in your bootstrap recipe file `Singularity` you can do that like this
 
@@ -18,10 +19,19 @@ Bootstrap:docker
 From: ubuntu:latest
 
 %environment
-VARIABLE_NAME=VARIABLE_VALUE
-export VARIABLE_NAME
+    VARIABLE_NAME=VARIABLE_VALUE
+    export VARIABLE_NAME
 ```
 
+If you need to add an environment variable to your container during the `%post` section, you can do so using the `$SINGULARITY_ENVIRONMENT` variable with the following syntax:
+
+```
+%post
+    echo 'export VARIABLE_NAME=VARIABLE_VALUE' >>$SINGULARITY_ENVIRONMENT
+```
+
+Text in the `%environment` section will be appended to the file `/.singularity.d/env/90-environment.sh` while text redirected to `$SINGULARITY_ENVIRONMENT` will end up in the file `/.singularity.d/env/91-environment.sh`.  Because files in `/.singularity.d/env` are sourced in alpha-numerical order, this means that variables added using `$SINGULARITY_ENVIRONMENT` take precedence over those added via the `%environment` section.
+
 Forget something, or need a variable defined at runtime? You can set any variable you want inside the container by prefixing it with "SINGULARITYENV_". It will be transposed automatically and the prefix will be stripped. For example, let's say we want to set the variable `HELLO` to have value `WORLD`. We can do that as follows:
 
 ```bash
@@ -58,7 +68,7 @@ singularity exec centos7.img cat /.singularity.d/env/10-docker.sh
 export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 ```
 
-### Labels
+## Labels
 Your container stores metadata about it's build, along with Docker labels. You can see the data as follows:
 
 ```bash