Skip to content

Build Python Environment

Jump to bottom
Tao Chen edited this page Feb 12, 2019 · 5 revisions

Virtual Environments on r2d2

The old Python virtual environments are in /home/aardvark/tools/VENV/. Since aardvark folder would raise permission problems for normal users, new virtual environments are created in /home/envs/VENV. This folder is only for managing virtual environments on r2d2 and is also the home directory of the (virtual) user venv.

Activate a virtual environment

tao@r2d2:~$ source /home/envs/VENV/PYENV/bin/activate
(PYENV) tao@r2d2:~$ which python
/home/envs/VENV/PYENV/bin/python
(PYENV) tao@r2d2:~$ python
Python 3.6.4 (default, Mar  6 2018, 11:51:57)
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Try to install package under a normal user

...
  Downloading https://files.pythonhosted.org/packages/14/d0/a73c15bbeda3d2e7b381a36afb0d9cd770a9f4adc5d1532691013ba881db/toolz-0.9.0.tar.gz (45kB)
    100% |████████████████████████████████| 51kB 15.5MB/s
Building wheels for collected packages: regex, ujson, dill, cytoolz, wrapt, toolz
  Building wheel for regex (setup.py) ... done
  Stored in directory: /home/tao/.cache/pip/wheels/74/17/3f/c77bba99efd74ba1a19862c9dd97f4b6d735e2826721dc00ff
  Building wheel for ujson (setup.py) ... done
  Stored in directory: /home/tao/.cache/pip/wheels/28/77/e4/0311145b9c2e2f01470e744855131f9e34d6919687550f87d1
  Building wheel for dill (setup.py) ... done
  Stored in directory: /home/tao/.cache/pip/wheels/5b/d7/0f/e58eae695403de585269f4e4a94e0cd6ca60ec0c202936fa4a
  Building wheel for cytoolz (setup.py) ... done
  Stored in directory: /home/tao/.cache/pip/wheels/88/f3/11/9817b001e59ab04889e8cffcbd9087e2e2155b9ebecfc8dd38
  Building wheel for wrapt (setup.py) ... done
  Stored in directory: /home/tao/.cache/pip/wheels/48/5d/04/22361a593e70d23b1f7746d932802efe1f0e523376a74f321e
  Building wheel for toolz (setup.py) ... done
  Stored in directory: /home/tao/.cache/pip/wheels/f4/0c/f6/ce6b2d1aa459ee97cc3c0f82236302bd62d89c86c700219463
Successfully built regex ujson dill cytoolz wrapt toolz
Installing collected packages: cymem, preshed, msgpack, msgpack-numpy, plac, toolz, cytoolz, murmurhash, wrapt, dill, thinc, regex, ujson, spacy
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/home/envs/VENV/PYENV/lib/python3.6/site-packages/cymem'
Consider using the `--user` option or check the permissions.

Although tmp files are stored in /home/tao/.cache/pip/, the user tao has permission problems in /home/envs.

Now we have a virtual user venv on r2d2 only for managing virtual environments. When you want to install a new package for one Python virtual environment, switch to this user first (with password passwd4venv).

tao@r2d2:~$ su venv
Password:
...
venv@r2d2:~$ source /home/envs/VENV/PYENV/bin/activate
(PYENV) venv@r2d2:~$ pip install spacy
Collecting spacy
...
Successfully built ujson dill regex cytoolz wrapt toolz
Installing collected packages: cymem, preshed, ujson, dill, regex, murmurhash, plac, msgpack, toolz, cytoolz, wrapt, msgpack-numpy, thinc, spacy
Successfully installed cymem-2.0.2 cytoolz-0.9.0.1 dill-0.2.9 msgpack-0.5.6 msgpack-numpy-0.4.3.2 murmurhash-1.0.1 plac-0.9.6 preshed-2.0.1 regex-2018.1.10 spacy-2.0.18 thinc-6.12.1 toolz-0.9.0 ujson-1.35 wrapt-1.10.11

Remember to switch back to the normal user for other tasks.

Use existing Python to build virtual environment

There is a compiled Python 3.6 in /home/aardvark/tools/.local/python3/bin/python3.6 and also a system Python 3.2 in /usr/bin/python on r2d2. With an existing Python, one could easily build a Python virtual environment. (Additional info)

Link this Python to target folder (Optional)

Create a link file of Python3 to /home/envs/local/bin

venv@r2d2:~/local/bin$ ln -s /home/aardvark/tools/.local/python3/bin/python3.6 /home/envs/local/bin/python3.6
venv@r2d2:~/local/bin$ ln -s /home/envs/local/bin/python3.6 /home/envs/local/bin/python3

Create a virtual environment with this Python

Here I use the user venv as an example. If you want to create Python virtual environment for your own, you don't need to switch to venv.

Create a folder for virtual environment with command /path/to/python /path/to/virtualenv.py /path/to/folder (for virtualenv.py you can directly use /home/envs/packages/virtualenv-15.1.0/virtualenv.py). Remember to use the correct Python. So it is better to provide the full path to that Python.

venv@r2d2:~$ /home/envs/local/bin/python3 /home/envs/packages/virtualenv-15.1.0/virtualenv.py /home/envs/VENV/PYENV
Using base prefix '/home/aardvark/tools/.local/python3'
New python executable in /home/envs/VENV/PYENV/bin/python3
Also creating executable in /home/envs/VENV/PYENV/bin/python
Installing setuptools, pip, wheel...done.

Then in the folder which you set the path and name, pip has been generated inside (/home/envs/VENV/PYENV/bin/pip).

Activate this virtual environment and the default Python in this environment is the one you set when creating this environment.

venv@r2d2:~$ source /home/envs/VENV/PYENV/bin/activate
(PYENV) venv@r2d2:~$ which python
/home/envs/VENV/PYENV/bin/python

Now under this environment, you can install packages like pip install numpy and also do anything with Python.

Clone this wiki locally