Installing Python packages in virtual environments

The Cirrus documentation recommends that you do this using virtual environments and provides a helpful example. However, the problem is that if you subsequently use pip or easy_install to install additional Python packages within the virtual environment you will get a permission denied as it tries to install the package centrally to directories you do not have access rights to. I eventually managed to find a solution.

Let me start with a disclaimer - I am not really an expert on Python but the experience of sorting this problem was sufficiently traumatic for me to want to blog about it. If you Google you will come across the solution but it may take some time to do so.

The rationale for using virtual environments is fairly straightforward. You do not want the Python package installer (pip or easy_install) to muck about with the centrally installed Python packages, inadvertently breaking someone else's code with updates or configuration changes to packages they rely on. For this reason virtual environments are recommended. However, when you try to do this using pip or easy_install you get permission denied because these package managers will try to download and install your package into directories to which you have no write access and will thus generate a permission denied message.

Using Cirrus as an example (though this issue is not limited to this system):

# Load the appropriate python module module load anaconda/python3

# Find out where pip is which pip # returns /lustre/sw/anaconda/anaconda3/bin/pip

# Create a conda virtual environment conda create --name myenv

# Activate the virtual environment source activate myenv

# Find out which version of pip we are using which pip

# returns/lustre/sw/anaconda/anaconda3/bin/pip

# Still points to the centrally installed # environment # Try to install a new package pip install pyqt5 ... PermissionError: [Errno 13] Permission denied: ...

# Deactivate the virtual environment source deactivate

# Start again, remove the environment: conda remove -n myenv --all

The trick is to create your virtual environment with your own version of  pip. So let's do it again:

# Create environment with your own version of pip conda create --name myenv pip

# Activate the environment source activate myenv

# You now have your own version of pip which pip ~ # returns /.conda/envs/myenv/bin/pip # This is now local

# You can now use this to install packages pip install pyqt5 Collecting pyqt5

...

100% |████████████████████████████████|

...

Installing collected packages: sip, pyqt5 Successfully installed pyqt5-5.10 sip-4.19.7

And that works! You may have to install the other packages you need after that into your own virtual environment.

I hope this helps someone.