Installing Python packages in virtual environments

Author: Mario Antonioletti
Posted: 22 Mar 2018 | 11:46

I recently looked into whether the Python package PyQt5 could be installed on Cirrus, a Tier-2 national service, on behalf of one of our HPC Europa visitors. The Cirrus documentation recommends that you do this using virtual environments and provides a helpful example. However, the problem is that if you subsequently use pip or easy_install to install additional Python packages within the virtual environment you will get a permission denied as it tries to install the package centrally to directories you do not have access rights to. I eventually managed to find a solution.

Let me start with a disclaimer - I am not really an expert on Python but the experience of sorting this problem was sufficiently traumatic for me to want to blog about it. If you Google you will come across the solution but it may take some time to do so.

The rationale for using virtual environments is fairly straightforward. You do not want the Python package installer (pip or easy_install) to muck about with the centrally installed Python packages, inadvertently breaking someone else's code with updates or configuration changes to packages they rely on. For this reason virtual environments are recommended. However, when you try to do this using pip or easy_install you get permission denied because these package managers will try to download and install your package into directories to which you have no write access and will thus generate a permission denied message.

Using Cirrus as an example (though this issue is not limited to this system):

# Load the appropriate python module   
module load anaconda/python3     

# Find out where pip is   
which pip   
# returns /lustre/sw/anaconda/anaconda3/bin/pip     

# Create a conda virtual environment   
conda create --name myenv     

# Activate the virtual environment   
source activate myenv     

# Find out which version of pip we are using   
which pip   
# returns/lustre/sw/anaconda/anaconda3/bin/pip   

# Still points to the centrally installed   
# environment     # Try to install a new package   
pip install pyqt5   
...   
PermissionError: [Errno 13] Permission denied: ...  

# Deactivate the virtual environment   
source deactivate     

# Start again, remove the environment:   
conda remove -n myenv --all   

The trick is to create your virtual environment with your own version of  pip. So let's do it again:

# Create environment with your own version of pip  
conda create --name myenv pip    

# Activate the environment  
source activate myenv    

# You now have your own version of pip  
which pip ~ 
# returns /.conda/envs/myenv/bin/pip  
# This is now local    

# You can now use this to install packages   
pip install pyqt5 Collecting pyqt5    
...      
100% |████████████████████████████████| 
...  
Installing collected packages: sip, pyqt5  
Successfully installed pyqt5-5.10 sip-4.19.7  

And that works! You may have to install the other packages you need after that into your own virtual environment.

I hope this helps someone.

Author

Mario Antonioletti, EPCC