Installing Python packages in virtual environments
Posted: 22 Mar 2018 | 11:46
I recently looked into whether the Python package PyQt5 could be installed on Cirrus, a Tier-2 national service, on behalf of one of our HPC Europa visitors. The Cirrus documentation recommends that you do this using virtual environments and provides a helpful example. However, the problem is that if you subsequently use
easy_install to install additional Python packages within the virtual environment you will get a permission denied as it tries to install the package centrally to directories you do not have access rights to. I eventually managed to find a solution.
Let me start with a disclaimer - I am not really an expert on Python but the experience of sorting this problem was sufficiently traumatic for me to want to blog about it. If you Google you will come across the solution but it may take some time to do so.
The rationale for using virtual environments is fairly straightforward. You do not want the Python package installer (
easy_install) to muck about with the centrally installed Python packages, inadvertently breaking someone else's code with updates or configuration changes to packages they rely on. For this reason virtual environments are recommended. However, when you try to do this using
easy_install you get permission denied because these package managers will try to download and install your package into directories to which you have no write access and will thus generate a permission denied message.
Using Cirrus as an example (though this issue is not limited to this system):
# Load the appropriate python module module load anaconda/python3 # Find out where pip is which pip # returns /lustre/sw/anaconda/anaconda3/bin/pip # Create a conda virtual environment conda create --name myenv # Activate the virtual environment source activate myenv # Find out which version of pip we are using which pip # returns/lustre/sw/anaconda/anaconda3/bin/pip # Still points to the centrally installed # environment # Try to install a new package pip install pyqt5 ... PermissionError: [Errno 13] Permission denied: ... # Deactivate the virtual environment source deactivate # Start again, remove the environment: conda remove -n myenv --all
The trick is to create your virtual environment with your own version of
pip. So let's do it again:
# Create environment with your own version of pip conda create --name myenv pip # Activate the environment source activate myenv # You now have your own version of pip which pip ~ # returns /.conda/envs/myenv/bin/pip # This is now local # You can now use this to install packages pip install pyqt5 Collecting pyqt5 ... 100% |████████████████████████████████| ... Installing collected packages: sip, pyqt5 Successfully installed pyqt5-5.10 sip-4.19.7
And that works! You may have to install the other packages you need after that into your own virtual environment.
I hope this helps someone.
Mario Antonioletti, EPCC