Working With Multiple Environments in Python

Working within a single Python environment in which you're limited to a single version of python and a single version of all your modules is fine for small projects, but as the number of projects or age of projects increase it leaves you boxed in.  You can't test your software against a new version of a dependency without putting everything else on hold, and then you may have to revert back.  Eventually, you'll end up doing insufficient testing or spending far too much time on installing, uninstalling, and developing projects in series rather than parallel.

Ian Bicking's Virtualenv changed this in 2007 with the ability to create multiple environments for a Python application.  Then a year later Doug Hellman wrote Virtualenvwrapper which added a set of convenience functions that significantly enhance virtualenv.

The documentation for these products is generally very good, but it seems to me that something is missing - documentation that pulls everything together across products so that the beginning user has a simple recipe for success.  The closest would be The Hitchhiker's Guide to Python, but this doesn't address multiple Python versions.   And I really want to nail down how to do this with multiple versions of Python.

The following information is intended to be a first step in that direction for the linux developer.

Installation of multiple versions of Python

Supporting more than a single Python version is an option that takes a little extra work and provides quite a bit more flexibility.  I covered this in my last article, with a detailed description of installing via source code compile.

Installation of Python Distribute

  • $ mkdir ~/Downloads; cd ~/Downloads
  • $ curl -O http://python-distribute.org/distribute_setup.py
  • $ python2.6 distribute_setup.py
  • $ python2.7 distribute_setup.py
  • $ python3.3 distribute_setup.py

Installation of Pip

  • cd ~/Downloads
  • Download the most recent version of pip.
    • Get it from http://pypi.python.org/pypi/pip.
    • Most recent version is currently pip-1.2.1.tar.gz.
    • wget http://pypi.python.org/packages/source/p/pip/pip-1.2.1.tar.gz
  • $ tar -xvf pip-1.2.1.tar.gz
  • $ cd ~/Downloads/pip-1.2.1
  • $ python2.7 setup.py install
  • $ python2.6 setup.py install
  • $ python3.3 setup.py install

Installation of VirtualEnv

  • Commands:
    • pip-2.6 install virtualenv
    • pip-2.7 install virtualenv
    • pip-3.3 install virtualenv
  • Notes
    • This will install to the Python instance location - which in our case is ~/.local.  However, if we weren't using our own local Python instance and still wanted to install this software to .local we could do that with the --user pip option.

Installation of VirtualEnvWrapper

  • Commands:
    • pip-2.6 install virtualenvwrapper
    • pip-2.7 install virtualenvwrapper
    • pip-3.3 install virtualenvwrapper
  • Notes
    • This will install to the Python instance location - which in our case is ~/.local.  However, if we weren't using our own local Python instance and still wanted to install this software to .local we could do that with the --user pip option.
    • http://virtualenvwrapper.readthedocs.org/en/latest/


Create Virtual Environment Directory

  • $ mkdir ~/Envs

Create Environmental Variables in .bashrc or equiv

  • export WORKON_HOME=~/Projects
  • source ~/.local/bin/virtualenvwrapper.sh

Run your .bashrc

  • Either exit and log back in, or rerun your .bashrc, etc:
    • Fixes your path and points to your new Python version 
    • Gets all the environmental variables defined for you
    • virtualenvwrapper will create default scripts in ~/Envs

Modify Virtual Environment Default  Scripts

  • Update ~/Envs/postmkvirtualenv - any command in this file will get run after the new environment is created.  I typically put commonly-used modules in here. 
    • export PYTHONPATH=
    • pip install argparse      # should be unnecessary with python2.7 & 3
    • pip install docopt       
    • pip install pylint       
    • pip install pep8         
    • pip install pytest       
    • pip install envoy        
    • pip install sh           
    • pip install appdirs      
    • pip install sqlalchemy   
    • pip install pyyaml        
  • Update ~/Envs/postactivate - any command here will be run after you select an environment work with ("activate" it)
    • cdvirtualenv              
      • starts you off in your virtualenv
  • Update ~/Envs/postdeactivate - any command here will be run after your environment 'deactivation'
    • export PYTHONPATH=         
      • resets any override from pre or post activate

Creating A Sample Environment for DataGristle

  • DataGristle  is a side project of mine, lets use it as an example from here down.
  • To create the DataGristle virtual environment for 3 versions of python:
    • $ mkvirtualenv --python python2.6 datagristle_py2.6
    • $ mkvirtualenv --python python2.7 datagristle_py2.7
    • $ mkvirtualenv --python python3.3 datagristle_py3.3

Modify Virtual Environment Scripts

  • Update scripts inside your environment with anything you want to override from the defaults you already created in the ~/Envs scripts.
  • For example, lets update ~/Envs/datagristle/bin/postactivate   
    • cdvirtualenv
      • Since we're overriding the default postactivate, we should include anything from the default version we want to keep. 
    • export PYTHONPATH=~/Envs/datagristle_py2.7 
      • This might be useful for testing
    • export PATH=~/Envs/datagristle_py2.7/scripts:$PATH   
      • This might be useful if you want to run these scripts from the command line
      • Note that $PATH will be changed back to its original value when you deactivate

Other Cool Commands

    • workon [environment name] -
      to activate an environment
    • deactivate - to deactivate an environment
    • workon - to list all environments
    • lssitepackage - to list all installed packages for an environment
    • rmvirtualenv - to remove a virtual environment
    • cdvirtualenv - to cd to a virtualenv
    • cdsitepackages -
      to cd to a virtualenv site packages directory
    • toggleglobalsitepackages  -
      to turn global site packages on/off in python's path

 Futher References



  1. would it be hard to adapt this approach to a Windows environment ?

  2. I *think* the general approach is probably similar. However, I don't have a windows box to test this on.

    Also, it would probably warrant a separate guide for linux vs windows vs (windows + cygwin).

  3. Thanks for this guide. I found it most useful. Perhaps it could be added to the Hitchhikers guide.

  4. Thanks for the comments Lewis. I've been updating this page as well as the one on multiple python versions as I get input from others. Once I get all that input reviewed and tested out again I'll approach Kenneth Reitz to see if there's some value in adding this (in some format) to his Hitchiker's Guide.

  5. I really appreciate you for sharing this valuable information.I think you are a really skilled blogger.This information can help lot of user to update their knowledge.Thanks for sharing.keep sharing more blogs.

    Python Online Training

  6. Hi.!! You have been post an very informative information. And i am very happy to read your blog post. I found so many interesting stuff on your blog share. keep going. thanks for share your ideas for our vision....
    Java Training in Chennai | Software Testing Training in Chennai

  7. Hi, you have given really informative post. Thanks for sharing this post to our vision. Learn Python will helps you to reach your goal.
    Python Training in Marathahalli
    Dot Net Training in Chennai

  8. I ‘d mention that most of us visitors are endowed to exist in a fabulous place with very many wonderful individuals with very helpful things.

    python training in bangalore|

  9. ⦁ I simply wanted to write down a quick word to say thanks to you for those wonderful tips and hints you are showing on this site"Oracle Training in Bangalore"

  10. This comment has been removed by the author.

  11. Thanks a lot very much for the high your blog post quality and results-oriented help. I won’t think twice to endorse to anybody who wants and needs support about this area. Besant technologies Marathahalli
    Besant technologies Marathahalli

  12. Extraordinary explanation was given to the Dot Net Framework’s new features. Each and every framework was clearly understood through your examples. Keep updating more information on .Net frameworks.
    DOT NET Training in Chennai
    DOT NET Course in Chennai

  13. From many algorithms available, Panda and penguin is the most important algorithm than others. The impact on ranking will severe when these both panda and penguin roll outs whereas other algorithms updates impact will be minor.
    Digital Marketing Training in Chennai | Digital Marketing Chennai

  14. This comment has been removed by the author.

  15. I accept there are numerous more pleasurable open doors ahead for people that took a gander at your site.
    big data training in Chennai

  16. nice information its very useful to read your blog. We provide best Digital Transformation Services

  17. Very good informative article. Thanks for sharing such nice article, keep on up dating such good articles.
    Austere Technologies | Best Cloud Solution services


    NO.1 IOT Services | INTERNET OF THINGS | Best IOT Services |

  19. Hi, am a big follower of your blog. I am really happy to found such a helpful and fascinating post that is written in well manner.

    white label website builder

  20. wow...nice blog, very help full information. Thanks for sharing.

    Best Mobility Services | Austere Technologies

  21. Your new valuable key points simply much a person like me and extremely more to my office workers. With thanks; from every one of us.
    Best Python training Institute in chennai

  22. Python is one of the most significant to learn for the future.

    Python training in Chennai
    Big data training in Chennai

  23. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.
    python training in Chennai

  24. I simply wanted to write down a quick word to say thanks to you for those wonderful tips and hints you are showing on this site.
    It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.

    Python Training in Chennai | Python Training Institutes in Chennai

  25. Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.
    Python Training in Bangalore

  26. Nice post..Keep updating..
    Summer Course Training for Graphic Designing in Chennai | <a href= "https://www.alltechzsolutions.in/summer-courses-in-chennai.phpSummer Course Training for Graphic Designing in Velachery</a>