2013-01-21

Working With Multiple Environments in Python


Working within a single Python environment in which you're limited to a single version of python and a single version of all your modules is fine for small projects, but as the number of projects or age of projects increase it leaves you boxed in.  You can't test your software against a new version of a dependency without putting everything else on hold, and then you may have to revert back.  Eventually, you'll end up doing insufficient testing or spending far too much time on installing, uninstalling, and developing projects in series rather than parallel.

Ian Bicking's Virtualenv changed this in 2007 with the ability to create multiple environments for a Python application.  Then a year later Doug Hellman wrote Virtualenvwrapper which added a set of convenience functions that significantly enhance virtualenv.

The documentation for these products is generally very good, but it seems to me that something is missing - documentation that pulls everything together across products so that the beginning user has a simple recipe for success.  The closest would be The Hitchhiker's Guide to Python, but this doesn't address multiple Python versions.   And I really want to nail down how to do this with multiple versions of Python.

The following information is intended to be a first step in that direction for the linux developer.



Installation of multiple versions of Python

Supporting more than a single Python version is an option that takes a little extra work and provides quite a bit more flexibility.  I covered this in my last article, with a detailed description of installing via source code compile.


Installation of Python Distribute

  • $ mkdir ~/Downloads; cd ~/Downloads
  • $ curl -O http://python-distribute.org/distribute_setup.py
  • $ python2.6 distribute_setup.py
  • $ python2.7 distribute_setup.py
  • $ python3.3 distribute_setup.py

Installation of Pip

  • cd ~/Downloads
  • Download the most recent version of pip.
    • Get it from http://pypi.python.org/pypi/pip.
    • Most recent version is currently pip-1.2.1.tar.gz.
    • wget http://pypi.python.org/packages/source/p/pip/pip-1.2.1.tar.gz
  • $ tar -xvf pip-1.2.1.tar.gz
  • $ cd ~/Downloads/pip-1.2.1
  • $ python2.7 setup.py install
  • $ python2.6 setup.py install
  • $ python3.3 setup.py install

Installation of VirtualEnv

  • Commands:
    • pip-2.6 install virtualenv
    • pip-2.7 install virtualenv
    • pip-3.3 install virtualenv
  • Notes
    • This will install to the Python instance location - which in our case is ~/.local.  However, if we weren't using our own local Python instance and still wanted to install this software to .local we could do that with the --user pip option.

Installation of VirtualEnvWrapper

  • Commands:
    • pip-2.6 install virtualenvwrapper
    • pip-2.7 install virtualenvwrapper
    • pip-3.3 install virtualenvwrapper
  • Notes
    • This will install to the Python instance location - which in our case is ~/.local.  However, if we weren't using our own local Python instance and still wanted to install this software to .local we could do that with the --user pip option.
    • http://virtualenvwrapper.readthedocs.org/en/latest/

 

Create Virtual Environment Directory

  • $ mkdir ~/Envs

Create Environmental Variables in .bashrc or equiv

  • export WORKON_HOME=~/Projects
  • export VIRTUALENVWRAPPER_LOG_DIR="$WORKON_HOME"
  • export VIRTUALENVWRAPPER_HOOK_DIR="$WORKON_HOME"
  • source ~/.local/bin/virtualenvwrapper.sh
  • export PIP_VIRTUALENV_BASE=$WORKON_HOME
  • export PIP_RESPECT_VIRTUALENV=true

Run your .bashrc

  • Either exit and log back in, or rerun your .bashrc, etc:
    • Fixes your path and points to your new Python version 
    • Gets all the environmental variables defined for you
    • virtualenvwrapper will create default scripts in ~/Envs

Modify Virtual Environment Default  Scripts

  • Update ~/Envs/postmkvirtualenv - any command in this file will get run after the new environment is created.  I typically put commonly-used modules in here. 
    • export PYTHONPATH=
    • pip install argparse      # should be unnecessary with python2.7 & 3
    • pip install docopt       
    • pip install pylint       
    • pip install pep8         
    • pip install pytest       
    • pip install envoy        
    • pip install sh           
    • pip install appdirs      
    • pip install sqlalchemy   
    • pip install pyyaml        
  • Update ~/Envs/postactivate - any command here will be run after you select an environment work with ("activate" it)
    • cdvirtualenv              
      • starts you off in your virtualenv
  • Update ~/Envs/postdeactivate - any command here will be run after your environment 'deactivation'
    • export PYTHONPATH=         
      • resets any override from pre or post activate
     

Creating A Sample Environment for DataGristle

  • DataGristle  is a side project of mine, lets use it as an example from here down.
  • To create the DataGristle virtual environment for 3 versions of python:
    • $ mkvirtualenv --python python2.6 datagristle_py2.6
    • $ mkvirtualenv --python python2.7 datagristle_py2.7
    • $ mkvirtualenv --python python3.3 datagristle_py3.3

Modify Virtual Environment Scripts

  • Update scripts inside your environment with anything you want to override from the defaults you already created in the ~/Envs scripts.
  • For example, lets update ~/Envs/datagristle/bin/postactivate   
    • cdvirtualenv
      • Since we're overriding the default postactivate, we should include anything from the default version we want to keep. 
    • export PYTHONPATH=~/Envs/datagristle_py2.7 
      • This might be useful for testing
    • export PATH=~/Envs/datagristle_py2.7/scripts:$PATH   
      • This might be useful if you want to run these scripts from the command line
      • Note that $PATH will be changed back to its original value when you deactivate

Other Cool Commands

    • workon [environment name] -
      to activate an environment
    • deactivate - to deactivate an environment
    • workon - to list all environments
    • lssitepackage - to list all installed packages for an environment
    • rmvirtualenv - to remove a virtual environment
    • cdvirtualenv - to cd to a virtualenv
    • cdsitepackages -
      to cd to a virtualenv site packages directory
    • toggleglobalsitepackages  -
      to turn global site packages on/off in python's path

 Futher References




  



4 comments:

  1. would it be hard to adapt this approach to a Windows environment ?

    ReplyDelete
  2. I *think* the general approach is probably similar. However, I don't have a windows box to test this on.

    Also, it would probably warrant a separate guide for linux vs windows vs (windows + cygwin).

    ReplyDelete
  3. Thanks for this guide. I found it most useful. Perhaps it could be added to the Hitchhikers guide.

    ReplyDelete
  4. Thanks for the comments Lewis. I've been updating this page as well as the one on multiple python versions as I get input from others. Once I get all that input reviewed and tested out again I'll approach Kenneth Reitz to see if there's some value in adding this (in some format) to his Hitchiker's Guide.

    ReplyDelete