Working With Multiple Versions of Python

I've needed to run multiple python versions for a while now, and have finally bitten the bullet and invested the time to get this working.    The primary drivers of this need are two separate projects:

• At my office we're running Redhat Enterprise Linux 5.8 which uses Python 2.4.  We've got an alternate install of Python 2.6 available, but I'd like to move immediately to Python 2.7 and in six months to Python 3.3.   During the transition I may have some components running on one version and others running on another.   This will help me incrementally move over.
• One of my side projects, DataGristle, was written on Python 2.6, but is being used on Python 2.6 & Python 2.7 environments.   Testing it to ensure it works correctly on both previously required a second machine.   I'm getting tired of that and looking forward to eventual Python 3.1, 3.2, 3.3 compatibility.

But I've discovered that the documentation for this seems to only exist in bits and pieces distributed around the net.  The closest would be The Hitchhiker's Guide to Python, but this doesn't address multiple python versions and doesn't generally have command-by-command configuration instructions. The following information is intended to help put together a complementary, consolidated recipe to make this easier for a beginning Python developer on linux.

So, this is the first of two series about working with many environments.  The second series is about Working with Multiple Environments.

I've been updating these articles as update (aka corrections) are coming in, and experimentation with clean environments via VirtualBox slowly moves forward.  Thanks for the corrections and patience.

Installation Options Include:

• Installation via a package manager such as apt-get or yum
• Installation via Pythonbrew
• Installation via source code compile & make

All of these methods can potentially run into unintended system conflicts with system applications using python.  Especially if they use env.  Generally, speaking they should be directly referencing the python version they use and care on install can avoid most problems.  Lets get into the details of each.

Installation via Package Manager  

Details on this method are provided in the second part of this series here.

Installation via PythonBrew

Details on this method are provided in the second part of this series here.

Installation via Source Code

That leaves compiling it yourself and installing it to a local directory as the most practical solution, at least in my case.   This leaves only these small issues:

• Updates - you'll need to keep track of updates to Python versions and apply them manually.
• 'Make install' vs 'make altinstall' - make_install will create a local copy of the Python executable as well as one with the version suffix.  Make altinstall will only create the the copy with the version suffix.  You probably want one of your versions to be the default version, need to identify which one, and do a normal 'make install' on that and 'make altinstall' on everything else.
• Make dependencies - you may need to first install certain packages in order to include their functionality in your python executable.   This can be labor-intensive to track down.

 More info is available here.

The installation steps:

1. Install dependencies - here's a practical minimum subset
• On ubuntu:
• $ sudo apt-get install build-essential checkinstall
• $ sudo apt-get install zlib1g-dev
• if on 64-bit: 

   $ sudo ln -s /lib/x86_64-linux-gnu/libz.so.1 /lib/libz.so

• $ sudo apt-get install libreadline-dev
• $ sudo apt-get install sqlite3
• $ sudo apt-get install libsqlite3-dev 
• $ sudo apt-get install libbz2-dev 
• $ sudo apt-get install libssl-dev  
• On Redhat:
• $ sudo yum install readline-devel

1. Determine the version and its url
• http://www.python.org/download/
2. Download the software
• $ cd ~/Downloads
• $ wget http://www.python.org/ftp/python/X.Y.Z/Python-X.Y.Z.tgz
3. Confirm software
• $ md5sum Python-X.Y.Z.tgz
• the resulting number should be identical to md5 checksum on python download page
4. Unpack:
• $ tar -xvf Python-X.Y.Z.tgz
• $ cd ~/home/kenfar/Downloads/Python-X.Y.Z
5. Install:
• $ configure --prefix="/home/kenfar/.local"
• $ make clean (only if this is a repeat attempt)
• $ make
• Watch out for message: "Failed to find the necessary bits to build these modules" followed by a list of modules.  Especially, if zlib is on that list - because this will prevent other installations.  If this is the case, go back to step #1, and install build-essential and zlib.
• $ make install (for primary version) - or $ make altinstall (for everything else)
• Suggestion - log output and keep this in case you have to diagnose problems later.  Ex:  make &> make.log, then copy to ~/logs/python_build
6. Validation
• You should hypothetically check the config.log file, but it'll be about 20,000 lines long.  So, you might not see anything there.
• Also note make errors - zlib will cause quite a few issues.

Specific versions change over time, but here's some at the time of this writing, along with compile issues I encountered on Ubuntu 12.04

• Python 2.6  - currently, this is the most recent:  http://www.python.org/ftp/python/2.6.8/Python-2.6.8.tgz
• Make errors on Ubuntu: Failed to find the necessary bits to build these modules:
  _bsddb             _curses            _curses_panel  
  _hashlib           _tkinter           bsddb185           dbm                dl                 gdbm               imageop            linuxaudiodev      ossaudiodev    
  sunaudiodev     
  
• Python 2.7 - currently, this is the most recent:  http://python.org/ftp/python/2.7.3/Python-2.7.3.tgz
• Python build finished, but the necessary bits to build these modules were not found:
  _bsddb             _curses         _curses_panel                        _tkinter           bsddb185           dbm                dl                 gdbm               imageop        
  sunaudiodev       
• Python 3.3 - currently, this is the most recent:  http://python.org/ftp/python/3.3.0/Python-3.3.0.tgz
• Python build finished, but the necessary bits to build these modules were not found:                                                                                _curses            _curses_panel       _dbm               _gdbm              _lzma               _tkinter       
      
• Some of these missing modules can come back to haunt you - with errors that might not always be very intuitive.   So, it's definitely worth-while to eliminate any that you think you are likely to run into.  For me that's zlib, bz2, _sqlite3, _ssl.  Typically, resolution just requires figuring out the name of package to install on your os, and installing it via your package manager.   The challenge is when you don't have root access, need to build the package, install to a custom location - and need to communicate that location to Python.

The final configuration looks like this:

kenfar@watson:~/.local/bin$
lrwxrwxrwx 1 kenfar kenfar       8 Jan 20 19:56 2to3 -> 2to3-3.3
-rwxrwxr-x 1 kenfar kenfar     110 Jan 20 19:56 2to3-3.3
-rwxrwxr-x 1 kenfar kenfar     108 Jan 20 13:25 idle
lrwxrwxrwx 1 kenfar kenfar       7 Jan 20 19:56 idle3 -> idle3.3
-rwxrwxr-x 1 kenfar kenfar     108 Jan 20 19:56 idle3.3
-rwxrwxr-x 1 kenfar kenfar      93 Jan 20 13:25 pydoc
lrwxrwxrwx 1 kenfar kenfar       8 Jan 20 19:56 pydoc3 -> pydoc3.3
-rwxrwxr-x 1 kenfar kenfar      93 Jan 20 19:56 pydoc3.3
lrwxrwxrwx 1 kenfar kenfar       7 Jan 20 19:52 python -> python2
lrwxrwxrwx 1 kenfar kenfar       9 Jan 20 19:52 python2 -> python2.7
-rwxr-xr-x 1 kenfar kenfar 6123100 Jan 20 13:26 python2.6
-rwxr-xr-x 1 kenfar kenfar    1433 Jan 20 13:26 python2.6-config
-rwxr-xr-x 1 kenfar kenfar 6519369 Jan 20 19:52 python2.7
-rwxr-xr-x 1 kenfar kenfar    1633 Jan 20 19:52 python2.7-config
lrwxrwxrwx 1 kenfar kenfar      16 Jan 20 19:52 python2-config -> python2.7-config
lrwxrwxrwx 1 kenfar kenfar       9 Jan 20 19:56 python3 -> python3.3
-rwxr-xr-x 2 kenfar kenfar 8503848 Jan 20 19:55 python3.3
lrwxrwxrwx 1 kenfar kenfar      17 Jan 20 19:56 python3.3-config -> python3.3m-config
-rwxr-xr-x 2 kenfar kenfar 8503848 Jan 20 19:55 python3.3m
-rwxr-xr-x 1 kenfar kenfar    1979 Jan 20 19:56 python3.3m-config
lrwxrwxrwx 1 kenfar kenfar      16 Jan 20 19:56 python3-config -> python3.3-config
lrwxrwxrwx 1 kenfar kenfar      14 Jan 20 19:52 python-config -> python2-config
lrwxrwxrwx 1 kenfar kenfar      10 Jan 20 19:56 pyvenv -> pyvenv-3.3
-rwxrwxr-x 1 kenfar kenfar     245 Jan 20 19:56 pyvenv-3.3
-rwxrwxr-x 1 kenfar kenfar   18930 Jan 20 13:25 smtpd.py

Usage of these new versions of Python is very straight-forward:

• Add to your .bashrc:
• export PATH=~/.local/bin:$PATH
• At this point we can access Python 2.7 via python, python2 or python2.7; Python 2.6 via python2.6; and Python 3.3 via python3 or python3.3
• When you install Virtualenv and Virtualenvwrapper they'll be able to easily choose between different versions of Python - and copy the needed one directly into each environment.
• And most importantly, this doesn't interfere at all with system operations.

Well, one Gotcha:

• Applications that use the wonderful env program in their shebang (ex: #!/usr/bin/env python) that are part of the system or that you've installed - could pick up your new version of Python!
• The Debian community has identified the use of env as an issue - that system software should specify the major version of Python needed instead.  However, this advice isn't always followed.
• You generally won't notice much of a problem.  And especially if you're just adding Python 2.7 to a system that comes with Python 2.6.   
• But if you add Python 3.3 to a Python 2.7 system, or Python 2.6 to a Python 2.7 system AND you use make install rather than make altinstall - then you could run into complications with misbehaving software.
• I think the safest approach is to make the same version or a minor upgrade, and make alt the other versions.