I've needed to run multiple python versions for a while now, and have finally bitten the bullet and invested the time to get this working. The primary drivers of this need are two separate projects:
- At my office we're running Redhat Enterprise Linux 5.8 which uses Python 2.4. We've got an alternate install of Python 2.6 available, but I'd like to move immediately to Python 2.7 and in six months to Python 3.3. During the transition I may have some components running on one version and others running on another. This will help me incrementally move over.
- One of my side projects, DataGristle, was written on Python 2.6, but is being used on Python 2.6 & Python 2.7 environments. Testing it to ensure it works correctly on both previously required a second machine. I'm getting tired of that and looking forward to eventual Python 3.1, 3.2, 3.3 compatibility.
So, this is the first of two series about working with many environments. The second series is about Working with Multiple Environments.
I've been updating these articles as update (aka corrections) are coming in, and experimentation with clean environments via VirtualBox slowly moves forward. Thanks for the corrections and patience.
Installation Options Include:
- Installation via a package manager such as apt-get or yum
- Installation via Pythonbrew
- Installation via source code compile & make
All of these methods can potentially run into unintended system conflicts with system applications using python. Especially if they use env. Generally, speaking they should be directly referencing the python version they use and care on install can avoid most problems. Lets get into the details of each.
Installation via Package ManagerDetails on this method are provided in the second part of this series here.
Installation via PythonBrewDetails on this method are provided in the second part of this series here.
Installation via Source CodeThat leaves compiling it yourself and installing it to a local directory as the most practical solution, at least in my case. This leaves only these small issues:
- Updates - you'll need to keep track of updates to Python versions and apply them manually.
- 'Make install' vs 'make altinstall' - make_install will create a local copy of the Python executable as well as one with the version suffix. Make altinstall will only create the the copy with the version suffix. You probably want one of your versions to be the default version, need to identify which one, and do a normal 'make install' on that and 'make altinstall' on everything else.
- Make dependencies - you may need to first install certain packages in order to include their functionality in your python executable. This can be labor-intensive to track down.
The installation steps:
- Install dependencies - here's a practical minimum subset
- On ubuntu:
- $ sudo apt-get install build-essential checkinstall
- $ sudo apt-get install zlib1g-dev
- if on 64-bit:
$ sudo ln -s /lib/x86_64-linux-gnu/libz.so.1 /lib/libz.so
- $ sudo apt-get install libreadline-dev
- $ sudo apt-get install sqlite3
- $ sudo apt-get install libsqlite3-dev
$ sudo apt-get install libbz2-dev
- $ sudo apt-get install libssl-dev
- On Redhat:
- $ sudo yum install readline-deve
- Determine the version and its url
- $ cd ~/Downloads
- $ wget http://www.python.org/ftp/python/X.Y.Z/Python-X.Y.Z.tgz
- $ md5sum Python-X.Y.Z.tgz
- the resulting number should be identical to md5 checksum on python download page
- $ tar -xvf Python-X.Y.Z.tgz
- $ cd ~/home/kenfar/Downloads/Python-X.Y.Z
- $ configure --prefix="/home/kenfar/.local"
- $ make clean (only if this is a repeat attempt)
- $ make
- Watch out for message: "Failed to find the necessary bits to build these modules" followed by a list of modules. Especially, if zlib is on that list - because this will prevent other installations. If this is the case, go back to step #1, and install build-essential and zlib.
- $ make install (for primary version) - or $ make altinstall (for everything else)
- Suggestion - log output and keep this in case you have to diagnose problems later. Ex: make &> make.log, then copy to ~/logs/python_build
- You should hypothetically check the config.log file, but it'll be about 20,000 lines long. So, you might not see anything there.
- Also note make errors - zlib will cause quite a few issues.
Specific versions change over time, but here's some at the time of this writing, along with compile issues I encountered on Ubuntu 12.04
- Python 2.6 - currently, this is the most recent: http://www.python.org/ftp/python/2.6.8/Python-2.6.8.tgz
- Make errors on Ubuntu: Failed to find the necessary bits to build these modules:
_bsddb _curses _curses_panel
_hashlib _tkinter bsddb185 dbm dl gdbm imageop linuxaudiodev ossaudiodev
- Python 2.7 - currently, this is the most recent: http://python.org/ftp/python/2.7.3/Python-2.7.3.tgz
- Python build finished, but the necessary bits to build these modules were not found:
_bsddb _curses _curses_panel _tkinter bsddb185 dbm dl gdbm imageop
- Python 3.3 - currently, this is the most recent: http://python.org/ftp/python/3.3.0/Python-3.3.0.tgz
- Python build finished, but the necessary bits to build these modules were not found: _curses _curses_panel _dbm _gdbm _lzma _tkinter
- Some of these missing modules can come back to haunt you - with errors that might not always be very intuitive. So, it's definitely worth-while to eliminate any that you think you are likely to run into. For me that's zlib, bz2, _sqlite3, _ssl. Typically, resolution just requires figuring out the name of package to install on your os, and installing it via your package manager. The challenge is when you don't have root access, need to build the package, install to a custom location - and need to communicate that location to Python.
The final configuration looks like this:kenfar@watson:~/.local/bin$
lrwxrwxrwx 1 kenfar kenfar 8 Jan 20 19:56 2to3 -> 2to3-3.3
-rwxrwxr-x 1 kenfar kenfar 110 Jan 20 19:56 2to3-3.3
-rwxrwxr-x 1 kenfar kenfar 108 Jan 20 13:25 idle
lrwxrwxrwx 1 kenfar kenfar 7 Jan 20 19:56 idle3 -> idle3.3
-rwxrwxr-x 1 kenfar kenfar 108 Jan 20 19:56 idle3.3
-rwxrwxr-x 1 kenfar kenfar 93 Jan 20 13:25 pydoc
lrwxrwxrwx 1 kenfar kenfar 8 Jan 20 19:56 pydoc3 -> pydoc3.3
-rwxrwxr-x 1 kenfar kenfar 93 Jan 20 19:56 pydoc3.3
lrwxrwxrwx 1 kenfar kenfar 7 Jan 20 19:52 python -> python2
lrwxrwxrwx 1 kenfar kenfar 9 Jan 20 19:52 python2 -> python2.7
-rwxr-xr-x 1 kenfar kenfar 6123100 Jan 20 13:26 python2.6
-rwxr-xr-x 1 kenfar kenfar 1433 Jan 20 13:26 python2.6-config
-rwxr-xr-x 1 kenfar kenfar 6519369 Jan 20 19:52 python2.7
-rwxr-xr-x 1 kenfar kenfar 1633 Jan 20 19:52 python2.7-config
lrwxrwxrwx 1 kenfar kenfar 16 Jan 20 19:52 python2-config -> python2.7-config
lrwxrwxrwx 1 kenfar kenfar 9 Jan 20 19:56 python3 -> python3.3
-rwxr-xr-x 2 kenfar kenfar 8503848 Jan 20 19:55 python3.3
lrwxrwxrwx 1 kenfar kenfar 17 Jan 20 19:56 python3.3-config -> python3.3m-config
-rwxr-xr-x 2 kenfar kenfar 8503848 Jan 20 19:55 python3.3m
-rwxr-xr-x 1 kenfar kenfar 1979 Jan 20 19:56 python3.3m-config
lrwxrwxrwx 1 kenfar kenfar 16 Jan 20 19:56 python3-config -> python3.3-config
lrwxrwxrwx 1 kenfar kenfar 14 Jan 20 19:52 python-config -> python2-config
lrwxrwxrwx 1 kenfar kenfar 10 Jan 20 19:56 pyvenv -> pyvenv-3.3
-rwxrwxr-x 1 kenfar kenfar 245 Jan 20 19:56 pyvenv-3.3
-rwxrwxr-x 1 kenfar kenfar 18930 Jan 20 13:25 smtpd.py
Usage of these new versions of Python is very straight-forward:
- Add to your .bashrc:
- export PATH=~/.local/bin:$PATH
- At this point we can access Python 2.7 via python, python2 or python2.7; Python 2.6 via python2.6; and Python 3.3 via python3 or python3.3
- When you install Virtualenv and Virtualenvwrapper they'll be able to easily choose between different versions of Python - and copy the needed one directly into each environment.
- And most importantly, this doesn't interfere at all with system operations.
Well, one Gotcha:
- Applications that use the wonderful env program in their shebang (ex: #!/usr/bin/env python) that are part of the system or that you've installed - could pick up your new version of Python!
- The Debian community has identified the use of env as an issue - that system software should specify the major version of Python needed instead. However, this advice isn't always followed.
- You generally won't notice much of a problem. And especially if you're just adding Python 2.7 to a system that comes with Python 2.6.
- But if you add Python 3.3 to a Python 2.7 system, or Python 2.6 to a Python 2.7 system AND you use make install rather than make altinstall - then you could run into complications with misbehaving software.
- I think the safest approach is to make the same version or a minor upgrade, and make alt the other versions.