Setuptools Egg Repository

I've been in the process of moving all of our account management code away from a home brewed XMLRPC system and on to a Twisted instance. At first, we'll probably defer most of our calls in threads, but over time, I hope to make everything a bit more asynchronous in nature.

It uses a plug-in architecture. We've one main RPM we deploy (bdist_rpm) that grafts itself into the twisted/plugins directory. That component is then responsible for loading our actual account provisioning code.

I've been bundling each plug-in as an egg file. In order to make deployment easier, I wanted to run my own egg repository.

The simplest method is to dump all of our eggs into a directory and enable directory indexing within Apache. This gives us an automatically generated index of egg files that 'easy_install -f URL' can read from.

In our configuration, buildbot does the publishing. I've the setup.cfg setup such that it tags each egg with the SVN revision number. After the build has passed testing, buildbot copies the new egg into an HTTP accessible directory.

The problem I ran into is that I wanted to make all of our SVN code available to developers, but still rely on the easy directory index approach.

My solution?
[root@buildbot-slave01 eggs]# cat lister.cgi

#!/usr/bin/python

import os
import cgitb
cgitb.enable()

print 'Content-type: text/html'
print

for egg in os.listdir('.'):
   if egg.endswith('.egg'):
       print "<a href="%s">%s</a>" % (egg, egg)
   elif egg.endswith('svn'):
       print open(egg).read()

[root@buildbot-slave01 eggs]#

So now, I've included little text files named 'package.svn' that simply contain setuptools #package-dev links:

[root@buildbot-slave01 eggs]# cat package.svn
<a href="http://buildbot:password@svn/package/trunk#egg=package.dev">
      package trunk</a>

This results in an index that includes our buildbot tagged packages that we'll use by default, and our SVN links that are available for devlopers to follow.

Installing is as easy as:

easy_install -f http://buildbot/eggs/lister.cgi package

In turn, I've setup the unit tests to require certain eggs that exist in my repository. The buildbot update method is 'clobber', so each automated built results in the latest egg file being installed in the local build directory.

Very cool!