Building My Egg Repository

I'm not certain whether this is the proper approach or not, I had a bit
of a tough time finding documentation detailing how to do this.

Our account provisioning system is composed of a twistd daemon and
a series of eggs which plug in to it. Each egg is responsible
for exporting a series of methods which can be exposed via
XMLRPC or SOAP. It works pretty well in that adding a new service
is as simple as adding a new egg.

I had been manually installing all of the egg files via direct
URLs each time I updated or changed the software. I've also
manually installed the required versions directly onto my
buildbot system. Managing it was getting slightly difficult as
the number of packages has been growing steadily.

Over the past few days, I've deployed my own egg repository. I've
updated my buildbot rules to copy over all newly built eggs into it
as tests complete.

Deploying the repository itself was quite easy. I simply
created a directory below my Apache document root for for each
package I want to serve, for example:

mkdir /var/www/html/eggs/hostapi.core
mkdir /var/www/html/eggs/hostapi.platform
mkdir /var/www/html/eggs/hostapi.mysql

The next step was to update the Apache configuration to allow
for directory listings.

Options Indexes ExecCGI

Lastly, I've updated my buildbot master.cfg script for each
component. After a build completes, we simply copy all of
the eggs in the dist/ directory into each project's eggs
location.

 f1.addStep(ShellCommand(
          command="cp -vf dist/*.egg /var/www/html/eggs/hostapi.core"))

Now, I can install all of my packages directly via easy_install, which
eliminates a lot of the deployment burden

[root@virtapi01 ~]# easy_install -Ui http://buildslave01/eggs hostapi.core
Searching for hostapi.core
Reading http://buildslave01/eggs/hostapi.core/
Reading http://buildslave01/eggs/hostapi.core/?C=S;O=A
Reading http://buildslave01/eggs/hostapi.core/?C=D;O=A
Reading http://buildslave01/eggs/hostapi.core/?C=M;O=A
Reading http://buildslave01/eggs/hostapi.core/?C=N;O=D
Best match: hostapi.core 1.0-r75
Processing hostapi.core-1.0_r75-py2.4.egg
hostapi.core 1.0-r75 is already the active version in easy-install.pth

Using /usr/lib/python2.4/site-packages/hostapi.core-1.0_r75-py2.4.egg
Reading http://buildslave01/eggs
Processing dependencies for hostapi.core
Finished processing dependencies for hostapi.core
[root@virtapi01 ~]#

All seemed well and good, so I updated the setup.py scripts for each
package to refer to a deployment URL rather than rely on locally
installed packages. Time for a test build.

running test
Checking .pth file support in .
/usr/bin/python -E -c pass
Searching for hostapi.core
Reading http://buildslave01/eggs
Reading http://pypi.python.org/simple/hostapi.core/
Couldn't find index page for 'hostapi.core' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
No local packages or download links found for hostapi.core
error: Could not find suitable distribution for Requirement.parse('hostapi.core')
program finished with exit code 1
elapsedTime=2.605963

Ak! Wha? Not only did it not find my eggs, it sent requests to PyPI that
aren't going to do any good. In order to fix the problem, I created
the following little index.cgi script, which I've placed within my eggs
directory on the web server.

#!/usr/bin/python

import os
import cgitb
cgitb.enable()

print 'Content-type: text/html'
print
for dir in os.listdir('.'):
   try:
       for egg in os.listdir(dir):
           if egg.endswith('.egg'):
               print "%s" % (dir, egg, egg)
           elif egg.endswith('svn'):
               print open(egg).read()
   except OSError: pass

Now everything seems to be happy. My buildbot CI builds work
correctly as they can find their dependancy links, my easy_install
command lines work correctly as I've built a 'fake' PyPI, and I work
a little more correctly as I'm not copying egg files around any longer!

In order to reduce pull on PyPI, I'm actually locally hosting copies
of BeautifulSoup, zope.*, and some SQL libraries. It also helps to
ensure I don't get an unexpected update.

Is there a better way to do any of this? This seems to work pretty
well. I'm currently keeping two of these repositories. The first
one gets new eggs with each CI build while the second one only
contains eggs which have cleared QA and are production ready