I've moved my blog to jmcneil.net. This is no longer being updated!

Friday, November 14, 2008

Building My Egg Repository

I'm not certain whether this is the proper approach or not, I had a bit
of a tough time finding documentation detailing how to do this.

Our account provisioning system is composed of a twistd daemon and
a series of eggs which plug in to it. Each egg is responsible
for exporting a series of methods which can be exposed via
XMLRPC or SOAP. It works pretty well in that adding a new service
is as simple as adding a new egg.

I had been manually installing all of the egg files via direct
URLs each time I updated or changed the software. I've also
manually installed the required versions directly onto my
buildbot system. Managing it was getting slightly difficult as
the number of packages has been growing steadily.

Over the past few days, I've deployed my own egg repository. I've
updated my buildbot rules to copy over all newly built eggs into it
as tests complete.

Deploying the repository itself was quite easy. I simply
created a directory below my Apache document root for for each
package I want to serve, for example:
mkdir /var/www/html/eggs/hostapi.core
mkdir /var/www/html/eggs/hostapi.platform
mkdir /var/www/html/eggs/hostapi.mysql
The next step was to update the Apache configuration to allow
for directory listings.
Options Indexes ExecCGI
Lastly, I've updated my buildbot master.cfg script for each
component. After a build completes, we simply copy all of
the eggs in the dist/ directory into each project's eggs
command="cp -vf dist/*.egg /var/www/html/eggs/hostapi.core"))
Now, I can install all of my packages directly via easy_install, which
eliminates a lot of the deployment burden
[root@virtapi01 ~]# easy_install -Ui http://buildslave01/eggs hostapi.core
Searching for hostapi.core
Reading http://buildslave01/eggs/hostapi.core/
Reading http://buildslave01/eggs/hostapi.core/?C=S;O=A
Reading http://buildslave01/eggs/hostapi.core/?C=D;O=A
Reading http://buildslave01/eggs/hostapi.core/?C=M;O=A
Reading http://buildslave01/eggs/hostapi.core/?C=N;O=D
Best match: hostapi.core 1.0-r75
Processing hostapi.core-1.0_r75-py2.4.egg
hostapi.core 1.0-r75 is already the active version in easy-install.pth

Using /usr/lib/python2.4/site-packages/hostapi.core-1.0_r75-py2.4.egg
Reading http://buildslave01/eggs
Processing dependencies for hostapi.core
Finished processing dependencies for hostapi.core
[root@virtapi01 ~]#
All seemed well and good, so I updated the setup.py scripts for each
package to refer to a deployment URL rather than rely on locally
installed packages. Time for a test build.
running test
Checking .pth file support in .
/usr/bin/python -E -c pass
Searching for hostapi.core
Reading http://buildslave01/eggs
Reading http://pypi.python.org/simple/hostapi.core/
Couldn't find index page for 'hostapi.core' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
No local packages or download links found for hostapi.core
error: Could not find suitable distribution for Requirement.parse('hostapi.core')
program finished with exit code 1
Ak! Wha? Not only did it not find my eggs, it sent requests to PyPI that
aren't going to do any good. In order to fix the problem, I created
the following little index.cgi script, which I've placed within my eggs
directory on the web server.

import os
import cgitb

print 'Content-type: text/html'
for dir in os.listdir('.'):
for egg in os.listdir(dir):
if egg.endswith('.egg'):
print "%s" % (dir, egg, egg)
elif egg.endswith('svn'):
print open(egg).read()
except OSError: pass
Now everything seems to be happy. My buildbot CI builds work
correctly as they can find their dependancy links, my easy_install
command lines work correctly as I've built a 'fake' PyPI, and I work
a little more correctly as I'm not copying egg files around any longer!

In order to reduce pull on PyPI, I'm actually locally hosting copies
of BeautifulSoup, zope.*, and some SQL libraries. It also helps to
ensure I don't get an unexpected update.

Is there a better way to do any of this? This seems to work pretty
well. I'm currently keeping two of these repositories. The first
one gets new eggs with each CI build while the second one only
contains eggs which have cleared QA and are production ready


stephan said...

You might want to check out EggBasket.

Jeff McNeil said...

I've not seen that before, thanks. I'll yank it down and toy with it a bit. I don't really need permissions or upload support just yet, but this seems more solid than my collection of directories.

chrism said...

There's also "basketweaver" on PyPI (which has a makeindex script that just reads a bunch of egg distros or .tar.gz or .zip distros) and makes an index on disk for them that you can serve up via Apache. It's not a server, just a command-line utility.

Jeff McNeil said...

That sounds a lot like what I need.... I don't really want to run a special process for it as we're already using Apache for Yum repositories. I just want something to generate the structure and whatnot.

Nathen said...

you can restrict your easy_install to only a specific domain with -H*your_domain.tld switch.

Jeff McNeil said...

Yeah, I can pass it in when I use easy_install from the command line, but I wasn't sure how to do that when running 'python setup.py test' and triggering the nose test runner via a buildbot rule.

I couldn't find a way to pass/set options like that. I probably simply didn't look deep enough.