of a tough time finding documentation detailing how to do this.
Our account provisioning system is composed of a twistd daemon and
a series of eggs which plug in to it. Each egg is responsible
for exporting a series of methods which can be exposed via
XMLRPC or SOAP. It works pretty well in that adding a new service
is as simple as adding a new egg.
I had been manually installing all of the egg files via direct
URLs each time I updated or changed the software. I've also
manually installed the required versions directly onto my
buildbot system. Managing it was getting slightly difficult as
the number of packages has been growing steadily.
Over the past few days, I've deployed my own egg repository. I've
updated my buildbot rules to copy over all newly built eggs into it
as tests complete.
Deploying the repository itself was quite easy. I simply
created a directory below my Apache document root for for each
package I want to serve, for example:
mkdir /var/www/html/eggs/hostapi.coreThe next step was to update the Apache configuration to allow
for directory listings.
Options Indexes ExecCGILastly, I've updated my buildbot master.cfg script for each
component. After a build completes, we simply copy all of
the eggs in the dist/ directory into each project's eggs
f1.addStep(ShellCommand(Now, I can install all of my packages directly via easy_install, which
command="cp -vf dist/*.egg /var/www/html/eggs/hostapi.core"))
eliminates a lot of the deployment burden
[root@virtapi01 ~]# easy_install -Ui http://buildslave01/eggs hostapi.coreAll seemed well and good, so I updated the setup.py scripts for each
Searching for hostapi.core
Best match: hostapi.core 1.0-r75
hostapi.core 1.0-r75 is already the active version in easy-install.pth
Processing dependencies for hostapi.core
Finished processing dependencies for hostapi.core
package to refer to a deployment URL rather than rely on locally
installed packages. Time for a test build.
running testAk! Wha? Not only did it not find my eggs, it sent requests to PyPI that
Checking .pth file support in .
/usr/bin/python -E -c pass
Searching for hostapi.core
Couldn't find index page for 'hostapi.core' (maybe misspelled?)
Scanning index of all packages (this may take a while)
No local packages or download links found for hostapi.core
error: Could not find suitable distribution for Requirement.parse('hostapi.core')
program finished with exit code 1
aren't going to do any good. In order to fix the problem, I created
the following little index.cgi script, which I've placed within my eggs
directory on the web server.
#!/usr/bin/pythonNow everything seems to be happy. My buildbot CI builds work
print 'Content-type: text/html'
for dir in os.listdir('.'):
for egg in os.listdir(dir):
print "%s" % (dir, egg, egg)
except OSError: pass
correctly as they can find their dependancy links, my easy_install
command lines work correctly as I've built a 'fake' PyPI, and I work
a little more correctly as I'm not copying egg files around any longer!
In order to reduce pull on PyPI, I'm actually locally hosting copies
of BeautifulSoup, zope.*, and some SQL libraries. It also helps to
ensure I don't get an unexpected update.
Is there a better way to do any of this? This seems to work pretty
well. I'm currently keeping two of these repositories. The first
one gets new eggs with each CI build while the second one only
contains eggs which have cleared QA and are production ready