Building My Egg Repository

I'm not certain whether this is the proper approach or not, I had a bit
of a tough time finding documentation detailing how to do this.

Our account provisioning system is composed of a twistd daemon and
a series of eggs which plug in to it. Each egg is responsible
for exporting a series of methods which can be exposed via
XMLRPC or SOAP. It works pretty well in that adding a new service
is as simple as adding a new egg.

I had been manually installing all of the egg files via direct
URLs each time I updated or changed the software. I've also
manually installed the required versions directly onto my
buildbot system. Managing it was getting slightly difficult as
the number of packages has been growing steadily.

Over the past few days, I've deployed my own egg repository. I've
updated my buildbot rules to copy over all newly built eggs into it
as tests complete.

Deploying the repository itself was quite easy. I simply
created a directory below my Apache document root for for each
package I want to serve, for example:

mkdir /var/www/html/eggs/hostapi.core
mkdir /var/www/html/eggs/hostapi.platform
mkdir /var/www/html/eggs/hostapi.mysql

The next step was to update the Apache configuration to allow
for directory listings.

Options Indexes ExecCGI

Lastly, I've updated my buildbot master.cfg script for each
component. After a build completes, we simply copy all of
the eggs in the dist/ directory into each project's eggs
location.

 f1.addStep(ShellCommand(
          command="cp -vf dist/*.egg /var/www/html/eggs/hostapi.core"))

Now, I can install all of my packages directly via easy_install, which
eliminates a lot of the deployment burden

[root@virtapi01 ~]# easy_install -Ui http://buildslave01/eggs hostapi.core
Searching for hostapi.core
Reading http://buildslave01/eggs/hostapi.core/
Reading http://buildslave01/eggs/hostapi.core/?C=S;O=A
Reading http://buildslave01/eggs/hostapi.core/?C=D;O=A
Reading http://buildslave01/eggs/hostapi.core/?C=M;O=A
Reading http://buildslave01/eggs/hostapi.core/?C=N;O=D
Best match: hostapi.core 1.0-r75
Processing hostapi.core-1.0_r75-py2.4.egg
hostapi.core 1.0-r75 is already the active version in easy-install.pth

Using /usr/lib/python2.4/site-packages/hostapi.core-1.0_r75-py2.4.egg
Reading http://buildslave01/eggs
Processing dependencies for hostapi.core
Finished processing dependencies for hostapi.core
[root@virtapi01 ~]#

All seemed well and good, so I updated the setup.py scripts for each
package to refer to a deployment URL rather than rely on locally
installed packages. Time for a test build.

running test
Checking .pth file support in .
/usr/bin/python -E -c pass
Searching for hostapi.core
Reading http://buildslave01/eggs
Reading http://pypi.python.org/simple/hostapi.core/
Couldn't find index page for 'hostapi.core' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
No local packages or download links found for hostapi.core
error: Could not find suitable distribution for Requirement.parse('hostapi.core')
program finished with exit code 1
elapsedTime=2.605963

Ak! Wha? Not only did it not find my eggs, it sent requests to PyPI that
aren't going to do any good. In order to fix the problem, I created
the following little index.cgi script, which I've placed within my eggs
directory on the web server.

#!/usr/bin/python

import os
import cgitb
cgitb.enable()

print 'Content-type: text/html'
print
for dir in os.listdir('.'):
   try:
       for egg in os.listdir(dir):
           if egg.endswith('.egg'):
               print "%s" % (dir, egg, egg)
           elif egg.endswith('svn'):
               print open(egg).read()
   except OSError: pass

Now everything seems to be happy. My buildbot CI builds work
correctly as they can find their dependancy links, my easy_install
command lines work correctly as I've built a 'fake' PyPI, and I work
a little more correctly as I'm not copying egg files around any longer!

In order to reduce pull on PyPI, I'm actually locally hosting copies
of BeautifulSoup, zope.*, and some SQL libraries. It also helps to
ensure I don't get an unexpected update.

Is there a better way to do any of this? This seems to work pretty
well. I'm currently keeping two of these repositories. The first
one gets new eggs with each CI build while the second one only
contains eggs which have cleared QA and are production ready

"phrase from nearest book" meme

via http://agiletesting.blogspot.com/2008/11/phrase-from-nearest-book-meme.html

• Grab the nearest book.
• Open it to page 56.
• Find the fifth sentence.
• Post the text of the sentence in your journal along with these instructions.
• Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

Here’s mine, from The Art of Agile Development by James Shore & Shane Warden:

"Setup a projector so the whole team navigates while one person drives."

Somewhat anticlimactic, but still solid advice.

A subprocess.Popen Gotcha

In answering a recent question, I noticed a bit of a gotcha while using
subprocess.Popen. It's important to note the use of the 'args' parameter if
setting the shell argument to True.

When the shell parameter is set to True, the subprocess module essentially
does the following:

exec_list = ['/bin/sh',  '-c' ] +  args
os.execvp(exec_list[0], exec_list)

The 'args' parameter to the Popen initializer can be either a sequence or
a simple string. In most cases, args ought to be a string value here,
such that a shell is executed and in turn fires off the command with
its arguments:

subprocess.Popen('/bin/ls /var/log', shell=True)

Will result in:

/bin/sh -c '/bin/ls /var/log/'

However, it is still perfectly valid to use a exec-style sequence for
'args.' The resulting command executed will be different, however.

subprocess.Popen(('/bin/ls', '/var/log'), shell=True)

Will result in:


/bin/sh -c '/bin/ls' '/var/log'


The end result is quite different. In the first example, '/var/log' is
passed as an argument to '/bin/ls.' In the second example, both '/bin/ls'
and '/var/log' are passed as arguments to '/bin/sh.'

Since there is no immediate error generated, the caller simply winds up with
unexpected results. The same problem doesn't exist in reverse. If a string
with multiple arguments is passed when the default shell=False is used, an
immediate error is raised:

>>> subprocess.Popen('/bin/ls /var/log', shell=False)
Traceback (most recent call last):
 File "", line 1, in 
 File "/usr/local/lib/python2.6/subprocess.py", line 595, in __init__
   errread, errwrite)
 File "/usr/local/lib/python2.6/subprocess.py", line 1106, in _execute_child
   raise child_exception
OSError: [Errno 2] No such file or directory

My assumption is that this is because we're actually trying to execute
'/bin/ls /var/log' as one filename (including a space).

Something to keep in my "what the..?" pile.