Setuptools Egg Repository

I've been in the process of moving all of our account management code away from a home brewed XMLRPC system and on to a Twisted instance. At first, we'll probably defer most of our calls in threads, but over time, I hope to make everything a bit more asynchronous in nature.

It uses a plug-in architecture. We've one main RPM we deploy (bdist_rpm) that grafts itself into the twisted/plugins directory. That component is then responsible for loading our actual account provisioning code.

I've been bundling each plug-in as an egg file. In order to make deployment easier, I wanted to run my own egg repository.

The simplest method is to dump all of our eggs into a directory and enable directory indexing within Apache. This gives us an automatically generated index of egg files that 'easy_install -f URL' can read from.

In our configuration, buildbot does the publishing. I've the setup.cfg setup such that it tags each egg with the SVN revision number. After the build has passed testing, buildbot copies the new egg into an HTTP accessible directory.

The problem I ran into is that I wanted to make all of our SVN code available to developers, but still rely on the easy directory index approach.

My solution?
[root@buildbot-slave01 eggs]# cat lister.cgi

#!/usr/bin/python

import os
import cgitb
cgitb.enable()

print 'Content-type: text/html'
print

for egg in os.listdir('.'):
   if egg.endswith('.egg'):
       print "<a href="%s">%s</a>" % (egg, egg)
   elif egg.endswith('svn'):
       print open(egg).read()

[root@buildbot-slave01 eggs]#

So now, I've included little text files named 'package.svn' that simply contain setuptools #package-dev links:

[root@buildbot-slave01 eggs]# cat package.svn
<a href="http://buildbot:password@svn/package/trunk#egg=package.dev">
      package trunk</a>

This results in an index that includes our buildbot tagged packages that we'll use by default, and our SVN links that are available for devlopers to follow.

Installing is as easy as:

easy_install -f http://buildbot/eggs/lister.cgi package

In turn, I've setup the unit tests to require certain eggs that exist in my repository. The buildbot update method is 'clobber', so each automated built results in the latest egg file being installed in the local build directory.

Very cool!

HTTP Error Codes

This is probably next to ancient, but I got a good chuckle out of it.

http://www.flickr.com/photos/apelad/sets/72157594388426362/

Instance Specific Methods

Given an instance of a Python class, it's quite simple to bind a function to that object. One just binds a callable.

Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42)

[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> class X(object):

...     pass

...

>>> def f():

...     pass

...

>>> x = X()

>>> x.f

Traceback (most recent call last):

 File "", line 1, in 

AttributeError: 'X' object has no attribute 'f'

>>> x.f = f

>>> x.f



>>>

The downside here is that when referencing the newly bound attribute, it's not treated as a bound method but rather a standard function object. There are two simple ways to actually create a new bound method that is specific to an instance.

First, one can easily just call the __get__ method of the function object when binding to the instance. Python functions are non-overriding descriptors. When the __get__ method is called, a method object is returned. Various attributes of the new method object handle the object 'plumbing', so to speak: im_self, im_class, im_func. That's really another topic. Here's how it works:

Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42)

[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> class X(object): pass

...

>>> def f(self): pass

...

>>> x = X()

>>> x.f

Traceback (most recent call last):

 File "", line 1, in 

AttributeError: 'X' object has no attribute 'f'

>>> x.f = f.__get__(x, X)

>>> x.f

>

>>>

There we go. The 'im_func' is made available as the first argument to the function's __get__ method, as it is really the 'self' parameter that any standard descriptor would be passed.

The next way to make this all happen is via the types module. There is an 'MethodType' object that we can use to wrap a function:

Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42)

[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2

Type "help", "copyright", "credits" or "license" for more information.

>>> import types

>>> help (types.MethodType)



>>> class X(object): pass

...

>>> def f(self): pass

...

>>> x = X()

>>> x.f

Traceback (most recent call last):

 File "", line 1, in 

AttributeError: 'X' object has no attribute 'f'

>>> x.f = types.MethodType(f, x, X)

>>> x.f

>

>>> x.f()

>>>

It's my assumption that MethodType is simply calling f.__get__ internally, though I'm slightly too lazy to go look.

NFS Problems

All of my NFS problems have just gone away. We've recently updated our Kernels to the latest RHES available. Looks like there was something hidden in there that fixed it for us. Wonderful.

I do realize that it's normal to see delay. Both due to attribute cache semantics and mtime granularity of one second. The problem I had here was quite different in that the dentry caches *never* expired negative entries.

More NFS...

I am able to reproduce the issue at will now using both a Linux NFS
server or the NAS system. We can pretty safely rule out the NAS as the
culprit (though let's keep the ticket open, in case they've seen this
before).

As I've already explained, Linux caches file names as to avoid hitting
the disk on every look up (dentry cache). When there is a cache hit
(either negative or positive) the kernel will call a 'validate'
function, to ensure that the cached data is still good.

On local file systems, that function is simply a no-operation as all
updates need to go through the cache. Coherency is guaranteed.

On remote file systems (NFS, GFS, etc...) that call does "something" to
validate the contents of the cache. If the backing data has changed,
that entry is invalidated and the entire look up process starts over.

If there is a cache entry, the NFS validate function checks the cached
mtime on that entry's parent and compares it with current mtime as
reported by the NFS server. If there is a difference, the entry is
invalidated.

That process appears to not work correctly when dealing with negative
entries. It seems to work fine for positive entries (file *changes* show
up, just not new files in some cases).

So, after spending some time tracing how this whole process actually
works, I was able to reproduce the problem at will:

1. Stat a file that we know is non-existant. This populates the dentry
   cache with a negative entry.
   

   
   [root@cluster-util ~]# ssh root@cluster-www1a
   "stat /home/cluster1/data/f/f/nfsuser01/test_nofile"
   stat: cannot stat `/home/cluster1/data/f/f/nfsuser01/test_nofile': No
   such file or directory
   

   
   
   
2. Create that file on a different server, this will also update the
   mtime on that parent directory, so the NFS validation code on the dentry
   hit ought to catch that.
   

   
   [root@cluster-util ~]# ssh root@cluster-sshftp1a
   "touch /home/cluster1/data/f/f/nfsuser01/test_nofile"
   

   
   
   
3. Try and stat the file again. Still broken.
   

   
   [root@cluster-util ~]# ssh root@cluster-www1a
   "stat /home/cluster1/data/f/f/nfsuser01/test_nofile"
   stat: cannot stat `/home/cluster1/data/f/f/nfsuser01/test_nofile': No
   such file or directory
   

   
   
   
4. Read the parent directory, this doesn't actually *repopulate* the
   cache as I first thought. It actually invalidates the parent directory
   itself (which is what should have happened correctly the first time
   through). It doesn't invalidate it "the same way", though.
   

   
   [root@cluster-util ~]# ssh root@cluster-www1a
   "ls /home/cluster1/data/f/f/nfsuser01/ | wc -l"
   16
   

   
   
   
5. And now it's present, as evidenced by the stat command below.
   

   
   [root@cluster-util ~]# ssh root@cluster-www1a
   "stat /home/cluster1/data/f/f/nfsuser01/test_nofile"
     File: `/home/cluster1/data/f/f/nfsuser01/test_nofile'
     Size: 0               Blocks: 0          IO Block: 4096   regular
   empty file
   Device: 15h/21d Inode: 4046108346  Links: 1
   Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
   Access: 2008-05-28 10:07:28.963000000 -0400
   Modify: 2008-05-28 10:07:28.963000000 -0400
   Change: 2008-05-28 10:07:28.963000000 -0400
   [root@cluster-util ~]#
   

   
   

Next step is going to be to load up a debug kernel or stick a system tap script in there to attempt to determine why that negative entry isn't invalidated as it should be .

Linux Caching Issues

Recently, we started seeing issues where a file would exist on a couple of the NFS clients, but not the others. The front-end Apache instance would return different results depending on which cluster nodes our load balancers would direct us to. In some cases, we'd recieve 404 errors. In other cases, we'd get the actual content. In a third scnenario, we would get the default Red Hat Index page (as the NFS client couldn't access the *real* DirectoryIndex page).

At first, I thought it was an Apache problem as we run some custom modules which handle URL translation.

However, another issue popped up with a PHP script. That PHP script was attempting to include another as a function library. In this case, that include was failing with an ENOENT on some systems, but not on others. Clearly this wasn't an Apache problem.

I've yet to be able to reproduce the problem, but I've a few existing instances to test with.

Given a situation where we're getting a 404 half of the time, I ran the following test against our NFS clients:

# for i in  www1a www1b www1c www1d ; do echo $i; ssh cluster-$i-mgmt "stat /home/cluster1/data/s/c/user/html/index.htm"; done
www1a
stat: cannot stat `/home/cluster1/data/s/c/user/html/index.htm': No such file or directory
www1b
  File: `/home/cluster1/data/s/c/user/html/index.htm'
  Size: 18838           Blocks: 40         IO Block: 4096   regular file
Device: 15h/21d Inode: 2733856169  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (15953/ user)   Gid: (15953/ user)
Access: 2008-05-23 09:55:03.029000000 -0400
Modify: 2008-05-23 09:55:03.029000000 -0400
Change: 2008-05-23 09:55:03.029000000 -0400
www1c
  File: `/home/cluster1/data/s/c/user/html/index.htm'
  Size: 18838           Blocks: 40         IO Block: 4096   regular file
Device: 15h/21d Inode: 2733856169  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (15953/ user)   Gid: (15953/ user)
Access: 2008-05-23 09:55:03.029000000 -0400
Modify: 2008-05-23 09:55:03.029000000 -0400
Change: 2008-05-23 09:55:03.029000000 -0400 
www1d
  File: `/home/cluster1/data/s/c/user/html/index.htm'
  Size: 18838           Blocks: 40         IO Block: 4096   regular file
Device: 15h/21d Inode: 2733856169  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (15953/ user)   Gid: (15953/ user)
Access: 2008-05-23 09:55:03.029000000 -0400
Modify: 2008-05-23 09:55:03.029000000 -0400
Change: 2008-05-23 09:55:03.029000000 -0400

When logging in to 'www1a', it's just not possible to read the file directly. I can't cat it, stat it, or ls it. However, once I step into the containing directory (html, in this case) and run an 'ls', the file is now availble. Looks as though the readdir() triggered by my ls command updates the cache.

So, my assumption at this point is that it's a directory cache problem. For some reason, we're getting negative entries (NULL inode structure pointers). I've no idea why.

The problem came up again this morning. As it turns out, Linux 2.6.16 allows users to dump various caches in order to free the memory being used. I tried it out on one of the systems experiencing the problem:

[root@cluster-www1c vm]# ls
block_dump                 drop_caches           max_map_count
overcommit_ratio          swappiness
dirty_background_ratio     hugetlb_shm_group     min_free_kbytes
pagecache                 swap_token_timeout
dirty_expire_centisecs     laptop_mode           nr_hugepages
page-cluster              vdso_enabled
dirty_ratio                legacy_va_layout      nr_pdflush_threads
panic_on_oom              vfs_cache_pressure
dirty_writeback_centisecs  lowmem_reserve_ratio  overcommit_memory
percpu_pagelist_fraction
[root@cluster-www1c vm]# stat /home/cluster1/data/s/c/user/html/index.htm
stat: cannot stat `/home/cluster1/data/s/c/user/html/index.htm': No such file or directory
[root@cluster-www1c vm]# sync
[root@cluster-www1c vm]# sync
[root@cluster-www1c vm]# echo '2' > drop_caches
[root@cluster-www1c vm]# !st
stat /home/cluster1/data/s/c/user/html/index.htm
  File: `/home/cluster1/data/s/c/user/html/index.htm'
  Size: 18838           Blocks: 40         IO Block: 4096   regular file
Device: 15h/21d Inode: 2733856169  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (15953/ user)   Gid: (15953/ user)
Access: 2008-05-23 09:55:03.029000000 -0400
Modify: 2008-05-23 09:55:03.029000000 -0400
Change: 2008-05-23 09:55:03.029000000 -0400
[root@cluster-www1c vm]#

So, we've got a bit of a workaround, at least for the long weekend. I'll probably wind up setting up a cron job to dump the cache every half hour or so in order to avoid phone calls.

The problem is clearly related to the dcache. I've no idea what is causing it, however. It could be our bind mount system. The fancy NAS unit may also be returning invalid responses causing Linux to do the Right Thing at the Wrong Time.

Getting to Know Zope 3: Round 1

While I've done quite a bit of Python over the past few years, I've not really used any of the Zope based technologies. With the exception of zope.interface and a Plone proof of concept, I'm new to most everything Zope.

I've elected to go with pure play Zope as opposed to Grok. I like what I see from the Grok project, but I like to understand all of the internals. I get the impression that Grok attempts to shield that from me.

One of the things that I've been looking forward to in my quest to learn Zope 3 has been the egg-based install process. Over the past few months, I've been contemplating moving our RPM based system over to Python eggs. We can dynamically download and install them via HTTP when our application server  starts up... or something all cutting edge and automated like that.

Getting it Up And Running

It took me a bit to actually figure out *how* to download an egg-based Zope installation. The links off of the main web site still point to the monolithic tarball distribution. It took a bit of creative clicking to find the Zope 3 site, which is Wiki based. Even then, the majority of the articles on that site are geared towards earlier releases of Zope 3.

Then I found this: http://wiki.zope.org/zope3/Zope340c1. The release documentation for Zope 3.4, release candidate 1. Not exactly the official style I was looking for, but it points me in the  right direction regardless.

Note that I opted to install the latest version of Python 2.4. My first attempt was with 2.5.1, but that ended with a  mysterious "no module named script.command." I've done all of this with 2.4.5.

1. First, we'll install the 'zopeproject' utility via setuptools. There are a few dependancies that area also added when this runs: PasteScript, PasteDeploy, and Paste.

root@marvin jj]# /usr/local/bin/easy_install zopeproject
Searching for zopeproject
Reading http://pypi.python.org/simple/zopeproject/
Best match: zopeproject 0.4.1
Downloading http://pypi.python.org/packages/2.4/z/zopeproject/zopeproject-0.4.1-py2.4.egg#md5=3c0c590752c637ee047cc555b2e8f5c1
Processing zopeproject-0.4.1-py2.4.egg
creating /usr/local/lib/python2.4/site-packages/zopeproject-0.4.1-py2.4.egg
Extracting zopeproject-0.4.1-py2.4.egg to /usr/local/lib/python2.4/site-packages
Adding zopeproject 0.4.1 to easy-install.pth file
Installing zopeproject script to /usr/local/bin

... Paste, PasteScript, and PasteDeploy ...
Finished processing dependencies for zopeproject

Easy enough. As this all relies on zc.buildout, I should be able to drop root permissions now and install directly into my home directory.

2. Next, run the newly installed zopeproject utility in order to create a Zope install. It asks a few questions specific to the install and then goes about it's business downloading all of the eggs that do not already exist on my system. As this is a new install, there are exactly zero pre existing. This takes a while. Note that it does allow one to pass a shared egg directory. This is a plus in that multiple instances can share the  same install base.

jeff@marvin code]$ /usr/local/bin/zopeproject test_project
Enter user (Name of an initial administrator user): jeff
Enter passwd (Password for the initial administrator user): password
Enter eggs_dir (Location where zc.buildout will look for and place packages) ['/home/jeff/buildout-eggs']:
Creating directory ./test_project
Downloading zc.buildout...
Invoking zc.buildout...
zip_safe flag not set; analyzing archive contents...
warning: no files found matching '0.1.0-changes.txt'
no previously-included directories found matching 'docs-in-progress'
"optparse" module already present; ignoring extras/optparse.py.
"textwrap" module already present; ignoring extras/textwrap.py.
zip_safe flag not set; analyzing archive contents...
docutils.writers.html4css1.__init__: module references __file__
docutils.writers.pep_html.__init__: module references __file__
docutils.writers.s5_html.__init__: module references __file__
docutils.writers.newlatex2e.__init__: module references __file__
docutils.parsers.rst.directives.misc: module references __file__
[jeff@marvin code]$

At this point, there's not a lot of knowledge of paster or buildout required. I intend to learn them, but for now, it's rather easy to get started. Also of note, there's a holy hell of a lot less warning output using 2.4 as opposed to 2.5!

3. Lastly, If this works as expected, it should be possible to simply step into the newly created instance  directory and startup the server.

[jeff@marvin test_project]$ bin/test_project-ctl fg
bin/paster serve deploy.ini
------
2008-04-24T21:14:38 WARNING root Developer mode is enabled: this is a security risk and should NOT be enabled on production servers. Developer mode can be turned off in etc/zope.conf
/home/jeff/buildout-eggs/zope.configuration-3.4.0-py2.4.egg/zope/configuration/config.py:197: DeprecationWarning: ZopeSecurityPolicy is deprecated. It has moved to zope.securitypolicy.zopepolicy  This reference will be removed somedays
 obj = getattr(mod, oname)
Starting server in PID 16295.
------
2008-04-24T21:14:41 INFO paste.httpserver.ThreadPool Cannot use kill_thread_limit as ctypes/killthread is not available
serving on http://127.0.0.1:8080

Perfect! We've an instance of Zope running and the ZMI is accessible. When time permits, I'll step through the instance directory and get a good understanding of what was actually created. It's pretty straightforward now that we've got an instance up.