Guppy

Over the course of the the last two months or so, I've been consolidating
a small collection of proprietary XML-RPC daemons down into a series of
egg-based plugins for twistd. So far, I'm very happy with the result.

Yesterday, I started doing some initial load testing to watch for file
descriptor and memory leaks. I noticed that the 'twistd' process started
about at 19 MB of memory and slowly increased if I kept a steady stream
of method calls going.

I had some initial trouble trying to debut the leak. I'm actually
pretty happy with what I came up with.

I came across Guppy (http://guppy-pe.sourceforge.net/), which is a very
nice little tool that lets one dump Python heap statistics. The issue I had,
though, is that my application is a daemon. I need to trigger dumps as
I increase load.

My solution is as follows. I added the following to one of the modules that contains a subset of my XMLRPC methods.

try:
    from guppy import hpy
    @exposed
    def dumpheap():
        """
        Dumps Python heap information to a temporary file.

        Args: None
        Returns: string filename
        """
        filename = '/tmp/heap.%d.%5f' % (os.getpid(), time.time())
        context = hpy()
        context.heap().dump(filename)
        return filename 
     
    except ImportError, e:
        pass

The '@exposed' decorator is an internal function which wires up XMLRPC
plumbing for dumpheap. So now, I can trigger a heap statistics dump
by simply calling an XML-RPC method. Easy!