I've moved my blog to jmcneil.net. This is no longer being updated!

Thursday, October 23, 2008


Over the course of the the last two months or so, I've been consolidating
a small collection of proprietary XML-RPC daemons down into a series of
egg-based plugins for twistd. So far, I'm very happy with the result.

Yesterday, I started doing some initial load testing to watch for file
descriptor and memory leaks. I noticed that the 'twistd' process started
about at 19 MB of memory and slowly increased if I kept a steady stream
of method calls going.

I had some initial trouble trying to debut the leak. I'm actually
pretty happy with what I came up with.

I came across Guppy (http://guppy-pe.sourceforge.net/), which is a very
nice little tool that lets one dump Python heap statistics. The issue I had,
though, is that my application is a daemon. I need to trigger dumps as
I increase load.

My solution is as follows. I added the following to one of the modules that contains a subset of my XMLRPC methods.

from guppy import hpy
def dumpheap():
Dumps Python heap information to a temporary file.

Args: None
Returns: string filename
filename = '/tmp/heap.%d.%5f' % (os.getpid(), time.time())
context = hpy()
return filename

except ImportError, e:

The '@exposed' decorator is an internal function which wires up XMLRPC
plumbing for dumpheap. So now, I can trigger a heap statistics dump
by simply calling an XML-RPC method. Easy!

1 comment:

Jeff McNeil said...

Yes, I know about the tempfile module. I built the filename up like that to make processing the data slightly easier.