a small collection of proprietary XML-RPC daemons down into a series of
egg-based plugins for twistd. So far, I'm very happy with the result.
Yesterday, I started doing some initial load testing to watch for file
descriptor and memory leaks. I noticed that the 'twistd' process started
about at 19 MB of memory and slowly increased if I kept a steady stream
of method calls going.
I had some initial trouble trying to debut the leak. I'm actually
pretty happy with what I came up with.
I came across Guppy (http://guppy-pe.sourceforge.net/), which is a very
nice little tool that lets one dump Python heap statistics. The issue I had,
though, is that my application is a daemon. I need to trigger dumps as
I increase load.
My solution is as follows. I added the following to one of the modules that contains a subset of my XMLRPC methods.
from guppy import hpy
Dumps Python heap information to a temporary file.
Returns: string filename
filename = '/tmp/heap.%d.%5f' % (os.getpid(), time.time())
context = hpy()
except ImportError, e:
The '@exposed' decorator is an internal function which wires up XMLRPC
plumbing for dumpheap. So now, I can trigger a heap statistics dump
by simply calling an XML-RPC method. Easy!