Getting to Know Zope 3: Round 1

While I've done quite a bit of Python over the past few years, I've not really used any of the Zope based technologies. With the exception of zope.interface and a Plone proof of concept, I'm new to most everything Zope.

I've elected to go with pure play Zope as opposed to Grok. I like what I see from the Grok project, but I like to understand all of the internals. I get the impression that Grok attempts to shield that from me.

One of the things that I've been looking forward to in my quest to learn Zope 3 has been the egg-based install process. Over the past few months, I've been contemplating moving our RPM based system over to Python eggs. We can dynamically download and install them via HTTP when our application server  starts up... or something all cutting edge and automated like that.

Getting it Up And Running

It took me a bit to actually figure out *how* to download an egg-based Zope installation. The links off of the main web site still point to the monolithic tarball distribution. It took a bit of creative clicking to find the Zope 3 site, which is Wiki based. Even then, the majority of the articles on that site are geared towards earlier releases of Zope 3.

Then I found this: http://wiki.zope.org/zope3/Zope340c1. The release documentation for Zope 3.4, release candidate 1. Not exactly the official style I was looking for, but it points me in the  right direction regardless.

Note that I opted to install the latest version of Python 2.4. My first attempt was with 2.5.1, but that ended with a  mysterious "no module named script.command." I've done all of this with 2.4.5.

1. First, we'll install the 'zopeproject' utility via setuptools. There are a few dependancies that area also added when this runs: PasteScript, PasteDeploy, and Paste.

root@marvin jj]# /usr/local/bin/easy_install zopeproject
Searching for zopeproject
Reading http://pypi.python.org/simple/zopeproject/
Best match: zopeproject 0.4.1
Downloading http://pypi.python.org/packages/2.4/z/zopeproject/zopeproject-0.4.1-py2.4.egg#md5=3c0c590752c637ee047cc555b2e8f5c1
Processing zopeproject-0.4.1-py2.4.egg
creating /usr/local/lib/python2.4/site-packages/zopeproject-0.4.1-py2.4.egg
Extracting zopeproject-0.4.1-py2.4.egg to /usr/local/lib/python2.4/site-packages
Adding zopeproject 0.4.1 to easy-install.pth file
Installing zopeproject script to /usr/local/bin

... Paste, PasteScript, and PasteDeploy ...
Finished processing dependencies for zopeproject

Easy enough. As this all relies on zc.buildout, I should be able to drop root permissions now and install directly into my home directory.

2. Next, run the newly installed zopeproject utility in order to create a Zope install. It asks a few questions specific to the install and then goes about it's business downloading all of the eggs that do not already exist on my system. As this is a new install, there are exactly zero pre existing. This takes a while. Note that it does allow one to pass a shared egg directory. This is a plus in that multiple instances can share the  same install base.

jeff@marvin code]$ /usr/local/bin/zopeproject test_project
Enter user (Name of an initial administrator user): jeff
Enter passwd (Password for the initial administrator user): password
Enter eggs_dir (Location where zc.buildout will look for and place packages) ['/home/jeff/buildout-eggs']:
Creating directory ./test_project
Downloading zc.buildout...
Invoking zc.buildout...
zip_safe flag not set; analyzing archive contents...
warning: no files found matching '0.1.0-changes.txt'
no previously-included directories found matching 'docs-in-progress'
"optparse" module already present; ignoring extras/optparse.py.
"textwrap" module already present; ignoring extras/textwrap.py.
zip_safe flag not set; analyzing archive contents...
docutils.writers.html4css1.__init__: module references __file__
docutils.writers.pep_html.__init__: module references __file__
docutils.writers.s5_html.__init__: module references __file__
docutils.writers.newlatex2e.__init__: module references __file__
docutils.parsers.rst.directives.misc: module references __file__
[jeff@marvin code]$

At this point, there's not a lot of knowledge of paster or buildout required. I intend to learn them, but for now, it's rather easy to get started. Also of note, there's a holy hell of a lot less warning output using 2.4 as opposed to 2.5!

3. Lastly, If this works as expected, it should be possible to simply step into the newly created instance  directory and startup the server.

[jeff@marvin test_project]$ bin/test_project-ctl fg
bin/paster serve deploy.ini
------
2008-04-24T21:14:38 WARNING root Developer mode is enabled: this is a security risk and should NOT be enabled on production servers. Developer mode can be turned off in etc/zope.conf
/home/jeff/buildout-eggs/zope.configuration-3.4.0-py2.4.egg/zope/configuration/config.py:197: DeprecationWarning: ZopeSecurityPolicy is deprecated. It has moved to zope.securitypolicy.zopepolicy  This reference will be removed somedays
 obj = getattr(mod, oname)
Starting server in PID 16295.
------
2008-04-24T21:14:41 INFO paste.httpserver.ThreadPool Cannot use kill_thread_limit as ctypes/killthread is not available
serving on http://127.0.0.1:8080

Perfect! We've an instance of Zope running and the ZMI is accessible. When time permits, I'll step through the instance directory and get a good understanding of what was actually created. It's pretty straightforward now that we've got an instance up.

Dipping into 3.0

A couple of days ago I downloaded the latest alpha release of Python 3.0. It's still pretty early, but I decided it made a lot of sense to start going through the new features in order to get a feel for what type of porting work we'll have to do.

Using the What's New page as a guide, I compared some of the behavior in the new 3.0 branch with the current 2.5 release. A lot of the changes appear to be "decruftification" updates. For instance, moving to a print() built in removes some of the awkward looking "print >>sys.stderr," code that never feels quite right.

The first feature that really caught my eye is the function annotation addition. I like the fact that the Python developers chose to go with a generic system as opposed to static typing alternative. I especially like the fact that annotations can be any valid Python expression.

We store all of our Linux user attributes in LDAP. This includes POSIX data as well as some domain specific stuff that makes sense. The first thought I had after reading up on function annotations was streamlining access to an LDAP system. It seems that one may be able to do something like:

#!/home/jeff/py3k/bin/python

LDAP_CONNECTION = "localhost"

class LDAPProxy:
 def __init__(self, connection):
     self._connection = connection

 def do_ldap_lookup(self, dn, attr):
     """LDAP Query Mock"""
     print("query ldap server on on {0}:{1} for {2}".format(
                             self._connection, dn, attr))
     # For example, something that can go string or int...
     return 0

 def __call__(self, attribute, kind):
     """Return a tuple of functions for use as a property"""
     def get_value(inner_self, attr=attribute) -> kind:
         result = self.do_ldap_lookup(inner_self.dn, attr)
         return get_value.__annotations__['return'](result)

     def set_value(inner_self, value):
         pass

     return (get_value, set_value)

class User:
 m = LDAPProxy(LDAP_CONNECTION)

 def __init__(self, dn):
     self.dn = dn

 uid = property(*m("uidNumber", int))
 home = property(*m("homeDirectory", str))

u = User("cn=user,ou=accounts,dc=domain,dc=com")
uid = u.uid
home = u.home
print ("{0} is ({1})".format(uid, type(uid)))
print ("{0} is ({1})".format(home, type(home))

Ok, so perhaps that's not the best example as it looks very possible
without the annotation. Still a pretty cool new feature, especially
when dealing with XMLRPC & SOAP. Ought to do make introspection and
WSDL generation much easier.

 

Choosing a framework with API support in mind?

For the last couple of years or so, I've maintained a system for managing and manipulating various web hosting accounts. The code consists of a few Python modules and some supporting C extensions where necessary.

Each module contains a series of XMLRPC calls that are made available to remote clients. Using these XMLRPC calls, it's possible to "do stuff" to a standard account. Accounts can be created, deleted, removed, and so on. It's a fairly powerful API in that it's possible to tickle just about every attribute associated with a user's account.

The application relies on a few thousand lines of boilerplate 'infrastructural code.' By infrastructural code, I generally mean things like logging, daemonization, startup and shutdown, and wrappers around Python's standard SimpleXMLRPCServer modules. They're actually subclassed in order to add support for some nifty little features, but that's not all that important.

We run multiple instances for each core module, each on a different port. For example, web might run on 8181, SQL on 8282, and ecommerce on 8383. The whole thing is fronted by Apache which provides URL mapping, authentication, and SSL capabilities.

I've really been hoping to get rid of all of that custom code. It's a headache. I don't want to manage PID files, session management, or SQL connection pooling manually any longer. Our group can be much more productive if we can simply focus on our API development rather than bugs in our frameworks.

I've spent a great deal of time researching the various Python application servers out there. I think I've narrowed it down to two: Zope 3 or Django.

With the exception of Zope 3, the problem I have with these frameworks in general is that they're clearly intended for web *site* development and not necessarily web-aware *API* development.

From a holistic point of view, I like Django. A lot. It's clean, pragmatic, and gaining a lot of popularity. I like the template system (I've seen a lot of horrible PHP). The documentation is outstanding. It took me almost no time at all to setup a basic site, grind out some extremely ugly templates, and wire everything up to a little MySQL database. That's about where it ends, though. We need to support XMLRPC calls. We don't need a UI. We don't even need a database for the most part.

Zope 3 on the other hand, looks to be exactly what we need. It's a very extensive framework. It's built on eggs, so we only need to install what we need. We can simply create XMLRPC/SOAP views and tie directly into our existing Python modules. Sounds great.

The documentation is god awful. I want to use it. I know it's very powerful. The Zope crew has just made the barrier to entry very difficult to overcome. It's not difficult to grasp once I find some type of authoritative documentation. It's finding that documentation that is the problem.

I'm going to learn it. It's clearly one of the most powerful utilities in the Pythonic tool chest.

All of that said, what is the preferred framework for API-driven development? Am I barking up the wrong tree with Zope and Django? Does it make sense to leave it as is? I don't need templates, URL routes, AJAX, or browser compatibility hacks. We're developing a large scale remote API, not a web site.

Lastly, I did look into using Twisted, but I'm not sure our current code base fits into the async. model very well. It would simply be an XMLRPC controller (for each module) firing off our current methods using threaded deferreds.

Post #1!

Well, this is it. I've done it. I've finally entered the technical blogosphere.  More specifically, the software-engineering-using-Python-in-an-Agile-setting blogosphere.   I'm preparing to step back into the world of Internet services after a brief hiatus.  Lots of stuff to do, technologies to implement, and problems to fix.  Sounds like a perfect time to start a new blog. 

So, who am I and why do I matter?  I'm not certain I can answer that second part, but I'd be more than happy to share some of my qualifications: Experience, and some experience. 

I've spent about ten years in the web hosting industry.  I've watched my company start, grow, IPO, acquire, and eventually get acquired.  I've done everything from purchase servers and operate disk arrays to develop large-scale systems using the Python programming language.  Always Linux (except for that time when I was younger and did some of that FreeBSD stuff...).

I have a very deep interest in Python specifically.  Unless dictated by standard or convention,  all of the work I've done for the past 3-4 years has been done in Python. Provisioning systems, account utilities, test suites, web UI's (which look terrible -- never let a programmer create a UI). All Python.  I sometimes say that I actually think in Python.

That's it for the obligatory introductory post. More to come.