I've moved my blog to jmcneil.net. This is no longer being updated!

Saturday, March 13, 2010

Grok 1.0 Web Development

I'm in the process of moving my blog to http://www.jmcneil.net, please update links & readers if you're subscribed to mcjeff.blogspot.com.  Comments disabled on this version, but allowed at http://www.jmcneil.net/2010/03/review-grok-1-0-web-development/.

If there is one thing that the Zope community takes a lot of heat for, it's complexity. More accurately, the quality and coherence of the documentation regarding that complexity. Enter Grok. Grok is an attempt at reducing the complexity and barrier to entry that usually surrounds Zope application development.

I've just finished reading Mr. Carlos de la Guardia's Grok 1.0 Web Development and there are a few things I know I'm sure of.
  1. Zope is still very complex. Need it be? That's a topic for another post.
  2. ZODB is cool no matter how you look at it. Unless I'm missing something, it's surprising that it isn't used in more applications as a standalone object store. Reading this book has put that on my list of things I intend to try. It seems as though it ought to be the "SQLite" of the NoSQL movement from a Python standpoint.
  3. The Zope Component Architecture (ZCA) is also just as cool as ZODB, and the same question applies. Why don't we see this in more places? Maybe this is already taken place, but I would love to see those core Zope components really put on a platform of their own.
All of that said, there's clearly a place for the ZCA. Don't use it if you're writing a script to automate a simple task. However, it might fit in well in a large enterprise-y Python environment. My irritation? When writing a script to automate a simple task I need to care about adapters, utilities, interfaces, and registries. If it's not wanted, it really can make the automation of a systems administration task overly complex.

Now that my personal opinion is out of the way, let's take a deeper look at the book. We'll get the plusses and minuses out of the way nice and early as I tend to be long winded.

The Positives

Carlos did a great job explaining a very, very complex software system. A lot of the Zope specific details were left out as this was a book on Grok, but when the detail was required, it was very simple to digest and understand. I'll cover it chapter by chapter below, but the Key Concepts Behind Grok and Grok and the ZODB chapters were very informative. Not only did they help me digest what Grok is all about, they really underscored why some of the additional complexity is beneficial. See bullet points two and three above; I think the book worked.

I enjoyed the deployment chapter as well. That's more of my forte as I really build the systems that people use to build the systems (make sense? I work for a service provider). I'm not certain now NFS friendly ZODB is, but assuming they play well together, Grok ought to deploy alright via a FastCGI methodology on most shared hosts. Thank you, WSGI. Disclaimer time: I didn't try this and I worry slightly about multiple processes. Maybe that's a nice future post.

The Negatives

Not many. First, I did notice a few sections where the code didn't wrap so cleanly on the page. In one place, copying verbatim would have resulted in a SyntaxError. There were also some indention oddities, which were probably a means to avoid listing code errors. Not a big deal. The content was stellar.

Secondly, it really didn't cover non-Linux deployments. I'll never deploy it on Windows, AIX, an DGUX, or a hair dryer, but I can understand how a developer on one of these other systems may be disappointed in having to track down instructions somewhere on the Internet.


Here's a brief take on each of the chapters in the book. As I said above, a few of them really shine.

Grok Book Cover

Chapter 1: Getting to Know Grok

Provides a quality overview of the Grok framework. Also clears up something that I'm fairly sure bothers a few developers right up front - it is possible to access a relational database system via a Zope-based framework! There are also touches on how the framework differs from others, which is to be expected.

Chapter 2: Getting Started with Grok

A general run down of how to install Grok, the directory structure, and what you'll find on the inside. Note that this covers virtualenv, which is a huge plus. Mucking with a system Python install, in my opinion, is evil.

Chapter 3: Views

You'll learn TAL, that's for certain. Having never worked with TAL, I kind of like it's approach.

Chapter 4: Models

This chapter begins to really introduce ZODB, and it does a very good job. Containers, models, persistence. It's covered at a high Grok-level detail.

Chapter 5: Forms

Quite honestly, I had trouble following this one a bit. It's simply an interest problem. Being that I generally don't spend a whole lot of time dealing with web based applications and form configurations, this just wasn't relevant to what I do on a daily basis.

Chapter 6: An Object-Oriented Search Engine

Ah-ha! So that's how one manages to query a ZODB instance without "CREATE INDEX..." support! Carlos does a great job of making the catalog concept very clear as well as separating it from the other "noise" and terminology that surrounds a Zope application. The big take-away here? Catalogs == Good. They're akin to providing good indexing on your relational tables.

Chapter 7: Security

As it turns out, security is rather complex in a Zope application. Surprised? Again, a wonderful job of taking a topic that is hard to swallow and making it clear. Principals, roles, users, groups. pluggable architecture, interfaces, and more. All of this to just sign in? Yup. But that's okay, by the end of "#7", you'll understand how it all works.

Chapter 8: Application Presentation and Page Layout

This falls into the same category as chapter 5 does. Informative, well written, good coverage, just not something I'm very involved in. This chapter clears up how one segments views via TAL templates (think {% block %} or a Mako 'def' statement).

Chapter 9: Grok and the ZODB

Well, by about this time I really started to get excited about the ZODB system and what it can do. Why should I have to care about joins, tables, ORMs, Unit-of-Work vs Active Record, thinking "flat" versus "object", or correct escapes to avoid injection attacks? By about halfway through this chapter I had started to scheme up ways to start looking at ZODB for my own purposes.

Chapter 10: Grok and Relational Database

An introduction to SQLAlchemy and an implementation of authentication using a relational database. Solid information, especially if your concerned with integrating with existing data stores.

Chapter 11: Key Concepts Behind Grok

This was my favorite chapter. Grok aside, all of those Zope bells and whistles start to really shine here. Interfaces? What about duck-typing? Oh! I see. Adapters? Why formalize something like.. oh? A registry? Hey that saves me time! By the end of this chapter, it's apparent that there is quite a bit of value in the ZCA and it's worth investigating.

Chapter 12: Grokkers, Martian, and Agile Configuration

I'm going to be honest here. I largely skimmed this.

Chapter 13: Testing and Debugging

Provides an detailed rundown of the testing methodologies and utilities available when developing a Grok application.

Chapter 14: Deployment

I'm already rather familiar with WSGI and standard deployment techniques, but this gives a nice rundown for those that are not. The ZEO configuration parts did provide some new information, though.

To sum it all up...

A good resource to have if you intend to do anything with Grok. Additionally, it does a good job of describing some of the core Zope components from a Grok point of view. I can easily see myself grabbing this book when I want to refresh my Zope memory before I head over to the Zope site. Also, chapter 5 is available for free, so you can check it out before you buy it.

Changing Blog Locations

I've decided to move over to a self-hosted Wordpress install on my own domain.  I'm moving the blog over to http://www.jmcneil.net. Go there instead!

Of course, there's always the FeedBurner angle.

Monday, March 8, 2010

Review: Python Testing Beginner's Guide

I've just finished my copy of Python Testing: Beginner's Guide, by Daniel Arbuckle.  While I'll fully admit I didn't type each and every code sample into a text editor, I did read this one cover-to-cover. Overall, it serves as a good introduction to Test Driven Development from a Python angle.

I've been rather careful to ensure all of my code is wrapped in automated testing and that they pass. I also run Subversion commit triggers and build across multiple versions of Python. I'm the only Python developer employed and a good chunk of our infrastructure is Python based.  I really see testing as a necessity as I can't really afford to get behind with unexpected bugs.

Mr. Arbuckle's book provided a good reality check. Personally, the big take away for me was that I am doing this right.  My tests are segregated, my fixtures are complete, and I'm following a lot of the best practices outlined. I'm using Buildbot and I'm integrating coverage.py.

If you're not already using some sort of testing framework, this is a wonderful introduction. It's gentle enough to be of benefit to newcomers.

Here's a chapter-by-chapter rundown of the content covered.

Types of Testing
The introduction. Covers the major types of testing that programmers really concern themselves with. Unit, Integration, and System.  You'll understand the differences between the testing "subclasses" after finishing this chapter.

Doctest: The Easiest Testing Tool
I've honestly never written a single doctest. Ever. It was enlightening to see how it all actually works and how the doctest system gets around quirks of the approach with thing such as ellipses and white space normalization.  It also helps as you'll begin to notice the limitations of the system. I prefer the control the unittest module provides.

Unit Testing with Doctest
This chapter really reduces the development process to practice through the use of doctest.  Good solid information that is useful elsewhere, just swap out the technology.  The cycle applies to other languages as well.

Breaking Tight Coupling by using Mock Objects
I've purposefully avoided using mock frameworks in the past as they've always seemed to add more complexity than they're worth. Instead, I've opted for home-grown classes that provide the minimum adherence to a protocol necessary to allow for testing via dependency injection.   I'll be 100% honest here: I think I'm 100% wrong. The Mocker libraries seem rather easy to work with and I'll attempt to remove more of the custom code I've written and integrate Mocker.

When Doctest isn't Enough: Unittest to the Rescue
 Not much to comment on here.  I've been using the unittest module for quite a while, this service as a great introduction.

Running Your Tests: Follow Your Nose
Nose! I love nose! A while back, I integrated a bunch of legacy code with a Twisted server. Nose made that whole process much easier as I didn't have to worry about how tests were really structured. My fixtures were solid and nose just found them.

There were two points that I would have liked to have seen included as they've become quite useful to me over time:

1. Integration with SetupTools/Distribute in order to handle dependencies automagically.  It is possible to specify nose as your test runner.
2. Permissions on tests matter!! If they're incorrect, Nose won't discover them! I spent quite a while tracking this down the first time I ran into it.

Developing a Test-Driven Project
This serves as another reduce-to-practice chapter.

Testing Web Application Frontends using Twill
I've said it before - I don't do much web development.  I've learned Django, Pylons, and GAE as I feel that that's required knowledge, but I don't do much production web work at all.  I'm a back-end guy. Provisioning, statistics, systems management, server integration, and so on.    I've seen Twill, I've just never used it for anything more than tinkering. This chapter is a wonderful introduction!

The one part I especially liked was the integration with unittest. With that covered, it's possible to automate your unit, system, integration, and UI testing entirely. It's possible to validate an entire system from top to bottom without needing human intervention. Bad for QA clickers, good for developers and overall code quality.

Integration Testing and System Testing
This, along with the previous chapter, shore up the areas that I believe most developers overlook. We write unit tests, sure, but do we go beyond that? This really walks a developer through doing just that. Very important stuff.

Other Testing Tools and Techniques
The final chapter. Introduces coverage.py and post-commit hooks.  Also touches on Buildbot.  I honestly did not know that the distributed VCS packages allow local commit hooks. It's logical and makes perfect sense when one things about it, but we're still using SVN. I think I have a little more ammo to push for Mercurial.

One thing I've done, and I would have liked to have seem mentioned, was to throw an exception and cause a BuildBot error report if code coverage drops below a configurable percentage.  I don't have the code handy to do this now or I would post it.  Maybe if I can dig it up I will.

So, in summary? Good book. Worth putting on your shelf if you're interested in the topic.  Python testing is a book the community has needed for a while; this fills that gap. You won't walk away an expert, but you will be off to a great start.

Like I said before, one of the chapters has been placed online for free. If you're a try before you buy type, check it out: Chapter 5: When Doctest isn't Enough: Unittest to the Rescue.


Monday, March 1, 2010

Books, books, books..

Reading a couple of books from the folks at Packt Publishing.  They've made some chapters available online so I figured I'd simply include the links.

Grok 1.0 Web Development and Python Testing : Beginner's Guide. I'm actually quite excited about the Grok book as I've taken a recent (and somewhat unexplained) interest in "webby" development.

Being more of an infrastructure and tools guy, I don't do much user facing stuff.  Over the past month or so I've built some GAE & Django sites in order to improve my skill set.  I've tried using the Zope 3 (err... BlueBream) frameworks before and have simply given up.  The benefit just didn't match the complexity.

Chapters for each are available online:

Lastly, if the planets align as they appear to be, I may finally get the chance to write my own book. That's something I've wanted to do for quite a while.