Review: Python Testing Beginner's Guide

I've just finished my copy of Python Testing: Beginner's Guide, by Daniel Arbuckle.  While I'll fully admit I didn't type each and every code sample into a text editor, I did read this one cover-to-cover. Overall, it serves as a good introduction to Test Driven Development from a Python angle.

I've been rather careful to ensure all of my code is wrapped in automated testing and that they pass. I also run Subversion commit triggers and build across multiple versions of Python. I'm the only Python developer employed and a good chunk of our infrastructure is Python based.  I really see testing as a necessity as I can't really afford to get behind with unexpected bugs.

Mr. Arbuckle's book provided a good reality check. Personally, the big take away for me was that I am doing this right.  My tests are segregated, my fixtures are complete, and I'm following a lot of the best practices outlined. I'm using Buildbot and I'm integrating coverage.py.

If you're not already using some sort of testing framework, this is a wonderful introduction. It's gentle enough to be of benefit to newcomers.

Here's a chapter-by-chapter rundown of the content covered.

Types of Testing
The introduction. Covers the major types of testing that programmers really concern themselves with. Unit, Integration, and System.  You'll understand the differences between the testing "subclasses" after finishing this chapter.

Doctest: The Easiest Testing Tool
I've honestly never written a single doctest. Ever. It was enlightening to see how it all actually works and how the doctest system gets around quirks of the approach with thing such as ellipses and white space normalization.  It also helps as you'll begin to notice the limitations of the system. I prefer the control the unittest module provides.

Unit Testing with Doctest
This chapter really reduces the development process to practice through the use of doctest.  Good solid information that is useful elsewhere, just swap out the technology.  The cycle applies to other languages as well.

Breaking Tight Coupling by using Mock Objects
I've purposefully avoided using mock frameworks in the past as they've always seemed to add more complexity than they're worth. Instead, I've opted for home-grown classes that provide the minimum adherence to a protocol necessary to allow for testing via dependency injection.   I'll be 100% honest here: I think I'm 100% wrong. The Mocker libraries seem rather easy to work with and I'll attempt to remove more of the custom code I've written and integrate Mocker.

When Doctest isn't Enough: Unittest to the Rescue
 Not much to comment on here.  I've been using the unittest module for quite a while, this service as a great introduction.

Running Your Tests: Follow Your Nose
Nose! I love nose! A while back, I integrated a bunch of legacy code with a Twisted server. Nose made that whole process much easier as I didn't have to worry about how tests were really structured. My fixtures were solid and nose just found them.

There were two points that I would have liked to have seen included as they've become quite useful to me over time:

1. Integration with SetupTools/Distribute in order to handle dependencies automagically.  It is possible to specify nose as your test runner.
2. Permissions on tests matter!! If they're incorrect, Nose won't discover them! I spent quite a while tracking this down the first time I ran into it.

Developing a Test-Driven Project
This serves as another reduce-to-practice chapter.

Testing Web Application Frontends using Twill
I've said it before - I don't do much web development.  I've learned Django, Pylons, and GAE as I feel that that's required knowledge, but I don't do much production web work at all.  I'm a back-end guy. Provisioning, statistics, systems management, server integration, and so on.    I've seen Twill, I've just never used it for anything more than tinkering. This chapter is a wonderful introduction!

The one part I especially liked was the integration with unittest. With that covered, it's possible to automate your unit, system, integration, and UI testing entirely. It's possible to validate an entire system from top to bottom without needing human intervention. Bad for QA clickers, good for developers and overall code quality.

Integration Testing and System Testing
This, along with the previous chapter, shore up the areas that I believe most developers overlook. We write unit tests, sure, but do we go beyond that? This really walks a developer through doing just that. Very important stuff.

Other Testing Tools and Techniques
The final chapter. Introduces coverage.py and post-commit hooks.  Also touches on Buildbot.  I honestly did not know that the distributed VCS packages allow local commit hooks. It's logical and makes perfect sense when one things about it, but we're still using SVN. I think I have a little more ammo to push for Mercurial.

One thing I've done, and I would have liked to have seem mentioned, was to throw an exception and cause a BuildBot error report if code coverage drops below a configurable percentage.  I don't have the code handy to do this now or I would post it.  Maybe if I can dig it up I will.

So, in summary? Good book. Worth putting on your shelf if you're interested in the topic.  Python testing is a book the community has needed for a while; this fills that gap. You won't walk away an expert, but you will be off to a great start.

Like I said before, one of the chapters has been placed online for free. If you're a try before you buy type, check it out: Chapter 5: When Doctest isn't Enough: Unittest to the Rescue.