I've moved my blog to jmcneil.net. This is no longer being updated!

Saturday, March 13, 2010

Grok 1.0 Web Development

I'm in the process of moving my blog to http://www.jmcneil.net, please update links & readers if you're subscribed to mcjeff.blogspot.com.  Comments disabled on this version, but allowed at http://www.jmcneil.net/2010/03/review-grok-1-0-web-development/.

If there is one thing that the Zope community takes a lot of heat for, it's complexity. More accurately, the quality and coherence of the documentation regarding that complexity. Enter Grok. Grok is an attempt at reducing the complexity and barrier to entry that usually surrounds Zope application development.

I've just finished reading Mr. Carlos de la Guardia's Grok 1.0 Web Development and there are a few things I know I'm sure of.
  1. Zope is still very complex. Need it be? That's a topic for another post.
  2. ZODB is cool no matter how you look at it. Unless I'm missing something, it's surprising that it isn't used in more applications as a standalone object store. Reading this book has put that on my list of things I intend to try. It seems as though it ought to be the "SQLite" of the NoSQL movement from a Python standpoint.
  3. The Zope Component Architecture (ZCA) is also just as cool as ZODB, and the same question applies. Why don't we see this in more places? Maybe this is already taken place, but I would love to see those core Zope components really put on a platform of their own.
All of that said, there's clearly a place for the ZCA. Don't use it if you're writing a script to automate a simple task. However, it might fit in well in a large enterprise-y Python environment. My irritation? When writing a script to automate a simple task I need to care about adapters, utilities, interfaces, and registries. If it's not wanted, it really can make the automation of a systems administration task overly complex.

Now that my personal opinion is out of the way, let's take a deeper look at the book. We'll get the plusses and minuses out of the way nice and early as I tend to be long winded.

The Positives

Carlos did a great job explaining a very, very complex software system. A lot of the Zope specific details were left out as this was a book on Grok, but when the detail was required, it was very simple to digest and understand. I'll cover it chapter by chapter below, but the Key Concepts Behind Grok and Grok and the ZODB chapters were very informative. Not only did they help me digest what Grok is all about, they really underscored why some of the additional complexity is beneficial. See bullet points two and three above; I think the book worked.

I enjoyed the deployment chapter as well. That's more of my forte as I really build the systems that people use to build the systems (make sense? I work for a service provider). I'm not certain now NFS friendly ZODB is, but assuming they play well together, Grok ought to deploy alright via a FastCGI methodology on most shared hosts. Thank you, WSGI. Disclaimer time: I didn't try this and I worry slightly about multiple processes. Maybe that's a nice future post.

The Negatives

Not many. First, I did notice a few sections where the code didn't wrap so cleanly on the page. In one place, copying verbatim would have resulted in a SyntaxError. There were also some indention oddities, which were probably a means to avoid listing code errors. Not a big deal. The content was stellar.

Secondly, it really didn't cover non-Linux deployments. I'll never deploy it on Windows, AIX, an DGUX, or a hair dryer, but I can understand how a developer on one of these other systems may be disappointed in having to track down instructions somewhere on the Internet.

Chapter-by-Chapter

Here's a brief take on each of the chapters in the book. As I said above, a few of them really shine.

Grok Book Cover

Chapter 1: Getting to Know Grok

Provides a quality overview of the Grok framework. Also clears up something that I'm fairly sure bothers a few developers right up front - it is possible to access a relational database system via a Zope-based framework! There are also touches on how the framework differs from others, which is to be expected.

Chapter 2: Getting Started with Grok

A general run down of how to install Grok, the directory structure, and what you'll find on the inside. Note that this covers virtualenv, which is a huge plus. Mucking with a system Python install, in my opinion, is evil.

Chapter 3: Views

You'll learn TAL, that's for certain. Having never worked with TAL, I kind of like it's approach.

Chapter 4: Models

This chapter begins to really introduce ZODB, and it does a very good job. Containers, models, persistence. It's covered at a high Grok-level detail.

Chapter 5: Forms

Quite honestly, I had trouble following this one a bit. It's simply an interest problem. Being that I generally don't spend a whole lot of time dealing with web based applications and form configurations, this just wasn't relevant to what I do on a daily basis.

Chapter 6: An Object-Oriented Search Engine

Ah-ha! So that's how one manages to query a ZODB instance without "CREATE INDEX..." support! Carlos does a great job of making the catalog concept very clear as well as separating it from the other "noise" and terminology that surrounds a Zope application. The big take-away here? Catalogs == Good. They're akin to providing good indexing on your relational tables.

Chapter 7: Security

As it turns out, security is rather complex in a Zope application. Surprised? Again, a wonderful job of taking a topic that is hard to swallow and making it clear. Principals, roles, users, groups. pluggable architecture, interfaces, and more. All of this to just sign in? Yup. But that's okay, by the end of "#7", you'll understand how it all works.

Chapter 8: Application Presentation and Page Layout

This falls into the same category as chapter 5 does. Informative, well written, good coverage, just not something I'm very involved in. This chapter clears up how one segments views via TAL templates (think {% block %} or a Mako 'def' statement).

Chapter 9: Grok and the ZODB

Well, by about this time I really started to get excited about the ZODB system and what it can do. Why should I have to care about joins, tables, ORMs, Unit-of-Work vs Active Record, thinking "flat" versus "object", or correct escapes to avoid injection attacks? By about halfway through this chapter I had started to scheme up ways to start looking at ZODB for my own purposes.

Chapter 10: Grok and Relational Database

An introduction to SQLAlchemy and an implementation of authentication using a relational database. Solid information, especially if your concerned with integrating with existing data stores.

Chapter 11: Key Concepts Behind Grok

This was my favorite chapter. Grok aside, all of those Zope bells and whistles start to really shine here. Interfaces? What about duck-typing? Oh! I see. Adapters? Why formalize something like.. oh? A registry? Hey that saves me time! By the end of this chapter, it's apparent that there is quite a bit of value in the ZCA and it's worth investigating.

Chapter 12: Grokkers, Martian, and Agile Configuration

I'm going to be honest here. I largely skimmed this.

Chapter 13: Testing and Debugging

Provides an detailed rundown of the testing methodologies and utilities available when developing a Grok application.

Chapter 14: Deployment

I'm already rather familiar with WSGI and standard deployment techniques, but this gives a nice rundown for those that are not. The ZEO configuration parts did provide some new information, though.

To sum it all up...

A good resource to have if you intend to do anything with Grok. Additionally, it does a good job of describing some of the core Zope components from a Grok point of view. I can easily see myself grabbing this book when I want to refresh my Zope memory before I head over to the Zope site. Also, chapter 5 is available for free, so you can check it out before you buy it.

Changing Blog Locations

I've decided to move over to a self-hosted Wordpress install on my own domain.  I'm moving the blog over to http://www.jmcneil.net. Go there instead!

Of course, there's always the FeedBurner angle.

Monday, March 8, 2010

Review: Python Testing Beginner's Guide

I've just finished my copy of Python Testing: Beginner's Guide, by Daniel Arbuckle.  While I'll fully admit I didn't type each and every code sample into a text editor, I did read this one cover-to-cover. Overall, it serves as a good introduction to Test Driven Development from a Python angle.

I've been rather careful to ensure all of my code is wrapped in automated testing and that they pass. I also run Subversion commit triggers and build across multiple versions of Python. I'm the only Python developer employed and a good chunk of our infrastructure is Python based.  I really see testing as a necessity as I can't really afford to get behind with unexpected bugs.

Mr. Arbuckle's book provided a good reality check. Personally, the big take away for me was that I am doing this right.  My tests are segregated, my fixtures are complete, and I'm following a lot of the best practices outlined. I'm using Buildbot and I'm integrating coverage.py.

If you're not already using some sort of testing framework, this is a wonderful introduction. It's gentle enough to be of benefit to newcomers.

Here's a chapter-by-chapter rundown of the content covered.

Types of Testing
The introduction. Covers the major types of testing that programmers really concern themselves with. Unit, Integration, and System.  You'll understand the differences between the testing "subclasses" after finishing this chapter.

Doctest: The Easiest Testing Tool
I've honestly never written a single doctest. Ever. It was enlightening to see how it all actually works and how the doctest system gets around quirks of the approach with thing such as ellipses and white space normalization.  It also helps as you'll begin to notice the limitations of the system. I prefer the control the unittest module provides.

Unit Testing with Doctest
This chapter really reduces the development process to practice through the use of doctest.  Good solid information that is useful elsewhere, just swap out the technology.  The cycle applies to other languages as well.

Breaking Tight Coupling by using Mock Objects
I've purposefully avoided using mock frameworks in the past as they've always seemed to add more complexity than they're worth. Instead, I've opted for home-grown classes that provide the minimum adherence to a protocol necessary to allow for testing via dependency injection.   I'll be 100% honest here: I think I'm 100% wrong. The Mocker libraries seem rather easy to work with and I'll attempt to remove more of the custom code I've written and integrate Mocker.

When Doctest isn't Enough: Unittest to the Rescue
 Not much to comment on here.  I've been using the unittest module for quite a while, this service as a great introduction.

Running Your Tests: Follow Your Nose
Nose! I love nose! A while back, I integrated a bunch of legacy code with a Twisted server. Nose made that whole process much easier as I didn't have to worry about how tests were really structured. My fixtures were solid and nose just found them.

There were two points that I would have liked to have seen included as they've become quite useful to me over time:

1. Integration with SetupTools/Distribute in order to handle dependencies automagically.  It is possible to specify nose as your test runner.
2. Permissions on tests matter!! If they're incorrect, Nose won't discover them! I spent quite a while tracking this down the first time I ran into it.

Developing a Test-Driven Project
This serves as another reduce-to-practice chapter.

Testing Web Application Frontends using Twill
I've said it before - I don't do much web development.  I've learned Django, Pylons, and GAE as I feel that that's required knowledge, but I don't do much production web work at all.  I'm a back-end guy. Provisioning, statistics, systems management, server integration, and so on.    I've seen Twill, I've just never used it for anything more than tinkering. This chapter is a wonderful introduction!

The one part I especially liked was the integration with unittest. With that covered, it's possible to automate your unit, system, integration, and UI testing entirely. It's possible to validate an entire system from top to bottom without needing human intervention. Bad for QA clickers, good for developers and overall code quality.

Integration Testing and System Testing
This, along with the previous chapter, shore up the areas that I believe most developers overlook. We write unit tests, sure, but do we go beyond that? This really walks a developer through doing just that. Very important stuff.

Other Testing Tools and Techniques
The final chapter. Introduces coverage.py and post-commit hooks.  Also touches on Buildbot.  I honestly did not know that the distributed VCS packages allow local commit hooks. It's logical and makes perfect sense when one things about it, but we're still using SVN. I think I have a little more ammo to push for Mercurial.

One thing I've done, and I would have liked to have seem mentioned, was to throw an exception and cause a BuildBot error report if code coverage drops below a configurable percentage.  I don't have the code handy to do this now or I would post it.  Maybe if I can dig it up I will.

So, in summary? Good book. Worth putting on your shelf if you're interested in the topic.  Python testing is a book the community has needed for a while; this fills that gap. You won't walk away an expert, but you will be off to a great start.

Like I said before, one of the chapters has been placed online for free. If you're a try before you buy type, check it out: Chapter 5: When Doctest isn't Enough: Unittest to the Rescue.

  

Monday, March 1, 2010

Books, books, books..

Reading a couple of books from the folks at Packt Publishing.  They've made some chapters available online so I figured I'd simply include the links.

Grok 1.0 Web Development and Python Testing : Beginner's Guide. I'm actually quite excited about the Grok book as I've taken a recent (and somewhat unexplained) interest in "webby" development.

Being more of an infrastructure and tools guy, I don't do much user facing stuff.  Over the past month or so I've built some GAE & Django sites in order to improve my skill set.  I've tried using the Zope 3 (err... BlueBream) frameworks before and have simply given up.  The benefit just didn't match the complexity.

Chapters for each are available online:


Lastly, if the planets align as they appear to be, I may finally get the chance to write my own book. That's something I've wanted to do for quite a while.

Wednesday, February 24, 2010

My Big PyCon 2010 Review

Hello, world! Oh, wait a minute. My apologies. I've seen a good half-dozen example applications over week; I think it's starting to rub off.

In all seriousness, I've just wrapped up my first PyCon event. Yup, that's right. I've been tossing Python code around for almost six years now and this was my first convention. First impression? What an excellent experience.

I do need to throw in a caveat here. I'm the sole Python+Linux developer on a team of six. The other five individuals spend their days firmly implanted in a seat driving Visual Studio. It's not very often I get to talk about and get into "my stuff." This was a refreshing experience for that alone.

Before I step into each talk that I attended, I wanted to mention three non-technical things I picked up while there.

1. The term 'dunder methods.' Whenever I explain __XXX__ to any one, I'm usually at a loss as to what to call them. It usually comes out as "magic double underscore name double underscore method." I've now firmly picked up on this nomenclature.

2. The majority of the attendees live in a different world than I do. It seemed like many were on larger teams of Python coders. As this is not my case, I found it a little difficult to discuss some topics. Not because I didn't understand them; quite the contrary. I simply never have the opportunity to vocalize.

So, that brings me the final point.

3. I want to get more involved in the Python community (commit rights, please?). More specifically, the 3.x efforts. It's an opportunity to get into a niche that will eventually become the standard. At the same time, porting and lending coding expertise will help the adoption rate. It's a win-win.

Here's a talk-by-talk rundown of the sessions I attended.

Keynote: Building the Python Community
Mr. Steve Holden


I enjoyed this as it was (surprise!) community based. Steve introduced the concept of a premium associate PSF membership. I like the idea. I think. Generally speaking, anything that can be done to improve the quality of a professional community is a good thing.

I do wonder if it will create a bit of an unintended dichotomy between members and non-members, though. You know, "fix my bug first, I pay each year!" Not to worry, I'm confident any issues like that will be worked out.

Keynote
Mr. Guido van Rossum


Well, I had one of my questions answered, so I'm happy. I was curious as to whether we need some type of 2-to-3 resource available. I'm fairly certain a phrased my question in the wrong context so it was interpreted as "we need a porting guide."

What I really meant, was:

Should we put up some sort of community page such that people with 3.0 porting experience can look to lend a hand to the projects that need it? People seem to be waiting for others to do some of the porting work.

Optimizations and Micro-Optimizations in CPython
Mr. Larry Hastings


What a way to start the morning. As I told my employer, they earned their money back with this one alone. For starters, I had no idea that GCC supported goto references. On top of that, doing so could yield a very large performance increase.

The big take away here was that there are a lot of optimizations under the hood. Writing good, clean code is paramount. Sure, it makes sense to worry about O(n) vs O(1) vs O(n^2), but that's a good place to stop and avoid obfuscation.

Python in the Browser
Mr. Jimmy Schementi


I attended this one as I work with a gaggle of .Net'ers. I figured some of the Silverlight exposure and Microsoft stuffs would help me understand some of the stuff they're working on these days. You know, help with team dynamics and all that.

You know what? You can run Python within Silverlight! Did you know that? Really? I didn't! Had no idea. That's pretty awesome. I intend to install whateveritisineedtoinstall in order to try this out one of these days.

The talk was rather informative, especially for a Linuxy guy like myself. There were a few technical issues, but that's fine by me. I took a lot of cross-platform knowledge out of this.

Import This, That, and the Other Thing: Custom Importers
Mr. Brett Cannon


Ah. Nice and advanced. Nothing like some brain pain in the morning. The talk covered how to write importers and what needs to be done in order to do it correctly. Handling dotted-names, package 'dunder init' files (see, I told you I'd use that term from here on out), scattered locations, sys.meta_path, and so on.

This is one of those things that I'm not likely to ever do. If I decide I have an need, Brett here has already put in the time to create a nice little library which ought to handle all of the details for me.

This talk was valuable as it gave an introduction into how this is done. While I may not do it, others will. The benefit here is that when a 3rd party custom importer fails, I'll understand what's going on a bit more.

Python 3: The Next Generation
Mr. Wesley J. Chun


A nice, well-rounded overview of Py3K. I knew most of this already, but I've yet to utilize it from a practical standpoint as I've yet to have a reason to. We're running 2.4 on most machines and 2.6 on the remainder.

I would like to note that Mr. Chun is a pretty good speaker.

Cross Platform Application Development and Distribution
Mr. Stani Michiels & Ms. Nadia Alramli


I need to make a confession here. I initially started with Deconstruction of an Object. However, that talk turned out to be geared to a developer with less experience so I ducked out. Mr Rush did a great job detailing the class structures, but I felt I would absorb more in a different talk.

This was my least favorite talk. The presenters did a wonderful job and their slide deck was very well done, but the material really wasn't relevant to what I'm doing each day. I was hoping for more focus on the development cycle. For example, Build Bot slaves on both Windows & Linux, handing dependancies, and so on. This focused more on UI elements. Great material, just not my area of focus.

Powerful Pythonic Patterns
Mr. Alex Martelli


My hope going into this talk was that it would come from a practical standpoint. I wanted to see some hard and fast code examples. I don't think in terms of object diagrams and pattern components for the most part and I figured that this would be a nice introduction to implementation from that standpoint. I left the talk looking for patterns in everyday life.

When us software guys hear the word "pattern", I assume that most of us think Gang of Four. Specifics such as Factory, Abstract Factory, or Facade spring to mind. While he touched on those, he also touched on more of the generic patterns and idioms that naturally develop.

So, while I still can't tell you exactly what a Memento implemented in Python would look like without Googling the term "Memento", I've started looking at my reproducible units of work for extractable patterns. Hopefully, I can up and refractor them to get slightly better performance.

How to Write Cross-Interpreter Python Programs
Mr. Maciej Fijalkowski


I fit into the bucket that most Python developers probably fit into. We write Python code for CPython. Perhaps IronPython. We don't switch between interpreters very much at all, really. I thought that hearing a different viewpoint would be a bit of a professional helper. I have two direct action items on my plate now due to this little talk.

First, I rely on reference counting in a lot of places. Mostly in my unit test code. What do I mean by that? Here's an example:

open('record.txt', 'w').write(record)
   
Now, we initialize a new file object with the call to open, write to it, and then drop the reference count immediately when the write call returns. In CPython, the file object is closed and cleaned up as the count drops to zero. This apparently isn't so in Jython and PyPy. The open file object can hang around for quite a while as they're not reference counting systems. I'm going to fix this.

Next, if it's practical, I'm going to look into setting up a Jython or a PyPy Build Bot slave. I'm not sure where version compatibilities are at this point in time.

Oh, and don't do crazy shit in your __del__ methods.

Keynote: Cadence, Quality, and Design
Mr. Mark Shuttleworth


Well, I live here in Atlanta. I use my iPhone alarm and it's set to go off every weekday. Well, guess what? Saturday isn't a weekday. I woke up far too late and walked in just as this talk got under way. I missed the earlier talks.

Mark really reiterated a lot of things we're already doing. Testing. Predictable cycles. Fall into a cadence or a rhythm. All good points and all very true. However, the speech wasn't overly earth shattering.

There was one comment that sticks out. Mark said something (and I'm paraphrasing here) like "The Good Enough Culture is Bullshit." I spent a good part of the day trying to keep that in mind. Under many circumstances, we do things to minimize work or to shave a few minutes. Productivity is important, but so is ensuring we're putting forth the highest quality possible.

Demystifying Non-Bocking and Async. I/O
Mr. Peter A. Portante


I use Twisted internally for an account provisioning API. As all of our internal code is synchronous, I just wrapped old calls with DeferToThread and move out of the async space. This is another example of one of those technologies I've used in a limited fashion. I understand it, I've just never really implemented anything with it. This helped to solidify it through discussion.

Learning Hosting Best-Practices From WebFaction
Mr. Brandon Craig Rhodes


Due to my line of work, I had a lot of hope for this one, and it held up. Brandon is a rather good speaker and the subject matter was right up my alley. I work in the same space, but not as a direct competition. It was very interesting to see how WebFaction does it.

This really solidified a theory I've had for a while. There are two distinct types of "webmaster." First, the technical user we're all familiar with. They want the bits and the bytes, the apps and the ports. However, there's also a class of very non-technical users that simply wants a basic site and they want to do it themselves. I believe the two stomp on each other quite a bit; finding the right middle ground is difficult.

I understand the plumbing of the Web Hosting industry inside and out. I learned a few things here that I can apply to my own work. Nicely done.

Understanding the Python GIL
Mr. David Beazley


Hands down, this was the best talk of the weekend. I understand the GIL and have worked around it in the past. I use Python threading quite a bit, but only for IO driven applications. For example, I'm parsing web logs for 80,000 web sites using a Python thread pool. It works great.

I'm not going to try to paraphrase as I won't do it justice, go read the documentation yourself, it's at http://www.dabeaz.com/talks.html.

I attended the Open Space talk on the GIL after this concluded. This stuff is all theoretical to me. I don't work on schedulers or priority queues during the day (or ever, for that matter). I know more about the GIL now than I ever thought I would.

Now, Mr DictatorForLife also took part in this talk. For a split second, I felt important. But then I remembered that I didn't have a damn thing to interject, so I left to go cry. That's my attempt at mid-blog humor. Go ahead. Laugh.

To Relate or Not To Relate
Mr. Mark Ramm


This was surprisingly relevant. As an organization, our data set isn't overly large. However, our deployment methodologies are somewhat time consuming and it can take a while to get new SQL servers deployed. Standard corporation purchasing stuff.

As a way around that, I have been looking at implementing something such as MongoDB or Cassandra. As we have a somewhat low overhead, I'm weighing our ability to deploy these services using local storage on a few hundred existing servers. We hit a few birds (Spotted Owls?) with the same proverbial stone:

1.Easier access to database capacity.
2. Reuse of existing local storage.
3. I get to play with something superneatocrazycool.

I had been leaning towards Cassandra, but MongoDB may be a better choice as it's implementation story seems simpler. Once the auto-sharding is complete, it's going to be a very strong contender.

Lastly, while we don't maintain a lot of data ourselves beyond standard customer records, we do offer databases to our customers as part of a hosting package. I also wonder what the feasibility of offering these things as a service is?

Threading is Not a Model
Joe Gregorio


At the end of the talk, someone made a comment that I've had a bit of trouble decompiling. He stated that Python does support Promises via Twisted's deferred object. Now, I'm not a guru in this space, but I think I disagree with that statement.

A promise runs a long running task in the background and then blocks when a function is called or an attribute is accessed if the requested variable is not yet around. A deferred, on the other hand, is loaded with a series of callbacks (errbacks) that fire when an event has completed. Are the concepts not different?

Sure, it's probably possible to implement a promise via the deferred mechanism, but it's probably much easier just to do it via a standard thread.

Python's Dusty Corners
Jack Diederich


Generally an overview of 'dunder methods' (see, there it is again!). I already had most of this in my brain, with the exception of the following:

If you have two objects, A and B, with B being a subclass of A, the comparison order is reversed if B appears after A in an object comparison.

Huh?

Given that C is of a different type, A == C will fire A.__eq__(self, C). However, in the A == B case, Python executes it as B.__eq__(self, A). The answer is simple. B is a more specific implementation and therefore should know more about the classes it's comparing.

Here's a code example.
class A(object):
    def __eq__(self, other):
        print 'Compared by A'

class B(A):
    def __eq__(self, other):
        print 'Compared by B'

class C(object): pass

A() == C()
A() == B()
When we run it, we get the following output:
jmcneil$ python comp.py 
Compared by A
Compared by B

I did attend an Open Spaces talk chartered by Doug Hellmann. We covered writing about Python. Initially I had believed it would be more of a talk regarding blogging & community documentation. It turned out to be focused more on authoring books. That's honestly something I've always wanted to do so I just sat back and soaked it all in. In the interim, I may make an effort to do some technical reviews.

That wraps it up PyCon 2010 for me as I wasn't able to attend Sunday's sessions. What a wonderful experience. This year was covered by my employer. I've already decided that if they do not cover it next year, I'll be paying for it out of pocket. It's time to start working on that already.

If you're an IT manager and you have the power to let your development staff attend, do so. It's in your best interest. Sure, a lot of recruiting goes down, but I promise that it's eclipsed only by the amount of learning.

I'll leave you with a final thought. PyCon indeed does have a little something for you, no matter what your tastes:

Saturday, January 16, 2010

Unit Tests with App Engine

Posted mostly for my own benefit (I know where to look later). 

I've been working around the socket limitations on App Engine this morning trying to get outbound XML-RPC requests to work properly.   I rigged up my unit test structure and designed the following little tests:

def test_xmlrpc_call(self):
    """Run XMLRPC Call"""

    class T(threading.Thread):
        def run(self):

            def rpc_method(x):
                return x

            server = SimpleXMLRPCServer(("localhost", 5432))
            server.register_function(rpc_method)
            server.handle_request()

    # Start XMLRPC Thread
    T().start()
    time.sleep(0.25)
    proxy = rpc.xmlrpc_proxy(LIVE_URL)
    self.assertTrue(proxy.rpc_method(True))

Fairly simple. It just creates a background XMLRPC thread and calls an identity function. As long as that returns True, I consider the test passing. This exercises all of my custom urlfetch transport code.

I ran the test suite and it bombed with the following error:

ProtocolError: protocolerror for localhost:5432/RPC2: \
   500 failed to fetch http://localhost:5432/RPC2: No \
      api proxy found for service "urlfetch"

I apologize for the formatting. I found this confusing as I defined a method above in an 'rpc' module named 'xmlrpc_proxy.' After about 20 minutes of digging, I discovered that if you elect to run App Engine code from the command line, then you need to manually rig up some of the Google API stubs. I must have missed that little detail.

So, in the root __init__.py of tests/ directory, I now have this:

"""
Special setup is required for certain App Engine stuff, namely
urlfetch proxies and data store.
"""

from google.appengine.api import apiproxy_stub_map
from google.appengine.api import urlfetch_stub

# Create a stub map so we can build App Engine mock stubs.
apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()

# Register App Engine mock stubs.
apiproxy_stub_map.apiproxy.RegisterStub(
    'urlfetch', urlfetch_stub.URLFetchServiceStub()) 

This wires up the stubs correctly and my App Engine unit tests pass as they should.

It's going to take a while to get used to coding for App Engine.

Friday, January 15, 2010

Help? Anyone?

So, my day job involves building, scaling, managing, tuning, tweaking, coding, breaking, fixing, and generally running the high availability clusters for Web.com.

Over the past few weeks, I've been working on finally getting FastCGI running on our cluster.  As part of that, I'm just about to deploy virtualenv & setuptools across the web and customer access SSH farms.  Assuming I know what I'm doing, end-users should now be able to run utilities such as Django, Pylons, and TurboGears via FastCGI/WSGI on our distributed Linux-based hosting platform.

I could really use a volunteer or three to test it out once it goes live.  Nothing big, just spend 30 minutes or so and throw up a quick example site and drop me a line if you see something goofy.  Anyone interested?