Django Forms and jQuery UI Dialogs

It's difficult, at the best of times, to specify user interface behavior in HTML5-based front-ends.  That is, if we want to utilize the best new browser technology has to offer, we need to recognize and understand how to use it.  We could, however, simplify the matter greatly by simply using HTML5 and CSS3, without Javascript. Minimal Javascript interruptions in our user interface design eliminate the complexity by at least half.  Browsers don't work that way.  Applications built for the web don't work that way.  There is a world of rich interactivity that we'd be missing out on should we decide not to use any Javascript libraries out there.

This is where things get interesting because most browser vendors do a decent enough job of providing HTML5 support that we can actually realize some of their benefits without fear of standard support.  And now we're left with the challenge of figuring out how to best integrate some of the more useful HTML5 features into our web applications.  This is balanced against the fact that the Javascript libraries we've invested in take care of a lot for us.  The very challenges we've faced in the past with prior versions of the HTML standard aren't really hindrances anymore.

So to find some practical usage that your current Javascript library doesn't take care of, or doesn't do so optimally, we have to look at more than just the technologies sitting inside the browser.  We have to mesh our user interface patterns with those of the application serving up these pages so as to form a complete architecture.

Identifying Javascript Challenges
One hurdle I've had to overcome myself is forcing HTML5 capabilities where they aren't justified because the same capability already exists in the Javascript framework.  For example, HTML5 tries to address the common problem of validating user input.  This not only handy — your validation criteria go directly into the markup — but it's a standard way of doing validation that works in any browser. So, if I'm starting to write a new user interface, maybe this is the solution.  Maybe not.  In my case, this doesn't work for a couple reasons.

First, Javascript frameworks I use, and probably most useful Javascript frameworks in existence, do some sort of user input validation.  Now is the time to ask, do I really want to adopt the standard HTML5 way of doing user input validation at the browser level, or is what I have in place going to do the job well enough?

Second, what implications does adopting HTML5 standards have on your web application framework?  For example, what would you need to implement differently on the server — does your framework support the HTML5 functionality you're aiming to implement?  It's a similar situation to that of Javascript libraries that have solved some of the challenges that HTML5 have rectified.  Maybe your framework has already solved some particular issues common in several application domains. But does the web application server fit well with your Javascript?  Is there a pattern of behavior that the Javascript toolkit you're using provides that'll fit tightly inside your application?

An effective method of identifying Javascript challenges is to look not at the user interface code alone, nor the application framework code alone.  Look at how they work together.  If you're like me, there probably isn't a bundled solution that tackles all your requirements — something like that, something that addresses the problem of synchronizing using interface code and the communication with the web application itself.  In fact, bridging the implementation inside the browser with what you're building on the server is a prevalent design issue.  One that I'm not convinced will ever be standardized globally — there are simply too many problem domains, and too many configurations to support every possible scenario.  That's where you can utilize HTML5 as a bridge between the two.

Using Data Attributes
As simple as it may sound, data attributes are an elegant solution to some of the implementation challenges I've faced.  Take a simple example — a Django form. The Django form API allows me to deffer much of the form-handling tasks to the framework itself — presenting forms, validation, and so on.  Now suppose I want to present this form in a jQuery UI dialog.  That's easy enough to do for the most part, but I need to pass a few additional details to the dialog widget.

One thing I need to tell the dialog widget about are the buttons.  I'll typically have a button to submit the form, and a button to cancel the action and close the dialog. To set the labels for these buttons, all attach the text to the form itself — using HTML5 data attributes.  When it comes time to build the dialog widget, my Javascript code simply grabs these attributes — jQuery has a data API that'll make this easy.  But if this is just text we're setting, why not do it straight in the Javascript?  Well, if we're setting it in the template, we can utilize some of the rendering tools supplied through Django's template system — things like internationalization and other string formatting filters.

This might seem like a trivial matter — passing data from the web application framework to the Javascript presentation layer — but it's important to get it right. There are probably several ways of doing this correctly — for example, building an API for our Javascript code to query — but the template-to-Javascript method seems like the shortest path.  In the Django world, we're not limited to merely formatting strings and passing them along to the Javascript user interface, we can use other features pertinent to the user interface like URL formatting.  No matter the framework, using HTML5 data attributes to store meta-data about the user interface means we're able to bundle the web application with any number of web browser technologies in a cohesive way.

Django Navigation States

I've tried several approaches to implementing Django navigation correctly.  Each approach presents it's own unique challenges and limitations.  The toughest thing to get right is the state.  By state, I mean the menu item that's currently active. Active navigational items need to be conveyed visually - this is easy enough from Django's perspective because it often means inserting the correct CSS class in the template.

Furthering the complexity to designing solid navigation in Django user interfaces is the fact that we can have dynamic variables in the URL path.  This makes a simple equality test in determining the state arduous.

So with this in mind, let me demonstrate the best Django-friendly approach to implementing state management with navigation.  I say Django-friendly because my approach attempts to be of use for any Django application.

Overview
The approach I'm using is similar to this one.  I'm creating a template filter that's applied to the HTTP request object inside a given template.  The filter takes a parameter - an identifier for the item we want state applied to.  So long as the identifier is valid, we can use this filter to emit the appropriate CSS class.

The Template
Here is the navigation template.  Intentionally simple, it shows how the link_state filter is applied.

<ul>
    <li><a class="{{ request|link_state:'home' }}" >Home</a></li>
    <li><a class="{{ request|link_state:'products' }}" >Products</a></li>
    <li><a class="{{ request|link_state:'services' }}" >Services</a></li>
</ul>

The Filter
Here is the link_state template filter used in the template to apply navigation item state.

from django import template
from django.core.urlresolvers import reverse, resolve, NoReverseMatch

register = template.Library()

@register.filter
def link_state(request, key):
    
    keys = dict(
        home = (
            'home',
            'promotion_details',
        ),
        products = (
            'product_list',
            'product_details',
        ),
        services = (
            'service_list',
            'service_details',
        )
    )
    
    url = resolve(request.path)
    
    for candidate in keys.get(key, []):
        try:
            candidate = reverse(
                candidate,
                args=url.args,
                kwargs=url.kwargs
            )
        except NoReverseMatch:
            continue
        if candidate == request.path:
            return 'active'
    return 'default'

Explanation
The link_state filter works by taking the HTTP request path, the navigational item key, and comparing URLs.  If a URL in the set associated with the key matches the request path, the state is active.  Otherwise, the state is default.

The keys dictionary maps navigational item keys to their active state URLs.  Here, there are three keys - home, products, and services.  Notice the relation to these names and arguments passed to link_state in the template.  Each key stores a tuple of URL names - when you define a URL in Django, you can give it a name.

Next, we resolve the request path.  The reason for doing so is that we need any dynamic path values so when we do comparisons, we can match these URLs too. Now we can iterate through each URL in the specified key, checking for a match.

This approach is flexible because it allows for navigational state management even with dynamic path values, without the need for any specialized code.  For example, the product_details URL might look like /products/845/.  This URL would set the products link as active.

Dealing With Permissions In Generic Views

Generic views in Django cover a vast range of usage scenarios. That's why they're called generic views — because they implement the abstract patterns common to most applications that are using views. This a huge savings on code-writing cost because there isn't as much of it to write — or maintain thereafter. So, this all well and good, but what about the pre-generic days of writing Django views? Is there nothing of value there we can carry forward and exploit in our newly-fashioned generic views?

Of course, before we used class based views, there were still some degree of generic capabilities that could be attached to our view behavior. They came in the form of decorators. Decorators in Python are a realization of the decorator pattern — we're attaching additional responsibilities to the function. Or, in this case, our view function. But things aren't so straight forward in the brave new generic world of writing views that use classes. How can we deal with things like permissions in our generic views while still honoring the DRY principle?

The decorated permission approach
The authentication system that ships with Django includes the ability to only expose certain views to specific users.  If we're writing views as functions, the authentication framework has decorators we can use to ensure the current user has the appropriate permissions.  Like this...

from django.shortcuts import render_to_response
from django.contrib.auth.decorators import permission_required

@permission_required('my_app.can_do_stuff')
def my_view(request):
    return render_to_response('my_template.html', dict())

This is an elegant approach to ensuring that only authorized user have access to my_view.  Since the permission_required decorator is provided by the authentication framework, it's available for every view in our application.  And, we only need to implement one line of code where we need to handle permissions. One per view.

The trouble with permission_required in modern Django applications is that they don't fit in nicely with the newer class-based generic view methodology.  So how then, can we exploit the power of the generic views in Django while keeping the simplicity of the decorated permission handling?

The inherited permission approach
One thing that class-based views offer that decorators don't is the ability to define defaults that the rest of the view hierarchy in our application can inherit.  These defaults include both data attributes and behavior.  This is one approach we can use with our class based views to simplify permission handling...

from django.views.generic import TemplateView
from django.http import HttpResponseForbidden

class MyAppTemplateView(TemplateView):
    
    perms = dict()
    
    def dispatch(self, request, *args, **kwargs):
        
        perms = self.perms.get(request.method.lower(), None)
        
        if perms and not request.user.has_perms(perms):
            return HttpResponseForbidden()
        
        parent = super(MyAppTemplateView, self)
        return parent.dispatch(request, *args, **kwargs)
        
class MyAppNews(MyAppTemplateView):
    
    template_name = 'news.html'
    perms = dict(
        get = ('myapp.can_see_news',)
    )

With this approach, able to control permission-based access based on the HTTP method — all within a dictionary overridden by descendant classes.  Here is how it works.

First, we're re-creating the base TemplateView class — called MyAppTemplateView.  Any other template views used in my application are now going to inherit from this class instead of the standard TemplateView defined by Django.  This is how we decorate each view with the added responsibility of ensuring access control.

The dispatch() method is the first call to action for any generic view.  So it is here that we want to approve of any additional execution.  What were doing here is really simple.  We're checking if the user has the permissions defined by the perms attribute.  And this is all we need to override.  By default, this attribute is an empty dictionary — so no permissions will be validated.

The MyAppNews view simply overrides the perms attribute to ensure the requesting user has the myapp.can_see_news permission.  It's not exactly a decorator, but we're retaining some flexibility while staying true to the DRY principle.

Preferring Filters To Tags

In Django, there are two ways to extend the template language — tags and filters. Tags are named tokens that insert HTML, executing logic behind the scenes.  For instance, an if tag and a for tag will conditionally produce content or iterate over a list objects respectfully.  The if tag in Django takes an argument — the condition to evaluate.

Filters, on the other hand, are different from tags — they don't spit out HTML markup or alter the overall template logic.  Instead, filters modify existing values in the template context for the sake of presentation.  For example, Django ships with a fileformat filter that'll display the argument, expressed in bytes, as a more readable representation.  This type of functionality is best encapsulated inside a filter because we're simply modifying the presentation of a single value.

I used to find myself grappling over whether something should be implemented as a tag or as a filter.  On the one hand, custom tags capacious changes — chunks of HTML or logic for manipulating the template.  On the other hand, filters are concise. They take input and modify it.  Filters exchange one piece of data for another. Django provides the machinery to do this, to slice up monolithic templates into reusable, independent components.  My inclination is that custom template tags can often be avoided in favor of included templates and filters.

Modular templates
If every page in our application were rendered using a single gargantuan template, we wouldn't be burdened with choosing between custom tags and custom filters. We'd have a number of other problems, no doubt, but I digress.  Instead, Django treats template files much like Python treats modules.  Django templates are modular.  Why write one large template, duplicated for individual views?  That approach doesn't follow the DRY principle, so we need a mechanism that'll allow us to treat our templates as components.

A component can have sub-components.  This abstraction works well in the Django template architecture because components are made up of smaller components. They can be decomposed and reconstructed, using smaller, loosely-coupled components.  In Django-speak, this means that we can start with a template that represents the entire page.  Logically, based on the application for which this template was designed, we can map out sections of this template that are likely to appear on all pages.  Things such as navigation, footer, and so forth.

These sub-components need to fill slots.  These slots are called blocks, in Django. Typically, the master template is extended by descendant templates whose job is to define what fills these slots.  Now we're starting to move down the composite template structure.  Beyond blocks, we need something to fill them with.

Eventually, fine-grained HTML markup is generated.  Some of it conditional, some of it iterative, and some using variables from the template context.  But even at this level, where we're filling in blocks, there are smaller pieces still.  Pieces that don't necessarily belong to a specific block.  Down yet another level, we're producing markup that might even appear twice on the same page — in two distinct blocks.

As the user interface of your Django project evolves and forms it's shape, you'll begin to notice these smaller chunks — those that might be of interest to multifarious blocks.  The main blocks, the logical regions of the master template are easy to grasp early on in development.  It's the smaller template components — the unbound HTML markup — that are more difficult to identify.  These low-level template components must take into consideration both custom template tags and custom template filters.

Sharing data
The tools that the Django template system gives developers enables the sharing of data.  Sharing between what exactly?  You extend the template language because you want to reuse those elements — custom tags and filters — across templates in your application.  But these new elements don't generate new data.  They're not storing application data.  Rather, the data new template elements share is transformed data.  They take input and alter what the template ultimately sends to the browser.

Let's say you've got an application that lists events of interest to the user when they first login.  These are probably queried from the database and rendered inside a for tag.  But maybe the event abstraction isn't limited to a user.  Maybe there are different event types that pertain to other abstractions in the application and aren't exclusive to the user.  Here, we might want to reuse a much of the same logic that rendered events on the homepage.  We're only changing the query.

We could follow the DRY principle here and implement a custom eventlist tag. This tag would accept some type of flag, indicating the query to execute in order to render the appropriate event list.  Using this tag in our templates is then quite straightforward — {% eventlist 'home' %} or {% eventlist 'updates' %}.  This tag is available in all templates — an easy way to share data across Django templates.

Our tag could even take care of rendering the corresponding HTML markup.  We'd simply register these eventlist tag as an inclusion tag.  We've now got our own template, isolated from other templates in our user interface dedicated to rendering event lists.

There are, however, a couple problems with this approach.  One, the template tag itself is responsible for executing the query.  This means that our view context passed to the template during rendering cannot alter the output of the event list. So if we want to use generic views that'll generate a list of events to render, we're stuck.  The eventlist tag is expecting an arbitrary flag.  We've given the Django template the added responsibility of performing database queries — which isn't a good thing.  We could alter our template tag so that it accepts a query set from the template context.  This way the view is still responsible for retrieving objects.  This is a good thing.  However, this leads to another problem.  One where we're violating the DRY principle.

Including and filtering
Tags are good because they're a concise way of sharing rendered HTML with other templates.  Our eventlist tag does exactly that.  But they're also a gateway into bad Django practice.  Defining your own template tags means writing some Python code.  Which gives us direct access to models.  Which means we can query them. Not good.  Not good as far as templates go because templates should be able to render anything I give them.  If I want to pass to my tag context, say, a list of Python objects that merely emulate one of my models, it should be able to handle that.  This isn't true if the template tags we define are going directly to the database for objects to render.

As for remedying the problem, we mentioned passing a query set, a list, something, to the template tag as an argument.  This somewhat solves the problem because now eventlist is polymorphic to an extent. It doesn't care what objects it gets as long as they look like event objects.

Recall, however, that we're defining eventlist as an inclusion tag.  This means a template dedicated to the list of events will be rendered whenever the tag is used. Keeping the HTML in the template is a smart idea — another win for us.  The eventlist tag is now simply passing the list of events to render.  Perfect.  Except, what doesn't the template tag do now?  It takes a list of objects and passes them into another template context.  It does exactly what the Django include tag does.

The include tag takes a smaller template component — in our case a list of rendered event objects — and plugs it into a larger component.  This sounds like an ideal candidate for our application.  Our views can now control the context — the list of events that get displayed in any given block.  And we're not repeating ourselves — we've got a template component that can be injected anywhere, and no need to define a custom template tag.

How about the template itself — the one we're including all over the site?  Are there any drawbacks to simply including it somewhere else without using our own tag? One challenge with making template components generic enough to be used anywhere is that they need to support a variety of scenarios.  What happens when the event object is in this state?  When it doesn't have a title attribute?  How do I handle CSS classes for events on this page?

Trying to handle all this in a single template that's used everywhere leads to messy template code.  In a hurry, you've got if statements inside HTML attributes and deep nesting levels elsewhere.  This is where defining custom template filters can alleviate some of these challenges.  Typical Django filters will take an input value and return a slightly different version of it — like a formatted date.  But we're not limited to only modifying the display of data.  We can return different values entirely.  Values used for the sole purpose of template rendering.

Template filter definitions are small.  We might have an input value, we do a few checks, and we return a different value.  Think of filters as dynamic template context modifiers — modifiers that can be shared throughout the application. Filters can also justify not having to define your own custom tags, ultimately leading to cleaner template code.

Django Class-Based Views: A Design Perspective

The release of Django 1.3 introduced the concept of class-based generic views.  Prior to 1.3, generic views were written as functions.  What's the big fuss over class-based views?  What can they possibly offer that functional views can't?  Convenience, less code, and elegant design - a property that extends down through Django's core.   From a design perspective, class-based views in Django are a better way to implement common view patters with marginal effort than generic function views.  For me, having only used class-based views for a few months, I'm very impressed.  When I compare generic functions with generic classes, I'm amazed at the difference in quality.  This is one of those small enhancements that I think will change Python web development for the better.

Maybe a little history first, shall we?  First of all, what makes Django views so useful and easy to work with in the first place?  We can tell Django how to map HTTP requests to our view functions using URL configurations.  This is useful because we have URL definitions separate from the functions themselves - multiple URL patterns can be handled by a single view function.  The URL path segments and query parameters are passed to the function as positional arguments and keyword arguments respectfully.

Having a Python function handle each HTTP request sent to the application is intuitive, no two ways about it.  The job of the view function is to take the client's request, perform some business logic, and return a rendering context for the template.  For common scenarios, generic view functions can be passed to the URL configuration.  For example, listing specific objects or for a detailed view of a single object.  For this, generic view functions are perfect.  We don't need to write repetitive code that differs only slightly from view to view.

However, we can't use generic views for everything and this is where the Django developer needs to step in and write some custom stuff.  Stuff that'll refine a query or pass additional context to the template.

Prior to Django 1.3, where generic class views were introduced, generic functions were customized by passing an info dictionary to the URL configuration.  This isn't an ideal coupling since the view now depends on additional data being passed to the URL definition.  For instance, if I have a URL where I want to return a list of objects, I can use a generic view that does just that.  However, to customize the query, I need to pass additional information to the URL definition that tells the view how to alter the query.  So if I wanted to reuse this same view, with the same query modifications, I'd have to duplicate the additional generic view info.  Not that this is painfully tedious, just error-prone.

Class-based generic views aims to solve the trouble of binding view-specific data to the URL, and to that end, I think they've succeeded hugely.  The new class-based approach introduces the concept of inheritance.  This method of extending views is less susceptible to problems with configuring URLs which depend on additional information.  Plus, extending generic views is simply elegant from a design perspective.

For example, we can extend the ListView class provided by Django to render a list of objects from our model.  At the most basic level, all we really need to tell ListView about is the model to select from.  Say we have a Book class.  We can define ourselves a new BookList class view for returning lists for books.  We're overriding the default model value of ListView.

I find this superior to the decorated customization approach.  With decorators, we're wrapping the view function with additional logic.  For instance, depending on the request context, HTTP headers, GET values, the URL, and so forth, we'll execute the logic of the generic view differently.  Nothing is wrong with this except for the fact that it doesn't sit well with doing things generically for a database view.  If I'm writing a list or detail view for things stored in the database, I want as little customization as possible if I'm using generic views.  This is a key point - the views are generic because you're supposed to give them only as little information as possible.  Just the bare minimum required to produce the appropriate template context.

With the new class-based approach, it's easier to leave the defaults alone, only overriding what's necessary.  The model name, for instance.  Or if you need more control over the resulting query set, overriding the get_queryset() method.  It's best to leave the common stuff to the generic view framework.  This is it's selling point, after all.  Things like automatically finding the template based on the model name.  If you absolutely must change something about a view, there is an attribute or a method you can override.  If you find that you're extending the Django generic views and they're becoming large, chances are you shouldn't be using them.  That is, if a method you've overridden on a Django generic class view looks nothing like a basic customization, you should think hard about why you're using a generic view in the first place.  Maybe you're better off writing a regular, non-generic view.  There is nothing wrong with this approach - that's why standard, function-based views exist - for writing business logic.

So from a design perspective, what do class-based generic views bring to the table?  A cleaner, more elegant mechanism for displaying database data.  In addition to decoupling the customizations that go along with generic views from the URL configurations, class-based generic views embrace inheritance as a means to use default behaviour and override only when necessary.  Often the generic behaviour is enough, just tell the view about the basic necessities, and you're off to the races.  Django offers a plentiful selection of built-in class-based views to choose from, each of which can be extended cleanly.

Django HTTP Response

The three main elements of an HTTP response are the body, the headers, and possibly cookie values. The Django HttpResponse object has support for all three, which is a requirement considering Django is a web application framework.

Whats cool about this Django class is how easy it is to test if a header exists. This is done by implementing the __contains__ method. It seems trivial when building classes, to add something like this in later, but it is easy to just forget about it.

It makes interacting with HttpResponse instances that much easier because there is one less step. Instead of doing if "key" in response_instance.headers().keys(), or something like that, Django HttpResponse instance allow developers to do if "key" in response_instance. The reason this functionality is implemented for headers and not for cookies is because headers are used more frequently than cookies in mos circumstances.

Django Cache Nodes

As part of the Django Python web application framework is a powerful template system. Many other Python web application frameworks rely on external template rendering packages whereas Django includes this functionality. Normally, it is a good idea to not re-invent the wheel an use existing functionality provided in other packages. Especially specialized packages that do only one thing like render templates. Django, however, is a batteries-included type of framework that doesn't really have external package dependencies.

The Django templating system is not all that different from other Python template rendering systems. It is quite straightforward for both developers and for UI designers to use.

Down at the code level, everything in a template is considered to be a node. In fact, there is a Node class that every template component inherits from. It could have been named TemplateComponent but that isn't the best name for a class. Node sounds better.

One type of node that may be found in Django templates is a cache node. These are template fragments that may be stored in the Django caching system once they have been rendered. Underlying these template cache nodes is the CacheNode class and is illustrated below.



As mentioned, every Django template node type extends from the Node class and CacheNode is no different. Also, as is shown in the illustration, the render() method is overridden to provide the specific caching functionality.

Illustrated below is an activity depicting how the CacheNode.render() method will attempt to retrieve a cached version of the node that has already been rendered and return that value instead if possible.



In this illustration, we start off with two objects, cache and args. The cache object represents the Django cache system as a whole. The args object is the context of the specific node about to be rendered. Next, the context is turned into an MD5 value. The reason for this is to produce a suitable value that can used to construct a cache key. Once this operation has completed, we now have a converted copy of the context. Next, we construct a cache key. This key serves as the query we submit to the Django cache system. Next, we perform the query, asking the cache system for a value based on the key we have just constructed. Finally, if a value was returned by the cache system, this is the rendered value that we return. If now value exists in the cache system, we now need to give it one. This means that we must render the node and pass the result to the cache system and also return this rendered value.

Applying Django Middleware

The Django Python web application framework supports the notion of middleware. What exactly is middleware? In the Django context, middleware are Python packages that do not explicitly belong to any particular Django application. Nor do these middleware packages belong to the Django package although Django does ship with some common middleware. The middleware Python packages that are independently installed sit in the middle.

So whats the point of separately installing Django components if they aren't part of either Django or the applications that use them? Well, since middleware components can be enabled in a Django application's settings, these middleware components may be shared between different applications that use the same Django installation. This keeps the Django core small.

The actual middleware components themselves are composed of classes. These classes must implicitly provide a Django middleware method. These methods can be one of process_request(), process_view(), process_response(), and process_exception().

The middleware components that an application wants to use are invoked automatically by the Django framework. All the application needs to do is enable the desired middleware components. The Django framework uses the BaseHandler class and the WSGIHandler class to invoke the middleware behavior. These classes are illustrated below.



Below is an illustration showing how the to classes, BaseHandler and WSGIHandler, collaborate to process all enabled middleware methods.

Popular Python Frameworks

The Python programming language is great for building web application frameworks. The two main reasons for this are that it is a very simple language to use and understand and it has fantastic networking libraries. Given these two facts, it is no wonder that there exist dozens of relatively solid web application frameworks written in Python.

The more popular web frameworks are the stable ones that have been around for some years. These frameworks have stood the test of time and have a large feature set.

I found another interesting way to look at which frameworks are the popular frameworks by using the Python package index. I browsed the available packages by framework to see which ones have the most packages. The Zope world is still dominating the Python web application framework market. Django is slowly catching up. At the time of this writing, here are the top frameworks listed by number of available packages.

• Plone (880)
• Zope3 (872)
• Zope2 (475)
• Django (295)

This list also demonstrates which frameworks are extensible because the easier it is to extend a software package, the more developers are willing to extend it with other packages and release them. What is surprising is the small number of Twisted and Trac packages. Both frameworks are well written and easily extensible. Having said that, the number of packages listed isn't entirely accurate because not all conceivable framework package lives in the Python package index. Also, there are most likely some categorization errors to take into account.

Shrinking Python Frameworks

An older entry by Ian Bicking talks about the shrinking world of Python web application frameworks. It is still a valid statement today, I think. There is no shortage of Python web application frameworks in which to choose from. Quite the contrary, it seems that a new one springs into existence every month. This often happens because a set of developers have a very niche application to develop and the existing web application frameworks don't cut it. Either that or they are missing a feature or two, or they have too many moving parts and so they will make some modifications. Whatever the difference, some developers will release their frameworks as an open source project.

The shrinking aspect refers to the number of frameworks which are a realistic choice for implementing a production grade application. Most of the newer Python web application frameworks, still in their infancy, are simply not stable enough.

Take Pylons and TurboGears for instance. Both are still OK web frameworks, you can't have TurboGears without Pylons now. However, they are somewhat problematic to implement applications with. Even if stable enough, there are complexities that shouldn't exist in a framework. Besides, I have yet to see a stable TurboGears release.

Taking complexity to a new level is Zope. This framework has been around for a long time and is extremely stable. But unless you have been using it for several years, it isn't really worth it because of the potential for misuse is so high.

The choice of which Python web application framework to use really comes down to how much programming freedom you want. If you want everything included, Django does everything and is very stable. However, if there are still many unknowns in the application in question, there are many stable frameworks that will simply expose WSGI controllers and allow the developers to do as they will.

Django Form Fields

The Django Python web application framework comes with almost every component that one might need to construct a full-featured application. One common component of web applications are widgets. Widgets are similarly defined in most desktop GUI libraries. A widget is simply a smaller part of the GUI whole. Widgets typically have a tightly defined concept and purpose. For example, a button widget is meant to be clicked. Also, a button widget is also meant to be visually striking enough that it is obvious that it is meant to be clicked. Django comes with a set of widgets that are featured in almost every web application in one way or another.

Web applications generally contain forms. Nobody has gotten away with using the web without having to fill out a form at some point. The pieces of these forms are often referred to as fields. This is from a data or domain perspective. When talking about a field, you are generally talking about what data that field contains, how that data is validated and so on. When talking about widgets in a form, you are generally talking about the visual aspect, like what can this widget do and how does it look when it does it.

Django provides abstractions within the framework for both of these concepts. The two concepts are closely related and thus tightly coupled. In this particular situation, tight coupling is absolutely necessary. You can't have a field without a widget and vice-versa. The alternative would be to implement all the field functionality inside the widget abstraction which would give us something bloated and hard to understand. Modularity is a good thing even when tight coupling is necessary. The two classes representing these concepts are Widget and Field. The two classes and how they are related to one another are illustrated below.

Granular Django Cache

Like many other web application frameworks, Django has a built-in caching system. Unlike other web application frameworks, the Django cache system is relatively straightforward to configure and use. Configuring the cache system can be as simple as specifying where the cached items are stored. With the Django cache system, developers have plenty of options. There is even a dummy cache storage that can be used for development purposes. Whichever back-end cache system you decide to use, it can be specified in the CACHE_BACKEND configuration value.

Once the cache storage location has been setup, caching can be implemented at any number of levels from per-site to low-level. The most effective way to implement Django cache, I find, is to implement it on a per-view basis. Using this method to implement cache means that cached items are created for each URL that is requested if the view mapped to the URL is cached. Using the lower level Django cache constructs are nearly impossible to manage for larger, more complex applications. They do exist, however, for niche situations.

The cache_page() function is responsible for creating a page cache. The function takes a view to be cached and a timeout as parameters. Once the timeout has expired, any cached items are no longer valid. Although the cache_page() function can be used as a decorator on the view declaration, it makes more sense to pass the view as a parameter to cache_page() within the URL configuration. This is the more portable way of doing things and is better aligned logically since the URL serves as the cache key, not the view name.

Django Ajax Response

The Django Python web application framework is capable of many types of data serialization.  Be it, XML, or JSON, The built-in Django serialize() function can handle it.  The transformation typically starts with a Python dictionary or list but can also handle instances of user-defined types.  The end result is always a string.  The string is of course desired because that is what will be passed along inside the HTTP response.  It is nice to have this functionality, but what is it used for?  Why not just use standard templates with template variables and let the view render it for the client?  The main reason is that more and more non-browser clients are being used with web applications.  Even if the client is a web browser, there is a good chance that the request is coming from an ajax application and they don't always like HTML responses.  Most prefer the JSON format.

There is, however, a good chance that more than one format is going to be necessary for the same data set.  For instance, I might have a standard Python dictionary that I want to return to the client.  Depending on who the client is, that same data is going to be rendered differently.  This is so that the client can understand the response.  Say, for instance, that the client was a javascript application.  This client could, potentially, have the ability to selectively handle different response formats that server returns to it.  This responsibility shouldn't really be left to the client application.  It would be nice if the Django application could automatically determine if a javascript application is requesting the data instead of a standard web request.  Thankfully, Django can do this without much developer intervention.

Django HTTP request instances can determine if the request came from an ajax application.  Django does this under the hood by examining the HTTP headers.  As is illustrated by the following example, simple scenarios like this one can be implemented with a single controller.

#Example; Django Ajax response.

#Import Django components.
from django.http import HttpResponse
from django.template import Context, Template
from django.core.serializers import serialize

#Initialize the test data object.
data_obj={"first_name":"First Name", "last_name":"Last Name"}

#The test view.
def index(request):
    #Check if this is an ajax request.
    if request.is_ajax():

        #Set the result to serialized JSON data.
        result=serialize("json", data_obj)

    else:
        #Create a context object from the test data.
        context_obj=Context(data_obj)
        
        #Create a test template string.
        template_str="""<b>first_name</b>: {{first_name}}<br/>
                        <b>last_name</b>:  {{last_name}}"""
                        
        #Set the result to the rendered HTML.
        result=Template(template_str).render(context_obj)

    #Return the response.
    return HttpResponse(result)

Django Content Files

The Django web application framework written in Python defines a file abstraction used for working with files within the file system. Web applications often have to deal with many very different file formats. These aren't just static files that get served to the client using the system, they are also used to parse and retrieve useful file metadata. An example of this useful metadata would be the image dimensions of an image file. There is other useful file metadata that the clients of the system may not necessarily be concerned with although the system may be.

The File class is the base Django file abstraction. Instances of this class extend the concept of file-like objects in regular Python applications. This abstraction comes in handy when iterating through file contents. Django as a certain iteration style used throughout the framework and this abstraction helps maintain it. The File class is also helpful when dealing with images; it is the base class of the ImageFile class.

Code in Django applications can remain consistent with the file abstraction even when using regular content. Similar to how StringIO works. The ContentFile class inherits from the File class and ContentFile instances and behave just like File instances do. The main difference of course being that the ContentFile only uses raw data instead of data that lives on the file system. This gives Django a huge interface consistency boost when dealing with string data.

Django Boundary Iterators

The Django Python web application framework provides tools to parse multi-part form data as any web framework should. In fact, the developer responsible for creating Django web applications need not concern themselves with the underlying parsing machinery. However, it is still there and very accessible.

The Django boundary iterators are used to help parse multi-part form data. These iterators are also general enough to be used in different contexts. The BoundaryIter class collaborates with other iterator classes such as ChunkIter and LazyStream. An example of these classes collaborating are illustrated below.

#Example; Using the Django boundary iterator.

#Imports.
from StringIO import StringIO
from django.http.multipartparser import ChunkIter, BoundaryIter, LazyStream
                                    
if __name__=="__main__":
  #Boundary data.
  b_boundary="-B-"
  c_boundary="-C-"

  #Example message with two boundaries.
  _message="bdata%scdata%s"%(b_boundary, c_boundary)

  #Instantiate a chunking iterator for file-like objects.
  _chunk_iter=ChunkIter(StringIO(_message))

  #Instantiate a lazy data stream using the chunking iterator.
  _lazy_stream=LazyStream(_chunk_iter)

  #Instantiate two boundary iterators.
  _bboundary_iter=BoundaryIter(_lazy_stream, b_boundary)
  _cboundary_iter=BoundaryIter(_lazy_stream, c_boundary)

  #Display the parsed boundary data.
  for data in _bboundary_iter:
      print "%s:      %s"%(b_boundary, data)

  for data in _cboundary_iter:
      print "%s:      %s"%(c_boundary, data)

In this example, we have a sample message containing two boundaries that is to be parsed. In order to achieve this, we create three iterators, and a stream. The _chunk_iter iterator is a ChunkIter instance that is a very general iterator used to read a single chunk of data at a time. This iterator expects a file-like object to iterate over. The two boundary iterators, _bboundary_iter and _cboundary_iter, are instances of the BoundaryIter class. This iterators expect both a data stream and a boundary. In this case, a LazyStream instance is passed to the boundary iterators.

We finally display the results of iterating over the boundary iterators. By the time the second iterator is reached, the data stream is now shorter in length.

Sending Django Dispatch Signals

In any given software system, there exist events that take place. Without events, the system would in fact not be a system at all. Instead, we would have nothing more than a schema. In addition to events taking place, there are often, but not always, responses to those events. Events can be thought of abstractly or modeled explicitly. For instance, the method invocation "obj.do_something()" could be considered an invocation event or a "do something" event. This would be an abstract way of thinking about events in an object oriented system. Developers may not even think of a method invocation as an event taking place. However, the abstraction is there if needed. A method invocation is an event when it needs to be because it has a location in both space and time. Events can also be modeled explicitly in code. This is the case when designing a system that employs a publish-subscribe event system. Events are explicitly published while the responses to events can subscribe to them. Another form of event terminology that is often used is to replace event with signal. This is the terminology used by the Django Python web application framework dispatching system.

Django defines a single Signal base class in dispatcher.py and is a crucial part of the dispatching system. The responsibility of the Signal class is to serve as a base class for all signal types that may dispatched in the system. In the Django signal dispatching system, signal instances are dispatched to receivers. Signal instances can't just spontaneously decide to send themselves. There has to be some motivating party and in the Django signal dispatching system, this concept is referred to as the sender. Thus, the three core concepts of the Django signal dispatching system are signal, sender, and receiver. The relationship between the three concepts is illustrated below.



Senders of signals may dispatch a signal to zero or more receivers. The only way that zero receivers receive a given signal is if zero receivers have been connected to that signal. Additionally, receivers, once connected to a given signal, have the option of only accepting signals from a specific sender.

So how does one wire the required connections between these signal concepts in the Django signal dispatching system? Receivers can connect to specific signal types by invoking the Signal.connect() method on the desired signal instance. The receiver that is being connected to the signal is passed to this method as a parameter. If this receiver is to only accept these signals from specific senders, the sender can also be specified as an parameter to this method. Once connected, the receiver will be activated once any of these signal types have been sent by a sender. A sender can send a signal by invoking the Signal.send() method. The sender itself is passed as a parameter to this method. This is a required parameter even though the receiver may not necessarily care who sent the signal. However, it is good practice to not take these chances. If, from a signal sending point of view, there is always a consistency in regards to who the sender is, there is a new lever of flexibility on the receiving end. Illustrated below is a sample interaction between a sender and a receiver using the Django signal dispatching system to send a signal.



The fact that the signal instances themselves are responsible for connecting receivers to signals as well as the actual sending of the signals may seem counter-intuitive at first. Especially if one is used to working with publish-subscribe style event systems. In these event systems, the publishing and subscribing mechanisms are independent from the publisher and subscriber entities. However, in the end, the same effect is achieved.

Django Templates and NodeLists

Whether dealing with a complex web application or a simple web site with relatively few pages, using templates is generally a good idea. With complex page structures, this is unavoidable. The alternative would be to implement the page structure manually. Even with simplistic web pages, there is often a dynamic element that is undesirable to manually edit. If the web application is built with a Python web application, developers have several template engines to choose from. TuboGears, for instance, offers support for several third-party template engines. Django, on the other hand, offers their own template rendering system. The Django template engine provides many of the same features as do other Python template engines. The syntax used in Django templates is simplistic and easy to use. Inside the template engine are two closely related classes that represent key template concepts and are illustrated below. These classes are Template and NodeList.



The Template class represents a template in its' entirety. During the construction of Template instances, the template is compiled. Since this process takes place in the Template constructor, the template string is a required constructor parameter. Since any given Template instance is compiled, it may be rendered at any time. The template passed to the constructor is only compiled once since the compilation takes place in the constructor. Template instances support iteration. Each iteration through a particular Template instance yields a node from the NodeList instance that resulted from compiling the template. Invoking Template.render() will in turn invoke NodeList.render(), passing along the specified context.

The NodeList represents a collection of nodes that result from compiling a template string. The majority of these nodes are HTML elements, template variables, or some Python language construct such as an if statement. NodeList instances are also instances of the Python list primitive. This means that each node contained within the list behaves just like a Python list element. The NodeList.render() method will cumulatively invoke render() on each node contained within the list. The end result is the rendered template. This rendering behavior provided by the NodeList.render() method can be considered polymorphic. The method iteratively invokes render() on anonymous instances. All it cares about is the render() interface.

The Django paginator

Most, if not all, modern web applications need pagination in one form or another. Pagination is the act of transforming a large data set into pages of a more manageable size. Google gives us a perfect example case of pagination. Google constantly deals with enormous data sets. Users who perform searches using google would promptly switch to a different search engine if there were no pagination provided. On the other side of the coin, pagination also provides more manageable data sets for the server code to deal with. Instead of the client saying "give me this entire large data set" the client says "give me page one of this large data set, page size being ten". From the developer perspective, pagination isn't always the most enjoyable task. They are error prone if not implemented correctly and can pose challenges when different query constraints come into play. An additional challenge with pagination is the fact that different web application frameworks use slightly different approaches to their pagination implementation. Another approach to implementing pagination could be to implement the functionality directly into the ORM. This would obviously only work with database query results but this is probably the most common use for pagination. However, if an ORM like SQLAlchemy for instance, were to implement pagination functionality, it could be used by developers inside a web application framework while still being functional outside the framework. The Django Python web application framework offers a good pagination implementation. It is not restricted to database query results and is easy for developers to understand and use. There is also much room in the implementation for extended functionality if so desired. To two classes used to implement the pagination functionality in Django are Paginator and Page as illustrated below.





The main class used by the Django web application framework is the Paginator class. The Paginator class deals directly with the data set in question. The constructor accepts an object_list as a required parameter. The constructor also accepts a per_page parameter that specifies the page size. This parameter is also required. Once the Paginator class has been instantiated with a data set, it can be used to generate Page instances. This is done by invoking the Paginator.page() method, specifying the desired page number. The Paginator class also defines several managed properties that can be used to query the state of the Paginator instance. Managed properties in Python are simply methods that may be invoked as attributes. The count attribute will return the number of objects in the data set. The num_pages attribute will return the number of pages that the data set contains, based on the specified page size. Finally, the page_range attribute will return a list containing each page number.

There is room for extended functionality in the Paginator class. Mainly, there could be some operators overloaded to make Paginator instances behave more like new-style Python instances. The Paginator class could define a __len__() method. This way, developers could invoke the builtin len() function on Paginator instances to retrieve the number of pages. An __iter__() method could also be defined for the Paginator class. This would allow instances of this class to be used in iterations. Each iteration would yield a new Page instance. Finally, a __getitem__() method would allow Paginator instances to behave like Python lists. The desired page number could be specified as an index.

The Page class complements the Paginator class in that it represents a specific page within the data set managed by the Paginator instance. The has_next(), has_previous() and has_other_pages() methods of Page instances are useful in determining if the page has any neighbouring pages. The start_index() method will return the index in the original data set owned by the Paginator instance that created the Page instance. The end_index() will return the end index in the original data set.

Like the Paginator class, there is also room for extended functionality here to make Page instances behave more like new-style Python instances. The Page class could define a __len__() method that could return the page size. The Page class could also define an __iter__() method that could enable Page instances to be used in iterations. Finally, the __getitem__() method, if it were defined, could return the specified object from the original object list.

Magic methods of the Django QuerySet

The QuerySet class defined in the Django Python web application framework is used to manage query results returned from the database. From the developer perspective, QuerySet instances provide a high-level interface for dealing with query results. Part of the reason for this is that instances of this class behave very similarly to other Python primitive types such as lists. This is accomplished by defining "magic" methods, or, operator overloading. If an operator for a specific instance is overloaded, it simply means that the default behavior for that operator has been replaced when the operand involves this instance. This means that we can do things such as use two QuerySet instances in an and expression that will invoke custom behavior rather than the default behavior. This useful because it is generally easier and more intuitive to use operators in certain contexts than invoking methods. In the case of using two instances in an and expression, it makes more sense to use the built-in and Python keyword than it does to invoke some and() method on the instance. The Django provides several "magic" methods that do just this and we will discuss some of them below.

Python instances that need to provide custom pickling behavior need to implement the __getstate__() method. The QuerySet class provides an implementation of this method. The Django QuerySet implementation removes any references to self by copying the __dict__ attribute. Here is an example of how this method might get invoked.

import pickle
pickle.dumps(query_set_obj)

The representation of Python instances is provided by the __repr__() method if it is defined. The Django QuerySet implementation will call the builtin list() function on itself. The size of the resulting list is based on the REPR_OUTPUT_SIZE variable which defaults to 20. The result of calling the builtin repr() function on this new list is then returned. Here is an example of how this method might get invoked.

print query_set_obj

The length of the QuerySet instance can be obtained by calling the builtin len() function while using the instance as the parameter. In order for this to work, a __len__() method must be defined by the instance. In the case of Django QuerySet instances, the first thing checked is the length of the result cache. The result cache is simply a Python list that exists in Python memory to try and save on database query costs. However, if the result cache is empty, it is filled by the QuerySet iterator. If the result cache is not empty, it is extended by the QuerySet iterator. This means that the result cache is updated with any results that should be in the result cache but are not at the time of the __len__() invocation. The builtin len() function is then called on the result cache and returned. Here is an example of how this method might get invoked.

print len(query_set_obj)

Python allows user-defined instance to participate in iterations. In order to do so, the class must define an __iter__() method. This method defines the behavior for how individual elements in a set are returned in an iteration. The QuerySet implementation of this method will first check if the result cache exists. If the QuerySet result cache is empty, the iterator defined for the QuerySet instance is returned. The iterator does the actual SQL querying and so in this scenario, the iteration looses out on the performance gained by having cached results. However, if a result cache does exist in the QuerySet instance, the builtin iter() function is called on the cache and this new iterator is returned. Here is an example of how the __iter__() method might be invoked.

for row in query_set_obj:
   print row

Python instances can also take part in if statements and invoke custom behavior defined by the __nonzero__() method. If an instance defines this method, it will be invoked if the instance is an operand in a truth test. The Django QuerySet implementation of this method first checks for a result cache, as does most of the other "magic" methods. If the result cache does exist, it will return the result of calling the builtin bool() function on the result cache. If there is no result cache yet, the method will attempt to retrieve the first item by performing an iteration on the QuerySet instance. If the first item cannot be found, false is returned. Here is an example of how the __nonzero__() might be invoked.

if query_set_obj:
   print "NOT EMPTY"

Finally, Python instance may be part or and an or Python expressions. The Django QuerySet instance defines both __and__() and __or__() methods. When these methods are invoked, they will change the underlying SQL query used by returning a new QuerySet instance. Here is an example of how both these methods may be used.

print query_set_obj1 and query_set_obj2
print query_set_obj1 or query_set_obj2

Django WSGI handlers

WSGI handlers in the Django Python web application framework are used to provide the WSGI Python interface. This means that the Django framework can easily support middleware projects that manipulate the response sent back to the client. The code for the WSGIHandler class and the supporting WSGIRequest class can be found in the wsgi.py Python module. Here is a simple illustration of what the module provides.



Just as a brief overview of what this module contains, I'll simply describe each of the four classes shown above. First, the HttpRequest class provide the basic HTTP request attributes and methods. The BaseHandler class provides attributes and methods common to all handlers. The WSGIRequest class is a specialized HTTP request. The WSGIHandler class is a specialized handler.

The WSGIHandler class depends on the WSGIRequest class because it instantiates the request instances. This is done by setting the WSGIRequest class as the request_class attribute. There are two aspects of the WSGIHandler class that I find interesting.

First, this class isn't instantiated. There is no __init__() method defined and all attributes are class attributes. However, the __call__() method is defined so this class can actually be invoked is if we were creating an instance only we get a response object returned instead. At the beginning of the __call__() method, the WSGIHandler class acquires a threading lock. This lock is than released after the middleware has been loaded.

Second, each middleware method is than cumulatively invoked on the response so as to filter it. Very cool!