Python processing

I've discovered an interesting Python package called processing. The name may be a little misleading but it is nonetheless useful. The reason it is misleading is that from the developer API perspective it is a threading API which mimics the built-in Python threading module. The main difference being, of course, it uses processes instead of threads.

This comes in handy when dealing with the controversial global interpreter lock (although I have no problem with the GIL). On multi-core and multi-processor systems, multi-threaded applications do not distribute threads amongst these processors. This is because of the Python GIL. The GIL is used to ensure safety with mutable data types.

However, new processes are distributed amongst processors. This is the issue the processing module aims to solve. I recommend trying it out. I also issue a warning. Spawning new processes more computationally expensive than spawning new threads no matter what platform is in use. There is no substitute for increasing application responsiveness with threads. However, longer-running algorithms may be better suited in a new process.

Gaphor services

The Gaphor UML modeling tool offers a simple service framework under the covers. The Gaphor application itself defines some core services that fit within this framework. For example, it defines an ActionManager service that is used to handle various actions within the application.

All services that are defined within Gaphor (or as extensions to Gaphor), implement the IService interface. This interface is defined in the interfaces module. Here is a simple illustration of what this interface looks like.



As you can see, this straight-forward interfaces requires that any Gaphor services implement an init and a shutdown method.

The ActionManager service is a good example of a Gaphor service. This service is responsible for maintaining actions in the application. It is also a service that is responsible for loading other services. For example, the FileManager service is also an action provider since it implements the IActionProvider interface. The ActionManager service loads all components that provide the IActionProvider interface and registers them as action providers within the application.

PyXMI

A few years back, I had a failed attempt at creating an open-source UML activity diagram runner called PyXMI. As the name implies, it was a Python program that attempted to load UML modules that were saved in the XMI format, and execute activity diagrams.

I think this project failed mostly because of my lack of understanding in the UML specification area as well as my lack of any use cases for such a project. I still do think it would be quite useful for such a library to exist. Having the ability to draw work-flow visually and have it directly executed (as opposed to generating the source code, and executing that), is a powerful tool. Maybe not in production use but as far as testing new ideas goes, this could offer insight that is not available by other means. Besides, that is why the UML was created, to visualize systems.

What if, besides generating the XMI document and executing that, we were able to simulate the model that is held in memory by the modeling tool? I know of no open-source project that can do this but the existing projects offer the components necessary to realize this functionality. Ideally, this would be fairly trivial to do if the modeling language is written in a dynamic language such as Python. Gaphor is written in Python and the code is of high quality and easy to grasp.

One feature Gaphor that is missing is colour support in the diagram canvas. Umbrello does offer colour support and makes for a good example of simulating a running activity diagram. For example, consider the following activity diagram.



This diagram is extremely straight-forward so I'm not going to bother explaining what it represents. Next, if we were running a simulation of this diagram, the modeling software could step through the activities and change the colour of the currently executing activity. Once an activity transition is reached, however, the model would need to pause for at least half a second so as to not fly through the execution. The reader should have the opportunity to view the execution as is illustrated below.



Now we process the input.




Finally, we send the output.




Hopefully, Gaphor will soon implement colour support so I can start experimenting with a plugin that will make another attempt at realizing executable/simulatable activity diagrams.

boduch 0.0.2

The new boduch Python package is now available on pypi. Some changes include:

• Added a new Set class.
• EventManager.prioritize() will now prioritize the event handlers.
• Added some new event classes.
• Added some new unit tests.
• API documentation enhancements.

The new Set class is basically a Python list that emits events. The class supports push, pop, get, and sort methods each of which emit a different event type. Also, the EventManager class will now sort the handles for any given event according to their priority (which is the sum given by the handle).

In 0.0.3, I plan to add the ability to execute event handles in a threadless mode. This will most likely be a parameter passed to publish(). I'm also going to add some constants that will hopefully come in handy when setting the priority of event handles.

Resource design with twisted.web

The twisted.web Python package offers several resource abstractions that make for a quality RESTful design in my opinion. Putting aside the the Python web framework requirement of "does it do everything I could possibly need ever need?" and focusing on designing an API that cohesive and easily understandable, the twisted.web package is very useful indeed.

The package offers a Resource class that serves as the base class for all resources in your application. As a developer, you will extend (and redefine if needed) this class to provide the behavior of your resources. To create a uniform API, from the RESTful perspective, we implement render_GET(), render_POST(), etc. These the value returned from these methods is the response given to the client.

Here is a trivial example of using the Resource class to implement a blogging system.

#Resource design with twisted.web

from twisted.web import server, resource, util
from twisted.internet import reactor

blog_data=[{'title':'First Entry', 'body':'First Entry Body...'},\
         {'title':'Second Entry', 'body':'Second Entry Body...'}]

class Blog(resource.Resource):
  def getChild(self, path, request):
      return util.Redirect('index')   

class BlogIndex(resource.Resource):
  isLeaf=True
  def render_GET(self, request):
      content=''
      template='<a href="/entry/%s/">%s</a><br/>'
      cnt=0
      for i in blog_data:
          content+=template%(cnt, i['title'])
          cnt+=1
      return '<html>%s</html>'%(content)

class BlogEntry(resource.Resource):
  isLeaf=True
  def render_GET(self, request):
      template='<html><h1>%s</h1><p>%s</p></html>'
      blog_id=int(request.postpath[0])
      blog_entry=blog_data[blog_id]
      return template%(blog_entry['title'], blog_entry['body'])

blog_obj=Blog()
blog_obj.putChild('index', BlogIndex())
blog_obj.putChild('entry', BlogEntry())
reactor.listenTCP(8080, server.Site(blog_obj))
reactor.run()

Some things to note in this example:

• The Blog resource is the root resource. It redefines the getChild() method in order to return the Redirect resource which will take us to the BlogIndex resource.
• The BlogIndex resource simply displays a list of sample blog entries.
• The BlogEntry resource represents a specific blog entry. This resource requires a blog id (list index). It will then display the blog entry.
• The main program first creates the Blog resource and adds the child resources. The server is then started using the Blog resource as the root resource.
  

ECP 2.1.1

The latest bug fix release of ECP is now available. Here are some of the changes:

• Randomly generated mac addresses are now written to the machine XML at provision time.
• The available system memory is now checked against the required memory for new machines at provision time.
• Fixed a bug regarding the valet extension module not properly checking the hypervisor type.
• Fixed a bug that disallows a machines' XML definition to be edited.
• Fixed several misc. bugs in the valet extension module.

New boduch Python library.

I've made a new Python library called boduch available on pypi. Why did I call it boduch? Because it is simply a Python package where I can publish any cool Python tools and I couldn't think of a better name.

The current version has my first attempt at an event publishing system. Here is an example usage scenario.

#Event example using boduch.event

from boduch.event import Event, publish, subscribe, lock, unlock
from boduch.handle import Handle

class MyHandler(Handle):
   def __init__(self, *args, **kw):
       Handle.__init__(self, *args, **kw)
      
   def run(self):
       print "Got event",self.data["event"]

if __name__=="__main__":
   subscribe(Event, MyHandler)
   publish(Event)
  

Here, we have defined a simple handler that prints out the event it received. In the main program, we subscribe our handler to the base Event. We then publish our event.

Transactions in Gaphor

The Gaphor UML modeling tool has support for transactions. There is a decorator called transactional that wraps the specified function in a transaction. If an exception is raised, the entire transaction is rolled-back.

The transaction.py module also defines the Transaction class as well as a TransactionError exception.



Here we can see that the Transaction class provides the ITransaction interface and also depends on the TransactionError exception. This exception is raised by the Transaction._close() method if there are no transactions on the transaction stack or if the only transaction on the stack is not the current transaction being closed.

There are also three events that are initiated by the Transaction class; TransactionBegin, TransactionCommit, and TransactionRollback.

Here is an example using the transactional decorator.

#Gaphor transaction decorator.

from gaphor.transaction import transactional

class MyDB:
   @transactional
   def update(self, data):
       """Update my hypothetical DB with data."""
       print "Updating..."

if __name__=="__main__":
   db_obj=MyDB()
   db_obj.update(123)

An important note about the example and the transactional decorator. There are no return values from the transactional-decorated functions or methods. This makes sense since the return value may no longer be valid if the transaction was rolled-back.

Distributing unit tests.

In this context, I'm referring to distributing unit tests along with a software package, as opposed to executing unit tests in a distributed fashion (which is an interesting topic in its' own right). Unit testing has proven to be an essential development artifact to ensure all use cases are executed correctly. I wish I could say all open source projects distribute unit tests as a component of the package. This is simply not the case. Several projects do, however, include a testing facility which is usually composed of unit tests.

Why, if the authors of these software packages spend time writing all these unit tests, should they be distributed along with the software? After all, they write these unit tests for their own testing. Sometimes an entire testing team is responsible for this task.

I would say that no matter how big a testing team any given project has, they are never going to cover all possible corner cases. That goes without saying (but I'm saying it anyway). Custom operating environments, even some other piece of installed software, could cause unexpected behavior that is in no way handled correctly be the software in question. At least having unit testing available in these situations can offer some clue.

Trac provides a good example of RESTful resources

The Trac issue tracking and wiki system provides a good example of a RESTful web resource. Good RESTful resources are connected. This means that the associations between resources are within the resource. If a given resource is associated with another resource, the second resource should be navigable from the first.

This is the basic concept behind links in hypermedia.

Trac, however takes this a step further with a trivial feature that is much more valuable than it may seem at first glance. Trac integrates very nicely with subversion. In a large percentage of cases, one ore more changesets are associated with a ticket. It is also fairly trivial to link to a changeset from within a ticket. Here is the interesting feature. The title attribute of the anchor that links to the changeset is the message associated with the changeset. Here is what I mean.



This subtle feature has saved me the time required to actually follow a link to view the changeset resource on numerous occasions. Most of the time, we're only interested in the message. Trac has realized this usability issue and addressed it.

Testing for length in dynamic languages

Testing for the length of some object in dynamic languages, or, statically-typed languages for that matter, is a common task in every single application. Especially in statically-typed languages. In cases where some task needs to be carried out for every object in a set, we simply iterate through that set. We perform the task needed on every object, possibly invoking some operation with the object as a parameter.

In Python, and other similar languages, it is fairly trivial to iterate over a set and is done quite often. For example.

#Python iterate.

list_obj=[1,[2,3],[4,[5,6],7],8,9,10]
for i in list_obj:
print i

Here, we have a simple list of integers and sub-lists. We iterate through this list and print its' elements.

Now, say we want to generate some HTML output for a web page. We need to display a list of all list elements on this HTML page. I'll implement a simple function to generate list elements.

#Python iterate.

list_obj=[1,[2,3],[4,[5,6],7],8,9,10]
html_list='<ul>%s</ul>'
temp_content=''

def generate_list_element(obj):
return '<li>%s</li>'%(str(obj))

for i in list_obj:
temp_content+=generate_list_element(i)

print html_list%(temp_content)

Here, we set up a temporary variable to hold all li tags generated by the generate_list_element() function. We also have the html_list template variable defined which, after the iteration has completed, will give us the final ul tag.

One problem with this approach is that if list_obj is empty, we have an empty ul element. I most cases, this isn't the desired output. The better solution would be to initialize the template based on what is available to the template. For example.

#Python iterate.

class MyHTML:
  list_obj=[1,[2,3],[4,[5,6],7],8,9,10]
  html_list=False
  template='<ul>%s</ul>'
  error='Error.  List is Empty.'

  @classmethod
  def generate_list_element(cls, obj):
      return '<li>%s</li>'%(str(obj))

  @classmethod
  def _set(cls, content):
      cls.html_list=cls.template%(content)

  @classmethod
  def set(cls, content):
      content and cls._set(content)

  @classmethod
  def get(cls):
      return cls.html_list or cls.error

temp_content=''
for i in MyHTML.list_obj:
  temp_content+=MyHTML.generate_list_element(i)
MyHTML.set(temp_content)
print MyHTML.get()

This code will produce the exact same output as the previous example. The main difference being that here, we can handle empty sets. I've better organized the code by introducing a class; MyHTML. This class defines an error message in the case of an empty list. The message isn't the important idea here. What is important is that MyHTML.error could be an empty string or invoke some other behavior to produce the desired output in the case of an empty set.

If you take a look at the main program, ignoring the class for now, you'll notice that we do not test for length. We simply iterate through our set and build our content. If the set is empty, the loop is not entered and that is our indicator of a false value. In the context of an alternate output based on an object having any elements or not, boolean logic should be used.

Just because we use the and and or operators doesn't make our code any less readable. It is encapsulated by our MyHTML class. In my opinion, the length of an object really only needs to be calculated for logic that "does this object have a length?" does not support.

New tools in the TG controllers module

After taking a look at some changes made to the TurboGears controller module in trunk, I must say I'm impressed with the improvements over the current 1.0.x branch.

The first change I noticed was that all classes are now derived from the WSGIController class from the Pylons Python web framework. Also new, and the most interesting in my view, are the hook processing mechanism implemented by the DecoratedController class. What this means is that developers writing TurboGears applications can define hooks that are processed in any of these controller states:

• before validation
• before the controller method is called
• before the rendering takes place
• after the rendering has happened

If nothing else, I think this will add great value in monitoring the state transitions in larger TurboGears applications. Some requests can be quite large; especially during development time and it is handy to know where this requests are failing. You can now easily log attribute values of your controller instance before validation takes place. This could give some insight as to why the validation is failing with valid values. These hook processors also allow for pre and post processing for every state transition within the controller life-cycle.

It looks like using a controller is not all that different from the current TurboGears. Simply extend the TGController class and expose your methods as needed.

An interface/generalization design pattern

Last month, I wrote about an interface/generalization design pattern. I thought I'd post a diagram which I feel better illustrates the pattern.



Again, to some the pattern up:

• The base interface specifies the contract that all classes who realize this interface must implement.
• The base class which realizes the base interface. This instances of this class will ensure that the expected default behavior of some collaboration can always be carried out in the case that a more specialized sub-class cannot be instantiated.
• You may create a sub-class of the base class which realizes the base interface. This class has the ability to redefine certain methods of the base class while using the default implementation of others.
• You may create a regular class that realizes the base interface. It can be thought of as a child of the base class because it will behave the same as any other children of the base class. The main difference being that this class must implement the entire interface. There are no defaults here.
  

Inside zope.event

The zope.event Python package is extremely straightforward. Which is why I like it. It provides a publish-subscribe mechanism that is sufficient for most applications. The entire zope.event package consists of a list for storing subscribers and a single notify function to publish events. Here is what the module looks like.

#zope.event

subscribers = []

def notify(event):
   for subscriber in subscribers:
       subscriber(event)

Very handy functionality and very easy to understand. This could event be implemented within an application should the zope.event module not be available because of its' simplicity.