The data() jQuery function let's us store arbitrary data in an element. It doesn't actually store it in the DOM, but it's an elegant means to associate arbitrary application data with an element. It removes unnecessary code and complexity — let jQuery handle it. I often find myself having to create structures in my code that look similar to what I have in the DOM, essentially duplicating it. There's often no need for this, because if it's data that's specifically related to an element, that's what data() is for.
Showing posts with label data. Show all posts
Showing posts with label data. Show all posts
Thursday, March 27, 2014
Storing Backbone Views In Element Data
Friday, February 1, 2013
Geo Everywhere
Precise geographic data is invaluable when it comes to software that has to know either where the hardware its running on is, or where something else is. And it seems that this particular type of data is found in software everywhere. And if it isn't, you could probably find a valid use case after browsing the code for ten minutes. Phenomenons hit software development culture, some stay while some disappear. The ones that stay, generally never stop evolving in an effort to standardize their meaning. Everyone must be on the same page, otherwise, you might as well create your own ad-hoc formulation. Geo is different. The way we store, manipulate, and use the data is constantly in flux. But the data itself — longitude and latitude — unchanging, and unambiguous.
Monday, March 5, 2012
Data Recognition
We all recognize things, objects familiar to us. Familiarity is a comforting phenomenon — it tells us we've been here before and we can make certain assumptions. It's when we encounter the unfamiliar — the unrecognizable — that we begin to have second thoughts. How we get stuff done, our jobs, our daily routines, it's all been done before. When we learn something new — either at our job or something we've set out to do based solely on curiosity, we're building up our repertoire of certitudes.
Now imagine that we didn't have this ability to recognize things within context. Imagine every scenario we encounter was brand new. We'd have to learn on the fly. We'd have to take in external stimulus, compare it with only our must unchanging fundamental knowledge and build up a mental representation. No ability to react in a split second. No ability to store new knowledge so that next time, we have something to reference. What would that be like?
It's kind of like the fatal flaw in how we're approaching data recognition in software development. We merely have the illusion, as users, that the computer knows something we don't. That it's really able to determine facts for us that we're simply incapable of doing and are thus powerful aids in intelligence. But software doesn't recognize data. Not the way we do. Every time we give software input and tell in to act on that input, it goes through the same motions as every other time. Even if the input is the same. Now, this isn't true of all software — artificial intelligence is geared toward learning. In an effort to simulate real human behaviour, some software can actually recognize certain data, attach meaning to it, and store new information that is applied in the next round.
Should we be taking this type of software more seriously? That is, should we not view learning software as more of a fundamental necessity for all types of applications? I think the answer is yes, because until we can effectively build software that can recognize data, the current levels of complexity will soon become recalcitrant.
The Best Help
Imagine using software that could display meaningful error messages. I know, probably not going to happen in our lifetime, but one can only dream. What is a meaningful error message? Well, we're typically used to seeing things like "oops, you forgot the last name field". This is helpful enough, and probably won't go away until forms vanish from the face of the earth. But what about errors that tell the user something they have no control over — these usually take the form of "an unknown error occurred" or "the system is a experiencing higher that normal traffic levels...". You've got to wonder, why even bother. Why not just say, "this software has bugs, so not your fault". This is another category with which the user has no control over whether or not the error message gets generated. So until software just works, we're stuck with these annoyances.
But let's revisit the problem of form entry errors for a moment. If the user is providing the data, it's much easier to tell them they've done wrong. They can relate to the message because it's clearly of their own doing. Most software frameworks have form validation sub-components that supply a primitive form of data recognition. That is, the field validators can recognize if the supplied data is in fact an email address or if it is in fact a phone number. These little helpers make the lives of developers writing the software easier and the lives of users supplying input. The developers know that they can read phone numbers and email addresses from the form. The users know exactly what they've done wrong in the case of submission failure because the get back meaningful messages.
However, we could take this principle of recognizing what the user has supplied us with to make their lives even easier. Instead of returning an error message if the email address cannot be recognized. or at least before returning the message, maybe the framework makes an attempt to fix the user's typo. For example, perhaps the @ character is missing and the first part of the string begins with a human name such as davidcompany.com or david company.com. The data recognition framework could save the user some typing if all it needs to do is come back with a suggestion — did you mean david@company.com. Obviously, that's exactly what I meant — good job software.
So is this the best we can do with data recognition in terms of understanding what the user is trying to say? That is, is the confirmation really necessary? Do we really need to go back and check if that is in fact what the user meant? The problem with not checking is that a false positive could result in a much worse situation than if they had simply been given an opportunity to correct things. And I don't necessarily think that implies that the software is weak or that in stands in the way of the traditional method of input correction. What it does do is gives the user a sense that the software does know what they're trying to do. It leaves a feeling of being coached along an activity, not taunted in a guessing game.
I think its a huge win for not only users, and not only developers that create these systems, but more of a collective win for a society that relies on software interactions. Any good software system must produce fantastic documentation, enabling their users to navigate through the system themselves. The documentation is simply learning material, some of it will no doubt be memorized, other parts referred to when necessary. But is thorough text the best learning solution for using a piece of software? Or is it better to give the users a brief overview — just enough to get them started. And this is where the data recognition end of thing comes into being for providing users with the ideal help system. The ultimate goal being to recognize what the user is trying to do and to offer advice of how to better achieve their goals.
Effort Required
It's easy to dream about software that recognizes data, correspondingly, making assumptions about what we want to do. It's a completely different story when it comes to putting these ideas into code. There simply isn't enough demand for this type of software-assisted user experience. Well, that might not be entirely true — Apple has made it abundantly clear that we cannot expect anything less than cutting edge user experiences from any of their products. Even still, the old routine of having to understand, often in some level of detail, what the computer is expecting from us, is somewhat ingrained in our experience.
We can't use the software because it requires training — reading manuals and perhaps some guided practical exercises. Everyone knows experience is the best teacher — why not streamline that wisdom even further if we have the ability to do so? Build in some of the mistakes that you'll expect your users to make.
This is part of the challenge — at the intersection of time to market and innovative data recognition solutions. I have nothing but the utmost confidence that we're able to apply the most sophisticated recognition techniques to almost any software and that we'd see absolutely incredible results. When push comes to shove, it's hard to justify such efforts. Which is too bad in my opinion because an investment into data recognition is an investment into customer behaviour.
Data recognition isn't just about users, it isn't just about saving them from typing a few extra characters. More broadly, it's about flexibility in how intent is expressed between actors that exchange information. Standards are good but only if the two communicating systems speak the same standard. If we could simply think of more intelligent ways to lower barrier between what the user envisions doing in their head and what that means supplying the keyboard, we'd be well on our way to quality data recognition.
Now imagine that we didn't have this ability to recognize things within context. Imagine every scenario we encounter was brand new. We'd have to learn on the fly. We'd have to take in external stimulus, compare it with only our must unchanging fundamental knowledge and build up a mental representation. No ability to react in a split second. No ability to store new knowledge so that next time, we have something to reference. What would that be like?
It's kind of like the fatal flaw in how we're approaching data recognition in software development. We merely have the illusion, as users, that the computer knows something we don't. That it's really able to determine facts for us that we're simply incapable of doing and are thus powerful aids in intelligence. But software doesn't recognize data. Not the way we do. Every time we give software input and tell in to act on that input, it goes through the same motions as every other time. Even if the input is the same. Now, this isn't true of all software — artificial intelligence is geared toward learning. In an effort to simulate real human behaviour, some software can actually recognize certain data, attach meaning to it, and store new information that is applied in the next round.
Should we be taking this type of software more seriously? That is, should we not view learning software as more of a fundamental necessity for all types of applications? I think the answer is yes, because until we can effectively build software that can recognize data, the current levels of complexity will soon become recalcitrant.
The Best Help
Imagine using software that could display meaningful error messages. I know, probably not going to happen in our lifetime, but one can only dream. What is a meaningful error message? Well, we're typically used to seeing things like "oops, you forgot the last name field". This is helpful enough, and probably won't go away until forms vanish from the face of the earth. But what about errors that tell the user something they have no control over — these usually take the form of "an unknown error occurred" or "the system is a experiencing higher that normal traffic levels...". You've got to wonder, why even bother. Why not just say, "this software has bugs, so not your fault". This is another category with which the user has no control over whether or not the error message gets generated. So until software just works, we're stuck with these annoyances.
But let's revisit the problem of form entry errors for a moment. If the user is providing the data, it's much easier to tell them they've done wrong. They can relate to the message because it's clearly of their own doing. Most software frameworks have form validation sub-components that supply a primitive form of data recognition. That is, the field validators can recognize if the supplied data is in fact an email address or if it is in fact a phone number. These little helpers make the lives of developers writing the software easier and the lives of users supplying input. The developers know that they can read phone numbers and email addresses from the form. The users know exactly what they've done wrong in the case of submission failure because the get back meaningful messages.
However, we could take this principle of recognizing what the user has supplied us with to make their lives even easier. Instead of returning an error message if the email address cannot be recognized. or at least before returning the message, maybe the framework makes an attempt to fix the user's typo. For example, perhaps the @ character is missing and the first part of the string begins with a human name such as davidcompany.com or david company.com. The data recognition framework could save the user some typing if all it needs to do is come back with a suggestion — did you mean david@company.com. Obviously, that's exactly what I meant — good job software.
So is this the best we can do with data recognition in terms of understanding what the user is trying to say? That is, is the confirmation really necessary? Do we really need to go back and check if that is in fact what the user meant? The problem with not checking is that a false positive could result in a much worse situation than if they had simply been given an opportunity to correct things. And I don't necessarily think that implies that the software is weak or that in stands in the way of the traditional method of input correction. What it does do is gives the user a sense that the software does know what they're trying to do. It leaves a feeling of being coached along an activity, not taunted in a guessing game.
I think its a huge win for not only users, and not only developers that create these systems, but more of a collective win for a society that relies on software interactions. Any good software system must produce fantastic documentation, enabling their users to navigate through the system themselves. The documentation is simply learning material, some of it will no doubt be memorized, other parts referred to when necessary. But is thorough text the best learning solution for using a piece of software? Or is it better to give the users a brief overview — just enough to get them started. And this is where the data recognition end of thing comes into being for providing users with the ideal help system. The ultimate goal being to recognize what the user is trying to do and to offer advice of how to better achieve their goals.
Effort Required
It's easy to dream about software that recognizes data, correspondingly, making assumptions about what we want to do. It's a completely different story when it comes to putting these ideas into code. There simply isn't enough demand for this type of software-assisted user experience. Well, that might not be entirely true — Apple has made it abundantly clear that we cannot expect anything less than cutting edge user experiences from any of their products. Even still, the old routine of having to understand, often in some level of detail, what the computer is expecting from us, is somewhat ingrained in our experience.
We can't use the software because it requires training — reading manuals and perhaps some guided practical exercises. Everyone knows experience is the best teacher — why not streamline that wisdom even further if we have the ability to do so? Build in some of the mistakes that you'll expect your users to make.
This is part of the challenge — at the intersection of time to market and innovative data recognition solutions. I have nothing but the utmost confidence that we're able to apply the most sophisticated recognition techniques to almost any software and that we'd see absolutely incredible results. When push comes to shove, it's hard to justify such efforts. Which is too bad in my opinion because an investment into data recognition is an investment into customer behaviour.
Data recognition isn't just about users, it isn't just about saving them from typing a few extra characters. More broadly, it's about flexibility in how intent is expressed between actors that exchange information. Standards are good but only if the two communicating systems speak the same standard. If we could simply think of more intelligent ways to lower barrier between what the user envisions doing in their head and what that means supplying the keyboard, we'd be well on our way to quality data recognition.
Friday, September 23, 2011
Making Data Public
What sort of information should be made public by organizations? The question, phrased like this, means what type of information is displayed on the company's web site? Information for the web has been prepackaged, so to speak, for consumption by human readers. Perhaps the more interesting question is this — what I'm interested in anyway — what type of raw data should be exposed through an API? Who would use such data and for what purpose?
With pre-formatted information, it's difficult to make sense of it in large volumes. Google does this for their search index by crawling monumental assemblages of HTML pages and other web resources. But for those of us who don't have enormous computing centers and the software capable of deciphering these data sets, we need something a little more primitive. Something that'll let our software draw it's own conclusions about public data.
Who wants to expose their data?
The internet changed things for all organizations — large and small. Having a place on the web is no longer a nice-to-have — you simply cannot compete without one. For one thing, consumers have come to expect this — a place where they can gather information on products and or services. Also, the web is a social animal now. Companies need a social presence on the web if they're to engage with their customers.
This is just web technology — read-only web technology. Information on what the business does, how they're a cut above the competition, and so forth. Just the ignition for the social correspondence that follows. I won't bother going into the whole social end of it because it's fairly obvious — it's important.
What isn't so obvious, however — how the specific information organizations publish on their websites is chosen. Obviously, publishing damaging information will come back to haunt the organization commodiously. Hence the reluctancy to publish anything at all. So why take the chance in making more information available to consumers when it's much safer to make less data available?
A competitive edge
Making data public — useful data — will absolutely give your organization a competitive advantage. That of course assumes that you have both interesting data to make available and the know-how to design an API that other software makers can accommodate. If you've got those two things — you're ahead of the game because you're enabling third party software to be developed on your behalf. This software directly impacts how existing and potential customers connect with your organization.
Take Apple's App Store for instance. There is obviously no shortage of applications available for users to install and use. In fact, it is amazingly imbalanced — the number of applications available in the App Store relative to other device maker's software markets.
Developers who make these applications aren't doing it because of the interesting data Apple is exposing through an API — they're doing it because of the iPhone's popularity. There is a much larger potential user base. So how does this relate to making your data available to the general public through an API? Because this is the type of following you want from developers. How useful would the iPhone be without plethora third-party applications? It would still be great device, but it would be missing a lot of things that users want.
Another aspect to the competitive advantage of having third-party software built for you — they'll use your data in ways you haven't thought of. Could Apple really have thought of all the applications available for their devices? Probably, but not in a time frame of just a few years.
Fear of over-exposure
If our companies weren't scared of exposing data items they shouldn't, they probably would have done it years ago, right? I don't think that's necessarily true because we're only still discovering the neat things we can build with other organization's data.
How do you protect yourself from exposing too much information, perhaps leading to a competitive edge for your competition? I'll tell you what won't protect you from it — not making an API for interesting information. The reality is, it's easy to spot weaknesses — every company in the world has at least one. If the competition is smart enough to exploit these weaknesses, they won't need an API to do it.
So listen to your customers — what kind of applications would they find useful? Or, more generally, what information would they find beneficial? Expose that information through an API. Developers will build cool things if the data is valuable. Who knows what new ideas these applications will in turn bring forth.
With pre-formatted information, it's difficult to make sense of it in large volumes. Google does this for their search index by crawling monumental assemblages of HTML pages and other web resources. But for those of us who don't have enormous computing centers and the software capable of deciphering these data sets, we need something a little more primitive. Something that'll let our software draw it's own conclusions about public data.
Who wants to expose their data?
The internet changed things for all organizations — large and small. Having a place on the web is no longer a nice-to-have — you simply cannot compete without one. For one thing, consumers have come to expect this — a place where they can gather information on products and or services. Also, the web is a social animal now. Companies need a social presence on the web if they're to engage with their customers.
This is just web technology — read-only web technology. Information on what the business does, how they're a cut above the competition, and so forth. Just the ignition for the social correspondence that follows. I won't bother going into the whole social end of it because it's fairly obvious — it's important.
What isn't so obvious, however — how the specific information organizations publish on their websites is chosen. Obviously, publishing damaging information will come back to haunt the organization commodiously. Hence the reluctancy to publish anything at all. So why take the chance in making more information available to consumers when it's much safer to make less data available?
A competitive edge
Making data public — useful data — will absolutely give your organization a competitive advantage. That of course assumes that you have both interesting data to make available and the know-how to design an API that other software makers can accommodate. If you've got those two things — you're ahead of the game because you're enabling third party software to be developed on your behalf. This software directly impacts how existing and potential customers connect with your organization.
Take Apple's App Store for instance. There is obviously no shortage of applications available for users to install and use. In fact, it is amazingly imbalanced — the number of applications available in the App Store relative to other device maker's software markets.
Developers who make these applications aren't doing it because of the interesting data Apple is exposing through an API — they're doing it because of the iPhone's popularity. There is a much larger potential user base. So how does this relate to making your data available to the general public through an API? Because this is the type of following you want from developers. How useful would the iPhone be without plethora third-party applications? It would still be great device, but it would be missing a lot of things that users want.
Another aspect to the competitive advantage of having third-party software built for you — they'll use your data in ways you haven't thought of. Could Apple really have thought of all the applications available for their devices? Probably, but not in a time frame of just a few years.
Fear of over-exposure
If our companies weren't scared of exposing data items they shouldn't, they probably would have done it years ago, right? I don't think that's necessarily true because we're only still discovering the neat things we can build with other organization's data.
How do you protect yourself from exposing too much information, perhaps leading to a competitive edge for your competition? I'll tell you what won't protect you from it — not making an API for interesting information. The reality is, it's easy to spot weaknesses — every company in the world has at least one. If the competition is smart enough to exploit these weaknesses, they won't need an API to do it.
So listen to your customers — what kind of applications would they find useful? Or, more generally, what information would they find beneficial? Expose that information through an API. Developers will build cool things if the data is valuable. Who knows what new ideas these applications will in turn bring forth.
Friday, October 8, 2010
Cloud Data
Cloud data is probably a better term for what is conventionally referred to as cloud computing. Rather than treating the cloud as a computing resource, we're using it as massive data store. This is only logical given that we're running out of places to put our information. Under typical circumstances, cloud data will suffice. A place where we can instantaneously fetch the information we need. Despite monumental storage capabilities, applications in the cloud can only process so much data without more CPU cycles and memory. How, exactly, is cloud data complimented by cloud computing resources?
First, I want to walk through how I put my data in the cloud. I have some data, lots of data probably, that I want accessible from anywhere in the world. I find some cloud computing service that allows me to do this. I give them my data, I later retrieve the bits and pieces I need. Everything is good. This is a little over-simplified - we put structured information into the cloud. We give social networks profile information. We give photo sharing platforms categorized images. We give cloud service providers virtual machines that are run on our behalf. All these things amount to data we put into the cloud and later retrieve it.
Virtual machines are unique in that they represent both data and computing resources. The ultimate purpose behind a virtual machine is to run software. But they also occupy a significant amount of disk space, as does any other cloud data. If virtual machines fall into cloud computing category, how is running it in the cloud beneficial? In the end, a virtual machine can only run on one physical machine at a time.
The problem is figuring out how to better utilize computing resources in the cloud - giving applications more CPU cycles and more memory. The distributed nature of the cloud is crucial for concurrency. Each node in the cloud adds another level of concurrency for your application to capitalize on. This is how applications running in the cloud can utilize computing resources. Executing two or more things and the same time will increase your throughput. The nice thing about concurrency in the cloud is that it implies more CPU cycles and more memory because of the distributed physical nodes. Applications equipped to run on multiple nodes are cloud-worthy.
One way an application can adapt to multiple cloud nodes is by implementing distributed algorithms. A distributed algorithm is an algorithm that takes into account the notion of time and space. Time and space on a single node is taken care of by the operating system. Your application might use some form of a map/reduce algorithm. It might implement a spanning tree, or a routing table of some kind. These are all examples of how an application utilize cloud computing resources.
Cloud data by itself isn't enough meet the computing resource requirements of applications running in the cloud. The cloud is proficient when it comes to storing vast amounts of information and making it globally available. We have yet to utilize the available cloud computing resources to their full potential. The cloud, as a whole, is nothing more than a networked group of hardware nodes. We've put the hard disks to good use. Its up to the developers to design applications that appropriate the CPU and memory in a meaningful way.
First, I want to walk through how I put my data in the cloud. I have some data, lots of data probably, that I want accessible from anywhere in the world. I find some cloud computing service that allows me to do this. I give them my data, I later retrieve the bits and pieces I need. Everything is good. This is a little over-simplified - we put structured information into the cloud. We give social networks profile information. We give photo sharing platforms categorized images. We give cloud service providers virtual machines that are run on our behalf. All these things amount to data we put into the cloud and later retrieve it.
Virtual machines are unique in that they represent both data and computing resources. The ultimate purpose behind a virtual machine is to run software. But they also occupy a significant amount of disk space, as does any other cloud data. If virtual machines fall into cloud computing category, how is running it in the cloud beneficial? In the end, a virtual machine can only run on one physical machine at a time.
The problem is figuring out how to better utilize computing resources in the cloud - giving applications more CPU cycles and more memory. The distributed nature of the cloud is crucial for concurrency. Each node in the cloud adds another level of concurrency for your application to capitalize on. This is how applications running in the cloud can utilize computing resources. Executing two or more things and the same time will increase your throughput. The nice thing about concurrency in the cloud is that it implies more CPU cycles and more memory because of the distributed physical nodes. Applications equipped to run on multiple nodes are cloud-worthy.
One way an application can adapt to multiple cloud nodes is by implementing distributed algorithms. A distributed algorithm is an algorithm that takes into account the notion of time and space. Time and space on a single node is taken care of by the operating system. Your application might use some form of a map/reduce algorithm. It might implement a spanning tree, or a routing table of some kind. These are all examples of how an application utilize cloud computing resources.
Cloud data by itself isn't enough meet the computing resource requirements of applications running in the cloud. The cloud is proficient when it comes to storing vast amounts of information and making it globally available. We have yet to utilize the available cloud computing resources to their full potential. The cloud, as a whole, is nothing more than a networked group of hardware nodes. We've put the hard disks to good use. Its up to the developers to design applications that appropriate the CPU and memory in a meaningful way.
Friday, April 17, 2009
The Django paginator
Most, if not all, modern web applications need pagination in one form or another. Pagination is the act of transforming a large data set into pages of a more manageable size. Google gives us a perfect example case of pagination. Google constantly deals with enormous data sets. Users who perform searches using google would promptly switch to a different search engine if there were no pagination provided. On the other side of the coin, pagination also provides more manageable data sets for the server code to deal with. Instead of the client saying "give me this entire large data set" the client says "give me page one of this large data set, page size being ten". From the developer perspective, pagination isn't always the most enjoyable task. They are error prone if not implemented correctly and can pose challenges when different query constraints come into play. An additional challenge with pagination is the fact that different web application frameworks use slightly different approaches to their pagination implementation. Another approach to implementing pagination could be to implement the functionality directly into the ORM. This would obviously only work with database query results but this is probably the most common use for pagination. However, if an ORM like SQLAlchemy for instance, were to implement pagination functionality, it could be used by developers inside a web application framework while still being functional outside the framework. The Django Python web application framework offers a good pagination implementation. It is not restricted to database query results and is easy for developers to understand and use. There is also much room in the implementation for extended functionality if so desired. To two classes used to implement the pagination functionality in Django are Paginator and Page as illustrated below.
The main class used by the Django web application framework is the Paginator class. The Paginator class deals directly with the data set in question. The constructor accepts an object_list as a required parameter. The constructor also accepts a per_page parameter that specifies the page size. This parameter is also required. Once the Paginator class has been instantiated with a data set, it can be used to generate Page instances. This is done by invoking the Paginator.page() method, specifying the desired page number. The Paginator class also defines several managed properties that can be used to query the state of the Paginator instance. Managed properties in Python are simply methods that may be invoked as attributes. The count attribute will return the number of objects in the data set. The num_pages attribute will return the number of pages that the data set contains, based on the specified page size. Finally, the page_range attribute will return a list containing each page number.
There is room for extended functionality in the Paginator class. Mainly, there could be some operators overloaded to make Paginator instances behave more like new-style Python instances. The Paginator class could define a __len__() method. This way, developers could invoke the builtin len() function on Paginator instances to retrieve the number of pages. An __iter__() method could also be defined for the Paginator class. This would allow instances of this class to be used in iterations. Each iteration would yield a new Page instance. Finally, a __getitem__() method would allow Paginator instances to behave like Python lists. The desired page number could be specified as an index.
The Page class complements the Paginator class in that it represents a specific page within the data set managed by the Paginator instance. The has_next(), has_previous() and has_other_pages() methods of Page instances are useful in determining if the page has any neighbouring pages. The start_index() method will return the index in the original data set owned by the Paginator instance that created the Page instance. The end_index() will return the end index in the original data set.
Like the Paginator class, there is also room for extended functionality here to make Page instances behave more like new-style Python instances. The Page class could define a __len__() method that could return the page size. The Page class could also define an __iter__() method that could enable Page instances to be used in iterations. Finally, the __getitem__() method, if it were defined, could return the specified object from the original object list.
The main class used by the Django web application framework is the Paginator class. The Paginator class deals directly with the data set in question. The constructor accepts an object_list as a required parameter. The constructor also accepts a per_page parameter that specifies the page size. This parameter is also required. Once the Paginator class has been instantiated with a data set, it can be used to generate Page instances. This is done by invoking the Paginator.page() method, specifying the desired page number. The Paginator class also defines several managed properties that can be used to query the state of the Paginator instance. Managed properties in Python are simply methods that may be invoked as attributes. The count attribute will return the number of objects in the data set. The num_pages attribute will return the number of pages that the data set contains, based on the specified page size. Finally, the page_range attribute will return a list containing each page number.
There is room for extended functionality in the Paginator class. Mainly, there could be some operators overloaded to make Paginator instances behave more like new-style Python instances. The Paginator class could define a __len__() method. This way, developers could invoke the builtin len() function on Paginator instances to retrieve the number of pages. An __iter__() method could also be defined for the Paginator class. This would allow instances of this class to be used in iterations. Each iteration would yield a new Page instance. Finally, a __getitem__() method would allow Paginator instances to behave like Python lists. The desired page number could be specified as an index.
The Page class complements the Paginator class in that it represents a specific page within the data set managed by the Paginator instance. The has_next(), has_previous() and has_other_pages() methods of Page instances are useful in determining if the page has any neighbouring pages. The start_index() method will return the index in the original data set owned by the Paginator instance that created the Page instance. The end_index() will return the end index in the original data set.
Like the Paginator class, there is also room for extended functionality here to make Page instances behave more like new-style Python instances. The Page class could define a __len__() method that could return the page size. The Page class could also define an __iter__() method that could enable Page instances to be used in iterations. Finally, the __getitem__() method, if it were defined, could return the specified object from the original object list.
Wednesday, March 18, 2009
3D data and 3D UML
An interesting and experimental idea: The ability to design three-dimensional models using UML. This idea is shard among some visionary coders who have already demonstrated the ability to model UML in a 3D model space. Diagram elements can be layered, placed, and rotated in a three-dimensional manor. Glasshouse is another cutting edge GUI for visualizing relational data sets. Users can use SQL or spreadsheet data as input, and Glasshouse will present the user as an avatar within a 3D environment. The avatar then acts as a the user within the data environment, allowing the user to manipulate the data in an interactive way never seen before. The remaining question is, what is wrong with the current standard 2D visualization of data and UML models today?
The answer is that there is nothing inherently wrong with viewing data in two dimensions. The same holds true with UML models. Before the graphical user interface, common on most desktops today, there was the command line. There is also nothing inherently wrong with the command line. However, the GUI was invented for a reason, so that human users can quickly comprehend what is displayed in front of them. Trying to grasp a relational data set that is displayed in the console is possible, although it would most likely take a seasoned professional two weeks to understand it fully. If that same data set is presented graphically, many more features become available such as moving windows around etc. With a GUI it might take that same professional a day or less to fully understand the data. Now, imagine trying to display, edit, and understand a modest UML diagram in the console. For humans understanding data, the GUI was the a big fist step and understanding UML models followed shortly. The next step is another dimension.
The Glasshouse project is good example of how this first 3D data manipulation interface might be taken. Insights about data sets will most likely be made possible that never were before. Users much more freedom in the perspective in which they view the data. Will the UML be able to follow this direction? Some tools have already started by making individual diagrams rotatable and stackable within the modeling space. This is where the 3D functionality in the UML ends. There is currently no tool that offers 3D UML elements such as classes, objects, or interactions. Could avatars be used as actors in use cases? For instance, when simulating a use case realization, the avatar (the actor) could actually move about in the collaboration among the other 3D UML elements. An instance of some class could expand and contract in three dimensions according to how many resources it is using.
These are some incredibly complex design challenges to implement. Building a two dimensional UML modeling tool is by no means trivial. Building three dimensional interfaces are also not trivial. Combining the two could take decades just to get a functional demo working. Is something like this worth the effort? Would this new UML 3D modeling interface produce better software, faster? In the end, this amounts to a tough decision to make because of the risk involved. But that hasn't stopped other ingenious software projects from being built in the past.
The answer is that there is nothing inherently wrong with viewing data in two dimensions. The same holds true with UML models. Before the graphical user interface, common on most desktops today, there was the command line. There is also nothing inherently wrong with the command line. However, the GUI was invented for a reason, so that human users can quickly comprehend what is displayed in front of them. Trying to grasp a relational data set that is displayed in the console is possible, although it would most likely take a seasoned professional two weeks to understand it fully. If that same data set is presented graphically, many more features become available such as moving windows around etc. With a GUI it might take that same professional a day or less to fully understand the data. Now, imagine trying to display, edit, and understand a modest UML diagram in the console. For humans understanding data, the GUI was the a big fist step and understanding UML models followed shortly. The next step is another dimension.
The Glasshouse project is good example of how this first 3D data manipulation interface might be taken. Insights about data sets will most likely be made possible that never were before. Users much more freedom in the perspective in which they view the data. Will the UML be able to follow this direction? Some tools have already started by making individual diagrams rotatable and stackable within the modeling space. This is where the 3D functionality in the UML ends. There is currently no tool that offers 3D UML elements such as classes, objects, or interactions. Could avatars be used as actors in use cases? For instance, when simulating a use case realization, the avatar (the actor) could actually move about in the collaboration among the other 3D UML elements. An instance of some class could expand and contract in three dimensions according to how many resources it is using.
These are some incredibly complex design challenges to implement. Building a two dimensional UML modeling tool is by no means trivial. Building three dimensional interfaces are also not trivial. Combining the two could take decades just to get a functional demo working. Is something like this worth the effort? Would this new UML 3D modeling interface produce better software, faster? In the end, this amounts to a tough decision to make because of the risk involved. But that hasn't stopped other ingenious software projects from being built in the past.
Monday, February 23, 2009
An argument against XML
My argument against using XML as a data format in certain situations is that it is too verbose. In other situations, however, the verbosity provided by XML is needed. Such as for human consumption. This is why XML exists, it is easy to use and read by both humans and computers.
The verbosity problem with XML stems from the use of tags. Every entity represented in XML needs needs to be enclosed in a tag. The opening tag indicating that a new entity definition has started and the ending tag indicating the end of that definition. For example, consider the following XML.
This is a trivial example of a person entity with a single name attribute. Notice the duplication of the text "person" and "name" in the metadata. With XML this is required. However, tags may also have attributes. Our person definition could be expressed as follows.
Here there is no metadata duplication. But I think the second example negates the readability philosophy behind XML. What exactly is the difference between attributes and child entities in XML? Semantically, there is none. A child entity is still an attribute of the parent entity.
With JSON, there is no duplication of metadata or any confusion of how an entity is defined. This is because the JSON format is focused on lightweight data, not readability. For instance, here is our person in JSON.
Now, if a person were reading this, the chances of them getting the meaning right are greatly reduced when compared to the XML equivalent. However, it is much less verbose in most cases. And verbosity counts when data is being transferred over a network. Another plus, the XML is not lost. JSON can easily be converted to XML and back. So if JSON-formatted data must be edited by humans as XML, this is not difficult to achieve.
Here is a simple Python demonstration of reducing the size of XML data with JSON.
Finally, since XML is based on tags, there is no opportunity for sets of primitive types. For example, some client says to the server "give me a list of names and nothing else". The client will likely name something along the lines of the following.
Here is the JSON alternative.
The verbosity problem with XML stems from the use of tags. Every entity represented in XML needs needs to be enclosed in a tag. The opening tag indicating that a new entity definition has started and the ending tag indicating the end of that definition. For example, consider the following XML.
<person>
<name>adam</name>
</person>
<person name="adam"/>
With JSON, there is no duplication of metadata or any confusion of how an entity is defined. This is because the JSON format is focused on lightweight data, not readability. For instance, here is our person in JSON.
{person:{name:"adam"}}
Here is a simple Python demonstration of reducing the size of XML data with JSON.
#Example; XML string and JSON string
xml_string="""
<entry><title>mytitle</title><body>mybody</body></entry>
"""
json_string="""
{entry:{title:"mytitle",body:"mybody"}}
"""
if __name__=="__main__":
print 'XML Length:',len(xml_string)
print 'JSON Length:',len(json_string)
pcent=float(len(json_string))/len(xml_string)*100
print 'XML size as JSON:',pcent,'%'
<list>
<item name="name1"/>
<item name="name2"/>
<item name="name3"/>
</list>
["name1", "name2", "name3"]
Wednesday, February 18, 2009
Python memory Usage
Here is an example in Python of how to retrieve the system memory usage. This example was adapted from an entry on stackoverflow.
Here we have a simple class called MemUsage. The constructor initializes the attributes of the class needed to compute the memory usage. The init_data() method is what MemUsage invokes in order to retrieve the required system data. This is done by using the subprocess module to execute the free command. The resulting data is then mapped to the corresponding attributes. We compute the memory usage as a percentage by subtracting the buffers and cache from the used memory and dividing the result by the total memory.
#Example; Get the system memory usage.
import subprocess
class MemUsage(object):
def __init__(self):
self.total=0
self.used=0
self.free=0
self.shared=0
self.buffers=0
self.cached=0
self.init_data()
def init_data(self):
command="free"
process=subprocess.Popen(command,\
shell=True,\
stdout=subprocess.PIPE)
stdout_list=process.communicate()[0].split('\n')
for line in stdout_list:
data=line.split()
try:
print data
if data[0]=="Mem:":
self.total=float(data[1])
self.used=float(data[2])
self.free=float(data[3])
self.shared=float(data[4])
self.buffers=float(data[5])
self.cached=float(data[6])
except IndexError:
continue
def calculate(self):
return ((self.used-self.buffers-self.cached)/self.total)*100
def __repr__(self):
return str(self.calculate())
if __name__=="__main__":
print MemUsage()
Friday, February 13, 2009
Relational databases are not going anywhere
I stumbled upon this entry which argues against the existence of the RDBMS in distributed applications. I must say I disagree. The "key-value" database movement does attempt to solve some valid concerns. For instance, the complexity involved with deploying a clustered RDBMS can be daunting at best. The ease of which application developers can use these key-value databases is very powerful. I also agree with the assertion that a single schema distributed across n nodes will not be able to scale very easily.
However, at a lower level, there is still no substitute for the RDBMS when it comes to reading and writing persistent data. I often sense a stereotype among developers toward RDBMSes as being a bloated overhead. Again, I disagree. There exist several open source, lightweight RDBMSes that add zero or very little additional effort.
On the schema end of things, they are only fixed at the data level. The application can easily abstract new and dynamic shemas during its' lifetime.
I think several of the new "key-value" database concepts (such has the document abstraction) belong in the application or server layer, not the data layer. As far as nodes in the cloud, each node will benefit from an RDBMS for the foreseeable future.
However, at a lower level, there is still no substitute for the RDBMS when it comes to reading and writing persistent data. I often sense a stereotype among developers toward RDBMSes as being a bloated overhead. Again, I disagree. There exist several open source, lightweight RDBMSes that add zero or very little additional effort.
On the schema end of things, they are only fixed at the data level. The application can easily abstract new and dynamic shemas during its' lifetime.
I think several of the new "key-value" database concepts (such has the document abstraction) belong in the application or server layer, not the data layer. As far as nodes in the cloud, each node will benefit from an RDBMS for the foreseeable future.
Tuesday, January 13, 2009
The spreadsheet is 30 years old
It was 30 years ago that the spreadsheet application was invented. PC magazine has an entry about how the spreadsheet has brought society nothing but trouble. For example, the article blames the spreadsheet for many miscalculations in the past rather than the people responsible for building the spreadsheet.
I think the spreadsheet was an ingenious idea that has much broader applications than accounting tasks. The spreadsheet is simply a tool to visualize and manipulate data. In the end, it is the human interpretation of that data that leads to the undesirable consequences. Blaming the spreadsheet application for a financial crises is like placing the responsibility of car accidents on the invention of the automobile, rather than the people who drive them.
I think the spreadsheet was an ingenious idea that has much broader applications than accounting tasks. The spreadsheet is simply a tool to visualize and manipulate data. In the end, it is the human interpretation of that data that leads to the undesirable consequences. Blaming the spreadsheet application for a financial crises is like placing the responsibility of car accidents on the invention of the automobile, rather than the people who drive them.
Tuesday, July 22, 2008
JSON or XML?
Which is the better format for data representation? JSON? or XML? That is a tough one because there are several dimensions here that constitute better. Some easy differences:
Can the two data formats exist within the same context? Yes. Would it make sense to use both formats in the same application? Not likely. One exception I can think of is when using third-party libraries or tools that require one format or the other but you have already invested heavily in the other format. The two formats are isomorphic enough that there is no magic needed to convert between the two. It is just unnecessary if it can be avoided.
If I were to start developing an application from scratch today, and the choice between the two formats needed to be made, I would most likely choose JSON. There are few reasons to not use it. There exist JSON libraries for all the major programming languages. The only question is human interpretation. It would be pretty easy to build a JSON, tree-style, viewer/editor (I've never seen anything out there that does this, and only this). This JSON editor/viewer will be a topic of further discussion in later posts.
- XML is easier to understand, and therefore better for humans to read.
- JSON is more lightweight, and therefor better for software to read.
Can the two data formats exist within the same context? Yes. Would it make sense to use both formats in the same application? Not likely. One exception I can think of is when using third-party libraries or tools that require one format or the other but you have already invested heavily in the other format. The two formats are isomorphic enough that there is no magic needed to convert between the two. It is just unnecessary if it can be avoided.
If I were to start developing an application from scratch today, and the choice between the two formats needed to be made, I would most likely choose JSON. There are few reasons to not use it. There exist JSON libraries for all the major programming languages. The only question is human interpretation. It would be pretty easy to build a JSON, tree-style, viewer/editor (I've never seen anything out there that does this, and only this). This JSON editor/viewer will be a topic of further discussion in later posts.
Monday, July 21, 2008
A data pattern
A useful data pattern one can implement is to extract all behavioral features from the abstraction in question, and into a new behavioral super class. This trivial pattern will allow several data implementations of the class inherit the same behavior interface.
For example, here is a simplistic view of this pattern in the context of a blogging system.
Here, we have three classes; BlogBehavior, BlogDB, and BlogMemory. BlogBehavior defines the methods involved with what a blog does. BlogDB defines data access methods for interacting with the database. BlogMemory defines data access methods that will store and retrieve attributes in memory.
Methods defined by instances of BlogDB and BlogMemory may be invoked by methods in BlogBehavior. This would be ideal, because which of the two data instances is being used in the context of the blog system doesn't matter.
The concept of separating abstrations based on storage location, data access technonlogy (same database, different API), is not new. I have witnessed several ocasions, however, where the same behavior is not generalized. I think this is an important practice to keep in touch with when using object technology.
For example, here is a simplistic view of this pattern in the context of a blogging system.
Here, we have three classes; BlogBehavior, BlogDB, and BlogMemory. BlogBehavior defines the methods involved with what a blog does. BlogDB defines data access methods for interacting with the database. BlogMemory defines data access methods that will store and retrieve attributes in memory.
Methods defined by instances of BlogDB and BlogMemory may be invoked by methods in BlogBehavior. This would be ideal, because which of the two data instances is being used in the context of the blog system doesn't matter.
The concept of separating abstrations based on storage location, data access technonlogy (same database, different API), is not new. I have witnessed several ocasions, however, where the same behavior is not generalized. I think this is an important practice to keep in touch with when using object technology.
Subscribe to:
Posts
(
Atom
)