Backbone Collection Fetch Scheduler

It's tricky to properly manage the the fetching of several Backbone collections in a given application. There are factors at play that require close attention. We don't want to put too much load on the server. We don't want to put too much load on the client. We need to consider things such as concurrency, and the ability to prioritize based on collection freshness. This seems like a lot to take on, and it maybe overkill to worry about such matters in smaller applications. But as the sheer volume of collections grows, as the complexity between the collection relationships increases, you're going to need some sort of fetch scheduling strategy.

Here's one way to go about doing it. It's a generic utility called a fetcher. It simply takes an array of Backbone collections as an argument, and orchestrates how each of those collections are fetched based on a few factors.

Let's take a look at the Fetcher code, and how it schedules fetches.

function Fetcher( collections ) {
        
    this.wait = 3000;
        
    this.concurrency = 2;
        
    this.collections = _.map( collections, function( c ) {
        return {
            collection: c,
            timestamp: 0,
            xhr: false
        };
    });

    this.start = function() {
            
        this.fetchNext();
            
        this.interval = setInterval( (function( fetcher ) {
            return (function() {
                fetcher.fetchNext()
            });
        })( this ), this.wait);
            
    };
        
    this.stop = function() {
        clearInterval( this.interval );
    };
        
    this.fetchNext = function() {
            
        var collections = _.chain( this.collections ),
                
            next = collections.sortBy( "timestamp" )
                       .first()
                       .value(),
                
            pending = collections.map( "xhr" )
                          .without( false )
                          .value()
                          .length;
            
        next.timestamp = new Date();
            
        if ( pending > this.concurrency ) {
            return;
        }
            
        if ( next.xhr !== false ) {
            return this.fetchNext();
        }
            
        next.xhr = next.collection.fetch().done( function() {
            next.xhr = false;
        });
            
    }
        
};

The Fetcher "class" get's instantiated with an array of Backbone collections as it's argument. The instance also has two tweak-able properties — wait and concurrency. We'll see how those are put to use in the fetchNext() method. But first, we have to setup the collections property. All we're doing is wrapping each property with an object. This object has additional properties used by the fetcher. These are xhr and timestamp. The start() and stop() methods basically start and stop the polling loop, respectively.

At the heart of Fetcher is the fetchNext() method. All Fetcher does is call fetchNext() in a loop. The wait property specifies the duration of each interval. All fetchNext() does is finds the collection that should be fetched next, and calls fetch() on that collection. Nothing too fancy, the trick is in figuring out which collection should be fetched next.

We start with prioritizing the collections array based on freshness. That is, staler collections, marked with the timestamp property, take precedence over fresher collections. Now that we have the stalest collection, we need to make sure it's a good candidate to be fetched. So we'll check a couple things. First, we figure out how many XHR requests are outstanding. We can do this fairly easily by mapping the xhr collection property into a new array, and counting the ones that aren't false. If this number is greater than the concurrency property, we don't fetch anything. Instead, we return and give the XHR request backlog a chance to clear up.

If the candidate collection passes the concurrency check, we then have to make sure that there isn't a pending XHR request for that same collection that hasn't yet completed. If there is, we just move on to the next candidate. If, on the other hand, the candidate collection is good to go, we call the fetch() method and update the xhr property. When the request completes, the xhr property is set to false once again, freeing it up to be fetched again.