A couple of days ago I asked you how you'd implement showing links from an RSS feed on a web page (in this case: my new company web site). These are my requirements for this:
- It needs to be fast
- The fewer requests that are impacted by retrieving the feed data, the better
- If I publish a post, the links on the company website should contain the new link within 30 minutes
- The simpler the solution, the better
I came up with a very simple solution, which satisfies these requirements better than any other solution I could think of, or heard of from other people. It is extremely fast, doesn't delay any requests, and doesn't require me to deploy anything but the company website. I'm building the site with Express on Node.js, which means I can take full advantage of the asynchronous nature of Node.js to implement this.
Let's go over the code... in the script that starts the express server, I have the following code:
var express = require('express'),
app = module.exports = express.createServer(),
NodePie = require('nodepie'),
request = require('request'),
recentFeedItems = null;
app.dynamicHelpers({
getRecentFeedItems: function() {
return recentFeedItems;
}
});
// ... some extra configuration of Express that isn't relevant to this post
var processFeed = function(callback) {
request('http://feeds.feedburner.com/davybrion', function(err, response, body) {
if (!err && response.statusCode == 200) {
var feed = new NodePie(body);
feed.init();
recentFeedItems = feed.getItems(0, 5);
if (callback) callback();
};
});
};
setInterval(processFeed, 1800000); // process feed items every 30 minutes
processFeed(function() {
app.listen(3000);
console.log('Express started on port 3000');
});
I'll discuss the code in just a moment, but first I want to show the view code that renders the links:
<ul>
<% getRecentFeedItems.forEach(function(item) { %>
<li><time class="date"><%= item.getDate().getDate() + '/' + (item.getDate().getMonth() + 1) %></time><a href="<%= item.getPermalink() %>"><%= item.getTitle() %></a></li>
<% }); %>
</ul>
And that's all. This is the solution in its entirety!
If you're new to Node, this code probably requires some explanation. Let's start with this part:
app.dynamicHelpers({
getRecentFeedItems: function() {
return recentFeedItems;
}
});
Here I'm adding a dynamic helper to the Express application. It basically means that my views have access to the getRecentFeedItems function, which returns the value of the recentFeedItems variable. It's important to know that the getRecentFeedItems function creates a closure on the recentFeedItems variable created above it. That means that if the value of the recentFeedItems variable changes at any point in time, the getRecentFeedItems function will return that new value.
var processFeed = function(callback) {
request('http://feeds.feedburner.com/davybrion', function(err, response, body) {
if (!err && response.statusCode == 200) {
var feed = new NodePie(body);
feed.init();
recentFeedItems = feed.getItems(0, 5);
if (callback) callback();
};
});
};
This just creates a function that we can use later on. It retrieves the feed asynchronously, and when the result is retrieved, we parse the feed using the NodePie library and we get the 5 most recent items which we store in the recentFeedItems variable. Again, this creates a closure on the recentFeedItems variable which means that every time we assign a value to this variable, any subsequent call to the getRecentFeedItems function will return the value we just assigned to it because both functions point to the same memory thanks to the magic of closures. Finally, if a callback is provided as a parameter, the callback will be invoked.
setInterval(processFeed, 1800000); // process feed items every 30 minutes
processFeed(function() {
app.listen(3000);
console.log('Express started on port 3000');
});
The call to setInterval makes sure that the processFeed function is called every 30 minutes. After that, we call the processFeed function manually, and we pass in a callback where we start the Express server. This guarantees that the feed items will be in memory before the server starts processing requests.
What makes this solution so great is that we take full advantage of some of Node's benefits. Whenever we retrieve the RSS feed, Node.JS will retrieve that data asynchronously. As soon as it has fired the request to get the RSS feed, it just goes to the next event in its eventloop so no request is kept waiting while we wait for the data to be downloaded. Until the data from the RSS feed is returned, each request will just use the items that are stored in the recentFeedItems variable. Once the data has been returned, our callback is executed which overwrites the value of the recentFeedItems variable. We don't need to do any locking here because the Node.JS eventloop is single-threaded: while our callback is running, no other code that has access to the recentFeedItems variable can be executed anyway. And the actual parsing of the RSS feed is done by NodePie, which uses expat behind the scenes, which is supposedly the fastest C XML parser available.
Looking back on my initial requirements, I think this solution matches very well.
