Challenge: Displaying Feed Items On A Web Page

16 commentsWritten on December 17th, 2011 by
Categories: Performance

I'm finally getting around to implementing the website for my company, and there's one small part of it that's quite interesting from an implementation point of view. The website will have a footer on each page which displays links to my 5 most recent blog posts:

Of course, I don't want to update those links manually whenever I publish a new post, so they need to be retrieved from my blog's RSS feed, which is published by Feedburner. I was hoping to be able to retrieve only the metadata from the posts (date, title and URL is all I need) because my feed always contains the last 20 posts and its total size is usually above 100KB. I haven't found a way to do that, so getting the information I need has to be retrieved through the full feed. Sure, 100KB isn't much but keep in mind that you need to retrieve it and parse it and that I absolutely want to minimize the time each request takes and that I'd rather not see any visual delays on the page either.

I'm interested in hearing how you would implement this. You have total freedom to pick the technologies you'd like to use and no limits on how you'd use them. My only requirements are these:

  • It needs to be fast
  • The fewer requests that are impacted by retrieving the feed data, the better
  • If I publish a post, the links on the company website should contain the new link within 30 minutes
  • The simpler the solution, the better

My solution can be found here.

« Blue Pill vs Red PillNode.js For Dummies »

This entry was posted on Saturday, December 17th, 2011 at 16:17 and is filed under Performance. You can subscribe to any comments to this entry via RSS 2.0. You can leave a response, or trackback from your own site.

  • Jeff Cyr

    - Fetch the feeds on the server
    - Keep only the last 5 posts summary
    - Cache it on the server
    - Refresh the server cache every 15 minutes
    => The cached data can be returned to clients without additional requests

    • http://davybrion.com Davy Brion

      could you go into more details about how you would fetch the feeds, and how you would refresh the cache?

      • Jeff Cyr

        I would fetch the feeds server-side with ‘System.ServiceModel.Syndication.SyndicationFeed’.

        Then I would store the last 5 items informations in memory (static variable).
        Creates a System.Threading.Timer with a X minutes interval to refetch the feeds.
        This logic can be wrapped in a server control that will render the feeds from the static variable.
        When the web application starts, call an static method on the server control class to fetch the feeds and start the timer.

  • http://www.facebook.com/people/Roger-Knapp/100000454117205 Roger Knapp

    Write a command-line application to fetch your feed, grab the information needed for the first 5 posts, and rewrite it as json directly to the website folder.  Schedule the application with the windows scheduler to run every x minutes.  Then write a little client javascript with jquery to pull the now static content in json and render it on the page.

  • Frederik Vig

    Use Yahoo Query Language, you can customize what you get back and the format (and it’s super fast)

    • http://davybrion.com Davy Brion

      hadn’t heard about that yet, looks pretty interesting
      but the fact that you have to go through a yahoo server, in combination with the following notice on their page (http://developer.yahoo.com/yql/) “If we’re going to shut down YQL, we will give you at least 6 months notice with an announcement on this web page and in our forum”, that kinda scares me.

      • Frederik Vig

        It uses their CDN so should be much faster than hosting yourself, but I agree that the 6 month notice message can seem scary, but then again this is a problem you´ll face with all external services that you don´t control. If you for instance are pulling the jQuery library from Google´s CDN, you should also have a fallback in cases when it´s not available. Same goes for your feed. What Yahoo! does is cache the result in their CDN for super fast access, if you´re using WebRequest or similar to get it (either your feed directly or through YQL) you should have a maximum amount of time you wait before you timeout and either server the user an old cached versioned or something else, but don´t keep them waiting. Like others have said you can use async tasks, scheduled job etc, all have their pros and cons. Depends on how much time you want to use on this (coding) and how important it´s for your business. Does the user experience and business value suffer if the feed is unavilable for a periode of time?
        I still maintain that using YQL is the quickest and most reliable way to get it if you don´t want to do a whole bunch of extra coding.

        • http://davybrion.com Davy Brion

          i agree that a fallback scenario is necessary for every service you don’t control… but with Yahoo’s somewhat unpredictable and unclear future, that message only makes me feel worse about it. It sounds as if the team behind isn’t even clear whether this technology is going to be around for a while.

          As for the amount of coding required… i’m curious to hear what you’ll think of my solution in a few days :)

  • Anonymous

    Look at pubsubhubbub?

    • http://davybrion.com Davy Brion

      looks interesting :)

      • Steve Sheldon

        Looking at it further, I really think that might be exactly what you want.  Looks like it’s integrated into feedburner even.  Looks like you just have to implement a handler on your subscribing site that will receive the updates.

  • http://lightyear.be teranex

    Using Drupal you can install the Feeds module. This can pull feeds, process them and store them as nodes (=standard drupal content). Pulling those feeds is done using the standard Drupal cron job. Once the content is in Drupal it can be cached etc just as every other content in Drupal. Setting this up would take around 15 minutes :)
    But I guess you are probably not interested in using Drupal.. ;)

    • http://davybrion.com Davy Brion

      i’m interested in hearing what people come up with, regardless of technology

  • http://twitter.com/vkornov Victor Kornov

    I’d use http://code.google.com/apis/feed/. Bearing in mind that your requirements match it 100% I believe that’s what you went with.

  • http://twitter.com/vkornov Victor Kornov

    I’d use http://code.google.com/apis/feed/. Bearing in mind that your requirements match it 100% I believe that’s what you went with.

  • Pingback: Displaying Feed Items On A Web Page: My Solution

Meta

Recent Posts

Categories