Archive for 2011

Looking Back On 2011

3 commentsWritten on December 26th, 2011 by
Categories: Off Topic

This was my first year working as an independent, and it's been quite a year. Things have gone very well, and I'm amazed at how quickly it all went by. This time last year, I was looking forward to start working for my first client though I didn't quite know what to expect. I hoped it would be interesting, and it certainly has been. I've been able to work with interesting technologies, with the kind of technical variety that I've come to really enjoy. I wrote more code in JavaScript than in C# this year, and I've also been able to do quite a bit of Ruby as well. I certainly didn't expect that going into this contract, so that was a major bonus. Even better than the technologies I've been able to work with this year are the people I've worked with. The team I've worked with has a fantastic mix of experience, knowledge and talent and it feels like all of us continuously learn new things from each other which is a situation I can only recommend to every developer.

Another great part about being independent is that you're free to do extra things on the side as well. I'm very happy with the responses that I've gotten from the people who've taken my NHibernate course, and I'm going to continue to offer it in 2012. I initially thought it would be something I'd do once or twice, but I was able to do the course 4 times this year: 1 public and 3 private ones. I've got another private one scheduled in February and will organize another public course in 2012, though this time I'm taking care of the whole thing myself instead of working with another company to organize it.

The most important thing I've learned from this first year is that it's important not to try to do too much. The first 6 months of 2011 were way too busy and hectic, and I only spent one full day doing absolutely nothing work-related. I've been taking things much slower since July and I'm spending much more time enjoying other things in life outside of work. For 2012, my biggest priority is pacing myself much better and spreading things out more. I always seem to alternate between months where I'm always working on something, followed by months of doing as little as possible outside of the 40 hours a week I do for my client. Hopefully, I can finally find a maintainable balance between those two extremes in 2012 :)

On the blogging front, 2011 was a different year than the previous ones on this blog. In the previous years, I averaged about 150 posts a year but this year I only published 71 posts. And I'm actually happy about that. In fact, I hope to get that number down to 50 next year. I still like writing posts, but I no longer feel the need to like I did in the past while I was trying to build a bit of an audience. That often meant pushing myself to keep publishing new posts. Nowadays, I'll only write a post if I really feel like it, which in the long run is much more sustainable anyway. As far as topics covered this year, things have changed quite a bit as well. Some of my views, opinions and technical preferences have changed this year, which naturally reflects in the content I've published. I'm sure I've lost some readers this year because of it, but I also seem to have gained a lot more readers than I've lost. I don't know yet what I'll be writing about in 2012, but I will continue to push people to learn new things (or old things that people seem to forget about) and do better like I always have on this blog, in my own unique way of course :)

I hope you all enjoyed Christmas, and I wish you all a happy 2012. If it does turn out to be our last year on this planet, we might as well make it count.

Displaying Feed Items On A Web Page: My Solution

7 commentsWritten on December 20th, 2011 by
Categories: express.js, JavaScript, node.js, Performance

A couple of days ago I asked you how you'd implement showing links from an RSS feed on a web page (in this case: my new company web site). These are my requirements for this:

  • It needs to be fast
  • The fewer requests that are impacted by retrieving the feed data, the better
  • If I publish a post, the links on the company website should contain the new link within 30 minutes
  • The simpler the solution, the better

I came up with a very simple solution, which satisfies these requirements better than any other solution I could think of, or heard of from other people. It is extremely fast, doesn't delay any requests, and doesn't require me to deploy anything but the company website. I'm building the site with Express on Node.js, which means I can take full advantage of the asynchronous nature of Node.js to implement this.

Let's go over the code... in the script that starts the express server, I have the following code:

var express = require('express'),
    app = module.exports = express.createServer(),
    NodePie = require('nodepie'),
    request = require('request'),
    recentFeedItems = null;

app.dynamicHelpers({
    getRecentFeedItems: function() {
        return recentFeedItems;
    }
});

// ... some extra configuration of Express that isn't relevant to this post

var processFeed = function(callback) {
    request('http://feeds.feedburner.com/davybrion', function(err, response, body) {
        if (!err && response.statusCode == 200) {
            var feed = new NodePie(body);
            feed.init();
            recentFeedItems = feed.getItems(0, 5);
            if (callback) callback();
        };
    }); 
};

setInterval(processFeed, 1800000); // process feed items every 30 minutes

processFeed(function() {
    app.listen(3000);
    console.log('Express started on port 3000');    
});

I'll discuss the code in just a moment, but first I want to show the view code that renders the links:

<ul>
<% getRecentFeedItems.forEach(function(item) { %>
    <li><time class="date"><%= item.getDate().getDate() + '/' + (item.getDate().getMonth() + 1) %></time><a href="<%= item.getPermalink() %>"><%= item.getTitle() %></a></li>
<% }); %>
</ul>

And that's all. This is the solution in its entirety!

If you're new to Node, this code probably requires some explanation. Let's start with this part:

app.dynamicHelpers({
    getRecentFeedItems: function() {
        return recentFeedItems;
    }
});

Here I'm adding a dynamic helper to the Express application. It basically means that my views have access to the getRecentFeedItems function, which returns the value of the recentFeedItems variable. It's important to know that the getRecentFeedItems function creates a closure on the recentFeedItems variable created above it. That means that if the value of the recentFeedItems variable changes at any point in time, the getRecentFeedItems function will return that new value.

var processFeed = function(callback) {
    request('http://feeds.feedburner.com/davybrion', function(err, response, body) {
        if (!err && response.statusCode == 200) {
            var feed = new NodePie(body);
            feed.init();
            recentFeedItems = feed.getItems(0, 5);
            if (callback) callback();
        };
    }); 
};

This just creates a function that we can use later on. It retrieves the feed asynchronously, and when the result is retrieved, we parse the feed using the NodePie library and we get the 5 most recent items which we store in the recentFeedItems variable. Again, this creates a closure on the recentFeedItems variable which means that every time we assign a value to this variable, any subsequent call to the getRecentFeedItems function will return the value we just assigned to it because both functions point to the same memory thanks to the magic of closures. Finally, if a callback is provided as a parameter, the callback will be invoked.

setInterval(processFeed, 1800000); // process feed items every 30 minutes

processFeed(function() {
    app.listen(3000);
    console.log('Express started on port 3000');    
});

The call to setInterval makes sure that the processFeed function is called every 30 minutes. After that, we call the processFeed function manually, and we pass in a callback where we start the Express server. This guarantees that the feed items will be in memory before the server starts processing requests.

What makes this solution so great is that we take full advantage of some of Node's benefits. Whenever we retrieve the RSS feed, Node.JS will retrieve that data asynchronously. As soon as it has fired the request to get the RSS feed, it just goes to the next event in its eventloop so no request is kept waiting while we wait for the data to be downloaded. Until the data from the RSS feed is returned, each request will just use the items that are stored in the recentFeedItems variable. Once the data has been returned, our callback is executed which overwrites the value of the recentFeedItems variable. We don't need to do any locking here because the Node.JS eventloop is single-threaded: while our callback is running, no other code that has access to the recentFeedItems variable can be executed anyway. And the actual parsing of the RSS feed is done by NodePie, which uses expat behind the scenes, which is supposedly the fastest C XML parser available.

Looking back on my initial requirements, I think this solution matches very well.

Node.js For Dummies

14 commentsWritten on December 18th, 2011 by
Categories: JavaScript, node.js

I'm sure you've all heard of Node.js by now. Its popularity is increasing rapidly, which means it's a good idea to be aware of what Node.js is and especially how it differs from more traditional technology stacks. In this post, I'll try to give an easy-to-understand overview of what makes Node.js different and make it clear that it's more than just server-side JavaScript. Note that this overview is highly simplified and only meant to help people understand how Node.js works. This is definitely not a completely accurate description of the lower-level details of Node.js.

Evented/Asynchronous I/O

In most technology stacks, API calls for I/O operations are synchronous. As in, the thread that executes the operation is blocked for the duration of the I/O operation until that operation has completed. Once completed, execution of your code proceeds. Of course, a lot of technology stacks have asynchronous variants of those operations available as well, but generally speaking, they aren't used as often as the synchronous variants. In Node.js, it's the other way around. All I/O operations are asynchronous and there are only a few synchronous implementations available (and you're generally discouraged from using them).

This means that whenever you do an I/O operation (file manipulation, network requests, database operations, etc…), Node.js initiates the I/O operation through a lower-level C/C++ layer which will perform the operation asynchronously. Once the operation has completed, Node.js will execute your callback function that you passed as a parameter to the I/O operation's function call. The important thing here is that while the I/O operation is being executed, Node.js doesn't have to wait for the operation to complete, and is able to focus entirely on processing other events. And those events can be anything: incoming network requests, executing callbacks from other operations that have completed, or invoking whatever function that is assigned to a particular event.

Eventloop

The Node.js eventloop is what makes Node.js so interesting and powerful. Node.js basically just keeps reading from an event queue until that queue is empty. As it loops through the events to be processed, it invokes the JavaScript functions that have been assigned for those events. If any of those functions performs an I/O operation, Node.js will initiate the operation and then immediately move to the next event in the event queue. Once the I/O operation has completed, an event will be added to the event queue with a reference to your original callback. Once all preceding events have been processed, Node.js will get to the newly added event and invoke your callback. Because all I/O operations are asynchronous, this enables Node.js to maximize its efficiency as it processes events because it doesn't need to wait for slow I/O operations to complete.

Single-threaded

One thing that people don't always realize is that the Node.js eventloop is single-threaded. This has some nice benefits but there's a huge drawback as well. The biggest benefit is that you don't need to worry about concurrent access to shared state. After all, there is never more than 1 thread executing your JavaScript functions. This means you don't have to write any locking code to protect shared state. The drawback to the single-threaded eventloop is that you need to be careful not to block the event loop. If you're planning on doing heavy synchronous processing in your JavaScript code, you need to realize that no other events can be processed by Node.js until that synchronous block of code has completed. Obviously, since there's only one thread going through the eventloop, any delay you cause in your code can be very costly to overall throughput and performance. For now, it's best to execute synchronous processing routines as a child process, possibly even in a language that is more suitable for this than JavaScript. But it seems that future Node.js versions will provide a more integrated way to deal with this.

Why JavaScript?

JavaScript's support of closures and it treating functions as first class objects means it's ideally suited for the evented programming model that Node.js offers. Many people still think of JavaScript as a joke or a toy language, but it's a lot more powerful than many people think it is. Yes, it certainly has problems as well, but it's definitely worth learning. I do hope that this post has made it clear that there's a lot more to Node.js than simply being server-side JavaScript. What makes Node.js so interesting are the principles that I've tried to explain in this post. Those principles can be implemented with other languages as well, and could be made to work just as great as, or perhaps even greater than Node.js itself. But you'd be hard-pressed to find a language that's so ubiquitous, yet completely devoid of a pre-existing synchronous I/O infrastructure.

Anyways, I hope I succeeded at making it somewhat clear how Node.js works and why it's so different from most other technology stacks.

Challenge: Displaying Feed Items On A Web Page

16 commentsWritten on December 17th, 2011 by
Categories: Performance

I'm finally getting around to implementing the website for my company, and there's one small part of it that's quite interesting from an implementation point of view. The website will have a footer on each page which displays links to my 5 most recent blog posts:

Of course, I don't want to update those links manually whenever I publish a new post, so they need to be retrieved from my blog's RSS feed, which is published by Feedburner. I was hoping to be able to retrieve only the metadata from the posts (date, title and URL is all I need) because my feed always contains the last 20 posts and its total size is usually above 100KB. I haven't found a way to do that, so getting the information I need has to be retrieved through the full feed. Sure, 100KB isn't much but keep in mind that you need to retrieve it and parse it and that I absolutely want to minimize the time each request takes and that I'd rather not see any visual delays on the page either.

I'm interested in hearing how you would implement this. You have total freedom to pick the technologies you'd like to use and no limits on how you'd use them. My only requirements are these:

  • It needs to be fast
  • The fewer requests that are impacted by retrieving the feed data, the better
  • If I publish a post, the links on the company website should contain the new link within 30 minutes
  • The simpler the solution, the better

My solution can be found here.

Blue Pill vs Red Pill

36 commentsWritten on December 15th, 2011 by
Categories: Opinions

I've been somewhat critical of certain 'hot' MS technologies on Twitter this week. Particularly: Azure, Nuget and Entity Framework, and I didn't even bother to complain about the troubles I've had with the differences between 32-bit and 64-bit Powershell today. And I've noticed that whenever I voice a negative opinion on a Microsoft technology that is considered cool or great, I lose a few Twitter followers. My reaction to that is: good riddance. I prefer losing a few followers because of it, than having to deal with the responses from certain fanboys who either haven't stepped outside of the MSDN world, or are just trying to impress others with a devotion that they're hoping will increase their status within their organization or within the Microsoft developer community in general.

A lot of people think that I'm anti-Microsoft. And I'm really not. I'm not anti-anything. I am pro-quality. I have high standards because I've seen and experienced high quality solutions elsewhere and when I have to deal with lower quality alternatives to those solutions, I tend to compare them, which is only natural. If Microsoft releases good stuff, I'm more than happy to use it. I even considered creating an Entity Framework training course, similar to my NHibernate training course, because I was hearing many good things about it. But after trying to deal with a certain Entity Framework problem that came up this week in the one project at work that uses Entity Framework, I couldn't help but think "I don't want this mess in my life, no matter how much money I could make with it". I sincerely want to use good Microsoft technology, but too often it just disappoints me. And that in itself, isn't much of a problem since there are plenty of good Open Source alternatives around. What I do consider to be a problem is the reaction that many people have when you're critical of Microsoft technology.

I've often compared it to living in The Matrix. A lot of us are living in a world where they are being pushed into believing something that just isn't true. And some of us at some point get to choose between taking the blue pill or the red pill. The blue pill symbolizes blissful ignorance of illusion, while the red pill symbolizes the sometimes painful truth of reality. A lot of developers in the Microsoft world choose to keep their eyes closed and blindly believe whatever Microsoft tells them to believe. They'll run into a variety of problems with the technologies they've been told to use but a lot of people just accept it for what it is because they don't know any better, or because they're scared of the seemingly harsh world that awaits them should they choose to ignore Microsoft's guidance and venture out into a world that is more chaotic, yet offers more possibilities and flexibility. If you take the red pill, you learn a lot about what's really possible yet you face the added burden of having to deal with the people who've picked the blue pill and even worse, the technology that comes with it. Because that truly is the only bad part about taking the red pill. You'll start taking some things for granted, and when faced with technologies that don't quite match up to what you've recently become used to, you will get frustrated because of it. After all, you know things can be done much better with less friction, yet here you are, dealing with problems that have been solved by other libraries/frameworks already. That is the only sour taste you'll experience after having swallowed the red pill.

The choice between the blue pill or the red pill is one that everyone has to make for themselves. And honestly, I can't be bothered anymore to try to convince people to take the red pill instead of the blue pill. I've learned to adopt a "whatever floats your boat without sinking mine is fine with me" attitude, but sometimes, I can't help but wish for a world were people would try to think just a little bit more for themselves instead of blindly following what a dominant entity is telling them to do or use. Look around, see what other people and communities are doing and honestly ask yourself "are they doing things better than I am?". And if they are, put in the effort to figure out why and how they're doing what they're doing. The worst thing that could possibly happen is a temporarily sour taste, and there are many ways to wash that away.

« Older Entries

Meta

Recent Posts

Categories

Archives