testing

Take Advantage Of Your BDD Framework

10 commentsWritten on August 3rd, 2011 by
Categories: Code Quality, testing

NOTE: the examples in this post use JavaScript and Jasmine, but the advice is applicable to whatever BDD framework you use.

I'm using Jasmine for the automated tests in my breakable toy project. It's a BDD framework, very similar to RSpec so it makes it really easy to write your tests in the given-when-then style which i'm (finally) starting to like more and more. What makes given-when-then testing interesting is that you have 3 explicit steps in a test: setting up the 'given' part, doing something in the 'when' part, and asserting on it in the 'then' part. It makes it especially easy to reuse the 'givens' and even the 'whens' if you just want to add a few 'thens'.

Let's first start off with a bad example, one that i wrote about 2 weeks ago:

describe('given an existing customer', function() {

    describe('when it is retrieved from the database', function() {

        it('should contain the same values that have been inserted', function() {
            var customer = new CustomerBuilder()
                .withIncludeContactOnInvoice()
                .build();

            customer.save(function(err) {
                Customer.findById(customer.id, function(err, result) {
                    helper.customersShouldBeEqual(result, customer);                    
                    asyncSpecDone();
                });
            });
            asyncSpecWait();
        });

    });

});

It's organized in a given-when-then style, but in a bad way. The only benefit that i'm getting from it here, is that the structure is sort of easy to read: given an existing customer, when it is retrieved from the database, it should contain the same values that have been inserted. When i wrote it, i knew that saving the customer should be done in the 'given' step, retrieving it should be done in the 'when' step and comparing the fields of the inserted customer and the retrieved customer should be done in the 'then' step. In this case, everything is being done in the 'then' step.

When i wrote that code, i figured it would just be easier to do it this way, because on Node.JS every I/O call is asynchronous and i thought it would hurt readability if i were to split everything up according to the given-when-then rules due to the asynchronous calls. But then i wanted to add tests for updating and deleting a customer. In both cases, the 'given' part would again be 'given an existing customer'. So i wanted to add them in the right place, which meant i had to choose between duplicating the code to save the customer in each test, or bite the bullet and split it up properly and deal with the asynchronous calls.

Let's start with our original example, and move the saving of the customer to the 'given' step, and the retrieval to the 'when' step:

describe('given an existing customer', function() {

    var customer = null;

    beforeEach(function(err) {
        customer = new CustomerBuilder()
            .withIncludeContactOnInvoice()
            .build();

        customer.save(function(err) {
            expect(err).toBeNull();
            asyncSpecDone();
        });
        asyncSpecWait();
    });

    describe('when it is retrieved from the database', function() {

        var retrievedCustomer = null;

        beforeEach(function() {
            Customer.findById(customer.id, function(err, result) {
                expect(err).toBeNull();
                retrievedCustomer = result;
                asyncSpecDone();
            });
            asyncSpecWait();
        });

        it('should contain the same values that have been inserted', function() {
            helper.customersShouldBeEqual(retrievedCustomer, customer);
        });

    });

});

Despite ending up with more lines of code, there are some notable improvements here. We take advantage of the beforeEach method, which is executed once before each spec (in the case of Jasmine, a call to the 'it' method is a spec) is executed and once before each spec in each nested suite (in the case of Jasmine, a call to the 'describe' method creates a new suite) is executed. Most BDD-frameworks have something similar. Obviously, due to the asynchronous nature of Node.JS we use the asyncSpecWait() and asyncSpecDone() calls (added to Jasmine by jasmine-node) to wait until the asynchronous calls have completed before we move to the next step. In production Node.JS code, you really don't want to do this since that completely takes away the benefits of the platform, but for automated tests, it makes sense to do so. This enables us to put the right code in the right place: saving the customer is done in the 'given' step, retrieving it is done in the 'when' step, and the 'then' step only contains the code to verify that both instances are equal. If we need to verify something else about retrieved customers, we could easily add more specs (calls to the 'it' method) without having to repeat any of the setup work.

Now we can also add the tests for the update and delete scenario, within the context of the 'given an existing customer' scenario.

describe('given an existing customer', function() {

    var customer = null;

    beforeEach(function(err) {
        customer = new CustomerBuilder()
            .withIncludeContactOnInvoice()
            .build();
            
        customer.save(function(err) {
            expect(err).toBeNull();
            asyncSpecDone();
        });
        asyncSpecWait();
    });

    describe('when it is retrieved from the database', function() {

        var retrievedCustomer = null;

        beforeEach(function() {
            Customer.findById(customer.id, function(err, result) {
                expect(err).toBeNull();
                retrievedCustomer = result;
                asyncSpecDone();
            });
            asyncSpecWait();
        });
    
        it('should contain the same values that have been inserted', function() {
            helper.customersShouldBeEqual(retrievedCustomer, customer);
        });
        
    });
    
    describe('when it is modified and updated', function() {
            
        beforeEach(function() { 
            customer.name = 'some other customer';
            customer.vatNumber = '0456.876.235';
            customer.address = {
                street: 'some other street',
                postalCode: '12345',
                city: 'some other city'
            };
            customer.phoneNumber = '123456789';
            customer.contact = {
                name: 'some name',
                email: 'some_email@hotmail.com'
            };
            customer.save(function(err) {
                expect(err).toBeNull();
                asyncSpecDone();
            });
            asyncSpecWait();
        });

        it('contains the updated values in the database', function() {
            Customer.findById(customer.id, function(err, result) {
                helper.customersShouldBeEqual(result, customer);
                asyncSpecDone();
            });
            asyncSpecWait();
        });

    });

    describe('when it is deleted', function() {
        
        beforeEach(function() {
            customer.remove(function(err) {
                expect(err).toBeNull();
                asyncSpecDone();
            });
            asyncSpecWait();
        });     

        it('can no longer be retrieved', function() {
            Customer.findById(customer.id, function(err, result) {
                expect(result).toBeNull();
                asyncSpecDone();
            });
            asyncSpecWait();
        });

    });
    
});

In this case, the only duplication we have are the calls to asyncSpecWait and asyncSpecDone, which can't really be avoided with this style of testing on the Node.JS platform. Other than that, each part of the code is focused solely on that what it needs to do. If you're using a BDD framework, be sure you leverage it to make sure each part of your testcode is as focused on its task as it can be.

Sharing An IE Instance Among Multiple Tests With WatiN And MS Test

4 commentsWritten on March 24th, 2011 by
Categories: testing

Lost some time yesterday trying to get something working with MS Test (not my choice, but that's what my client uses) that i had expected to be easy. After all, it was especially easy to get working with NUnit. I wanted to create a base testing fixture which would instantiate one instance of Internet Explorer for the entire test run, and make that instance available to each test in the assembly. Sounds easy, no?

First problem: MS Test runs each test on a different thread.

When you use IE through WatiN, it uses COM behind the scenes. Accessing COM objects from different threads is not a safe thing to do and can lead to the following exception: System.Runtime.InteropServices.InvalidComObjectException: COM object that has been separated from its underlying RCW cannot be used.

Running each test individually worked, but running the entire suit made every test except for the first one fail with that exception because MS Test uses a different thread for each test (i suppose the development team did that to make sure it was enterprisey). Quite annoying, but luckily for me, the only other guy in the world who uses MS Test with WatiN also ran into the same problem and he described his workaround on his blog.

I made minor modifications to his IEStaticInstanceHelper class (basically just turned it into a static class) so my version looks like this:

    public static class IEStaticInstanceHelper
    {
        // TODO: move this to a config file
        public const string ROOT_URL = "http://localhost:13834/";

        private static IE _ie;
        private static int _previouslyKnownIeThreadHashCode;
        private static string _ieHwnd;

        public static void Initialize()
        {
            IE = new IE(ROOT_URL);
        }

        public static IE IE
        {
            get
            {
                if (GetCurrentThreadHashCode() != _previouslyKnownIeThreadHashCode)
                {
                    _ie = Browser.AttachTo<IE>(Find.By("hwnd", _ieHwnd));
                    _previouslyKnownIeThreadHashCode = GetCurrentThreadHashCode();
                }
                return _ie;
            }
            private set
            {
                _ie = value;
                _ieHwnd = _ie.hWnd.ToString();
                _previouslyKnownIeThreadHashCode = GetCurrentThreadHashCode();
            }
        }

        private static int GetCurrentThreadHashCode()
        {
            return Thread.CurrentThread.GetHashCode();
        }
    }

I also had the following AssemblyInitialize and AssemblyCleanup methods:

        [AssemblyInitialize]
        public static void AssemblyInitialize(TestContext testContext)
        {
            IEStaticInstanceHelper.Initialize();
        }

        [AssemblyCleanup]
        public static void AssemblyCleanup()
        {
            if (IEStaticInstanceHelper.IE != null)
            {
                IEStaticInstanceHelper.IE.Close();
                IEStaticInstanceHelper.IE.Dispose();
            }
        }

MS Test will call the AssemblyInitialize method before any test in the assembly is executed, provided that you don't forget to add the TestContext parameter to your method or it will silently be ignored (WTF?!). It'll also call the AssemblyCleanup method after each test in the assembly has finished executing.

Second problem: MS Test runs the AssemblyCleanup method in an MTA thread, even though each test is executed in STA threads by default.

As you can see in my AssemblyCleanup method, i access the IE property of IEStaticInstanceHelper. That property getter contains the following line:

    _ie = Browser.AttachTo<IE>(Find.By("hwnd", _ieHwnd));

That line works perfectly during the execution of the tests. When it is called from the AssemblyCleanup method, it times out after 30 seconds because it can't seem to find the IE window with the handle (_ieHwnd) that is known to be valid. And this, apparently, is because the current thread is an MTA thread when we're within the AssemblyCleanup method instead of an STA thread. I can't for the life of me figure out why they'd use an MTA thread for the AssemblyCleanup method while they use STA threads for the tests, but i will again assume it was done to keep up to the high enterprisey standard that people expect from something like MS Test.

The solution, while a horrible hack, is quite simple and works perfectly:

        [AssemblyCleanup]
        public static void AssemblyCleanup()
        {
            var thread = new Thread(() =>
            {
                if (IEStaticInstanceHelper.IE != null)
                {
                    IEStaticInstanceHelper.IE.Close();
                    IEStaticInstanceHelper.IE.Dispose();
                }
            });

            thread.SetApartmentState(ApartmentState.STA);
            thread.Start();
            thread.Join();
        }

There... nice and enterprisey.

Using Generic TestFixtures To Run Tests In Multiple Browsers With WatiN

6 commentsWritten on February 14th, 2011 by
Categories: testing

Just ran into something that i thought was pretty cool. If you're using WatiN, it's relatively easy to write browser-based automated tests without resorting to recorded tests. And since WatiN supports multiple browsers, you can write those tests in a browser-agnostic manner. And if you make use of NUnit's Generic Fixtures (introduced in NUnit 2.5), you can very easily run those tests in multiple browsers as well. Suppose you have the following base test fixture:

    public abstract class ViewTest<TBrowser> where TBrowser : Browser, new()
    {
        [TestFixtureSetUp]
        public void FixtureSetUp()
        {
            Browser = new TBrowser();
            Browser.GoTo(RootUrl);
        }

        [TestFixtureTearDown]
        public void FixtureTearDown()
        {
            if (Browser != null)
            {
                Browser.Dispose();
            }
        }

        [SetUp]
        public void SetUp()
        {
            DoSetUp();
        }

        protected TBrowser Browser { get; private set; }

        protected string RootUrl { get { return ConfigurationManager.AppSettings["RootUrl"]; } }

        protected virtual void DoSetUp() { }
    }

You can then write a test fixture like this:

    [TestFixture(typeof(IE))]
    [TestFixture(typeof(FireFox))]
    public class ListCustomersTests<TBrowser> : ViewTest<TBrowser> where TBrowser : Browser, new()
    {
        protected override void DoSetUp()
        {
            Browser.GoTo(RootUrl + "Customers");
        }

        [Test]
        public void ShowsErrorMessageWhenClickingProceedWithoutSelectingAnItemFromGrid()
        {
            var proceedButton = Browser.Button(button => button.Value == "Proceed");
            Assert.IsFalse(Browser.Para("selection_required").Exists);
            proceedButton.Click();
            Assert.That(Browser.Para("selection_required").Exists);
        }
    }

And when you run your tests, it will run this test once in IE, and once in Firefox.

Resharper's TestRunner has issues with this though... it does run the tests, but it doesn't report any feedback on them. The normal NUnit testrunner does show the feedback correctly though.

First Experiences With RSpec/BDD

2 commentsWritten on August 27th, 2010 by
Categories: Ruby, testing

I wanted to write some tests for the EventPublisher Ruby module i've been playing around with, so i figured i'd just use RSpec for it since that appears to be the most popular testing library in the Ruby world. Now, in the .NET world i never really got into the whole BDD thing and i stuck with TDD because i was quite happy with the coverage that it gave me. In Ruby however, due to the whole dynamic environment i think it's more important to test functionality as completely as possible with as little knowledge as possible of implementation details while mocking/stubbing/faking as little as possible. That doesn't mean i wouldn't mock anything in Ruby tests... it just means that i would try to follow my own rules on the subject as much as possible, whereas in the .NET world many of us (myself included) probably go a little overboard with the whole mocking/stubbing/faking thing.

Something to keep in mind for the rest of this post: i did not write my tests first for this thing. I know, i know, test-first is better. I generally prefer to write my tests before my real code as well, but in this case, the EventPublisher code was the result of just some first time Ruby experiments, and since i'm pretty happy with the code i don't want to get rid of it just so i could do it "right" by re-writing it test-first. So these tests were not meant to drive the design, only to verify the correctness of the code. Also note that the tests are not complete yet. More should be added, but i thought i had enough to post here and hopefully collect some feedback from you guys/gals.

When i started with these tests for the EventPublisher module, i instinctively wanted to test on a too technical level, like i often do in .NET. For instance, i wrote a test that proved that when you called the subscribe method, that the passed in method was actually added to the Event instance that the EventPublisher uses. The thing is: if you use the EventPublisher, you never directly use Event instances. So why on earth should i even know about them in my tests, right? After all, they are an implementation detail. I had to switch my reasoning from "is the code doing what i, a software developer, think it should do?" to something along the lines of "what needs to happen when i trigger an event?". For instance, if i trigger an event, all i should care about is that the subscribed methods are called correctly and that they receive their arguments correctly. How that actually happens is something that i probably shouldn't care about at all in these tests.

I eventually ended up with the following:

class Publisher
    include EventPublisher
    event :my_first_event
    event :my_second_event
    
    def trigger_first_event(args)
        trigger :my_first_event, args
    end
    
    def trigger_second_event(arg1, arg2)
        trigger :my_second_event, arg1, arg2
    end
end

describe EventPublisher, ": triggering event" do
    before(:each) do
      @publisher = Publisher.new
    end

    it "should not fail without any subscribers" do
      @publisher.trigger_first_event "testing"
    end

    it "should pass single event arg correctly to subscribed method with one argument" do
        @args = nil
      def my_first_event_handler(args);
            @args = args
        end

        @publisher.subscribe :my_first_event, method(:my_first_event_handler)
        @publisher.trigger_first_event "testing!"
        @args.should == "testing!"
    end
    
    it "should pass multiple event args correctly to subscribed method with multiple arguments" do
        @args2_1, @args2_2 = nil, nil
        def my_second_event_handler(arg1, arg2)
            @args2_1, @args2_2 = arg1, arg2
        end
        
        @publisher.subscribe :my_second_event, method(:my_second_event_handler)
        @publisher.trigger_second_event "second", "event"
        @args2_1.should == "second"
        @args2_2.should == "event"
    end

    it "should pass single event arg correctly to subscribed block with one argument" do
        event_args = nil
        @publisher.subscribe(:my_first_event) { |args| event_args = args }
        @publisher.trigger_first_event "test"
      event_args.should == "test"
    end
    
    it "should pass multiple event args correctly to subscribed block with two arguments" do
      first_arg, second_arg = nil, nil
        @publisher.subscribe(:my_second_event) { |arg1,arg2| first_arg, second_arg = arg1, arg2 }
        @publisher.trigger_second_event "first", "second"
        first_arg.should == "first"
        second_arg.should == "second"
    end
    
    it "should call subscribed method once for each time it was subscribed" do
      @counter1 = 0
        def my_first_event_handler(args)
            @counter1 += 1
        end
        
        2.times { @publisher.subscribe :my_first_event, method(:my_first_event_handler) }
        @publisher.trigger_first_event "test"
        @counter1.should == 2
    end
    
    it "should call all subscribed methods" do
        @counter1, @counter2 = 0, 0
        def handler1(args)
            @counter1 += 1
        end
        
        def handler2(args)
            @counter2 += 1
        end
        
        @publisher.subscribe :my_first_event, method(:handler1)
        @publisher.subscribe :my_first_event, method(:handler2)
        @publisher.trigger_first_event "first_event"
        @counter1.should == 1
        @counter2.should == 1
    end
    
end

There are a couple of things i like about this. For starters, the output of running this code looks like this:

EventPublisher: triggering event should not fail without any subscribers should pass single event arg correctly to subscribed method with one argument should pass multiple event args correctly to subscribed method with multiple arguments should pass single event arg correctly to subscribed block with one argument should pass multiple event args correctly to subscribed block with two arguments should call subscribed method once for each time it was subscribed should call all subscribed methods

Anyone can read that and understand what kind of functionality is supported.

Another big benefit of these tests is that they contain zero knowledge of the actual implementation of the EventPublisher module. They merely initiate its functionality, and verify whether the expected behavior in the given functional context occurred. I could seriously refactor (or even rewrite) the actual EventPublisher code and i wouldn't have to change my tests as long as i don't change the name and arguments of the subscribe and trigger methods.

For now, i'm pretty happy with this style and organization of tests and will probably stick with it for a while in my Ruby coding. Unless one (or some) of you tell me how i can improve it :)

The Downside Of Providing An API Through Extension Methods

9 commentsWritten on August 26th, 2010 by
Categories: Code Quality, testing

I recently used the excellent Moq mocking library for the first time, and i noticed a difference between Moq and Rhino Mocks (what i usually use) that i found interesting.

Consider the following useless and contrived example code:

    public interface ISomeComponent
    {
        void DoSomething();
    }
 
    public class SomeClass
    {
        private ISomeComponent someComponent;
 
        public SomeClass(ISomeComponent someComponent)
        {
            this.someComponent = someComponent;
        }
 
        public void DoSomethingReallyImportant()
        {
            someComponent.DoSomething();
        }
    } 

Now suppose that we want to verify in a test that the DoSomethingReallyImportant method of SomeClass actually calls the DoSomething method of its ISomeComponent dependency.

With Moq, we could do that like this:

    [TestFixture]
    public class TestWithMoq
    {

        [Test]
        public void CallsDoSomethingOnSomeComponent()
        {
            var mock = new Mock<ISomeComponent>();
            var someObject = new SomeClass(mock.Object);
 
            someObject.DoSomethingReallyImportant();
 
            mock.Verify(m => m.DoSomething());
        }
    }

And with Rhino Mocks, it would look like this:

    [TestFixture]
    public class TestWithMoq
    {
        [Test]
        public void CallsDoSomethingOnSomeComponent()
        {
            var mock = new Mock<ISomeComponent>();
            var someObject = new SomeClass(mock.Object);
 
            someObject.DoSomethingReallyImportant();
 
            mock.Verify(m => m.DoSomething());
        }
    }

Not much of a difference, right? Except that Rhino Mocks provides you with a proxy that implements the ISomeComponent interface and Moq provides you with a generic Mock object, which contains a proxy that implements the ISomeComponent interface and is exposed through the Object property. Other than that, the tests are very similar.

The key difference is what you experience when you write the tests, as the 2 pictures below will illustrate:

Since Moq's API is not fully based on Extension Methods, you get a normal and clean IntelliSense experience. Rhino Mocks on the other hand provides its API (at least the non-legacy stuff) solely through extension methods, which leads to all of them being included in your IntelliSense, even when they don't make any sense at all.It's obviously not a major issue, but i was suprised with how much i liked not seeing all of the extension methods all the time while writing tests.