I wanted to write some tests for the EventPublisher Ruby module i’ve been playing around with, so i figured i’d just use RSpec for it since that appears to be the most popular testing library in the Ruby world. Now, in the .NET world i never really got into the whole BDD thing and i stuck with TDD because i was quite happy with the coverage that it gave me. In Ruby however, due to the whole dynamic environment i think it’s more important to test functionality as completely as possible with as little knowledge as possible of implementation details while mocking/stubbing/faking as little as possible. That doesn’t mean i wouldn’t mock anything in Ruby tests… it just means that i would try to follow my own rules on the subject as much as possible, whereas in the .NET world many of us (myself included) probably go a little overboard with the whole mocking/stubbing/faking thing.

Something to keep in mind for the rest of this post: i did not write my tests first for this thing. I know, i know, test-first is better. I generally prefer to write my tests before my real code as well, but in this case, the EventPublisher code was the result of just some first time Ruby experiments, and since i’m pretty happy with the code i don’t want to get rid of it just so i could do it “right” by re-writing it test-first. So these tests were not meant to drive the design, only to verify the correctness of the code. Also note that the tests are not complete yet. More should be added, but i thought i had enough to post here and hopefully collect some feedback from you guys/gals.

When i started with these tests for the EventPublisher module, i instinctively wanted to test on a too technical level, like i often do in .NET. For instance, i wrote a test that proved that when you called the subscribe method, that the passed in method was actually added to the Event instance that the EventPublisher uses. The thing is: if you use the EventPublisher, you never directly use Event instances. So why on earth should i even know about them in my tests, right? After all, they are an implementation detail. I had to switch my reasoning from “is the code doing what i, a software developer, think it should do?” to something along the lines of “what needs to happen when i trigger an event?”. For instance, if i trigger an event, all i should care about is that the subscribed methods are called correctly and that they receive their arguments correctly. How that actually happens is something that i probably shouldn’t care about at all in these tests.

I eventually ended up with the following:

class Publisher
	include EventPublisher
	event :my_first_event
	event :my_second_event

	def trigger_first_event(args)
		trigger :my_first_event, args
	end

	def trigger_second_event(arg1, arg2)
		trigger :my_second_event, arg1, arg2
	end
end

describe EventPublisher, ": triggering event" do
	before(:each) do
	  @publisher = Publisher.new
	end

	it "should not fail without any subscribers" do
	  @publisher.trigger_first_event "testing"
	end

	it "should pass single event arg correctly to subscribed method with one argument" do
		@args = nil
	  def my_first_event_handler(args);
			@args = args
		end

		@publisher.subscribe :my_first_event, method(:my_first_event_handler)
		@publisher.trigger_first_event "testing!"
		@args.should == "testing!"
	end

	it "should pass multiple event args correctly to subscribed method with multiple arguments" do
		@args2_1, @args2_2 = nil, nil
		def my_second_event_handler(arg1, arg2)
			@args2_1, @args2_2 = arg1, arg2
		end

		@publisher.subscribe :my_second_event, method(:my_second_event_handler)
		@publisher.trigger_second_event "second", "event"
		@args2_1.should == "second"
		@args2_2.should == "event"
	end

	it "should pass single event arg correctly to subscribed block with one argument" do
		event_args = nil
		@publisher.subscribe(:my_first_event) { |args| event_args = args }
		@publisher.trigger_first_event "test"
	  event_args.should == "test"
	end

	it "should pass multiple event args correctly to subscribed block with two arguments" do
	  first_arg, second_arg = nil, nil
		@publisher.subscribe(:my_second_event) { |arg1,arg2| first_arg, second_arg = arg1, arg2 }
		@publisher.trigger_second_event "first", "second"
		first_arg.should == "first"
		second_arg.should == "second"
	end

	it "should call subscribed method once for each time it was subscribed" do
	  @counter1 = 0
		def my_first_event_handler(args)
			@counter1 += 1
		end

		2.times { @publisher.subscribe :my_first_event, method(:my_first_event_handler) }
		@publisher.trigger_first_event "test"
		@counter1.should == 2
	end

	it "should call all subscribed methods" do
		@counter1, @counter2 = 0, 0
		def handler1(args)
			@counter1 += 1
		end

		def handler2(args)
			@counter2 += 1
		end

		@publisher.subscribe :my_first_event, method(:handler1)
		@publisher.subscribe :my_first_event, method(:handler2)
		@publisher.trigger_first_event "first_event"
		@counter1.should == 1
		@counter2.should == 1
	end

end

There are a couple of things i like about this. For starters, the output of running this code looks like this:

EventPublisher: triggering event
should not fail without any subscribers
should pass single event arg correctly to subscribed method with one argument
should pass multiple event args correctly to subscribed method with multiple arguments
should pass single event arg correctly to subscribed block with one argument
should pass multiple event args correctly to subscribed block with two arguments
should call subscribed method once for each time it was subscribed
should call all subscribed methods

Anyone can read that and understand what kind of functionality is supported.

Another big benefit of these tests is that they contain zero knowledge of the actual implementation of the EventPublisher module. They merely initiate its functionality, and verify whether the expected behavior in the given functional context occurred. I could seriously refactor (or even rewrite) the actual EventPublisher code and i wouldn’t have to change my tests as long as i don’t change the name and arguments of the subscribe and trigger methods.

For now, i’m pretty happy with this style and organization of tests and will probably stick with it for a while in my Ruby coding. Unless one (or some) of you tell me how i can improve it

I recently used the excellent Moq mocking library for the first time, and i noticed a difference between Moq and Rhino Mocks (what i usually use) that i found interesting.

Consider the following useless and contrived example code:

Now suppose that we want to verify in a test that the DoSomethingReallyImportant method of SomeClass actually calls the DoSomething method of its ISomeComponent dependency.

With Moq, we could do that like this:

And with Rhino Mocks, it would look like this:

Not much of a difference, right? Except that Rhino Mocks provides you with a proxy that implements the ISomeComponent interface and Moq provides you with a generic Mock object, which contains a proxy that implements the ISomeComponent interface and is exposed through the Object property. Other than that, the tests are very similar.

The key difference is what you experience when you write the tests, as the 2 pictures below will illustrate:

Since Moq’s API is not fully based on Extension Methods, you get a normal and clean IntelliSense experience. Rhino Mocks on the other hand provides its API (at least the non-legacy stuff) solely through extension methods, which leads to all of them being included in your IntelliSense, even when they don’t make any sense at all.

It’s obviously not a major issue, but i was suprised with how much i liked not seeing all of the extension methods all the time while writing tests.

One of several interesting things in Mark Meyers’ QuickNet project is the whole input generation thing that you need for property-based testing.  It turns out that those input generators are very usable for far more purposes than just property-based testing, so it’s evolved into its own library.  It can generate object instances of almost any kind, while you can still have fine-grained control over the generation if you want to.  You can use it for simple types, complex objects or even entire object graphs. I wish i had time to write a more in-depth post about this, but for now i’m just gonna point you guys in the right direction, and i hope that you’ll see the value in this

The announcement of the first release can be found here, and an example can be found here.  Here’s a little glimpse at the code of one the examples:

I think that piece of code is a nice illustration of how powerful and flexible this is

A couple of months ago, i showed a sneak preview of a Silverlight unit testing Resharper plugin that one of my coworkers was working on.  I’m glad to say it’s finally been released and you can get it here

Those are Silverlight tests, executed through Resharper within the Silverlight runtime

In this post i’m going to give you a very detailed explanation of a QuickNet test that i wrote for Agatha’s caching layer.  If i do my job well, you’ll have a much better view on what QuickNet does, how it works, and how it can help you.  I do want to ask you to keep a very open mind and to forget pretty much everything that you know about automated testing, including your opinions on what should or should not be done in an automated test.  If you have problems with that, you might be interested in more conservative reading on automated testing.  Still here? Alright, let’s get started.  Please stay focused throughout, because you will need it :p

First of all, requests whose response is eligible for caching must override the Equals method and the GetHashCode method.  I’ll use the following two request/response combinations for this test:

Agatha’s Request Processor should cache responses for requests that are eligible for caching, and when subsequent requests need to be processed which are equal (in their values) to previous requests whose response has been cached already, the cached response needs to be returned instead of handling the request again.

In order to inspect the usage of the caching layer during tests, i wrote the following CacheManagerSpy class which inherits from my normal CacheManager class (which is used by Agatha):

This Spy basically gives me the ability to see which entries are stored in the cache, and which responses have been returned.

Now i can create a QuickNet Acid test.  I’m going to go over the code step by step instead of listing it all at once because each step probably needs quite a bit of explanation if you don’t understand QuickNet yet.

First of all, here’s the definition of the test:

The 10 means that QuickNet will perform 10 testruns.  The 2000 means that each testrun will contain 2000 executions of the defined transitions.  Don’t worry yet about what a transition is, i’ll get to that later on in the post.

We’ll also need some fields to access some of the classes that will be used in this test:

Yeah, i know what you’re thinking.  Static is bad right?  Well, not in this case since we need to be able to use these instances in some inner classes that we’ll go over soon enough and these references will hold the same instance for each testrun (which, as mentioned consists of 2000 transition executions).  Anyway, forget about the static keyword for a second and keep reading.

Now we can implement the SetUp of our testrun:

This setup code will be executed for each testrun.  So QuickNet will execute this 10 times, and each time it will run 2000 transition executions on the classes that we set up for the current testrun. 

So what exactly is a transition? It’s just a piece of code that you want to execute.  That piece of code can receive input and it can return output.  It’s basically the code that you’re testing.  Our transition will simply consist of calling Agatha’s Request Processor and instructing it to process 2 requests.  Those requests would be the two request types shown earlier which are eligible for caching.  After QuickNet executes the transition, it will also verify that our defined specifications for that particular transition are indeed correct.

Let’s define the input that our transition will receive.  In this particular case, we need a Request instance, an instance of a Request Handler for that Request, and the Response that the Request Handler needs to return if there’s no valid Response available in the cache.  So we’ll use the following helper class to define this input:

An instance of this class basically combines the Request, its Request Handler and the Response that needs to be returned.  It also contains a helper method so we can easily instruct the mocked Request Handler to return the given Response when it is asked to handle the given Request.  And we’ll also set a simple flag to true if the Request Handler has indeed been called.

As i mentioned a few times already, QuickNet will execute our transition (which i’ll show later on in the post) 2000 times for each testrun.  Obviously, we can’t be expected to create these instances manually all the time, so we need to define some generators that QuickNet will be able to use to randomly generate the input that it can pass to the transition.  We first need two generators which are capable of generating random instances of the FirstCachedRequest and SecondCachedRequest types:

The FirstRequestGenerator class is capable of returning a new instance of FirstCachedRequest whenever it is asked to do so, and it will populate the FirstCachedRequest instance’s String property with a random string consisting of one character.   Some of the generated instances will be considered equals of each other if the generated string value is equal, and some will not be considered as equals if the generated string value is not equal.  The SecondRequestGenerator basically does the same thing, except that it will populate the SecondCachedRequest instances’ Integer property with a random int value between 0 and 5.  So again, some of the generated instances will be considered as equals, and some won’t.   The important part to remember is that QuickNet will generate random instances for us and that we don’t have to worry about it.

Now that we have the generators for the request instances, we can write a generator which can generate random ProcessInputElement instances:

Actually, this generator doesn’t just generate one instance of ProcessInputElement, it generates a tuple consisting of two ProcessInputElement instances.  The first value in the tuple will always contain a random instance of FirstCachedRequest, its Request Handler and a FirstCachedResponse instance that needs to be returned by the Request Handler if there’s no cached response yet.  The same goes for the second value in the tuple, except that it has a random instance of SecondCachedRequest, its Request Handler and a SecondCachedResponse instance.

Again, every time QuickNet will execute our transition, it will ask the ProcessInputTupleGenerator to generate a new tuple, and that tuple will contain the random request instances.  The generated tuple is then passed into the transition and the transition will be executed.

Now i can finally show you the transition that we’ll define:

As you can see, the ProcessRequestsTransition inherits from the MetaTransition<TInput, TOutput> class.  So we basically just defined that the input of the transition is a Tuple<ProcessInputElement, ProcessInputElement> and the output is an array of Response objects.  In the constructor of the transition, we define that the generator to be used to generate the input values is the ProcessInputTupleGenerator.  And we also define what actually needs to be executed by the transition.  In this case, the piece of code that will always be executed will first clear the CacheManagerSpy instance (so we can safely inspect it during our specification verification), it will then prepare both Request Handler mocks and then it simply calls the Process method of the Request Processor.  It’s output (a Response array) will be the output of the transition since that is our return value.

After all that work, we can finally define our specifications for this transition.  Let’s start with the first one:

The SpecFor attribute makes it possible to define which Transition this Spec belongs to.  The method has 2 parameters, which correspond with the input that will be passed into the transition, and the output that the transition returned.  It’s very important to realize that this method is executed before the transition is executed (yet after the input has been generated), but the block of code that we pass into the Spec instance will only be executed after the transition has been executed.  Actually, that block of code will only be executed if the Spec’s precondition and postcondition have been satisfied.  The precondition (which can be defined with the Spec class’ If method) will be evaluated before the transition is executed.  If it evaluates to false, the spec will be ignored for the current execution of the transition.  The postcondition (which can be defined with the Spec class’ IfAfter method) is evaulated after the transition is executed.  If it evaulates to false, the spec will be ignored.  If it evaluates to true, the spec will be verified using the input that was passed into the transition and the output that was returned from the transition.

For this particular spec, we define a postcondition that the spec only needs to be verified if the CacheManager returned one or more null values instead of cached responses.  If the CacheManager returns a null, it means that there was no cached response for the request it was given.  And that is when we need to verify the following things:

1. The request was handled by the Request Handler
2. The response that was returned by the Request Handler has been put in the cache
3. The response array that was returned by the Request Processor contains the expected response

We perform this check for both requests, or only one of them, depending on which one was not in the cache yet.  Remember that the transition will be executed 2000 times using the same request processor and the same cache manager.  Which means that cached responses from previous transition executions might still be in the cache.  Also keep in mind that one of the request types has a defined expiration of 1 second, and the other an expiration of 2 seconds which means that there will be plenty of transition executions where they are either both in the cache, only one of them, or none of them.  All of those cases are now covered with the code in this spec.

Here’s another spec:

Here we verify that when a cached response was returned by the CacheManager:

1. The Request Handler was not executed
2. The response array returned by the Request Processor contains the cached response

We again perform this check for both requests, or only one of them depending on which one had a cached response.

You might think that this was a lot of work, but was it really?  It’s about 160 lines of code, and i’m quite sure that the functionality that i’m testing is covered much more thoroughly than it would’ve been with 160 lines of code for classical unit tests.   In fact, i challenge each one of you to come up with a more thorough test for this in less than 160 lines of code.  Also, i can keep adding specs quite easily and those really don’t contain that much code.  But my functional coverage would increase dramatically, much more than it would with classical unit tests.

Granted, the learning curve is steep as i mentioned already in a previous post.  But once you start to get it, you become pretty productive with it and you’ll catch a lot more bugs before you deliver your software because of it.

Now, i tried hard to make everything here as clear as possible, so i’d love to hear from you whether or not:

1. you understood it
2. you like it
3. you think it’s way over the top
4. you want to start doing this too

I recently posted the first draft of the caching implementation of Agatha.  You might want to take another look at the piece of code that deals with processing requests in the Request Processor.  I thought it was alright.  And if only 10% of my readers actually read that code, then it means that about 200 people thought it was alright too because nobody mentioned a possible problem with that part.  Today i was adding some cached requests/responses to the RequestProcessor’s QuickNet exception handling test.  And lo and behold, 2 bugs showed up that i hadn’t anticipated.

Before i discuss the two bugs, i’d like to explain what the exception handling quicknet test does, and how it works.  It basically calls the request processing code a bunch of times, each time with a set of requests where either none, one or more of the request handlers will throw an exception during the handling of the requests.  The order of the exceptions or which handlers will throw an exception is completely randomized (thanks to QuickNet) and thus it varies from test to test.  This particular QuickNet test will perform 50 testruns, where each testrun consists of 50 transitions that will be executed.  Each transition is a piece of functionality that will be executed (in this case, the processing of requests) with randomized input (in this case, there is a fixed set of requests but which ones will fail will differ with each run of the transition) and after a transition is executed, the relevant specs are verified (in this case, a bunch of checks to make sure that the whole error handling functionality always does what it needs to do) to make sure that the piece of code that you’re testing (your transition, in this case the processing of requests) actually works.

Instead of testing with a mocked caching layer, i used the real thing in the tests.  I also added 2 cached request types to the test request types, and gave one of them an expiration of 1 second, the other an expiration of 2 seconds.  This means that during the entire QuickNet testrun, some requests will return cached responses, some won’t, some cached responses will expire and subsequent requests of that particular type need to be executed and have their responses cached again until they expire.   In the meantime, QuickNet is constantly firing requests at the RequestProcessor, and some of them will randomly fail.  Our specs need to verify that the returned responses always contain the correct exception information for the batch of requests that was processed.

And here’s the beauty of the whole thing.  I hadn’t even thought about how exception handling also influences how you deal with cached responses.  I didn’t take exception handling into account when adding the caching code, and without even adding specific tests to see whether the caching worked correctly, my QuickNet specs that intended to cover something that doesn’t really have anything to do with caching uncovered 2 issues with my caching code.   I was suddenly getting error messages from QuickNet that in some cases, the exception information wasn’t correct.   And it frequently took over 30 executions of the transition, or sometimes multiple testruns before some of those specs would fail.  The longer it takes for a bug to show up in a QuickNet test, the better the bug is at hiding and the harder it is to find it manually because it almost certainly is an edge-case that you hadn’t considered at all and that only occurs in certain situations.

And those are the type of bugs that you typically don’t find with traditional unit tests.  When you write traditional tests that are aimed at asserting correct behavior, you typically only write tests for the problems that you thought about.  Sometimes you’ll get lucky and traditional unit tests will inform you of incorrect behavior that you didn’t think of,  but it rarely happens for the edge-cases.  And those are the bugs that can be really hard and painful to find and fix.

By now you’re probably wondering what the two bugs are.  The first one is that if an exception occurs during the handling of a request which is eligible for caching, the response with the exception information would be stored in the cache, and the next time that that particular request would be processed while the cached response hadn’t expired yet, you’d simply get the original response with the exception information again instead of actually handling the request again (which might have been handled successfully this time).  The second bug was that if an earlier request in the batch of requests already failed, it could possibly return a valid response later on in the same batch for one of the cached requests if its response hadn’t expired yet.   Pretty stupid huh?  But they were there nevertheless, and sooner or later somebody would’ve tripped over it and it would’ve caused frustration for the user who would run into it, and pain for me because i’d have to go looking for it long after i’d written the code.

Again, i seriously doubt that traditional unit tests would’ve prevented the existence of these 2 problems since it never occurred to me while i was writing it.  The QuickNet tests did find it, even before i added specific tests for the caching functionality to them, which i found pretty impressive and a huge benefit to this sort of testing.  Granted, the learning curve for writing valuable QuickNet tests is steep (i’m not even halfway there IMO) and it takes a lot of effort when you’re starting out with it.  But if it helps me prevent the sort of bugs that are otherwise easily missed and could negatively influence the perception of my project, then i definitely consider it worth it.

This is something that Steven De Kock and Tom Ceulemans (two coworkers of mine without a blog, so no link) have been working on.  Those are Silverlight tests, executing in the Silverlight runtime, within a full browser context.

And yes, they will release it as an open source project soon

I was asked to show how you can easily do CRUD tests, so here’s a base class that makes it very easy

Simply inherit from this class, implement the BuildEntity, ModifyEntity, AssertAreEqual and AssertValidId methods and that’s it.  Those methods are usually pretty simple.  In BuildEntity you just create an unpersisted entity and assign values to the properties, in ModifyEntity you modify the properties, and in AssertAreEqual you compare the properties of both instances.  In AssertValidId, you make sure that the ID value is ok (depending on your identifier strategy).

This is good for regular CRUD operations, though we typically add extra tests when we want to test cascades or one-to-many associations mapped with inverse=”true”.

Grant Palin recently asked me for an in-depth article on TDD’ing an NHibernate application. While this post won’t be very in-depth, it might be helpful already.  There are basically two approaches that i’ve seen used with good results, though there are obviously more approaches that you can use.  I’m going to limit the scope of this post to the following two approaches though, and i’ll also discuss exactly what we test.

Creating a new database for each test (or testfixture)

This approach creates the database at the beginning of each test (or testfixture), runs the test (or tests in the fixture), and then destroys the database after the test (or testfixture) completed.  The easiest way to do this is to create a base test class that all of you data access test class should inherit from.  Here’s a simple example:

This class will create the database from scratch once for each TestFixture, which means that each test in the fixture will use the same database.  It also destroys the database at the end of the fixture.  It creates the database based on your mappings, and as you can see, you really don’t have to do a lot for this.  If you want the database to be recreated and dropped for each test, then you obviously need to move the code in the TestFixtureSetup and TestFixtureTearDown methods to your regular SetUp and TearDown methods.  If you go that route, i’d advise you to include empty template methods before and after the setup and teardown so you can plug in some extra code before and after these operations in your derived test classes.

The biggest benefit of this approach is that you don’t have any possibly present state in the database that can influence your tests.  The downside is that you can’t rely on certain data (eg reference data) to be present and you have to recreate it whenever you need it.   You can also use multiple transactions in your tests, though you are also responsible for cleaning any data that is left in the database at the end of each test.  You also need to guarantee that this always happens because any data that is left by one test might influence another one.   Also, if you leave data in the database, that might lead to problems when dropping the database, so you really need to be careful with this.

Another problem with this example is that there is no automatic way to push the ISession instance to your data access components.  That could easily be added though, depending on how your data access components retrieve a reference to a valid ISession.

Tests that automatically roll back their transactions

This is the approach that we always use at work.  These tests require your database to be set up in a valid manner before your tests begin.  Each test uses one transaction, which is automatically rolled back at the end of the test to prevent the possibility of any test data remaining in the database.   With this way of testing, it’s also possible to provide some kind of ‘known state’ in the database (eg reference data) that you can use from within your code.

Here’s the NHibernateTest class that our NHibernate test classes all inherit from:

Now, this example uses our UnitOfWork and ActiveSessionManager classes (read this if you haven’t seen those yet) to make sure that our data access components can access the current ISession instance.  Each test has a valid ISession present, which has already created a transaction and we can create/modify/delete data, run our queries, modify some stuff again, run our queries again and perform our assertions, all in the same transaction.  After the test is completed, the transaction is never committed (and thus, automatically rolled back) so none of that data ever remains in the database.

What Exactly Do We Test?

Well, we test everything basically.  We test all of our CRUD operations (again, with a simple base class which only requires you to implement the BuildEntity, ModifyEntity and AssertEqual methods and does all of the operations and checks automatically) for each entity.  That’s right, for each entity.  The extra work that this requires really doesn’t take a lot of time and it lets us know for a fact that our mappings are valid.

We also test every custom query that we write, always using the following approach: create some test data, flush the session, perform your query and verify that the expected data has indeed been returned by the query, and also that data that shouldn’t have been returned isn’t there.

And that’s pretty much it.  We mock the data layer in all of our other tests, so our CRUD and query tests are the only ones that actually use NHibernate and the database.  But CRUD actions and queries are tested very extensively.

Mark just made the announcement

I hope some of you will give this a shot and try it out, i know i will