In this post i’m going to give you a very detailed explanation of a QuickNet test that i wrote for Agatha’s caching layer.  If i do my job well, you’ll have a much better view on what QuickNet does, how it works, and how it can help you.  I do want to ask you to keep a very open mind and to forget pretty much everything that you know about automated testing, including your opinions on what should or should not be done in an automated test.  If you have problems with that, you might be interested in more conservative reading on automated testing.  Still here? Alright, let’s get started.  Please stay focused throughout, because you will need it :p

First of all, requests whose response is eligible for caching must override the Equals method and the GetHashCode method.  I’ll use the following two request/response combinations for this test:

Agatha’s Request Processor should cache responses for requests that are eligible for caching, and when subsequent requests need to be processed which are equal (in their values) to previous requests whose response has been cached already, the cached response needs to be returned instead of handling the request again.

In order to inspect the usage of the caching layer during tests, i wrote the following CacheManagerSpy class which inherits from my normal CacheManager class (which is used by Agatha):

This Spy basically gives me the ability to see which entries are stored in the cache, and which responses have been returned.

Now i can create a QuickNet Acid test.  I’m going to go over the code step by step instead of listing it all at once because each step probably needs quite a bit of explanation if you don’t understand QuickNet yet.

First of all, here’s the definition of the test:

The 10 means that QuickNet will perform 10 testruns.  The 2000 means that each testrun will contain 2000 executions of the defined transitions.  Don’t worry yet about what a transition is, i’ll get to that later on in the post.

We’ll also need some fields to access some of the classes that will be used in this test:

Yeah, i know what you’re thinking.  Static is bad right?  Well, not in this case since we need to be able to use these instances in some inner classes that we’ll go over soon enough and these references will hold the same instance for each testrun (which, as mentioned consists of 2000 transition executions).  Anyway, forget about the static keyword for a second and keep reading.

Now we can implement the SetUp of our testrun:

This setup code will be executed for each testrun.  So QuickNet will execute this 10 times, and each time it will run 2000 transition executions on the classes that we set up for the current testrun. 

So what exactly is a transition? It’s just a piece of code that you want to execute.  That piece of code can receive input and it can return output.  It’s basically the code that you’re testing.  Our transition will simply consist of calling Agatha’s Request Processor and instructing it to process 2 requests.  Those requests would be the two request types shown earlier which are eligible for caching.  After QuickNet executes the transition, it will also verify that our defined specifications for that particular transition are indeed correct.

Let’s define the input that our transition will receive.  In this particular case, we need a Request instance, an instance of a Request Handler for that Request, and the Response that the Request Handler needs to return if there’s no valid Response available in the cache.  So we’ll use the following helper class to define this input:

An instance of this class basically combines the Request, its Request Handler and the Response that needs to be returned.  It also contains a helper method so we can easily instruct the mocked Request Handler to return the given Response when it is asked to handle the given Request.  And we’ll also set a simple flag to true if the Request Handler has indeed been called.

As i mentioned a few times already, QuickNet will execute our transition (which i’ll show later on in the post) 2000 times for each testrun.  Obviously, we can’t be expected to create these instances manually all the time, so we need to define some generators that QuickNet will be able to use to randomly generate the input that it can pass to the transition.  We first need two generators which are capable of generating random instances of the FirstCachedRequest and SecondCachedRequest types:

The FirstRequestGenerator class is capable of returning a new instance of FirstCachedRequest whenever it is asked to do so, and it will populate the FirstCachedRequest instance’s String property with a random string consisting of one character.   Some of the generated instances will be considered equals of each other if the generated string value is equal, and some will not be considered as equals if the generated string value is not equal.  The SecondRequestGenerator basically does the same thing, except that it will populate the SecondCachedRequest instances’ Integer property with a random int value between 0 and 5.  So again, some of the generated instances will be considered as equals, and some won’t.   The important part to remember is that QuickNet will generate random instances for us and that we don’t have to worry about it.

Now that we have the generators for the request instances, we can write a generator which can generate random ProcessInputElement instances:

Actually, this generator doesn’t just generate one instance of ProcessInputElement, it generates a tuple consisting of two ProcessInputElement instances.  The first value in the tuple will always contain a random instance of FirstCachedRequest, its Request Handler and a FirstCachedResponse instance that needs to be returned by the Request Handler if there’s no cached response yet.  The same goes for the second value in the tuple, except that it has a random instance of SecondCachedRequest, its Request Handler and a SecondCachedResponse instance.

Again, every time QuickNet will execute our transition, it will ask the ProcessInputTupleGenerator to generate a new tuple, and that tuple will contain the random request instances.  The generated tuple is then passed into the transition and the transition will be executed.

Now i can finally show you the transition that we’ll define:

As you can see, the ProcessRequestsTransition inherits from the MetaTransition<TInput, TOutput> class.  So we basically just defined that the input of the transition is a Tuple<ProcessInputElement, ProcessInputElement> and the output is an array of Response objects.  In the constructor of the transition, we define that the generator to be used to generate the input values is the ProcessInputTupleGenerator.  And we also define what actually needs to be executed by the transition.  In this case, the piece of code that will always be executed will first clear the CacheManagerSpy instance (so we can safely inspect it during our specification verification), it will then prepare both Request Handler mocks and then it simply calls the Process method of the Request Processor.  It’s output (a Response array) will be the output of the transition since that is our return value.

After all that work, we can finally define our specifications for this transition.  Let’s start with the first one:

The SpecFor attribute makes it possible to define which Transition this Spec belongs to.  The method has 2 parameters, which correspond with the input that will be passed into the transition, and the output that the transition returned.  It’s very important to realize that this method is executed before the transition is executed (yet after the input has been generated), but the block of code that we pass into the Spec instance will only be executed after the transition has been executed.  Actually, that block of code will only be executed if the Spec’s precondition and postcondition have been satisfied.  The precondition (which can be defined with the Spec class’ If method) will be evaluated before the transition is executed.  If it evaluates to false, the spec will be ignored for the current execution of the transition.  The postcondition (which can be defined with the Spec class’ IfAfter method) is evaulated after the transition is executed.  If it evaulates to false, the spec will be ignored.  If it evaluates to true, the spec will be verified using the input that was passed into the transition and the output that was returned from the transition.

For this particular spec, we define a postcondition that the spec only needs to be verified if the CacheManager returned one or more null values instead of cached responses.  If the CacheManager returns a null, it means that there was no cached response for the request it was given.  And that is when we need to verify the following things:

1. The request was handled by the Request Handler
2. The response that was returned by the Request Handler has been put in the cache
3. The response array that was returned by the Request Processor contains the expected response

We perform this check for both requests, or only one of them, depending on which one was not in the cache yet.  Remember that the transition will be executed 2000 times using the same request processor and the same cache manager.  Which means that cached responses from previous transition executions might still be in the cache.  Also keep in mind that one of the request types has a defined expiration of 1 second, and the other an expiration of 2 seconds which means that there will be plenty of transition executions where they are either both in the cache, only one of them, or none of them.  All of those cases are now covered with the code in this spec.

Here’s another spec:

Here we verify that when a cached response was returned by the CacheManager:

1. The Request Handler was not executed
2. The response array returned by the Request Processor contains the cached response

We again perform this check for both requests, or only one of them depending on which one had a cached response.

You might think that this was a lot of work, but was it really?  It’s about 160 lines of code, and i’m quite sure that the functionality that i’m testing is covered much more thoroughly than it would’ve been with 160 lines of code for classical unit tests.   In fact, i challenge each one of you to come up with a more thorough test for this in less than 160 lines of code.  Also, i can keep adding specs quite easily and those really don’t contain that much code.  But my functional coverage would increase dramatically, much more than it would with classical unit tests.

Granted, the learning curve is steep as i mentioned already in a previous post.  But once you start to get it, you become pretty productive with it and you’ll catch a lot more bugs before you deliver your software because of it.

Now, i tried hard to make everything here as clear as possible, so i’d love to hear from you whether or not:

1. you understood it
2. you like it
3. you think it’s way over the top
4. you want to start doing this too

I recently posted the first draft of the caching implementation of Agatha.  You might want to take another look at the piece of code that deals with processing requests in the Request Processor.  I thought it was alright.  And if only 10% of my readers actually read that code, then it means that about 200 people thought it was alright too because nobody mentioned a possible problem with that part.  Today i was adding some cached requests/responses to the RequestProcessor’s QuickNet exception handling test.  And lo and behold, 2 bugs showed up that i hadn’t anticipated.

Before i discuss the two bugs, i’d like to explain what the exception handling quicknet test does, and how it works.  It basically calls the request processing code a bunch of times, each time with a set of requests where either none, one or more of the request handlers will throw an exception during the handling of the requests.  The order of the exceptions or which handlers will throw an exception is completely randomized (thanks to QuickNet) and thus it varies from test to test.  This particular QuickNet test will perform 50 testruns, where each testrun consists of 50 transitions that will be executed.  Each transition is a piece of functionality that will be executed (in this case, the processing of requests) with randomized input (in this case, there is a fixed set of requests but which ones will fail will differ with each run of the transition) and after a transition is executed, the relevant specs are verified (in this case, a bunch of checks to make sure that the whole error handling functionality always does what it needs to do) to make sure that the piece of code that you’re testing (your transition, in this case the processing of requests) actually works.

Instead of testing with a mocked caching layer, i used the real thing in the tests.  I also added 2 cached request types to the test request types, and gave one of them an expiration of 1 second, the other an expiration of 2 seconds.  This means that during the entire QuickNet testrun, some requests will return cached responses, some won’t, some cached responses will expire and subsequent requests of that particular type need to be executed and have their responses cached again until they expire.   In the meantime, QuickNet is constantly firing requests at the RequestProcessor, and some of them will randomly fail.  Our specs need to verify that the returned responses always contain the correct exception information for the batch of requests that was processed.

And here’s the beauty of the whole thing.  I hadn’t even thought about how exception handling also influences how you deal with cached responses.  I didn’t take exception handling into account when adding the caching code, and without even adding specific tests to see whether the caching worked correctly, my QuickNet specs that intended to cover something that doesn’t really have anything to do with caching uncovered 2 issues with my caching code.   I was suddenly getting error messages from QuickNet that in some cases, the exception information wasn’t correct.   And it frequently took over 30 executions of the transition, or sometimes multiple testruns before some of those specs would fail.  The longer it takes for a bug to show up in a QuickNet test, the better the bug is at hiding and the harder it is to find it manually because it almost certainly is an edge-case that you hadn’t considered at all and that only occurs in certain situations.

And those are the type of bugs that you typically don’t find with traditional unit tests.  When you write traditional tests that are aimed at asserting correct behavior, you typically only write tests for the problems that you thought about.  Sometimes you’ll get lucky and traditional unit tests will inform you of incorrect behavior that you didn’t think of,  but it rarely happens for the edge-cases.  And those are the bugs that can be really hard and painful to find and fix.

By now you’re probably wondering what the two bugs are.  The first one is that if an exception occurs during the handling of a request which is eligible for caching, the response with the exception information would be stored in the cache, and the next time that that particular request would be processed while the cached response hadn’t expired yet, you’d simply get the original response with the exception information again instead of actually handling the request again (which might have been handled successfully this time).  The second bug was that if an earlier request in the batch of requests already failed, it could possibly return a valid response later on in the same batch for one of the cached requests if its response hadn’t expired yet.   Pretty stupid huh?  But they were there nevertheless, and sooner or later somebody would’ve tripped over it and it would’ve caused frustration for the user who would run into it, and pain for me because i’d have to go looking for it long after i’d written the code.

Again, i seriously doubt that traditional unit tests would’ve prevented the existence of these 2 problems since it never occurred to me while i was writing it.  The QuickNet tests did find it, even before i added specific tests for the caching functionality to them, which i found pretty impressive and a huge benefit to this sort of testing.  Granted, the learning curve for writing valuable QuickNet tests is steep (i’m not even halfway there IMO) and it takes a lot of effort when you’re starting out with it.  But if it helps me prevent the sort of bugs that are otherwise easily missed and could negatively influence the perception of my project, then i definitely consider it worth it.

This is something that Steven De Kock and Tom Ceulemans (two coworkers of mine without a blog, so no link) have been working on.  Those are Silverlight tests, executing in the Silverlight runtime, within a full browser context.

And yes, they will release it as an open source project soon

I was asked to show how you can easily do CRUD tests, so here’s a base class that makes it very easy

Simply inherit from this class, implement the BuildEntity, ModifyEntity, AssertAreEqual and AssertValidId methods and that’s it.  Those methods are usually pretty simple.  In BuildEntity you just create an unpersisted entity and assign values to the properties, in ModifyEntity you modify the properties, and in AssertAreEqual you compare the properties of both instances.  In AssertValidId, you make sure that the ID value is ok (depending on your identifier strategy).

This is good for regular CRUD operations, though we typically add extra tests when we want to test cascades or one-to-many associations mapped with inverse=”true”.

Grant Palin recently asked me for an in-depth article on TDD’ing an NHibernate application. While this post won’t be very in-depth, it might be helpful already.  There are basically two approaches that i’ve seen used with good results, though there are obviously more approaches that you can use.  I’m going to limit the scope of this post to the following two approaches though, and i’ll also discuss exactly what we test.

Creating a new database for each test (or testfixture)

This approach creates the database at the beginning of each test (or testfixture), runs the test (or tests in the fixture), and then destroys the database after the test (or testfixture) completed.  The easiest way to do this is to create a base test class that all of you data access test class should inherit from.  Here’s a simple example:

This class will create the database from scratch once for each TestFixture, which means that each test in the fixture will use the same database.  It also destroys the database at the end of the fixture.  It creates the database based on your mappings, and as you can see, you really don’t have to do a lot for this.  If you want the database to be recreated and dropped for each test, then you obviously need to move the code in the TestFixtureSetup and TestFixtureTearDown methods to your regular SetUp and TearDown methods.  If you go that route, i’d advise you to include empty template methods before and after the setup and teardown so you can plug in some extra code before and after these operations in your derived test classes.

The biggest benefit of this approach is that you don’t have any possibly present state in the database that can influence your tests.  The downside is that you can’t rely on certain data (eg reference data) to be present and you have to recreate it whenever you need it.   You can also use multiple transactions in your tests, though you are also responsible for cleaning any data that is left in the database at the end of each test.  You also need to guarantee that this always happens because any data that is left by one test might influence another one.   Also, if you leave data in the database, that might lead to problems when dropping the database, so you really need to be careful with this.

Another problem with this example is that there is no automatic way to push the ISession instance to your data access components.  That could easily be added though, depending on how your data access components retrieve a reference to a valid ISession.

Tests that automatically roll back their transactions

This is the approach that we always use at work.  These tests require your database to be set up in a valid manner before your tests begin.  Each test uses one transaction, which is automatically rolled back at the end of the test to prevent the possibility of any test data remaining in the database.   With this way of testing, it’s also possible to provide some kind of ‘known state’ in the database (eg reference data) that you can use from within your code.

Here’s the NHibernateTest class that our NHibernate test classes all inherit from:

Now, this example uses our UnitOfWork and ActiveSessionManager classes (read this if you haven’t seen those yet) to make sure that our data access components can access the current ISession instance.  Each test has a valid ISession present, which has already created a transaction and we can create/modify/delete data, run our queries, modify some stuff again, run our queries again and perform our assertions, all in the same transaction.  After the test is completed, the transaction is never committed (and thus, automatically rolled back) so none of that data ever remains in the database.

What Exactly Do We Test?

Well, we test everything basically.  We test all of our CRUD operations (again, with a simple base class which only requires you to implement the BuildEntity, ModifyEntity and AssertEqual methods and does all of the operations and checks automatically) for each entity.  That’s right, for each entity.  The extra work that this requires really doesn’t take a lot of time and it lets us know for a fact that our mappings are valid.

We also test every custom query that we write, always using the following approach: create some test data, flush the session, perform your query and verify that the expected data has indeed been returned by the query, and also that data that shouldn’t have been returned isn’t there.

And that’s pretty much it.  We mock the data layer in all of our other tests, so our CRUD and query tests are the only ones that actually use NHibernate and the database.  But CRUD actions and queries are tested very extensively.

Mark just made the announcement

I hope some of you will give this a shot and try it out, i know i will

As most of you already know, i’m a proponent of unit testing and automated tests in general. There is however one inherent flaw that comes with the unit testing approach: if you didn’t think of a problem, you didn’t write a test for it, and thus, you’re not protected from that problem.

In the past two years we’ve seen the .NET world gradually accept the concept of TDD, and it has been extended somewhat with the whole BDD thing. However, there hasn’t been a single fundamental change which allows you to take away that inherent flaw. Every single variation that has been introduced in the TDD/BDD world is honestly very minor (dare i even say pointless?) because it all boils down to different syntax and slightly different ways of organizing your test fixtures. But again, even if you are doing the very latest flavor of the month when it comes to automated tests, you’re still not protected from problems that you didn’t think of.

Mark Meyers, a former coworker of mine, has been working hard on porting QuickCheck to .NET and the result is the QuickNet project. Instead of being yet another minor evolutionary step in the automated testing world for .NET, you could actually consider this to be a revolutionary improvement. For the .NET world that is, since other developer communities have adopted this new approach already some time ago.

I highly encourage you to check out what QuickNet is about. I’m about to get much more involved with the project, from a development point of view as well as using it and trying to spread the good word. This is something that actually can offer you substantial benefits over classical unit testing approaches so you really would be doing yourself a huge favor in giving this a shot. Keep in mind that it’s still pretty early and that the codebase is still going through some major changes. There is no official release yet, so breaking changes can (and probably will) occur if you already try it out.

There also isn’t any official documentation yet either, but Mark has some great posts about QuickNet’s usage and there’s also a couple of nice examples in the code already. In fact, we also have a few QuickNet tests to test… well… QuickNet itself. Head on over to some or all of the links i put in this post, and start learning

Just read a post by Jonathan Oliver where he talks about a solution he saw to tackle the problem of date-dependent code in tests. As you probably know, code that uses the current date can very easily cause testability problems. Code that accesses DateTime.Now or DateTime.UctNow can quite easily cause tests to fail for no valid reason when the tests happen to run on a certain date, on weekend days, at the end of the month, etc…

The problem obviously is that you can’t override the value that DateTime.Now will return during your tests. Many people seem to resort to using some kind of DateTimeService dependency which each piece of code that needs the current date will use. Basically, something like this:

The implementation that will be used at runtime then looks like this:

Code that needs the current date simply declares a dependency on IDateTimeService and at run-time the IOC container will inject an instance of DateTimeService to fulfill the IDateTimeService dependency. At test-time, the IDateTimeService is mocked so you can easily set which date should be returned for each test.

Personally, i’m not a fan of this approach. I mean, i use the same approach for most dependencies but for simply getting the current date this is a bit too much IMO.

Instead, we simply use something like this:

And our real code simply calls DateTimeProvider.Now instead of DateTime.Now. In our tests, we call the SetDateTimeToReturn method to provide the date that we want to use for a particular test. At the end of the test, simply call the ResetCurrentDateTime method and that’s it.

I generally don’t like static methods but in this case, this is a very simple solution to this specific problem. And you know how the saying goes: “do the simplest thing that could possibly work”. And as long as that works, resist the urge to change it.

When you hear people talk about automated testing, the subject of code coverage typically comes up as well. Code coverage is a metric which indicates how much of your production code is executed while your automated tests are running. A lot of people think that high code coverage is a good thing, because it means that a high percentage of their production code is being executed by their tests. That’s gotta be a good thing, right?

Well, not necessarily. You can very easily write bad tests which execute most, or even all of your production code, while the tests themselves probably don’t cover everything that really needs to be covered. Even if you’re a disciplined developer who is very careful not to make mistakes, the odds that your code will contain bugs that your tests didn’t catch are still relatively likely. For those of you who write automated tests (either test-first or test-after): how often has it happened to you that a bug was discovered later on for some edge case that you didn’t think of when writing the production code or the test code? No matter how good you are, this has happened to you on more than one occasion. And it will happen again in the future.

Here’s the important question though: how frequently does it happen that bugs are discovered in pieces of code which have high coverage during the test-runs? It might not happen every day, it might not happen every week but it definitely happens occasionally and in some cases even frequently. What exactly does that tell you about the value of a metric such as code coverage? You might think that if this situation comes up, you don’t have proper tests in place but that’s not necessarily the case. You can have an incredibly good test fixture (or several) for a certain piece of code but if you and/or your teammates didn’t happen to think of a certain edge-case or even something that might be obvious, then you didn’t write tests for that. Even though the build report might indicate 100% coverage for that piece of code.

I’ve always thought that code coverage is only meaningful when it’s low. Low code coverage can at least definitively tell you that certain parts of code are never executed during test-runs and are thus completely untested. However, it will never definitively tell you which parts of code (or how much of it) are guaranteed to be covered correctly by your tests. As such, i would say that code coverage can only be used as an indication that something is wrong, but never as an indication that something is good/right/correct/great.

I mentioned QuickNet before and i’m happy to say that the project is now available on Google Code. I’m also going to be contributing to QuickNet as of next week, so you can probably expect some future blog posts about this as well

Be sure to read Mark’s announcement about this if you’re interested in playing with QuickNet.