Testing Agatha’s Caching Functionality With QuickNet
Posted by Davy Brion on December 24th, 2009
In this post i’m going to give you a very detailed explanation of a QuickNet test that i wrote for Agatha’s caching layer. If i do my job well, you’ll have a much better view on what QuickNet does, how it works, and how it can help you. I do want to ask you to keep a very open mind and to forget pretty much everything that you know about automated testing, including your opinions on what should or should not be done in an automated test. If you have problems with that, you might be interested in more conservative reading on automated testing. Still here? Alright, let’s get started. Please stay focused throughout, because you will need it :p
First of all, requests whose response is eligible for caching must override the Equals method and the GetHashCode method. I’ll use the following two request/response combinations for this test:
[EnableResponseCaching(Seconds = 1)]
public class FirstCachedRequest : Request
{
public string String { get; set; }
public bool Equals(FirstCachedRequest other)
{
if (ReferenceEquals(null, other)) return false;
if (ReferenceEquals(this, other)) return true;
return Equals(other.String, String);
}
public override bool Equals(object obj)
{
if (ReferenceEquals(null, obj)) return false;
if (ReferenceEquals(this, obj)) return true;
if (obj.GetType() != typeof(FirstCachedRequest)) return false;
return Equals((FirstCachedRequest)obj);
}
public override int GetHashCode()
{
return (String != null ? String.GetHashCode() : 0);
}
}
public class FirstCachedResponse : Response { }
[EnableResponseCaching(Seconds = 2)]
public class SecondCachedRequest : Request
{
public int Integer { get; set; }
public bool Equals(SecondCachedRequest other)
{
if (ReferenceEquals(null, other)) return false;
if (ReferenceEquals(this, other)) return true;
return other.Integer == Integer;
}
public override bool Equals(object obj)
{
if (ReferenceEquals(null, obj)) return false;
if (ReferenceEquals(this, obj)) return true;
if (obj.GetType() != typeof(SecondCachedRequest)) return false;
return Equals((SecondCachedRequest)obj);
}
public override int GetHashCode()
{
return Integer;
}
}
public class SecondCachedResponse : Response {}
Agatha’s Request Processor should cache responses for requests that are eligible for caching, and when subsequent requests need to be processed which are equal (in their values) to previous requests whose response has been cached already, the cached response needs to be returned instead of handling the request again.
In order to inspect the usage of the caching layer during tests, i wrote the following CacheManagerSpy class which inherits from my normal CacheManager class (which is used by Agatha):
public class CacheManagerSpy : CacheManager
{
private List<CacheEntry> cacheEntries;
private List<Response> returnedCachedResponses;
public IEnumerable<CacheEntry> CacheEntries
{
get { return cacheEntries; }
}
public IEnumerable<Response> ReturnedCachedResponses
{
get { return returnedCachedResponses; }
}
public CacheManagerSpy(CacheConfiguration configuration, ICacheProvider cacheProvider) : base(configuration, cacheProvider)
{
Clear();
}
public void Clear()
{
cacheEntries = new List<CacheEntry>();
returnedCachedResponses = new List<Response>();
}
protected override Response GetCachedResponseFor(Request request, string region)
{
var cachedResponse = base.GetCachedResponseFor(request, region);
returnedCachedResponses.Add(cachedResponse);
return cachedResponse;
}
protected override void StoreInCache(Request request, Response response, TimeSpan expiration, string region)
{
cacheEntries.Add(new CacheEntry(request, response, expiration, region));
base.StoreInCache(request, response, expiration, region);
}
public class CacheEntry
{
public Request Request { get; private set; }
public Response Response { get; private set; }
public TimeSpan Expiration { get; private set; }
public string Region { get; private set; }
public CacheEntry(Request request, Response response, TimeSpan expiration, string region)
{
Request = request;
Response = response;
Expiration = expiration;
Region = region;
}
}
}
This Spy basically gives me the ability to see which entries are stored in the cache, and which responses have been returned.
Now i can create a QuickNet Acid test. I’m going to go over the code step by step instead of listing it all at once because each step probably needs quite a bit of explanation if you don’t understand QuickNet yet.
First of all, here’s the definition of the test:
public class RequestProcessorCachingSpecs : AcidTest
{
public RequestProcessorCachingSpecs() : base(10, 2000) { }
The 10 means that QuickNet will perform 10 testruns. The 2000 means that each testrun will contain 2000 executions of the defined transitions. Don’t worry yet about what a transition is, i’ll get to that later on in the post.
We’ll also need some fields to access some of the classes that will be used in this test:
protected static IRequestHandler<FirstCachedRequest> firstCachedRequestHandler;
protected static IRequestHandler<SecondCachedRequest> secondCachedRequestHandler;
protected static IRequestProcessor requestProcessor;
protected static CacheManagerSpy cacheManager;
Yeah, i know what you’re thinking. Static is bad right? Well, not in this case since we need to be able to use these instances in some inner classes that we’ll go over soon enough and these references will hold the same instance for each testrun (which, as mentioned consists of 2000 transition executions). Anyway, forget about the static keyword for a second and keep reading.
Now we can implement the SetUp of our testrun:
public override void SetUp()
{
IoC.Container = new Agatha.Castle.Container();
var serviceLayerConfiguration = new ServiceLayerConfiguration(Assembly.GetExecutingAssembly(),
Assembly.GetExecutingAssembly(), IoC.Container)
{
BusinessExceptionType = typeof(BusinessException),
SecurityExceptionType = typeof(SecurityException),
CacheManagerImplementation = typeof(CacheManagerSpy)
};
serviceLayerConfiguration.Initialize();
// i want to take advantage of the automatic initialization, so i’m just resolving the requestprocessor instead of creating it
requestProcessor = IoC.Container.Resolve<IRequestProcessor>();
// the cache manager is a singleton so i can just resolve it and it’ll be the same one the request processor uses
cacheManager = (CacheManagerSpy)IoC.Container.Resolve<ICacheManager>();
firstCachedRequestHandler = MockRepository.GenerateMock<IRequestHandler<FirstCachedRequest>>();
secondCachedRequestHandler = MockRepository.GenerateMock<IRequestHandler<SecondCachedRequest>>();
IoC.Container.RegisterInstance(firstCachedRequestHandler);
IoC.Container.RegisterInstance(secondCachedRequestHandler);
}
This setup code will be executed for each testrun. So QuickNet will execute this 10 times, and each time it will run 2000 transition executions on the classes that we set up for the current testrun.
So what exactly is a transition? It’s just a piece of code that you want to execute. That piece of code can receive input and it can return output. It’s basically the code that you’re testing. Our transition will simply consist of calling Agatha’s Request Processor and instructing it to process 2 requests. Those requests would be the two request types shown earlier which are eligible for caching. After QuickNet executes the transition, it will also verify that our defined specifications for that particular transition are indeed correct.
Let’s define the input that our transition will receive. In this particular case, we need a Request instance, an instance of a Request Handler for that Request, and the Response that the Request Handler needs to return if there’s no valid Response available in the cache. So we’ll use the following helper class to define this input:
public class ProcessInputElement
{
public IRequestHandler RequestHandler { get; private set; }
public Request Request { get; private set; }
public Response Response { get; private set; }
public bool RequestHandlerWasExecuted { get; private set; }
public ProcessInputElement(IRequestHandler requestHandler, Request request, Response response)
{
RequestHandler = requestHandler;
Request = request;
Response = response;
}
public void StubHandler()
{
RequestHandler
.Stub(r => r.Handle(Arg<Request>.Is.Same(Request))) // WHY: reference check iso equality check
.Return(Response)
.WhenCalled(a => RequestHandlerWasExecuted = true)
.Repeat.Once();
}
}
An instance of this class basically combines the Request, its Request Handler and the Response that needs to be returned. It also contains a helper method so we can easily instruct the mocked Request Handler to return the given Response when it is asked to handle the given Request. And we’ll also set a simple flag to true if the Request Handler has indeed been called.
As i mentioned a few times already, QuickNet will execute our transition (which i’ll show later on in the post) 2000 times for each testrun. Obviously, we can’t be expected to create these instances manually all the time, so we need to define some generators that QuickNet will be able to use to randomly generate the input that it can pass to the transition. We first need two generators which are capable of generating random instances of the FirstCachedRequest and SecondCachedRequest types:
public class FirstRequestGenerator : BaseGenerator<FirstCachedRequest>
{
public FirstRequestGenerator()
{
AddGeneratorForProperty(r => r.String, new StringGenerator(1, 1)); // generates a random string of 1 character
}
}
public class SecondRequestGenerator : BaseGenerator<SecondCachedRequest>
{
public SecondRequestGenerator()
{
AddGeneratorForProperty(r => r.Integer, new IntGenerator(0, 5));
}
}
The FirstRequestGenerator class is capable of returning a new instance of FirstCachedRequest whenever it is asked to do so, and it will populate the FirstCachedRequest instance’s String property with a random string consisting of one character. Some of the generated instances will be considered equals of each other if the generated string value is equal, and some will not be considered as equals if the generated string value is not equal. The SecondRequestGenerator basically does the same thing, except that it will populate the SecondCachedRequest instances’ Integer property with a random int value between 0 and 5. So again, some of the generated instances will be considered as equals, and some won’t. The important part to remember is that QuickNet will generate random instances for us and that we don’t have to worry about it.
Now that we have the generators for the request instances, we can write a generator which can generate random ProcessInputElement instances:
public class ProcessInputTupleGenerator : BaseGenerator<Tuple<ProcessInputElement, ProcessInputElement>>
{
private FirstRequestGenerator firstRequestGenerator = new FirstRequestGenerator();
private SecondRequestGenerator secondRequestGenerator = new SecondRequestGenerator();
protected override Tuple<ProcessInputElement, ProcessInputElement> GetDefaultInstance()
{
var firstElement = new ProcessInputElement(firstCachedRequestHandler, firstRequestGenerator.GetRandomValue(),
new FirstCachedResponse());
var secondElement = new ProcessInputElement(secondCachedRequestHandler, secondRequestGenerator.GetRandomValue(),
new SecondCachedResponse());
return Tuple.New(firstElement, secondElement);
}
}
Actually, this generator doesn’t just generate one instance of ProcessInputElement, it generates a tuple consisting of two ProcessInputElement instances. The first value in the tuple will always contain a random instance of FirstCachedRequest, its Request Handler and a FirstCachedResponse instance that needs to be returned by the Request Handler if there’s no cached response yet. The same goes for the second value in the tuple, except that it has a random instance of SecondCachedRequest, its Request Handler and a SecondCachedResponse instance.
Again, every time QuickNet will execute our transition, it will ask the ProcessInputTupleGenerator to generate a new tuple, and that tuple will contain the random request instances. The generated tuple is then passed into the transition and the transition will be executed.
Now i can finally show you the transition that we’ll define:
public class ProcessRequestsTransition : MetaTransition<Tuple<ProcessInputElement, ProcessInputElement>, Response[]>
{
public ProcessRequestsTransition()
{
Generator = new ProcessInputTupleGenerator();
Execute =
input =>
{
cacheManager.Clear(); // clears the spy for every execution of this transition
input.First.StubHandler();
input.Second.StubHandler();
return requestProcessor.Process(new[] {input.First.Request, input.Second.Request});
};
}
}
As you can see, the ProcessRequestsTransition inherits from the MetaTransition<TInput, TOutput> class. So we basically just defined that the input of the transition is a Tuple<ProcessInputElement, ProcessInputElement> and the output is an array of Response objects. In the constructor of the transition, we define that the generator to be used to generate the input values is the ProcessInputTupleGenerator. And we also define what actually needs to be executed by the transition. In this case, the piece of code that will always be executed will first clear the CacheManagerSpy instance (so we can safely inspect it during our specification verification), it will then prepare both Request Handler mocks and then it simply calls the Process method of the Request Processor. It’s output (a Response array) will be the output of the transition since that is our return value.
After all that work, we can finally define our specifications for this transition. Let’s start with the first one:
[SpecFor(typeof(ProcessRequestsTransition))]
public Spec ResponsesAreCachedIfTheyArentInTheCacheYet(Tuple<ProcessInputElement, ProcessInputElement> input, Response[] output)
{
return new Spec(() =>
{
Action<ProcessInputElement> verify =
element =>
{
Ensure.Equal(element.Response, cacheManager.CacheEntries.First(e => e.Request.Equals(element.Request)).Response);
Ensure.True(output.Contains(element.Response));
Ensure.True(element.RequestHandlerWasExecuted);
};
if (cacheManager.CacheEntries.Any(e => e.Request.Equals(input.First.Request)))
{
verify(input.First);
}
if (cacheManager.CacheEntries.Any(e => e.Request.Equals(input.Second.Request)))
{
verify(input.Second);
}
})
.IfAfter(() => cacheManager.ReturnedCachedResponses.Any(r => r == null));
}
The SpecFor attribute makes it possible to define which Transition this Spec belongs to. The method has 2 parameters, which correspond with the input that will be passed into the transition, and the output that the transition returned. It’s very important to realize that this method is executed before the transition is executed (yet after the input has been generated), but the block of code that we pass into the Spec instance will only be executed after the transition has been executed. Actually, that block of code will only be executed if the Spec’s precondition and postcondition have been satisfied. The precondition (which can be defined with the Spec class’ If method) will be evaluated before the transition is executed. If it evaluates to false, the spec will be ignored for the current execution of the transition. The postcondition (which can be defined with the Spec class’ IfAfter method) is evaulated after the transition is executed. If it evaulates to false, the spec will be ignored. If it evaluates to true, the spec will be verified using the input that was passed into the transition and the output that was returned from the transition.
For this particular spec, we define a postcondition that the spec only needs to be verified if the CacheManager returned one or more null values instead of cached responses. If the CacheManager returns a null, it means that there was no cached response for the request it was given. And that is when we need to verify the following things:
- The request was handled by the Request Handler
- The response that was returned by the Request Handler has been put in the cache
- The response array that was returned by the Request Processor contains the expected response
We perform this check for both requests, or only one of them, depending on which one was not in the cache yet. Remember that the transition will be executed 2000 times using the same request processor and the same cache manager. Which means that cached responses from previous transition executions might still be in the cache. Also keep in mind that one of the request types has a defined expiration of 1 second, and the other an expiration of 2 seconds which means that there will be plenty of transition executions where they are either both in the cache, only one of them, or none of them. All of those cases are now covered with the code in this spec.
Here’s another spec:
[SpecFor(typeof(ProcessRequestsTransition))]
public Spec CachedResponsesAreReturnedWhenAvailableInsteadOfCallingTheHandler(Tuple<ProcessInputElement, ProcessInputElement> input, Response[] output)
{
return new Spec(() =>
{
Action<ProcessInputElement> verify =
element =>
{
Ensure.False(element.RequestHandlerWasExecuted);
Ensure.True(output.Contains(cacheManager.ReturnedCachedResponses.First(r => r != null && r.GetType() == element.Response.GetType())));
};
if (cacheManager.ReturnedCachedResponses.Any(r => r != null && r.GetType() == input.First.Response.GetType()))
{
verify(input.First);
}
if (cacheManager.ReturnedCachedResponses.Any(r => r != null && r.GetType() == input.Second.Response.GetType()))
{
verify(input.Second);
}
})
.IfAfter(() => cacheManager.ReturnedCachedResponses.Any(r => r != null));
}
Here we verify that when a cached response was returned by the CacheManager:
- The Request Handler was not executed
- The response array returned by the Request Processor contains the cached response
We again perform this check for both requests, or only one of them depending on which one had a cached response.
You might think that this was a lot of work, but was it really? It’s about 160 lines of code, and i’m quite sure that the functionality that i’m testing is covered much more thoroughly than it would’ve been with 160 lines of code for classical unit tests. In fact, i challenge each one of you to come up with a more thorough test for this in less than 160 lines of code. Also, i can keep adding specs quite easily and those really don’t contain that much code. But my functional coverage would increase dramatically, much more than it would with classical unit tests.
Granted, the learning curve is steep as i mentioned already in a previous post. But once you start to get it, you become pretty productive with it and you’ll catch a lot more bugs before you deliver your software because of it.
Now, i tried hard to make everything here as clear as possible, so i’d love to hear from you whether or not:
- you understood it
- you like it
- you think it’s way over the top
- you want to start doing this too
December 24th, 2009 at 9:29 am
[...] Testing Agatha’s Caching Functionality With QuickNet – Davy Brion delves deeper into the tests he has written for Agatha’s caching layer which use QuickNet, along the way providing a good introduction to the capabilities of QuickNet and an understanding of how to use it. [...]
December 24th, 2009 at 10:42 am
Although I do not understand everything, I think I get the basic idea, and I can really see the potential. It seems like writing unit tests will be less repetitive, and the coverage will be higher… I’m looking forward to using it!
December 25th, 2009 at 12:52 pm
If I understand correctly, QuickNet is an infrastructure for random tests creation which is pretty cool, still I haven’t managed to find any more material on it. can you direct me to some ?
A couple of things I would like to know are :
1. Do I have any control on the random distribution of values ?
2. In case of failure what do I get ? what is the exception message ? Is it reproducable ?
December 25th, 2009 at 5:14 pm
code and download : http://code.google.com/p/quicknet/
It’s still a pretty young framework, in beta, and the upcoming next release allready contains a ‘small’ breaking change (SetUp became Setup). I’m trying to keep these things to an absolute minimum though.
Documentation is not yet available, buth forthcoming in the very near future. Follow the links on google code for the blog and discussion group.
1. Yes, full control if you want, default generators are available if you don’t need full control.
2. The framework reports the simplest failcase :
An acid test for this buggy class :
public class BugHouse
{
private int count;
public bool Run(int a)
{
if (count++ >= 3 && a == 6)
throw new Exception();
return true;
}
}
returns the following console output :
—— Test started: Assembly: QuickNet.Example.dll ——
TestCase ‘BugRun ShrinksIrrelevantInput’ failed: System.Exception : Exception of type ‘System.Exception’ was thrown.
D:\DotNetDev\QN\QuickNet.Example\SimpleTransitionShrinking.cs(16,0): at QuickNet.Example.BugHouse.Run(Int32 a)
D:\DotNetDev\QN\QuickNet.Example\SimpleTransitionShrinking.cs(58,0): at QuickNet.Example.UsingGenerator.BugRun.b__3(Int32 input)
D:\DotNetDev\QN\QuickNet\Transition.cs(55,0): at QuickNet.Transition`2.PerformTransition(TestRunReport report)
Output from BugRun ShrinksIrrelevantInput:
———————————————————-
BugRun ShrinksIrrelevantInput
Falsifiable after 1 test(s), 40 Transition(s).
——————–Simplest Fail Case——————–
Transition : BugRun, Output : True
Transition : BugRun, Output : True
Transition : BugRun, Output : True
Transition : BugRun, Output : True
6
———————————————————-
0 passed, 1 failed, 0 skipped, took 1,13 seconds.
This means that calling BugHouse.Run 3 times with any input and then once again with 6 as a parameter causes an exception.
‘Is it reproducable ?’
Once it hits a fail case, it will hit it again. You might have to tweak some values to get to the edge-cases more often. Currently it is not very debugger-friendly. You pretty much want to debug the simplest fail case and not the entire run, this is currently not possible. I’m working on improving that.
December 26th, 2009 at 12:04 pm
@Garfield
it doesn’t really create random tests… it uses randomly generated data (which as Mark said, you have full control over if you need/want it) to execute the transitions and verify the specs that you’ve defined. There’s more to it because the previous sentence is quite an over-simplification but i just want to avoid any confusion with regards to creating random tests.
December 27th, 2009 at 9:09 am
Kilfour –
can you show the test code for that acid test?
In other words – how much work did you need to put in to catch that bug?
December 27th, 2009 at 12:43 pm
public class UsingGenerator : AcidTest
{
static BugHouse bugHouse;
public override void Setup()
{
bugHouse = new BugHouse();
}
class BugRun : MetaTransition
{
public BugRun()
{
Generator = new IntGenerator(0, 20);
Execute = input => bugHouse.Run(input);
}
}
[SpecFor(typeof(BugRun))]
public Spec ShrinksIrrelevantInput(int input, bool output)
{
return new Spec(() => Ensure.True(output));
}
}
December 27th, 2009 at 12:50 pm
I just posted this on my blog as the formatting in these comments makes my eyes squint :
See http://kilfour.wordpress.com/2009/12/27/mr-bad-example/.
December 27th, 2009 at 4:12 pm
[...] Testing Agatha’s Caching Functionality With QuickNet [...]