Just read a post from a former co-worker about how to test exceptions. He uses the ExpectedException attribute (of MS Test) which has at least the following issues (IMO):
- There is some confusion about the message parameter in the attribute. In NUnit, the message you provide is the expected exception message that you want to test. With MS Test, it is the message that should be displayed when the exception is not thrown. This effectively removes the possibility to use the attribute to test the message of the exception when using MS Test. And yes, it can definitely be useful to test the content of the message.
- It is not immediately clear which line of code is supposed to throw the exception. In the example he provides, when you only have 2 lines of code (one of which is instantiating the object) it's not hard to figure out where the exception is thrown. But when you have larger test methods, it's often confusing to see where the exception should be thrown. And yes, we all like to avoid large test methods, but sometimes it's hard to avoid. And with large, i mean 10+ lines.
- You can't test for the general Exception type. Whether it is a good idea or not to throw a general Exception is not really relevant to this discussion. But i do want to be able to test for it when the situation calls for it
- If you have a custom exception with contains some properties that you want to test, using the ExpectedException attribute is insufficiant... yes, you can test that the exception is thrown, but if the exception has extra properties, the contents of those properties should be validated as well
Let's use his Order example, but with some more logic, to show some better ways to test exceptions. Suppose we have the following code:
public class Order
{
public IEnumerable<OrderLine> OrderLines { get; set; }
public Customer Customer { get; set; }
public Order() : this(null, null) {}
public Order(Customer customer) : this(customer, null) {}
public Order(Customer customer, IEnumerable<OrderLine> orderLines)
{
Customer = customer;
OrderLines = orderLines;
}
public decimal CalculateTotal()
{
return OrderLines.Sum(o => o.Price) * Customer.DiscountRate;
}
}
For maximum flexibility, we've provided 3 ways to create an Order object: without any of its dependencies, with one dependency (Customer) and with both dependencies (Customer and OrderLines). Depending on which constructor you've used, you might need to provide one or two dependencies through their setters before you can call the CalculateTotal method. With the code as it is right now, we'll get a NullReferenceException if one of the dependencies hasn't been provided. So we'll modify the CalculateTotal method to guard against this. I don't like creating Exception-derived types for everything that could possibly go wrong, so i'll define an enum with the validation problems that the Order class could have:
public enum OrderValidationProblem
{
OrderLinesNotSet,
OrderLinesHasNoItems,
CustomerNotSet
}
Now we introduce an OrderValidationException type which will contain the type of validation problem:
public class OrderValidationException : Exception
{
public OrderValidationProblem Problem { get; set; }
public OrderValidationException(OrderValidationProblem problem)
{
Problem = problem;
}
}
Since we try to be responsible programmers, we immediately write the tests so we can't ever forget to properly guard against these conditions:
[TestFixture]
public class OrderValidationTests
{
[Test]
[ExpectedException(typeof(OrderValidationException))]
public void CustomerNotSetThrowsValidationException()
{
var order = new Order();
order.OrderLines = EntityTestFactory.CreateDummyOrderLines();
order.CalculateTotal();
}
[Test]
[ExpectedException(typeof(OrderValidationException))]
public void OrderLinesNotSetThrowsValidationException()
{
var order = new Order(new Customer());
order.CalculateTotal();
}
[Test]
[ExpectedException(typeof(OrderValidationException))]
public void OrderLinesWithNoItemsThrowsValidationException()
{
var order = new Order(new Customer(), new OrderLine[] {});
order.CalculateTotal();
}
}
Now it's time to make the tests pass... so we modify the CalculateTotal method to provide the necessary guard clauses:
public decimal CalculateTotal()
{
if (OrderLines == null) throw new OrderValidationException(OrderValidationProblem.OrderLinesNotSet);
if (OrderLines.Count() == 0) throw new OrderValidationException(OrderValidationProblem.OrderLinesHasNoItems);
if (Customer == null) throw new OrderValidationException(OrderValidationProblem.CustomerNotSet);
return OrderLines.Sum(o => o.Price) * Customer.DiscountRate;
}
At this point, the tests pass... but what do they prove? Sure, we throw the right exception when we need to. But we still don't know if the exception has been constructed properly. If we want to test that, we have to stop using the ExpectedException attribute. But i don't wan't to litter my tests with try/catch constructs either. We could create a helper method to perform the necessary check:
private void CheckExceptionAndProblem(Func<Decimal> function,
OrderValidationProblem expectedOrderValidationProblem)
{
try
{
function();
}
catch (OrderValidationException e)
{
Assert.AreEqual(expectedOrderValidationProblem, e.Problem);
return;
}
Assert.Fail("Exception was not thrown");
}
The generic Func type allows us to easily pass a delegate to the CalculateTotal method instead of having to declare a specific delegate for it first.
Then we'd modify our tests like this:
[Test]
public void CustomerNotSetThrowsValidationException()
{
var order = new Order();
order.OrderLines = EntityTestFactory.CreateDummyOrderLines();
CheckExceptionAndProblem(order.CalculateTotal, OrderValidationProblem.CustomerNotSet);
}
[Test]
public void OrderLinesNotSetThrowsValidationException()
{
var order = new Order(new Customer());
CheckExceptionAndProblem(order.CalculateTotal, OrderValidationProblem.OrderLinesNotSet);
}
[Test]
public void OrderLinesWithNoItemsThrowsValidationException()
{
var order = new Order(new Customer(), new OrderLine[] {});
CheckExceptionAndProblem(order.CalculateTotal, OrderValidationProblem.OrderLinesHasNoItems);
}
That's already much better i think... We can test that the exception is thrown, and that its Problem property is set correctly, and we only have one try/catch clause in our tests.
I'm still not happy with it though... The CheckExceptionAndProblem method is not reusable for anything other than OrderValidation, yet testing exceptions is a common problem so we should strive to provide something that'll help us anytime we need to test exceptions.
How about a custom assert method that asserts that any piece of code that is passed to it throws the expected exception, and then returns the exception so you can easily assert anything else you wanna check in the returned exception? Sounds pretty good to me... let's give it a shot:
private T GetThrownException<T>(Action code) where T : Exception
{
try
{
code();
}
catch (T expectedException)
{
return expectedException;
}
catch (Exception e) {}
Assert.Fail("Expected exception of type {0} was not thrown", typeof(T).FullName);
return null;
}
This method runs the code that was passed in, catches the expected exception and returns it. If the expected exception is not caught, it fails the current test.
Now we can modify our CheckExceptionAndProblem method so it looks like this:
private void CheckExceptionAndProblem(Func<Decimal> function,
OrderValidationProblem expectedOrderValidationProblem)
{
var expectedException = GetThrownException<OrderValidationException>(() => function());
Assert.AreEqual(expectedOrderValidationProblem, expectedException.Problem);
}
Now we can use the GetThrownException method pretty much anywhere where want to test exceptions and their properties.
Anyways, this is just one possible approach of testing exceptions in a much better way than the ExpectedException attribute offers us.
Btw, i think the xUnit.Net testing framework already provides similar approaches to what i used in this post
... depending on how much free time i'll have obviously