Test Doubles: When To (Not) Use Them

9 commentsWritten on August 13th, 2008 by
Categories: testing

I'm currently introducing somewhat 'advanced' testing techniques such as Dependency Injection and using Mocks/Stubs to my team members. They're getting the hang of it pretty fast, but the typical "do i need to mock this object for this test?" question came up pretty frequently at first, and sometimes it still does. Which is only normal when you're starting out with these techniques. So i figured it would be useful to list some guidelines about it in a post.

First of all, i'll use the term test double instead of mock or stub or whatever for the remainder of this post to (hopefully) avoid any confusion. Gerard Meszaros (author of the excellent book xUnit Test Patterns) provides the following definition of a test double:

A Test Double is any object or component that we install in place of the real component for the express purpose of running a test.

One thing that's not clear from just that sentence (it is very clear in the book, but not when you merely look at this sentence in isolation) is the fact that it usually only makes sense to use a double for what Gerard calls Dependent-On Components (more commonly referred to as dependencies) of the code you are testing.

So, when does it make sense to use a test double, and when are you better off just using the real component? A lot of people think differently about this, but i have found the following guidelines to be the most efficient with regards to keeping the tests focused, fast, easy to set up, and most importantly: meaningful.

If a component uses an external resource (a database, a file, a remote service, ...) either directly or indirectly, then i always replace the component with a double when i'm testing code that is dependent on it. An external resource usually means that the component is either slow (keep in mind that you want to run a couple hundred tests per second), hard to set up (required data or files or whatever), or a possible source of unpredictability (tests should never fail and then work again without changing code). Using doubles allows you to avoid the slowness, makes it easier to set up any required state, and they are always predictable.

Components that do not use external resources yet still require a lot of work to set up (for complex in-memory calculations for example) are also good candidates to be replaced with test doubles. Even if the real component is lightning fast, if you need to write a lot of set up code just to get the real component to behave the way you want it to for a specific test, then you're better off using a double for it. Tests should be focused, so you don't want a lot of code in them (either in the test method itself or in the setup/teardown methods) if that code doesn't really pertain to the functionality that you are testing in that fixture. Performing a bunch of set up for dependencies is simply a waste of time while writing it, while maintaining it, and it's also a distraction while reading it. So, always avoid that, even if the components don't use external resources.

That's pretty much all for when you should use doubles. Now, when should you not use doubles? My answer to that is very simple: when there is no clear-cut benefit to it. If you're using classes that do not have external resources and hardly require any set-up, then by all means, use the real components in your tests. Using test doubles is a great technique, but it is only a means to an end. Using doubles merely for the sake of using doubles will not lead to better tests. It's very important to keep that in mind.

I often hear developers say something like "but if you're not using the real components in your tests, how do you know for sure that they work?". Well, the answer to that is also pretty simple. Each component should have its own dedicated tests. Yes, even components that use the database, or the file system or even remote services or whatever else you can think of. These components will be used at runtime, therefore they need to be tested. Test them as thoroughly as you can, but no more than that. Test them in isolation, perhaps even in a different test suite that you don't need to run every time. Perhaps you don't even need to run these tests automatically? But that would be a nice subject for a different post :)

Anyways, the guidelines (or rules of thumb) are just that... they're only guidelines, not strict rules. As we all know, when it comes to coding, there are always exceptions (no pun intended). If you follow these guidelines and some things still feel like it requires too much work or are causing too many issues, then there's probably a better solution for that specific problem still waiting to be found. Don't be afraid to spend a bit of extra time searching for it. It might save you a hell of a lot of time later on.

  • anon

    The problem we found is that developers are too used to working closely with databases, so when we introduced unit tests, a lot more tests than was necessary had db dependancies.

    Of course this doesn’t scale well, takes a lot more setup/teardown work and is generally a bad idea, but the hard part is convincing people that separation of concerns (and single responsibility principle) is often the root cause of this database dependency problem. One person I work with has adopted (db dependent quasi-unit) testing and loves it, so I don’t want to discourage them. I’d rather have slow tests than no tests at all.

  • http://davybrion.com Davy Brion

    true, up to a point. We currently have 2 projects at work where practically all of the tests use the database. One has about 11000 tests, and the tests take 20 minutes to run. Nobody runs the tests on their own machine, they always wait for the buildserver to run them. Apart from being horribly slow, the developers of that team actually do a great job of keeping that build ‘green’ as much as possible. Still, i would much rather see them having a test suite that can be ran by anyone at any time without wasting time on it.

    The second project has about 7000 tests, which take about 10 minutes to run. The biggest problem with this project is not the time wasted running the tests, but the time wasted on writing the setup for each test because they have a horribly complicated data model. For every new test, they are wasting a lot of time just to be able to write tests. There’s also a large maintenance cost involved… i’ve heard some developers claim that they often spend 4 hours or more fixing/modifying tests that are related to a fix/modification in the code which only took them about an hour to do. Those tests are also extremely fragile, up to the point where i’m actually suprised if the project has a completely green build. It doesn’t happen often. And it never lasts long.

  • Pingback: Dew Drop - August 14, 2008 | Alvin Ashcraft's Morning Dew

  • Pingback: Reflective Perspective - Chris Alcock » The Morning Brew #159

  • Pingback: Nocturn vision » Blog Archive » Introduction to test-doubles

  • http://dgoyani.blogspot.com/ Dhananjay Goyani

    While browsing for something, I come across this post somehow. Excellent one, as usual – Davy.

    >>> If a component uses an external resource (a database, a file, a remote service, …) either directly or indirectly, then i always replace the component with a double when i’m testing code that is dependent on it.

    Now above statement go against Mark’s.

    Test Double in your post seems to be PURE stub however by ‘mocking’ Mark is referring to stub in “verifying behavior/interaction” context.

    Can you make it little bit more clear?

  • http://davybrion.com Davy Brion

    @Dhananjay

    no, a test double can be a mock or a stub… it’s just a term that means that the implementation of one of the dependencies will be replaced with something else. Regardless of whether it’s a mock or a stub.

    And i still recommend the same guidelines as i did originally in this post, regardless of the point Mark makes in his post

  • http://dgoyani.blogspot.com/ Dhananjay Goyani

    Yes, I agree. I much liked – replace external resource with a double. That way you simply making your test focused and independent.

    As per Mark’s reasoning (pasted at bottom), I still think that by mocking he means to verify behavior/interaction on the 3rd party components. Unit testing somebody else’s code is not one’s responsibility and secondly writing a wrapper on top of each library for testability seems weird to me IMO.

    Well, I follow your recommendation just that I want to decipher Mark’s suggestion.

    ———————-
    There are a couple of reasons why doing this isn’t a great idea:

    1) We have no idea what the correct method calls are in the first place so we’re just guessing based on looking through the Hibernate code and selecting the methods that we think make it work correctly.

    2) If the library code gets changed then our tests break even though functionally the code might still work

    ———————-

  • Pingback: First Experiences With RSpec/BDD