I can’t wait to see how things progress. Thanks for posting all of this.

Analyzing build process performance can be done by using the appropriate tools when executing your unit or integration tests. Profilers, Coverage Tools or diagnostics solution can identify which methods take how long and how often those methods are executed by your unit tests. This helps you to identify methods that already have a very good coverage – and also lets you find the impact a single method has on the overall test execution.

Another interesting aspect is optimizing compile time. I’ve recently worked with a client that is running a build for every single check-in that is done by the developers. Here it is very critical to optimize compile times and unit test execution times. They use dynatrace to analyze test execution and are now also looking into diagnostics of their compile times.
Alois – a colleague of mine – wrote an interesting article about <a href=”http://blog.dynatrace.com/2009/05/04/performance-management-in-continuous-integration/”Performance Management in CI.

Hope my comment helps

i think we’re averaging around 8 builds a day… it would be more if we allowed concurrent builds of the same project on our buildserver but we’ve disabled that so the builds for this project won’t prevent other projects’ builds from running frequently

as for performance bottlenecks in the old tests… we did that once, there was one part that was executed before every test which caused a lot of slowdown. We modified that last year, which actually took the build time from 90 minutes to around 40 minutes back then. Other than that, there aren’t really any specific performance issues we can easily fix for those tests. Going with several instances of the CI server is unfortunately not an option.

We’re not writing unit tests for the generated code… the codegenerator actually generates a whole bunch of tests for the actual code it generates. I seriously dislike that (IMO the codegen should have its own tests to make sure it generates correct code) but there’s really not that much we can change about the whole thing at this point. The whole codegen process is way too complicated for what it originally should’ve been and we’re kind of stuck with it for parts that haven’t been rewritten yet.

Generating stubbed code purely for testing would make a lot of those old tests useless as well… the generated code is basically an extensive datalayer (and a pretty bad one at that) and a lot of those tests use parts that are generated, and parts that were added to it (such as custom SQL queries). If we were to generate stubbed code, the tests would fail the moment they would execute any custom query which happens very frequently.

Making changes to the codegen is something we try to avoid as much as possible. Every time we try to make a change there, it causes problems and we end up reverting to the way it was before. It’s also hard to convince management to spend time on the codegen since the idea is to phase it out and basically rewrite everything eventually.

If you’re trying to cut running time, though, here are some things I’d look into:
- Have you profiled the test run to find the perf bottleneck(s)?
- Can any of the tests be parallelized?
- Assuming the tests have to run serially, is splitting the CI server into several instances (virtual servers or whatever), each running different subsets of the tests an option?
- Is there a better way to test your generated code than hand-crafting unit tests for it? Maybe that’s not what you’re doing and I misunderstood you. If so, I apologize.
- Alternately, is there a subset of those tests that give you the biggest test bang for the time it takes to run them (80/20 rule)? If a big bunch of tests always passes, it’s probably not helping you to find new problems to run everything as frequently as tests for problem areas. Yes, this is similar to Whaley’s suggestion. I like his suggestion, too, though.
- Again, maybe I misunderstood you, but if you’re having problems stubbing db accesses in the generated code, why not create a “test build” wherein the generator creates stubbed code? Sure, that’s not exactly the same as release code and it’s a good idea to run against your “real” code, too, but if you ran against a test build frequently and a real build less frequently I bet you could speed up the feedback.

Also, the build server itself is plenty beefy… most of the time is wasted on running those old tests, during which the CPU activity of the build server isn’t really high (around 30%) so upgrading the buildserver isn’t going to make a huge difference

I have no clue how modular your system is but perhaps keep your builds on commits strictly relegated to compiling and running *unit tests* for a particular module/library and do a nightly/twice a day/four times a day “full” build that compiles and unit+integration test of everything.

Also, how beefy is your build box/cluster? It never hurts to upgrade that thing. Hardware is cheap compared to developer time.