There are essentially 2 big problems with this project. One is a mountain of legacy code with so much technical debt (due to the evils of code generation) it sorta resembles the current economic recession (as in: it'll take a long time to get everything sorted out). The other problem is more a matter of organization. We're a pretty small company and while we have great ideas for products, we simply can't assign a steady, stable team of developers to work on any of these projects on a continuous basis since we all often need to work on stuff that is simply more lucrative at that particular point in time. The result is that this particular project typically has an ever rotating group of developers working on it, usually for short periods of time. Some people work almost full-time on it, but that list is pretty limited.

Not exactly the ideal situation for a large project to apply CI and other agile development practices to, right? Luckily for us, we're quite stubborn and we try to make the best out of every situation. So back when this project's CI success rate was only showing a lack of success, we all agreed to follow the CI rules and at this point, i'm pretty proud to be able to show you guys the following picture:

Note: we moved to Team City in July 2008 so i can't show you any earlier data than that.

Anyways, as you can see, after the dismal months of February and March, the success rate of the CI build gradually started improving again. The average time to fix failing tests also decreased sharply. We still have failing tests from time to time, but at least now they're all dealt with in a timely fashion. We still have build failures, though those have decreased a lot as well and are always dealt with pretty soon now. I'm not sure if this is because of my 'Continuous Bitching' whenever the build fails, or that everyone bought into the concept of CI again (for my own sake, i'll just assume that it's the latter instead of the former) but i'm pretty happy with the results that we're getting.

Also, take a look at the number of tests (we're at 16000+ now) and the duration of the build. The build time is about 48 minutes on average now, which is obviously way above the recommended 10 minutes. I'd love to see this go down to about 20 minutes (which i'd find very acceptable considering the size of the project) but that's gonna take a long while. Of those 16000 tests, there are about 13000 tests that cover the legacy code and they all use the database. And since those 13000 tests use a generated data layer, we can't just let it run on an in-memory database nor can we mock the database in those tests because all of that generated code, and pretty much everything that was written on top of it, is coupled more tightly than Siamese Twins. We also lose a couple of minutes of build time due to our Genesis processing but that is simply something that we can't go without anymore so we don't really mind the extra build time of that part.

So there you have it... the reason i wanted to post this is because when the topic of CI comes up, you always read about 'instant feedback' and really quick builds and things like that. It's simply not always like that in the real world. But with a bit of effort and focus, you can get many of the benefits that are usually attributed to CI, even on huge projects with lots of legacy code.

Before you commit your changes to the repository:

1. You update your local copy to get all of the changes that were committed since you last updated your local copy
2. If there are conflicts, you resolve them in your local copy... you DO NOT simply overwrite other people's changes
3. You compile your code (You'd think nobody would forget this one, right? Think again...)
4. If it doesn't compile, you fix it (Again, you'd think nobody would forget this one....)
5. If it compiles, you run the tests. At least run the fast tests, and i do mean all of them, not the ones you just wrote or modified. If you have thousands of slow tests, then you obviously can't run them every single time you need to commit. But you can and should definitely run the fast tests. Yes, all of them!
6. If a test fails, fix the code that broke it, or fix the test if it's no longer correct. Repeat this process until none of the fast tests fail.
7. Commit your code, and begin working on your next task, take a break, go home, whatever is most appropriate at the time.

For the next section, i'm going to assume that everyone has at least something that continuously visualizes the state of the automated build, either TV's on the wall (like we have), or at least a system tray notifier (like we also have).

When the build breaks:

1. If none of your team members beat you to it, look at the build results to figure out why it broke
2. If it's a compile error, alert the team (so they don't accidentally update their local copy with the broken code), and figure out who's responsible. MAKE SURE THE COMPILE ERROR(S) ARE FIXED ASAP. As a team, punish the guilty one for wasting the team's time (make the guilty one buy donuts, pizza, make them wear a dress for a day, whatever works...)
3. If it broke because of failing tests, look at the failures. Where you the last one to touch those tests or the code that was being tested by them? Then fix it ASAP. Was it someone else? Discuss this with the person who's responsible. If the developer who's responsible has time to fix it, then he/she must fix it at once. If the developer can't fix it instantly, then someone else has to step up and fix it. DO NOT WAIT FOR SOMEBODY ELSE TO FIX THEM. Discuss it with the team and agree on someone to fix the tests. If you're not sitting with the rest of the team, pick up the phone, use IM, whatever. But at least do something!
4. Repeat these steps until the build is green again.

As a member of the team, you have certain responsibilities. Every team member has them. You have to take your responsibilities seriously, if not, you're just wasting your team members' time. At the same time, you should hold your team members accountable when they fail to take their responsibility serious. Don't fight about it, but talk about it as a team. When people don't do what they are supposed to do, discuss it with the entire team and after a couple of times (at most) people will feel that social pressure to not let the team down again. If they don't, well then maybe they shouldn't be part of the team in the first place but that's a whole other story.

Those are obviously important goals. There's another reason why a fast build (with that i mean building the code and running all of the tests) is important. I believe the duration of a build can be a good indication of the complexity and the quality of the code base of your project. If a build takes a long time, it's usually a strong indication that something is definitely not right in the code.

Let's apply the technique of Root Cause Analysis to the problem of a slow build of a fictional (yet very common) project:

Question: Why is the build so slow? Answer: Because the tests take a long time to run

Question: Why do the tests take so long? Answer: Because we need a lot of set up in the code to be able to run the tests

Question: Why do we need all that set up? Answer: Because the code under test requires a lot of stuff to be in place before we can run it

Question: Why does the code under test require a lot of stuff to be in place before we can run it? Answer: Because those classes do a lot of stuff... they get data from the database, do something with it and put it back into the database.

Question: Why do those classes do so much? Answer: They instantiate Data Access Components and they don't work without real data in the database.

Bingo.

On a code level, what is the problem here? The last 2 answers indicate that the classes that are being tested have too many responsibilities and that there probably is a lot of highly coupled code in there. By looking at the build time and asking those questions to the developers, you're able to learn a lot about the quality of the code without having any prior knowledge about the project or the code.

Obviously, it's not always possible to reach valid conclusions based purely on the build time of a project. But again, it can be a great indication.

Now, the project mentioned above might actually have a really low number of bugs in it. The application might even be perfect according to its users. But that doesn't mean that the code of the application is in good shape. If the code were in better shape, it might have been cheaper to develop the application. It might mean that the maintenance of the code would be cheaper too. But the slow build and the answers to the question do indicate complexity in the code. We can't always eliminate complexity but we should strive to minimize it, because it leads to cheaper development in the long term.

Granted, a fast build time doesn't always mean that a code base is in great shape either. That's a conclusion you could only reach by looking at the code and going through the tests. But for slow builds, you'd be amazed what you could quickly learn from a few questions.

I don't understand why someone would even have to choose between them. Setting up a regular CI build (with that i mean a simple compile + running all the tests) for a project should never take more than 10 minutes, tops. That is assuming that you already have a build server in place, obviously. But with Team City being so incredibly easy to install and configure, that can no longer be an issue.

So what does it take then? Drop in a simple, standard build script which builds your project and runs the tests, add a build configuration and you're done. Setting up a basic CI build should never take more work than that, unless you have some very specific needs, or the project structure is somewhat messy... as in: it doesn't compile if you don't have certain assemblies in folders that are either hardcoded or 'assumed' to be there... which should be avoided at all times

Seriously though, long build scripts or requiring a lot of time to set up a new CI build are all signs that your CI process in general needs some refracturing (nope, not a typo).

And in case you needed some motivation: even TFS 2008 allows you to set up a CI build in less than 5 minutes of work

There was one issue at work that was somewhat hard to fix though. TeamCity has fantastic integrated support for NUnit, but unfortunately it doesn't write NUnit's output to XML files like nunit-console.exe does. After a bit of browsing, i found a post on the TeamCity forums that discussed a workaround on how to get the output. Unfortunately the workaround is a bit cumbersome as it requires you to you create an XML file which contains the arguments that TeamCity would pass to its NUnitLauncher task.

I believe in 'script reuse' as much as i believe in 'code reuse', so every project's build script merely imports a generic build script and then it just overwrites some variables that the generic script uses and then it kicks off the usual build process. Since all of our projects will now be built by TeamCity Build Agents, and we definitely need to have NUnit's results xml file i wanted to automate this whole thing as much as possible.

So instead of having to create (and thus, maintain) a cumbersome xml file for each project, i wrote an MSBuild task that would generate the xml file containing the arguments on the fly during the build, and would then pass those xml arguments to TeamCity's NUnitLauncher. This way we get to keep all of TeamCity's NUnit integration goodness, while still keeping the NUnit result files around as well.

So now in my MSBuild file, i can just do this:

As you can see, there is nothing project-specific in there so this just integrates nicely into each build that we need. The only limitation to this approach is that TeamCity's NUnitLauncher will write one NUnit result file for each assembly containing tests. Apparently there is no way in the current version of TeamCity to get it to combine those results. We use an extra tool to analyze the NUnit output after the tests have run, and unfortunately it requires all of the results to be in one file. I looked around for a way to merge the output of several NUnit result files, but i didn't find something that was already available. So i wrote another MSBuild task that would merge the output into one xml file:

Now we finally have all of TeamCity's goodness while we still get to run our post-test analysis on the NUnit result file that contains the testresults of all the test assemblies

You can find the code of the MSBuild tasks here. They are BSD-licensed so feel free to use them if you need them.