Estimates Are A Double-Edged Sword
Posted by Davy Brion on July 28th, 2010
Most people in the software development industry will (hopefully) agree that estimates are important. You need estimates to predict the cost of development and how long it’s going to take. Without them, there is no way you could promise a delivery date to a customer or come up with a total cost estimate. Both of which are tremendously important when your core business is based on delivering software to clients. It is therefore very important to make sure your estimates are as good as they can be, and that you review your estimates with the final actual figures so you can figure out where you need to start improving (either your development process, or your estimation practices).
But there are some things you need to watch out for in any organization which takes estimates seriously. Most development shops have estimates at the task-level. These task-level estimates are often used during the planning of the next iteration, but were most likely also used in the initial total estimate of the project. I certainly don’t doubt the importance of those task-level estimates, but i do think they should be used cautiously.
If too much importance is placed on staying below the estimates, or at least not going over them, you run the risk of growing a culture where developers let the estimates have too much influence on the quality of the work they’re doing. Some developers will strive hard to routinely stay below the estimates. Some developers will work in fear of going over the estimate. In both cases, it leads to technical shortcuts that will be taken. Those shortcuts will consist of skipping some tests here and there, skipping a bunch of tests altogether, decreasing willingness to refactor code that could use it and worst of all, no longer keeping a clean design in mind. Granted, i’m painting a very bleak picture here but it does happen with some people and the negative results can not be underestimated.
Let’s consider the overall effect of these shortcuts. A shortcut here or there is not abnormal in some circumstances, but it should be exceptional. It should not become routine, it should be frowned upon and i’d even go as far as recommending a “no shortcuts allowed unless the team approves of it” policy. A shortcut here or there doesn’t take a lot of effort to fix and because of this, it can often be done as soon as the original reason of the existence of the shortcut (typically: a deadline) is no longer an issue. An accumulation of shortcuts however leads to not only an obviously bigger workload to fix all of the shortcuts, but it also impacts a lot of other code that shouldn’t have been impacted in the first place. Once this happens, you’ve got yourself an unhealthy code base and it’s only going to get worse until it eventually dies at a younger age than it could have reached.
Now, imagine a culture where estimates are given as much importance as quality. If you’re approaching the estimate and you’ve still got a lot of work left to fully complete the task (and with fully completed, i really don’t mean watching it work on your machine as you manually test it), then you consult with the rest of your team members. Estimate how much time it would take to fully complete the task, and see if the increased time over the original estimate is doable. If you’re already behind your overall schedule, it’s probably not doable unless there are many subsequent tasks that are dependent on this particular task. But if you’re ahead of schedule, you’re probably better off accepting the fact that you’re going to go over this task’s original estimate in order to do it right. If you do so, the extra hours you spent on this task should (in most cases at least) not lead to extra hours on future tasks. Conversely, if you’ve finished a task well below the estimate, it might actually be a good investment to refactor some ‘bad’ code that you’ve come across while working on your task. Or to add some missing tests here or there. That certainly doesn’t mean that you need to spend all the available remaining time you have but you shouldn’t run to start on a new task as soon as humanly possible either. Follow the boy scout rule here… you won’t spend that much extra time on it, but overall quality can improve greatly from this.
In fact, i’d even bet that the time it takes to do it right will in a large majority of cases still be less than the sum of the original time that was spent on a task hastily done and the time that will be spent on bugs that resulted from that hastily done task, not to mention the impact it could have on the development of future tasks simply because of the suboptimal implementation of the hastily done task.
Finally, it’s also important to think about why there was a need to go over an estimate. There can be a variety of reasons for this, and one of them is that the task was simply badly estimated. If that indeed turns out to be the case, make sure that you learn from it to prevent this from happening in the future. When we find bugs in our code, we try to prevent those bugs from ever occurring again. We should have a similar attitude to our estimating practices as well. If an estimate was simply too low, make sure that a similar task in the future won’t be estimated too low again or you will again introduce a possible problem in your (future) project.
And that, in my opinion, is also a good reason to decide to go over the estimate if you can. If you go over the estimate and all people involved learn from the bad estimate, then everyone basically benefits from it. Future estimates should become more accurate, and no code was harmed in the process. You might have lost some money on it, but at least you only lost it once instead of losing it over and over again in similar future circumstances.
July 29th, 2010 at 7:57 am
I have to disagree with you on this one. When I look at the word ‘estimate’, the word ‘gamble’ is the next one that pops into my head and I hate to rely on gambles. Human beings suck in providing long-term estimates and they will never ever be good at it. Long-term estimates are usually never accurate, and if they do, it’s merely by accident. We can however be more accurate with short-term estimates and they can be effective (e.g. what can we do in the next week or two weeks), but that’s about it. Sure there are a couple of books written full of techniques on how we can improve estimates, but then again, there are also books written on how to be good on black jack and I don’t think they are making millionaires on the spot. Try to guess the distance from the earth to the moon. If you’re off by more than 5%, try again and estimate the distance from the earth to mars by applying what you learned from your first estimate. I don’t think you can ever guess the correct distance for 5 other planets in row.
I realize however why estimates are important for customers and management because it’s about the money. But what I don’t understand is why they are holding on to ‘gambles’ as the sole criteria for executing/evaluating a project. I don’t see estimates being that important to the software industry either. Most win-win projects I have ever been a part of didn’t need estimates, but they fully employed the agile mind-set which was enough for these customers to go with it and mark it as a success afterwards because they added business value instead of merely making an arbitrary deadline. But I don’t have to preach to you why agile works.
My apologies for the long reply.
July 29th, 2010 at 9:38 am
@Jan
Those long-term estimates are still important, if only to get an idea on how much money will be required to do the whole thing, or to get an idea of what can be done for a certain price. They’ll never be 100% correct, but it shouldn’t be that far from reality either. Even if you do follow the agile-mindset, there will (in almost all cases anyway) be a total estimate and the closer it turns out to be to reality, the better for everyone.
Also, the company you work for does software to support its core business, but software is not its core business. There really is a big difference in how projects, budgets and schedules are handled between companies whose development supports core business, versus companies where development _is_ the core business.
But this post was more about possible dangers of short-term estimates anyway
July 29th, 2010 at 11:43 am
I agree that there’s a difference regarding importance of estimates between companies where IT is supportive and companies where IT is the core business. But in the end, software is supporting a business whether it’s your own or someone else’s. The problem with any kind of estimate is that they’re mistaken for the truth, whether it’s a customer or management, and that success or failure is often measured according to these estimates which is just nuts. A link can also be made with ‘fixed-price’ projects which is the sad reality of our industry today, as this is often (according to my experience) a lose-lose deal.
As usual I’m ruining the conversation by bending it from short-term to long-term estimates
, but it’s a very interesting discussion nonetheless.
July 29th, 2010 at 11:51 am
@Jan
“The problem with any kind of estimate is that they’re mistaken for the truth, whether it’s a customer or management, and that success or failure is often measured according to these estimates which is just nuts.”
I fully agree that it definitely should not be a measure of success or failure
“A link can also be made with ‘fixed-price’ projects which is the sad reality of our industry today, as this is often (according to my experience) a lose-lose deal. ”
Well, that depends on how well you (as an organization) have learned to estimate. For instance, we keep track of time spent on each task and we can compare them to the original estimates. We also know how much overhead (both technical and project management) we typically need to estimate for a given workload based on our historic figures. With that data, we’re in a position were we can usually make a reasonably accurate estimate for fixed-price projects, and in fact, we prefer to do projects that way. Of course, that does mean that we have to make sure that our task-level estimates are reasonably accurate and when they aren’t (give or take… certainly doesn’t have to be 100% all the time since that’s impossible), we need to figure out why there was a big difference so we can prevent that from happening in the future.
But yeah, if you don’t do any of those things then you’d be stupid to do fixed-price
and no worries, it is indeed an interesting discussion
July 29th, 2010 at 12:01 pm
I have to admit that the approach you’re describing is a more scientific approach based on actual numbers and sounds quite interesting. I also think that most companies don’t work that way (at least not the ones that I worked for). I also wonder how you deal with changing requirements, priorities and feedback in general using the approach you mentioned?
July 29th, 2010 at 12:16 pm
@Jan
as usual, it all depends
When a customer wants changes to parts that have already been developed, we look at what it’ll cost and how much budget we have left. If we still have enough left to do the change without impacting what still needs to be developed, then we just do it. If there’s not enough left, we either take the hit and do it anyway or propose to do it a future phase… Obviously, this depends on a few factors
If requirements change of parts that we haven’t developed yet, estimates are basically adjusted according to the changes. If that turns out to put us over budget, we again see whether we’ll take the hit or propose it for a future change. If the changes end up giving us more budget flexibility, then that’s obviously good for other future changes that might be requested.
Generally speaking, we’re very flexible as long as the budget allows it. Once we go over budget, it sorta depends
Priorities can be changed as well… if the priority of a task or a set of tasks is moved up, we just need to make sure that all tasks that the newly-moved-up tasks are dependent on are moved up as well. This typically doesn’t influence task-level estimates though, unless a temporary solution is introduced somewhere to be replaced by a final solution later on in the project. And again, then it all depends