We Underestimate Nonlinearity, Not Effort

People have linear minds. We expect and imagine all things to be straight. I live many of my weekends in Brussels (Belgium). It has one of the craziest street plans I ever came across. Many roads seem to be straight, but most of them aren't. I often end up in weird and unexpected places because the roads take me 90 degrees to the left or to the right of where I expected them to take me. In Brussels, my estimates of direction are always wrong.

I think it is the same with software estimates. People, and software developers in particular, have linear minds. We expect projects to go straight, while in reality the road ahead is as crooked and bent as the average Belgian politician.

Here, let me show you:

This graph pictures the relative size of 127 of our projects according to the estimates that were given by our developers. As you can see, the line follows a power law distribution, which is expected in such cases. But here's another one:

The second graph shows you the number of days our people worked on 320 real projects. Naturally, this line also depicts a power low distribution. However, there is one major difference between the two graphs: The line in the second graph is much steeper than in the first, therefore… the effort estimated by our developers is more linear than the real effort! These findings support a theory I have had for some time now:

People do not underestimate effort, they underestimate nonlinearity.

It is often said that people typically underestimate project size. But I don't think that's the real story. Underestimation of project size is just a side-effect. From the graphs above it seems that our people overestimate small projects, while underestimating big projects. We all know that even the smallest tasks can sometimes take much longer than expected. Usually this is because of unexpected complications. It is never the work we underestimate, though. We underestimate the chance of something unexpected to happen that doesn't match our linear thinking. And the bigger a project, the bigger its complexity and nonlinearity, and the larger the impact of any unexpected surprises.

To compensate for the lack of understanding of nonlinearity and complexity, our people simply tend to overestimate effort. "Ehm, let's see. We need to build twenty features. They usually take two days a piece, but let's make that three, just to be safe. So, the estimate will be 60 days. And if they want only half of them, it will be 30 days." Wrong! This kind of linear thinking leads to simple projects being overestimated, while large projects are still underestimated. This would be better: "Ehm, let's see. The features take two days a piece, but complexity increases exponentially when doing more. So, if they want twenty features it will be 40 days + a complexity margin of maybe 25 days, making it 65 days. If they want only half of that it will be 20 days + a complexity margin of maybe 8 days, making it 28 days."

The shorter a street in Brussels, the better the chance I can reach my destination, even with my eyes closed (and not considering all the potholes I normally wish to evade). But the longer a street in Brussels, the more it deviates from my linear estimation, and the likelier it is that I end up in some politician's backyard swimming pool.