I have a bit of a problem with a recent survey in Dr. Dobb's Journal, carried out by Scott Ambler. The survey is supposed to show that agile projects are more successful than traditional projects. I think this is probably true, but I also think the survey doesn't prove that. I have commented on it on Scott's blog, but I will repeat my argument here, for your convencience (and my own administration):
A. Comparisons between agile and traditional projects should only be made for projects of equal size. Because, by their very nature (complexity) big projects have a higher chance of failure than small projects, whatever best practices you throw at them. Even a team of blind monkeys would be more successful with small projects. Craig Larman has described this nicely in his book Agile & Iterative Development. Sure, I believe that agile practices increase the chance of project success, also for large projects. But that is not what these simple surveys are measuring! They just compare *all* traditional projects versus *all* agile projects. This is unfair, because I believe that research has shown that agile practices are still more often applied to small projects. Traditional approaches are more often used in large projects. (Please don't ask me what research — I am quite sure I have seen it somewhere. If it doesn't exist, I stand corrected and I'll eat my hamster.) So the outcome of the Dr. Dobb's survey is already skewed. Therefore, the results simply don't prove anything.
B. When is a project agile and when is it traditional? If I were to apply half of the agile best practices, and half of any traditional practices, in which category would my project fit then? Agitional? Tragile? I wouldn't know. So I find these polls and survey useless.
I would like to see a survey carried out in the following way:
Make a list of best practices that people can (and do) apply individually (TDD, stand-ups, continuous integration, etc…)
Ask the respondents to answer the questions for a few specific projects. Ask about the project size and let them explain which practices they applied to it.
In the answers received, correlate the best practices with project success rates, and distinguish between different categories of project size.
I am quite sure that many agile best practices will turn out to be major contributors to project successes. I just haven't seen proof of it yet.