Traditional versus Agile: Where’s the Proof?

I have a bit of a problem with a recent survey in Dr. Dobb's Journal, carried out by Scott Ambler. The survey is supposed to show that agile projects are more successful than traditional projects. I think this is probably true, but I also think the survey doesn't prove that. I have commented on it on Scott's blog, but I will repeat my argument here, for your convencience (and my own administration):

A. Comparisons between agile and traditional projects should only be made for projects of equal size. Because, by their very nature (complexity) big projects have a higher chance of failure than small projects, whatever best practices you throw at them. Even a team of blind monkeys would be more successful with small projects. Craig Larman has described this nicely in his book Agile & Iterative Development. Sure, I believe that agile practices increase the chance of project success, also for large projects. But that is not what these simple surveys are measuring! They just compare *all* traditional projects versus *all* agile projects. This is unfair, because I believe that research has shown that agile practices are still more often applied to small projects. Traditional approaches are more often used in large projects. (Please don't ask me what research — I am quite sure I have seen it somewhere. If it doesn't exist, I stand corrected and I'll eat my hamster.) So the outcome of the Dr. Dobb's survey is already skewed. Therefore, the results simply don't prove anything.

B. When is a project agile and when is it traditional? If I were to apply half of the agile best practices, and half of any traditional practices, in which category would my project fit then? Agitional? Tragile? I wouldn't know. So I find these polls and survey useless.

I would like to see a survey carried out in the following way:

  1. Make a list of best practices that people can (and do) apply individually (TDD, stand-ups, continuous integration, etc…)
  2. Ask the respondents to answer the questions for a few specific projects. Ask about the project size and let them explain which practices they applied to it.
  3. In the answers received, correlate the best practices with project success rates, and distinguish between different categories of project size.

I am quite sure that many agile best practices will turn out to be major contributors to project successes. I just haven't seen proof of it yet.

  • Competence Development: The Cone of Incompetence
  • Flight of the Software Bugs
Related Posts
free book
“How to Change the World”
  • Jeff Anderson

    I certainly agree that any survey of this nature is difficult to conduct, and subject to bias.
    Adding to the difficulty is the reality that most organizations in traditional markets like banking, manufacturing, and public sector do not follow agile or traditional but a hybrid of the two. At least that is the case for every client that I’ve come across.
    I will add that I think is relevant to compare projects of different sizes, because one of the agile best practices is to shrink the size of a project, using a variety of mechanisms.
    -quality over quantity (in terms of resources and amount of work delivered a specific period of time)
    -focusing on implementing code (instead of a ton of documentation)
    the list goes on. On every large project I’ve ever been on, I’ve always come to the conclusion that I could’ve accomplished the same results with a vastly smaller team. Of course this would mean that time would have to slip, or the scope.
    However this would not necessarily mean that real value would be reduced, as I have also noticed that on most large projects only a very small portion of the actual end software had what I would call “core value”. Again, this type of value gets better realized if you’re putting real code in front of your users more frequently.

  • Jurgen Appelo

    Jeff, thanks for your feedback!
    You are right, agile approaches tend to make projects smaller, which makes it even more difficult to compare agile projects and traditional projects. Nevertheless, I hope some research will someday turn up that digs a level deeper by comparing individual practices.

  • Stefan Billiet

    My girlfriend compels you to not eat George; she insists that imaginary hamsters have feelings too 🙂

How to Change the World - free Workout - free