The Decline and Fall of Agilists

There once was a time when being agile was being a visionary. Agile experts wrote books on how adaptive software development is about managing complex systems (Jim Highsmith), why agility is a cooperative game (Alistair Cockburn), and how being agile (or lean) requires a new way of thinking (Mary & Tom Poppendieck). The Agile Manifesto itself was a visionary masterpiece. So simple, so powerful, and so… Well, I wanted to say 'perfect', but nothing is perfect. Not even me.

Then came the agilists…

Nowadays, agilists are telling me that in order to do Scrum or XP or any form of Agile successfully, you must refactor. Sorry, not optional. Necessary. They tell me that everybody needs unit tests, and that Scrum makes things worse by ignoring important (but hard) agile engineering practices, and that you're not agile if you don't have a build integration at least once a day. According to these agilists, being agile is not about being adaptable and doing whatever it takes to make your project a long-lasting success. These days, agilists simply claim that agile is about following practices X, Y and Z. Extreme Programmers often joke that Scrum is just XP without the technical practices that make it work.

And my feelings of inspiration changed into guilt…

We have fully adopted Scrum in our organization, but our adoption of XP practices is still far from completed. Are we now inferior to others? The agilists keep saying that Scrum without XP practices will result in a build-up of technical debt, lower productivity and failed projects. Every time I was reading this, I felt there was something not right about the picture that was being painted. I felt I had to respond, but I didn't know how.

Until today…

Today all our project managers agreed that the introduction of Scrum has increased the success of our projects. I've heard our software developers confirm that (because of Scrum) they now finally have at least a chance to keep technical debt away. A chance they didn't have before. Our customers tell us that our projects are now of higher quality. And last week our CEO complimented everyone in the organization on the measurable progress we've made. (Actually, I'm still doubtful about those hard figures, but who am I to complain when the gold medals are handed out?)

And all that without XP. How?

Why is it that every agilist and their uncle are warning people not to adopt Scrum without XP practices, while our organization seems to be quite successful doing exactly that? Well, to understand what's going on we need to do what the average agilist hasn't done in many years… which is to THINK, and go back to the roots of agility.

I am referring to complexity theory…

The Edge of Chaos

Agile software development has some of its roots in complex adaptive systems theory (also called complexity science). One property of complex systems is that they find themselves at the edge of chaos. That name is so badly chosen, one might think that Microsoft had something to do with it. You see, the edge of chaos is also the edge of order. It is actually the narrow edge between order and chaos. These days, scientists prefer to use the name complexity for that specific part of the spectrum.

In biology, psychology, economics, and other scientific disciplines, complex systems are the most creative, most adaptable and therefore most successful systems of all. In software development it is no different. Agility is about moving software projects to the area of complexity, right between order and chaos. And bla bla bla, there's nothing new about that. Jim Highsmith and Ken Schwaber already wrote about that years ago. So far, so good.

Now, software companies that are used to producing software in an orderly, structured fashion, will find that agile methods push them from the left side of the graph to the right, in the direction of chaos, but (if all is done well) not into chaos. This is probably what most agilists mean when they say that "Scrum is not enough". If you don't adopt a properly balanced set of practices, including technical ones, trying to be agile can harm your organization by moving it to far, from the ordered to the chaotic regime. Some parts of the system (like quality) might go out of control.

But…

Many organizations don't find themselves in the ordered regime. Most organizations that I know find themselves in the chaotic part of the spectrum. (Don't ask me why, either I am attracted to chaos, or it finds me wherever I am.) In these cases, agile methods will push them, at the very least, from the right side of the graph to the left, in the direction of order. Again, this shift must come to a halt when the projects have become complex instead of chaotic. Going too far into the ordered regime is equal to adding needless rigid structures and practices to a system that needs to stay adaptable. A requirement of having to follow practices X, Y and Z, without consideration to environmental context, would be an example of adding such needless bureaucracy. Less elegantly, context is king, and I don't give a rat's ass about Ron's lovely blue convertible. I already have a nice and shiny black monster.

So you see, for chaotic organizations, adoption of Scrum will be good, even without the XP practices. And that's what happened in our organization. We came from the chaotic part of the spectrum, and Scrum helped us to become complex. Sure, Scrum is not enough, but it's much better than nothing at all!

But, wait! I hear you say… in the end the absolute perfect process for a software project would be a combination of Scrum and XP practices, right?

Well… no!

Enter game theory…

Evolutionary Stable Strategies

An evolutionary stable strategy (ESS) is any combination of practices that enables a system to be successful and survive, in some environment. The ESS is a concept defined in game theory, but it applies just as well to economics, biology, psychology, and software development. Evolutionary stable strategies depend on their environment, and on any other ESS-es in that same environment.

With the possible exception of adaptability, there is no other single property that every ESS needs to have in order to survive. Yes, organisms benefit from having eyes and feet, but plants and bacteria seem to be thriving well enough without them. Many companies benefit from marketing and advertising, but some simply don't bother with that kind of stuff. Many software development projects benefit from Automated Tests, On-site Customers and Pair Programming. But there are plenty of others out there who are doing well enough without them, thank-you-very-much.

(Twenty years ago, long before XP, I wrote a bookkeeping program of 100,000 lines of code. I still use it almost every day. And in all these years I have found only two bugs in it. Required XP practices, My Foot!)

Game theory teaches us that there cannot be one best strategy for all participants in an environment. There is no such thing as one specific string of genes that every organism on earth must have. Because, if there was, it would be a weakness that could be exploited by other ESS-es (viruses and parasites, for example). There is also not one business model that beats all others in a capitalist economic system. If there was, everyone would be using it, which would make everyone's behavior predictable, which would be a weakness… Likewise, there is no such thing as one specific sequence of practices that every software project on earth must have to be successful.

If there was one best set of agile practices, it would be a weakness of the system!

In environments full of complex adaptive systems, the weakness of one system will be exploited by the other. That is how these systems co-exist, and co-evolve. When you know exactly how other systems (competing businesses, competing projects) are working internally, they will be predictable, and exploitable, and you can find ways of using that information for your benefit. This is why there will never be one best way of doing software projects. The idea of "required agile practices" borders on the naivety (or arrogance) of thinking that the human genome is the only best possible DNA string in the world. (See: why ants rule the world.)

Agile vs. Agilists

Agility is about staying successful in ever-changing environments. That's it. There's nothing more to it.

This includes defying any expert who claims that practice X is essential for your organization. Because, quite likely, practice X happens to be something this expert is very good at, and is willing to assist you with, for a considerable consultancy fee. (Remember, I was talking about viruses and parasites…)

Some agilists are saying that perhaps we should start by giving up on the "Agile" brand name. It's never been clearly defined, which has allowed a lot of dysfunctional projects to call themselves "Agile." But what these agilists mean, is that it has never been clearly defined which practices are the core of agile. And rightfully so, because there are none! If there were, it would mean prescribing one evolutionary stable strategy for all systems. Which would defeat the very purpose of being agile. Agile has never been some specific set of practices. Nowhere on the Agile Manifesto does it say that you have to do automated testing, pair programming, refactoring, or whatever. In fact, an "agile practice" should be considered a contradiction in terms as soon as people consider it to be mandatory!

It would be reasonable for us to expect from agilists that they understand the basics of complexity theory and game theory. That stuff has been at the very roots of agile software development. If they knew that stuff, they wouldn't be telling us silly things like "do practice X or you won't be agile" and "doing just Scrum will only make things worse". Unfortunately, that is not how things are these days. The glory days of agile visionaries seem to be gone. The agilists squabble over xp vs. scrum, lean vs. scrum, kanban vs. scrum, fdd vs. scrum, and who-knows-what-else-and-my-mother-in-law vs. scrum. Scrum is the norm. So if you find some faults with Scrum, people will probably think you're very smart. Or something.

Oh well, the agilists are not what they used to be.

We are faced with a global armada of agilists who know absolutely nothing of complexity theory and game theory. Yes, they know words like emergence and self-organization, because everybody's using them. But do they know where those words came from? And do any of them know what a phase transition is? Or a fitness landscape? Or an adaptive walk? Or an attractor?

I do. (And I bet most of them don't.)

Some of them are even thinking of dropping the agile name. Well, I couldn't care less, because it doesn't matter if you call it rapid, adaptive, agile, lean, or gobbledegook software development.

The agilists will fade away. But complexity is here to stay…

OK, now it's time to sit back, relax, and enjoy the comments section of this post. Actually no, I will not relax. There's still plenty of stuff to do. Like, for example, figuring out how to introduce those bloody XP practices in our organization. Because I believe that some of them could actually be quite useful for us as well.

(picture by jurvetson)

Follow me on Twitter – Subscribe with a reader – Subscribe by email!

Latest, greatest and favoritest posts:
Simple vs. Complicated vs. Complex vs. Chaotic
Eat Chaos, Poop Order
A Theory of Everything for Software Development