The “Optimal” Fallacy

You cannot “optimize the whole”. The best you can do is sub-optimize, cooperate, and iterate.

Allow me to share with you the trouble of speaking at conferences and organizing courses. When conference organizers invite speakers, the optimal thing for them to do is to make speakers commit to their conference as early as possible, while the organizers themselves try to defer their own decisions for as long as possible. At the same time speakers who organize courses want to commit to conferences as late as possible, while trying to get participants for courses to commit as early as possible.

As a speaker and trainer my daily life is a neverending stream of decisions that all tend to be compromises between customers, partners, suppliers, and other stakeholders. People ask me for decisions about speaking engagements, while I’m waiting for courses to be finalized. I attempt to have partners commit to courses, while they defer decisions waiting for participants. We are all selfish bastards, optimizing for ourselves. And therefore we are all sub-optimizing.

There is no other way.

Optimize the Whole

There exists a Lean principle called “Optimize the Whole”, which is too often interpreted as an instruction to only optimize “the whole of the system” and to refrain from optimizing subsystems in an organization. Again and again I come across suggestions to “consider the whole value stream”, to “measure results end-to-end”, and to “focus on customer value”. The assumption is that the customer’s view on a business is one that includes the whole system and therefore a relentless focus on customer value delivery will “optimize the whole”.

This is not true.

A business is a complex system involving customers, employees, shareholders, suppliers, partners, local communities, and natural resources. (And a number of very confused managers.) They all try to optimize for themselves. No observer of a system can claim to have an objective view on the whole, wrote Jerry Weinberg in An Introduction to General Systems Thinking. And therefore, the customer’s view differs from the shareholder’ view. The shareholder’s view differs from the view of the local community, and so on, and so on. Movements like stakeholder management and sustainable development attempt to address the need for reconciliation of multiple views on businesses, from different perspectives.

The Optimizing Whole

We cannot “optimize the whole” by focusing only on the customer. The best we can do is sub-optimize, negotiate, cooperate, and re-optimize, in a neverending stream of compromises. All the parts try to optimize for themselves, and through interdependencies between the parts the whole system tends to evolve to an optimal situation, wrote Stuart Kauffman in At Home in the Universe. That’s how complex systems optimize a whole. Not as a principle to be followed, but as the outcome of interdependent parts, cooperation, and iteration.

The optimizing whole is emergent, not directed.

I call this kind of reasoning complexity thinking, which is slightly different from systems thinking. Some systems thinkers and Lean gurus seem to optimize their own consultancy firms instead of teaching people the nature of complex adaptive systems. But I get paid for helping people think, instead of selling solutions. Lean tools are nice, but counterproductive when applied without proper thinking, wrote John Seddon in Freedom from Command & Control.

Feel free to contact me and add an invitation to my very sub-optimal calendar.

  • 8 Reasons NOT to do an Agile Management Course
  • Top 15 Systems Thinking Books
Related Posts
free book
“How to Change the World”
  • Glen B. Alleman

    While anecdotally interesting and possibly anecdotally useful, you may want to look to the actual discipline of systems engineering as practiced by Systems Engineers, for the discussion of the issues around optimization and sub-optimization, rather than the populist view of this topic.
    This topic is something we deal with in our aerospace and defense software intensive domain, with constrained resources (but people and machines), size, weight, performance, and “customers” demanding ever more capabilities for their emerging mission needs.

  • Jurgen Appelo

    The paper you are referring to deals with _designed_ systems.
    I am referring to complex systems, which are not designed by a single authority.
    The fact that you refer my sources (all professors and academic researchers) as “populist views” reveals a lot about your level of thinking.

  • Nick Gall

    Glen, I skimmed the paper you cite and for me the showstopper is the requirement of a perfect rank order of performance preferences agreed to by a set of stakeholders with diverse interests. This is the essential point Jurgen is making.
    The reason it’s a showstopper is Arrow’s Impossibility Theorem: . “In short, the theorem proves that no voting system can be designed that satisfies these three “fairness” criteria:
    If every voter prefers alternative X over alternative Y, then the group prefers X over Y.
    If every voter’s preference between X and Y remains unchanged, then the group’s preference between X and Y will also remain unchanged (even if voters’ preferences between other pairs like X and Z, Y and Z, or Z and W change).
    There is no “dictator”: no single voter possesses the power to always determine the group’s preference.”
    Since there is no way to fairly aggregate multiple stakeholder preferences into a single ranked list of performance preferences (ie PR0), there is no way to agree on an “optimal” system across all stakeholders. QED.

  • Glen B. Alleman

    The condition of “designed by a single authority” would seem to limit the system to the simple.
    By populist I mean non-mathematically based. And you are observant, that is my level of thinking, since I earn my living doing Systems Engineering and the Program Management activities around Systems Engineering as found in the discipline described in

  • Glen B. Alleman

    I’m speak in the domain of technical systems. You seem to be speaking in the domain of non-technical systems. A common gap on this blog. My mistake.
    However, it may be interesting to explore the “practice” of system design as practiced in today’s world of Systems Engineering beyond an untested 1951 thesis.

How to Change the World - free Workout - free