Tag Archives: Failure Analysis

The Resilience Conundrum

From the Webb Space Telescope to Dishwashing Liquids

The Resilience Conundrum

GUEST POST from Pete Foley

Many of us have been watching the spectacular photos coming from Webb Space Telescope this week. It is a breathtaking example of innovation in action. But what grabbed my attention almost as much as the photos was the challenge of deploying it at the L2 Lagrange point. That not only required extraordinary innovation of core technologies, but also building unprecedented resilience into the design. Deploying a technology a million miles from Earth leaves little room for mistakes, or the opportunity for the kind of repairs that rescued the Hubble mission. Obviously the Webb team were acutely aware of this, and were painstaking in identifying and pre-empting 344 single points of failure, any one of which had the potential to derail it. The result is a triumph.  But it is not without cost. Anticipating and protecting against those potential failures played a significant part in taking Webb billions over budget, and years behind it’s original schedule.

Efficiency versus Adaptability: Most of us will never face quite such an amazing but  daunting challenge, or have the corresponding time and budget flexibility. But as an innovation community, and a planet, we are entering a phase of very rapid change as we try to quickly address really big issues, such as climate change and AI. And the speed, scope and interconnected complexity of that change make it increasingly difficult to build resilience into our innovations. This is compounded because a need for speed and efficiency often drives us towards narrow focus and increased specialization.  That focus can help us move quickly, but we know from nature that the first species to go extinct in the face of environmental change are often the specialists, who are less able to adapt with their changing world. Efficiency often reduces resilience, it’s another conundrum.

Complexity, Systems Effects and Collateral Damage. To pile on the challenges a little, the more breakthrough an innovation is, the less we understand about how interacts at a systems level, or secondary effects it may trigger.  And secondary failures can be catastrophic. Takata airbags, or the batteries in Samsung Galaxy phones were enabling, not core technologies, but they certainly derailed the core innovations.

Designed Resiliency. One answer to this is to be more systematic about designing resilience into innovation, as the Webb team were. We may not be able to reach the equivalent of 344 points of failure, but we can be systematic about scenario planning, anticipating failure, and investing up front in buffering ourselves against risk. There are a number of approaches we can adopt to achieve this, which I’ll discuss in detail later.

The Resiliency Conundrum. But first let’s talk just a little more about the Resilience conundrum. For virtually any innovation, time and money are tight. Conversely, taking time to anticipate potential failures is often time consuming and expensive. Worse, it rarely adds direct, or at least marketable value. And when it does work, we often don’t see the issues it prevents, we only notice them when resiliency fails. It’s a classic trade off, and one we face at all levels of innovation. For example, when I worked on dishwashing liquids at P&G, a slightly less glamorous field than space exploration, an enormous amount of effort went into maintaining product performance and stability under extreme conditions. Product could be transported in freezing or hot temperatures, and had to work extreme water hardness or softness. These conditions weren’t typical, but they were possible. But the cost of protecting these outliers was often disproportionately high.

And there again lies the trade off. Design in too much resiliency, and we are become inefficient and/or uncompetitive. But too little, and we risk a catastrophic failure like the Takata airbags. We need to find a sweet spot. And finding it is still further complicated because we are entering an era of innovation and disruption where we are making rapid changes to multiple systems in parallel. Climate change is driving major structural change in energy, transport and agriculture, and advances in computing are changing how those systems are managed. With dishwashing, we made changes to the formula, but the conditions of use remained fairly constant, meaning we were pretty good at extrapolating what the product would have to navigate. The same applies with the Webb telescope, where conditions at the Lagrange point have not changed during the lifetime of the project. We typically have a more complex, moving target.

Low Carbon Energy. Much of the core innovation we are pursuing today is interdependent. As an example, consider energy. Simply replacing hydrocarbons with, for example, solar, is far more complex than simply swapping one source of energy for another. It impacts the whole energy supply system. Where and how it links into our grid, how we store it, unpredictable power generation based on weather, how much we can store, maintenance protocols, and how quickly we can turn up or down the supply are just a few examples. We also create new feedback loops, as variables such as weather can impact both power generation and power usage concurrently. But we are not just pursuing solar, but multiple alternatives, all of which have different challenges. And concurrent to changing our power source, we are also trying to switch automobiles and transport in general from hydrocarbons to electric power, sourced from the same solar energy. This means attempting significant change in both supply and a key usage vector, changing two interdependent variables in parallel. Simply predicting the weather is tricky, but adding it to this complex set of interdependent variables makes surprises inevitable, and hence dialing in the right degree of resilience pretty challenging.

The Grass is Always Greener: And even if we anticipate all of that complexity, I strongly suspect, we’ll see more, rather than less surprises than we expect.   One lesson I’ve learned and re-learned in innovation is that the grass is always greener. We don’t know what we don’t know, in part because we cannot see the weeds from a distance. The devil often really is in the details, and there is nothing like moving from theory to practice, or from small to large scale to ferret out all of the nasty little problems that plague nearly every innovation, but that are often unfathomable when we begin. Finding and solving these is an inherent part of virtually any innovation process, but it usually adds time and cost to the process. There are reasons why more innovations take longer than expected than are delivered ahead of schedule!

It’s an exciting, but also perilous time to be innovating. But ultimately this is all manageable. We have a lot of smart people working on these problems, and so most of the obvious challenges will have contingencies.   We don’t have the relative time and budget of the Webb Space Telescope, and so we’ll inevitably hit a few unanticipated bumps, and we’ll never get everything right. But there are some things we can do to tip the odds in our favor, and help us find those sweet spots.

  1. Plan for over capacity during transitions. If possible, don’t shut down old supply chins until the new ones are fully established. If that is not possible, stockpile heavily as a buffer during the transition. This sounds obvious, but it’s often a hard sell, as it can be a significant expense. Building inventory or capacity of an old product we don’t really want to sell, and leaving it in place as we launch doesn’t excite anybody, but the cost of not having a buffer can be catastrophic.
  2. In complex systems, know the weakest link, and focus resilience planning on it. Whether it’s a shortage of refills for a new device, packaging for a new product, or charging stations for an EV, innovation is only as good as its weakest link. This sounds obvious, but our bias is to focus on the difficult, core and most interesting parts of innovation, and pay less attention to peripherals. I’ve known a major consumer project be held up for months because of a problem with a small plastic bottle cap, a tiny part of a much bigger project. This means looking at resilience across the whole innovation, the system it operates in and beyond. It goes without saying that the network of compatible charging stations needs to precede any major EV rollout. But never forget, the weakest link may not be within our direct control. We recently had a bunch of EV’s stranded in Vegas because a huge group of left an event at a time when it was really hot. The large group overwhelmed our charging stations, and the high temperatures meant AC use limited the EV’s range, requiring more charging. It’s a classic multivariable issue where two apparently unassociated triggers occur at once.   And that is a case where the weakest link is visible. If we are not fully vertically integrated, resilience may require multiple sources or suppliers to protect against potential failure points we are not aware of, just to protect us against things we cannot control.
  3. Avoid over optimization too early. It’s always tempting to squeeze as much cost out of innovation prior to launch. But innovation by its very nature disrupts a market, and creates a moving target. It triggers competitive responses, changes in consumer behavior, supply chain, and raw material demand. If we’ve optimized to the point of removing flexibility, this can mean trouble. Of course, some optimization is always needed as part of the innovation process, but nailing it down too tightly and too early is often a mistake. I’ve lost count of the number of initiatives I’ve seen that had to re-tool or change capacity post launch at a much higher cost than if they’d left some early flexibility and fine-tuned once the initial dust had settled.
  4. Design for the future, not the now. Again this sounds obvious, but we often forget that innovation takes time, and that, depending upon our cycle-time, the world may be quite different when we are ready to roll out than it was when we started. Again, Webb has an advantage here, as the Lagrange point won’t have changed much even in the years the project has been active. But our complex, interconnected world is moving very quickly, especially at a systems level, and so we have to build in enough flexibility to account for that.
  5. Run test markets or real world experiments if at all possible. Again comes with trade offs, but no simulation or lab test beats real world experience. Whether its software, a personal care product, or a solar panel array, the real world will throw challenges at us we didn’t anticipate. Some will matter, some may not, but without real world experience we will nearly always miss something. And the bigger our innovation, generally the more we miss. Sometimes we need to slow down to move fast, and avoid having to back track.
  6. Engage devils advocates. The more interesting or challenging an innovation is, the easier it is to slip into narrow focus, and miss the big picture. Nobody loves having people from ‘outside’ poke holes in the idea they’ve been nurturing for months or years, but that external objectiveness is hugely valuable, together with different expertise, perspectives and goals. And cast the net as wide as possible. Try to include people from competing technologies, with different goals, or from the broad surrounding system. There’s nothing like a fierce competitor, or people we disagree with to find our weaknesses and sharpen an idea. Welcome the naysayers, and listen to them. Just because they may have a different agenda doesn’t mean the issues they see don’t exist.

Of course, this is all a trade off. I started this with the brilliant Webb Space telescope, which is amazing innovation with extraordinary resilience, enabled by an enormous budget and a great deal or time and resource. As we move through the coming years we are going to be attempting innovation of at least comparable complexity on many fronts, on a far more planetary scale, and with far greater implications if we get it wrong. Resiliency was a critical part of the Webb Telescopes success. But with stakes as high as they are with much of today’s innovation, I passionately believe we need to learn from that. And a lot of us can contribute to building that resiliency. It’s easy to think of Carbon neutral energy, EV’s, or AI as big, isolated innovations. But in reality they comprise and interface with many, many sub-projects. That’s a lot of innovation, a lot of complexity, a lot of touch-points, a lot of innovators, and a lot of potential for surprises. A lot of us will be involved in some way, and we can all contribute. Resiliency is certainly not a new concept for innovation, but given the scale, stakes and implications of what we are attempting, we need it more than ever.

Image Credit: NASA, ESA, CSA, and STScl

Subscribe to Human-Centered Change & Innovation WeeklySign up here to get Human-Centered Change & Innovation Weekly delivered to your inbox every week.