Resilience Engineering #2: The Danger of Drift

Last week introduced resilience engineering and started with two critical concepts, robustness and resilience, with robust systems being unchanged but pushed to provide performance in a challenging environment and resilient systems adapting to the challenge and evolving.

This week we’ll look at the costs that can accrue with robust systems and the potential for introducing a potentially dangerous behavior called drift. Before getting to drift a little background information will help.

Resilience engineering is especially useful in resource-limited, constrained situations; situations where trade-offs must be made almost on an ongoing basis. One such trade-off that must be considered is how far to push the current system in terms of both technology and people vs. making necessary changes.

The Importance of Time Horizons

The distance to the client’s, customer’s, senior manager’s, or any other powerful stakeholder’s time horizon has a big influence on whether or not a robust or resilient approach is used. A client situation that had very real consequences might help explain. The client firm wanted to purchase another company. Due diligence was performed. However, it was constrained by shortsightedness. The client wanted to enter the market and generated emotionality regarding the issue. As the urge to purchase increased so did the shortsightedness.

The financials looked fine. The concern for the client was the physical plant, comprising four locations, needed addressed to the same depth. The shortsightedness mentioned earlier won out and the purchase went through. The entire situation ended up slowly turning into a nightmare ending with the client selling at a very reduced price several years later to get out of the situation.

The Cost of Robust Behavior

What had occurred was a classic case of the seller making the company look enticingly resilient while actually working it in a robust manner. A simple metaphor for the situation is brakes on a car. Imagine you want to buy a used car and you ask, “Do the brakes work?” Truth be told; the answer is, “Yes, they always have.” Sounds good. But what if a different question were asked, “How much life does the braking system have left in it?” That might yield a totally different answer, e.g., the rotors/drums need replaced, the hydraulic fittings are corroded and will need replaced if a wrench is put to them, etc., etc. In actuality the braking system is on the verge of failure and an additional $1,000 or more is needed to make needed repairs…and the repairs can’t be piecemeal, the entire system needs replaced at the same time. However, if the purchaser is satisfied with the fact the car has always stopped well in the past then the issue of overuse, of being pushed beyond a safe limit, will be missed and the dreams of where this car can take him will continue. This overuse of the braking system to the point where it is close to being a safety issue is called drift. Formally,

Drift is the incremental movement of a system towards, and eventual crossing of, a failure boundary. This all occurring while belief is maintained that all is well.

Using the brake metaphor, the seller had let the system (physical plant) drift towards failure while increasing performance pressure in order to be able to say, “See, it is giving the desired results.” In the previous blog it was mentioned there was a cost associated with this behavior. In this case it was an insidious cost. Money that should have gone into maintaining the physical plant was shifted towards the bottom line.

The seller pushed the system to perform in a robust manner, i.e., continue generating profit and have them falsely increase by siphoning off money needed for the physical plant to maintain and adapt to the increasing performance pressure.

This made the purchase look that much more enticing causing the client, only looking at the bottom line and blinded by emotionality, to pay too much, essentially taking a mortgage to cover profits extracted by the seller – profits that really weren’t profits but maintenance dollars. On top of that the needed repairs and equipment costs still needed to be incurred.

Another issue was inability to grow since there was no resilience. With the assumption that the plant was fine the belief that current systems could be integrated into the changes envisioned turned out to be bankrupt. Not only did current systems need work, they were close to obsolescence.

Probably enough said for such a dark and dreary topic. Next week we will look at a brighter story, a story where a firm split but did it amicably through a resilient approach.