This morning, one of my largest health technology projects experienced several disasters.
It all started with one piece of key information being left out of an initial patient record data set. This omission resulted in several statistical models being wildly inaccurate. On top of this, the quality assurance team used the wrong validation criteria, and our client-side counterparts ignored several automatic warnings generated by our error-checking algorithms. Frustratingly, this event triggered a timeline extension for the project, which originally needed to be delivered by the end of the month. We immediately undertook emergency measures, placing this project and several others in danger of breaking the high-profile promises of executive champions.
If my team was Mission Control, we just witnessed our rocket explode during the launch sequence.
Fortunately, all of these failures were intentionally planned…on a whiteboard.
All of these tragic events took place during a stress test session of our technology implementation within hypothetical client operations. They were part of a series of internal exercises with a large team of stakeholders to identify risks, anticipate failure, and plan for unanticipated events.
In last week’s post, I focused on the importance of identifying success metrics during pilots and how to recognize success when it occurs. It’s equally important to also focus on the opposite of success.
I’ve helped shepherd dozens of health technology implementations in enterprise settings. These stress test and failure planning sessions are an essential, but overlooked, piece of the strategy planning process. While they may take up valuable meeting time, the benefits of these disciplined risk mitigation efforts far outweigh the nominal time they take to complete.
Every operational plan for new health technology deployment should have a stress testing phase, where every stakeholder aggressively challenges the innovation’s viability. Oftentimes, the difference between a smooth execution and total catastrophe rests in our ability to anticipate and prepare for when things go wrong.
When people discuss an innovation, they focus on the incredible benefits and wonderful opportunities that a new process or technology brings. What is left out of the discussion is mention of the inherent risk associated with transition and implementation. Stress testing should systematically address all of the areas of risk associated with the technology. Everything that can go wrong should be envisioned and discussed in an outline of anticipated failures. Afterwards, rigorous contingency and emergency plans should be put into place.
Conducting Stress Testing and Failure Planning
Essentially, project planners should plan to fail by closely examining every factor that contributes to success.
For predictive analytics, this means creating an inventory of all the things that can go wrong, by looking at the downstream events that would be triggered in the event of a particular problem or failure. An inventory of the resulting consequences should also be created. This can be done through data collection, ETL, model training, analysis, and reporting. For example, if a dataset were to significantly change within the initial population, what implications would this have for the overall analysis reporting? How would an organization respond from a crisis management standpoint?
We’ve challenged our underlying assumptions and conducted sensitivity testing for every potential point of project failure. This effort ranges from re-evaluating the basic innovation premise to creating what-if scenarios. Much of this work may be reused for contingency planning, but it’s primary purpose is to make sure that the initial launch occurs smoothly.
From my team’s perspective, our role as Mission Control is to continually check and re-check the launch. It’s prudent to make sure failure planning takes place, so that none of the hypothetical scenarios never become a reality.
Some basic questions that your Mission Control should ask include:
New technology adoption, especially in healthcare, is fraught with risk in both planning and execution. Non-healthcare industries do not have to contend with the number of stakeholders and multi-dimensional pressures that exist within hospital systems and payers. The implementation of new technologies will inevitably introduce a level of uncertainty that organizations typically do not experience on a regular basis. Stress testing and failure planning are essential tools for mitigating innovation execution risk.
By: Wil Yu, VP of Business Development
Wil Yu heads innovation strategy, business, and partnership development for Lumiata. His work focuses on predictive data analytics, care management transformation, and related emerging health technologies. Previously at the U.S. Department of Health and Human Services, he led nationwide healthcare innovation efforts.