Evaluation of Sepsis Prediction Tool Raises Questions About Proprietary Tools

A new report is sparking debate on the role of transparency in clinical decision support tools, and on the role of such tools in general.

Last week, JAMA Internal Medicine published a study from investigators at the University of Michigan evaluating the Epic Sepsis Model (ESM), a widely used prediction model designed to quickly alert providers at the onset of sepsis. Corresponding author Karandeep Singh, MD, MMSc, and colleagues, said that because of the proprietary nature of ESM, there is a lack of publicly available data about its performance. The investigators therefore sought to conduct an independent analysis of the model to see how it compared with the company’s claims about the product.

“Without independent evaluation, it’s hard to know how much to trust a model’s confidential information sheet, because what is performed and reported in the sheet is determined entirely by the model developer,” Singh told Contagion.

The investigators used a retrospective study design, looking at the experiences of 27,697 adult patients who sought care at the University of Michigan’s health system between December 6, 2018 and October 20, 2019. The patients represented 38,455 hospitalizations. Sepsis occurred in 2,552 of the patients.

Singh and colleagues said the ESM had a hospitalization-level area under the curve (AUC) of 0.63, compared to the company’s publicly reported ACU of 0.76-0.83. The system identified 183 patients with sepsis (7%) who did not receive timely antibiotics, and did not identify 1,709 patients (67%), the report said. This was despite the fact that the system generated alerts for 18% of hospitalizations, creating a “large burden of alert fatigue,” the investigators said.

Singh said one reason for the disparity between the company’s AUC and the investigators’ AUC is the way each defined sepsis onset.

“[W]e arrived at a different result than what is reported by the vendor because of our use of clinically guided definitions of sepsis, rather than billing codes alone,” he said. Using billing codes to define sepsis is less precise and less fitting in a real-world scenario, he argued.

However, the company argued that the investigators’ approach “casts a wide net to include more patients as possibly septic, yielding more false positives.” In a statement provided to Contagion, the company said clinicians will recognize many patients who are septic, but the purpose of the model is to identify those patients whose cases are harder to identify. They noted that the Michigan analysis indeed suggested the model would have identified 183 patients who otherwise might have been missed. In real-world clinical practice, they said, the model can identify patients becoming septic up to 4 hours earlier than a clinician, which they said can mean the difference between life and death. Epic also pointed to other published research that supports the model.

Singh and colleagues have also evaluated other models, including Epic’s Deterioration Index, which they found to be just as effective as the company claimed.

However, he said the evaluations highlight a larger problem, “which is that technical advances in implementing proprietary models have outpaced independent, scientific evaluations of model performance.”

He said the problem is not unique to Epic, but one that has proliferated as hundreds of companies large and small have developed tools like ESM.

“These problems can be corrected, but require more open engagement with clinical groups, such as those in a position to construct clinical guidelines,” he said.

Epic, in its statement, said it works in close collaboration with clients and academics, and other experts to explain and improve its models.

“We are constantly improving implementations of the model including strategies for how to operationalize it to achieve the best possible outcomes,” the company said. “We use this information to evaluate how the model and clinicians together can improve outcomes.”

Singh, however, said while there is a role for tools like ESM, it is also important to understand what value such tools provide. When a hospital or provider is already providing high-quality care, the models may not provide a meaningful benefit to patients.

“Our work highlights that it is important to consider what a clinician would have done in the absence of a model to understand the value that a model would add,” he said.