Machine Learning Models for COVID-19 Not Yet Suitable for Clinical Use

A recent systematic review of a host of scientific manuscripts, conducted by investigators from the University of Cambridge, has found that machine learning models for detecting or diagnosing COVID-19 are not yet suitable compared to standard medical imaging. The research was published in the journal Nature Machine Intelligence.

"In the early days of the pandemic, there was such a hunger for information, and some publications were no doubt rushed," James Rudd, a co-author on the study said. "But if you're basing your model on data from a single hospital, it might not work on data from a hospital in the next town over: the data needs to be diverse and ideally international, or else you're setting your machine learning model up to fail when it's tested more widely."

For the review, the investigators identified over 2,000 studies published between January and October of 2020, that claimed an ability to diagnose or prognosticate for COVID-19 from chest radiographs (CXR) and computed tomography (CT) images. While many did undergo peer review, the majority of them did not.

Of identified studies, the investigators narrowed the number down to 62 after several screenings. They found that none of the studies had the potential for clinical use due to biases, methodological flaws, lack of reproducibility, and erroneous datasets.

"Whether you're using machine learning to predict the weather or how a disease might progress, it's so important to make sure that different specialists are working together and speaking the same language, so the right problems can be focused on," Michael Roberts, a co-author on the study said.

In many of the studies, the did not specify where they had gotten their data, or if the models and tests were used from the same data, which made it impossible to reproduce the initial results.

However, the investigators stress that this area of research shows much promise with some key modifications and can potentially be a powerful tool in the fight against the pandemic.

"The international machine learning community went to enormous efforts to tackle the COVID-19 pandemic using machine learning," Rudd said. "These early studies show promise, but they suffer from a high prevalence of deficiencies in methodology and reporting, with none of the literature we reviewed reaching the threshold of robustness and reproducibility essential to support use in clinical practice."