One Algorithm May Not Fit All: How Selection Bias Affects Machine Learning Performance.

Abstract

Machine learning (ML) algorithms have demonstrated high diagnostic accuracy in identifying and categorizing disease on radiologic images. Despite the results of initial research studies that report ML algorithm diagnostic accuracy similar to or exceeding that of radiologists, the results are less impressive when the algorithms are installed at new hospitals and are presented with new images. This phenomenon is potentially the result of selection bias in the data that were used to develop the ML algorithm. Selection bias has long been described by clinical epidemiologists as a key consideration when designing a clinical research study, but this concept has largely been unaddressed in the medical imaging ML literature. The authors discuss the importance of selection bias and its relevance to ML algorithm development to prepare the radiologist to critically evaluate ML literature for potential selection bias and understand how it might affect the applicability of ML algorithms in real clinical environments. ©RSNA, 2020.