Deep learning from \u201cpassive feeding\u201d to \u201cselective eating\u201d of real-world data

Zhongwen Li,Chuan Chen,Yu Han,Pisong Yan,Fabao Xu,Chong Guo,Haotian Lin,Yi Zhu,Lanqin Zhao,Chenjin Jin,Danyao Nie,Duoru Lin,Ping Zhang,Xiaohang Wu,Meimei Dongye

doi:10.1038/s41746-020-00350-y

Abstract

Artificial intelligence (AI) based on deep learning has shown excellent diagnostic performance in detecting various diseases with good-quality clinical images. Recently, AI diagnostic systems developed from ultra-widefield fundus (UWF) images have become popular standard-of-care tools in screening for ocular fundus diseases. However, in real-world settings, these systems must base their diagnoses on images with uncontrolled quality (“passive feeding”), leading to uncertainty about their performance. Here, using 40,562 UWF images, we develop a deep learning–based image filtering system (DLIFS) for detecting and filtering out poor-quality images in an automated fashion such that only good-quality images are transferred to the subsequent AI diagnostic system (“selective eating”). In three independent datasets from different clinical institutions, the DLIFS performed well with sensitivities of 96.9%, 95.6% and 96.6%, and specificities of 96.6%, 97.9% and 98.8%, respectively. Furthermore, we show that the application of our DLIFS significantly improves the performance of established AI diagnostic systems in real-world settings. Our work demonstrates that “selective eating” of real-world data is necessary and needs to be considered in the development of image-based AI systems.

Highlights

Artificial intelligence (AI) platforms provide substantial opportunities to improve population health due to their high efficiencies in disease detection and diagnosis[1,2,3,4,5]
There were 679 disputed images that were arbitrated by the senior retina specialist, of which, 223 images were assigned to the poor-quality group, and the remaining 456 images were assigned to the good-quality group
When the deep learning–based image filtering system (DLIFS) is applied in the clinic, photographers can be immediately notified if a poor-quality image is detected, and the photographer can reimage for better quality

Summary

Introduction

Artificial intelligence (AI) platforms provide substantial opportunities to improve population health due to their high efficiencies in disease detection and diagnosis[1,2,3,4,5]. The performances of these systems in detecting ocular fundus diseases are ideal in laboratory settings, their performances in real-world settings are unclear because the systems have to make a diagnosis based on images of varying quality. In the real-world clinic, it is necessary to filter out poor-quality images to ensure that the subsequent AI diagnostic analyses can be based on good-quality images. Manual image quality analysis often requires experienced doctors and can be time-consuming and labourintensive, especially in high-throughput settings (e.g., disease screenings and multicentre studies). An automated approach to detect and filter out poor-quality images becomes crucial

Objectives

Methods

Results

Conclusion