Medical Image Retrieval: Applications and Resources

Abstract

Motivation: Medical imaging is one of the largest data producers in the world and over the last 30 years this production increased exponentially via a larger number of images and a higher resolution, plus totally new types of images. Most images are used only in the context of a single patient and a single time point, besides a few images that are used for publications or in teaching. Data are usually scattered across many institutions and cannot be combined even for the treatment of a single patient. Much knowledge is stored in these medical archives of images and other clinical information and content-based medical image retrieval has from the start aimed at making such knowledge accessible using visual information in combination with text or structured data. With the digitization of radiology that started in the mid 1990s the foundation for broader use was laid out. Problem statement: This keynote presentation aims at giving a historical perspective of how medical image retrieval has evolved from a few prototypes using first only text, then global visual features to the current multimodal systems that can index many types of images in large quantities and use deep learning as a basis for the tools [1,2,3,4]. It also aims at looking at what the place of image retrieval is in medicine, where it is currently still only sparsely used in clinical practice. It seems that it is mainly a tool for teaching and research. Certified medical tools for decision support rather make use of specific approaches for detection and classification. Approach: The presentation follows a systematic review of the domain that includes many examples of systems and approaches that changed over time when better performing tools became available. Medical mage retrieval has evolved strongly, and many tools linked to mage retrieval are now employed as clinical decision support but mainly for detection and classification. Retrieval remains useful but is often integrated with tools and thus has become almost invisible. A second aspect of the presentation includes a presentations of existing data sets and other resources that were difficult to obtain even ten years ago, but that have been shared via repositories such as TCGA (The Cancer Genome Atlas, https://www.cancer.gov/about-nci/organization/ccg/ research/structural-genomics/tcga), TCIA (The Cancer Imaging Archive, https://www.cancerimagingarchive.net), or via scientific challenges such ImageCLEF [5] or listed in the Grand Challenges web page (https://grand-challenge.org). Medical data are now easily accessible in many fields and often even in large quantities. Discussion: Medical retrieval has gone from single text or image retrieval to multimodal approaches [6], really aiming to use all data available for a case, similar to what a physician would do by looking at a patient holistically. The limiting factor in terms of data access is now rather linked to limited manual annotations, as the time of clinicians for annotations is expensive. Global labels for images usually exist with the associated text reports that describe images and outcomes. Still, these weak labels need to be made usable with deep learning approaches that possibly require large amounts of data to generalize well. Conclusions: Medical image retrieval has evolved strongly over the past 30 years and can be integrated with several tools. For real clinical decision support, it is still rarely used, also because the certification process is tedious and commercial benefit is not as easy to show, as with detection or classification in a clear and limited scenario. In terms of research many resources are available that allow advances also in the future. Still, certification and ethical aspects also need to be taken into account to limit risks for individuals.