Open access database for artificial intelligence research

Abstract

Artificial intelligence (AI) for gastrointestinal endoscopy is an important and rapidly developing area of research. In particular, AI research in colonoscopy has attracted significant attention, with several AI medical devices already on the market in the United States and worldwide. With the help of AI models based on machine learning, endoscopists can appreciate improved and operator-independent colonoscopy quality, such as AI-driven detection and characterization of colorectal polyps.1Mori Y. Bretthauer M. Kalager M. Hopes and hypes for artificial intelligence in colorectal cancer screening.Gastroenterology. 2021; 161: 774-777Abstract Full Text Full Text PDF PubMed Scopus (14) Google Scholar The core of machine learning is data, which are used for 2 different purposes: data for developing the AI model and data for evaluating the performance of the developed AI model. Whereas many researchers collect these data from electric hospital records with patient consent, they also have the option of using publicly available databases for research purposes. These publicly available databases are especially important for researchers in the field of informatics because access to healthcare data is quite limited. Regardless of how one collects the data in AI research, the quality and quantity of the data greatly affect the performance of the AI models developed. The optimal data for machine learning should be large in scale and diverse in terms of patient and disease characteristics.2Vinsard D.G. Mori Y. Misawa M. et al.Quality assurance of computer-aided detection and diagnosis in colonoscopy.Gastrointest Endosc. 2019; 90: 55-63Abstract Full Text Full Text PDF PubMed Scopus (70) Google Scholar In this regard, it is very important for both researchers and users to understand what kinds of data have been used to develop and evaluate AI models for colonoscopy. In this issue of Gastrointestinal Endoscopy, Houwen et al3Houwen B. Nass K.J. Vleugels J.L.A. et al.Comprehensive review of publicly available colonoscopic imaging databases for artificial intelligence research: availability, accessibility and usability.Gastrointest Endosc. 2023; 97: 184-199.e15Abstract Full Text Full Text PDF Scopus (0) Google Scholar provide an exciting insight into how to interpret the quality and quantity of publicly available colonoscopy databases. Through a high-quality systematic review, they identified a total of 22 publicly available colonoscopy image databases. One of the authors' goals was to identify the potential limitations of each database to facilitate their appropriate use, which could help accelerate AI research in colonoscopy. The authors encountered several serious problems that could hinder the generalization of research with the use of these databases. These problems include the limited accessibility and variety of the data. Overestimation of AI performance often occurs when the performance of an AI model is evaluated on the basis of such biased, small datasets.4van der Sommen F. de Groof J. Struyvenberg M. et al.Machine learning in GI endoscopy: practical guidance in how to interpret a novel field.Gut. 2020; 69: 2035-2045Crossref PubMed Scopus (48) Google Scholar In relation to this matter, a major problem with currently available databases is the discrepancy between the number of images and the number of different polyps/patients. The authors clearly mentioned this point in their article. Although the number of images registered in these databases often exceeded 10,000, the median number of unique patients per database was only 42, and the median number of unique polyps per database was only 73.5. Only 1 database contained >100 distinct polyps and patients, namely, the POLAR database,3Houwen B. Nass K.J. Vleugels J.L.A. et al.Comprehensive review of publicly available colonoscopic imaging databases for artificial intelligence research: availability, accessibility and usability.Gastrointest Endosc. 2023; 97: 184-199.e15Abstract Full Text Full Text PDF Scopus (0) Google Scholar which had 802 patients and 2069 polyps registered. Obviously, there is a lack of diversity of polyps and patients in the currently available databases in general. The authors also pointed out several ethical concerns with these databases. Given that these databases are to be used for research, they should be compliant with research ethics. In this context, several relevant pieces of information may need to be disclosed, such as how the developers obtained informed consent from patients, how they de-identified the endoscopy images, and when and where they obtained ethics committee approval. However, the situation is discouraging. For example, only 5 of the 22 (32%) databases reported methods for de-identifying the data. In addition, 10 of 22 (45%) databases did not indicate in their publication/database description that they had obtained ethical approval. The absence of this type of information may lead to unexpected issues related to research integrity. In addition, potential legal issues may also occur. Given that the images registered in these databases are completely anonymized, there is a risk that some researchers may use the data even without indicating the source of the data. To avoid this unfavorable situation, a strict data-sharing policy should be discussed. Obtaining a signature on a legal document may be one of the solutions, but it has not been widely adopted. Considering the above drawbacks, it is very important to identify truly reliable databases for AI research. To achieve this goal, the authors proposed a novel method to objectively evaluate database quality. They developed an original checklist to evaluate the usability of databases by referring to a structured metadata reporting checklist, CLAIM (Artificial Intelligence in Medical Imaging).5Mongan J. Moy L. Kahn Jr., C.E. Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers.Radiol Artif. 2020; 2e200029Google Scholar This approach can be an innovative solution to both finding useful databases and developing trustworthy databases for the future. Their proposed checklist includes metadata such as participants’ consent, ethical approval, data collection period, number of participating centers, eligibility criteria, de-identification methods, and number of included patients. The authors' efforts to standardize and improve the quality of databases deserve considerable attention. After all, standardized high-quality data are a prerequisite for developing and accessing a powerful machine learning model. At the same time, we should show respect and gratitude to all contributors to the publicly available databases presented in this study. Their dedication and their painstaking work, often unpaid, are the source of the driving force in science, especially in the field of AI medicine. Data really do matter in health research. Dr Mori is a consultant for, and user of equipment on loan from, Olympus Corp and has an ownership interest in Cybernet System Corp. The other author disclosed no financial relationships.