Automatic image and text-based description for colorectal polyps using BASIC classification

Roger Fonollà,Quirine E.W Van Der Zander,Ramon M Schreuder,Sharmila Subramaniam,Pradeep Bhandari,Ad A.M Masclee,Erik J Schoon,Fons Van Der Sommen,Peter H.N De With

doi:10.1016/j.artmed.2021.102178

Abstract

Colorectal polyps (CRP) are precursor lesions of colorectal cancer (CRC). Correct identification of CRPs during in-vivo colonoscopy is supported by the endoscopist's expertise and medical classification models. A recent developed classification model is the Blue light imaging Adenoma Serrated International Classification (BASIC) which describes the differences between non-neoplastic and neoplastic lesions acquired with blue light imaging (BLI). Computer-aided detection (CADe) and diagnosis (CADx) systems are efficient at visually assisting with medical decisions but fall short at translating decisions into relevant clinical information. The communication between machine and medical expert is of crucial importance to improve diagnosis of CRP during in-vivo procedures. In this work, the combination of a polyp image classification model and a language model is proposed to develop a CADx system that automatically generates text comparable to the human language employed by endoscopists. The developed system generates equivalent sentences as the human-reference and describes CRP images acquired with white light (WL), blue light imaging (BLI) and linked color imaging (LCI). An image feature encoder and a BERT module are employed to build the AI model and an external test set is used to evaluate the results and compute the linguistic metrics. The experimental results show the construction of complete sentences with an established metric scores of BLEU-1 = 0.67, ROUGE-L = 0.83 and METEOR = 0.50. The developed CADx system for automatic CRP image captioning facilitates future advances towards automatic reporting and may help reduce time-consuming histology assessment.

Highlights

Colorectal polyps (CRP) are precursor lesions and indicators of colorectal cancer (CRC)
We present a CADx system, that incorporates an image captioning block based on the Bidirectional Encoder Representations from Transformers (BERT) language model [44]
The first group consisted of six expert endoscopists from the international blue light imaging (BLI)-expert group, who were knowledgeable in using BLI and BLI Adenoma Serrated International Classification (BASIC) [19,48] (Table 1) and had an experience of more than 2000 colonoscopies

Summary

Introduction

Colorectal polyps (CRP) are precursor lesions and indicators of colorectal cancer (CRC). Current medical pro tocols dictate that all detected CRPs should be resected to undergo histological evaluation, but this protocol has two considerable drawbacks, since (1) unnecessary removal of benign polyps exposes the patient to additional risks of polypectomy-related complications, and (2) histological examination of all resected polyps leads to significantly increased costs. To minimize both the cost and risk, the strategy of resectand-discard has been proposed for diminutive (≤5 mm) adenomatous polyps [2,3,4] and the diagnose-and-leave strategy for diminutive hyper plastic polyps [5,6] in the left colon

Methods

Results

Discussion

Conclusion