Abstract
We propose a novel pathology-sensitive deep learning model (PS-DeVCEM) for frame-level anomaly detection and multi-label classification of different colon diseases in video capsule endoscopy (VCE) data. Our proposed model is capable of coping with the key challenge of colon apparent heterogeneity caused by several types of diseases. Our model is driven by attention-based deep multiple instance learning and is trained end-to-end on weakly labeled data using video labels instead of detailed frame-by-frame annotation. This makes it a cost-effective approach for the analysis of large capsule video endoscopy repositories. Other advantages of our proposed model include its capability to localize gastrointestinal anomalies in the temporal domain within the video frames, and its generality, in the sense that abnormal frame detection is based on automatically derived image features. The spatial and temporal features are obtained through ResNet50 and residual Long short-term memory (residual LSTM) blocks, respectively. Additionally, the learned temporal attention module provides the importance of each frame to the final label prediction. Moreover, we developed a self-supervision method to maximize the distance between classes of pathologies. We demonstrate through qualitative and quantitative experiments that our proposed weakly supervised learning model gives a superior precision and F1-score reaching, 61.6% and 55.1%, as compared to three state-of-the-art video analysis methods respectively. We also show our model’s ability to temporally localize frames with pathologies, without frame annotation information during training. Furthermore, we collected and annotated the first and largest VCE dataset with only video labels. The dataset contains 455 short video segments with 28,304 frames and 14 classes of colorectal diseases and artifacts. Dataset and code supporting this publication will be made available on our home page.
Highlights
There are several colorectal diseases and abnormalities that interfere with the normal working of the colon
We exploit the ordering of the attention weights to minimize the similarities between high and low attention frames by training a two-layered neural network which acts as a self-supervision method
We show that by using within bag similarity as self-supervision, we can boost the performance of frame localization and video capsule endoscopy (VCE) video classification
Summary
There are several colorectal diseases and abnormalities that interfere with the normal working of the colon. A number of image processing based methods have addressed colorectal pathology detection problems in literature (Mohammed et al, 2018b; Bernal et al, 2017; Tajbakhsh et al, 2015; Ronneberger et al, 2015) These methods do not consider long-term temporal dependencies between frames to improve the performance of detection algorithms. The dataset used in training such models lacks class variety to be used in clinical application To address these challenges, we propose PS-DeVCEM, a new weakly supervised learning approach for learning frame-level multi-label classification from a given video label. We exploit the ordering of the attention weights to minimize the similarities between high and low attention frames by training a two-layered neural network which acts as a self-supervision method.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
