A deep learning based fusion of RGB camera information and magnetic localization information for endoscopic capsule robots

Jahanzaib Shabbir,Metin Sitti,Mehmet Turan,Helder Araujo,Ender Konukoglu

doi:10.1007/s41315-017-0039-1

Abstract

A reliable, real time localization functionality is crutial for actively controlled capsule endoscopy robots, which are an emerging, minimally invasive diagnostic and therapeutic technology for the gastrointestinal (GI) tract. In this study, we extend the success of deep learning approaches from various research fields to the problem of sensor fusion for endoscopic capsule robots. We propose a multi-sensor fusion based localization approach which combines endoscopic camera information and magnetic sensor based localization information. The results performed on real pig stomach dataset show that our method achieves sub-millimeter precision for both translational and rotational movements.

Highlights

Robot localization denotes the robot’s ability to establish its position and orientation within the frame of reference
We evaluate the performance of our system both quantitatively and qualitatively in terms of trajectory estimation
The absolute trajectory (ATE) root-mean-square error metric (RMSE) is used for quantitative comparisons, which measures the root-mean-square of Euclidean distances between all estimated endoscopic capsule robot poses and the ground truth poses

Summary

Introduction

Robot localization denotes the robot’s ability to establish its position and orientation within the frame of reference. Localization techniques for endoscopic capsule robots can be categorized into three main groups: electromagnetic wave-based techniques; magnetic field strength-based techniques and hybrid techniquesUmay et al (2017). One of the major advantages of utilizing magnetic field strength-based localization techniques is their successful coupling with magnetic locomotion systems. This could be achieved using magnetic steering, magnetic levitation, and remote magnetic manipulation. Another group of endoscopic capsule robot localization techniques is the hybrid techniques These implement an integration of different sources at once such as RF sensors, magnetic sensors, and RGB sensors. Inspired by the recent success of deep-learning models for processing raw, high-dimensional data, we propose in this paper a sequence-to-sequence deep sensor fusion approach for endoscopic capsule robot localization

System architecture details

Multi‐scale vessel enhancement

Keyframe selection

Magnetic localization system

Deep CNN‐RNN architecture for sensor fusion

Dataset

Trajectory estimation

Conclusion