Transformer-based Feature Matching for Fetoscopic Mosaicking

Abstract

Twin-to-twin transfusion syndrome (TTTS) is a rare fetal anomaly that affects the twins sharing a monochronic placenta. It is caused by abnormal placental vascular anastomoses on the placenta, leading to uneven flow of blood between the two fetuses [1]. Fetoscopic Laser Pho- tocoagulation (FLP) is used to treat TTTS, however, this procedure is hindered because of difficulty in visualizing the surgical environment due to limited surgical field-of- view, unusual placenta position, limited manoeuvrability of the fetoscope and poor visibility due to fluid turbidity and occlusions (Fig. 1. This adds to the surgeon’s cogni- tive load and may result in increased procedural time and missed treatment, leading to persistent TTTS. Fetoscopic video mosaicking can create a virtual expanded field-of- view (FOV) image of the fetoscopic environment, which may support the surgeons in localizing the vascular anastomoses during the FLP procedure. Classical video mosaicking techniques perform handcrafted feature detection, description (i.e. SIFT, SURF, ORB, etc) and feature matching in consecutive frames and homography estimation for image stitching. However, these methods perform poorly on the in vivo fetoscopic videos due to low resolution, poor visibility, floating particles and texture paucity or repetitive texture. Deep learning-based sequential mosaicking [2] method overcomes the limitation of feature-based mosaicking methods, but results in drifting error when stitching non-planar views. A recent intensity-based image registration [3] method relies on placental vessel segmentation maps for registration. This facilitated in overcoming some visibility challenges, however, this method fails when the predicted segmentation map is inaccurate or inconsistent across frames or in views with thin or no vessels. In the paper, we propose the use of transformer-based detector-free local feature matching (LoFTR) method [4] as a dense feature matching technique for creating reliable mosaics with minimal drifting error. Using the publicly available dataset [3], we experimentally show the robustness of the proposed method over the state-of-the-art vessel-based method.