Synthetic laparoscopic video generation for machine learning-based surgical instrument segmentation from real laparoscopic video and virtual surgical instruments

Abstract

ABSTRACT This paper proposes a synthetic laparoscopic image generation for machine-learning-based surgical instrument segmentation from laparoscopic videos. Recently surgical instrument extraction methods from laparoscopic videos have been studied using deep learning, fuelling the creation of a large amount of training data for better performance. However, it is difficult to collect a massive amount of data on surgical instruments that are used infrequently during surgery. Their recognition accuracy may be reduced by the lack of training data. This paper solves this problem by increasing the training data with an image synthesis technique. Pairs of synthetic laparoscopic videos and their labelled data are automatically generated by superimposing 3D virtual surgical instrument models on real laparoscopic videos. The synthetic laparoscopic images are translated using CycleGAN so that the appearance of the surgical instruments closely resembles those in the real laparoscopic videos. Additionally, we extracted surgical instruments from laparoscopic videos using 2D U-Net based network. This network was trained using both the synthetic laparoscopic images and the manually labelled, real laparoscopic video data. The trained model extracted the surgical instruments from the laparoscopic videos. Our experimental result showed that the recognition accuracy of the surgical instruments, which are used infrequently during surgery, was improved using synthetic laparoscopic images generated by our proposed method.