Style transfer-enabled Sim2Real framework for efficient learning of robotic transesophageal echocardiography using simulation

Abstract

Recently, various machine learning techniques have been proposed to realize automatic ultrasound (US) image analysis for robotic US acquisition tasks. However, obtaining large amounts of real US images for training is usually expensive or even infeasible in some clinical applications, such as transesophageal echocardiography (TEE). An alternative is to build a simulator to generate synthetic US data for training, but the differences between simulated and real US images may result in poor model performance. This work presents a novel Sim2Real framework to learn robotic US image analysis based on simulated data to address the challenges of limited real US data for robotic TEE. A style transfer model is proposed based on unsupervised contrastive learning to convert real US images into the simulation style. Moreover, a task-relevant model is designed to combine CNNs with vision transformers and trained only on the simulated data to generate task-dependent predictions. We demonstrate the effectiveness of our method in an image regression task to predict the probe position based on US images in robotic TEE. Our results show that using only simulated US data and a small amount of unlabelled real data for training, our method can achieve comparable performance to semi-supervised and fully supervised learning methods. Moreover, the effectiveness of our previously proposed CT-based US image simulation method is also indirectly confirmed.