Augmented Reality Visualization and Interaction for COVID-19 CT-Scan NN Automated Segmentation: A Validation Study

Abstract

Although medical imaging technology has persisted in evolving over the last decades, the techniques and technologies used for analytical and visualisation purposes have remained constant. Manual or semi-automatic segmentation is, in many cases, complicated. It requires the intervention of a specialist and is time-consuming, especially during the Coronavirus disease (COVID-19) pandemic, which has had devastating medical and economic consequences. Processing and visualising medical images with advanced techniques represent medical professionals’ breakthroughs. This paper studies how augmented reality (AR) and artificial intelligence (AI) can transform medical practice during COVID-19 and post-COVID-19 pandemic. Here we report an augmented reality visualisation and interaction platform; it covers the whole process from uploading chest Ct-scan images to automatic segmentation-based deep learning, 3D reconstruction, 3D visualisation, and manipulation. AR provides a more realistic 3D visualisation system, allowing doctors to effectively interact with the generated 3D model of segmented lungs and COVID-19 lesions. We use the U-Net Neural Network (NN) for automated segmentation. The statistical measures obtained using the Dice score, pixel accuracy, sensitivity, G-mean, and specificity are 0.749, 0.949, 0.956, 0.955, and 0.954, respectively. The user-friendliness and usability are objectified by a formal user study that compared our augmented reality-assisted design to the standard diagnosis setup. One hundred and six doctors and medical students, including eight senior medical lecturers, volunteered to assess our platform. The platform could be used as an aid diagnosis tool to identify and analyse the COVID-19 infectious or as a training tool for residents and medical students. The prototype can be extended to other pulmonary pathologies.