Specular highlight removal for endoscopic images using partial attention network

Abstract

Objective. Endoscopic imaging is a visualization method widely used in minimally invasive surgery. However, owing to the strong reflection of the mucus layer on the organs, specular highlights often appear to degrade the imaging performance. Thus, it is necessary to develop an effective highlight removal method for endoscopic imaging. Approach. A specular highlight removal method using a partial attention network (PatNet) for endoscopic imaging is proposed to reduce the interference of bright light in endoscopic surgery. The method is designed as two procedures: highlight segmentation and endoscopic image inpainting. Image segmentation uses brightness threshold based on illumination compensation to divide the endoscopic image into the highlighted mask and the non-highlighted area. The image inpainting algorithm uses a partial convolution network that integrates an attention mechanism. A mask dataset with random hopping points is designed to simulate specular highlight in endoscopic imaging for network training. Through the filtering of masks, the method can focus on recovering defective pixels and preserving valid pixels as much as possible. Main results. The PatNet is compared with 3 highlight segmentation methods, 3 imaging inpainting methods and 5 highlight removal methods for effective analysis. Experimental results show that the proposed method provides better performance in terms of both perception and quantification. In addition, surgeons are invited to score the processing results for different highlight removal methods under realistic reflection conditions. The PatNet received the highest score of 4.18. Correspondingly, the kendall’s W is 0.757 and the asymptotic significance p = 0.000 < 0.01, revealing that the subjective scores have good consistency and confidence. Significance. Generally, the method can realize irregular shape highlight reflection removal and image restoration close to the ground truth of endoscopic images. This method can improve the quality of endoscopic imaging for accurate image analysis.