GAN-Based Super Resolution for Accurate 3D Surface Reconstruction from Light Field Skin Images Towards Haptic Palpation

Abstract

The development of vision technology for observation of skin surface and diagnosis of skin disease for preventing secondary infections caused by direct skin touch has consistently been in the medical field spotlight. Many studies have been conducted to acquire three dimensional (3D) data through stereo images, multiple images, and lasers because (3D) data of in-vivo skin image is essential for accurate medical diagnosis. However, stereo vision systems or 3D laser systems for obtaining 3D information require high cost and have high computational complexity, and hence they have not been used universally. Additionally, the use of such systems is still not preferred in the medical field due to limitations on visual decision making. Therefore, a haptic diagnosis system that can blend vision information from a camera and palpation information from a dermatologist has been considered. In this study, we propose a 3D skin surface reconstruction method using a light field camera for haptic rendering and palpation. To achieve this goal, we addressed the low resolution problem, which has been consistently present in light field cameras, through the generative adversarial nets (GANs)-based super resolution method, and exploited the light field system which has been applied only to the object scene for obtaining 3D skin surface texture. Experimental results show that the method proposed in this study is promising and offers sufficient potential for haptic diagnosis.