Restoration of Lighting Parameters in Mixed Reality Systems Using Convolutional Neural Network Technology Based on RGBD Images

Abstract

One of the main problems of mixed reality devices is the lack of universal methods and algorithms for the visualization of virtual world objects in real space. The key point of natural perception of virtual objects in the real world is the creation of natural lighting conditions for virtual world objects by light sources located in the real world, i.e. the formation of natural glares on virtual objects and shadows cast by these objects in the real world. The paper proposes a method for adequately determining the position of the main light sources of the real world in mixed reality systems. Modern technologies that combine the capability of forming 2.5D images created by depth cameras and their subsequent computer processing using neural networks make it possible to identify real-world objects, recognize their shadows, and correctly restore the light sources that create these shadows. The results of the proposed method are presented, the accuracy of restoring the position of the light sources is estimated, and the visual difference between the image of the scene with the original light sources and the same scene with the restored parameters of the light sources is demonstrated.