Restoration of light parameters and forming criteria for the quality estimation of visual perception in mixed reality systems using neural networks

Abstract

One of the main problems of mixed reality devices is the physically correct representation of the luminance distribution for virtual objects and their shadows in the real world. In other words, restoring the correct distribution of scene luminance is one of the key parameters that allows solving the problem of correct interaction between the virtual and real worlds. The paper proposes methods for restoring the parameters of light sources. Also, the work is aimed at the creation and formation of criteria for the quality of visual perception, allowing to evaluate the synthesized image of mixed reality and to decide how natural it is from the point of view of the observer. In this article, surveys were used to create datasets using realistic software tools. In work the neural network is trained to recognize those areas of the image that do not fit into the environment and to classify this image as a class that causes visual discomfort. As criteria for the quality of visual perception, it is proposed to use estimates of the mismatch of the parameters of shadows cast by virtual objects and the distribution of luminance over these objects in the images of scenes containing models of "real" and "virtual" objects. The level of misalignment is estimated concerning the true lighting conditions of the real world. In this work, criteria for the quality of visual perception were formed and a neural network was trained, which makes it possible to decide and analyze the quality of a synthesized mixed reality image.