Visual Discomfort Prediction Research Articles

Multi-task visual discomfort prediction model for stereoscopic images based on multi-view feature representation

Multi-task visual discomfort prediction model for stereoscopic images based on multi-view feature representation

Bi-disparity sparse feature learning for 3D visual discomfort prediction

Viewing stereoscopic images sometimes causes viewers to feel inconvenience, which is called 3D visual discomfort. Excessive horizontal disparity, misalignment between the left and right views, or depth cues conflicts are some of the important factors involved in 3D visual discomfort. The ability to estimate the degree of 3D visual discomfort can be used to improve the 3D display systems and provide acceptable binocular visual quality. Most of the existing visual discomfort prediction (VDP) approaches extract hand-crafted features based on perceptual modeling and statistical analysis of disparities. We have proposed a simple yet effective VDP model based on unsupervised learning of sparse features which are highly predictive of subjective discomfort levels. These features are extracted from the aggregation of left and right disparity maps. This aggregation effectively highlights the areas with sudden changes and high levels of disparities where discomfort is most likely to occur. The regression model trained by the features, predicts high correlated 3D visual discomfort scores on each dataset. The cross-database results are also superior to other reported ones.

A Risk-Aware Pairwise Rank Learning Approach for Visual Discomfort Prediction of Stereoscopic 3D

For visual discomfort prediction (VDP) of stereoscopic 3D images, a common two-stage framework is to first extract features that are predictive of the experienced visual discomfort level when viewing stereoscopic images and then use typical regression tools to learn the mapping from the extracted features to visual discomfort scores. Most existing approaches for stereoscopic 3D VDP focus on the former stage, i.e., feature extraction, while limited efforts have dedicated to exploiting more powerful and robust learning algorithms in this field. In this letter, inspired by the pairwise comparison-based subjective evaluation methodology, we propose a novel Risk-Aware Pairwise Rank Learning (RAPRL) approach to further improve the prediction accuracy. Unlike the traditional VDP approaches using different regression tools for feature-score mapping, our proposed RARL method addresses this problem based on a completely different pairwise rank learning framework with a risk-aware constraint. Experiments have verified the effectiveness and robustness of our proposed VDP model using RAPRL as the learning algorithm.

Optimal Region Selection for Stereoscopic Video Subtitle Insertion

Stereoscopic subtitle insertion is a fundamental and essential element in stereoscopic film and TV industry. However, little work has been dedicated to the optimal region selection for stereoscopic subtitle insertion. In addition, there is no public database reported for the performance evaluation of it. In this paper, we build the first large-scale video database (TJU3D) for stereoscopic video subtitle insertion, which includes 50 video sequences with rich screen scenes. Compared with 2D subtitle region selection, there are several problems we have to consider in stereoscopic subtitle region selection: 1) the subtitle should avoid depth cue collision and occlusion from objects in stereoscopic video sequences; 2) the disparity value of the subtitle must be minimized to reduce visual discomfort; and 3) the temporal coherence constraint must be considered during region selection for subtitles in video sequences. By considering these constraints, we propose an optimal region selection algorithm for stereoscopic subtitle insertion. First, we compute the disparity map of each video frame in video sequences. For each frame, the optimal position and disparity value of the subtitle are determined by a subtitle region selection algorithm, which contains two parts (i.e., the coarse selection and fine selection). After that, by considering the temporal consistency between adjacent frames, the position and disparity value of each frame are further classified and processed in order to avoid the subtitle jitter. We evaluate the proposed method on TJU3D video database through two visual discomfort prediction metrics and one subjective experiment. To further verify the effectiveness of the proposed method, we also validate the performance of the proposed method on video comfort assessment database, i.e., IEEE-SA Stereo Database. Experimental results demonstrate that the visual discomfort is greatly reduced when using the proposed method compared with the basic method.

Deep Visual Discomfort Predictor for Stereoscopic 3D Images.

Most prior approaches to the problem of stereoscopic 3D (S3D) visual discomfort prediction (VDP) have focused on the extraction of perceptually meaningful handcrafted features based on models of visual perception and of natural depth statistics. Towards advancing performance on this problem, we have developed a deep learning based VDP model named Deep Visual Discomfort Predictor (DeepVDP). DeepVDP uses a convolutional neural network (CNN) to learn features that are highly predictive of experienced visual discomfort. Since a large amount of reference data is needed to train a CNN, we develop a systematic way of dividing S3D image into local regions defined as patches, and model a patch-based CNN using two sequential training steps. Since it is very difficult to obtain human opinions on each patch, instead a proxy ground-truth label that is generated by an existing S3D visual discomfort prediction algorithm called 3D-VDP is assigned to each patch. These proxy ground-truth labels are used to conduct the first stage of training the CNN. In the second stage, the automatically learned local abstractions are aggregated into global features via a feature aggregation layer. The learned features are iteratively updated via supervised learning on subjective 3D discomfort scores, which serve as ground-truth labels on each S3D image. The patchbased CNN model that has been pretrained on proxy groundtruth labels is subsequently retrained on true global subjective scores. The global S3D visual discomfort scores predicted by the trained DeepVDP model achieve state-of-the-art performance as compared to previous VDP algorithms.

3D visual discomfort predictor based on subjective perceived-constraint sparse representation in 3D display system

Three-dimensional (3D) display systems have been widely adopted due to the recent increased availability of an increasing 3D contents. However, viewers may experience visual discomfort due to the limited viewing zone available of 3D display systems. Therefore, 3D visual discomfort prediction is important for optimizing 3D display systems. In this paper, we propose a 3D visual discomfort predictor (3D-VDP) that is based on the visual discomfort features of the primary visual cortex (V1) and the properties of subjective perceived-constraint sparse representation (SPCSR). Embedding subjective values of visual discomfort as a constraint into sparse representation such that the dictionary is more suitable for visual perception is the major technical contribution of this study. Specifically, the proposed 3D-VDP with SPCSR consists of two phases. In the training phase, first, the neural activity mechanism of V1 is considered, and the features of visually important disparity and spatial frequency disparity are extracted to highlight the influence of disparity on the comfort of stereoscopic images. Second, by considering the visual properties of the receptive field and learning mechanism, a perceived dictionary of visual discomfort and the corresponding perceived value of visual discomfort are obtained based on the subjective value of visual discomfort as a constraint condition applied to the construction of a supervised dictionary learning algorithm. In the testing phase, the sparse coefficient of visual discomfort of the stereoscopic image is computed according to the perceived dictionary of visual discomfort by using the sparse coding algorithm, and the final visual discomfort score of the stereoscopic image is obtained from the weighted sparse coefficients and the perceived value of visual discomfort. Experimental results obtained with the IVY LAB database and the NBU database demonstrate that, in comparison with closely related existing models, the proposed 3D-VDP with SPCSR achieves a high consistency of subjective assessment.

Saliency Detection of Stereoscopic 3D Images with Application to Visual Discomfort Prediction

Visual saliency detection is potentially useful for a wide range of applications in image processing and computer vision fields. This paper proposes a novel bottom-up saliency detection approach for stereoscopic 3D (S3D) images based on regional covariance matrix. As for S3D saliency detection, besides the traditional 2D low-level visual features, additional 3D depth features should also be considered. However, only limited efforts have been made to investigate how different features (e.g. 2D and 3D features) contribute to the overall saliency of S3D images. The main contribution of this paper is that we introduce a nonlinear feature integration descriptor, i.e., regional covariance matrix, to fuse both 2D and 3D features for S3D saliency detection. The regional covariance matrix is shown to be effective for nonlinear feature integration by modelling the inter-correlation of different feature dimensions. Experimental results demonstrate that the proposed approach outperforms several existing relevant models including 2D extended and pure 3D saliency models. In addition, we also experimentally verified that the proposed S3D saliency map can significantly improve the prediction accuracy of experienced visual discomfort when viewing S3D images.

Visual discomfort prediction on stereoscopic 3D images without explicit disparities

Almost all existing 3D visual discomfort prediction models are based, at least in part, on features that are extracted from computed disparity maps. These include such estimated quantities such as the maximum disparity, disparity range, disparity energy and other measures of the disparity distribution. A common first step when implementing a 3D visual discomfort model is some form of disparity calculation, whence the accuracy of prediction largely depends on the accuracy of the disparity result. Unfortunately, most algorithms that compute disparity maps are expensive, and are not guaranteed to deliver sufficiently accurate or perceptually relevant disparity data. This raises the question of whether it is possible to build a 3D discomfort prediction model without explicit disparity calculation. Towards this possibility, we have developed a new feature map, called the percentage of un-linked pixels (PUP), that is descriptive of the presence of disparity, and which can be used to accurately predict experienced 3D visual discomfort without the need for actually calculating disparity values. Instead, PUP features are extracted by predicting the percentage of un-linked pixels in corresponding retinal patches of image pairs. The un-linked pixels are determined by feature classification on orientation and luminance distributions. Calculation of PUP maps is much faster than traditional disparity computation, and the experimental results demonstrate that the predictive power attained using the PUP map is highly competitive with prior models that rely on computed disparity maps.

Experimental validation of ray tracing as a means of image-based visual discomfort prediction

Image-based visual discomfort analysis has strong potential to detect glare in order to predict occupant satisfaction with a space. However, in order to design buildings with occupant visual comfort in mind, architects need the ability to predict glare through renderings, rather than photographs. In this paper, we compare glare measurements from high dynamic range photographs and computer renderings of the same daylit space under 240 clear and 38 cloudy sky conditions in order to assess how accurately architects can predict glare though simulation. We use two ray tracing engines, Radiance and Accelerad, on scenes illuminated by a mathematically defined sky model or by recorded outdoor luminance levels from photographs. Depending on the scene, these methods predict daylight glare probability levels due to bright sources with between 93% and 99% accuracy and discomfort glare due to contrast with between 71% and 99% accuracy. Glare predictions are more sensitive to modeling and measurement accuracy than to simulation parameters, so accurate results can come from relatively fast simulations. Furthermore, with graphics acceleration, we are able to produce accurate glare predictions within minutes using Accelerad, achieving a speedup over Radiance of between 16 and 44 times.

Stereoscopic 3D Visual Discomfort Prediction: A Dynamic Accommodation and Vergence Interaction Model.

The human visual system perceives 3D depth following sensing via its binocular optical system, a series of massively parallel processing units, and a feedback system that controls the mechanical dynamics of eye movements and the crystalline lens. The process of accommodation (focusing of the crystalline lens) and binocular vergence is controlled simultaneously and symbiotically via cross-coupled communication between the two critical depth computation modalities. The output responses of these two subsystems, which are induced by oculomotor control, are used in the computation of a clear and stable cyclopean 3D image from the input stimuli. These subsystems operate in smooth synchronicity when one is viewing the natural world; however, conflicting responses can occur when viewing stereoscopic 3D (S3D) content on fixed displays, causing physiological discomfort. If such occurrences could be predicted, then they might also be avoided (by modifying the acquisition process) or ameliorated (by changing the relative scene depth). Toward this end, we have developed a dynamic accommodation and vergence interaction (DAVI) model that successfully predicts visual discomfort on S3D images. The DAVI model is based on the phasic and reflex responses of the fast fusional vergence mechanism. Quantitative models of accommodation and vergence mismatches are used to conduct visual discomfort prediction. Other 3D perceptual elements are included in the proposed method, including sharpness limits imposed by the depth of focus and fusion limits implied by Panum's fusional area. The DAVI predictor is created by training a support vector machine on features derived from the proposed model and on recorded subjective assessment results. The experimental results are shown to produce accurate predictions of experienced visual discomfort.

3D Visual Discomfort Prediction: Vergence, Foveation, and the Physiological Optics of Accommodation

To achieve clear binocular vision, neural processes that accomplish accommodation and vergence are performed via two collaborative, cross-coupled processes: accommodation-vergence (AV) and vergence-accommodation (VA). However, when people watch stereo images on stereoscopic displays, normal neural functioning may be disturbed owing to anomalies of the cross-link gains. These anomalies are likely the main cause of visual discomfort experienced when viewing stereo images, and are called Accommodation-Vergence Mismatches (AVM). Moreover, the absence of any useful accommodation depth cues when viewing 3D content on a flat panel (planar) display induces anomalous demands on binocular fusion, resulting in possible additional visual discomfort. Most prior efforts in this direction have focused on predicting anomalies in the AV cross-link using measurements on a computed disparity map. We further these contributions by developing a model that accounts for both accommodation and vergence, resulting in a new visual discomfort prediction algorithm dubbed the 3D-AVM Predictor. The 3D-AVM model and algorithm make use of a new concept we call local 3D bandwidth (BW) which is defined in terms of the physiological optics of binocular vision and foveation. The 3D-AVM Predictor accounts for anomalous motor responses of both accommodation and vergence, yielding predictive power that is statistically superior to prior models that rely on a computed disparity distribution only.

Experimental investigation of discomfort combination: toward visual discomfort prediction for stereoscopic videos

Automatic prediction of visual discomfort for stereoscopic videos is important to address the viewing safety issues in stereoscopic displays. In literature, disparity and motion characteristics have been known as major factors of visual discomfort caused by stereoscopic content. For developing an algorithm to accurately predict visual discomfort in stereoscopic videos, this study investigates how to combine the individual prediction values of disparity- and motion-induced discomforts into an overall prediction value of visual discomfort. To that end, subjective experiments were performed with various stereoscopic videos, and four possible combination methods were compared based on the results of subjective experiments. The combination methods used in this study were the weighted summation, multiplication, Minkowski summation, and max combination methods. Experimental results showed that the Minkowski summation combination with a high exponent and max combination yielded the best accuracy in predicting the overall level of visual discomfort. The results indicated that the overall level of perceived discomfort could be dominantly affected by the most significant discomfort factor, i.e., the winner-takes-all mechanism. Our results could be useful for the future development of an accurate and reliable algorithm for visual discomfort prediction.

Visual discomfort visualizer using stereo vision and time-of-flight depth cameras

Visual discomfort monitoring is an important technique for comfortable viewing of stereoscopic images. This paper proposes a novel monitoring system of visual discomfort that takes advantages of a fusion camera system equipped with stereo vision cameras and a time-of-flight camera. We improve disparity map enhancement technique to obtain a high-quality disparity map that is aligned to the right view color image coordinate. The captured stereo-plusdisparity images are used for visual discomfort visualizer that automatically predicts overall degree of perceived visual discomfort and visualizes which regions are problematic in terms of visual discomfort. The proposed visual discomfort visualizer is based on visual importance of image scenes, resulting in considerable improvement in prediction performance of visual discomfort. The experimental results showed that the proposed system could generate enhanced disparity maps and achieve visual discomfort prediction accuracy that is feasible for practical applications.