Cyclopean View Research Articles

Deep neural network based distortion parameter estimation for blind quality measurement of stereoscopic images

Stereoscopic/3D image quality measurement (SIQM) has emerged as an active and important research branch in image processing/computer vision field. Existing methods for blind/no-reference SIQM often train machine-learning models on degraded stereoscopic images for which human subjective quality ratings have been obtained, and they are thus constrained by the fact that only a limited number of 3D image quality datasets currently exist. Although methods have been proposed to overcome this restriction by predicting distortion parameters rather than quality scores, the approach is still limited to the time-consuming, hand-crafted features extracted to train the corresponding classification/regression models as well as the rather complicated binocular fusion/rivalry models used to predict the cyclopean view. In this paper, we explore the use of deep learning to predict distortion parameters, giving rise to a more efficient opinion-unaware SIQM technique. Specifically, a deep fusion-and-excitation network which takes into account the multiple-distortion interactions is proposed to perform distortion parameter estimation, thus avoiding hand-crafted features by using convolution layers while simultaneously accelerating the algorithm by using the GPU. Moreover, we measure distortion parameter values of the cyclopean view by using support vector regression models which are trained on the data obtained from a newly-designed subjective test. In this way, the potential errors in computing the disparity map and cyclopean view can be prevented, leading to a more rapid and precise 3D-vision distortion parameter estimation. Experimental results tested on various 3D image quality datasets demonstrate that our proposed method, in most cases, offers improved predictive performance over existing state-of-the-art methods.

3D saliency guided deep quality predictor for no-reference stereoscopic images

The use of 3D technologies is growing rapidly, and stereoscopic imaging is usually used to display the 3D contents. However, compression, transmission and other necessary treatments may reduce the quality of these images. Stereo Image Quality Assessment (SIQA) has attracted more attention to ensure good viewing experience for the users and thus several methods have been proposed in the literature with a clear improvement for deep learning-based methods. This paper introduces a new deep learning-based no-reference SIQA using cyclopean view hypothesis and human visual attention. First, the cyclopean image is constructed considering the presence of binocular rivalry that covers the asymmetric distortion case. Second, the saliency map is computed considering the depth information. The latter aims to extract patches on the most perceptual relevant regions. Finally, a modified version of the pre-trained Convolutional Neural Network (CNN) is fine-tuned and used to predict the quality score through the selected patches. Five distinct pre-trained models were analyzed and compared in term of results. The performance of the proposed metric has been evaluated on four commonly used datasets (3D LIVE phase I and phase II databases as well as Waterloo IVC 3D Phase 1 and Phase 2). Compared with the state-of-the-art metrics, the proposed method gives better outcomes. The implementation code will be made accessible to the public at: https://github.com/o-messai/3D-NR-SIQA

Human gaze is systematically offset from the center of cone topography

The small physical depression of the human retina, the fovea, is the retinal locus of prime visual resolution, achieved by a peaking topography of the light-sensitive cone photoreceptor outer segments1-3 and a post-receptor wiring scheme preserving high-density sampling.4,5 Humans dynamically direct their gaze such that the retinal images of objects of interest fall onto the foveola, the central one-degree diameter of the fovea,6-8 but it is yet unclear whether a relationship between the individual photoreceptor topography at this location and visual fixation behavior exists.9,10 By employing adaptive optics invivo imaging and micro-stimulation,11-13 we created topographical maps of the complete foveolar cone mosaics in both eyes of 20 healthy participants while simultaneously recording the retinal location of a fixated visual object in a psychophysical experiment with cellular resolution. We found that the locus of fixation was systematically shifted away from the topographical center toward a naso-superior quadrant on the retina, about 5min of arc of visual angle on average, with a mirror symmetrical trend between fellow eyes. In cyclopean view, the topographical centers were superior to the fixated target, corresponding to areas in the visual field usually more distant14,15 and thus containing higher spatial frequencies. Given the large variability in foveal topography between individuals, and the surprising precision with which fixation is repeatedly directed to just a small bouquet of cones in the foveola, these findings demonstrate a finely tuned, functionally relevant link between the development of the cellular mosaic of photoreceptors and visual behavior.

Quality assessment of multiply and singly distorted stereoscopic images via adaptive construction of cyclopean views

A challenging problem confronted when designing a blind/no-reference (NR) stereoscopic image quality assessment (SIQA) algorithm is to simulate the quality assessment (QA) behavior of the human visual system (HVS) during binocular vision. An effective way to solve this problem is to estimate the quality of the merged single view created in the human brain which is also referred to as the cyclopean image. However, due to the difficulty in modeling the binocular fusion and rivalry properties of the HVS, obtaining effective cyclopean images for QA is non-trivial, and consequently previous NR SIQA algorithms either require the MOS/DMOS values of the distorted 3D images for training or ignore the quality analysis of the merged cyclopean view. In this paper, we focus on (1) constructing accurate and appropriate cyclopean views for QA of stereoscopic images by adaptively analyzing the distortion information of two monocular views, and (2) training NR SIQA models without requiring the assistance of the MOS/DMOS values in existing databases. Accordingly, we present an effective opinion-unaware SIQA algorithm called MUSIQUE-3D, which blindly assesses the quality of multiply and singly distorted stereoscopic images by analyzing quality degradations of both monocular and cyclopean views. The monocular view quality is estimated by an extended version of the MUSIQUE algorithm, and the cyclopean view quality is computed from the distortion parameter values predicted by a two-layer classification-regression model trained on a large 3D image dataset. Tests on various 3D image databases demonstrate the superiority of our method as compared with other state-of-the-art SIQA algorithms.

Deep Quality evaluator guided by 3D Saliency for Stereoscopic Images

Due to the use of 3D contents in various applications, Stereo Image Quality Assessment (SIQA) has attracted more attention to ensure good viewing experience for the users. Several methods have been thus proposed in the literature with a clear improvement for deep learning-based methods. This paper introduces a new deep learning-based no-reference SIQA using cyclopean view hypothesis and human visual attention. First, the cyclopean image is built considering the presence of binocular rivalry that covers the asymmetric distortion case. Second, the saliency map is computed taking into account the depth information. The latter aims to extract patches on the most perceptual relevant regions. Finally, a modified version of the pre-trained vgg-19 is fine-tuned and used to predict the quality score through the selected patches. The performance of the proposed metric has been evaluated on 3D LIVE phase I and phase II databases. Compared with the state-of-the-art metrics, our method gives better outcomes.

Learning Local Quality-Aware Structures of Salient Regions for Stereoscopic Images via Deep Neural Networks

The perceptual quality of stereoscopic images plays an essential role in the human perception of visual information. However, most available stereoscopic image quality assessment (SIQA) methods evaluate 3D visual experience using hand-crafted features or shallow architectures, which cannot model the visual properties of stereo images well. In this paper, we use convolutional neural networks (CNNs) to learn deeper local quality-aware structures for stereo images. With different inputs, two CNN models are designed for no-reference SIQA tasks. The one-column CNN model directly accepts a cyclopean view as the input, and the three-column CNN model jointly considers the cyclopean, left and right views as CNN inputs. The two SIQA frameworks share the same implementation approach: First, to overcome the obstacle of limited SIQA datasets, we accept image patches that have been cropped from corresponding stereopairs as inputs for local quality-sensitive feature extraction. Next, a local feature selection algorithm is used to remove related features on non-salient patches, which could cause large prediction errors. Finally, the reserved local visual structures of salient regions are aggregated into a final quality score in an end-to-end manner. Experimental results on three public SIQA databases demonstrate that our method outperforms most state-of-the-art no-reference (NR) SIQA methods. The results of a cross-database experiment also show the robustness and generality of the proposed method.

AdaBoost neural network and cyclopean view for no-reference stereoscopic image quality assessment

Stereoscopic imaging is widely used in many fields. In many scenarios, stereo images quality could be affected by various degradations, such as asymmetric distortion. Accordingly, to guarantee the best quality of experience, robust and accurate reference-less metrics are required for quality assessment of stereoscopic content. Most existing stereo no-reference Image Quality Assessment (IQA) models are not consistent with asymmetrical distortions. This paper presents a new no-reference stereoscopic image quality assessment metric using a human visual system (HVS) modeling and an advanced machine-learning algorithm. The proposed approach consists of two stages. In the first stage, cyclopean image is constructed considering the presence of binocular rivalry in order to cover the asymmetrically distorted part. In the second stage, gradient magnitude, relative gradient magnitude, and gradient orientation are extracted. These are used as a predictive source of information for the quality. In order to obtain the best overall performance against different databases, Adaptive Boosting (AdaBoost) idea of machine learning combined with artificial neural network model has been adopted. The benchmark LIVE 3D phase-I, phase-II, and IRCCyN/IVC 3D databases have been used to evaluate the performance of the proposed approach. Experimental results have demonstrated that the proposed metric performance achieves high consistency with subjective assessment and outperforms the blind stereo IQA over various types of distortion.

Blind Stereo Image Quality Assessment Based on Binocular Visual Characteristics and Depth Perception

The quality prediction of stereo images has great challenges without reference images. In this paper, we propose a novel no-reference stereo image quality assessment (NR-SIQA) model based on binocular visual characteristics and depth perception, which can effectively evaluate the quality of symmetric distortion and asymmetric distortion images. To be specific, we discriminate the different binocular behaviors by analyzing binocular visual characteristics, and construct the corresponding cyclopean view instead of single cyclopean view to simulate different binocular behaviors. Then, we extract monocular and binocular visual features from the left view, the right view and the synthetic cyclopean view. Furthermore, in order to evaluate the depth quality of the stereo image accurately, we extract the depth perception features from the weighted disparity map and the longitudinal correlation coefficient map. Finally, we construct the mapping relationship model from quality perception feature domain to quality score domain by training an adaptive enhancement algorithm based on support vector regression (SVR). We evaluate the performance of the proposed algorithm on four stereo image databases. The experimental results show that compared with the state-of-the-art full reference(FR), reduced reference(RR) and NR-SIQA algorithms, the proposed algorithm achieves highly competitive performance for both symmetric and asymmetric distortions.

Blind Stereo Image Quality Evaluation Based on Convolutional Network and Saliency Weighting

With the rapid development of stereo image applications, there is an increasing demand to develop a versatile tool to evaluate the perceived quality of stereo images. Therefore, in this study, a blind stereo image quality evaluation (SIQE) algorithm based on convolutional network and saliency weighting is proposed. The main network framework used by the algorithm is the quality map generation network, which is used to train the distortion image dataset and quality map label to obtain an optimal network framework. Finally, the left view, right view, and cyclopean view of the stereo image are used as inputs to the network frame, respectively, and then weighted fusion for the final stereo image quality score. The experimental results reveal that the proposed SIQE algorithm can improve the accuracy of the image quality prediction and prediction score to a certain extent and has good generalization ability.

Toward an unsupervised blind stereoscopic 3D image quality assessment using joint spatial and frequency representations

Existing blind stereoscopic 3D (S3D) image quality assessment (IQA) metrics usually require supervised learning methods to predict S3D image quality, which limits their applicability in practice. In this paper, we propose an unsupervised blind S3D IQA metric that utilizes the joint spatial and frequency representations of visual perception. The metric proposed in this work was inspired by the binocular visual mechanism; furthermore, it is unsupervised and does not require subject-rated samples for training. To be more specific, first, the various binocular quality-aware features in spatial and frequency domains are extracted from the monocular and cyclopean views of natural S3D image patches. Subsequently, these features are utilized to establish a pristine multivariate Gaussian (MVG) model to characterize natural S3D image regularities. Finally, with the learned MVG model, the final quality score for a distorted S3D image can be yielded using a Bhattacharyya-like distance. Our experimental results illustrate that, compared to related existing metrics, the devised metric achieves competitive prediction performance.

Local and global sparse representation for no-reference quality assessment of stereoscopic images

No-reference/blind quality assessment of stereoscopic 3D images is much more challenging than 2D images due to the poor understanding of binocular vision. In this paper, we propose a BLind Quality Evaluator for stereoscopic 3D images by learning Local and Global Sparse Representations (BLQELGSR). Specifically, at the training stage, we first construct a large-scale training set by simulating some common distortions that are likely encountered by stereoscopic images, and propose a multi-modal sparse representation framework to characterize the relationship between the feature and quality spaces for all sources of information from left, right and cyclopean views in local and global manners. At the testing stage, based on the derived 3D quality prediction framework, the local and global quality scores from different sources are predicted and combined to drive a final 3D quality score. Experimental results on three 3D image quality databases show that in comparison with the existing methods, the devised BLQELGSR can achieve better prediction performance to be in line with subjective assessment.

3D-MAD: A Full Reference Stereoscopic Image Quality Estimator Based on Binocular Lightness and Contrast Perception.

Algorithms for a stereoscopic image quality assessment (IQA) aim to estimate the qualities of 3D images in a manner that agrees with human judgments. The modern stereoscopic IQA algorithms often apply 2D IQA algorithms on stereoscopic views, disparity maps, and/or cyclopean images, to yield an overall quality estimate based on the properties of the human visual system. This paper presents an extension of our previous 2D most apparent distortion (MAD) algorithm to a 3D version (3D-MAD) to evaluate 3D image quality. The 3D-MAD operates via two main stages, which estimate perceived quality degradation due to 1) distortion of the monocular views and 2) distortion of the cyclopean view. In the first stage, the conventional MAD algorithm is applied on the two monocular views, and then the combined binocular quality is estimated via a weighted sum of the two estimates, where the weights are determined based on a block-based contrast measure. In the second stage, intermediate maps corresponding to the lightness distance and the pixel-based contrast are generated based on a multipathway contrast gain-control model. Then, the cyclopean view quality is estimated by measuring the statistical-difference-based features obtained from the reference stereopair and the distorted stereopair, respectively. Finally, the estimates obtained from the two stages are combined to yield an overall quality score of the stereoscopic image. Tests on various 3D image quality databases demonstrate that our algorithm significantly improves upon many other state-of-the-art 2D/3D IQA algorithms.

An efficient human visual system based quality metric for 3D video

Stereoscopic video technologies have been introduced to the consumer market in the past few years. A key factor in designing a 3D system is to understand how different visual cues and distortions affect the perceptual quality of stereoscopic video. The ultimate way to assess 3D video quality is through subjective tests. However, subjective evaluation is time consuming, expensive, and in some cases not possible. The other solution is developing objective quality metrics, which attempt to model the Human Visual System (HVS) in order to assess perceptual quality. Although several 2D quality metrics have been proposed for still images and videos, in the case of 3D efforts are only at the initial stages. In this paper, we propose a new full-reference quality metric for 3D content. Our method mimics HVS by fusing information of both the left and right views to construct the cyclopean view, as well as taking to account the sensitivity of HVS to contrast and the disparity of the views. In addition, a temporal pooling strategy is utilized to address the effect of temporal variations of the quality in the video. Performance evaluations showed that our 3D quality metric quantifies quality degradation caused by several representative types of distortions very accurately, with Pearson correlation coefficient of 90.8 %, a competitive performance compared to the state-of-the-art 3D quality metrics.

Quantifying the importance of cyclopean view and binocular rivalry-related features for objective quality assessment of mobile 3D video

3D video is expected to provide an enhanced user experience by using the impression of depth to bring greater realism to the user. Quality assessment plays an important role in the design and optimization of 3D video processing systems. In this paper, a new 3D image quality model that is specifically tailored for mobile 3D video is proposed. The model adopts three quality components, called the cyclopean view, binocular rivalry, and the scene geometry, in which the quality must be quantified. The cyclopean view formation process is simulated and its quality is evaluated using the three proposed approaches. Binocular rivalry is quantified over the distorted stereo pairs, and the scene quality is quantified over the disparity map. Based on the model, the 3D image quality can then be assessed using state-of-the-art 2D quality measures selected appropriately through a machine learning approach. To make the metric simple, fast, and efficient, final selection of the quality features is accomplished by also considering the computational complexity and the CPU running time. The metric is compared with several currently available 2D and 3D metrics. Experimental results show that the compound metric gives a significantly high correlation with the mean opinion scores that were collected through large-scale subjective tests run on mobile 3D video content.

Efficient Dense Stereo with Occlusions for New View-Synthesis by Four-State Dynamic Programming

A new algorithm is proposed for efficient stereo and novel view synthesis. Given the video streams acquired by two synchronized cameras the proposed algorithm synthesises images from a virtual camera in arbitrary position near the physical cameras. The new technique is based on an improved, dynamic-programming, stereo algorithm for efficient novel view generation. The two main contributions of this paper are: (i) a new four state matching graph for dense stereo dynamic programming, that supports accurate occlusion labelling; (ii) a compact geometric derivation for novel view synthesis by direct projection of the minimum cost surface. Furthermore, the paper presents an algorithm for the temporal maintenance of a background model to enhance the rendering of occlusions and reduce temporal artefacts (flicker); and a cost aggregation algorithm that acts directly in the three-dimensional matching cost space. The proposed algorithm has been designed to work with input images with large disparity range, a common practical situation. The enhanced occlusion handling capabilities of the new dynamic programming algorithm are evaluated against those of the most powerful state-of-the-art dynamic programming and graph-cut techniques. Four-state DP is also evaluated against the disparity-based Middlebury error metrics and its performance found to be amongst the best of the efficient algorithms. A number of examples demonstrate the robustness of four-state DP to artefacts in stereo video streams. This includes demonstrations of cyclopean view synthesis in extended conversational sequences, synthesis from a freely translating virtual camera and, finally, basic 3D scene editing.

Kanizsa's Shrinkage Illusion Produced by a Misapplied 3-D Corrective Mechanism

In order to include the monocular areas from the left and the right eye in the cyclopean view, the visual system displaces the occluded elements which would result in a horizontal elongation of the shape but does not occur thanks to a correction mechanism which preserves the shape. We hypothesised that this mechanism causes Kanizsa's amodal shrinkage illusion (the apparent elongation of a partially occluded square) when it is incorrectly applied by the visual system to a two-dimensional stimulus. Four experiments tested this hypothesis: (i) one-eyed observers were less susceptible to the illusion than people with normal binocular vision because, for them, the correction for shape is unnecessary; (ii) the illusion was stronger with binocular than with monocular vision since binocularity induces the visual system to correct for the shape distortion; (iii) the illusion diminished when the stimulus was rotated 90 degrees given that displacement and compression are not required for vertical occlusion; (iv) the magnitude of the illusion was a function of the width of the occluder because, as previous research has shown, the edges of a partially occluded square are less displaced the farther they are from the edges of the occluder. The data from the four experiments support our hypothesis even though no condition was able to eliminate the illusion; other possible causes are discussed.

Stereo disparity improves color constancy

Binocular disparity is an aspect of natural viewing. This research investigates whether disparity affects surface color perception. Achromatic settings were obtained and compared for two stereograms of a scene with specular reflections, one stereogram with binocular disparity and one without it (cyclopean view). Binocular disparity was found to improve color constancy. Next, the geometry of specular highlights, which is distorted without binocular disparity, was specifically examined. Measurements compared color constancy with specular reflections that were either normal (with stereo disparity) or distorted (cyclopean view of the specularities). No significant change in constancy was found due to the geometrical distortion of specular highlights that occurs without stereo disparity, suggesting that constancy depends on other features of the percept affected by disparity. The results are discussed in terms of illuminant estimation in surface color perception.

Further consideration of size illusions in random dot stereograms.

It has been suggested that many geometric illusions are caused by the application of depth or size constancy rules to an image which does not have sufficient cues to establish that the elements lie in a flat plane. Thus, converging lines are taken as depth cues, and the attributed depth provides the basis for adjusting the perceived size of stimulus elements. It this is the case, one should not see a distortion of relative size if the disparity cues provide for strong statification, i.e., localization in depth of the linear perspective cues. This expectation is challenged by demonstrations that show distortions of relative size using random-dot stereograms. In 1971 Julesz provided such examples but did not comment on the implications for theories of depth. Here we redemonstrate these distortion of length and size in autostereograms which contain the Ponzo and Corridor configurations. The illusory distortions can be seen in the cyclopean view even though the linear perspective elements are well stratified. We suggest that the processing of binocular disparity cues, as required for judgments of absolute distance, may involve the dorsal stream of vision, i.e., activity passing into and including the parietal lobe. Pictorial cues, on the other hand, are likely passed through the ventral stream into the temporal lobe. The analysis of depth by this system provides for size constancy and, possibly, the calibration of relative motion.

FURTHER CONSIDERATION OF SIZE ILLUSIONS IN RANDOM DOT STEREOGRAMS

It has been suggested that many geometric illusions are caused by the application of depth or size-constancy rules to an image which does not have sufficient cues to establish that the elements lie in a flat plane. Thus, converging lines are taken as depth cues, and the attributed depth provides the basis for adjusting the perceived size of stimulus elements. If this is the case, one should not see a distortion of relative size if the disparity cues provide for strong stratification, i.e., localization in depth, of the linear perspective cues. This expectation is challenged by demonstrations that show distortions of relative size using random-dot stereograms. In 1971 Julesz provided such examples but did not comment on the implications for theories of depth. Here we redemonstrate these distortions of length and size in autostereograms which contain the Ponzo and Corridor configurations. The illusory distortions can be seen in the cyclopean view even though the linear perspective elements are well stratified. We suggest that the processing of binocular disparity cues, as required for judgments of absolute distance, may involve the dorsal stream of vision, i.e., activity passing into and including the parietal lobe. Pictorial cues, on the other hand, are likely passed through the ventral stream into the temporal lobe. The analysis of depth by this system provides for size constancy and, possibly, the calibration of relative motion.

Temporal events in cyclopean vision.

The majority of neurons in the primary visual cortex of primates can be activated by stimulation of either eye; moreover, the monocular receptive fields of such neurons are located in about the same region of visual space. These well-known facts imply that binocular convergence in visual cortex can explain our cyclopean view of the world. To test the adequacy of this assumption, we examined how human subjects integrate binocular events in time. Light flashes presented synchronously to both eyes were compared to flashes presented alternately (asynchronously) to one eye and then the other. Subjects perceived very-low-frequency (2 Hz) asynchronous trains as equivalent to synchronous trains flashed at twice the frequency (the prediction based on binocular convergence). However, at higher frequencies of presentation (4-32 Hz), subjects perceived asynchronous and synchronous trains to be increasingly similar. Indeed, at the flicker-fusion frequency (approximately 50 Hz), the apparent difference between the two conditions was only 2%. We suggest that the explanation of these anomalous findings is that we parse visual input into sequential episodes.