Stereoscopic Image Quality Assessment Research Articles

Joint binocular energy-contrast perception for quality assessment of stereoscopic images

. Binocular visual system (BVS) can perceive the difference between left and right retinal images to create a mental image with depth perception, which is in consequence of two binocular interactions, i.e., binocular fusion and rivalry. To study the effective method of accounting for binocular fusion and rivalry in stereoscopic image quality assessment (SIQA) design, in this paper, a novel full reference (FR) SIQA metric is proposed by jointly considering binocular energy-contrast perception (BECP). As a major technical contribution, we design a dual-channel model for SIQA that more effectively mimic binocular fusion and rivalry mechanisms of the BVS. Specifically, since the binocular visual sensitivity of stimulus at different spatial frequencies is different, each image of the reference and distorted stereopairs is first filtered independently by a contrast sensitivity function (CSF). Constructively, the weights of relative contribution of each view for binocular fusion are calculated based on a magnitude response of Log-Gabor filtering measure. Further, the weights of relative contribution of each view for dominant perception are calculated by utilizing a block-based contrast measure. Finally, the overall perceived quality of a stereoscopic image is obtained by the quality scores combining of the BECP. Experiments are performed on publicly available symmetric and asymmetric subjected stereoscopic image databases, which demonstrate that the proposed metric achieves high consistency with human opinions and significantly higher prediction accuracy than the state-of-the-art FR-SIQA methods.

No-reference stereoscopic image quality assessment guided by visual hierarchical structure and binocular effects.

Stereoscopic image quality assessment (SIQA) is an essential technique for modern 3D image and video processing systems serving as performance evaluators and monitors. However, the study on SIQA remains immature due to the complexity of the human visual system (HVS) and binocular effects that binocular vision brings about. To overcome the difficulties, a novel method is proposed that extracts and quantifies image quality-aware features related to cortex areas in charge of visual quality perception, rather than attempting to rigorously simulate the biological processing in HVS, so that the predicting accuracy is preserved while the computational complexity remains moderate. Meanwhile, binocular effects including binocular rivalry and visual discomfort are taken into consideration. Moreover, the proposed method can be operated completely without the assistance of reference images, indicating its wide practical usages. Compared to state-of-the-art works, our method shows evident superiority in terms of effectiveness and robustness.

Bivariate analysis of 3D structure for stereoscopic image quality assessment

Human visual system (HVS) identifies each unit of stereo-pair into binocular or monocular perception depending on distortions and depth perception. However, these HVS properties still have a large room for exploration in stereoscopic image quality assessment (SIQA) research field. In this paper, a bivariate natural scene statistics (NSS) model is proposed to capture image quality by extracting features from binocular and monocular perception regions, respectively. In the implementation details, the stereo-pair is first segmented into various regions based on spatial information of its disparity. Then the regions are classified into categories of binocular fusion, binocular rivalry and binocular suppression. Bivariate statistics of spatially adjacent Gabor response of image are extracted from each category of regions, based on which features are calculated for image quality representation. In particular, the extraction strategy depends on the type of image patch. Experimental results show that the proposed model is promising at handling the task of SIQA on LIVE 3D Image Quality Database.

A Neural-Based Stereoscopic Image Quality Assessment with Reference

Image quality is an important aspect for several applications (biometrics, tracking, object detection and so on). Several methods have been proposed in the literature to estimate it. These methods are able to predict subjective judgments according to different characteristics. The goal of this paper is to present a framework for stereoscopic image quality metric with reference based on Neural Networks (CNN & ANN). The proposed CNN model is composed of 3 convolutional layers and two Fully Connected (FC) layers and it is used to identify the degradation type in the image. The quality is then estimated using an ANN model. Its inputs are some computed features, selected according to the identified degradation type. The results obtained through two common datasets show the relevance of the proposed approach.

No-Reference Stereoscopic Image Quality Assessment Using Convolutional Neural Network for Adaptive Feature Extraction

The pervasion of 3-D technologies over the years gives rise to the increasing demands of accurate and efficient stereoscopic image quality assessment (SIQA) methods, designed to automatically supervise and optimize 3-D image and video processing systems. Though 2-D IQA has attracted considerable attention, its 3-D counterpart is yet to be well explored. In this paper, a no-reference SIQA method using convolution neural network (CNN) for feature extraction is proposed. In the proposed method, a CNN model is trained from scratch to classify images according to their perceptual quality, with quality-aware monocular features extracted from a higher level layer of the network. Then, visual saliency models are utilized to fuse the captured monocular features. In the meanwhile, multi-scale statistical features are derived from the binocular disparity maps. Finally, the fused CNN features and the disparity features are synthesized by support vector regression into the objective quality score of the stereoscopic image. Experimental results on two public databases demonstrate the superior performance of the proposed method over other state-of-the-art methods, in terms of its accuracy in predicting stereoscopic image quality as well as its robustness across different databases and distortion types.

Stereoscopic Image Quality Assessment by Considering Binocular Visual Mechanisms

With the booming of 3-D image processing in the entertainment industry and 3-D multimedia applications todays, the technology for assessing the quality of stereoscopic image faces more challenging tasks than its 2-D counterparts, such as binocular combination, stereo matching, and binocular rivalry. In this paper, a novel stereoscopic image quality assessment method is proposed by jointly exploring binocular fusion and rivalry models. In the quality-aware feature extraction stage, multi-scale binocular combination and binocular rivalry energy responses are first generated from reference and distorted stereopairs. In addition, considering that the changes of luminance affect the quality of stereoscopic image greatly, multi-scale visual features relating to image quality are obtained from its luminance maps as another binocular combination and rivalry features. Then, the dissimilarity of quality-aware features between the reference stereopair and its distorted version is quantified. Finally, such dissimilarities are mapped into an objective score to represent the perceptual quality of stereoscopic image through the pooling strategy of support vector regression. Experiments conducted on LIVE 3-D databases demonstrate that the proposed method achieves 96.61% and 96.03% in terms of Pearson’s linear correlation coefficient on Database Phase I and Phase II, respectively, outperforming most of the state-of-the-art methods.

基于小波变换与结构特征的立体图像质量评价

基于小波变换与结构特征的立体图像质量评价

Reduced-Reference Stereoscopic Image Quality Assessment Using Natural Scene Statistics and Structural Degradation

Perceptual stereo image quality assessment (SIQA) aims to design computational models to measure the stereo image quality in accordance with human opinions. In this paper, a novel reduced-reference (RR) SIQA is proposed by characterizing the statistical and perceptual properties of the stereo image in both the spatial and gradient domains. To be specific, in the spatial domain, we extract the parameters of the generalized Gaussian distribution fits of luminance wavelet coefficients to form the underlying features. In the gradient domain, the modified gradient magnitudes maps are generated by jointly considering human visual system’s contrast sensitivity and neighborhood gradient information to weight the gradient magnitudes in a locally adaptive manner. Afterward, perceptual features are extracted based on the entropy of discrete wavelet transform coefficients of modified gradient magnitudes. Furthermore, we consolidate the left and right features into a single set of features per stereo image pair. Finally, the qualities of both the spatial and gradient domains are combined to obtain the overall quality of stereo image. Extensive experiments performed on popular data sets demonstrate that the proposed RR-SIQA method achieves highly competitive performance as compared with the state-of-the-art RR-SIQA models as well as full-reference ones for both symmetric and asymmetric distortions.

No-Reference Stereoimage Quality Assessment for Multimedia Analysis Towards Internet-of-Things

With continuous progress of Internet of Things, multimedia analysis in it has attracted more and more attention. Specially, stereoscopic display technology plays an important role in the multimedia analysis processing. In the Internet of Things system, the quality of stereoscopic image will be reduced in the transmission process. In this mode, it will have a great impact on multimedia analysis to judge whether the quality of stereoscopic image meets the requirements. In this paper, a new no-reference stereoscopic image quality assessment model for multimedia analysis towards Internet of Things is built, which is based on a deep learning model to learn from the class labels and image representations. In our framework, images are represented by natural scene statistics features that are extracted from discrete cosine transform domain, and a regression model is employed to shine upon the quality from the feature vector. The training process of the proposed model contains an unsupervised pretraining phase and a supervised fine-tuning phase, enabling it to generalize over the whole distortion types and severity. The proposed model greatly shows the correlation with subjective assessment as demonstrated by experiments on the LIVE 3-D Image Quality Database and IVC 3-D Image Quality Database.

Quaternion representation based visual saliency for stereoscopic image quality assessment

In this paper, a novel visual saliency detection method for stereoscopic images is proposed for the stereoscopic image quality assessment (SIQA) by considering the disparity map and difference image between the stereo image pairs. Firstly, a new quaternion representation (QR) of each stereo image (left/right view image) is constructed, which comprises the image content, the inter-view disparity, and the difference map. The quaternion Fourier transform (QFT) is performed on the constructed QR to generate the visual saliency maps for left and right views of stereoscopic image pairs, respectively. The generated visual saliency maps are further incorporated into the quality metrics for SIQA. Experimental results demonstrate that the visual saliency maps generated by the proposed method can help significantly boost the performance of SIQA, comparing with other visual saliency models proposed for stereoscopic images. It further confirms that the proposed visual saliency model can accurate depict the acuity property of human visual system (HVS) in judging the perceptual quality of stereoscopic images.

No‐reference quality assessment for stereoscopic images considering visual discomfort and binocular rivalry

A no-reference stereoscopic image quality assessment method is proposed that takes two primary binocular effects, visual discomfort, and binocular rivalry, into consideration. The adaptive segmentation, local energy, and visual saliency are employed for estimating binocular effects and constructing a quality baseline, which together composes a comprehensive image quality description. Moreover, a compact parallel framework is designed for efficiency. Experiments have validated the high accuracy and robustness of the proposed method.

Stereoscopic Image Quality Assessment Based on Depth and Texture Information

With the increasing growth of multimedia applications over the networking in recent years, users have put forward much higher requirements for multimedia signals, particularly for image/video signal's quality of experience (QoE) than before. The objective approach of image quality assessment (IQA) plays an important role for the development of compression standards and various multimedia applications. However, the quality assessment of stereoscopic (3-D) images faces more new challenges, such as depth perception, virtual view synthesis, and asymmetric stereo compression. In this paper, we propose a new full-reference (FR) 3-D IQA method to measure the quality of the distorted images. The properties of the depth component, structure component, and gradient component are taken into account to establish the proposed metric. The experimental results show that the proposed metric is highly consistent with the subjective test scores compared with the existing related metrics. In addition, the main significance of the proposed metric is that it not only could effectively evaluate the quality of 3-D image but also has a satisfied effect for measuring the quality of 2-D image.

Sparse representation based stereoscopic image quality assessment accounting for perceptual cognitive process

In this paper, we propose a sparse representation based Reduced-Reference Image Quality Assessment (RR-IQA) index for stereoscopic images from the following two perspectives: 1) Human visual system (HVS) always tries to infer the meaningful information and reduces uncertainty from the visual stimuli, and the entropy of primitive (EoP) can well describe this visual cognitive progress when perceiving natural images. 2) Ocular dominance (also known as binocularity) which represents the interaction between two eyes is quantified by the sparse representation coefficients. Inspired by previous research, the perception and understanding of an image is considered as an active inference process determined by the level of “surprise”, which can be described by EoP. Therefore, the primitives learnt from natural images can be utilized to evaluate the visual information by computing entropy. Meanwhile, considering the binocularity in stereo image quality assessment, a feasible way is proposed to characterize this binocular process according to the sparse representation coefficients of each view. Experimental results on LIVE 3D image databases and MCL database further demonstrate that the proposed algorithm achieves high consistency with subjective evaluation.

Full-reference quality assessment of stereoscopic images by learning binocular visual properties.

Stereoscopic imaging technology has been growingly prevalent driven by both the entertainment industry and scientific applications in today's world. But objective quality assessment of stereoscopic images is a challenging task. In this paper, we propose a novel stereoscopic image quality assessment (SIQA) method by jointly considering monocular perception and binocular interaction. As the most significant contribution of this study, binocular perceptual properties of simple and complex cells are considered for full-reference (FR) SIQA. Specifically, the proposed scheme first simulates the receptive fields of simple cells (one class of V1 neurons) using a push-pull combination of receptive fields response, which is used to represent a monocular cue. Further, the receptive fields of complex cells (the other class of V1 neurons) are simulated by using binocular energy response and binocular rivalry response, which are used to represent a binocular cue. Subsequently, various quality-aware features are extracted from the response of area V1 by calculating the self-weighted histogram of the local binary pattern on four types of feature maps of similarity measurement that will change in the presence of distortions. Finally, kernel ridge regression is used to simulate a nonlinear relationship between the quality-aware features and objective quality scores. The performance of our method is evaluated over popular stereoscopic image databases and shown to be competitive with the state-of-the-art FR SIQA algorithms.

No‐reference stereoscopic image quality assessment based on saliency‐guided binocular feature consolidation

Different from traditional methods depending on the procedure of intermediate ‘cyclopean’ view construction, a novel framework based on saliency-guided multi-scale feature consolidation for stereoscopic image quality assessment is proposed. For quality representation, the underlying features are extracted from three aspects: (i) global natural statistics features, (ii) local spatial and spectral entropy features and (iii) the kurtosis and skew of disparity distribution. Then the binocular features are consolidated by a saliency-guided weighted process. Finally, a machine learning technique of support vector regression is used for objective quality mapping. Experimental results demonstrate the promising performance of the proposed method.

Local and global sparse representation for no-reference quality assessment of stereoscopic images

No-reference/blind quality assessment of stereoscopic 3D images is much more challenging than 2D images due to the poor understanding of binocular vision. In this paper, we propose a BLind Quality Evaluator for stereoscopic 3D images by learning Local and Global Sparse Representations (BLQELGSR). Specifically, at the training stage, we first construct a large-scale training set by simulating some common distortions that are likely encountered by stereoscopic images, and propose a multi-modal sparse representation framework to characterize the relationship between the feature and quality spaces for all sources of information from left, right and cyclopean views in local and global manners. At the testing stage, based on the derived 3D quality prediction framework, the local and global quality scores from different sources are predicted and combined to drive a final 3D quality score. Experimental results on three 3D image quality databases show that in comparison with the existing methods, the devised BLQELGSR can achieve better prediction performance to be in line with subjective assessment.

Structured entropy of primitive: big data‐based stereoscopic image quality assessment

The ultimate receiver of image and video is human visual system (HVS). It is an important problem in the domain of image and video processing that how to establish visual information representation model meeting the HVS perception property. In this study, authors give theory analysis and experiment results to prove that l_1 norm-based entropy of primitive (EoP) is superior to the l_0 norm-based EoP for the monocular cue in image quality assessment. By developing the concept of mutual information of primitive (MIP) as the binocular cue, an l_1 EoP-based stereoscopic image quality assessment metric is proposed. With EoP as monocular cue and MIP as binocular cue, the relative entropy between the original stereoscopic image and the distorted one is explored to predict the quality score with support vector regression. To avoid destroying image's structured information, the structured EoP (SEoP) is further explored to measure the stereoscopic image information. Extensive experimental results demonstrate that the stereoscopic image quality assessment algorithm with SEoP as monocular cue and MIP as binocular cue outperforms many state-of-the-art ones.

Using distortion and asymmetry determination for blind stereoscopic image quality assessment strategy

Predicting the perceived quality of stereoscopic 3D images is a challenging task, especially when the stereo-pair is asymmetrically distorted. Despite the considerable efforts to fix this issue, there is no commonly accepted metric. Most of the attempts consisted in developing full reference quality metrics, while very few efforts have been dedicated to blind/no-reference (NR) quality assessment of stereoscopic images. In this paper, we propose a blind/NR quality assessment strategy for stereoscopic images based on the identification of the distortion type in order to select the most efficient impairment measure in addition to the determination of whether a stereo-pair is symmetrically or asymmetrically distorted to account for the binocular fusion properties. Finally, the last step combines the two key information derived from previous steps and estimates the 3D image quality appropriately using different binocular combination strategies. Experimental results on four publicly available 3D image quality assessment databases showed that the proposed strategy reaches significant prediction consistency and accuracy when compared to state-of-the-art metrics.

Objective quality assessment of stereoscopic images with vertical disparity using EEG

Objective. Neurophysiological correlates of vertical disparity in 3D images are studied in an objective approach using EEG technique. These disparities are known to negatively affect the quality of experience and to cause visual discomfort in stereoscopic visualizations. Approach. We have presented four conditions to subjects: one in 2D and three conditions in 3D, one without vertical disparity and two with different vertical disparity levels. Event related potentials (ERPs) are measured for each condition and the differences between ERP components are studied. Analysis is also performed on the induced potentials in the time frequency domain. Main results. Results show that there is a significant increase in the amplitude of P1 components in 3D conditions in comparison to 2D. These results are consistent with previous studies which have shown that P1 amplitude increases due to the depth perception in 3D compared to 2D. However the amplitude is significantly smaller for maximum vertical disparity (3D-3) in comparison to 3D with no vertical disparity. Our results therefore suggest that the vertical disparity in 3D-3 condition decreases the perception of depth compared to other 3D conditions and the amplitude of P1 component can be used as a discriminative feature. Significance. The results show that the P1 component increases in amplitude due to the depth perception in the 3D stimuli compared to the 2D stimulus. On the other hand the vertical disparity in the stereoscopic images is studied here. We suggest that the amplitude of P1 component is modulated with this parameter and decreases due to the decrease in the perception of depth.

Stereoscopic image quality assessment by learning non-negative matrix factorization-based color visual characteristics and considering binocular interactions

In this paper, we propose a novel stereoscopic image quality assessment (SIQA) method by learning non-negative matrix factorization (NMF)-based color visual characteristics for monocular perception and considering binocular interactions. In training phase, a feature basis matrix is learned based on NMF by considering color information and a feature detector is designed by performing Schmidt orthogonalization on the feature basis matrix. In construction of SIQA phase, for monocular perception, visual saliency regions are selected and parts-based feature similarity indexes of left and right views based on the feature vectors extracted by the feature detector are calculated. For binocular interactions, we calculate cyclopean feature similarity index by considering binocular fusion and rivalry. Finally, support vector regression is used to simulate nonlinear relationship between monocular perception and binocular interactions. Experimental results on LIVE 3D image databases and NBU 3D IQA database demonstrate that the proposed SIQA method is more consistent with human perception.