Stereoscopic Image Quality Assessment Method Research Articles

Image2Brain: a cross-modality model for blind stereoscopic image quality ranking

Objective. Human beings perceive stereoscopic image quality through the cerebral visual cortex, which is a complex brain activity. As a solution, the quality of stereoscopic images can be evaluated more accurately by attempting to replicate the human perception from electroencephalogram (EEG) signals on image quality in a machine, which is different from previous stereoscopic image quality assessment methods focused only on the extraction of image features. Approach. Our proposed method is based on a novel image-to-brain (I2B) cross-modality model including a spatial-temporal EEG encoder (STEE) and an I2B deep convolutional generative adversarial network (I2B-DCGAN). Specifically, the EEG representations are first learned by STEE as real samples of I2B-DCGAN, which is designed to extract both quality and semantic features from the stereoscopic images by a semantic-guided image encoder, and utilize a generator to conditionally create the corresponding EEG features for images. Finally, the generated EEG features are classified to predict the image perceptual quality level. Main results. Extensive experimental results on the collected brain-visual multimodal stereoscopic image quality ranking database, demonstrate that the proposed I2B cross-modality model can better emulate the visual perception mechanism of the human brain and outperform the other methods by achieving an average accuracy of 95.95. Significance. The proposed method can convert the learned stereoscopic image features into brain representations without EEG signals during testing. Further experiments verify that the proposed method has good generalization ability on new datasets and the potential for practical applications.

Blind Stereoscopic Image Quality Evaluator Based on Binocular Semantic and Quality Channels

Human beings always evaluate the perceptual quality of an image coupled with identifying the semantic content of images. This paper addresses the correlation issue between stereoscopic image quality assessment (SIQA) and semantic recognition. In contrast to the previous SIQA methods that relied on binocular quality-aware features of a stereoscopic image, our approach also extracts binocular semantic features using a pre-trained deep convolutional neural network (DCNN) on a large dataset like ImageNet dataset, as well as the manually designed binocular quality-aware features. It can solve the problem of limited SIQA dataset size and facilitate better prediction on the quality. Experimental results demonstrate that the binocular semantic features are a good predictor for the stereoscopic image quality. The proposed method outperforms the state-of-the-art SIQA methods on four benchmark SIQA datasets. Significantly, all Spearman rank-order correlation coefficients (SROCCs) between the predicted scores and the subjective scores on the four datasets exceed 0.95. The MATLAB source code of the proposed method is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/kyohoonsim/Blind-Stereoscopic-Image-Quality-Evaluator-based-on-Binocular-Semantic-and-Quality-Channels</uri> <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/kyohoonsim</uri> .

A no-Reference Stereoscopic Image Quality Assessment Network Based on Binocular Interaction and Fusion Mechanisms.

In contemporary society full of stereoscopic images, how to assess visual quality of 3D images has attracted an increasing attention in field of Stereoscopic Image Quality Assessment (SIQA). Compared with 2D-IQA, SIQA is more challenging because some complicated features of Human Visual System (HVS), such as binocular interaction and binocular fusion, must be considered. In this paper, considering both binocular interaction and fusion mechanisms of the HVS, a hierarchical no-reference stereoscopic image quality assessment network (StereoIF-Net) is proposed to simulate the whole quality perception of 3D visual signals in human cortex, including two key modules: BIM and BFM. In particular, Binocular Interaction Modules (BIMs) are constructed to simulate binocular interaction in V2-V5 visual cortex regions, in which a novel cross convolution is designed to explore the interaction details in each region. In the BIMs, different output channel numbers are designed to imitate various receptive fields in V2-V5. Furthermore, a Binocular Fusion Module (BFM) with automatic learned weights is proposed to model binocular fusion of the HVS in higher cortex layers. The verification experiments are conducted on the LIVE 3D, IVC and Waterloo-IVC SIQA databases and three indices including PLCC, SROCC and RMSE are employed to evaluate the assessment consistency between StereoIF-Net and the HVS. The proposed StereoIF-Net achieves almost the best results compared with advanced SIQA methods. Specifically, the metric values on LIVE 3D, IVC and WIVC-I are the best, and are the second-best on the WIVC-II.

Deep network based stereoscopic image quality assessment via binocular summing and differencing

With the development of deep networks in dealing with various visual tasks, the deep network based on binocular vision is expected to tackle the issue of stereoscopic image quality assessment. Here, we present a stereoscopic image quality assessment method using the deep network with four channels together, which takes the left view, right view, binocular summing view, and binocular differencing view as the inputs of the network. The visual features are enhanced through the concatenation in a weighted way, so that the binocular vision can be adequately included in the binocular addition and subtraction information. Compared with the state-of-the-art metrics, the proposed method exhibits relatively high performances on four benchmark databases.

Stereoscopic image quality assessment considering visual mechanism and multi-loss constraints

In this paper, a convolutional neural network (CNN) with multi-loss constraints is designed for stereoscopic image quality assessment (SIQA). A stereoscopic image not only contains monocular information, but also provides binocular information which is as identically crucial as the former. So we take the image patches of left-view images, right-view images and the difference images as the inputs of the network to utilize monocular information and binocular information. Moreover, we propose a method to obtain proxy label of each image patch. It preserves the quality difference between different regions and views. In addition, the multiple loss functions with adaptive loss weights are introduced in the network, which consider both local features and global features and constrain the feature learning from multiple perspectives. And the adaptive loss weights also make the multi-loss CNN more flexible. The experimental results on four public SIQA databases show that the proposed method is superior to other existing SIQA methods with state-of-the-art performance.

Learning Local Quality-Aware Structures of Salient Regions for Stereoscopic Images via Deep Neural Networks

The perceptual quality of stereoscopic images plays an essential role in the human perception of visual information. However, most available stereoscopic image quality assessment (SIQA) methods evaluate 3D visual experience using hand-crafted features or shallow architectures, which cannot model the visual properties of stereo images well. In this paper, we use convolutional neural networks (CNNs) to learn deeper local quality-aware structures for stereo images. With different inputs, two CNN models are designed for no-reference SIQA tasks. The one-column CNN model directly accepts a cyclopean view as the input, and the three-column CNN model jointly considers the cyclopean, left and right views as CNN inputs. The two SIQA frameworks share the same implementation approach: First, to overcome the obstacle of limited SIQA datasets, we accept image patches that have been cropped from corresponding stereopairs as inputs for local quality-sensitive feature extraction. Next, a local feature selection algorithm is used to remove related features on non-salient patches, which could cause large prediction errors. Finally, the reserved local visual structures of salient regions are aggregated into a final quality score in an end-to-end manner. Experimental results on three public SIQA databases demonstrate that our method outperforms most state-of-the-art no-reference (NR) SIQA methods. The results of a cross-database experiment also show the robustness and generality of the proposed method.

No-reference stereoscopic image quality assessment based on global and local content characteristics

No-reference stereoscopic images quality assessment (NR-SIQA) via deep learning has gained increasing attention. In this paper, we propose a no-reference stereoscopic image quality assessment method based on global and local content characteristics. The proposed method simulates the perception route of human visual system, and derives features from the fused view and single view through the global feature fusion sub-network and local feature enhancement sub-network. As for the fused view, a cross-fusion strategy is applied to model the process in the V1 visual cortex, and the multi-scales pooling (MSP) is utilized to integrate context information under different sub-regions for effective global feature extraction. As for the single view, the asymmetric convolution block (ACB) is introduced to strengthen the local information description. By jointly considering the fused view and single view, the proposed network can efficiently extract the features for quality assessment. Finally, a weighted average strategy is applied to estimate the visual quality of stereoscopic image. Experimental results on 3D quality databases demonstrate that the proposed network is superior to the state-of-the-art metrics, and achieves an excellent performance.

No-reference stereoscopic image quality assessment based on cyclopean image and enhanced image

In order to effectively evaluate the quality of stereoscopic images, we propose a no-reference stereoscopic image quality assessment (SIQA) method. Firstly, considering the characteristics of binocular fusion, binocular rivalry, and binocular suppression of human visual system, we propose a new color cyclopean image which is suitable for symmetric and asymmetric distortion images. And then, based on the importance of the disparity map, the enhanced image is generated according to the cyclopean image and the disparity map. Next, the natural statistical features are extracted from the enhanced image and the cyclopean image weighted by the gradient of disparity map (named weighted cyclopean image) in the spatial domain. The kurtosis and skewness are extracted from disparity map. Finally, the extracted features are fused and the quality of stereoscopic image is obtained by support vector regression. Experimental results show that the proposed algorithm is superior to most existing objective SIQA methods and can maintain a high degree of consistency with the subjective scores.

Stereoscopic image quality assessment by analysing visual hierarchical structures and binocular effects

In the design of three-dimensional image processing systems, stereoscopic image quality assessment (SIQA) plays an indispensable role as a performance evaluator and a supervisor, yet the study upon which remains immature due to the complexity of the human visual system (HVS). In this study, a novel SIQA method is proposed by extracting quality-aware image features according to the properties of the hierarchical structure in the HVS. Especially, the interests of the primary and secondary visual cortex are taken into consideration, so that the image quality representation is constructed in a way both accurate and efficient. Moreover, influences caused by binocular effects including binocular rivalry and binocular visual discomfort are accounted to further improve the performance of the proposed method. The superiority of the proposed method is validated through experiments on public databases in comparison to state-of-the-art works in terms of accuracy and robustness.

Multiple just-noticeable-difference-based no-reference stereoscopic image quality assessment

Just-noticeable difference (JND) is an important characteristic of the human visual system (HVS), and some established JND models imitating the perception of human eyes already exist. However, their utilization in stereoscopic image quality assessment (SIQA) remains limited. To better simulate how HVS senses 3D images under a no-reference situation, a novel SIQA method based on multiple JND models is proposed in this paper. In our metric, the stereoscopic image pairs are decomposed into multi-scale monocular views and binocular views. Then, texture and edge information of these multi-scale images is extracted. Next, a monocular JND model, a binocular JND model, and a depth JND model are separately applied to the extracted features and the depth map. Finally, these features are synthesized and mapped to objective scores. Through experiment and comparison on public 3D image databases, the proposed method shows a competitive advantage over most state-of-the-art SIQA methods, which indicates that it has a promising prospect in practical applications.

Stereoscopic image quality assessment combining statistical features and binocular theory

Stereoscopic image quality evaluation is extremely significance as a performance evaluator of modern 3D display technology. Due to the complexity of human visual system (HVS) and the incomprehensive study of stereoscopic perception of human eyes, stereoscopic image quality assessment (SIQA) is still a challenging task. In this paper, combining binocular characteristics, we propose an efficient no-reference stereoscopic image quality assessment according to binocular adding and subtracting channels. Distinguished from other SIQA methods, which pay attention to complex binocular visual properties, the visual information which is closely bound up with image distortion from adding and subtracting channels to describe ocular dominance (alternate name binocularity) is proposed. To estimate the contribution of each channel in SIQA, a dynamic weighting system is proposed for binocular fusion according to local energy. Furthermore, quality awareness features based on multi-scale and multi-orientation are extracted to describe visual degradation. Comparing with existing methods, experimental results on public 3D image databases demonstrate the proposed framework achieves high consistent with the subjective quality scores.

Multistage Pooling for Blind Quality Prediction of Asymmetric Multiply-Distorted Stereoscopic Images

Quality prediction for asymmetric multiply-distorted stereoscopic images (MDSIs) confronts more challenges than previous stereoscopic image quality assessment (SIQA) issues, whereas the existing no-reference SIQA methods have been limited to understand the asymmetric distortions and multiple distortions simultaneously for general-purpose blind quality prediction. In this paper, we propose a multistage pooling (MUSP) model for quality prediction of asymmetric MDSIs. In the training stage, we establish multimodal sparse representation framework for phase and amplitude components, respectively. In the testing stage, we use an MUSP strategy to simulate the pooling procedure undergoing multimodal quality pooling, feature pooling, binocular pooling, and phase-amplitude quality pooling in order. Experimental results on our new established database (NBU-MDSID Phase-II) demonstrate the effectiveness of our blind metric.

Perceptual stereoscopic image quality assessment method with tensor decomposition and manifold learning

Perceptual quality assessment of stereoscopic images is a challenge in three-dimensional video systems. Existing studies suggest that simply averaging the quality of left and right views can effectively predict the quality of symmetrically distorted stereoscopic images, but prediction deviation occurs in the case of asymmetrically distorted stereoscopic images. Most previous stereoscopic image quality assessment (SIQA) methods have been based only on the luminance component of the images; in addition, the basis of human visual perception is critical to image quality assessment and lies on the low-dimensional manifold. Inspired by this, a new perceptual SIQA method is proposed, which includes two stages: training stage and quality prediction stage. In the training stage, the authors apply Tucker decomposition to RGB images to reduce dimensions along colour channels to produce training sets, and the projection matrix is obtained through manifold learning. In the quality prediction stage, considering the binocular visual characteristics of visual perception, the overall stereoscopic estimate depends on the monocular image quality via a local energy ratio based pooling strategy and cyclopean based binocular quality. Extensive experiments on three available benchmark databases demonstrate that the proposed metric has better performance and achieves highly consistent alignment with subjective assessment compared with state-of-the-art SIQA metrics.

Stereoscopic Image Quality Assessment by Considering Binocular Visual Mechanisms

With the booming of 3-D image processing in the entertainment industry and 3-D multimedia applications todays, the technology for assessing the quality of stereoscopic image faces more challenging tasks than its 2-D counterparts, such as binocular combination, stereo matching, and binocular rivalry. In this paper, a novel stereoscopic image quality assessment method is proposed by jointly exploring binocular fusion and rivalry models. In the quality-aware feature extraction stage, multi-scale binocular combination and binocular rivalry energy responses are first generated from reference and distorted stereopairs. In addition, considering that the changes of luminance affect the quality of stereoscopic image greatly, multi-scale visual features relating to image quality are obtained from its luminance maps as another binocular combination and rivalry features. Then, the dissimilarity of quality-aware features between the reference stereopair and its distorted version is quantified. Finally, such dissimilarities are mapped into an objective score to represent the perceptual quality of stereoscopic image through the pooling strategy of support vector regression. Experiments conducted on LIVE 3-D databases demonstrate that the proposed method achieves 96.61% and 96.03% in terms of Pearson’s linear correlation coefficient on Database Phase I and Phase II, respectively, outperforming most of the state-of-the-art methods.

No-Reference Stereoscopic Image Quality Assessment Using Convolutional Neural Network for Adaptive Feature Extraction

The pervasion of 3-D technologies over the years gives rise to the increasing demands of accurate and efficient stereoscopic image quality assessment (SIQA) methods, designed to automatically supervise and optimize 3-D image and video processing systems. Though 2-D IQA has attracted considerable attention, its 3-D counterpart is yet to be well explored. In this paper, a no-reference SIQA method using convolution neural network (CNN) for feature extraction is proposed. In the proposed method, a CNN model is trained from scratch to classify images according to their perceptual quality, with quality-aware monocular features extracted from a higher level layer of the network. Then, visual saliency models are utilized to fuse the captured monocular features. In the meanwhile, multi-scale statistical features are derived from the binocular disparity maps. Finally, the fused CNN features and the disparity features are synthesized by support vector regression into the objective quality score of the stereoscopic image. Experimental results on two public databases demonstrate the superior performance of the proposed method over other state-of-the-art methods, in terms of its accuracy in predicting stereoscopic image quality as well as its robustness across different databases and distortion types.

No‐reference quality assessment for stereoscopic images considering visual discomfort and binocular rivalry

A no-reference stereoscopic image quality assessment method is proposed that takes two primary binocular effects, visual discomfort, and binocular rivalry, into consideration. The adaptive segmentation, local energy, and visual saliency are employed for estimating binocular effects and constructing a quality baseline, which together composes a comprehensive image quality description. Moreover, a compact parallel framework is designed for efficiency. Experiments have validated the high accuracy and robustness of the proposed method.

Stereoscopic image quality assessment by learning non-negative matrix factorization-based color visual characteristics and considering binocular interactions

In this paper, we propose a novel stereoscopic image quality assessment (SIQA) method by learning non-negative matrix factorization (NMF)-based color visual characteristics for monocular perception and considering binocular interactions. In training phase, a feature basis matrix is learned based on NMF by considering color information and a feature detector is designed by performing Schmidt orthogonalization on the feature basis matrix. In construction of SIQA phase, for monocular perception, visual saliency regions are selected and parts-based feature similarity indexes of left and right views based on the feature vectors extracted by the feature detector are calculated. For binocular interactions, we calculate cyclopean feature similarity index by considering binocular fusion and rivalry. Finally, support vector regression is used to simulate nonlinear relationship between monocular perception and binocular interactions. Experimental results on LIVE 3D image databases and NBU 3D IQA database demonstrate that the proposed SIQA method is more consistent with human perception.

Parts-based stereoscopic image assessment by learning binocular manifold color visual properties

Existing stereoscopic image quality assessment (SIQA) methods are mostly based on the luminance information, in which color information is not sufficiently considered. Actually, color is part of the important factors that affect human visual perception, and nonnegative matrix factorization (NMF) and manifold learning are in line with human visual perception. We propose an SIQA method based on learning binocular manifold color visual properties. To be more specific, in the training phase, a feature detector is created based on NMF with manifold regularization by considering color information, which not only allows parts-based manifold representation of an image, but also manifests localized color visual properties. In the quality estimation phase, visually important regions are selected by considering different human visual attention, and feature vectors are extracted by using the feature detector. Then the feature similarity index is calculated and the parts-based manifold color feature energy (PMCFE) for each view is defined based on the color feature vectors. The final quality score is obtained by considering a binocular combination based on PMCFE. The experimental results on LIVE I and LIVE Π 3-D IQA databases demonstrate that the proposed method can achieve much higher consistency with subjective evaluations than the state-of-the-art SIQA methods.

Stereoscopic image quality assessment method based on binocular combination saliency model

The objective quality assessment of stereoscopic images plays an important role in three-dimensional (3D) technologies. In this paper, we propose an effective method to evaluate the quality of stereoscopic images that are afflicted by symmetric distortions. The major technical contribution of this paper is that the binocular combination behaviors and human 3D visual saliency characteristics are both considered. In particular, a new 3D saliency map is developed, which not only greatly reduces the computational complexity by avoiding calculation of the depth information, but also assigns appropriate weights to the image contents. Experimental results indicate that the proposed metric not only significantly outperforms conventional 2D quality metrics, but also achieves higher performance than the existing 3D quality assessment models.

Binocular vision based objective quality assessment method for stereoscopic images

Human visual system (HVS) can perceive the difference between two retinal images to create a mental image with depth perception, which is the result of two binocular interactions, i.e., binocular fusion and suppression. According to perceptual attributes of binocular interactions, in this paper, a full-reference stereoscopic image quality assessment (SIQA) method is proposed based on the mechanisms of binocular fusion and suppression. There are two kinds of information in stereoscopic images: monocular information which is visible in only one view, and binocular information which is visible in two views. HVS adopts two ways to deal with the binocular information, one is binocular fusion which deals with the information with similar content and small disparity, the other is binocular suppression which deals with the information with dissimilar content or large disparity. Therefore, the proposed method firstly divides a distorted stereoscopic image into occluded, pseudo-binocular fusion and pseudo-binocular suppression regions. Then three methods are respectively adopted to assess the quality of the three regions and the three quality indices combine into one to represent the overall quality of the distorted stereoscopic image. Finally, the predictive performance of the proposed method is evaluated and compared with existing methods in terms of consistency, cross-image and cross-distortion, and robustness. Experimental results show that the proposed SIQA method outperforms other methods and can predict human visual perception of stereoscopic image more accurately.