No-reference Stereoscopic Image Quality Assessment Research Articles

Two-stream interactive network based on local and global information for No-Reference Stereoscopic Image Quality Assessment

Nowadays, stereoscopic image quality assessment (SIQA) based on convolutional neural network (CNN) has become the mainstream model of image quality assessment (IQA). Compared with the two-dimensional quality evaluation model, stereoscopic image quality evaluation is more challenging due to the effects of depth and parallax information. In this paper, we propose a two-stream interactive network model to perform quality evaluation, which can well simulate the process of human stereo visual perception. Meanwhile, we enhance the extraction of local and global features of images by asymmetric convolution kernel and interactive sub-networks of inter-layers, respectively, which can further optimize our network model. Our proposed algorithm was evaluated on four public databases. The final experimental results show that our proposed algorithm exhibits good performance not only on the whole database but also on each single distortion type.

A no-Reference Stereoscopic Image Quality Assessment Network Based on Binocular Interaction and Fusion Mechanisms.

In contemporary society full of stereoscopic images, how to assess visual quality of 3D images has attracted an increasing attention in field of Stereoscopic Image Quality Assessment (SIQA). Compared with 2D-IQA, SIQA is more challenging because some complicated features of Human Visual System (HVS), such as binocular interaction and binocular fusion, must be considered. In this paper, considering both binocular interaction and fusion mechanisms of the HVS, a hierarchical no-reference stereoscopic image quality assessment network (StereoIF-Net) is proposed to simulate the whole quality perception of 3D visual signals in human cortex, including two key modules: BIM and BFM. In particular, Binocular Interaction Modules (BIMs) are constructed to simulate binocular interaction in V2-V5 visual cortex regions, in which a novel cross convolution is designed to explore the interaction details in each region. In the BIMs, different output channel numbers are designed to imitate various receptive fields in V2-V5. Furthermore, a Binocular Fusion Module (BFM) with automatic learned weights is proposed to model binocular fusion of the HVS in higher cortex layers. The verification experiments are conducted on the LIVE 3D, IVC and Waterloo-IVC SIQA databases and three indices including PLCC, SROCC and RMSE are employed to evaluate the assessment consistency between StereoIF-Net and the HVS. The proposed StereoIF-Net achieves almost the best results compared with advanced SIQA methods. Specifically, the metric values on LIVE 3D, IVC and WIVC-I are the best, and are the second-best on the WIVC-II.

No-reference stereoscopic image quality assessment on both complex contourlet and spatial domain via Kernel ELM

Stereoscopic imaging is widely used in many fields. To guarantee the best quality of experience, it is necessary to design a robust and accurate quality assessment model for stereoscopic content. In this paper, we proposed a no-reference stereoscopic image quality assessment (NR-SIQA) model using both complex contourlet and spatial domain features of monocular and binocular images. Monocular features extracted from the CIELAB color space are exploited to characterize quality degradation, including the across-scale and across-orientation correlation in complex contourlet domain and natural scene statistics-based (NSS) features in spatial domain. Then, the binocular features consist of energy along with energy difference and structural correlation in complex contourlet domain, and statistics distribution in spatial domain extracted from the synthesized cyclopean image and the 3D visual discomfort measure based on the statistics of disparity map. Finally, the above features are mapped into the predicted quality scores by the Kernel ELM (KELM) regression model. Experimental results on four public datasets show that the proposed model is highly consistent with human subjective perception in terms of accuracy and generalization.

No-reference stereoscopic image quality evaluator based on human visual characteristics and relative gradient orientation

Stereoscopic image quality assessment (SIQA) is of great significance to the development of modern three-dimensional (3D) display technology. In this work, by further mining the relationship between visual features and stereoscopic image quality perception, we build a new no-reference SIQA model, which combines the monocular and binocular features. Statistical quality-aware structural features from relative gradient orientation (RGO) map and texture features from the histogram of the weighted local binary pattern (LBP) in the texture image (TLBP) are not only extracted from both monocular view, but also extracted from binocular views to predict binocular quality perception. Meanwhile, the color statistical features ignored by most models and the binocularity feature is extracted to complement the monocular features and the above binocular features, respectively. Finally, all the extracted features and subjective scores are used to predict the objective quality score through the support vector regression (SVR) model. Experiments on four popular stereoscopic image databases show that the proposed model achieves high consistency with subjective assessment, and the performance of the model is very competitive with the latest models.

No-reference stereoscopic image quality assessment using quaternion wavelet transform and heterogeneous ensemble learning

As the demand for high-quality stereo images has grown in recent years, stereoscopic image quality assessment (SIQA) has become an important research area in modern image processing technology.In this paper, we propose a no-reference stereoscopic image quality assessment (NR-SIQA) model using heterogeneous ensemble learning ‘quality-aware’ features from luminance image, chrominance image, disparity and cyclopean images via quaternion wavelet transform (QWT). Firstly, luminance image and chrominance image are generated by CIELAB color space as monocular perception, and the novel disparity and cyclopean images are utilized to complement with monocular information. Then, a number of ‘quality-aware’ features in the quaternion wavelet domain are discovered, including entropy, texture features, energy features, energy differences features and MSCN coefficients of high frequency sub-band. Finally, a heterogeneous ensemble model via support vector regression (SVR) & extreme learning machine (ELM) & random forest (RF) is proposed to predict quality score, and bootstrap sampling and rotated feature space are used to increase the diversity of data distribution. Comparing with the state-of-the-art NR-SIQA models, experimental results on four public databases prove the accuracy and robustness of the proposed model.

No-reference stereoscopic image quality assessment using 3D visual saliency maps fused with three-channel convolutional neural network

No-reference stereoscopic image quality assessment using 3D visual saliency maps fused with three-channel convolutional neural network

No-Reference Stereoscopic Image Quality Assessment Based on Binocular Statistical Features and Machine Learning

Learning a deep structure representation for complex information networks is a vital research area, and assessing the quality of stereoscopic images or videos is challenging due to complex 3D quality factors. In this paper, we explore how to extract effective features to enhance the prediction accuracy of perceptual quality assessment. Inspired by the structure representation of the human visual system and the machine learning technique, we propose a no-reference quality assessment scheme for stereoscopic images. More specifically, the statistical features of the gradient magnitude and Laplacian of Gaussian responses are extracted to form binocular quality-predictive features. After feature extraction, these features of distorted stereoscopic image and its human perceptual score are used to construct a statistical regression model with the machine learning technique. Experimental results on the benchmark databases show that the proposed model generates image quality prediction well correlated with the human visual perception and delivers highly competitive performance with the typical and representative methods. The proposed scheme can be further applied to the real-world applications on video broadcasting and 3D multimedia industry.

Learning Local Quality-Aware Structures of Salient Regions for Stereoscopic Images via Deep Neural Networks

The perceptual quality of stereoscopic images plays an essential role in the human perception of visual information. However, most available stereoscopic image quality assessment (SIQA) methods evaluate 3D visual experience using hand-crafted features or shallow architectures, which cannot model the visual properties of stereo images well. In this paper, we use convolutional neural networks (CNNs) to learn deeper local quality-aware structures for stereo images. With different inputs, two CNN models are designed for no-reference SIQA tasks. The one-column CNN model directly accepts a cyclopean view as the input, and the three-column CNN model jointly considers the cyclopean, left and right views as CNN inputs. The two SIQA frameworks share the same implementation approach: First, to overcome the obstacle of limited SIQA datasets, we accept image patches that have been cropped from corresponding stereopairs as inputs for local quality-sensitive feature extraction. Next, a local feature selection algorithm is used to remove related features on non-salient patches, which could cause large prediction errors. Finally, the reserved local visual structures of salient regions are aggregated into a final quality score in an end-to-end manner. Experimental results on three public SIQA databases demonstrate that our method outperforms most state-of-the-art no-reference (NR) SIQA methods. The results of a cross-database experiment also show the robustness and generality of the proposed method.

No-reference stereoscopic image quality assessment based on global and local content characteristics

No-reference stereoscopic images quality assessment (NR-SIQA) via deep learning has gained increasing attention. In this paper, we propose a no-reference stereoscopic image quality assessment method based on global and local content characteristics. The proposed method simulates the perception route of human visual system, and derives features from the fused view and single view through the global feature fusion sub-network and local feature enhancement sub-network. As for the fused view, a cross-fusion strategy is applied to model the process in the V1 visual cortex, and the multi-scales pooling (MSP) is utilized to integrate context information under different sub-regions for effective global feature extraction. As for the single view, the asymmetric convolution block (ACB) is introduced to strengthen the local information description. By jointly considering the fused view and single view, the proposed network can efficiently extract the features for quality assessment. Finally, a weighted average strategy is applied to estimate the visual quality of stereoscopic image. Experimental results on 3D quality databases demonstrate that the proposed network is superior to the state-of-the-art metrics, and achieves an excellent performance.

No-reference stereoscopic image quality evaluator with segmented monocular features and perceptual binocular features

No-reference stereoscopic image quality assessment (NR SIQA) has aroused much attention in the multimedia application, but most current works cannot achieve reliable accuracy due to the non-visual meaning image patches and the weak biological visual models. Inspired by this, we propose a novel NR SIQA model by the introduction of the meaningful superpixel visual patches and the significant binocular perception models. Following the visual perceptual posterior probability theory, local monocular superpixel spatial entropy and the natural scene statistics (NSS) features are firstly utilized to present the high level semantic perceptual mechanism and image naturalness. Then the binocular biological models are applied to extract the binocular features to simulate the inner binocular interaction information, which well complement with the monocular features. Finally, a support vector regression (SVR) model is utilized to predict the objective quality score. Experiments are conducted on four popular stereoscopic image Databases, and the results demonstrate that the proposed model obtains a good consistency with the subjective scores, and has more reliable performance than previous works in terms of prediction accuracy and generalization capability.

AdaBoost neural network and cyclopean view for no-reference stereoscopic image quality assessment

Stereoscopic imaging is widely used in many fields. In many scenarios, stereo images quality could be affected by various degradations, such as asymmetric distortion. Accordingly, to guarantee the best quality of experience, robust and accurate reference-less metrics are required for quality assessment of stereoscopic content. Most existing stereo no-reference Image Quality Assessment (IQA) models are not consistent with asymmetrical distortions. This paper presents a new no-reference stereoscopic image quality assessment metric using a human visual system (HVS) modeling and an advanced machine-learning algorithm. The proposed approach consists of two stages. In the first stage, cyclopean image is constructed considering the presence of binocular rivalry in order to cover the asymmetrically distorted part. In the second stage, gradient magnitude, relative gradient magnitude, and gradient orientation are extracted. These are used as a predictive source of information for the quality. In order to obtain the best overall performance against different databases, Adaptive Boosting (AdaBoost) idea of machine learning combined with artificial neural network model has been adopted. The benchmark LIVE 3D phase-I, phase-II, and IRCCyN/IVC 3D databases have been used to evaluate the performance of the proposed approach. Experimental results have demonstrated that the proposed metric performance achieves high consistency with subjective assessment and outperforms the blind stereo IQA over various types of distortion.

No-reference stereoscopic image quality assessment using a multi-task CNN and registered distortion representation

Scene discrepancy between the left and right views presents more challenges for image quality assessment (IQA) of stereoscopic images as opposed to monocular ones. Existing no-reference stereoscopic IQA (NR-SIQA) metrics cannot achieve a good performance on asymmetrically distorted stereoscopic images. In this paper, we propose an NR-SIQA index that first addresses scene discrepancy by means of image registration. It then uses a registered distortion representation based on the left and registered right views to represent the distortion in the stereoscopic image. Because different distortion types influence image quality differently, a multi-task convolutional neural network (CNN) is employed to learn image quality prediction and distortion-type identification simultaneously. We first design a one-column multi-task CNN model, that learns from the registered distortion representation. Then, we extend the one-column model to a three-column model, which also learns from the left and right views. Our experimental results validate the effectiveness of the proposed registered distortion representation and multi-task CNN architecture. The proposed one- and three-column models outperform the state-of-the-art NR-SIQA metrics, especially for asymmetrically distorted stereoscopic images.

No-reference stereoscopic image quality assessment based on cyclopean image and enhanced image

In order to effectively evaluate the quality of stereoscopic images, we propose a no-reference stereoscopic image quality assessment (SIQA) method. Firstly, considering the characteristics of binocular fusion, binocular rivalry, and binocular suppression of human visual system, we propose a new color cyclopean image which is suitable for symmetric and asymmetric distortion images. And then, based on the importance of the disparity map, the enhanced image is generated according to the cyclopean image and the disparity map. Next, the natural statistical features are extracted from the enhanced image and the cyclopean image weighted by the gradient of disparity map (named weighted cyclopean image) in the spatial domain. The kurtosis and skewness are extracted from disparity map. Finally, the extracted features are fused and the quality of stereoscopic image is obtained by support vector regression. Experimental results show that the proposed algorithm is superior to most existing objective SIQA methods and can maintain a high degree of consistency with the subjective scores.

Dual-Stream Interactive Networks for No-Reference Stereoscopic Image Quality Assessment.

The goal of objective stereoscopic image quality assessment (SIQA) is to predict the human perceptual quality of stereoscopic/3D images automatically and accurately. Compared with traditional 2D image quality assessment, the quality assessment of stereoscopic images is more challenging because of complex binocular vision mechanisms and multiple quality dimensions. In this paper, inspired by the hierarchical dual-stream interactive nature of the human visual system, we propose a stereoscopic image quality assessment network (StereoQA-Net) for no-reference stereoscopic image quality assessment. The proposed StereoQA-Net is an end-to-end dual-stream interactive network containing left and right view sub-networks, where the interaction of the two sub-networks exists in multiple layers. We evaluate our method on the LIVE stereoscopic image quality databases. The experimental results show that our proposed StereoQA-Net outperforms state-of-the-art algorithms on both symmetrically and asymmetrically distorted stereoscopic image pairs of various distortion types. In a more general case, the proposed StereoQA-Net can effectively predict the perceptual quality of local regions. In addition, cross-dataset experiments also demonstrate the generalization ability of our algorithm.

No-Reference Stereoscopic Image Quality Assessment Based on Image Distortion and Stereo Perceptual Information

In this paper, we propose a novel no-reference three-dimensional (3-D) image quality assessment (NR-SIQA) method, which takes three types of visual perceptual characteristics into consideration, including image distortion, depth perception, and binocular combination perception. The estimation of image distortion is based on the observation that blurriness, noisiness, and blockiness are considered as three important factors that affect image quality. Moreover, the value difference between any two neighboring pixels for any image satisfies a generalized Laplace distribution, which will be changed with various types of distortion existing in the image. Since the major difference between 2-D images and 3-D images is the depth, a disparity search algorithm is a key element for 3-D perception assessment. To address this, a robust disparity search algorithm is first designed based on the Gaussian average SSIM, followed by the generation of cyclopean image, rivalry map, depth map, and weight map. These four maps have taken the visual perceptual characteristics into consideration. Three types of features relating to image distortion, depth perception, and binocular disparity are extracted from these maps and stereo-pairs. In the end, machine learning is utilized to map these features to human perceptual score. Experimental results show that the proposed algorithm outperforms the recent full reference SIQA and NR-SIQA metrics on two publicly available stereoscopic image quality assessment databases.

Multiple just-noticeable-difference-based no-reference stereoscopic image quality assessment

Just-noticeable difference (JND) is an important characteristic of the human visual system (HVS), and some established JND models imitating the perception of human eyes already exist. However, their utilization in stereoscopic image quality assessment (SIQA) remains limited. To better simulate how HVS senses 3D images under a no-reference situation, a novel SIQA method based on multiple JND models is proposed in this paper. In our metric, the stereoscopic image pairs are decomposed into multi-scale monocular views and binocular views. Then, texture and edge information of these multi-scale images is extracted. Next, a monocular JND model, a binocular JND model, and a depth JND model are separately applied to the extracted features and the depth map. Finally, these features are synthesized and mapped to objective scores. Through experiment and comparison on public 3D image databases, the proposed method shows a competitive advantage over most state-of-the-art SIQA methods, which indicates that it has a promising prospect in practical applications.

No-Reference Stereoscopic Image Quality Assessment Based on Visual Attention and Perception

In recent years, the methods of no-reference stereoscopic image quality assessment (NR-SIQA) have been well investigated, but there still remain challenges due to the inaccurate extraction of binocular perception information. In this paper, we propose an NR-SIQA method based on visual attention and perception. We combine saliency and just noticeable difference (JND) to model visual attention and perception, respectively, and weight the global and local features extracted from the left and right views. Meanwhile, in order to obtain the accurate binocular perception information, the global structural features reflecting spatial correlation are extracted from the cyclopean map that is synthesized by the left and right views. Then, a regression model is learned based on a support vector machine regression (SVR) to evaluate the quality of stereoscopic images. The experiments on popular SIQA datasets demonstrate that the proposed NR-SIQA method has better and more reliable performance than the state-of-the-art methods.

Stereoscopic image quality assessment combining statistical features and binocular theory

Stereoscopic image quality evaluation is extremely significance as a performance evaluator of modern 3D display technology. Due to the complexity of human visual system (HVS) and the incomprehensive study of stereoscopic perception of human eyes, stereoscopic image quality assessment (SIQA) is still a challenging task. In this paper, combining binocular characteristics, we propose an efficient no-reference stereoscopic image quality assessment according to binocular adding and subtracting channels. Distinguished from other SIQA methods, which pay attention to complex binocular visual properties, the visual information which is closely bound up with image distortion from adding and subtracting channels to describe ocular dominance (alternate name binocularity) is proposed. To estimate the contribution of each channel in SIQA, a dynamic weighting system is proposed for binocular fusion according to local energy. Furthermore, quality awareness features based on multi-scale and multi-orientation are extracted to describe visual degradation. Comparing with existing methods, experimental results on public 3D image databases demonstrate the proposed framework achieves high consistent with the subjective quality scores.

Multistage Pooling for Blind Quality Prediction of Asymmetric Multiply-Distorted Stereoscopic Images

Quality prediction for asymmetric multiply-distorted stereoscopic images (MDSIs) confronts more challenges than previous stereoscopic image quality assessment (SIQA) issues, whereas the existing no-reference SIQA methods have been limited to understand the asymmetric distortions and multiple distortions simultaneously for general-purpose blind quality prediction. In this paper, we propose a multistage pooling (MUSP) model for quality prediction of asymmetric MDSIs. In the training stage, we establish multimodal sparse representation framework for phase and amplitude components, respectively. In the testing stage, we use an MUSP strategy to simulate the pooling procedure undergoing multimodal quality pooling, feature pooling, binocular pooling, and phase-amplitude quality pooling in order. Experimental results on our new established database (NBU-MDSID Phase-II) demonstrate the effectiveness of our blind metric.

No-reference stereoscopic image quality assessment based on hue summation-difference mapping image and binocular joint mutual filtering.

The no-reference (NR) quality assessment for stereoscopic images plays a significant role in 3D technology, but it also faces great challenges. In this paper, a novel NR stereo image quality assessment (SIQA) method is proposed. Based on the human visual system, this method mimics the summation and difference channels, which consider the binocular interactive perception property, to process the visual information. Especially, the summation and difference images are calculated by the contrast of hue and luminance in color patches. Meanwhile, considering the interactive filtering between the left and right viewpoints, this method uses the filtered information as the weighting factor to integrate the visual information of the summation and difference channels to form the summation-difference mapping image (SDMI). Then, energy entropy, bivariate generalized Gaussian distribution for the joint distribution of SDMI and the depth map subband coefficients, and the local log-Euclidean multivariate Gaussian descriptor are detected as the feature descriptors. Support vector regression, trained by the features, is utilized to predict the quality of stereoscopic images. Experimental results demonstrate that the proposed algorithm achieves high consistency with subjective assessment on four SIQA databases.