Stereoscopic Image Quality Assessment Research Articles

Quality assessment of stereoscopic images under acousto-optic diffraction in paratellurite crystal

Quality assessment of stereoscopic images under acousto-optic diffraction in paratellurite crystal

Image2Brain: a cross-modality model for blind stereoscopic image quality ranking

Objective. Human beings perceive stereoscopic image quality through the cerebral visual cortex, which is a complex brain activity. As a solution, the quality of stereoscopic images can be evaluated more accurately by attempting to replicate the human perception from electroencephalogram (EEG) signals on image quality in a machine, which is different from previous stereoscopic image quality assessment methods focused only on the extraction of image features. Approach. Our proposed method is based on a novel image-to-brain (I2B) cross-modality model including a spatial-temporal EEG encoder (STEE) and an I2B deep convolutional generative adversarial network (I2B-DCGAN). Specifically, the EEG representations are first learned by STEE as real samples of I2B-DCGAN, which is designed to extract both quality and semantic features from the stereoscopic images by a semantic-guided image encoder, and utilize a generator to conditionally create the corresponding EEG features for images. Finally, the generated EEG features are classified to predict the image perceptual quality level. Main results. Extensive experimental results on the collected brain-visual multimodal stereoscopic image quality ranking database, demonstrate that the proposed I2B cross-modality model can better emulate the visual perception mechanism of the human brain and outperform the other methods by achieving an average accuracy of 95.95. Significance. The proposed method can convert the learned stereoscopic image features into brain representations without EEG signals during testing. Further experiments verify that the proposed method has good generalization ability on new datasets and the potential for practical applications.

A Novel No-Reference Quality Assessment Metric for Stereoscopic Images with Consideration of Comprehensive 3D Quality Information.

Recently, stereoscopic image quality assessment has attracted a lot attention. However, compared with 2D image quality assessment, it is much more difficult to assess the quality of stereoscopic images due to the lack of understanding of 3D visual perception. This paper proposes a novel no-reference quality assessment metric for stereoscopic images using natural scene statistics with consideration of both the quality of the cyclopean image and 3D visual perceptual information (binocular fusion and binocular rivalry). In the proposed method, not only is the quality of the cyclopean image considered, but binocular rivalry and other 3D visual intrinsic properties are also exploited. Specifically, in order to improve the objective quality of the cyclopean image, features of the cyclopean images in both the spatial domain and transformed domain are extracted based on the natural scene statistics (NSS) model. Furthermore, to better comprehend intrinsic properties of the stereoscopic image, in our method, the binocular rivalry effect and other 3D visual properties are also considered in the process of feature extraction. Following adaptive feature pruning using principle component analysis, improved metric accuracy can be found in our proposed method. The experimental results show that the proposed metric can achieve a good and consistent alignment with subjective assessment of stereoscopic images in comparison with existing methods, with the highest SROCC (0.952) and PLCC (0.962) scores being acquired on the LIVE 3D database Phase I.

Blind Quality Assessment of Stereoscopic Images Considering Binocular Perception Based on Shearlet Decomposition

Blind Quality Assessment of Stereoscopic Images Considering Binocular Perception Based on Shearlet Decomposition

Binocular rivalry-based stereoscopic images quality assessment relevant to its asymmetric and distorted contexts

Binocular rivalry-based stereoscopic images quality assessment relevant to its asymmetric and distorted contexts

Two-stream interactive network based on local and global information for No-Reference Stereoscopic Image Quality Assessment

Nowadays, stereoscopic image quality assessment (SIQA) based on convolutional neural network (CNN) has become the mainstream model of image quality assessment (IQA). Compared with the two-dimensional quality evaluation model, stereoscopic image quality evaluation is more challenging due to the effects of depth and parallax information. In this paper, we propose a two-stream interactive network model to perform quality evaluation, which can well simulate the process of human stereo visual perception. Meanwhile, we enhance the extraction of local and global features of images by asymmetric convolution kernel and interactive sub-networks of inter-layers, respectively, which can further optimize our network model. Our proposed algorithm was evaluated on four public databases. The final experimental results show that our proposed algorithm exhibits good performance not only on the whole database but also on each single distortion type.

Monocular and Binocular Interactions Oriented Deformable Convolutional Networks for Blind Quality Assessment of Stereoscopic Omnidirectional Images

Stereoscopic omnidirectional content, as a novel visual media, has drawn wide attention in recent years due to its ability in providing strong immersive experience. Since Stereoscopic Omnidirectional Images (SOIs) involve the properties from panoramic and stereoscopic visual perception, it is very challenging to establish an efficient and effective visual quality evaluation model for SOIs. To better measure the user’s experience in virtual reality, we put forward a novel deep learning framework to assess the quality of SOIs in this paper. Firstly, the deformable convolutions instead of standard convolutions are adopted to ensure the invariant receptive fields of convolutional kernels on Equi-Rectangular Projection (ERP). Secondly, according to the stereoscopic property, we use binocular-difference information and a coarse-to-fine mechanism to construct the binocular feature extraction network. Thirdly, a three-channel network involving left-view, right-view and binocular-difference channels is presented to simulate the process of monocular and binocular interactions, in which independent quality labels are provided for each channel to reflect the individual effect of monocular and binocular visions on the whole visual quality. Finally, experimental results on two available benchmark databases demonstrate the superiority of the proposed metric over the state-of-the-art blind quality assessment models in predicting the quality of SOIs. Moreover, our model is efficient in computational cost as the feature extraction is directly applied on ERP images.

Blind stereoscopic image quality assessment using 3D saliency selected binocular perception and 3D convolutional neural network

Blind stereoscopic image quality assessment using 3D saliency selected binocular perception and 3D convolutional neural network

Rich Structural Index for Stereoscopic Image Quality Assessment.

The human visual system (HVS), affected by viewing distance when perceiving the stereo image information, is of great significance to study of stereoscopic image quality assessment. Many methods of stereoscopic image quality assessment do not have comprehensive consideration for human visual perception characteristics. In accordance with this, we propose a Rich Structural Index (RSI) for Stereoscopic Image objective Quality Assessment (SIQA) method based on multi-scale perception characteristics. To begin with, we put the stereo pair into the image pyramid based on Contrast Sensitivity Function (CSF) to obtain sensitive images of different resolution. Then, we obtain local Luminance and Structural Index (LSI) in a locally adaptive manner on gradient maps which consider the luminance masking and contrast masking. At the same time we use Singular Value Decomposition (SVD) to obtain the Sharpness and Intrinsic Structural Index (SISI) to effectively capture the changes introduced in the image (due to distortion). Meanwhile, considering the disparity edge structures, we use gradient cross-mapping algorithm to obtain Depth Texture Structural Index (DTSI). After that, we apply the standard deviation method for the above results to obtain contrast index of reference and distortion components. Finally, for the loss caused by the randomness of the parameters, we use Support Vector Machine Regression based on Genetic Algorithm (GA-SVR) training to obtain the final quality score. We conducted a comprehensive evaluation with state-of-the-art methods on four open databases. The experimental results show that the proposed method has stable performance and strong competitive advantage.

Hierarchical multi-scale stereoscopic image quality assessment based on visual mechanism

Hierarchical multi-scale stereoscopic image quality assessment based on visual mechanism

Blind Stereoscopic Image Quality Evaluator Based on Binocular Semantic and Quality Channels

Human beings always evaluate the perceptual quality of an image coupled with identifying the semantic content of images. This paper addresses the correlation issue between stereoscopic image quality assessment (SIQA) and semantic recognition. In contrast to the previous SIQA methods that relied on binocular quality-aware features of a stereoscopic image, our approach also extracts binocular semantic features using a pre-trained deep convolutional neural network (DCNN) on a large dataset like ImageNet dataset, as well as the manually designed binocular quality-aware features. It can solve the problem of limited SIQA dataset size and facilitate better prediction on the quality. Experimental results demonstrate that the binocular semantic features are a good predictor for the stereoscopic image quality. The proposed method outperforms the state-of-the-art SIQA methods on four benchmark SIQA datasets. Significantly, all Spearman rank-order correlation coefficients (SROCCs) between the predicted scores and the subjective scores on the four datasets exceed 0.95. The MATLAB source code of the proposed method is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/kyohoonsim/Blind-Stereoscopic-Image-Quality-Evaluator-based-on-Binocular-Semantic-and-Quality-Channels</uri> <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/kyohoonsim</uri> .

A no-Reference Stereoscopic Image Quality Assessment Network Based on Binocular Interaction and Fusion Mechanisms.

In contemporary society full of stereoscopic images, how to assess visual quality of 3D images has attracted an increasing attention in field of Stereoscopic Image Quality Assessment (SIQA). Compared with 2D-IQA, SIQA is more challenging because some complicated features of Human Visual System (HVS), such as binocular interaction and binocular fusion, must be considered. In this paper, considering both binocular interaction and fusion mechanisms of the HVS, a hierarchical no-reference stereoscopic image quality assessment network (StereoIF-Net) is proposed to simulate the whole quality perception of 3D visual signals in human cortex, including two key modules: BIM and BFM. In particular, Binocular Interaction Modules (BIMs) are constructed to simulate binocular interaction in V2-V5 visual cortex regions, in which a novel cross convolution is designed to explore the interaction details in each region. In the BIMs, different output channel numbers are designed to imitate various receptive fields in V2-V5. Furthermore, a Binocular Fusion Module (BFM) with automatic learned weights is proposed to model binocular fusion of the HVS in higher cortex layers. The verification experiments are conducted on the LIVE 3D, IVC and Waterloo-IVC SIQA databases and three indices including PLCC, SROCC and RMSE are employed to evaluate the assessment consistency between StereoIF-Net and the HVS. The proposed StereoIF-Net achieves almost the best results compared with advanced SIQA methods. Specifically, the metric values on LIVE 3D, IVC and WIVC-I are the best, and are the second-best on the WIVC-II.

Deep network based stereoscopic image quality assessment via binocular summing and differencing

With the development of deep networks in dealing with various visual tasks, the deep network based on binocular vision is expected to tackle the issue of stereoscopic image quality assessment. Here, we present a stereoscopic image quality assessment method using the deep network with four channels together, which takes the left view, right view, binocular summing view, and binocular differencing view as the inputs of the network. The visual features are enhanced through the concatenation in a weighted way, so that the binocular vision can be adequately included in the binocular addition and subtraction information. Compared with the state-of-the-art metrics, the proposed method exhibits relatively high performances on four benchmark databases.

No-reference stereoscopic image quality assessment on both complex contourlet and spatial domain via Kernel ELM

Stereoscopic imaging is widely used in many fields. To guarantee the best quality of experience, it is necessary to design a robust and accurate quality assessment model for stereoscopic content. In this paper, we proposed a no-reference stereoscopic image quality assessment (NR-SIQA) model using both complex contourlet and spatial domain features of monocular and binocular images. Monocular features extracted from the CIELAB color space are exploited to characterize quality degradation, including the across-scale and across-orientation correlation in complex contourlet domain and natural scene statistics-based (NSS) features in spatial domain. Then, the binocular features consist of energy along with energy difference and structural correlation in complex contourlet domain, and statistics distribution in spatial domain extracted from the synthesized cyclopean image and the 3D visual discomfort measure based on the statistics of disparity map. Finally, the above features are mapped into the predicted quality scores by the Kernel ELM (KELM) regression model. Experimental results on four public datasets show that the proposed model is highly consistent with human subjective perception in terms of accuracy and generalization.

No-reference stereoscopic image quality evaluator based on human visual characteristics and relative gradient orientation

Stereoscopic image quality assessment (SIQA) is of great significance to the development of modern three-dimensional (3D) display technology. In this work, by further mining the relationship between visual features and stereoscopic image quality perception, we build a new no-reference SIQA model, which combines the monocular and binocular features. Statistical quality-aware structural features from relative gradient orientation (RGO) map and texture features from the histogram of the weighted local binary pattern (LBP) in the texture image (TLBP) are not only extracted from both monocular view, but also extracted from binocular views to predict binocular quality perception. Meanwhile, the color statistical features ignored by most models and the binocularity feature is extracted to complement the monocular features and the above binocular features, respectively. Finally, all the extracted features and subjective scores are used to predict the objective quality score through the support vector regression (SVR) model. Experiments on four popular stereoscopic image databases show that the proposed model achieves high consistency with subjective assessment, and the performance of the model is very competitive with the latest models.

No-reference stereoscopic image quality assessment using quaternion wavelet transform and heterogeneous ensemble learning

As the demand for high-quality stereo images has grown in recent years, stereoscopic image quality assessment (SIQA) has become an important research area in modern image processing technology.In this paper, we propose a no-reference stereoscopic image quality assessment (NR-SIQA) model using heterogeneous ensemble learning ‘quality-aware’ features from luminance image, chrominance image, disparity and cyclopean images via quaternion wavelet transform (QWT). Firstly, luminance image and chrominance image are generated by CIELAB color space as monocular perception, and the novel disparity and cyclopean images are utilized to complement with monocular information. Then, a number of ‘quality-aware’ features in the quaternion wavelet domain are discovered, including entropy, texture features, energy features, energy differences features and MSCN coefficients of high frequency sub-band. Finally, a heterogeneous ensemble model via support vector regression (SVR) & extreme learning machine (ELM) & random forest (RF) is proposed to predict quality score, and bootstrap sampling and rotated feature space are used to increase the diversity of data distribution. Comparing with the state-of-the-art NR-SIQA models, experimental results on four public databases prove the accuracy and robustness of the proposed model.

Stereoscopic image quality assessment considering visual mechanism and multi-loss constraints

In this paper, a convolutional neural network (CNN) with multi-loss constraints is designed for stereoscopic image quality assessment (SIQA). A stereoscopic image not only contains monocular information, but also provides binocular information which is as identically crucial as the former. So we take the image patches of left-view images, right-view images and the difference images as the inputs of the network to utilize monocular information and binocular information. Moreover, we propose a method to obtain proxy label of each image patch. It preserves the quality difference between different regions and views. In addition, the multiple loss functions with adaptive loss weights are introduced in the network, which consider both local features and global features and constrain the feature learning from multiple perspectives. And the adaptive loss weights also make the multi-loss CNN more flexible. The experimental results on four public SIQA databases show that the proposed method is superior to other existing SIQA methods with state-of-the-art performance.

No-reference stereoscopic image quality assessment using 3D visual saliency maps fused with three-channel convolutional neural network

No-reference stereoscopic image quality assessment using 3D visual saliency maps fused with three-channel convolutional neural network

Projection Invariant Feature and Visual Saliency-Based Stereoscopic Omnidirectional Image Quality Assessment

In this article, we propose a quality assessment model-based on the projection invariant feature and the visual saliency for Stereoscopic Omnidirectional Images (SOIs). Firstly, the projection invariant monocular and binocular features of SOI are derived from the Scale-Invariant Feature Transform (SIFT) points to tackle the inconsistency between the stretched projection formats and the viewports. Secondly, the visual saliency model, which combines the chrominance and contrast perceptual factors, is used to facilitate the prediction accuracy. Thirdly, according to the characteristics of the panoramic image, we generate the weight map and utilize it as a location prior, which can be adapted to different projection formats. Finally, the proposed SOI quality assessment model fuses the projection invariant features, visual saliency, and location prior. Experimental results on both the NingBo University SOI Database (NBU-SOID) and Stereoscopic OmnidirectionaL Image quality assessment Database (SOLID) demonstrate the proposed metric on equi-rectangular projection format outperforms the state-of-the-art schemes, the pearson linear correlation coefficient and spearman rank order correlation coefficient performance are 0.933 and 0.933 on SOLID, and 0.907 and 0.910 on NBU-SOID, respectively. Meanwhile, the proposed algorithm is extended to another five representative projection formats and achieves superior performance.

Quality assessment of multiply and singly distorted stereoscopic images via adaptive construction of cyclopean views

A challenging problem confronted when designing a blind/no-reference (NR) stereoscopic image quality assessment (SIQA) algorithm is to simulate the quality assessment (QA) behavior of the human visual system (HVS) during binocular vision. An effective way to solve this problem is to estimate the quality of the merged single view created in the human brain which is also referred to as the cyclopean image. However, due to the difficulty in modeling the binocular fusion and rivalry properties of the HVS, obtaining effective cyclopean images for QA is non-trivial, and consequently previous NR SIQA algorithms either require the MOS/DMOS values of the distorted 3D images for training or ignore the quality analysis of the merged cyclopean view. In this paper, we focus on (1) constructing accurate and appropriate cyclopean views for QA of stereoscopic images by adaptively analyzing the distortion information of two monocular views, and (2) training NR SIQA models without requiring the assistance of the MOS/DMOS values in existing databases. Accordingly, we present an effective opinion-unaware SIQA algorithm called MUSIQUE-3D, which blindly assesses the quality of multiply and singly distorted stereoscopic images by analyzing quality degradations of both monocular and cyclopean views. The monocular view quality is estimated by an extended version of the MUSIQUE algorithm, and the cyclopean view quality is computed from the distortion parameter values predicted by a two-layer classification-regression model trained on a large 3D image dataset. Tests on various 3D image databases demonstrate the superiority of our method as compared with other state-of-the-art SIQA algorithms.