Stereoscopic Video Quality Assessment Research Articles

Stereoscopic Video Quality Assessment in the Context of Internet of Things

Under the background of Internet of Things (IoT), users pursue a higher quality of immer-sive visual experience, including virtual reality (VR), augmented reality and stereoscopic information. In order to ensure the quality of experience (QoE), an effective quality assessment method is proposed to accurately determine the quality of information. Specifically, for the large amount of stereo video data, the representative sequence of stereo video is extract to reduce the data volume. Inspired by binocular channel theory, operators of the difference map and the novel fusion map are executed on these sequences. Then a dynamic texture descriptor, known as volume local binary pattern (VLBP), is employed to represent the spatio-temporal domain to predict the stereo video quality. Experiments have been executed on two public databases and results show that our method has excellent quality prediction capability.

No-Reference Quality Assessment of Stereoscopic Video Based on Temporal Adaptive Model for Improved Visual Communication

An objective stereo video quality assessment (SVQA) strives to be consistent with human visual perception while ensuring a low time and labor cost of evaluation. The temporal-spatial characteristics of video make the data processing volume of quality evaluation surge, making an SVQA more challenging. Aiming at the effect of distortion on the stereoscopic temporal domain, a stereo video quality assessment method based on the temporal-spatial relation is proposed in this paper. Specifically, a temporal adaptive model (TAM) for a video is established to describe the space-time domain of the video from both local and global levels. This model can be easily embedded into any 2D CNN backbone network. Compared with the improved model based on 3D CNN, this model has obvious advantages in operating efficiency. Experimental results on NAMA3DS1-COSPAD1 database, WaterlooIVC 3D Video Phase I database, QI-SVQA database and SIAT depth quality database show that the model has excellent performance.

Three-Stream 3D deep CNN for no-Reference stereoscopic video quality assessment

• An end-to-end learning structure for the Stereoscopic Video Quality Assessment with three pipelines to extract the quality degradation from spatial, temporal, and disparity information. • A non-homogeneous data augmentation system that splits the stereoscopic video into small cube patches and removes the outliers from the created dataset using entropy criteria. • A robust deep depth quality-aware feature extractor used to extract motion and depth maps jointly and feed them synchronously to the deep network to extract quality discriminatory features. • Two popular datasets are used to validate the proposed learning architecture for Stereoscopic Video Quality Assessment and received state-of-the-art results in both datasets. Convolutional Neural Networks (CNNs) have achieved great success in learning computer vision tasks, particularly 3D CNNs, for extracting spatio-temporal features from the given videos. However, 3D CNNs have not been well-examined for the Stereoscopic Video Quality Assessment (SVQA). To our best knowledge, most of the state-of-the-art methods used the traditional hand-crafted feature extraction methods for the SVQA. Very few methods used the power of deep learning for SVQA, and they just considered the spatial information, ignoring the disparity and motion information. In this paper, we propose a No-Reference (NR) deep 3D CNN architecture that jointly focuses on spatial, disparity, and temporal information between consecutive frames. A 3-Stream 3D CNN, shortly 3S-3DCNN , by performing 3D CNNs, extracts features from spatial, motion, and depth channels to estimate the stereo video’s quality. It captures the degradations in the quality of the stereoscopic video in multiple dimensions. Firstly, the scene flow, which is the joint prediction of the optical flow and stereo disparity, is calculated. Then, the spatial information, optical flow, and disparity map of a given video are used as input to the 3S-3DCNN model. The extracted features are concatenated and utilized as inputs to the fully connected layers for doing the regression. We split the input videos into cube patches for data augmentation and remove the cubes that confuse our model from the training and testing sets. Two standard stereoscopic video quality assessment benchmarks of LFOVIAS3DPh2 and NAMA3DS1-COSPAD1 were used to evaluate our method. Experimental results show that our 3S-3DCNN method’s objective score significantly correlates with the subjective SVQ scores in multiple video datasets. The RMSE for NAMA3DS1-COSPAD1 dataset is 0.2757, which outperforms other methods by a large margin. The SROCC value for the blur distortion of the LFOVIAS3DPh2 dataset is more than 98%, indicating that the 3S-3DCNN is consistent with human visual perception.

Full-Reference Stereoscopic Video Quality Assessment Using a Motion Sensitive HVS Model

Stereoscopic video quality assessment has become a major research topic in recent years. Existing stereoscopic video quality metrics are predominantly based on stereoscopic image quality metrics extended to the time domain via for example temporal pooling. These approaches do not explicitly consider the motion sensitivity of the Human Visual System (HVS). To address this limitation, this paper introduces a novel HVS model inspired by physiological findings characterising the motion sensitive response of complex cells in the primary visual cortex (V1 area). The proposed HVS model generalises previous HVS models, which characterised the behaviour of simple and complex cells but ignored motion sensitivity, by estimating optical flow to measure scene velocity at different scales and orientations. The local motion characteristics (direction and amplitude) are used to modulate the output of complex cells. The model is applied to develop a new type of full-reference stereoscopic video quality metrics which uniquely combine non-motion sensitive and motion sensitive energy terms to mimic the response of the HVS. A tailored two-stage multi-variate stepwise regression algorithm is introduced to determine the optimal contribution of each energy term. The two proposed stereoscopic video quality metrics are evaluated on three stereoscopic video datasets. Results indicate that they achieve average correlations with subjective scores of 0.9257 (PLCC), 0.9338 and 0.9120 (SRCC), 0.8622 and 0.8306 (KRCC), and outperform previous stereoscopic video quality metrics including other recent HVS-based metrics.

No-Reference Quality Evaluation of Stereoscopic Video Based on Spatio-Temporal Texture

Due to the wide application of stereoscopic display technology, stereoscopic video quality assessment (SVQA) is facing great challenges, but worthwhile. Stereoscopic videos contain a great deal of information, which involves not only the spatial domain but also the spatio-temporal domain. Motion in stereoscopic video plays a critical role in quality perception, while the existing SVQA methods rarely refer to motion factors, and the performance of these methods is restrained. In this article, a novel SVQA based on motion perception is introduced and its performance is superior to that of existing excellent methods. Particularly, to appropriately reduce the amount of data processing, we extract the key-frame sequences according to the influence of movement intensity on binocular visual quality perception. The binocular summation and difference operations are implemented on extracted sequences, and then spatial texture and spatio-temporal texture statistic measurement are extracted simultaneously with local binary patterns from three orthogonal planes (LBP-TOP). Experiments are implemented on two publicly available databases and the results demonstrate the effectiveness and robustness of our algorithm for various categories of distortion stereoscopic video pairs.

Stereoscopic Video Quality Assessment Using Oriented Local Gravitational Force Statistics

We develop a new no-reference (NR) stereoscopic video quality assessment model that adopts oriented local gravitational force (OLGF) statistics in the space-time domain. The OLGF is a novel extension of an existing local gravitational force descriptor and includes two new components: relative local gravitational force magnitude and relative local gravitational force orientation. The resulting algorithm, called Stereoscopic Video Integrity Predictor using OLGF Statistics (SVIPOS), first uses our previous work to generate a cyclopean image and a product image in spatial domain from a stereoscopic video sequence pair, and calculates a frame difference image in temporal domain from a left video sequence. Then the OLGF model is deployed to compute various local gravitational force responses on the cyclopean image, the product image and the frame difference image. Finally, a support vector regression is used to map statistical features extracted from the computed gravitational force responses to stereoscopic video quality predications. SVIPOS is thoroughly tested on three publicly stereoscopic video quality databases. The experimental results show that SVIPOS outperforms state-of-the-art stereoscopic video quality methods.

No-Reference Quality Assessment of Stereoscopic Videos With Inter-Frame Cross on a Content-Rich Database

With the wide application of stereoscopic video technology, the quality of stereoscopic video has attracted people’s attention. Objective stereoscopic video quality assessment (SVQA) is highly challenging, but essential, particularly the no-reference (NR) SVQA method, where reference information is not needed and a large number of samples are required for training and testing sets. However, as far as we know, there are only a few samples in the established stereo video database, which is unsuitable for NR quality assessment and seriously hampers the development of NR-SVQA method. For these difficulties that we encountered, we carry out a comprehensive subjective evaluation of stereoscopic video quality in our newly established TJU-SVQA databases that contain various contents, mixed resolution coding and symmetrically/asymmetrically distorted stereoscopic videos. Furthermore, we propose a new inter-frame cross map to predict the objective quality scores. We compare and analyze the performance of several state-of-the-art 2D and 3D quality evaluation methods on our new databases. The experimental results on our established databases and a public database demonstrate that the proposed method can robustly predict the quality of stereoscopic videos.

Quality assessment of stereoscopic video in free viewpoint video system

It is challenging to develop an effective quality assessment method for the stereoscopic video (SV) in the free viewpoint video (FVV) system because of asymmetric distortion and the interactively changing combinations. This paper proposes a quality assessment method that takes the characteristics of the SV in FVV systems into consideration. Specifically, considering the distortion introduced by the rendering process, the proposed method extracts a critical distortion area that can be perceivable by the human eye. The distortion degree of each critical distortion area is quantified and the overall results are pooled to obtain the initial quality score. The video pairs are classified into different combination types and the initial quality score is refined to obtain the final quality score. The experimental results show that the proposed method outperforms conventional objective methods.

No Reference Quality Assessment of Stereo Video Based on Saliency and Sparsity

With the popularity of video technology, stereoscopic video quality assessment (SVQA) has become increasingly important. Existing SVQA methods cannot achieve good performance because the videos’ information is not fully utilized. In this paper, we consider various information in the videos together, construct a simple model to combine and analyze the diverse features, which is based on saliency and sparsity. First, we utilize the 3-D saliency map of sum map, which remains the basic information of stereoscopic video, as a valid tool to evaluate the videos’ quality. Second, we use the sparse representation to decompose the sum map of 3-D saliency into coefficients, then calculate the features based on sparse coefficients to obtain the effective expression of videos’ message. Next, in order to reduce the relevance between the features, we put them into stacked auto-encoder, mapping vectors to higher dimensional space, and adding the sparse restraint, then input them into support vector machine subsequently, and finally, get the quality assessment scores. Within that process, we take the advantage of saliency and sparsity to extract and simplify features. Through the later experiment, we can see the proposed method is fitting well with the subjective scores.

Stereoscopic video quality assessment based on 3D convolutional neural networks

Abstract The research of stereoscopic video quality assessment (SVQA) plays an important role for promoting the development of stereoscopic video system. Existing SVQA metrics rely on hand-crafted features, which is inaccurate and time-consuming because of the diversity and complexity of stereoscopic video distortion. This paper introduces a 3D convolutional neural networks (CNN) based SVQA framework that can model not only local spatio-temporal information but also global temporal information with cubic difference video patches as input. First, instead of using hand-crafted features, we design a 3D CNN architecture to automatically and effectively capture local spatio-temporal features. Then we employ a quality score fusion strategy considering global temporal clues to obtain final video-level predicted score. Extensive experiments conducted on two public stereoscopic video quality datasets show that the proposed method correlates highly with human perception and outperforms state-of-the-art methods by a large margin. We also show that our 3D CNN features have more desirable property for SVQA than hand-crafted features in previous methods, and our 3D CNN features together with support vector regression (SVR) can further boost the performance. In addition, with no complex preprocessing and GPU acceleration, our proposed method is demonstrated computationally efficient and easy to use.

Blind Stereoscopic Video Quality Assessment: From Depth Perception to Overall Experience.

Stereoscopic video quality assessment (SVQA) is a challenging problem. It has not been well investigated on how to measure depth perception quality independently under different distortion categories and degrees, especially exploit the depth perception to assist the overall quality assessment of 3D videos. In this paper, we propose a new depth perception quality metric (DPQM) and verify that it outperforms existing metrics on our published 3D video extension of High Efficiency Video Coding (3D-HEVC) video database. Furthermore, we validate its effectiveness by applying the crucial part of the DPQM to a novel blind stereoscopic video quality evaluator (BSVQE) for overall 3D video quality assessment. In the DPQM, we introduce the feature of auto-regressive prediction-based disparity entropy (ARDE) measurement and the feature of energy weighted video content measurement, which are inspired by the free-energy principle and the binocular vision mechanism. In the BSVQE, the binocular summation and difference operations are integrated together with the fusion natural scene statistic measurement and the ARDE measurement to reveal the key influence from texture and disparity. Experimental results on three stereoscopic video databases demonstrate that our method outperforms state-of-the-art SVQA algorithms for both symmetrically and asymmetrically distorted stereoscopic video pairs of various distortion types.

A no-reference optical flow-based quality evaluator for stereoscopic videos in curvelet domain

Most of the existing 3D video quality assessment (3D-VQA/SVQA) methods only consider spatial information by directly using an image quality evaluation method. In addition, a few take the motion information of adjacent frames into consideration. In practice, one may assume that a single data-view is unlikely to be sufficient for effectively learning the video quality. Therefore, integration of multi-view information is both valuable and necessary. In this paper, we propose an effective multi-view feature learning metric for blind stereoscopic video quality assessment (BSVQA), which jointly focuses on spatial information, temporal information and inter-frame spatio-temporal information. In our study, a set of local binary patterns (LBP) statistical features extracted from a computed frame curvelet representation are used as spatial and spatio-temporal description, and the local flow statistical features based on the estimation of optical flow are used to describe the temporal distortion. Subsequently, a support vector regression (SVR) is utilized to map the feature vectors of each single view to subjective quality scores. Finally, the scores of multiple views are pooled into the final score according to their contribution rate. Experimental results demonstrate that the proposed metric significantly outperforms the existing metrics and can achieve higher consistency with subjective quality assessment.

Stereoscopic video quality assessment based on visual attention and just-noticeable difference models

With the consideration that incorporating visual saliency information appropriately can benefit image quality assessment metrics, this paper proposes an objective stereoscopic video quality assessment (SVQA) metric by incorporating stereoscopic visual attention (SVA) to SVQA metric. Specifically, based upon the multiple visual masking characteristics of HVS, a stereoscopic just-noticeable difference model is proposed to compute the perceptual visibility for stereoscopic video. Next, a novel SVA model is proposed to extract stereoscopic visual saliency information. Then, the quality maps are calculated by the similarity of the original and distorted stereoscopic videos’ perceptual visibility. Finally, the quality score is obtained by incorporating visual saliency information to the pooling of quality maps. To evaluate the proposed SVQA metric, a subjective experiment is conducted. The experimental result shows that the proposed SVQA metric achieves better performance in comparison with the existing SVQA metrics.

Towards the Standardization of Stereoscopic Video Quality Assessment: An Application for Objective Algorithms

Towards the Standardization of Stereoscopic Video Quality Assessment: An Application for Objective Algorithms

Toward an Impairment Metric for Stereoscopic Video: A Full-Reference Video Quality Metric to Assess Compressed Stereoscopic Video

The quality assessment of impaired stereoscopic video is a key element in designing and deploying advanced immersive media distribution platforms. A widely accepted quality metric to measure impairments of stereoscopic video is, however, still to be developed. As a step toward finding a solution to this problem, this paper proposes a full reference stereoscopic video quality metric to measure the perceptual quality of compressed stereoscopic video. A comprehensive set of subjective experiments is performed with 14 different stereoscopic video sequences, which are encoded using both the H.264 and high efficiency video coding compliant video codecs, to develop a subjective test results database of 116 test stimuli. The subjective results are analyzed using statistical techniques to uncover different patterns of subjective scoring for symmetrically and asymmetrically encoded stereoscopic video. The subjective result database is subsequently used for training and validating a simple but effective stereoscopic video quality metric considering heuristics of binocular vision. The proposed metric performs significantly better than state-of-the-art stereoscopic image and video quality metrics in predicting the subjective scores. The proposed metric and the subjective result database will be made publicly available, and it is expected that the proposed metric and the subjective assessments will have important uses in advanced 3D media delivery systems.