Full-reference Video Quality Assessment Research Articles

Full-Reference Video Quality Assessment for Military Surveillance Video Compressed with Ultra Narrow Bandwidth

Full-Reference Video Quality Assessment for Military Surveillance Video Compressed with Ultra Narrow Bandwidth

Screen content video quality assessment based on spatiotemporal sparse feature

In recent years, the explosive growth of application scenarios have occurred in screen content videos (SCVs), in which the SCVs are unavoidably suffered from the quality degradation and the video quality assessment (VQA) of the SCVs becomes very essential. In view of this, a full-reference VQA algorithm, called the spatiotemporal sparse feature-based model (SSFM) is proposed in this paper, aiming to give a precise and efficient quality evaluation about the distorted SCVs. Note that the SCVs are full of edge information which the human eyes are highly sensitive to, and the sparse coding can provide accurate quantitative predictions which are consistent with the perception resulted from the cerebral cortex in various receptive field models of the visual cortex. With these considerations, three dimensional Difference of Gaussian (3D-DOG) filter and 3D Sparse Dictionary are developed to extract multi-scale spatiotemporal features and obtain spatiotemporal sparse features respectively, from the reference and distorted SCVs. Based on these features, the spatiotemporal sparse feature similarity can be measured and followed by generating the quality scores of the distorted SCVs under evaluation. Compared to other classic and state-of-the-art image/video quality evaluation metrics, the experimental results of the proposed SSFM on the screen content video database (SCVD) are more consistent with the perceived evaluation of SCVs according to the human visual system (HVS).

Full Reference Video Quality Assessment Based on Statistical Based Transform Coefficients

Full Reference Video Quality Assessment Based on Statistical Based Transform Coefficients

Revised One, a Full Reference Video Quality Assessment Based on Statistical Based Transform Coefficient

Revised One, a Full Reference Video Quality Assessment Based on Statistical Based Transform Coefficient

A Gabor feature-based full reference video quality assessment model based on spatiotemporal slice of videos

A Gabor feature-based full reference video quality assessment model based on spatiotemporal slice of videos

Enhanced optical flow-based full reference video quality assessment algorithm

Enhanced optical flow-based full reference video quality assessment algorithm

Screen Content Video Quality Assessment Model Using Hybrid Spatiotemporal Features.

In this paper, a full-reference video quality assessment (VQA) model is designed for the perceptual quality assessment of the screen content videos (SCVs), called the hybrid spatiotemporal feature-based model (HSFM). The SCVs are of hybrid structure including screen and natural scenes, which are perceived by the human visual system (HVS) with different visual effects. With this consideration, the three dimensional Laplacian of Gaussian (3D-LOG) filter and three dimensional Natural Scene Statistics (3D-NSS) are exploited to extract the screen and natural spatiotemporal features, based on the reference and distorted SCV sequences separately. The similarities of these extracted features are then computed independently, followed by generating the distorted screen and natural quality scores for screen and natural scenes. After that, an adaptive screen and natural quality fusion scheme through the local video activity is developed to combine them for arriving at the final VQA score of the distorted SCV under evaluation. The experimental results on the Screen Content Video Database (SCVD) and Compressed Screen Content Video Quality (CSCVQ) databases have shown that the proposed HSFM is more in line with the perceptual quality assessment of the SCVs perceived by the HVS, compared with a variety of classic and latest IQA/VQA models.

Video Quality Assessment With Serial Dependence Modeling

Video quality assessment (VQA) is much more challenging than image quality assessment, due to the difficulty of modeling temporal influence among frames. Most of the existing VQA methods usually isolate each moment within the video (i.e., it neglects the sequential nature), leading to a large gap from the subjective perception. Recent research on neuroscience suggests a serially dependent perception (SDP) mechanism in the human visual system (HVS). Namely, the HVS tends to incorporate the recent past visual experience to predict the present perception. Inspired by the SDP, we suggest that the HVS prefers stable and continuous degradations in videos due to their predictability, and exhibits less tolerance to interrupted and unpredictable disturbances. Thus, we introduce a novel serial dependence modeling (SDM) framework for full-reference VQA in this paper. Firstly, the instantaneous degradation is measured on both the static appearance and motion information for each glimpse of scenes. Since motion plays an important role in videos, two types of structures are extracted for motion representation, namely, an explicit content-based 3D structure and an implicit feature-based 2D structure. Next, an assessment-directed long-short term memory (A-LSTM) is proposed to capture the serial dependence among instantaneous degradations. With the consideration of the perceptual effect from the previous moment on the current one, especially the effect from the perceptually worst moment, the serially dependent degradation is characterized. Finally, by mimicking the subjective rating for video-viewing, an attention-based quality decision procedure is presented to acquire the final video quality. Experimental results on publicly available VQA databases demonstrate that the proposed method maintains good consistency with the subjective perception.

Blind video quality assessment based on multilevel video perception

Blind video quality assessment (VQA) metrics predict the quality of videos without the presence of reference videos. This paper proposes a new blind VQA model based on multilevel video perception, abbreviated as MVP. The model fuses three levels of video features occurring in natural video scenes to predict video quality: natural video statistics (NVS) features, global motion features and motion temporal correlation features. They represent video scene characteristics, video motion types, and video temporal correlation variations. In the process of motion feature extraction, motion compensation filtering video enhancement is adopted to highlight the motion characteristics of videos so as to improve the perceptual correlations of the video features. The experimental results on the LIVE and CSIQ video databases show that the predicted video scores of the new model are highly correlated with human perception and have low root mean square errors. MVP obviously outperforms state-of-art blind VQA metrics, and particularly demonstrates competitive performance even compared against top-performing full reference VQA metrics.

No-reference video quality assessment based on modeling temporal-memory effects

This study presents a hybrid network for no-reference (NR) video quality assessment (VQA). Besides spatial cues, the network concerns temporal motion effect and temporal hysteresis effect on the visual quality estimation, and two modules are embedded. One module is dedicated to incorporate short-term spatio-temporal features based on spatial quality maps and temporal quality maps, and the follow-up module explores graph convolutional network to quantify the relationship between image frames in a sequence. The proposed network and several popular models are evaluated on three video quality databases (CSIQ, LIVE, and KoNViD-1K). Experimental results indicate that the network outperforms other involved NR models, and its competitive performance is close to that of state-of-the-art full-reference VQA models. Conclusively, short-term spatio-temporal feature fusion benefits the modeling of interaction between spatial and temporal cues in VQA tasks, long-term sequence fusion further improves the performance, and a strong correlation with human subjective judgment is achieved.

Joint model of gradient magnitude and Gabor features via Spatio-Temporal slice

To form a high-performance video quality predictor, we developed a framework for full-reference (FR) video quality assessment that integrates Spatio-temporal slice analysis (STS) to create a high-performance predictor of video quality. However, both gradient and Gabor are spatial–temporal structural capturers used for the simultaneous extraction of both spatial and temporal features. In this paper, we proposed a novel VQA algorithm via a joint model of gradient magnitude and Gabor features (JMG) between the STS images of the reference videos and their distorted counterparts to assess the degradation of video quality effectively. Firstly, gradient magnitude and the Gabor filter were constructed to extract the spatiotemporal features of the video sequence. However, the two-feature model combined to predict the perceptual quality of frames. This new proposed VQA model is known as the horizontal and time STS (HT-JMG) model. To further investigate the influence of spatial dissimilarity, we combined the frame-by-frame spatial T-JMG(S) factor with the HT-JMG and propose another VQA model, called the time, horizontal, and vertical STS (THV-JMG) model. Finally, the results of the experiment showed that the proposed method has a strong correlation with subjective perception and is competitive with state-of-the-art full reference VQA models.

Video quality assessment with dense features and ranking pooling

Benefiting with the rapid development of communication networks, effective video quality assessment (VQA) models which provide guidance for video transmission and compression technologies are highly demanded. This paper proposes a general-purpose full-reference VQA method combining DenseNet with spatial pyramid pooling and RankNet to not only extract high-level distortion representation and global spatial information of samples but also characterize the temporal correlation among frames. Firstly, the pretrained DenseNet is modified and finetuned to extract high-level features of distorBenefiting with the rapid development of communication networks, effective video quality assessment (VQA) models which provide guidance for video transmission and compression technologies are highly demanded. This paper proposes a general-purpose full-reference VQA method combining DenseNet with spatial pyramid pooling and RankNet to not only extract high-level distortion representation and global spatial information of samples but also characterize the temporal correlation among frames. Firstly, the pretrained DenseNet is modified and finetuned to extract high-level features of distorted videos. Then, spatial pyramid pooling is equipped in the DenseNet module to process flexible inputs with arbitrary spatial resolution. Thus, this kind of input which has the same spatial resolution as the original distorted video is processed by the well-trained DenseNet to generate frame-level quality, which considers the global spatial information of videos directly. Finally, learning to rank is introduced to explore the high-level temporal correlation of distorted videos by taking the RankNet as the temporal pooling function. The experimental results on two public VQA databases show that the proposed algorithm performs consistently with human visual perception.

Predicting the Quality of Compressed Videos With Pre-Existing Distortions.

Because of the increasing ease of video capture, many millions of consumers create and upload large volumes of User-Generated-Content (UGC) videos to social and streaming media sites over the Internet. UGC videos are commonly captured by naive users having limited skills and imperfect techniques, and tend to be afflicted by mixtures of highly diverse in-capture distortions. These UGC videos are then often uploaded for sharing onto cloud servers, where they are further compressed for storage and transmission. Our paper tackles the highly practical problem of predicting the quality of compressed videos (perhaps during the process of compression, to help guide it), with only (possibly severely) distorted UGC videos as references. To address this problem, we have developed a novel Video Quality Assessment (VQA) framework that we call 1stepVQA (to distinguish it from two-step methods that we discuss). 1stepVQA overcomes limitations of Full-Reference, Reduced-Reference and No-Reference VQA models by exploiting the statistical regularities of both natural videos and distorted videos. We also describe a new dedicated video database, which was created by applying a realistic VMAF-Guided perceptual rate distortion optimization (RDO) criterion to create realistically compressed versions of UGC source videos, which typically have pre-existing distortions. We show that 1stepVQA is able to more accurately predict the quality of compressed videos, given imperfect reference videos, and outperforms other VQA models in this scenario.

An optical flow direction-based full reference video quality assessment algorithm

Modelling of functions of visual cortex and optical flow field has made it possible to develop many sophisticated human visual system-based visual quality assessment algorithms. There exist very few video quality assessment algorithms which consider spatio-temporal distortions effectively. In this article, based on the direction of optical flow field, we propose a full reference video quality algorithm. We claim that some temporal distortions, which are present in other directions than the majority of optical flow field directions, can not be easily perceivable by the human observer; hence it results in reduced video quality scores. Our proposed method achieved high video quality score and higher correlation with formally existed algorithm and near to the objective scores. Our experimental results demonstrate that the proposed algorithm improves the state of art.

An optical flow direction-based full reference video quality assessment algorithm

An optical flow direction-based full reference video quality assessment algorithm

Screen Content Video Quality Assessment: Subjective and Objective Study.

In this paper, we make the first attempt to study the subjective and objective quality assessment for the screen content videos (SCVs). For that, we construct the first large-scale video quality assessment (VQA) database specifically for the SCVs, called the screen content video database (SCVD). This SCVD provides 16 reference SCVs, 800 distorted SCVs, and their corresponding subjective scores, and it is made publicly available for research usage. The distorted SCVs are generated from each reference SCV with 10 distortion types and 5 degradation levels for each distortion type. Each distorted SCV is rated by at least 32 subjects in the subjective test. Furthermore, we propose the first full-reference VQA model for the SCVs, called the spatiotemporal Gabor feature tensor-based model (SGFTM), to objectively evaluate the perceptual quality of the distorted SCVs. This is motivated by the observation that 3D-Gabor filter can well stimulate the visual functions of the human visual system (HVS) on perceiving videos, being more sensitive to the edge and motion information that are often-encountered in the SCVs. Specifically, the proposed SGFTM exploits 3D-Gabor filter to individually extract the spatiotemporal Gabor feature tensors from the reference and distorted SCVs, followed by measuring their similarities and later combining them together through the developed spatiotemporal feature tensor pooling strategy to obtain the final SGFTM score. Experimental results on SCVD have shown that the proposed SGFTM yields a high consistency on the subjective perception of SCV quality and consistently outperforms multiple classical and state-of-the-art image/video quality assessment models.

Video quality assessment using space–time slice mappings

We develop a full-reference (FR) video quality assessment framework that integrates analysis of space–time slices (STSs) with frame-based image quality measurement (IQA) to form a high-performance video quality predictor. The approach first arranges the reference and test video sequences into a space–time slice representation. To more comprehensively characterize space–time distortions, a collection of distortion-aware maps are computed on each reference–test video pair. These reference-distorted maps are then processed using a standard image quality model, such as peak signal-to-noise ratio (PSNR) or Structural Similarity (SSIM). A simple learned pooling strategy is used to combine the multiple IQA outputs to generate a final video quality score. This leads to an algorithm called Space–TimeSlice PSNR (STS-PSNR), which we thoroughly tested on three publicly available video quality assessment databases and found it to deliver significantly elevated performance relative to state-of-the-art video quality models. Source code for STS-PSNR is freely available at: http://live.ece.utexas.edu/research/Quality/STS-PSNR_release.zip.

Bitrate-Based No-Reference Video Quality Assessment Combining the Visual Perception of Video Contents

In video communication, the quality of video is mainly determined by bitrate in general. Moreover, the effect of video contents and their visual perception on video quality assessment (VQA) is often overlooked. However, in fact, for different videos, although the bitrates are the same, their VQA scores are still significantly different. Hence, it is assumed that the bitrate, video contents, and human visual characteristics mainly affect the VQA. Based on the above three aspects, in this paper, we designed a bitrate-based no-reference (NR) VQA metric combining the visual perception of video contents, namely, BRVPVC. In this metric, first an initial VQA model was proposed by only considering the bitrate alone. Then, the visual perception model for video contents was designed based on the texture complexity and local contrast of image, temporal information of video, and their visual perception features. Finally, two models were synthesized by adding certain weight coefficients into an overall VQA metric, namely, BRVPVC. Furthermore, ten reference videos and 150 distorted videos in the LIVE video database were used to test the metric. Moreover, based on the results of evaluating the videos in LIVE, VQEG, IRCCyN, EPFL-PoliMI, IVP, CSIQ, and Lisbon databases, the performance of BRVPVC is respectively compared with that of six full-reference (FR) metrics and ten NR VQA metrics. The results show that our VQA metric has a higher accuracy than six common FR VQA metrics and eight NR VQA metrics, and it is close to other two NR VQA metrics in accuracy. The corresponding values of Pearson linear correlation coefficient and Spearman rank order correlation coefficient reached 0.8547 and 0.8260, respectively. In addition, the computational complexity of proposed VQA metric is lower than video signal-to-noise ratio, video quality model, motion-based video integrity evaluation, spatiotemporal most apparent distortion, V-BLINDS, and V-CORNIA metrics. Moreover, the proposed metric has a better generalization property than these metrics.

Blind Video Quality Assessment With Weakly Supervised Learning and Resampling Strategy

Due to the 3D spatiotemporal regularities of natural videos and small-scale video quality databases, effective objective video quality assessment (VQA) metrics are difficult to obtain but highly desirable. In this paper, we propose a general-purpose no-reference VQA framework that is based on weakly supervised learning with a convolutional neural network (CNN) and a resampling strategy. First, an eight-layer CNN is trained by weakly supervised learning to construct the relationship between the deformations of the 3D discrete cosine transform of video blocks and the corresponding weak labels judged by a full-reference (FR) VQA metric. Thus, the CNN obtains the quality assessment capacity converted from the FR-VQA metric, and the effective features of the distorted videos can be extracted through the trained network. Then, we map the frequency histogram calculated from the quality score vectors predicted by the trained network onto the perceptual quality. Especially, to improve the performance of the mapping function, we transfer the frequency histogram of the distorted images and videos to resample the training set. The experiments are carried out on several widely used VQA databases. The experimental results demonstrate that the proposed method is on a par with some state-of-the-art VQA metrics and has promising robustness.

Full reference video quality evaluation using foveated vision and multiple fixation points

In video applications it is necessary to continuously measure the video quality perceived by the end-user. Thus it is desirable to know which parts of video frame, i.e. which contents, attract viewers’ attention. If this information is known, then it is possible to estimate perceived video quality in a meaningful way. However, automatic detection of viewers’ fixation points is time-consuming process and often is omitted in objective video quality assessment (VQA) metrics. Based on our previous work, in which we proposed Foveation-based content Adaptive Root Mean Squared Error (FARMSE) VQA metric, in this work we propose two new full-reference (FR) VQA metrics called Multi-Point FARMSE (MP-FARMSE), and Simple-FARMSE (S-FARMSE). Both new-proposed metrics are based on foveated-vision features of human visual system and spatio-temporal features of video signal. In MP-FARMSE, by using an engineering approach, we implemented the fact that viewer’s attention can be directed out of the center of the frame, thus covering use-cases when objects of interest are not located in the center of the frame. The main idea when creating the S-FARMSE metric was to reduce the computational complexity of the final algorithm and to make S-FARMSE metric capable of processing high-resolution video signals in real-time. Performances of the new-proposed metrics are compared to performances of seven existing VQA metrics on two different video quality databases. The results show that performances achieved by MP-FARMSE and S-FARMSE are quite close to those of state-of-the-art VQA metrics, whereas at the same time their computational complexity level is significantly lower.