Video Quality Assessment Methods Research Articles

Subjective Video QoE With a Hybrid Cognitive-Motor Task

This paper reports on the subjective video quality analysis with and without a parallel task. It aims to understand how the quality perceived by the human observer brain is affected when the subject is not entirely focused on the judgment task. For this purpose, two subjective video quality tests were performed according to the ITU-T P.910 recommendation - Subjective video quality assessment methods for multimedia applications. They consisted of two parts involving a total of 24 subjects. Unlike the first test, the second test included an additional (parallel) psychomotor task where subjects had to sort different colors of pasta into three bowls. Various analyses were provided, including preliminary analysis, group analysis, and RMSE analysis. Surprisingly, the results show that not all the spectrum of quality is affected equally by parallel task introduction.

PEA265: Perceptual Assessment of Video Compression Artifacts

The most widely used video encoders share a common hybrid coding framework that includes block-based motion estimation/compensation and block-based transform coding. Despite their high coding efficiency, the encoded videos often exhibit visually annoying artifacts, denoted as Perceivable Encoding Artifacts (PEAs), which significantly degrade the visual Qualityof- Experience (QoE) of end users. To monitor and improve visual QoE, it is crucial to develop subjective and objective measures that can identify and quantify various types of PEAs. In this work, we make the first attempt to build a large-scale subjectlabelled database composed of H.265/HEVC compressed videos containing various PEAs. The database, namely the PEA265 database, includes 4 types of spatial PEAs (i.e. blurring, blocking, ringing and color bleeding) and 2 types of temporal PEAs (i.e. flickering and floating). Each containing at least 60,000 image or video patches with positive and negative labels. To objectively identify these PEAs, we train Convolutional Neural Networks (CNNs) using the PEA265 database. It appears that state-of-theart ResNeXt is capable of identifying each type of PEAs with high accuracy. Furthermore, we define PEA pattern and PEA intensity measures to quantify PEA levels of compressed video sequence. We believe that the PEA265 database and our findings will benefit the future development of video quality assessment methods and perceptually motivated video encoders.

No-Reference Nonuniform Distorted Video Quality Assessment Based on Deep Multiple Instance Learning

Each part of a nonuniform distorted video (NUDV) has a unique distortion degree. When NUDV blocks are used as inputs, traditional machine-learning-based video quality assessment (VQA) methods frequently do not work effectively. Because these methods directly assign the label of the entire video to blocks, causing the unreliability of labels. We creatively propose video bag, a collection of video blocks, to deal with this unreliability. We develop a novel multiple instance learning (MIL) based model, VQA-MIL, which dynamically adjusts the weights by a block-wise attention module and enriches the features of video bags by a MI Pooling layer. Furthermore, we apply the mixup data-augmentation strategy to address the lack of human labels in common video datasets. We test our method on LIVE and CSIQ, and on a relatively large-scale dataset, named NUDV-KT, that we have collected. Results show that our method outperforms popular state-of-the-art no-reference VQA methods on NUDVs.

Error sensitivity model based on spatial and temporal features

Packet loss and error propagation induced by it are significant causes of visual impairments in video applications. Most of the existing video quality assessment models are developed at frame or sequence level, which can not accurately describe the impact of packet loss on the local regions in one frame. In this paper, we propose an error sensitivity model to evaluate the impact of a single packet loss. We also make full use of the spatio-temporal correlation of the video and analyze a set of features that directly impact the perceptual quality of videos, based on the specific situation of video packet loss. With the aid of the support vector regression (SVR), these features are used to predict the error sensitivity of the local region. The proposed model is tested on six video sequences. Experimental results show that the proposed model predicts sensitivity of videos to different packet loss cases with certain reasonable accuracy, and provides good generalization ability, which turns out outperform the state-of-art image and video quality assessment methods.

Look ahead to improve QoE in DASH streaming

When a video is encoded with constant quality, the resulting bitstream will have variable bitrate due to the inherent nature of the video encoding process. This paper proposes a video Adaptive Bitrate Streaming (ABR) algorithm, called Look Ahead, which takes into account this bitrate variability in order to calculate, in real time, the appropriate quality level that minimizes the number of interruptions during the playback. The algorithm is based on the Dynamic Adaptive Streaming over HTTP (DASH) standard for on-demand video services. In fact, it has been implemented and integrated into ExoPlayer v2, the latest version of the library developed by Google to play DASH contents. The proposed algorithm is compared to the Müller and Segment Aware Rate Adaptation (SARA) algorithms as well as to the default ABR algorithm integrated into ExoPlayer. The comparison is carried out by using the most relevant parameters that affect the Quality of Experience (QoE) in video playback services, that is, number and duration of stalls, average quality of the video playback and number of representation switches. These parameters can be combined to define a QoE model. In this sense, this paper also proposes two new QoE models for the evaluation of ABR algorithms. One of them considers the bitrate of every segment of each representation, and the second is based on VMAF (Video Multimethod Assessment Fusion), a Video Quality Assessment (VQA) method developed by Netflix. The evaluations presented in the paper reflect: first, that Look Ahead outperforms the Müller, SARA and the ExoPlayer ABR algorithms in terms of number and duration of video playback stalls, with hardly decreasing the average video quality; and second, that the two QoE models proposed are more accurate than other similar models existing in the literature.

IJNM Special Issue—International Journal of Network Management: QoE‐centric analysis and management of communication networks

IJNM Special Issue—International Journal of Network Management: QoE‐centric analysis and management of communication networks

No‐reference video quality assessment method based on spatio‐temporal features using the ELM algorithm

This work presents an application of the extreme learning machine (ELM) algorithm based on a single-hidden layer feedforward neural network for no-reference video quality assessment. The present research introduces an augmented version of ELM through simple stop criteria, which proved the effectiveness of the video quality assessment method. The authors present empirical studies using LIVE video data base show that the proposed method delivers accuracy (Pearson's correlation coefficient) and monotonicity (Spearman's correlation coefficient) with subjective scores against no-reference, Joint Photographic Experts Group No-Reference, metric and full-reference metrics, for instance, peak signal-to-noise ratio, structural similarity (SSIM) and multi-scale-SSIM indexes, and the proposed method is suitable for quality monitoring of video transmission and reception system.

Panoramic Video Quality Assessment Based on Non-Local Spherical CNN

Panoramic video and stereoscopic panoramic video are essential carriers of virtual reality content, so it is very crucial to establish their quality assessment models for the standardization of virtual reality industry. However, it is very challenging to evaluate the quality of the panoramic video at present. One reason is that the spatial information of the panoramic video is warped due to the projection process, and the conventional video quality assessment (VQA) method is difficult to deal with this problem. Another reason is that the traditional VQA method is problematic to capture the complex global time information in the panoramic video. In response to the above questions, this paper presents an end-to-end neural network model to evaluate the quality of panoramic video and stereoscopic panoramic video. Compared to other panoramic video quality assessment methods, our proposed method combines spherical convolutional neural networks (CNN) and non-local neural networks, which can effectively extract complex spatiotemporal information of the panoramic video. We evaluate the method in two databases, VRQ-TJU and VR-VQA48. Experiments show the effectiveness of different modules in our method, and our method outperforms state-of-the-art other related methods.

No Reference Video Quality Assessment with authentic distortions using 3-D Deep Convolutional Neural Network

Video Quality Assessment (VQA) is an essential topic in several industries ranging from video streaming to camera manufacturing. In this paper, we present a novel method for No-Reference VQA. This framework is fast and does not require the extraction of hand-crafted features. We extracted convolutional features of 3-D C3D Convolutional Neural Network and feed one trained Support Vector Regressor to obtain a VQA score. We did certain transformations to different color spaces to generate better discriminant deep features. We extracted features from several layers, with and without overlap, finding the best configuration to improve the VQA score. We tested the proposed approach in LIVE-Qualcomm dataset. We extensively evaluated the perceptual quality prediction model, obtaining one final Pearson correlation of 0:7749±0:0884 with Mean Opinion Scores, and showed that it can achieve good video quality prediction, outperforming other state-of-the-art VQA leading models.

Non-reference underwater video quality assessment method for small size samples

Non-reference underwater video quality assessment method for small size samples

A No-Reference Video Quality Assessment Model for Underwater Networks

Underwater imagery is increasingly drawing attention from the scientific community, since pictures and videos are invaluable tools in the study of the vast unknown oceanic environment that covers 90% of the planetary biosphere. However, underwater sensor networks must cope with the harsh channel that seawater constitutes. Medium range communication is only possible using acoustic modems that have limited transmission capabilities and peak bitrates of only a few dozens of kilobits per second. These reduced bitrates force heavy compression on videos, yielding much higher levels of distortion than in other video services. Furthermore, underwater video users are ocean researchers, and therefore their quality perception is also different from the generic viewers that typically take part in subjective quality assessment experiments. Computational efficiency is also important since the underwater nodes must run on batteries and their recovery is very expensive. In this paper, we propose a pixel-based no-reference video quality assessment method that addresses the described challenges and achieves good correlations against subjective scores of users of underwater videos.

Stochastic QoE-aware optimization of multisource multimedia content delivery for mobile cloud

The increasing popularity of mobile video streaming in wireless networks has stimulated growing demands for efficient video streaming services. However, due to the time-varying throughput and user mobility, it is still difficult to provide high quality video services for mobile users. Our proposed optimization method considers key factors such as video quality, bitrate level, and quality variations to enhance quality of experience over wireless networks. The mobile network and device parameters are estimated in order to deliver the best quality video for the mobile user. We develop a rate adaptation algorithm using Lyapunov optimization for multi-source multimedia content delivery to minimize the video rate switches and provide higher video quality. The multi-source manager algorithm is developed to select the best stream based on the path quality for each path. The node joining and cluster head election mechanism update the node information. As the proposed approach selects the optimal path, it also achieves fairness and stability among clients. The quality of experience feature metrics like bitrate level, rebuffering events, and bitrate switch frequency are employed to assess video quality. We also employ objective video quality assessment methods like VQM, MS-SSIM, and SSIMplus for video quality measurement closer to human visual assessment. Numerical results show the effectiveness of the proposed method as compared to the existing state-of-the-art methods in providing quality of experience and bandwidth utilization.

How to Assess the Quality of Compressed Surveillance Videos Using Face Recognition

Video surveillance plays an important role in public security. To store the growing volume of surveillance videos, video compression is beneficial for reducing video volume; however, it is simultaneously harmful to the video quality. Video quality assessment (VQA) methods help to achieve a tradeoff between the data volume and perceptual quality of compressed surveillance videos. Generally speaking, surveillance video quality assessment (SVQA) is different from conventional VQA, because surveillance videos are usually used for specific tasks, e.g., pedestrian recognition, rather than for entertainment purposes. Therefore, in this paper, we propose two full-reference SVQA methods based on the concept of quality of recognition. We first design two new tasks, distorted face verification (DFV) and distorted face identification (DFI), based on which we further propose two SVQA methods, DFV-SVQA and DFI-SVQA, and corresponding quality metrics. The core components of the DFV-SVQA and DFI-SVQA methods are feature extractors (a DFV model and a DFI model), which we construct using convolutional-neural-network-based face recognition models. In addition, we construct a real-world surveillance video data set, based on which we analyze how various factors, including the video codec, compression level, face resolution, and light intensity, affect the quality of compressed surveillance videos. We find that, compared with conventional VQA methods, our methods are more effective in measuring the quality of surveillance videos while maintaining an acceptable time efficiency.

Network video quality assessment method using fuzzy decision tree

At present the service of network video is increasing rapidly. The video quality of experience is usually defined as user experience. However, the objective quality assessment for network video is unsure but is an urgent need. To improve the objective method, this study proposes an assessment method using a fuzzy decision tree. This work has two parts: (i) It considers different impair factors. Next, the decision tree is built by the set of measured factors from the different network environments. This tree can define the quality of network video, but cannot give the concrete objective scores. (ii) In order to give the accuracy objective scores, all the impair factors of leaf nodes are divided into two groups and inputted to fuzzy logic systems. The outputs are used as the next inputs and inputted into another fuzzy logic system to give the objective score. At last, the performance of the proposed method is tested on network videos with different network environments. At the same time, other methods are compared with it. The experimental results show that this method can get more accurate objective scores. This means the proposed method can improve the objective assessment model.

Sparse Representation based Video Quality Assessment for Synthesized 3D Videos.

The temporal flicker distortion is one of the most annoying noises in synthesized virtual view videos when they are rendered by compressed multi-view video plus depth in Three Dimensional (3D) video system. To assess the synthesized view video quality and further optimize the compression techniques in 3D video system, objective video quality assessment which can accurately measure the flicker distortion is highly needed. In this paper, we propose a full reference sparse representation based video quality assessment method towards synthesized 3D videos. Firstly, a synthesized video, treated as a 3D volume data with spatial (X-Y) and temporal (T) domains, is reformed and decomposed as a number of spatially neighboring temporal layers, i.e., X-T or Y-T planes. Gradient features in temporal layers of the synthesized video and strong edges of depth maps are used as key features in detecting the location of flicker distortions. Secondly, dictionary learning and sparse representation for the temporal layers are then derived and applied to effectively represent the temporal flicker distortion. Thirdly, a rank pooling method is used to pool all the temporal layer scores and obtain the score for the flicker distortion. Finally, the temporal flicker distortion measurement is combined with the conventional spatial distortion measurement to assess the quality of synthesized 3D videos. Experimental results on synthesized video quality database demonstrate our proposed method is significantly superior to other state-of-the-art methods, especially on the view synthesis distortions induced from depth videos.

Crowdsourced subjective 3D video quality assessment

This article proposes a new method for subjective 3D video quality assessment based on crowdsourced workers—Crowd3D. The limitations of traditional laboratory-based grade collection procedures are outlined, and their solution through the use of a crowd-based approach is described. Several conceptual and technical requirements of crowd-based 3D video quality assessment methods are identified and the solutions adopted described in detail. The system built takes the form of a web-based platform that supports 3D video monitors, and orchestrates the entire process of observer validation, content presentation and quality, depth, and comfort grade recording in a remote database. The crowdsourced subjective 3D quality assessment system uses as source contents a set of 3D video and grades database assembled earlier in a laboratory setting. To evaluate the validity of the crowd-based approach, the grades gathered using the crowdsourced system were analysed and compared to a set of grades obtained in laboratory settings using the same data set. Results show that it is possible to obtain Pearson’s and Spearman’s correlation up to 0.95 for quality Difference Mean Opinion Score and 0.96 for quality Mean Opinion Score, when comparing with laboratory grades. Apart from the present study, the 3D video quality assessment platform proposed can be used with advantage for further related research activities, reducing the time and cost compared to the traditional laboratory-based quality assessments.

Objective Video Quality Assessment Combining Transfer Learning With CNN.

Nowadays, video quality assessment (VQA) is essential to video compression technology applied to video transmission and storage. However, small-scale video quality databases with imbalanced samples and low-level feature representations for distorted videos impede the development of VQA methods. In this paper, we propose a full-reference (FR) VQA metric integrating transfer learning with a convolutional neural network (CNN). First, we imitate the feature-based transfer learning framework to transfer the distorted images as the related domain, which enriches the distorted samples. Second, to extract high-level spatiotemporal features of the distorted videos, a six-layer CNN with the acknowledged learning ability is pretrained and finetuned by the common features of the distorted image blocks (IBs) and video blocks (VBs), respectively. Notably, the labels of the distorted IBs and VBs are predicted by the classic FR metrics. Finally, based on saliency maps and the entropy function, we conduct a pooling stage to obtain the quality scores of the distorted videos by weighting the block-level scores predicted by the trained CNN. In particular, we introduce a preprocessing and a postprocessing to reduce the impact of inaccurate labels predicted by the FR-VQA metric. Due to feature learning in the proposed framework, two kinds of experimental schemes including train-test iterative procedures on one database and tests on one database with training other databases are carried out. The experimental results demonstrate that the proposed method has high expansibility and is on a par with some state-of-the-art VQA metrics on two widely used VQA databases with various compression distortions.

An objective and subjective quality assessment study of passive gaming video streaming

SummaryPassive gaming video‐streaming applications have recently gained much attention as evident with the rising popularity of many Over The Top (OTT) providers such as Twitch.tv and YouTube Gaming. For the continued success of such services, it is imperative that the user Quality of Experience (QoE) remains high, which is usually assessed using subjective and objective video quality assessment methods. Recent years have seen tremendous advancement in the field of objective video quality assessment (VQA) metrics, with the development of models that can predict the quality of the videos streamed over the Internet. A study on the performance of objective VQA on gaming videos, which are artificial and synthetic and have different streaming requirements than traditionally streamed videos, is still missing. Towards this end, we present in this paper an objective and subjective quality assessment study on gaming videos considering passive streaming applications. Subjective ratings are obtained for 90 stimuli generated by encoding six different video games in multiple resolution‐bitrate pairs. Objective quality performance evaluation considering eight widely used VQA metrics is performed using the subjective test results and on a data set of 24 reference videos and 576 compressed sequences obtained by encoding them in 24 resolution‐bitrate pairs. Our results indicate that Video Multimethod Assessment Fusion (VMAF) predicts subjective video quality ratings the best, while Naturalness Image Quality Evaluator (NIQE) turns out to be a promising alternative as a no‐reference metric in some scenarios.

A framework for computationally efficient video quality assessment

Objective video quality assessment (VQA) methods are essentially algorithms that estimate video quality. Recent quality assessment methods aim to provide quality predictions that are well correlated with subjective quality scores. However, most of these methods are computationally costly, which limits their use in real-time applications. A possible solution to this problem is to decrease the video resolution (spatial, temporal or both) in order to reduce the amount of processed data. Although reducing the video resolution is a simple way of decreasing the running time of a VQA method, this approach might impact the prediction accuracy of the VQA method. In this paper, we analyze this impact. More specifically, we analyze the effects of resolution reduction on the performance of the VQA methods. Based on this analysis, we propose a framework that decreases the overall processing time of VQA methods, without decreasing significantly the performance accuracy. We test the framework using six different VQA methods and four different video quality databases. Results show that the proposed framework reduces the average runtime performance of the tested VQA methods, without considerably altering their performance accuracy.

A Just Noticeable Difference-Based Video Quality Assessment Method with Low Computational Complexity

A Just Noticeable Difference (JND)-based video quality assessment (VQA) method is proposed. This method, termed as JVQ, applies JND concept to structural similarity (SSIM) index to measure the spatial quality. JVQ incorporates three features, i.e. luminance adaptation, contrast masking, and texture masking. In JVQ, the concept of JND is refined and more features are considered. For the spatial part, minor distortions in the distorted frames are ignored and considered imperceptible. For the temporal part, SSIM index is simplified and used to measure the temporal video quality. Then, a similar JND concept which comprises of temporal masking is also applied in the temporal quality evaluation. Pixels with large variation over time are considered as not distorted because the distortions in these pixels are hardly perceivable. The final JVQ index is the arithmetic mean of both spatial and temporal quality indices. JVQ is found to achieve good correlation with subjective scores. In addition, this method has low computational cost as compared to existing state-of-the-art metrics.