Video Quality Assessment Methods Research Articles

Retina-based quality assessment of tile-coded 360-degree videos

Nowadays, omnidirectional content, which delivers 360-degree views of scenes, is a significant aspect of Virtual Reality systems. While 360 video requires a lot of bandwidth, users only see visible tiles, therefore a large amount of bitrate can be saved without affecting the user’s experience on the service. The fact leads to current video adaptation solutions to filter out superfluous parts and extraneous bandwidth. To form a good basis for these adaptations, it is necessary to understand human’s video quality perception. In our research, we contribute to building an effective omnidirectional video database that can be applied to study the effects of the five zones of the human retina. We also design a new video quality assessment method to analyze the impacts of those zones of a 360 video according to the human retina. The proposed scheme is found to outperform 22 current objective quality measures by 11 to 31% in terms of the PCC parameter.

Tile-Based Panoramic Video Quality Assessment

As part of video transmission, video quality assessment plays an important role in improving the user’s visual quality and reducing transmission bitrate. However, the existing assessment schemes for the tile-based transmission mechanism cannot accurately predict the video quality. In this paper, we propose a tile-based panoramic video quality assessment method to optimize the transmission strategy for tile-based streaming. Considering the distortion of panoramic video projection, the inverse of gnomonic projection and bilinear interpolation are utilized to convert the planar tiles to sphere. Video Multimethod Assessment Fusion (VMAF) is applied to obtain the tile quality score. Besides, the user’s head movement and eye movement are combined to generate a tile weight map, and generate the final score of the panoramic video through weighted averaging. In addition, considering the impact of the quality fluctuation of the tiles within field of view, the tile quality loss model is established from multiple dimensions including the encoding parameter, viewport position and video content. Experimental results show that there is a high correlation between the predicted results and actual subjective scores, under two video quality assessment datasets. Furthermore, we integrate the method into the panoramic video transmission system. Compared with the tile-based adaptive rate algorithm, the proposed method degrades the video quality by only 3%, while reducing the bandwidth consumption by 16.99%.

Objective Video Quality Assessment Method for Face Recognition Tasks

Nowadays, there are many metrics for overall Quality of Experience (QoE), both those with Full Reference (FR), such as Peak Signal-to-Noise Ratio (PSNR) or Structural Similarity (SSIM), and those with No Reference (NR), such as Video Quality Indicators (VQI), which are successfully used in video processing systems to evaluate videos whose quality is degraded by different processing scenarios. However, they are not suitable for video sequences used for recognition tasks (Target Recognition Videos, TRV). Therefore, correctly estimating the performance of the video processing pipeline in both manual and Computer Vision (CV) recognition tasks is still a major research challenge. There is a need for objective methods to evaluate video quality for recognition tasks. In response to this need, we show in this paper that it is possible to develop the new concept of an objective model for evaluating video quality for face recognition tasks. The model is trained, tested and validated on a representative set of image sequences. The set of degradation scenarios is based on the model of a digital camera and how the luminous flux reflected from the scene eventually becomes a digital image. The resulting degraded images are evaluated using a CV library for face recognition as well as VQI. The measured accuracy of a model, expressed as the value of the F-measure parameter, is 0.87.

Learning Generalized Spatial-Temporal Deep Feature Representation for No-Reference Video Quality Assessment

In this work, we propose a no-reference video quality assessment method, aiming to achieve high-generalization capability in cross-content, -resolution and -frame rate quality prediction. In particular, we evaluate the quality of a video by learning effective feature representations in spatial-temporal domain. In the spatial domain, to tackle the resolution and content variations, we impose the Gaussian distribution constraints on the quality features. The unified distribution can significantly reduce the domain gap between different video samples, resulting in more generalized quality feature representation. Along the temporal dimension, inspired by the mechanism of visual perception, we propose a pyramid temporal aggregation module by involving the short-term and long-term memory to aggregate the frame-level quality. Experiments show that our method outperforms the state-of-the-art methods on cross-dataset settings, and achieves comparable performance on intra-dataset configurations, demonstrating the high-generalization capability of the proposed method. The codes are released at <i><uri>https://github.com/Baoliang93/GSTVQA</uri></i>

The optimization network video quality assessment method based on fuzzy inference

The optimization network video quality assessment method based on fuzzy inference

The critical review of image and video quality assessment methods

The critical review of image and video quality assessment methods

Video Quality Assessment With Serial Dependence Modeling

Video quality assessment (VQA) is much more challenging than image quality assessment, due to the difficulty of modeling temporal influence among frames. Most of the existing VQA methods usually isolate each moment within the video (i.e., it neglects the sequential nature), leading to a large gap from the subjective perception. Recent research on neuroscience suggests a serially dependent perception (SDP) mechanism in the human visual system (HVS). Namely, the HVS tends to incorporate the recent past visual experience to predict the present perception. Inspired by the SDP, we suggest that the HVS prefers stable and continuous degradations in videos due to their predictability, and exhibits less tolerance to interrupted and unpredictable disturbances. Thus, we introduce a novel serial dependence modeling (SDM) framework for full-reference VQA in this paper. Firstly, the instantaneous degradation is measured on both the static appearance and motion information for each glimpse of scenes. Since motion plays an important role in videos, two types of structures are extracted for motion representation, namely, an explicit content-based 3D structure and an implicit feature-based 2D structure. Next, an assessment-directed long-short term memory (A-LSTM) is proposed to capture the serial dependence among instantaneous degradations. With the consideration of the perceptual effect from the previous moment on the current one, especially the effect from the perceptually worst moment, the serially dependent degradation is characterized. Finally, by mimicking the subjective rating for video-viewing, an attention-based quality decision procedure is presented to acquire the final video quality. Experimental results on publicly available VQA databases demonstrate that the proposed method maintains good consistency with the subjective perception.

Efficient video quality assessment with deeper spatiotemporal feature extraction and integration

The challenge of video quality assessment (VQA) modeling for user-generated content (UGC) (i.e., UGC-VQA) is how to accurately extract discriminative features and elaborately quantify interfeature interactions by following the behavior patterns of human eye-brain vision perception. To address this issue, we propose the Deeper Spatial–Temporal Scoring Network (DSTS-Net) to give a precise VQA. Concretely, we first deploy the multiscale feature extraction module to characterize content-aware features accounting for nonlinear reverse hierarchy theory in video perception process, which is not fully considered in the reported UGC-VQA models. Hierarchical handcraft and semantic features are simultaneously considered using content adaptive weighting. Second, we develop a feature integration structure, i.e., deeper gated recurrent unit (DGRU), to fully imitate the interfeature interactions in visionary perception, including feedforward and feedback processes. Third, the dual DGRU structure is employed to further account for interframe interactions of hierarchical features, imitating the nonlinearity of perception as much as possible. Finally, improved pooling is achieved in the local adaptive smoothing module accounting for the temporal hysteresis. Holistic validation of the proposed method on four public challenging UGC-VQA datasets presents a comparable performance over the state-of-the-art no-reference VQA methods, especially our method can give an accurate prediction of the low quality videos with weak temporal correlation. To promote reproducible research and public evaluation, an implementation of our method has been made available online: https://github.com/liu0527aa/DSTS-Net.

DVL2021: An ultra high definition video dataset for perceptual quality study

This paper describes an ultra high definition (UHD) video dataset named DVL2021 for the perceptual study of video quality assessment (VQA). To our knowledge, DVL2021 is the first authentically distorted 4K (3840 × 2160) UHD video quality dataset. The dataset contains 206 versatile 4K UHD video sequences, which are all collected in in-the-wild scenarios. Each sequence is captured at 50 frames per second (fps), stored in raw 10-bit 4:2:0 YUV format, and has a duration of 10 s. Following the subjective evaluation method of TV image quality granted by ITU-R BT.500-13, 32 unique participants take part in the manual annotation process, whose ages are from teenage to sixties (32.7 years old on average). DVL2021 has the following merits: (1) enormous variety of video contents, (2) captured by different types of cameras, (3) complex types and multiple levels of authentic distortion, (4) broadly distributed temporal/spatial information, and (5) a wide spectrum of mean opinion scores (MOS) distribution. Furthermore, we conduct a benchmark experiment by evaluating several mainstream VQA methods on DVL2021. The baseline results are higher than 0.75 in Spearman’s rank order correlation coefficient (SROCC) metric. Our study provides a basis for the UHD VQA problem. DVL2021 is publicly available at https://github.com/GZHU-DVL/DVL2021.

CNN-based no-reference video quality assessment method using a spatiotemporal saliency patch selection procedure

We propose a yet lightweight no-reference (NR) video quality assessment (VQA) method, which uses a convolution neural network (CNN) architecture. The proposed method implements a spatiotemporal saliency patch selection procedure that crops the frame into small nonoverlapping blocks of images (patches) and selects the most perceptually relevant ones. The selected patches are then forwarded to the CNN. To determine which patches are the most relevant, spatial and temporal saliency features are computed for each frame. The proposed method does not require subjective scores to train the CNN. It uses objective quality scores as target quality scores for each video frame, which are computed using an NR image quality assessment method. Given the lack of large annotated video quality databases, this is an advantage of the proposed method. Finally, although it has much smaller cost of data-processing, compared with other state-of-the-art methods, the proposed NR-VQA obtains robust and competitive results.

No-reference video quality assessment for user generated content based on deep network and visual perception

Video quality assessment (VQA) is an important technique in video service systems. In recent years, the development of deep learning has provided further possibilities for VQA. A no-reference VQA (NR-VQA) method that combines the attention mechanism and human visual perception is proposed for in-the-wild videos. First, a deep network consisting of a convolutional neural network and attention mechanism is constructed to extract depth perception features for frame-level images, and global covariance pooling is integrated into the downsampled features to extract the second-order information of the features. Second, a Transformer network is used for temporal modeling to learn the long-term dependence of the perceptual quality prediction. Finally, a temporal weighting strategy for visual perception is used for weighted summation of the frame-level scores to obtain the final video quality scores. The results of experiments on three public user-generated content authentic distorted video databases, namely KoNViD-1k, CVD2014, and LIVE-VQC, demonstrate that the proposed method can achieve effective quality assessment in authentic distortion and outperforms other partially recent NR-VQA methods.

On the Semantic Dependency of Video Quality Assessment Methods

Blind assessment of video quality is a widely covered topic in computer vision. In this work, we perform an analysis of how much the effectiveness of some of the current No-Reference VQA (NR-VQA) methods varies with respect to specific types of scenes. To this end, we automatically annotated the videos from two video quality datasets with user-generated videos whose content is unknown and then estimated the correlation for the different categories of scenes. The results of the analysis highlight that the prediction errors are not equally distributed among the different categories of scenes and indirectly suggest what next generation NR-VQA methods should take into account and model.

The research of OTT video quality assessment method

The research of OTT video quality assessment method

Multi-Dimensional Feature Fusion Network for No-Reference Quality Assessment of In-the-Wild Videos.

Over the past few decades, video quality assessment (VQA) has become a valuable research field. The perception of in-the-wild video quality without reference is mainly challenged by hybrid distortions with dynamic variations and the movement of the content. In order to address this barrier, we propose a no-reference video quality assessment (NR-VQA) method that adds the enhanced awareness of dynamic information to the perception of static objects. Specifically, we use convolutional networks with different dimensions to extract low-level static-dynamic fusion features for video clips and subsequently implement alignment, followed by a temporal memory module consisting of recurrent neural networks branches and fully connected (FC) branches to construct feature associations in a time series. Meanwhile, in order to simulate human visual habits, we built a parametric adaptive network structure to obtain the final score. We further validated the proposed method on four datasets (CVD2014, KoNViD-1k, LIVE-Qualcomm, and LIVE-VQC) to test the generalization ability. Extensive experiments have demonstrated that the proposed method not only outperforms other NR-VQA methods in terms of overall performance of mixed datasets but also achieves competitive performance in individual datasets compared to the existing state-of-the-art methods.

Video quality assessment with dense features and ranking pooling

Benefiting with the rapid development of communication networks, effective video quality assessment (VQA) models which provide guidance for video transmission and compression technologies are highly demanded. This paper proposes a general-purpose full-reference VQA method combining DenseNet with spatial pyramid pooling and RankNet to not only extract high-level distortion representation and global spatial information of samples but also characterize the temporal correlation among frames. Firstly, the pretrained DenseNet is modified and finetuned to extract high-level features of distorBenefiting with the rapid development of communication networks, effective video quality assessment (VQA) models which provide guidance for video transmission and compression technologies are highly demanded. This paper proposes a general-purpose full-reference VQA method combining DenseNet with spatial pyramid pooling and RankNet to not only extract high-level distortion representation and global spatial information of samples but also characterize the temporal correlation among frames. Firstly, the pretrained DenseNet is modified and finetuned to extract high-level features of distorted videos. Then, spatial pyramid pooling is equipped in the DenseNet module to process flexible inputs with arbitrary spatial resolution. Thus, this kind of input which has the same spatial resolution as the original distorted video is processed by the well-trained DenseNet to generate frame-level quality, which considers the global spatial information of videos directly. Finally, learning to rank is introduced to explore the high-level temporal correlation of distorted videos by taking the RankNet as the temporal pooling function. The experimental results on two public VQA databases show that the proposed algorithm performs consistently with human visual perception.

Evaluation of objective video quality assessment methods on video sequences with different spatial and temporal activity encoded at different spatial resolutions

With the development of Video on Demand applications due to the availability of high-speed internet access, adaptive streaming algorithms have been developing and improving. The focus is on improving user’s Quality of Experience (QoE) and taking it into account as one of the parameters for the adaptation algorithm. Users often experience changing network conditions, so the goal is to ensure stable video playback with satisfying QoE level. Although subjective Video Quality Assessment (VQA) methods provide more accurate results regarding user’s QoE, objective VQA methods cost less and are less time-consuming. In this article, nine different objective VQA methods are compared on a large set of video sequences with various spatial and temporal activities. VQA methods used in this analysis are: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), MultiScale Structural Similarity Index (MS-SSIM), Video Quality Metric (VQM), Mean Sum of Differences (DELTA), Mean Sum of Absolute Differences (MSAD), Mean Squared Error (MSE), Netflix Video Multimethod Assessment Fusion (Netflix VMAF) and Visual Signal-to-Noise Ratio (VSNR). The video sequences used for testing purposes were encoded according to H.264/AVC with twelve different target coding bitrates, at three different spatial resolutions (resulting in a total of 190 sequences). In addition to objective quality assessment, subjective quality assessment was performed for these sequences. All results acquired by objective VQA methods have been compared with subjective Mean Opinion Score (MOS) results using Pearson Linear Correlation Coefficient (PLCC). Measurement results obtained on a large set of video sequences with different spatial resolutions show that VQA methods like SSIM and VQM correlate better with MOS results compared to PSNR, SSIM, VSNR, DELTA, MSE, VMAF and MSAD. However, the PLCC results for SSIM and VQM are too low (0.7799 and 0.7734, respectively), for the usage of these methods in streaming services instead of subjective testing. These results suggest that more efficient VQA methods should be developed to be used in streaming testing procedures as well as to support the video segmentation process. Furthermore, when comparing results obtained for different spatial resolutions, it can be concluded that the quality of video sequences encoded at lower spatial resolutions in cases of lower target coding bitrate is higher compared to the quality of video sequences encoded at higher spatial resolutions at the same target coding bitrate, particularly when video sequences with higher spatial and temporal information are used.

Reference-Free DIBR-Synthesized Video Quality Metric in Spatial and Temporal Domains

Depth image-based rendering (DIBR) techniques play an important role in free viewpoint videos (FVVs), which have a wide range of applications including immersive entertainment, remote monitoring, education, etc. FVVs are usually synthesized by DIBR techniques in a &#x201C;blind&#x201D; environment (without a reference video). Thus, an effective reference-free synthesized video quality assessment (VQA) metric is vital. At present, many image quality assessment (IQA) algorithms for DIBR-synthesized images have been proposed, but limited researches have been concerned about the quality assessment of DIBR-synthesized videos. To this end, this paper proposes a novel reference-free VQA method for synthesized videos, which operates in Spatial and Temporal Domains, dubbed as STD. The design fundamental of the proposed STD metric considers the effects of two major distortions introduced by DIBR techniques on the visual quality of synthesized videos. First, considering the geometric distortion introduced by DIBR technologies can increase high-frequency contents of the synthesized frame, the influence of the geometric distortion on the visual quality of a synthesized video can be effectively evaluated by estimating high-frequency energies of each synthesized frame in spatial domain. Second, temporal inconsistency caused by DIBR techniques brings the temporal flicker distortion, which is one of the most annoying artifacts in DIBR-synthesized videos. In temporal domain, we quantify temporal inconsistency by measuring motion differences between consecutive frames. Specifically, optical flow method is first used to estimate the motion field between adjacent frames. Then, we calculate the structural similarity of adjacent optical flow fields and further adopt the structural similarity value to weight the pixel differences of adjacent optical flow fields. Experiments show that the above two features are able to well perceive the visual quality of DIBR-synthesized videos. Furthermore, since the two features are extracted from spatial and temporal domains, respectively, we integrate them using a linear weighting strategy to obtain our STD metric, which proves advantageous over two components and the competing state-of-the-art I/VQA methods. The source code is available at <uri>https://github.com/wgc-vsfm/DIBR-video-quality-assessment</uri>.

No‐reference quality assessment of dynamic sports videos based on a spatiotemporal motion model

This paper proposes an approach to improve the performance of no-reference video quality assessment for sports videos with dynamic motion scenes using an efficient spatiotemporal model. In the proposed method, we divide the video sequences into video blocks and apply a 3D shearlet transform that can efficiently extract primary spatiotemporal features to capture dynamic natural motion scene statistics from the incoming video blocks. The concatenation of a deep residual bidirectional gated recurrent neural network and logistic regression is used to learn the spatiotemporal correlation more robustly and predict the perceptual quality score. In addition, conditional video block-wise constraints are incorporated into the objective function to improve quality estimation performance for the entire video. The experimental results show that the proposed method extracts spatiotemporal motion information more effectively and predicts the video quality with higher accuracy than the conventional no-reference video quality assessment methods.

KonVid-150k: A Dataset for No-Reference Video Quality Assessment of Videos in-the-Wild

Video quality assessment (VQA) methods focus on particular degradation types, usually artificially induced on a small set of reference videos. Hence, most traditional VQA methods under-perform in-the-wild. Deep learning approaches have had limited success due to the small size and diversity of existing VQA datasets, either artificial or authentically distorted. We introduce a new in-the-wild VQA dataset that is substantially larger and diverse: KonVid-150k. It consists of a coarsely annotated set of 153,841 videos having five quality ratings each, and 1,596 videos with a minimum of 89 ratings each. Additionally, we propose new efficient VQA approaches (MLSP-VQA) relying on multi-level spatially pooled deep-features (MLSP). They are exceptionally well suited for training at scale, compared to deep transfer learning approaches. Our best method, MLSP-VQA-FF, improves the Spearman rank-order correlation coefficient (SRCC) performance metric on the commonly used KoNViD-1k in-the-wild benchmark dataset to 0.82. It surpasses the best existing deep-learning model (0.80 SRCC) and hand-crafted feature-based method (0.78 SRCC). We further investigate how alternative approaches perform under different levels of label noise, and dataset size, showing that MLSP-VQA-FF is the overall best method for videos in-the-wild. Finally, we show that the MLSP-VQA models trained on KonVid-150k sets the new state-of-the-art for cross-test performance on KoNViD-1k and LIVE-Qualcomm with a 0.83 and 0.64 SRCC, respectively. For KoNViD-1k this inter-dataset testing outperforms intra-dataset experiments, showing excellent generalization.

Subjective Video QoE With a Hybrid Cognitive-Motor Task

This paper reports on the subjective video quality analysis with and without a parallel task. It aims to understand how the quality perceived by the human observer brain is affected when the subject is not entirely focused on the judgment task. For this purpose, two subjective video quality tests were performed according to the ITU-T P.910 recommendation - Subjective video quality assessment methods for multimedia applications. They consisted of two parts involving a total of 24 subjects. Unlike the first test, the second test included an additional (parallel) psychomotor task where subjects had to sort different colors of pasta into three bowls. Various analyses were provided, including preliminary analysis, group analysis, and RMSE analysis. Surprisingly, the results show that not all the spectrum of quality is affected equally by parallel task introduction.