No-reference Video Quality Assessment Method Research Articles

A multi-scale no-reference video quality assessment method based on transformer

A multi-scale no-reference video quality assessment method based on transformer

MRET: Multi-resolution transformer for video quality assessment

No-reference video quality assessment (NR-VQA) for user generated content (UGC) is crucial for understanding and improving visual experience. Unlike video recognition tasks, VQA tasks are sensitive to changes in input resolution. Since large amounts of UGC videos nowadays are 720p or above, the fixed and relatively small input used in conventional NR-VQA methods results in missing high-frequency details for many videos. In this paper, we propose a novel Transformer-based NR-VQA framework that preserves the high-resolution quality information. With the multi-resolution input representation and a novel multi-resolution patch sampling mechanism, our method enables a comprehensive view of both the global video composition and local high-resolution details. The proposed approach can effectively aggregate quality information across different granularities in spatial and temporal dimensions, making the model robust to input resolution variations. Our method achieves state-of-the-art performance on large-scale UGC VQA datasets LSVQ and LSVQ-1080p, and on KoNViD-1k and LIVE-VQC without fine-tuning.

Study on no-reference video quality assessment method incorporating dual deep learning networks

Study on no-reference video quality assessment method incorporating dual deep learning networks

Learning Generalized Spatial-Temporal Deep Feature Representation for No-Reference Video Quality Assessment

In this work, we propose a no-reference video quality assessment method, aiming to achieve high-generalization capability in cross-content, -resolution and -frame rate quality prediction. In particular, we evaluate the quality of a video by learning effective feature representations in spatial-temporal domain. In the spatial domain, to tackle the resolution and content variations, we impose the Gaussian distribution constraints on the quality features. The unified distribution can significantly reduce the domain gap between different video samples, resulting in more generalized quality feature representation. Along the temporal dimension, inspired by the mechanism of visual perception, we propose a pyramid temporal aggregation module by involving the short-term and long-term memory to aggregate the frame-level quality. Experiments show that our method outperforms the state-of-the-art methods on cross-dataset settings, and achieves comparable performance on intra-dataset configurations, demonstrating the high-generalization capability of the proposed method. The codes are released at <i><uri>https://github.com/Baoliang93/GSTVQA</uri></i>

No-Reference Video Quality Assessment Using Distortion Learning and Temporal Attention

The rapid growth of video consumption and multimedia applications has increased the interest of the academia and industry in building tools that can evaluate perceptual video quality. Since videos might be distorted when they are captured or transmitted, it is imperative to develop reliable methods for no-reference video quality assessment (NR-VQA). To date, most NR-VQA models in prior art have been proposed for assessing a specific category of distortion, such as authentic distortions or traditional distortions. Moreover, those developed for both authentic and traditional distortions video databases have so far led to poor performances. This resulted in the reluctance of service providers to adopt multiple NR-VQA approaches, as they prefer a single algorithm capable of accurately estimating video quality in all situations. Furthermore, many existing NR-VQA methods are computationally complex and therefore impractical for various real-life applications. In this paper, we propose a novel deep learning method for NR-VQA based on multi-task learning where the distortion of individual frames in a video and the overall quality of the video are predicted by a single neural network. This enables to train the network with a greater amount and variety of data, thereby improving its performance in testing. Additionally, our method leverages temporal attention to select the frames of a video sequence which contribute the most to its perceived quality. The proposed algorithm is evaluated on five publicly-available video quality assessment (VQA) databases containing traditional and authentic distortions. Results show that our method outperforms the state-of-the-art on traditional distortion databases such as LIVE VQA and CSIQ video, while also delivering competitive performance on databases containing authentic distortions such as KoNViD-1k, LIVE-Qualcomm and CVD2014.

Efficient video quality assessment with deeper spatiotemporal feature extraction and integration

The challenge of video quality assessment (VQA) modeling for user-generated content (UGC) (i.e., UGC-VQA) is how to accurately extract discriminative features and elaborately quantify interfeature interactions by following the behavior patterns of human eye-brain vision perception. To address this issue, we propose the Deeper Spatial–Temporal Scoring Network (DSTS-Net) to give a precise VQA. Concretely, we first deploy the multiscale feature extraction module to characterize content-aware features accounting for nonlinear reverse hierarchy theory in video perception process, which is not fully considered in the reported UGC-VQA models. Hierarchical handcraft and semantic features are simultaneously considered using content adaptive weighting. Second, we develop a feature integration structure, i.e., deeper gated recurrent unit (DGRU), to fully imitate the interfeature interactions in visionary perception, including feedforward and feedback processes. Third, the dual DGRU structure is employed to further account for interframe interactions of hierarchical features, imitating the nonlinearity of perception as much as possible. Finally, improved pooling is achieved in the local adaptive smoothing module accounting for the temporal hysteresis. Holistic validation of the proposed method on four public challenging UGC-VQA datasets presents a comparable performance over the state-of-the-art no-reference VQA methods, especially our method can give an accurate prediction of the low quality videos with weak temporal correlation. To promote reproducible research and public evaluation, an implementation of our method has been made available online: https://github.com/liu0527aa/DSTS-Net.

No-reference video quality assessment for user generated content based on deep network and visual perception

Video quality assessment (VQA) is an important technique in video service systems. In recent years, the development of deep learning has provided further possibilities for VQA. A no-reference VQA (NR-VQA) method that combines the attention mechanism and human visual perception is proposed for in-the-wild videos. First, a deep network consisting of a convolutional neural network and attention mechanism is constructed to extract depth perception features for frame-level images, and global covariance pooling is integrated into the downsampled features to extract the second-order information of the features. Second, a Transformer network is used for temporal modeling to learn the long-term dependence of the perceptual quality prediction. Finally, a temporal weighting strategy for visual perception is used for weighted summation of the frame-level scores to obtain the final video quality scores. The results of experiments on three public user-generated content authentic distorted video databases, namely KoNViD-1k, CVD2014, and LIVE-VQC, demonstrate that the proposed method can achieve effective quality assessment in authentic distortion and outperforms other partially recent NR-VQA methods.

3D-DWT cross-band statistics and features for No-Reference Video Quality Assessment (NR-VQA)

This paper presents a robust, novel, and computationally efficient noise estimation-based NR-VQA model. It uses four novel sub-band features; namely cross-band statistics, sub-band kurtosis, sub-band energy ratios, and natural video statistics in three-dimensional discrete wavelet transform (3D-DWT) domain. As the HVS shows high sensitivity towards the structural and textural information, a three-dimensional gray level co-occurrence matrix (3D-GLCM) is used to select top 25% video blocks having maximal textural information. We validate the performance of the proposed method on five publicly available and widely used video quality databases. Extensive experiments show that the proposed model yields considerably better and robust performance than other contemporary state-of-the-art NR-VQA methods at lower computational costs.

Multi-Dimensional Feature Fusion Network for No-Reference Quality Assessment of In-the-Wild Videos.

Over the past few decades, video quality assessment (VQA) has become a valuable research field. The perception of in-the-wild video quality without reference is mainly challenged by hybrid distortions with dynamic variations and the movement of the content. In order to address this barrier, we propose a no-reference video quality assessment (NR-VQA) method that adds the enhanced awareness of dynamic information to the perception of static objects. Specifically, we use convolutional networks with different dimensions to extract low-level static-dynamic fusion features for video clips and subsequently implement alignment, followed by a temporal memory module consisting of recurrent neural networks branches and fully connected (FC) branches to construct feature associations in a time series. Meanwhile, in order to simulate human visual habits, we built a parametric adaptive network structure to obtain the final score. We further validated the proposed method on four datasets (CVD2014, KoNViD-1k, LIVE-Qualcomm, and LIVE-VQC) to test the generalization ability. Extensive experiments have demonstrated that the proposed method not only outperforms other NR-VQA methods in terms of overall performance of mixed datasets but also achieves competitive performance in individual datasets compared to the existing state-of-the-art methods.

No‐reference quality assessment of dynamic sports videos based on a spatiotemporal motion model

This paper proposes an approach to improve the performance of no-reference video quality assessment for sports videos with dynamic motion scenes using an efficient spatiotemporal model. In the proposed method, we divide the video sequences into video blocks and apply a 3D shearlet transform that can efficiently extract primary spatiotemporal features to capture dynamic natural motion scene statistics from the incoming video blocks. The concatenation of a deep residual bidirectional gated recurrent neural network and logistic regression is used to learn the spatiotemporal correlation more robustly and predict the perceptual quality score. In addition, conditional video block-wise constraints are incorporated into the objective function to improve quality estimation performance for the entire video. The experimental results show that the proposed method extracts spatiotemporal motion information more effectively and predicts the video quality with higher accuracy than the conventional no-reference video quality assessment methods.

No-Reference Nonuniform Distorted Video Quality Assessment Based on Deep Multiple Instance Learning

Each part of a nonuniform distorted video (NUDV) has a unique distortion degree. When NUDV blocks are used as inputs, traditional machine-learning-based video quality assessment (VQA) methods frequently do not work effectively. Because these methods directly assign the label of the entire video to blocks, causing the unreliability of labels. We creatively propose video bag, a collection of video blocks, to deal with this unreliability. We develop a novel multiple instance learning (MIL) based model, VQA-MIL, which dynamically adjusts the weights by a block-wise attention module and enriches the features of video bags by a MI Pooling layer. Furthermore, we apply the mixup data-augmentation strategy to address the lack of human labels in common video datasets. We test our method on LIVE and CSIQ, and on a relatively large-scale dataset, named NUDV-KT, that we have collected. Results show that our method outperforms popular state-of-the-art no-reference VQA methods on NUDVs.

A No-Reference Video Quality Assessment Model for Underwater Networks

Underwater imagery is increasingly drawing attention from the scientific community, since pictures and videos are invaluable tools in the study of the vast unknown oceanic environment that covers 90% of the planetary biosphere. However, underwater sensor networks must cope with the harsh channel that seawater constitutes. Medium range communication is only possible using acoustic modems that have limited transmission capabilities and peak bitrates of only a few dozens of kilobits per second. These reduced bitrates force heavy compression on videos, yielding much higher levels of distortion than in other video services. Furthermore, underwater video users are ocean researchers, and therefore their quality perception is also different from the generic viewers that typically take part in subjective quality assessment experiments. Computational efficiency is also important since the underwater nodes must run on batteries and their recovery is very expensive. In this paper, we propose a pixel-based no-reference video quality assessment method that addresses the described challenges and achieves good correlations against subjective scores of users of underwater videos.

Two-Level Approach for No-Reference Consumer Video Quality Assessment.

Smartphones and other consumer devices capable of capturing video content and sharing it on social media in nearly real time are widely available at a reasonable cost. Thus, there is a growing need for no-reference video quality assessment (NR-VQA) of consumer produced video content, typically characterized by capture impairments that are qualitatively different from those observed in professionally produced video content. To date, most of the NR-VQA models in prior art have been developed for assessing coding and transmission distortions, rather than capture impairments. In addition, the most accurate NR-VQA methods known in prior art are often computationally complex, and therefore impractical for many real life applications. In this paper, we propose a new approach for learning-based video quality assessment, based on the idea of computing features in two levels so that low complexity features are computed for the full sequence first, and then high complexity features are extracted from a subset of representative video frames, selected by using the low complexity features. We have compared the proposed method against several relevant benchmark methods using three recently published annotated public video quality databases, and our results show that the proposed method can predict subjective video quality more accurately than the benchmark methods. The best performing prior method achieves nearly similar accuracy, but at substantially higher computational cost.

An experimental survey of no-reference video quality assessment methods

Purpose The Video Quality Metric (VQM) is one of the most used objective methods to assess video quality, because of its high correlation with the human visual system (HVS). VQM is, however, not viable in real-time deployments such as mobile streaming, not only due to its high computational demands but also because, as a Full Reference (FR) metric, it requires both the original video and its impaired counterpart. In contrast, No Reference (NR) objective algorithms operate directly on the impaired video and are considerably faster but loose out in accuracy. The purpose of this paper is to study how differently NR metrics perform in the presence of network impairments. Design/methodology/approach The authors assess eight NR metrics, alongside a lightweight FR metric, using VQM as benchmark in a self-developed network-impaired video data set. This paper covers a range of methods, a diverse set of video types and encoding conditions and a variety of network impairment test-cases. Findings The authors show the extent by which packet loss affects different video types, correlating the accuracy of NR metrics to the FR benchmark. This paper helps identifying the conditions under which simple metrics may be used effectively and indicates an avenue to control the quality of streaming systems. Originality/value Most studies in literature have focused on assessing streams that are either unaffected by the network (e.g. looking at the effects of video compression algorithms) or are affected by synthetic network impairments (i.e. via simulated network conditions). The authors show that when streams are affected by real network conditions, assessing Quality of Experience becomes even harder, as the existing metrics perform poorly.

휘도, 색차의 분포도 분석을 이용한 인지적 무기준법 영상 화질 평가방법

With advances in video technology, many researchers are interested in video quality assessment to prove better performance of proposed algorithms. Since human visual system is too complex to be formulated exactly, many researches about video quality assessment are in progressing. No-reference video-quality assessment is suitable for various video streaming services, because of no requested additional data and network capacity to perform quality assessment. In this paper, we propose a novel no-reference video-quality assessment method with the estimation of dynamic range distortion. To measure the performance, we obtain mean opinion score (MOS) data by subject video quality test with the ITU-T P.910 Absolute Category Rating (ACR) method. And, we compare it with proposed algorithm using 363 video sequences. Experimental results show that the proposed algorithm has a higher correlation with obtained MOS.