Video Quality Metrics Research Articles

Modeling and estimating the subjects\u2019 diversity of opinions in video quality assessment: a neural network based approach

Subjective experiments are considered the most reliable way to assess the perceived visual quality. However, observers’ opinions are characterized by large diversity: in fact, even the same observer is often not able to exactly repeat his first opinion when rating again a given stimulus. This makes the Mean Opinion Score (MOS) alone, in many cases, not sufficient to get accurate information about the perceived visual quality. To this aim, it is important to have a measure characterizing to what extent the observed or predicted MOS value is reliable and stable. For instance, the Standard deviation of the Opinions of the Subjects (SOS) could be considered as a measure of reliability when evaluating the quality subjectively. However, we are not aware of the existence of models or algorithms that allow to objectively predict how much diversity would be observed in subjects’ opinions in terms of SOS. In this work we observe, on the basis of a statistical analysis made on several subjective experiments, that the disagreement between the quality as measured by means of different objective video quality metrics (VQMs) can provide information on the diversity of the observers’ ratings on a given processed video sequence (PVS). In light of this observation we: i) propose and validate a model for the SOS observed in a subjective experiment; ii) design and train Neural Networks (NNs) that predict the average diversity that would be observed among the subjects’ ratings for a PVS starting from a set of VQMs values computed on such a PVS; iii) give insights into how the same NN based approach can be used to identify potential anomalies in the data collected in subjective experiments.

A low-complexity psychometric curve-fitting approach for the objective quality assessment of streamed game videos

The increasing popularity of video gaming competitions, the so called eSports, has contributed to the rise of a new type of end-user: the passive game video streaming (GVS) user. This user acts as a passive spectator of the gameplay rather than actively interacting with the content. This content, which is streamed over the Internet, can suffer from disturbing network and encoding impairments. Therefore, assessing the user’s perceived quality, i.e the Quality of Experience (QoE), in real-time becomes fundamental. For the case of natural video content, several approaches already exist that tackle the client-side real-time QoE evaluation. The intrinsically different expectations of the passive GVS user, however, call for new real-time quality models for these streaming services. Therefore, this paper presents a real-time Reduced-Reference (RR) quality assessment framework based on a low-complexity psychometric curve-fitting approach. The proposed solution selects the most relevant, low-complexity objective feature. Afterwards, the relationship between this feature and the ground-truth quality is modelled based on the psychometric perception of the human visual system (HVS). This approach is validated on a publicly available dataset of streamed game videos and is benchmarked against both subjective scores and objective models. As a side contribution, a thorough accuracy analysis of existing Objective Video Quality Metrics (OVQMs) applied to passive GVS is provided. Furthermore, this analysis has led to interesting insights on the accuracy of low-complexity client-based metrics as well as to the creation of a new Full-Reference (FR) objective metric for GVS, i.e. the Game Video Streaming Quality Metric (GVSQM).

Full-Reference Stereoscopic Video Quality Assessment Using a Motion Sensitive HVS Model

Stereoscopic video quality assessment has become a major research topic in recent years. Existing stereoscopic video quality metrics are predominantly based on stereoscopic image quality metrics extended to the time domain via for example temporal pooling. These approaches do not explicitly consider the motion sensitivity of the Human Visual System (HVS). To address this limitation, this paper introduces a novel HVS model inspired by physiological findings characterising the motion sensitive response of complex cells in the primary visual cortex (V1 area). The proposed HVS model generalises previous HVS models, which characterised the behaviour of simple and complex cells but ignored motion sensitivity, by estimating optical flow to measure scene velocity at different scales and orientations. The local motion characteristics (direction and amplitude) are used to modulate the output of complex cells. The model is applied to develop a new type of full-reference stereoscopic video quality metrics which uniquely combine non-motion sensitive and motion sensitive energy terms to mimic the response of the HVS. A tailored two-stage multi-variate stepwise regression algorithm is introduced to determine the optimal contribution of each energy term. The two proposed stereoscopic video quality metrics are evaluated on three stereoscopic video datasets. Results indicate that they achieve average correlations with subjective scores of 0.9257 (PLCC), 0.9338 and 0.9120 (SRCC), 0.8622 and 0.8306 (KRCC), and outperform previous stereoscopic video quality metrics including other recent HVS-based metrics.

No‐reference quality assessment for live broadcasting videos in temporal and spatial domains

Nowadays, live broadcasting video has become increasingly popular and high-quality live broadcasting video is highly needed. In practice, live broadcasting videos usually undergo several processing stages, which inevitably introduce multiple distortions, e. g. frame freezing and intensity mutation, causing the degraded quality of experience. However, little work has been done to the quality evaluation of live broadcasting videos, which may hinder the further development of more advanced live broadcasting video delivery systems. Motivated by this, this study presents a no-reference quality evaluation model for live broadcasting videos (LBVQA) in temporal and spatial domains. In the temporal domain, statistic features are extracted to measure the frame freezing and intensity mutation, and the entropy-based feature is extracted to describe the global jitter. In the spatial domain, blurring is measured based on phase coherence, and abnormal exposure ratio is calculated based on an adaptive threshold. Finally, all features are fed into a backpropagation neural network to train the quality prediction model. Experimental results on the Live Broadcasting Video Database demonstrate the advantages of the proposed metric over the state-of-the-art image and video quality metrics.

Video Multimethod Assessment Fusion (VMAF) on 360VR Contents

This paper describes the subjective experiments and subsequent analysis carried out to validate the application of one of the most robust and influential video quality metrics, Video Multimethod Assessment Fusion (VMAF), to 360VR contents. VMAF is a full reference metric initially designed to work with traditional 2D contents. Hence, at first, it cannot be assumed to be compatible with the particularities of the scenario where omnidirectional content is visualized using a Head-Mounted Display (HMD). Therefore, through a complete set of tests, we prove that this metric can be successfully used without any specific training or adjustments to obtain the quality of 360VR sequences actually perceived by users.

Analyzing the performance of autoencoder-based objective quality metrics on audio-visual content

The development of audio-visual quality models faces a number of challenges, including the integration of audio and video sensory channels and the modeling of their interaction characteristics. Commonly, objective quality metrics estimate the quality of a single component (audio or video) of the content. Machine learning techniques, such as autoencoders, offer as a very promising alternative to develop objective assessment models. This paper studies the performance of a group of autoencoder-based objective quality metrics on a diverse set of audio-visual content. To perform this test, we use a large dataset of audio-visual content (The UnB-AV database), which contains degradations in both audio and video components. The database has accompanying subjective scores collected on three separate subjective experiments. We compare our autoencoder-based methods, which take into account both audio and video components (multi-modal), against several objective (single-modal) audio and video quality metrics. The main goal of this work is to verify the gain or loss in performance of these single-modal metrics, when tested on audio-visual sequences.

Open Software Framework for Collaborative Development of No Reference Image and Video Quality Metrics

This paper describes ongoing work within the video quality experts group (VQEG) to develop no-reference (NR) audiovisual video quality analysis (VQA) metrics. VQEG provides an open forum that encourages knowledge sharing and collaboration. The VQEG no-reference Metric (NORM) group’s goal is to develop open-source NR-VQA metrics that meet industry requirements for scope, accuracy, and capability. This paper presents industry specifications from discussions at VQEG face-to-face meetings among industry, academic, and government participants. This paper also announces an open software framework for collaborative development of NR image quality Analysis (IQA) and VQA metrics <ext-link ext-link-type="url" xlink:href="https://github.com/NTIA/NRMetricFramework"><https://github.com/NTIA/NRMetricFramework></ext-link>. This framework includes the support tools necessary to begin research and avoid common mistakes. VQEG’s goal is to produce a series of NR-VQA metrics with progressively improving scope and accuracy. This work draws upon and enables IQA metric research, as both use the human visual system to analyze the quality of audiovisual media on modern displays. Readers are invited to participate.

Investigating Prediction Accuracy of Full Reference Objective Video Quality Measures through the ITS4S Dataset

Large subjectively annotated datasets are crucial to the development and testing of objective video quality measures (VQMs). In this work we focus on the recently released ITS4S dataset. Relying on statistical tools, we show that the content of the dataset is rather heterogeneous from the point of view of quality assessment. Such diversity naturally makes the dataset a worthy asset to validate the accuracy of video quality metrics (VQMs). In particular we study the ability of VQMs to model the reduction or the increase of the visibility of distortion due to the spatial activity in the content. The study reveals that VQMs are likely to overestimate the perceived quality of processed video sequences whose source is characterized by few spatial details. We then propose an approach aiming at modeling the impact of spatial activity on distortion visibility when objectively assessing the visual quality of a content. The effectiveness of the proposal is validated on the ITS4S dataset as well as on the Netflix public dataset.

The Performance of Quality Metrics in Assessing Error-Concealed Video Quality.

In highly-interactive video streaming applications such as video conferencing, tele-presence, or tele-operation, retransmission is typically not used, due to the tight deadline of the application. In such cases, the lost or erroneous data must be concealed. While various error concealment techniques exist, there is no defined rule to compare their perceived quality. In this paper, the performance of 16 existing image and video quality metrics (PSNR, SSIM, VQM, etc.) evaluating errorconcealed video quality is studied. The encoded video is subjected to packet loss and the loss is concealed using various error concealment techniques. We show that the subjective quality of the video cannot be necessarily predicted from the visual quality of the error-concealed frame alone. We then apply the metrics to the error-concealed images/videos and evaluate their success in predicting the scores reported by human subjects. The errorconcealed videos are judged by image quality metrics applied on the lossy frame, or by video quality metrics applied on the video clip containing that lossy frame; this way, the impact of error propagation is also considered by the objective metrics. The measurement and comparison of the results show that, mostly though not always, measuring the objective quality of the video is a better way to judge the error concealment performance. Moreover, our experiments show that when the objective quality metrics are used for the assessment of the performance of an error concealment technique, they do not behave as they would for general quality assessment. In fact, some newly developed metrics show the correct decision only about 60% of the time, leading to an unacceptable error rate of as much as 40%. Our analysis shows which specific quality metrics are relatively more suitable for error-concealed videos.

A Framework for in-Network QoE Monitoring of Encrypted Video Streaming

With the amount of global network traffic steadily increasing, mainly due to video streaming services, network operators are faced with the challenge of efficiently managing their resources while meeting customer demands and expectations. A prerequisite for such Quality-of-Experience-driven (QoE) network traffic management is the monitoring and inference of application-level performance in terms of video Key Performance Indicators (KPIs) that directly influence end-user QoE. Given the persistent adoption of end-to-end encryption, operators lack direct insights into video quality metrics such as start-up delays, resolutions, or stalling events, which are needed to adequately estimate QoE and drive resource management decisions. Numerous solutions have been proposed to tackle this challenge on individual use-cases, most of them relying on machine learning (ML) for inferring KPIs from observable traffic patterns and statistics. In this paper, we summarize the key findings in state-of-the-art research on the topic. Going beyond previous work, we devise the concept of a generic framework for ML-based QoE/KPI monitoring of HTTP adaptive streaming (HAS) services, including model training, deployment, and re- evaluation. Components of the framework are designed in a generic way, independent of a particular streaming service and platform. The methodology for applying different framework components is discussed across various use-cases. In particular, we demonstrate framework applicability in a concrete use-case involving the YouTube service delivered to smartphones via the mobile YouTube app, as this presents one of the most prominent examples of accessing YouTube. We tackle both QoE/KPI estimation on a per-video-session level (utilizing the validated ITU-T P.1203 QoE model), as well as “real-time” KPI estimation over short time intervals. Obtained results provide important insights and challenges related to the deployment of a generic in-network QoE monitoring framework for encrypted video streams.

Global Rate-Distortion Optimization-Based Rate Control for HEVC HDR Coding

High dynamic range (HDR) video compression technology, which is capable of delivering a wider range of luminance and a larger colour gamut than standard dynamic range (SDR) technology, has been widely used in recent years in many fields, including industrial image processing, digital entertainment, and machine vision. Rate control (RC) is of paramount importance to HDR compression and transmission; accordingly, an RC scheme for HDR in High Efficiency Video Coding (HEVC) is proposed in this paper. First, considering the HDR characteristics, we propose an HDR-Visual Difference Predictor (VDP)-2-based rate-distortion (R-D) model to improve the coding performance. Second, we directly utilize $\lambda $ rather than the bit rate in the optimization process to obtain the optimal solution. Finally, we propose a new model parameter estimation method to further reduce the RC errors. According to our experimental results, significant bit rate reductions in terms of HDR-VDP-2, the Video Quality Metric (VQM) and the mean peak-signal-to-noise ratio (mPSNR) can be achieved on average compared with the state-of-the-art algorithm used in HM16.19.

A Metric for Video Blending Quality Assessment.

We propose an objective approach to assess the quality of video blending. Blending is a fundamental operation in video editing, which can smooth the intensity changes of relevant regions. However blending also generates artefacts such as bleeding and ghosting. To assess the quality of the blended videos, our approach considers the illuminance consistency as a positive aspect while regard the artefacts as a negative aspect. Temporal coherence between frames is also considered. We evaluate our metric on a video blending dataset where the results of subjective evaluation are available. Experimental results validate the effectiveness of our proposed metric, and shows that this metric gives superior performance over existing video quality metrics.

Experimental QoE Evaluation of Multicast Video Delivery over IEEE 802.11aa WLANs

The IEEE 802.11aa amendment standardised the Group Addressed Transmission Service (GATS), which extends 802.11 WLANs with a novel set of MAC mechanisms to support an effective and efficient multicast video service. The key challenge with GATS is the selection of the best scheme and its configuration for a given network scenario, as the standard does not provide any guidelines nor any assessment of the performance of each mechanism. Although some previous studies have addressed this challenge, their evaluation is either via analysis or simulations under non-realistic assumptions, or based on Quality of Service (QoS) metrics instead of video quality metrics, which are required for a proper video performance assessment. In this paper, we deploy a mid-size real-life testbed and develop a thoughtful methodology to perform an extensive Quality of Experience (QoE) evaluation of GATS under a variety of scenarios. We analyse the performance of the novel schemes under ideal conditions, as well as under controlled and non-controlled interference, assessing their ability to provide an adequate QoE and quantifying the resources left for other type of traffic. Ours is the first thorough QoE evaluation of GATS in a real-life scenario, providing key insights on their performance, and can be used to derive configuration guidelines for the schemes.

AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience

User expectations have a crucial impact on the levels of quality of experience (QoE) that they consider acceptable or satisfying. Measuring acceptability and annoyance has mainly been performed in separate or multi-step experiments without any control over participants’ expectations. This paper introduces a simple methodology to obtain the information about both of the entities in a single step and compares several data processing strategies useful for results interpretation. A specifically designed subjective experiment, conducted on compressed videos, has shown that the multi-step procedures could be replaced by our proposed single-step approach, regardless of the viewing conditions, while the novel approach is significantly preferred by observers for its low time requirements and higher intuitiveness. The test has simultaneously proven that user expectations can be altered by the instructions and it is, therefore, possible to simulate different user profiles regardless of the participants’ real habits. The acceptability/annoyance experimental results are also used to benchmark the state-of-the-art objective video quality metrics in predicting acceptability/annoyance of QoE. A case study on the determination of the threshold of acceptability/annoyance for objective quality metrics is conducted, which can be served as a guideline for video streaming service providers.

Quality of Experience (QoE)-Aware Fast Coding Unit Size Selection for HEVC Intra-Prediction

The exorbitant increase in the computational complexity of modern video coding standards, such as High Efficiency Video Coding (HEVC), is a compelling challenge for resource-constrained consumer electronic devices. For instance, the brute force evaluation of all possible combinations of available coding modes and quadtree-based coding structure in HEVC to determine the optimum set of coding parameters for a given content demand a substantial amount of computational and energy resources. Thus, the resource requirements for real time operation of HEVC has become a contributing factor towards the Quality of Experience (QoE) of the end users of emerging multimedia and future internet applications. In this context, this paper proposes a content-adaptive Coding Unit (CU) size selection algorithm for HEVC intra-prediction. The proposed algorithm builds content-specific weighted Support Vector Machine (SVM) models in real time during the encoding process, to provide an early estimate of CU size for a given content, avoiding the brute force evaluation of all possible coding mode combinations in HEVC. The experimental results demonstrate an average encoding time reduction of 52.38%, with an average Bjøntegaard Delta Bit Rate (BDBR) increase of 1.19% compared to the HM16.1 reference encoder. Furthermore, the perceptual visual quality assessments conducted through Video Quality Metric (VQM) show minimal visual quality impact on the reconstructed videos of the proposed algorithm compared to state-of-the-art approaches.

On the suitability of VMAF for quality assessment of medical videos: Medical ultrasound & wireless capsule endoscopy

With the rapid evolution in modern multimedia networks and systems, services such as telemedicine and tele-surgery are becoming more popular. Quality estimation and monitoring of medical videos is becoming important not only in the field of research, but also in real-time applications and services. The state-of-the-art video quality metric (VQM) called Video Multimethod Assessment Fusion (VMAF) is a promising solution for quality estimation of videos impaired by compression and scaling artifacts. The metric was developed by Netflix for entertainment video content and its good performance does not necessarily extend to medical videos. This paper focuses on evaluating the performance of VMAF in the context of quality assessment (QA) for medical videos. We consider in this paper medical videos compressed via High Efficiency Video Coding (HEVC) and refer in particular to medical ultrasound videos and wireless capsule endoscopy (WCE) videos for the performance estimation of VMAF. The correlation between the subjective scores of these two datasets and VMAF's quality estimates is studied and presented. The results show that VMAF outperforms other state-of-the-art VQMs in the context of WCE videos, but this is not the case for medical ultrasound videos.

No-reference artifacts measurements based video quality metric

In multimedia delivery, perceived quality of video signal has a significant role in overall users Quality of Experience (QoE). Therefore, multimedia service providers must constantly measure and monitor perceived video quality, which is usually performed using no-reference (NR) video quality metrics. In this paper, a novel NR objective video quality metric named Artifacts Measurements Based Video Quality Metric (AMB-VQM) is proposed. In addition to artifacts measures computed by blocking (BL), packet-loss (PL), and freezing (FZ) artifacts detection algorithms, the metric incorporates artifacts masking based on spatial and temporal video content complexity. Furthermore, a newly created FERIT-RTRK-2 video quality database, which contains 486 Full HD test video sequences impaired by video compression (MPEG-2, H.264/AVC, and H.265), packet-loss, frame freezing, and combinations of these procedures, is presented in this paper. FERIT-RTRK-2 database is publicly available for scientific community at http://www.rt-rk.com/other/VideoDBReadme.html. Additionally, newly created user-friendly subjective video quality assessment tool with graphical user interface, which can be used for conducting of subjective video quality experiments, is presented. In experimental part the performance of the proposed AMB-VQM is compared to this of 10 other objective video quality metrics (PSNR, SSIM, VSNR, PSNRHVS, PSNRHVSM, VIFP, ViS3, ST-MAD, BRISQUE, VIIDEO) using distorted video sequences from four different video quality databases: CSIQ, LIVE, FERIT-RTRK and FERIT-RTRK-2. The results show that AMB-VQM achieves high performance when predicting the video quality for videos distorted in different manners. AMB-VQM results outperform the results of most of the analyzed popular and very often used video quality metrics.

No-reference prediction of quality metrics for H.264-compressed infrared sequences for unmanned aerial vehicle applications

We propose a no-reference (NR) method for estimating the scores of metrics assessing the quality of infrared (IR) sequences compressed with H.264 for low-complexity unmanned aerial vehicle (UAV) applications. The scenario studied is to estimate the quality on an on-ground computer to avoid performing the processing onboard, due to the computational and memory limitations of the onboard hardware. For low complexity and fast feedback, a bitstream-based (BB) approach was chosen. The original IR sequences are captured by UAV, and then BB and pixel-based (PB) features are computed. Thereafter, a feature selection process is applied and the selected features are mapped using support vector regression, to predict the quality scores of full reference metrics. The method is evaluated for the NR prediction of four image and one video quality metrics. A set of five UAV- and three ground-IR sequences are used for evaluation. The proposed NR method consistently achieves robust results for the different objective metrics tested (Spearman rank-order correlation coefficients ranging from 0.91 to 0.99). A comparison with estimations based on features from three NR models from the literature proves to be in favor of the proposed method.

Optical Flow Prediction for Blind and Non-Blind Video Error Concealment Using Deep Neural Networks

A novel optical flow prediction model using an adaptable deep neural network architecture for blind and non-blind error concealment of videos degraded by transmission loss is presented. The two-stream network model is trained by separating the horizontal and vertical motion fields which are passed through two similar parallel pipelines that include traditional convolutional (Conv) and convolutional long short-term memory (ConvLSTM) layers. The ConvLSTM layers extract temporally correlated motion information while the Conv layers correlate motion spatially. The optical flows used as input to the two-pipeline prediction network are obtained through a flow generation network that can be easily interchanged, increasing the adaptability of the overall end-to-end architecture. The performance of the proposed model is evaluated using real-world packet loss scenarios. Standard video quality metrics are used to compare frames reconstructed using predicted optical flows with those reconstructed using “ground-truth” flows obtained directly from the generator.

A multi-metric approach for block-level video quality assessment

Developing an objective video quality metric that accurately estimates perceived video quality is challenging. Developing a metric that can additionally be embedded in the rate distortion optimization process of a video codec can be even harder given that decisions have to be made locally. In this paper, we present a method for combining a number of existing state of the art objective video quality metrics at the coding block level by employing a fusion of local content features for deciding how to best utilize the chosen metrics. Our results indicate promising performance in terms of the correlation of the developed locally-acting quality metric with the overall perceived quality of the video.