Objective Video Quality Metrics Research Articles

VQEG Column: New topics

Welcome to the fourth column on the ACM SIGMM Records from the Video Quality Experts Group (VQEG). During the last VQEG plenary meeting (14-18 Dec. 2020) various interesting discussions arose regarding new topics not addressed up to then by VQEG groups, which led to launching three new sub-projects and a new project related to: 1) clarifying the computation of spatial and temporal information (SI and TI), 2) including video quality metrics as metadata in compressed bitstreams, 3) Quality of Experience (QoE) metrics for live video streaming applications, and 4) providing guidelines on implementing objective video quality metrics to the video compression community. The following sections provide more details about these new activities and try to encourage interested readers to follow and get involved in any of them by subscribing to the corresponding reflectors.

VQEG column: VQEG meeting Dec. 2020 (virtual/online)

Welcome to the third column on the ACM SIGMM Records from the Video Quality Experts Group (VQEG). The last VQEG plenary meeting took place online from 14 to 18 December. Given the current circumstances, it was organized all online for the second time, with multiple sessions distributed over five to six hours each day allowing remote participation of people from different time zones. About 130 participants from 24 different countries registered to the meeting and could attend the several presentations and discussions that took place in all working groups. This column provides an overview of this meeting, while all the information, minutes, files (including the presented slides), and video recordings from the meeting are available online in the VQEG meeting website. As highlights of interest for the SIGMM community, apart from several interesting presentations of state-of-the-art works, relevant contributions to ITU recommendations related to multimedia quality assessment were reported from various groups (e.g., on adaptive bitrate streaming services, on subjective quality assessment of 360-degree videos, on statistical analysis of quality assessments, on gaming applications, etc.), the new group on quality assessment for health applications was launched, and an interesting session on 5G use cases took place, as well as a workshop dedicated to user testing during Covid-19. In addition, new efforts have been launched related to the research on quality metrics for live media streaming applications, and to provide guidelines on implementing objective video quality metrics (ahead of PSNR) to the video compression community. We encourage those readers interested in any of the activities going on in the working groups to check their websites and subscribe to the corresponding reflectors, to follow them and get involved.

Modeling and estimating the subjects\u2019 diversity of opinions in video quality assessment: a neural network based approach

Subjective experiments are considered the most reliable way to assess the perceived visual quality. However, observers’ opinions are characterized by large diversity: in fact, even the same observer is often not able to exactly repeat his first opinion when rating again a given stimulus. This makes the Mean Opinion Score (MOS) alone, in many cases, not sufficient to get accurate information about the perceived visual quality. To this aim, it is important to have a measure characterizing to what extent the observed or predicted MOS value is reliable and stable. For instance, the Standard deviation of the Opinions of the Subjects (SOS) could be considered as a measure of reliability when evaluating the quality subjectively. However, we are not aware of the existence of models or algorithms that allow to objectively predict how much diversity would be observed in subjects’ opinions in terms of SOS. In this work we observe, on the basis of a statistical analysis made on several subjective experiments, that the disagreement between the quality as measured by means of different objective video quality metrics (VQMs) can provide information on the diversity of the observers’ ratings on a given processed video sequence (PVS). In light of this observation we: i) propose and validate a model for the SOS observed in a subjective experiment; ii) design and train Neural Networks (NNs) that predict the average diversity that would be observed among the subjects’ ratings for a PVS starting from a set of VQMs values computed on such a PVS; iii) give insights into how the same NN based approach can be used to identify potential anomalies in the data collected in subjective experiments.

A low-complexity psychometric curve-fitting approach for the objective quality assessment of streamed game videos

The increasing popularity of video gaming competitions, the so called eSports, has contributed to the rise of a new type of end-user: the passive game video streaming (GVS) user. This user acts as a passive spectator of the gameplay rather than actively interacting with the content. This content, which is streamed over the Internet, can suffer from disturbing network and encoding impairments. Therefore, assessing the user’s perceived quality, i.e the Quality of Experience (QoE), in real-time becomes fundamental. For the case of natural video content, several approaches already exist that tackle the client-side real-time QoE evaluation. The intrinsically different expectations of the passive GVS user, however, call for new real-time quality models for these streaming services. Therefore, this paper presents a real-time Reduced-Reference (RR) quality assessment framework based on a low-complexity psychometric curve-fitting approach. The proposed solution selects the most relevant, low-complexity objective feature. Afterwards, the relationship between this feature and the ground-truth quality is modelled based on the psychometric perception of the human visual system (HVS). This approach is validated on a publicly available dataset of streamed game videos and is benchmarked against both subjective scores and objective models. As a side contribution, a thorough accuracy analysis of existing Objective Video Quality Metrics (OVQMs) applied to passive GVS is provided. Furthermore, this analysis has led to interesting insights on the accuracy of low-complexity client-based metrics as well as to the creation of a new Full-Reference (FR) objective metric for GVS, i.e. the Game Video Streaming Quality Metric (GVSQM).

AccAnn: A New Subjective Assessment Methodology for Measuring Acceptability and Annoyance of Quality of Experience

User expectations have a crucial impact on the levels of quality of experience (QoE) that they consider acceptable or satisfying. Measuring acceptability and annoyance has mainly been performed in separate or multi-step experiments without any control over participants’ expectations. This paper introduces a simple methodology to obtain the information about both of the entities in a single step and compares several data processing strategies useful for results interpretation. A specifically designed subjective experiment, conducted on compressed videos, has shown that the multi-step procedures could be replaced by our proposed single-step approach, regardless of the viewing conditions, while the novel approach is significantly preferred by observers for its low time requirements and higher intuitiveness. The test has simultaneously proven that user expectations can be altered by the instructions and it is, therefore, possible to simulate different user profiles regardless of the participants’ real habits. The acceptability/annoyance experimental results are also used to benchmark the state-of-the-art objective video quality metrics in predicting acceptability/annoyance of QoE. A case study on the determination of the threshold of acceptability/annoyance for objective quality metrics is conducted, which can be served as a guideline for video streaming service providers.

No-reference artifacts measurements based video quality metric

In multimedia delivery, perceived quality of video signal has a significant role in overall users Quality of Experience (QoE). Therefore, multimedia service providers must constantly measure and monitor perceived video quality, which is usually performed using no-reference (NR) video quality metrics. In this paper, a novel NR objective video quality metric named Artifacts Measurements Based Video Quality Metric (AMB-VQM) is proposed. In addition to artifacts measures computed by blocking (BL), packet-loss (PL), and freezing (FZ) artifacts detection algorithms, the metric incorporates artifacts masking based on spatial and temporal video content complexity. Furthermore, a newly created FERIT-RTRK-2 video quality database, which contains 486 Full HD test video sequences impaired by video compression (MPEG-2, H.264/AVC, and H.265), packet-loss, frame freezing, and combinations of these procedures, is presented in this paper. FERIT-RTRK-2 database is publicly available for scientific community at http://www.rt-rk.com/other/VideoDBReadme.html. Additionally, newly created user-friendly subjective video quality assessment tool with graphical user interface, which can be used for conducting of subjective video quality experiments, is presented. In experimental part the performance of the proposed AMB-VQM is compared to this of 10 other objective video quality metrics (PSNR, SSIM, VSNR, PSNRHVS, PSNRHVSM, VIFP, ViS3, ST-MAD, BRISQUE, VIIDEO) using distorted video sequences from four different video quality databases: CSIQ, LIVE, FERIT-RTRK and FERIT-RTRK-2. The results show that AMB-VQM achieves high performance when predicting the video quality for videos distorted in different manners. AMB-VQM results outperform the results of most of the analyzed popular and very often used video quality metrics.

A multi-metric approach for block-level video quality assessment

Developing an objective video quality metric that accurately estimates perceived video quality is challenging. Developing a metric that can additionally be embedded in the rate distortion optimization process of a video codec can be even harder given that decisions have to be made locally. In this paper, we present a method for combining a number of existing state of the art objective video quality metrics at the coding block level by employing a fusion of local content features for deciding how to best utilize the chosen metrics. Our results indicate promising performance in terms of the correlation of the developed locally-acting quality metric with the overall perceived quality of the video.

Overview of full-reference video quality metrics and their performance evaluations for videoconferencing application

With the development of video services available on various devices (mobile, PC, tablet, television, etc.), the end user experiences and requirements have changed. Objective video quality assessment tools and metrics are increasingly complex and adapted to different types of service context. Compared with other reviews of quality metrics, the full-reference (FR) video quality metrics reviewed (particularly, ViS3, SSIMplus, video multimethod assessment fusion, and open perceptual video quality metric) are more recent and have not been compared with each other yet. We compare and evaluate the performance of 10 FR metrics with respect to their accuracy in predicting the subjective perceived quality for the videoconferencing application. Emphasis is given to the metrics evaluating the visual quality (from human perception). A detailed statistical study is carried out on four subjective databases: Ecole polytechnique federale de Lausanne database, live mobile database, and two Orange Lab databases, with a large sample of distortion types (transmission errors, encoding bit rates, etc.).

Performance analysis of a video quality ruler methodology for subjective quality assessment

Objective video quality metrics are designed to be as reliable as the subjective quality assessments on which they are calibrated and validated. However, existing standard methodologies for subjective video quality assessment provide low reliable results for some conditions. We investigate whether an extension of the quality ruler experimental methodology, originally defined for images and shown to be more reliable than, e.g., standard single stimulus (SS) methods, can be adapted to reliably assess the quality of videos. The video quality ruler methodology allows subjects to assess video quality using a set of reference anchor images (the ruler), spanning a wide range of quality altogether, but closely spaced in function of quality one from the other. Subjects are asked to compare the quality of the displayed test video with the quality of these anchor images, displayed on a tablet, and indicate which of the reference images matches in quality the test video. As a result, the video quality assessment task is reduced to a set of visual comparisons between video and reference image quality. We describe how to adapt the original quality ruler methodology to video quality assessment, and we compare the proposed methodology with two other, widely used experimental methodologies: the single stimulus (SS) and the double stimulus (DS) method. Our results show that video quality ruler is a reliable method to assess video quality according to a multitude of criteria.

Choosing Encoding Parameters for High-Dynamic Range Streaming

This paper provides data and analysis on the impact of various high-dynamic range (HDR) encoding options on video quality and bitrate. Specifically, this paper examines permutations of perceptual quantizer (PQ), hybrid-log gamma (HLG), <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Y}^\prime ~C_{B}C_{R}$ </tex-math></inline-formula> , IC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$_{T}C_{P}$ </tex-math></inline-formula> , encoded resolution, and bitrate using the open-source encoder <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times 265$ </tex-math></inline-formula> . Video distortion for each permutation is measured using publicly available nonproprietary objective video quality metrics. Today, we have two different ways of mapping light to HDR code values: PQ and HLG. We also have the option of representing color and luminance as either <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Y}^\prime ~C_{B}C_{R}$ </tex-math></inline-formula> or the newer alternative IC <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$_{T}C_{P}$ </tex-math></inline-formula> . For HDR adaptive bitrate streaming, we need to choose which compression levels and encoding resolutions to use when we create adaptation sets; but we do not yet have a complete understanding of how resolution and compression could interact to alter the highlights and deep darks that make HDR so visually potent. When viewed altogether, the many permutations of HDR encoding can make designing an HDR streaming service complicated and uncertain. This paper provides practical data and analysis that can be used to make designing HDR streaming services easier and more predictable. <disp-quote xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> This paper provides practical data and analysis that can be used to make designing HDR streaming services easier and more predictable. </disp-quote>

Quality assessment for wireless capsule endoscopy videos compressed via HEVC: From diagnostic quality to visual perception

Maintaining the quality of medical images and videos is an essential part of the e-services provided by the healthcare sector. The convergence of modern communication systems and the healthcare industry necessitates the provision of better quality of service and experience by the service provider. Recent inclusion and standardization of the high efficiency video coder (HEVC) has made it possible for medical data to be compressed and transmitted over wireless networks with minimal compromise of the quality. Quality evaluation and assessment of these medical videos transmitted over wireless networks is another important research area that requires further exploration and attention. In this paper, we have conducted an in-depth study of video quality assessment for compressed wireless capsule endoscopy (WCE) videos. Our study includes the performance evaluation of several state-of-the-art objective video quality metrics in terms of determining the quality of compressed WCE videos. Subjective video quality experiments were conducted with the assistance of experts and non-experts in order to predict the diagnostic and visual quality of these medical videos, respectively. The evaluation of the metrics is based on three major performance metrics that include, correlation between the subjective and objective scores, relative statistical performance and computation time. Results show that the metrics information fidelity criterion (IFC), visual information fidelity-(VIF) and especially pixel based VIF stand out as best performing metrics. Furthermore, our paper reports the performance of HEVC compression on medical videos and according to the results, it performs optimally in preserving the diagnostic and visual quality of WCE videos at Quantization Parameter (QP) values of up to 35 and 37 respectively.

Disparity map estimation and view synthesis using temporally adaptive triangular meshes

This paper presents a new method for spatio-temporally coherent disparity map estimation and view interpolation for multiview linear camera arrays based on 2D domain triangulation. In the first frame of the sequence, a 3D mesh is computed for each camera, leading to a spatially coherent view interpolation. For the remaining frames of the sequence, a new scheme is proposed to update the 3D mesh dynamically, by moving, deleting and inserting vertices based on the optical flow and the previously computed disparity map. With this approach it is possible to relate triangles of the mesh across time, and a combination of Hidden Markov Models (HMMs) applied to time-persistent triangles with the Kalman Filter applied to the vertices produces temporally coherent disparity maps and interpolated views. Experimental results indicate that our approach was able to generate visually coherent in-between interpolated views for challenging, real-world videos with natural lighting and camera movement. Also, quantitative evaluations using objective video quality metrics show that our interpolated videos are typically better than competitive approaches.

Do gaze disruptions indicate the perceived quality of non-uniformly coded natural scenes?

Subjective video quality evaluation techniques usually involves a subject voluntarily attending to several regions in a video in order to scrutinize its quality. These techniques often tend to over-estimate the visual thresholds and in cases of non-uniform quality/coding, subjects mostly perceive the underlying video content in worse quality. This occurs due to a process known as Attentional Modulation occurring in the higher visual cortical areas V2 to V4. Examining disruptions in free viewing gaze patterns on the other hand, are said to be a more naturalistic method to measure the perceived video quality in such cases. To explore the feasibility of such a gaze disruption based quality metric, we examine the dependency between the two indicators: gaze disruptions and perceived subjective quality, obtained from a carefully controlled subjective test. By the examination of eye-tracking data and subsequent statistical analysis of difference opinion scores given by users, we are able to see that disruptions are indeed excellent indicators of perceived quality achieving a correlation 0.84. Several state of the art objective video quality metrics like SSIM, VIFp, VQM and PSNR-HVS (designed mostly for evaluation of uniform-quality) on the other hand, only produce a correlation ranging from 0.01 to 0.10. We conclude therefore that gaze disruptions may be used as excellent natural indicators of perceived quality in cases where quality is non-uniform, and may serve as new ground truth indicators for objective algorithms like Scanpath disruption(prediction) metrics that measure video quality in a more real-world like(naturalistic) manner.

Study of Saliency in Objective Video Quality Assessment.

Reliably predicting video quality as perceived by humans remains challenging and is of high practical relevance. A significant research trend is to investigate visual saliency and its implications for video quality assessment. Fundamental problems regarding how to acquire reliable eye-tracking data for the purpose of video quality research and how saliency should be incorporated in objective video quality metrics (VQMs) are largely unsolved. In this paper, we propose a refined methodology for reliably collecting eye-tracking data, which essentially eliminates bias induced by each subject having to view multiple variations of the same scene in a conventional experiment. We performed a large-scale eye-tracking experiment that involved 160 human observers and 160 video stimuli distorted with different distortion types at various degradation levels. The measured saliency was integrated into several best known VQMs in the literature. With the assurance of the reliability of the saliency data, we thoroughly assessed the capabilities of saliency in improving the performance of VQMs, and devised a novel approach for optimal use of saliency in VQMs. We also evaluated to what extent the state-of-the-art computational saliency models can improve VQMs in comparison to the improvement achieved by using "ground truth" eye-tracking data. The eye-tracking database is made publicly available to the research community.

Mathematical Bottom-to-Up Approach in Video Quality Estimation Based on PHY and MAC Parameters

This paper presents an extension of research scope of the relationship between quality of service (QoS) and quality of experience (QoE), which is based on contribution of lower open system interconnection layers, such as physical and media access control, in overall QoS/QoE paradigm. Degradations that inevitably occur in transmission channel are an important reason for appearance of QoS distortion and therefore low values of video quality. Various channel quality indicators (CQIs) relating to domain of communication channel can be used for notification of different interferences in the channel. In order to extend the relationship between QoS and QoE to the transmission channel, the paper proposed the mathematical model that used CQI to estimate values of QoS indicators by using statistical analysis. The model was also expanded by objective video quality metrics in order to evaluate QoE. Verification of the model was checked by experimental method with consideration of Internet protocol television (IPTV) service delivery in Digital Subscriber Line network.

An Innovative No-Reference Metric for Real-Time 3D Stereoscopic Video Quality Assessment

3D video technologies have been widely adopted by video service providers and consumer electronics stakeholders due to their potential of offering an immersive user experience. In case of 3D video streaming, the dynamic network conditions are the bottleneck that limits the content delivery at good perceived quality levels and an effective solution is to employ advanced 3D video adaptation schemes. Accurate real-time objective 3D video quality assessment is a critical factor in adaptive decision making. State-of-the-art objective 3D video quality assessment methods are in general reference-based and require the availability of the original 3D video sequence, which makes them not suitable for real-time applications. This paper proposes the extended no reference objective video quality metric (eNVQM), an innovative metric for real-time 3D video quality assessment. eNVQM estimates the 3D video quality by taking as the input parameters network packet loss, video transmission bitrate, and frame rate. Based on extensive subjective tests, eNVQM models the impact of network packet loss on 3D video at different bitrates and frame rates on the perceived stereoscopic 3D video quality. The performance of eNVQM is investigated by comparing its results with two state-of-the-art objective video quality metrics: 1) structural similarity index and 2) video quality metric. Results show that eNVQM maintains similar accuracy level in estimating 3D video quality with the alternative reference-based metrics.

VQAMap: A Novel Mechanism for Mapping Objective Video Quality Metrics to Subjective MOS Scale

The reliable estimation of video quality has become increasingly important with the proliferation of online video services and users becoming more quality aware. A multitude of objective video quality assessment (VQA) metrics with various performance and complexity have been proposed. However, their applicability in real-world scenarios is limited by the lack of clear interpretations of how the metric values reflect the subjective user-perceived video quality. This paper proposes a novel mechanism called VQAMap, that uses data from public VQA databases and enables to automatically create generic rules for mapping the values of objective VQA metrics to the subjective MOS scale (i.e., 1–bad, 2–poor, 3–fair, 4–good, and 5–excellent). An extensive evaluation study of VQAMap was conducted using data from three public VQA databases, considering six objective VQA metrics. The results analysis has shown that VQAMap provides mapping rules with quality estimation accuracy as high as 95%, while the variation in performance being caused by the varying accuracy of the different objective metrics.

Low-Complexity and Context-Aware Estimation of Spatial and Temporal Activity Parameters for Automotive Camera Rate Control

Rate control in video compression adjusts the encoding parameters to reach a certain target bitrate for the encoded video. State-of-the-art rate controllers for hybrid video coding typically employ content-dependent video bitrate models and video quality metrics (VQMs). To capture the content characteristics, temporal and spatial video activity measures are determined from the raw video using computationally complex algorithms that require access to the uncompressed source video. In automotive deployments, however, full access to the uncompressed source video and the internal functions of video encoders is typically not possible. As a remedy, in this paper, we present a low-complexity approach to estimate the temporal activity (TA) and spatial activity (SA) measures for videos that are captured by a front-facing camera of a vehicle, based on the context information of the vehicle. To this end, we exploit information about the dynamics of the vehicle and other vehicles in the field-of-view of the front-facing camera. We apply the estimated TA and SA values to a video bitrate model and an objective VQM and use these models to solve the rate control problem to determine the optimal encoding settings for given bitrate constraints. The proposed low-complexity solution offers a similar accuracy in achieving rate constraints and similar perceptual quality characteristics as a solution that uses the computed TA and SA values, with the advantage that no access to the uncompressed source video stream or the internal functions of the video encoder is required.

Applicability of Existing Objective Metrics of Perceptual Quality for Adaptive Video Streaming

Objective video quality metrics are designed to estimate thequality of experience of the end user. However, these objectivemetrics are usually validated with video streams degraded undercommon dist ...

Design of Objective Video Quality Metrics Using Spatial and Temporal Informations

Digital videos are subject to several types of distortions, which may occur during the processes of acquisition, processing, compression, storage and transmission of videos, resulting in loss of visual quality. Video quality assessment is important to set system parameters and propose the reduction of degradation, to improve the systems quality. The most appropriate way to assess the visual quality of a video sequence is subjective assessment. This study proposes new metrics for assessment of videos. The proposed metrics that uses spatial and temporal information to evaluate videos are more closely to the subjective assessment. For the evaluation of videos the BD-TPW-SSIM and TP-VQI metric obtained the best result for wireless degradation. For MPEG-2 encoded videos, T-PW-SSIM e T-MOVIE was the best, and IP systems the metric with the best result was the TPW-SSIM. Finally, the metric T-VimSSIM was the best for H.264 encoding.