Video Quality Experts Group Research Articles

Large-scale Research on Quality of Experience (QoE) Algorithms

The large variety of video data sources means variability not only in terms ofincluded content, but also in terms of quality. Therefore, quality assessment pro-vides an additional dimension. The paper describes a comprehensive evaluationexperiment on perceived video quality. Consequently, in summary, 19 200 000video frames will be processed. Given the scale of the experiment, it is setup on a computer cluster in order to accelerate the calculations significantly.This work on Quality of Experience (QoE) is synchronized with that conductedby the Video Quality Experts Group (VQEG), in particular the Joint EffortsGroup (JEG) – Hybrid group project.

The Relationship Among Video Quality, Screen Resolution, and Bit Rate

How much bandwidth is required for good quality video for a given screen resolution? Data acquired during two Video Quality Experts Group (VQEG) projects allow at least a partial answer to this question. This international subjective testing produced large amounts of mean opinion score (MOS) data for the screen resolutions QIF, CIF, VGA, and HD; for H.264 and similar modern codecs; and for many bit rates. Those data are assembled in the present report. For each screen resolution, MOS is plotted as a function of bit rate. A plot of all four data sets together shows the bit rate that would be required to achieve a given level of video quality for a given screen resolution. Relations among the four data sets are regular, suggesting that interpolation across screen resolutions might be reasonable. Based on these data, it would be reasonable to choose a bit rate, given a screen resolution; it would not be reasonable to choose a screen resolution given a bit rate.

55.3: Crosstalk Measurements of Shutter Glasses 3D Displays

AbstractCrosstalk is probably one of the main perceptual factors contributing to perceived image quality and visual comfort. The Video Quality Experts Group (VQEG) within their 3D video quality project is specifying a practical measurement procedure that will produce consistent results across laboratories. This paper is part of that effort. Two different method of measuring cross talk on shutter glasses stereo displays have been studied. One is based on time average luminance measurements and the other on temporal measurements. The results show that crosstalk is roughly 0.5% but that there are differences in the crosstalk between the two eyes in the shutter glasses.

Content-weighted video quality assessment using a three-component image model

Objective image and video quality measures play important roles in numerous image and video processing applications. In this work, we propose a new content-weighted method for full-reference (FR) video quality assessment using a three-component image model. Using the idea that different image regions have different perceptual significance relative to quality, we deploy a model that classifies image local regions according to their image gradient properties, then apply variable weights to structural similarity image index (SSIM) [and peak signal-to-noise ratio (PSNR)] scores according to region. A frame-based video quality assessment algorithm is thereby derived. Experimental results on the Video Quality Experts Group (VQEG) FR-TV Phase 1 test dataset show that the proposed algorithm outperforms existing video quality assessment methods.

Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos

There has recently been a great deal of interest in the development of algorithms that objectively measure the integrity of video signals. Since video signals are being delivered to human end users in an increasingly wide array of applications and products, it is important that automatic methods of video quality assessment (VQA) be available that can assist in controlling the quality of video being delivered to this critical audience. Naturally, the quality of motion representation in videos plays an important role in the perception of video quality, yet existing VQA algorithms make little direct use of motion information, thus limiting their effectiveness. We seek to ameliorate this by developing a general, spatio-spectrally localized multiscale framework for evaluating dynamic video fidelity that integrates both spatial and temporal (and spatio-temporal) aspects of distortion assessment. Video quality is evaluated not only in space and time, but also in space-time, by evaluating motion quality along computed motion trajectories. Using this framework, we develop a full reference VQA algorithm for which we coin the term the MOtion-based Video Integrity Evaluation index, or MOVIE index. It is found that the MOVIE index delivers VQA scores that correlate quite closely with human subjective judgment, using the Video Quality Expert Group (VQEG) FRTV Phase 1 database as a test bed. Indeed, the MOVIE index is found to be quite competitive with, and even outperform, algorithms developed and submitted to the VQEG FRTV Phase 1 study, as well as more recent VQA algorithms tested on this database.

VQeg validation and ITU standardization of objective perceptual video quality metrics [Standards in a Nutshell

For industry, the need to access accurate and reliable objective video metrics has become more pressing with the advent of new video applications and services such as mobile broadcasting, Internet video, and Internet Protocol television (IPTV). Industry-class objective quality- measurement models have a wide range of uses, including equipment testing (e.g., codec evaluation), transmission- planning and network-dimensioning tasks, head-end quality assurance, in- service network monitoring, and client-based quality measurement. The Video Quality Experts Group (VQEG) is the primary forum for validation testing of objective perceptual quality models. The work of VQEG has resulted in International Telecommunication Union (ITU) standardization of objective quality models designed for standard- definition television and for multimedia applications. This article reviews VQEG's work, paying particular attention to the group's approach to validation testing.

구조적 유사도 기반의 인간의 시각적 특성을 이용한 비디오 품질 측정 기준

Recently, the structural similarity (SSIM) index metric is proposed. In the present paper, a new framework, which is called visual SSIM (VSSIM), is proposed by incorporating crucial human factors into the SSIM. The human factors are foveation, luminance, frequency and motion information. The performance of VSSIM is evaluated by subjective quality test compliant with the Video Quality Expert Group (VQEG) multimedia group test plan. It shows that the visual SSIM is more correlated with the subjective quality result than the conventional SSIM.

The Evolution of Video Quality Measurement: From PSNR to Hybrid Metrics

This paper reviews the evolution of video quality measurement techniques and their current state of the art. We start with subjective experiments and then discuss the various types of objective metrics and their uses. We also introduce V-Factor, a hybrid metric using both transport- and bitstream information. Finally, we summarize the main standardization activities, such as the work of the video quality experts group (VQEG), and we take a look at emerging trends in quality measurement, including image preference, visual attention, and audiovisual quality.

Current Status and Trend of VQEG (Video Quality Experts Group)

Current Status and Trend of VQEG (Video Quality Experts Group)

Video quality assessment using a statistical model of human visual speed perception

Motion is one of the most important types of information contained in natural video, but direct use of motion information in the design of video quality assessment algorithms has not been deeply investigated. Here we propose to incorporate a recent model of human visual speed perception [Nat. Neurosci. 9, 578 (2006)] and model visual perception in an information communication framework. This allows us to estimate both the motion information content and the perceptual uncertainty in video signals. Improved video quality assessment algorithms are obtained by incorporating the model as spatiotemporal weighting factors, where the weight increases with the information content and decreases with the perceptual uncertainty. Consistent improvement over existing video quality assessment algorithms is observed in our validation with the video quality experts group Phase I test data set.

Subjective Multimedia Quality Assessment

The Video Quality Experts Group (VQEG) is preparing a programme of subjective multimedia quality tests. The results from these tests will be used to evaluate the performance of competing objective multimedia quality metrics. The reliability of the subjective test data is of great importance for VQEG's task. This paper provides an overview of VQEG's multimedia ad-hoc group. The work of this group will require subjective tests to be performed by laboratories located in Europe, Asia and North America. For VQEG's multimedia work to be successful, the subjective assessment methodology must be precisely defined and produce reliable and repeatable subjective quality data. Although international standards covering multimedia quality assessment methods are in force, there remains some uncertainty regarding the most effective approach to assessing the subjective quality of multimedia. A review of existing methods is provided. Two experiments are presented investigating the suitability of alternative subjective assessment methods (single-stimulus ACR and SAMVIQ). The results of these experiments are discussed within the context of the VQEG multimedia testing programme.

A Convolutional Neural Network Approach for Objective Video Quality Assessment

This paper describes an application of neural networks in the field of objective measurement method designed to automatically assess the perceived quality of digital videos. This challenging issue aims to emulate human judgment and to replace very complex and time consuming subjective quality assessment. Several metrics have been proposed in literature to tackle this issue. They are based on a general framework that combines different stages, each of them addressing complex problems. The ambition of this paper is not to present a global perfect quality metric but rather to focus on an original way to use neural networks in such a framework in the context of reduced reference (RR) quality metric. Especially, we point out the interest of such a tool for combining features and pooling them in order to compute quality scores. The proposed approach solves some problems inherent to objective metrics that should predict subjective quality score obtained using the single stimulus continuous quality evaluation (SSCQE) method. This latter has been adopted by video quality expert group (VQEG) in its recently finalized reduced referenced and no reference (RRNR-TV) test plan. The originality of such approach compared to previous attempts to use neural networks for quality assessment, relies on the use of a convolutional neural network (CNN) that allows a continuous time scoring of the video. Objective features are extracted on a frame-by-frame basis on both the reference and the distorted sequences; they are derived from a perceptual-based representation and integrated along the temporal axis using a time-delay neural network (TDNN). Experiments conducted on different MPEG-2 videos, with bit rates ranging 2-6 Mb/s, show the effectiveness of the proposed approach to get a plausible model of temporal pooling from the human vision system (HVS) point of view. More specifically, a linear correlation criteria, between objective and subjective scoring, up to 0.92 has been obtained on a set of typical TV videos.

Objective video quality assessment

We propose a new method for an objective measurement of video quality. By analyzing subjective scores of various video sequences, we find that the human visual system is particularly sensitive to degradation around edges. In other words, when edge areas of a video sequence are degraded, evaluators tend to give low quality scores to the video, even though the overall mean squared error is not large. Based on this observation, we propose an objective video quality measurement method based on degradation around edges. In the proposed method, we first apply an edge detection algorithm to videos and locate edge areas. Then, we measure degradation of those edge areas by computing mean squared errors and use it as a video quality metric after some postprocessing. Experiments show that the proposed method significantly outperforms the conventional peak signal-to-noise ratio (PSNR). This method was also independently evaluated by independent laboratory groups in the Video Quality Experts Group (VQEG) Phase 2 test. The method consistently provided good performances. As a result, the method was included in international recommendations for objective video quality measurement.

A new standardized method for objectively measuring video quality

The National Telecommunications and Information Administration (NTIA) General Model for estimating video quality and its associated calibration techniques were independently evaluated by the Video Quality Experts Group (VQEG) in their Phase II Full Reference Television (FR-TV) test. The NTIA General Model was the only video quality estimator that was in the top performing group for both the 525-line and 625-line video tests. As a result, the American National Standards Institute (ANSI) adopted the NTIA General Model and its associated calibration techniques as a North American Standard in 2003. The International Telecommunication Union (ITU) has also included the NTIA General Model as a normative method in two Draft Recommendations. This paper presents a description of the NTIA General Model and its associated calibration techniques. The independent test results from the VQEG FR-TV Phase II tests are summarized, as well as results from eleven other subjective data sets that were used to develop the method.

Visual Distortion Assessment With Emphasis on Spatially Transitional Regions

It is known that the human visual system (HVS) does not pay equal attention to each error and even region in judging picture quality. In this paper, we combine a perceptual model with an integrated detection of the spatially transitional regions in visual distortion evaluation to better match the HVS perception to visual quality. For decompressed images or video, the spatially transitional regions are the regions where major perceptually disturbing artefacts caused by edge impairments (mainly due to blurring and locations where the edge information is not adequately represented) and the presence of false edges (mainly due to blockiness and the presence of strong rippling effects of ringing) usually occur. Such regions are efficiently detected based on a single two-dimensional spatial high-pass filter in our work. Good correlation between the proposed method and the human perception has been demonstrated with the full set of 50-Hz video quality expert group test data.

Accuracy and cross-calibration of video quality metrics: new methods from ATIS/T1A1

Video quality metrics (VQMs) have often been evaluated and compared using simple measures of correlation to subjective mean opinion scores from panels of observers. However, this approach does not fully take into account the variability implicit in the observers. We present techniques for determining the statistical resolving power of a VQM, defined as the minimum change in the value of the metric for which subjective test scores show a significant change. Resolving power is taken as a measure of accuracy. These techniques have been applied to the video quality experts group (VQEG) data set and incorporated into the recent Alliance for Telecommunications Industry Solutions (ATIS) Committee T1A1 series of technical reports (TRs), which provide a comprehensive framework for characterizing and validating full-reference VQM. These approved TRs, while not standards, will enable the US telecommunications industry to incorporate VQMs into contracts and tariffs for compressed video distribution. New methods for assessing VQM accuracy and cross-calibrating VQMs are an integral part of the framework. These methods have been applied to two VQMs at this point: peak-signal-to-noise ratio and the version of Sarnoff's just noticeable difference metric (JNDmetrix ®) tested by VQEG (Rapporteur Q11/12 (VQEG): Final report from the VQEG on the validation of objective models of video quality assessment, June 2000). The framework is readily extensible to additional VQMs.

Video quality assessment based on structural distortion measurement

Objective image and video quality measures play important roles in a variety of image and video processing applications, such as compression, communication, printing, analysis, registration, restoration, enhancement and watermarking. Most proposed quality assessment approaches in the literature are error sensitivity-based methods. In this paper, we follow a new philosophy in designing image and video quality metrics, which uses structural distortion as an estimate of perceived visual distortion. A computationally efficient approach is developed for full-reference (FR) video quality assessment. The algorithm is tested on the video quality experts group Phase I FR-TV test data set.

PVQM – A perceptual video quality measure

Abstract Modern video coding systems such as ISO-MPEG1,2,4 exploit properties of the human visual system, to reduce the bit rate at which a video sequence is coded, given a certain required video quality. As a result, to the degree in which such exploitation is successful, accurate prediction of the quality of the output video of such systems, should also take the human visual system into account. In this paper, we propose a perceptual video quality system, that uses a linear combination of three indicators. The indicators are, the “edginess” of the luminance, the normalized color error and the temporal decorrelation. In the benchmark by the Video Quality Expert Group (VQEG), a combined ITU-T and ITU-R expert group, the model showed the highest variance weighted regression overall correlation of all models.

VQEG Evaluation of Objective Methods of Video Quality Assessment

This paper briefly discusses objective and subjective methods for video quality assessment by focusing on the current work plan and schedule of the Video Quality Experts Group (VQEG). Subjective assessment methods have been used reliably for many years to evaluate video image quality, yet, some issues arise with subjective assessment including the cost of conducting the evaluations and the fact that these methods cannot easily be used to monitor video image quality in realtime. Traditional analog objective measurements, while still necessary, are not sufficient to measure the quality of digitally compressed video systems. Thus, there is a need to develop new objective methods utilizing the characteristics of the human visual system. While several new objective methods have been developed, there is no internationally standardized method. The VQEG is comprised of experts from various fields and organizations working on the evaluation and standardization of methods for video quality assessment. The first and current task of VQEG is the evaluation of objective model performance against subjective results for video quality in the bit rate range of 768 kbits/sec to 50 Mbits/sec.