Video Quality Experts Group Research Articles

VQEG Column: VQEG Meeting December 2022

This column provides an overview of the last Video Quality Experts Group (VQEG) plenary meeting, which took place from 12 to 16 December 2022. Around 100 participants from 21 different countries around the world registered for the meeting that was organized online by Brightcove (United Kingdom). During the five days, there were more than 40 presentations and discussions among researchers working on topics related to the projects ongoing within VQEG. All the related information, minutes, and files from the meeting are available online on the VQEG meeting website, and video recordings of the meeting are available on Youtube.

VQEG Column: VQEG Meeting May 2022

Welcome to this new column on the ACM SIGMM Records from the Video Quality Experts Group (VQEG), which will provide an overview of the last VQEG plenary meeting that took place from 9 to 13 May 2022. It was organized by INSA Rennes (France), and it was the first face-to-face meeting after the series of online meetings due to the Covid-19 pandemic. Remote attendance was also offered, which made possible that around 100 participants, from 17 different countries, attended the meeting (more than 30 of them attended in person). During the meeting, more than 40 presentations were provided, and interesting discussion took place. All the related information, minutes, and files from the meeting are available online in the VQEG meeting website, and video recordings of the meeting are available in Youtube.

Subjective Evaluation of Visual Quality and Simulator Sickness of Short 360$^\circ$ Videos: ITU-T Rec. P.919

Recently an impressive development in immersive technologies, such as Augmented Reality (AR), Virtual Reality (VR) and 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> video, has been witnessed. However, methods for quality assessment have not been keeping up. This paper studies quality assessment of 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> video from the cross-lab tests (involving ten laboratories and more than 300 participants) carried out by the Immersive Media Group (IMG) of the Video Quality Experts Group (VQEG). These tests were addressed to assess and validate subjective evaluation methodologies for 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> video. Audiovisual quality, simulator sickness symptoms, and exploration behavior were evaluated with short (from 10 seconds to 30 seconds) 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> sequences. The following factors&#x2019; influences were also analyzed: assessment methodology, sequence duration, Head-Mounted Display (HMD) device, uniform and non-uniform coding degradations, and simulator sickness assessment methods. The obtained results have demonstrated the validity of Absolute Category Rating (ACR) and Degradation Category Rating (DCR) for subjective tests with 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> videos, the possibility of using 10-second videos (with or without audio) when addressing quality evaluation of coding artifacts, as well as any commercial HMD (satisfying minimum requirements). Also, more efficient methods than the long Simulator Sickness Questionnaire (SSQ) have been proposed to evaluate related symptoms with 360<inline-formula><tex-math notation="LaTeX">${^\circ }$</tex-math></inline-formula> videos. These results have been instrumental for the development of the ITU-T Recommendation P.919. Finally, the annotated dataset from the tests is made publicly available for the research community.

VQEG Column: VQEG Meeting Jun. 2021 (virtual/online)

Welcome to the fifth column on the ACM SIGMM Records from the Video Quality Experts Group (VQEG). The last VQEG plenary meeting took place online from 7 to 11 June 2021. As the previous meeting celebrated in December 2020, it was organized online (this time by Kingston University) with multiple sessions spread over five days, allowing remote participation of people from 22 different countries of America, Asia, and Europe. More than 100 participants registered to the meeting and they could attend the 40 presentations and several discussions that took place in all working groups. This column provides an overview of the recently completed VQEG plenary meeting, while all the information, minutes and files (including the presented slides) from the meeting are available online in the VQEG meeting website.

VQEG Column: New topics

Welcome to the fourth column on the ACM SIGMM Records from the Video Quality Experts Group (VQEG). During the last VQEG plenary meeting (14-18 Dec. 2020) various interesting discussions arose regarding new topics not addressed up to then by VQEG groups, which led to launching three new sub-projects and a new project related to: 1) clarifying the computation of spatial and temporal information (SI and TI), 2) including video quality metrics as metadata in compressed bitstreams, 3) Quality of Experience (QoE) metrics for live video streaming applications, and 4) providing guidelines on implementing objective video quality metrics to the video compression community. The following sections provide more details about these new activities and try to encourage interested readers to follow and get involved in any of them by subscribing to the corresponding reflectors.

VQEG column: VQEG meeting Dec. 2020 (virtual/online)

Welcome to the third column on the ACM SIGMM Records from the Video Quality Experts Group (VQEG). The last VQEG plenary meeting took place online from 14 to 18 December. Given the current circumstances, it was organized all online for the second time, with multiple sessions distributed over five to six hours each day allowing remote participation of people from different time zones. About 130 participants from 24 different countries registered to the meeting and could attend the several presentations and discussions that took place in all working groups. This column provides an overview of this meeting, while all the information, minutes, files (including the presented slides), and video recordings from the meeting are available online in the VQEG meeting website. As highlights of interest for the SIGMM community, apart from several interesting presentations of state-of-the-art works, relevant contributions to ITU recommendations related to multimedia quality assessment were reported from various groups (e.g., on adaptive bitrate streaming services, on subjective quality assessment of 360-degree videos, on statistical analysis of quality assessments, on gaming applications, etc.), the new group on quality assessment for health applications was launched, and an interesting session on 5G use cases took place, as well as a workshop dedicated to user testing during Covid-19. In addition, new efforts have been launched related to the research on quality metrics for live media streaming applications, and to provide guidelines on implementing objective video quality metrics (ahead of PSNR) to the video compression community. We encourage those readers interested in any of the activities going on in the working groups to check their websites and subscribe to the corresponding reflectors, to follow them and get involved.

VQEG column: Recent contributions to ITU recommendations

Welcome to the second column on the ACM SIGMM Records from the Video Quality Experts Group (VQEG). VQEG plays a major role in research and the development of standards on video quality and this column presents examples of recent contributions to International Telecommunication Union (ITU) recommendations, as well as ongoing contributions to recommendations to come in the near future. In addition, the formation of a new group within VQEG addressing Quality Assessment for Health Applications (QAH) has been announced.

Standards column: VQEG

The Video Quality Experts Group (VQEG) was born from a need to bring together experts in subjective video quality assessment and objective quality measurement. The first VQEG meeting, held in Turin in 1997, was attended by a small group of experts drawn from ITU-T and ITU-R Study Groups. VQEG was first grounded in basic subjective methodology and objective tool development/verification for video quality assessment such that the industry could be moved forward with standardization and implementation. At the beginning it was focused around measuring the perceived video quality since the distribution path for video and audio were limited and known. Over the last 20 years from the formation of VQEG the ecosystem has changed dramatically and thus so must the work. Multimedia is now pervasive on all devices and methods of distribution from broadcast to cellular data networks. This shift has the expertise within VQEG to move from the visual (no-audio) quality of video to Quality of Experience (QoE). The march forward of technologies means that VQEG needs to react and be on the leading edge of developing, defining and deploying methods and tools that help address these new technologies and move the industry forward. This also means that we need to embrace both qualitative and quantitative ways of defining these new spaces and terms. Taking a holistic approach to QoE will enable VQEG to drive forward and faster with unprecedented collaboration and execution. VQEG is open to all interested from industry, academia, government organizations and Standard-Developing Organizations (SDOs). There are no fees involved, no membership applications and no invitations are needed to participate in VQEG activities. Subscription to the main VQEG email list (ituvidq@its.bldrdoc.gov) constitutes membership in VQEG. VQEG conducts work via discussions over email reflectors, regularly scheduled conference calls and, in general, two face-to-face meetings per year. There are currently more than 500 people registered across 11 email reflectors, including a main reflector for general announcements relevant to the entire group, and different project reflectors dedicated to technical discussions of specific projects. A LinkedIn group exists as well.

Open Software Framework for Collaborative Development of No Reference Image and Video Quality Metrics

This paper describes ongoing work within the video quality experts group (VQEG) to develop no-reference (NR) audiovisual video quality analysis (VQA) metrics. VQEG provides an open forum that encourages knowledge sharing and collaboration. The VQEG no-reference Metric (NORM) group’s goal is to develop open-source NR-VQA metrics that meet industry requirements for scope, accuracy, and capability. This paper presents industry specifications from discussions at VQEG face-to-face meetings among industry, academic, and government participants. This paper also announces an open software framework for collaborative development of NR image quality Analysis (IQA) and VQA metrics &lt;ext-link ext-link-type="url" xlink:href="https://github.com/NTIA/NRMetricFramework"&gt;&lt;https://github.com/NTIA/NRMetricFramework&gt;&lt;/ext-link&gt;. This framework includes the support tools necessary to begin research and avoid common mistakes. VQEG’s goal is to produce a series of NR-VQA metrics with progressively improving scope and accuracy. This work draws upon and enables IQA metric research, as both use the human visual system to analyze the quality of audiovisual media on modern displays. Readers are invited to participate.

Multi-Model Standard for Bitstream-, Pixel-Based and Hybrid Video Quality Assessment of UHD/4K: ITU-T P.1204

The paper presents a series of three new video quality model standards for the assessment of sequences of up to UHD/4K resolution. They were developed in a competition within the International Telecommunication Union (ITU-T), Study Group 12, in collaboration with the Video Quality Experts Group (VQEG), over a period of more than two years. A large video quality test set with a total of 26 individual databases was created, with 13 used for training and 13 for validation and selection of the winning models. For each database, video quality laboratory tests were run with at least 24 subjects each. The 5-point Absolute Category Rating (ACR) scale was used for rating, calculating Mean Opinion Scores (MOS) as ground-truth. To represent today's commonly applied HTTP-based adaptive streaming context, the test sequences comprise a variety of encoding settings, bitrates, resolutions and framerates for the three codecs H.264/AVC, H.265/HEVC and VP9, applied to a wide range of source sequences of around 8 s duration. Processing was carried out with an FFmpeg-based processing chain developed specifically for the competition, and via upload and encoding through exemplary online streaming services. The resulting data represents the largest, lab-test-based dataset used for video quality model development to date, with a total of around 5,000 test sequences. The paper addresses the three models ultimately standardized in the P.1204 Recommendation series, resulting in different model types and for different applications: (i) Rec. P.1204.3, no-reference bitstream-based, with access to encoded bitstream information; (ii) P.1204.4, pixel-based, using information from the reference and the processed video, and (iii) P.1204.5, no-reference hybrid, using both bitstream-and pixel-information without knowledge of the reference. The paper outlines the development process and provides holistic details about the statistical evaluation, test databases, model algorithms and validation results, as well as a performance comparison with state-of-the-art models.

A common framework for the evaluation of psychophysiological visual quality assessment

The assessment of perceived quality based on psychophysiological methods recently gained attraction as it potentially overcomes certain flaws of psychophysical approaches. Although studies report promising results, it is not possible to arrive at decisive and comparable conclusions that recommend the use of one or another method for a specific application or research question. The video quality expert group started a project on psychophysiological quality assessment to study these novel approaches and to develop a test plan that enables more systematic research. This test plan comprises of a specifically designed set of quality annotated video sequences, suggestions for psychophysiological methods to be studied in quality assessment, and recommendations for the documentation and publications of test results. The test plan is presented in this article.

Interpreting MOS scores, when can users see a difference? Understanding user experience differences for photo quality

The use of no-reference image quality evaluation tools that produce MOS scores, like the VIQET tool which was released by the Video Quality Expert Group, raises the question of whether the produced MOS differences between images correspond to noticeable differences in quality by the consumers. In this work, we attempted to approximate the minimum MOS difference that is required in order for people to be able to distinguish between a higher and a lower quality image under realistic conditions that are commonly encountered in the current consumer space. 91 people participated in a subjective just-noticeable-differences study across three countries that used non-simulated image stimuli, produced and evaluated through crowd sourcing for the validation of the VIQET no-reference image quality tool. The image dataset consisted of 15 different scenes belonging to three different scene types, with a total of 210 different image pairs being used. After evaluating the quality of the collected data, a logistic regression analysis approach was employed in order to estimate the minimum MOS difference required between two images in order for a given percentage of people to be able to detect the higher quality image.

Performance evaluation of different techniques to estimate subjective quality in live video streaming applications over LTE-Advance mobile networks

Current mobile service providers are offering Gigabit Internet access over LTE-Advanced networks. Traditional services, such as live video streaming, over wired networks are feasible on these networks. However different aspects should be taken into account due to the fast changing network conditions as well as the constrained resources of the mobile phones, in order to provide a good subjective video quality in terms of Mean Opinion Score (MOS). Our goal is to estimate and predict this subjective metric without information or reference from the original video, known as Non Reference approach. This approach is important for the Service Provider from a practical point of view, because it can keep the customer satisfaction at good levels. We analyze different estimation techniques running over a set of monitored variables throughout the whole steaming system, from the streaming server to the mobile phone. We have gathered variables related to bit stream, basic video quality metrics as well as Quality of Services variables. These variables are used to estimate MOS in a reliable and robust way. We compare three techniques such as Artificial Neural Networks (ANN), Factor Analysis (FA) and Multinomial Linear Regression, at different time scales and with Full Reference and Non Reference approaches. We carry out a performance evaluation of these techniques, concluding that the behavior of MOS estimation based on FA is more accurate, unless we had a lossless scenario related to Guaranteed Bit Rate services, where ANN performs better. The subjective video quality has been evaluated through surveys. Finally, we evaluate the accuracy of the estimated MOS against well known publicly available video quality algorithms following the recommendations given by Video Quality Experts Group (VQEG).

Monitoring of audio visual quality by key indicators

Over 10 billion hours of video are watched online every month. Together with high definition television broadcasting and the rise in high quality video on demand, this makes quality assessment a key task in the global multimedia market. Automating quality checking is currently based on finding major audiovisual artefacts. The Monitoring Of Audio Visual quality by key Indicators (MOAVI) subgroup of the Video Quality Experts Group (VQEG) is an open collaborative project for developing No-Reference models for monitoring audiovisual service quality. The purpose of this paper is to report on the development of the audiovisual part of this project, which includes the detection of muting, clipping and lip synchronization (also known as lip sync) artefacts.

Assessing Perceptual Video Quality in Wimax Networks

This paper presents an approach for assessing the perceptual quality of wireless video networking applications transmitted via WiMAX, the Worldwide Interoperability for Microwave Access air interface standard. The Video Quality Model developed by Information Administration’s Video Quality Expert Group is used to benchmark perceptual video quality.

Assessing Perceptual Video Quality in Wimax Networks

This paper presents an approach for assessing the perceptual quality of wireless video networking applications transmitted via WiMAX, the Worldwide Interoperability for Microwave Access air interface standard. The Video Quality Model developed by Information Administration’s Video Quality Expert Group is used to benchmark perceptual video quality.

A Perception-Based Hybrid Model for Video Quality Assessment

It is known that the human visual system (HVS) employs independent processes (distortion detection and artifact perception—also often referred to as near-threshold and suprathreshold distortion perception) to assess video quality for various distortion levels. Visual masking effects also play an important role in video distortion perception, especially within spatial and temporal textures. In this paper, a novel perception-based hybrid model for video quality assessment is presented. This simulates the HVS perception process by adaptively combining noticeable distortion and blurring artifacts using an enhanced nonlinear model. Noticeable distortion is defined by thresholding absolute differences using spatial and temporal tolerance maps that characterize texture masking effects, and this makes a significant contribution to quality assessment when the quality of the distorted video is similar to that of the original video. Characterization of blurring artifacts, estimated by computing high frequency energy variations and weighted with motion speed, is found to further improve metric performance. This is especially true for low quality cases. All stages of our model exploit the orientation selectivity and shift invariance properties of the dual-tree complex wavelet transform. This not only helps to improve the performance but also offers the potential for new low complexity in-loop application. Our approach is evaluated on both the Video Quality Experts Group (VQEG) full reference television Phase I and the Laboratory for Image and Video Engineering (LIVE) video databases. The resulting overall performance is superior to the existing metrics, exhibiting statistically better or equivalent performance with significantly lower complexity.

Advances in video quality estimation models: An Overview

In recent years many advances and standardizations have been performed in the field of video quality estimation. This paper presents an overview of these recent advances, explaining the main ideas and contributions of the video quality estimation models. The projects of VQEG (Video Quality Experts Group) are discussed, along with the recent standards approved by ITU (International Telecommunication Union), as well as other proposed models by different group of authors. Finally, the future work in this area is outlined.

Quaternion Model of Fast Video Quality Assessment Based on Structural Similarity Normalization

Video quality evaluation methods have been widely studied because of an increasing need in variety of video processing applications, such as compression, analysis, communication, enhancement and restoration. The quaternion models are also widely used to measure image or video quality. In this paper, we proposed a new quaternion model which mainly describes the contour feature, surface feature and temporal information of the video. We use structure similarity comparison to normalize four quaternion parts respectively, because each part of the quaternion use different metric. Structure similarity comparison is also used to measure the difference between reference videos and distortion videos. The results of experiments show that the new method has good correlation with perceived video quality when tested on the video quality experts group (VQEG) Phase I FR-TV test data set.

Quality assessment of perceptual color video based on a top-down framework and quaternion

Objective video quality assessment is of great importance in a variety of video processing applications. Most existing video quality metrics either focus primarily on capturing spatial artifacts in the video signal, or are designed to assess only grayscale video thereby ignoring important chrominance information. In this paper, on the basis of the top-down visual analysis of cognitive understanding and video features, we propose and develop a novel full-reference perceptual video assessment technique that accepts visual information inputs in the form of a quaternion consisting of contour, color and temporal information. Because of the more important role of chrominance information in the "border-to-surface" mechanism at early stages of cognitive visual processing, our new metric takes into account the chrominance information rather than the luminance information utilized in conventional video quality assessment. Our perceptual quaternion model employs singular value decomposition (SVD) and utilizes the human visual psychological features for SVD block weighting to better reflect perceptual focus and interest. Our major contributions include: a new perceptual quaternion that takes chrominance as one spatial feature, and temporal information to model motion or changes across adjacent frames; a three-level video quality measure to reflect visual psychology; and the two weighting methods based on entropy and frame correlation. Our experimental validation on the video quality experts' group (VQEG) Phase I FR-TV test dataset demonstrated that our new assessment metric outperforms PSNR, SSIM, PVQM (P8) and has high correlation with perceived video quality.