Full-reference Video Quality Metrics Research Articles

QoS monitoring in real-time streaming overlays based on lock-free data structures

Peer-to-peer streaming is a well-known technology for the large-scale distribution of real-time audio/video contents. Delay requirements are very strict in interactive real-time scenarios (such as synchronous distance learning), where playback lag should be of the order of seconds. Playback continuity is another key aspect in these cases: in presence of peer churning and network congestion, a peer-to-peer overlay should quickly rearrange connections among receiving nodes to avoid freezing phenomena that may compromise audio/video understanding. For this reason, we designed a QoS monitoring algorithm that quickly detects broken or congested links: each receiving node is able to independently decide whether it should switch to a secondary sending node, called “fallback node”. The architecture takes advantage of a multithreaded design based on lock-free data structures, which improve the performance by avoiding synchronization among threads. We will show the good responsiveness of the proposed approach on machines with different computational capabilities: measured times prove both departures of nodes and QoS degradations are promptly detected and clients can quickly restore a stream reception. According to PSNR and SSIM, two well-known full-reference video quality metrics, QoE remains acceptable on receiving nodes of our resilient overlay also in presence of swap procedures.

Overview of full-reference video quality metrics and their performance evaluations for videoconferencing application

With the development of video services available on various devices (mobile, PC, tablet, television, etc.), the end user experiences and requirements have changed. Objective video quality assessment tools and metrics are increasingly complex and adapted to different types of service context. Compared with other reviews of quality metrics, the full-reference (FR) video quality metrics reviewed (particularly, ViS3, SSIMplus, video multimethod assessment fusion, and open perceptual video quality metric) are more recent and have not been compared with each other yet. We compare and evaluate the performance of 10 FR metrics with respect to their accuracy in predicting the subjective perceived quality for the videoconferencing application. Emphasis is given to the metrics evaluating the visual quality (from human perception). A detailed statistical study is carried out on four subjective databases: Ecole polytechnique federale de Lausanne database, live mobile database, and two Orange Lab databases, with a large sample of distortion types (transmission errors, encoding bit rates, etc.).

Using multiple spatio-temporal features to estimate video quality

In this paper, we propose a new video quality metric based on a set of multiple features that incorporate texture, saliency, spatial activity, and temporal attributes. A random forest regression algorithm is used to combine these features and obtain a video quality score. Experimental results show that the proposed metric has a good performance when tested on several benchmark video quality databases, outperforming current state-of-the-art full-reference video quality metrics.

QoE rating performance evaluation of ITU-T recommended video quality metrics in the context of video freezes

In real-time video streaming, video quality can be degraded due to network performance issues. Among other artefacts, video freezing and video jumping are factors that influence user experience. Service providers, operators and manufacturers are interested in evaluating the quality of experience (QoE) objectively because subjective assessment of QoE is expensive and, in many user cases, subjective assessment is not possible to perform. Different algorithms have been proposed and implemented in this regard. Some of them are in the recommendation list of the ITU Telecommunication Standardization Sector (ITU-T). In this paper, we study the effect of the freezing artefact on user experience and compare the mean opinion score of these videos with the results of two algorithms, the perceptual evaluation of video quality (PEVQ) and temporal quality metric (TQM). Both metrics are part of the ITU-T Recommendation J.247 Annex B and C. PEVQ is a full-reference video quality metric, whereas TQM is a no-reference quality metric. Another contribution of this paper is the study of the impact of different resolutions and frame rates on user experience and how accurately PEVQ and TQM measure varying frame rates.

Video quality assessment using content-weighted spatial and temporal pooling method

Video quality assessment plays an important role in video processing and communication applications. We propose a full reference video quality metric by combining a content-weighted spatial pooling strategy with a temporal pooling strategy. All pixels in a frame are classified into edge, texture, and smooth regions, and their structural similarity image index (SSIM) maps are divided into increasing and saturated regions by the curve of their SSIM values, then a content weight method is applied to increasing regions to get the score of an image frame. Finally, a temporal pooling method is used to get the overall video quality. Experimental results on the LIVE and IVP video quality databases show our proposed method works well in matching subjective scores.

Full-reference audio-visual video quality metric

The goal of this work is to present a full-reference (FR) audio-visual quality metric. We performed three psychophysical experiments in order to obtain a better understanding of how audio and video components interact with each other and how these interactions affect the overall audio-visual quality. In experiment I, subjects evaluated the quality of videos (without any audio) compressed at different video bitrates. In experiment II, subjects evaluated the quality of audio (without any video) compressed at different audio bitrates. In experiment III, subjects evaluated the quality of videos (audio-visual signals), which had their audio and video components compressed at different bitrates. Based on the data gathered from these experiments, we obtain a set of subjective models for audio-visual quality. Inspired by these subjective models, we propose a set of FR audio-visual quality metrics composed of a combination of a video quality metric and an audio quality metric. The proposed metrics have good performance and present better results when compared to simple FR video quality metrics.

Video quality assessment using visual attention computational models

A recent development in the area of image and video quality consists of trying to incorporate aspects of visual attention in the design of visual quality metrics, mostly using the assumption that visual distortions appearing in less salient areas might be less visible and, therefore, less annoying. This research area is still in its infancy and results obtained by different groups are not yet conclusive. Among the works that have reported some improvements, most use subjective saliency maps, i.e., saliency maps generated from eye-tracking data obtained experimentally. Other works address the image quality problem, not focusing on how to incorporate visual attention into video signals. We investigate the benefits of incorporating bottom-up video saliency maps (obtained using Itti's computational model) into video quality metrics. In particular, we com- pare the performance of four full-reference video quality metrics with their modified versions, which had saliency maps incorporated into the algorithm. Results show that the addition of video saliency maps improve the per- formance of most quality metrics tested, but the highest gains were obtained for the metrics that only took into consideration spatial degradations. © 2014 SPIE and IS&T (DOI: 10.1117/1.JEI.23.6.061107)

A Framework to Measure and Estimate Video Quality in SVC Real-Time Adaptive Systems

Effectively adapting the content to network conditions in real-time is an important matter in best-effort networks like the Internet. Scalable Video Coding (SVC) is an interesting alternative to implement such systems. However, some problems of the performance evaluation of SVC based adaptive systems have not been solved. The authors review the main efforts directed to measure video quality on SVC related systems and discuss the limitations of each one. This paper elaborates a framework to measure video quality metrics in real adaptive SVC based streams. An estimation method for full reference video quality metrics is proposed. This method reduces reference information required and it is able to provide real-time accurate results simply using metadata regarding the video quality of the reference layers. The video quality of several streams that have been generated using a real-time adaptive system is first measured with the elaborated framework and then estimated with the proposed method.

Full-Reference Video Quality Assessment by Decoupling Detail Losses and Additive Impairments

Video quality assessment plays a fundamental role in video processing and communication applications. In this paper, we study the use of motion information and temporal human visual system (HVS) characteristics for objective video quality assessment. In our previous work, two types of spatial distortions, i.e., detail losses and additive impairments, are decoupled and evaluated separately for spatial quality assessment. The detail losses refer to the loss of useful visual information that will affect the content visibility, and the additive impairments represent the redundant visual information in the test image, such as the blocking or ringing artifacts caused by data compression and so on. In this paper, a novel full-reference video quality metric is developed, which conceptually comprises the following processing steps: 1) decoupling detail losses and additive impairments within each frame for spatial distortion measure; 2) analyzing the video motion and using the HVS characteristics to simulate the human perception of the spatial distortions; and 3) taking into account cognitive human behaviors to integrate frame-level quality scores into sequence-level quality score. Distinguished from most studies in the literature, the proposed method comprehensively investigates the use of motion information in the simulation of HVS processing, e.g., to model the eye movement, to predict the spatio-temporal HVS contrast sensitivity, to implement the temporal masking effect, and so on. Furthermore, we also prove the effectiveness of decoupling detail losses and additive impairments for video quality assessment. The proposed method is tested on two subjective quality video databases, LIVE and IVP, and demonstrates the state-of-the-art performance in matching subjective ratings.

Full-Reference Video Quality Metric for Fully Scalable and Mobile SVC Content

Universal Multimedia Access (UMA) aims at enabling a straightforward consumption of multimedia content in heterogeneous usage environments. These usage environments may range from mobile devices in a wireless network to high-end desktop computers with wired network connectivity. Scalable video content can be used to deal with the restrictions and capabilities of diverse usage environments. However, in order to optimally tailor scalable video content along the temporal, spatial, or perceptual quality axis, a metric is needed that reliably models subjective video quality. The major contribution of this paper is the development of a novel full-reference quality metric for scalable video bit streams that are compliant with the H.264/AVC Scalable Video Coding (SVC) standard. The scalable video bit streams are intended to be used in mobile usage environments (e.g., adaptive video streaming to mobile devices). The proposed quality metric allows modeling the temporal, spatial, and perceptual quality characteristics of SVC-compliant bit streams by taking into account several properties of the compressed bit streams. These properties include the temporal and spatial variance of the video content, the frame rate, the spatial resolution, and PSNR values. An extensive number of subjective experiments have been conducted to construct and validate our quality metric. Experimental results show that the average correlation coefficient for the video sequences tested is as high as 0.95 (compared to a value of 0.60 when only using the traditional PSNR quality metric). The proposed quality metric also shows a performance that is a uniformly high for video sequences with different temporal and spatial characteristics.

Overt visual attention for free-viewing and quality assessment tasks: Impact of the regions of interest on a video quality metric

The aim of this study is to understand how people watch a video sequence during free-viewing and quality assessment tasks. To this end, two eye tracking experiments were carried out. The video dataset is composed of 10 original video sequences and 50 impaired video sequences (five levels of impairments obtained by a H.264 video compression). A first experiment consisted in recording eye movements in a free-viewing task. The 10 original video sequences were used. The second experiment concerned an eye tracking experiment in a context of a subjective quality assessment. Eye movements were recorded while observers judged on the quality of the 50 impaired video sequences. The comparison between gaze allocations indicates the quality task has a moderate impact on the visual attention deployment. This impact increases with the presentation number of impaired video sequences. The locations of regions of interest remain highly similar after several presentations of the same video sequence, suggesting that eye movements are still driven by the low level visual features after several viewings. In addition, the level of distortion does not significantly alter the oculomotor behavior. Finally, we modified the pooling of an objective full-reference video quality metric by adjusting the weight applied on the distortions. This adjustment depends on the visual importance (the visual importance is deduced from the eye tracking experiment realized on the impaired video sequences). We observe that a saliency-based distortion pooling does not significantly improve the performances of the video quality metric.