Full-reference Metrics Research Articles

Objective Video Quality Assessment Combining Transfer Learning With CNN.

Nowadays, video quality assessment (VQA) is essential to video compression technology applied to video transmission and storage. However, small-scale video quality databases with imbalanced samples and low-level feature representations for distorted videos impede the development of VQA methods. In this paper, we propose a full-reference (FR) VQA metric integrating transfer learning with a convolutional neural network (CNN). First, we imitate the feature-based transfer learning framework to transfer the distorted images as the related domain, which enriches the distorted samples. Second, to extract high-level spatiotemporal features of the distorted videos, a six-layer CNN with the acknowledged learning ability is pretrained and finetuned by the common features of the distorted image blocks (IBs) and video blocks (VBs), respectively. Notably, the labels of the distorted IBs and VBs are predicted by the classic FR metrics. Finally, based on saliency maps and the entropy function, we conduct a pooling stage to obtain the quality scores of the distorted videos by weighting the block-level scores predicted by the trained CNN. In particular, we introduce a preprocessing and a postprocessing to reduce the impact of inaccurate labels predicted by the FR-VQA metric. Due to feature learning in the proposed framework, two kinds of experimental schemes including train-test iterative procedures on one database and tests on one database with training other databases are carried out. The experimental results demonstrate that the proposed method has high expansibility and is on a par with some state-of-the-art VQA metrics on two widely used VQA databases with various compression distortions.

Reduced-reference quality assessment of image super-resolution by energy change and texture variation

In this paper, we propose a novel reduced-reference quality assessment metric for image super-resolution (RRIQA-SR) based on the low-resolution (LR) image information. With the pixel correspondence, we predict the perceptual similarity between image patches of LR and SR images by two components: the energy change in low-frequency regions, which can be used to capture the global distortion in SR images, and texture variation in high-frequency regions, which can be used to capture the local distortion in SR images. The overall quality of SR images is estimated by perceptual similarity calculated by energy change and texture variation between local image patches of LR and HR images. Experimental results demonstrate that the proposed method can obtain better performance of quality prediction for SR images than other existing ones, even including some full-reference (FR) metrics.

Image Quality Assessment through FSIM, SSIM, MSE and PSNR—A Comparative Study

Quality is a very important parameter for all objects and their functionalities. In image-based object recognition, image quality is a prime criterion. For authentic image quality evaluation, ground truth is required. But in practice, it is very difficult to find the ground truth. Usually, image quality is being assessed by full reference metrics, like MSE (Mean Square Error) and PSNR (Peak Signal to Noise Ratio). In contrast to MSE and PSNR, recently, two more full reference metrics SSIM (Structured Similarity Indexing Method) and FSIM (Feature Similarity Indexing Method) are developed with a view to compare the structural and feature similarity measures between restored and original objects on the basis of perception. This paper is mainly stressed on comparing different image quality metrics to give a comprehensive view. Experimentation with these metrics using benchmark images is performed through denoising for different noise concentrations. All metrics have given consistent results. However, from representation perspective, SSIM and FSIM are normalized, but MSE and PSNR are not; and from semantic perspective, MSE and PSNR are giving only absolute error; on the other hand, SSIM and PSNR are giving perception and saliency-based error. So, SSIM and FSIM can be treated more understandable than the MSE and PSNR.

No-Reference Video Quality Estimation Based on Machine Learning for Passive Gaming Video Streaming Applications

Recent years have seen increasing growth and popularity of gaming services, both interactive and passive. While interactive gaming video streaming applications have received much attention, passive gaming video streaming, in-spite of its huge success and growth in recent years, has seen much less interest from the research community. For the continued growth of such services in the future, it is imperative that the end user gaming quality of experience (QoE) is estimated so that it can be controlled and maximized to ensure user acceptance. Previous quality assessment studies have shown not so satisfactory performance of existing No-reference (NR) video quality assessment (VQA) metrics. Also, due to the inherent nature and different requirements of gaming video streaming applications, as well as the fact that gaming videos are perceived differently from non-gaming content (as they are usually computer generated and contain artificial/synthetic content), there is a need for application-specific light-weight, no-reference gaming video quality prediction models. In this paper, we present two NR machine learning-based quality estimation models for gaming video streaming, NR-GVSQI, and NR-GVSQE, using NR features, such as bitrate, resolution, and temporal information. We evaluate their performance on different gaming video datasets and show that the proposed models outperform the current state-of-the-art no-reference metrics, while also reaching a prediction accuracy comparable to the best known full reference metric.

Convolutional Autoencoder-Based Multispectral Image Fusion

This paper presents a deep learning-based pansharpening method for fusion of panchromatic and multispectral images in remote sensing applications. This method can be categorized as a component substitution method in which a convolutional autoencoder network is trained to generate original panchromatic images from their spatially degraded versions. Low resolution multispectral images are then fed into the trained convolutional autoencoder network to generate estimated high resolution multispectral images. The fusion is achieved by injecting the detail map of each spectral band into the corresponding estimated high resolution multispectral bands. Full reference and no-reference metrics are computed for the images of three satellite datasets. These measures are compared with the existing fusion methods whose codes are publicly available. The results obtained indicate the effectiveness of the developed deep learning-based method for multispectral image fusion.

Toward an objective benchmark for video completion

Video-completion methods aim to complete selected regions of a video sequence in a natural looking manner with little to no additional user interaction. Numerous algorithms were proposed to solve this problem; however, a unified benchmark to quantify the progress in the field is still lacking. Video-completion results are usually judged by their plausibility and aren’t expected to adhere to one ground-truth result, which complicates measuring the video-completion performance. In this paper, we address this problem by proposing a set of full-reference quality metrics that outperform naive approaches and an online benchmark for video-completion algorithms. We construct seven test sequences with ground-truth video-completion results by composing various foreground objects over a set of background videos. Using this dataset, we conduct an extensive comparative study of video-completion perceptual quality involving six algorithms and over 300 human participants. Finally, we show that by relaxing the requirement of complete adherence to ground truth and by taking into account temporal consistency we can increase the correlation of objective quality metrics with perceptual completion quality on the proposed dataset.

Combining Local and Global Measures for DIBR-Synthesized Image Quality Evaluation.

Depth-Image-Based-Rendering (DIBR) techniques are significant for three-dimensional (3D) video applications, e.g., 3D television and free viewpoint video (FVV). Unfortunately, the DIBR-synthesized image suffers from various distortions, which induce an annoying viewing experience for the entire FVV. Proposing a quality evaluator for DIBR-synthesized images is fundamental for the design of perceptual friendly FVV systems. Since the associated reference image is usually not accessible, full-reference (FR) methods cannot be directly applied for quality evaluation of the synthesized image. In addition, most traditional no-reference (NR) methods fail to effectively measure the specifically DIBR-related distortions. In this paper, we propose a novel NR quality evaluation method accounting for two categories of DIBR-related distortions, i.e., geometric distortions and sharpness. First, the disoccluded regions, as one of the most obvious geometric distortions, are captured by analyzing local similarity. Then, another typical geometric distortion (i.e., stretching) is detected and measured by calculating the similarity between it and its equal-size adjacent region. Second, considering the property of scale invariance, the global sharpness is measured as the distance between the distorted image and its downsampled version. Finally, the perceptual quality is estimated by linearly pooling the scores of two geometric distortions and sharpness together. Experimental results verify the superiority of the proposed method over the prevailing FR and NR metrics. More specifically, it is superior to all competing methods except APT in terms of effectiveness, but greatly outmatches APT in terms of implementation time.

Benchmarking Single Image Dehazing and Beyond.

In this paper, we present a comprehensive study and evaluation of existing single image dehazing algorithms, using a new large-scale benchmark consisting of both synthetic and real-world hazy images, called REalistic Single Image DEhazing (RESIDE). RESIDE highlights diverse data sources and image contents, and is divided into five subsets, each serving different training or evaluation purposes. We further provide a rich variety of criteria for dehazing algorithm evaluation, ranging from full-reference metrics, to no-reference metrics, to subjective evaluation and the novel task-driven evaluation. Experiments on RESIDE shed light on the comparisons and limitations of stateof- the-art dehazing algorithms, and suggest promising future directions.

Referenceless image quality assessment by saliency, color-texture energy, and gradient boosting machines

In most practical multimedia applications, processes are used to manipulate the image content. These processes include compression, transmission, or restoration techniques, which often create distortions that may be visible to human subjects. The design of algorithms that can estimate the visual similarity between a distorted image and its non-distorted version, as perceived by a human viewer, can lead to significant improvements in these processes. Therefore, over the last decades, researchers have been developing quality metrics (i.e., algorithms) that estimate the quality of images in multimedia applications. These metrics can make use of either the full pristine content (full-reference metrics) or only of the distorted image (referenceless metric). This paper introduces a novel referenceless image quality assessment (RIQA) metric, which provides significant improvements when compared to other state-of-the-art methods. The proposed method combines statistics of the opposite color local variance pattern (OC-LVP) descriptor with statistics of the opposite color local salient pattern (OC-LSP) descriptor. Both OC-LVP and OC-LSP descriptors, which are proposed in this paper, are extensions of the opposite color local binary pattern (OC-LBP) operator. Statistics of these operators generate features that are mapped into subjective quality scores using a machine-learning approach. Specifically, to fit a predictive model, features are used as input to a gradient boosting machine (GBM). Results show that the proposed method is robust and accurate, outperforming other state-of-the-art RIQA methods.

Reduced-reference Video Quality Metric Using Spatio-temporal Activity Information

Monitoring and maintaining acceptable Quality of Experience is of great importance to video service providers. Perceived visual quality of transmitted video via wireless networks can be degraded by transmission errors. This paper presents a reduced-reference video quality metric of very low complexity and overhead that makes use of frame based spatial (SI) and temporal (TI) activity levels to monitor the effect of channel errors on video transmitted over error prone networks. The performance of the metric is evaluated relative to that of a number of full and reduced reference metrics. The proposed metric outperforms some of the most popular full reference metrics whilst requiring very little overhead.

Modelling the non-stationary videos for performance assessment of frame reconstruction

Usually public datasets involving non-stationary video sequences do not have the corresponding ground-truth stationary videos. This means that the performance assessment of reconstructed videos using the wide spread full-reference metrics is impossible. For this purpose, we propose a set of models simulating the shaking camera motion in real-life applications, such as the hand-held shooting and shooting from unmanned aerial vehicles and moving platforms. First, the stationary video sequence is analyzed carefully. 3D camera path is built using the pseudo-disparity values between the consecutive frames based on the corresponding feature points. This high cost procedure permits to build a relatively smooth camera path in 3D space with the non-significant jitters. Second, the end-user chooses the duration, position, and type of shake and/or jitters in video sequence. Third, the desired simulation is implemented in wavelet domain. The distortions, such as displacements, rotations, and local/global blurring, are simulated depending on the chosen situation. We use the physically-based and sample models. In order to reach the ground-truth demonstration, we crop the original frames. If RGB-D video is available, then realistic perspective projections of shaking synthesized video can be created.

Metrics and methods of video quality assessment: a brief review

With rapid development of video acquisition devices and wide applications of video data, more and more requirements are established to use video data query. video quality assessment and improvement become popular and important research issues which attract lots of researchers. The video quality can influence technique application, user experience, and application results. This paper firstly reviews research work on video query based applications. Then, various metrics of video quality assessment are reviewed according to the requirement of reference video, including full-reference metrics, no-reference metrics and reduced-reference metrics. In addition, methods of video quality assessment are reviewed by the features, which include visual features, bitstream-based or packet-based features, data features. Finally, a number of video quality improvement methods are briefly introduced.

Blind quality estimator for 3D images based on binocular combination and extreme learning machine

Over recent years, blind quality estimators for three-dimensional (3D) images have received increasing attention. In this paper, we describe a blind quality estimator for 3D images based on binocular combination and an extreme learning machine (ELM) for more accurate alignment with subjective human experience. First, two binocular combinations of stimuli are generated using different combination strategies. Various binocular quality-aware features of these combinations are then extracted by local binary pattern operators, which give an effective description of the degradation pattern. Finally, these features are mapped to the subjective quality score of the distorted 3D image using an ELM. Experimental results using three benchmark databases confirm that the proposed metric is effective and achieves competitive prediction performance when compared with most current full reference and blind metrics.

No-reference perceived image quality measurement for multiple distortions

No-reference image quality assessment (NR-IQA) metrics play an important role in real-world multimedia applications. Accordingly, many NR-IQA metrics have been proposed in the literature. Most of these proposed metrics predict the visual quality only of images distorted with a single distortion, whereas in the real-world, digital images undergo various steps that contaminate them with multiple distortions before reaching consumers. However, only a few NR-IQA metrics have been proposed for multiple distortions. In this paper, we propose an NR-IQA metric for images distorted with blur and noise, which are added during the image acquisition and transmission process. In the proposed metric, perceived blur and noise scores are estimated separately. These scores are then mapped to the final quality score considering the perceived degree of each type of distortion, the nonlinear functions of the distortions, and the joint effects of the distortions. Weights for each mapping component are learned with subjective data input. Performance comparison of the proposed metric on the blur-noise image dataset of the LIVE multiply distorted (LIVEMD) image quality database confirm that the proposed metric is more effective than the state-of-the-art full-reference and NR IQA metrics.

An extensive performance evaluation of full-reference HDR image quality metrics

High dynamic range (HDR) image and video technology has recently attracted a great deal of attention in the multimedia community, as a mean to produce truly realistic video and further improve the quality of experience (QoE) of emerging multimedia services. In this context, measuring the quality of compressed HDR content plays a fundamental role. However, full-reference (FR) HDR visual quality assessment poses new challenges with respect to the conventional low dynamic range case. Quality metrics have to be redesigned or adapted to HDR, and understanding their reliability to predict users’ judgments is even more critical due to the still limited availability of HDR displays to perform subjective evaluations. The goal of this paper is to provide a complete and thorough survey of the performance of the most popular HDR FR image quality metrics. To this end, we gather several existing HDR image databases with subjective quality annotations, in addition to a new one created by ourselves. After aligning the scores in these databases, we obtain an extensive set of 690 compressed HDR images, along with their subjective quality. Next, we analyze in depth many FR metrics, including those used in MPEG standardization, using both classical correlation analyses and classification accuracy. We believe that our results could serve as the most complete and comprehensive benchmark of image quality metrics in the field of HDR image compression.

Learning a no-reference quality metric for single-image super-resolution

Numerous single-image super-resolution algorithms have been proposed in the literature, but few studies address the problem of performance evaluation based on visual perception. While most super-resolution images are evaluated by full-reference metrics, the effectiveness is not clear and the required ground-truth images are not always available in practice. To address these problems, we conduct human subject studies using a large set of super-resolution images and propose a no-reference metric learned from visual perceptual scores. Specifically, we design three types of low-level statistical features in both spatial and frequency domains to quantify super-resolved artifacts, and learn a two-stage regression model to predict the quality scores of super-resolution images without referring to ground-truth images. Extensive experimental results show that the proposed metric is effective and efficient to assess the quality of super-resolution images based on human perception.

Estimation of the Image Quality under Different Distortions

The evaluation of the image quality is very important. The best subjective evaluation of the image is done by the human eyes as they are the good receivers. Objective analysis of the image is done by using full reference metric. The results of the objective measurements are validated by the subjective measurements. The objective evaluation of image in this paper has been done using PSNR (Peak Signal to Noise Ratio), MSE (Mean Squared Error) and SSIM (Structural Similarity Index Metric). These algorithms are applied on the different images. If the value of PSNR increases, the corresponding value of SSIM also increases and the value of MSE decreases. In the proposed wok, if the value of MSE increases by 8%, then the corresponding PSNR decreases by 10%. This is not applicable on all images because the MSE, PSNR and SSIM changes according to the complexity of the image. Every image has different coefficient of complexity. More complex image gets distorted first than the less complex image.

An experimental survey of no-reference video quality assessment methods

Purpose The Video Quality Metric (VQM) is one of the most used objective methods to assess video quality, because of its high correlation with the human visual system (HVS). VQM is, however, not viable in real-time deployments such as mobile streaming, not only due to its high computational demands but also because, as a Full Reference (FR) metric, it requires both the original video and its impaired counterpart. In contrast, No Reference (NR) objective algorithms operate directly on the impaired video and are considerably faster but loose out in accuracy. The purpose of this paper is to study how differently NR metrics perform in the presence of network impairments. Design/methodology/approach The authors assess eight NR metrics, alongside a lightweight FR metric, using VQM as benchmark in a self-developed network-impaired video data set. This paper covers a range of methods, a diverse set of video types and encoding conditions and a variety of network impairment test-cases. Findings The authors show the extent by which packet loss affects different video types, correlating the accuracy of NR metrics to the FR benchmark. This paper helps identifying the conditions under which simple metrics may be used effectively and indicates an avenue to control the quality of streaming systems. Originality/value Most studies in literature have focused on assessing streams that are either unaffected by the network (e.g. looking at the effects of video compression algorithms) or are affected by synthetic network impairments (i.e. via simulated network conditions). The authors show that when streams are affected by real network conditions, assessing Quality of Experience becomes even harder, as the existing metrics perform poorly.

QoE rating performance evaluation of ITU-T recommended video quality metrics in the context of video freezes

In real-time video streaming, video quality can be degraded due to network performance issues. Among other artefacts, video freezing and video jumping are factors that influence user experience. Service providers, operators and manufacturers are interested in evaluating the quality of experience (QoE) objectively because subjective assessment of QoE is expensive and, in many user cases, subjective assessment is not possible to perform. Different algorithms have been proposed and implemented in this regard. Some of them are in the recommendation list of the ITU Telecommunication Standardization Sector (ITU-T). In this paper, we study the effect of the freezing artefact on user experience and compare the mean opinion score of these videos with the results of two algorithms, the perceptual evaluation of video quality (PEVQ) and temporal quality metric (TQM). Both metrics are part of the ITU-T Recommendation J.247 Annex B and C. PEVQ is a full-reference video quality metric, whereas TQM is a no-reference quality metric. Another contribution of this paper is the study of the impact of different resolutions and frame rates on user experience and how accurately PEVQ and TQM measure varying frame rates.

Perceptual quality estimation of H.264/AVC videos using reduced-reference and no-reference models

Reduced-reference (RR) and no-reference (NR) models for video quality estimation, using features that account for the impact of coding artifacts, spatio-temporal complexity, and packet losses, are proposed. The purpose of this study is to analyze a number of potentially quality-relevant features in order to select the most suitable set of features for building the desired models. The proposed sets of features have not been used in the literature and some of the features are used for the first time in this study. The features are employed by the least absolute shrinkage and selection operator (LASSO), which selects only the most influential of them toward perceptual quality. For comparison, we apply feature selection in the complete feature sets and ridge regression on the reduced sets. The models are validated using a database of H.264/AVC encoded videos that were subjectively assessed for quality in an ITU-T compliant laboratory. We infer that just two features selected by RR LASSO and two bitstream-based features selected by NR LASSO are able to estimate perceptual quality with high accuracy, higher than that of ridge, which uses more features. The comparisons with competing works and two full-reference metrics also verify the superiority of our models.