Human Opinion Scores Research Articles

Attentive Deep Image Quality Assessment for Omnidirectional Stitching

Omnidirectional images or videos are commonly generated via the stitching of multiple images or videos, and the quality of omnidirectional stitching strongly influences the quality of experience (QoE) of the generated scenes. Although there were many studies research the omnidirectional image quality assessment (IQA), the evaluation of the omnidirectional stitching quality has not been sufficiently explored. In this paper, we focus on the IQA for the omnidirectional stitching of dual fisheye images. We first establish an omnidirectional stitching image quality assessment (OSIQA) database, which includes 300 distorted images and 300 corresponding reference images generated from 12 raw scenes. The database contains a variety of distortion types caused by omnidirectional stitching, including color distortion, geometric distortion, blur distortion, and ghosting distortion, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">etc.</i> A subjective quality assessment study is conducted on the database and human opinion scores are collected for the distorted omnidirectional images. We then devise a deep learning based objective IQA metric termed Attentive Multi-channel IQA Net. In particular, we extend hyper-ResNet by developing a subnetwork for spatial attention and propose a spatial regularization item. Experimental results show that our proposed FR and NR models achieve the best performance compared with the state-of-the-art FR and NR IQA metrics on the OSIQA database. The OSIQA database as well as the proposed Attentive Multi-channel IQA Net will be released to facilitate future research.

Toward a blind image quality evaluator in the wild by learning beyond human opinion scores

Nowadays, most existing blind image quality assessment (BIQA) models inthewild heavily rely on human ratings, which are extraordinarily labor-expensive to collect. Here, we propose an opinion−free BIQA method that learns from multiple annotators to assess the perceptual quality of images captured in the wild. Specifically, we first synthesize distorted images based on the pristine counterparts. We then randomly assemble a set of image pairs from the synthetic images, and use a group of IQA models to assign pseudo-binary labels for each pair indicating which image has higher quality as the supervisory signal. Based on the newly established pseudo-labeled dataset, we train a deep neural network (DNN)-based BIQA model to rank the perceptual quality, optimized for consistency with the binary rank labels. Since there exists domain shift, e.g., distortion shift and content shift, between the synthetic and in-the-wild images, we leverage two ways to alleviate this issue. First, the simulated distortions should be similar to authentic distortions as much as possible. Second, an unsupervised domain adaptation (UDA) module is further applied to encourage learning domain-invariant features between two domains. Extensive experiments demonstrate the effectiveness of our proposed opinion−free BIQA model, yielding SOTA performance in terms of correlation with human opinion scores, as well as gMAD competition. Our code is available at: https://github.com/wangzhihua520/OF_BIQA.

Understanding the perceived quality of video predictions

The study of video prediction models is believed to be a fundamental approach to representation learning for videos. While a plethora of generative models for predicting the future frame pixel values given the past few frames exist, the quantitative evaluation of the predicted frames has been found to be extremely challenging. In this context, we study the problem of quality assessment of predicted videos. We create the Indian Institute of Science Predicted Videos Quality Assessment (IISc PVQA) Database consisting of 300 videos, obtained by applying different prediction models on different datasets, and accompanying human opinion scores. We collected subjective ratings of quality from 50 human participants for these videos. Our subjective study reveals that human observers were highly consistent in their judgments of quality of predicted videos. We benchmark several popularly used measures for evaluating video prediction and show that they do not adequately correlate with these subjective scores. We introduce two new features to effectively capture the quality of predicted videos, motion-compensated cosine similarities of deep features of predicted frames with past frames, and deep features extracted from rescaled frame differences. We show that our feature design leads to state-of-the-art quality prediction in accordance with human judgments on our IISc PVQA Database. The database and code are publicly available on our project website: https://nagabhushansn95.github.io/publications/2020/pvqa.

On the performance of objective quality metrics for lightfields

Lightfield (LF) technology has attained significant attention in recent years due to its capability to capture much richer textural and geometric information in the scene compared to the classical 2D representation. The resampling and compression operations on LFs often lead to visual quality degradation, thus, sophisticated visual quality assessment methods play a crucial role to ensure a pleasant viewing experience. To this end, it is necessary to examine the performance of quality assessment methods for LF contents. The paper provides a comprehensive study on the reliability of various objective algorithms for LF quality prediction. Three subjectively-annotated LF data sets were selected and an extensive quality estimation analysis has been conducted using several objective quality assessment methods. In total, 250 LFs (more than 48000 perspective images) were evaluated. The results were compared against human opinion scores using various correlation indices and their statistical significance. Next, a decision-making strategy was adopted to choose the most reliable quality metrics for evaluation of LFs and finally, a metric fusion framework was proposed to further improve the quality prediction accuracy. To best of our knowledge, the benchmark and the analytical methodologies used in this paper are the most comprehensive study on the objective quality assessment methods for LF application.

Predicting Spatio-Temporal Entropic Differences for Robust No Reference Video Quality Assessment

We consider the problem of robust no reference (NR) video quality assessment (VQA) where the algorithms need to have good generalization performance when they are trained and tested on different datasets. We specifically address this question in the context of predicting video quality for compression and transmission applications. Motivated by the success of the spatio-temporal entropic differences video quality predictor in this context, we design a framework using convolutional neural networks to predict spatial and temporal entropic differences without the need for a reference or human opinion score. This approach enables our model to capture both spatial and temporal distortions effectively and allows for robust generalization. We evaluate our algorithms on a variety of datasets and show superior cross database performance when compared to state of the art NR VQA algorithms.

A novel approach for image focus measure

In this paper, a new no-reference image quality assessment (NR-IQA) metric is proposed. This method is based on the probability coefficients and modified entropy. The probability coefficients are used to capture the significant features and represent the important dynamics of an image. Moreover, it is used to quantify the possible loss of naturalness in a distorted image. The proposed approach has been tested on synthetic images with blurring effects and under noisy conditions. It is observed that the proposed focus measure is invariant to contrast and noisy conditions. The effectuality of the proposed approach is tested on various experimentations carried out on real-time images. The performance of proposed focus measure is better as compared to the existing image focus metrics. It also shows a good consistency and a better response under blurring effects and noises at different levels. Moreover, it exhibits a good working range, unimodality and monotonicity properties and hence satisfying the conditions of a good focus measure. Further, the various experiments on image datasets such as LIVE, TID2008 and CSIQ have shown the effectiveness of proposed NR-IQA metric. It is also observed that the presented approach has good correlation with the human opinion score as compared to the existing methods.

Predicting the Empirical Distribution of Video Quality Scores Using Recurrent Neural Networks

Video quality assessment is a crucial routine in the broadcasting industry. Due to the duration and the excessive number of video files, a computer-based video quality assessment mechanism is the only solution. While it is common to measure the quality of a video file at the compression stage by comparing it against the raw data, at later stages, no reference video is available for comparison. Therefore, a no-reference (Blind) video quality assessment (NR-VQA) technique is essential. The current NR-VQA methods predict only the mean opinion score (MOS) and do not provide further information about the distribution of people score. However, this distribution is informative for the evaluation of QoE. In this paper, we propose a method for predicting the empirical distribution of human opinion scores in the assessment of video quality. To this end, we extract some frame-level features, and next, we feed these features to a recurrent neural network. Finally, the distribution of opinion score is predicted in the last layer of the RNN. The experiments show that averages of predicted distributions have comparable or better results with previous methods on the KonVid-1k dataset.

Perceptual Quality Assessment of Digital Images Using Deep Features

Perceptual quality assessment is a tough task especially in the absence of reference information. No-reference image quality assessment is more challenging than full-reference or reduced reference methods, as the system has to model the different image distortions in the form of a quality score. Most of the approaches are based on handcrafted features which are based on natural scene statistics and are specific to some distortion types. These approaches provide high correlation with human opinion score for datasets containing specific distortions, but they fail to generalize well in scenarios were multiple distortions or real-time distortions are present in images. Deep learning algorithms, on the other hand, demonstrated their abilities to learn expert features with better discriminatory power for various classification and regression tasks. It is a big challenge to use those deep learning methods for image quality assessment as the image datasets with human opinion score are very small and cannot be used effectively to train a deep learning algorithm. We experimented with activations of different deep layers of thirteen pre-trained models and checked for their suitability for the task of no-reference quality assessment. Fine-tuning of these models on quality assessment datasets provided even better performance. A Gaussian process regression model is trained on these activations to perform the quality assessment and it provided state-of-the-art performance. Cross-dataset validation demonstrated its performance further and also provided further prospects of research in this direction.

PFF-RVM: A new no reference image quality measure

In this paper, a new no-reference image quality metric is proposed and developed. A set of ten perceptual features are extracted from each distorted image, then Relevance Vector Machine (RVM) is employed to learn the mapping between the combined features and human opinion scores. Validation tests and simulations are conducted on the LIVE II and MDID 2013 databases. The predictive performances of this new metric (we called PFF-RVM for perceptual features fusion using relevance vector machine based metric) are compared to the most recent no-reference metrics in terms of correlation and monotonicity. Results show that the proposed metric has satisfactory and comparable performances to the most sophisticated and commonly used no-reference quality metrics in the state-of-the-art.

A No-Reference Image Quality Measure for Blurred and Compressed Images Using Sparsity Features

Images can be distorted in the real world via many sources like faulty sensors, artifacts generated by compression algorithms, defocus, faulty lens, and poor lighting conditions. Our biological vision system can identify the quality of image by looking at the images, but developing an algorithm to assess the quality of an image is a very challenging task as an image can be corrupted by different types of distortions and statistical properties of different types of distortions are dissimilar. The main objective of this article is to propose an image quality assessment technique for images corrupted by blurring and compression-based artifacts. Machine learning-based approaches have been used in recent times to perform this task. Images can be analyzed in different transform domains like discrete cosine transform domain, wavelet domains, curvelet domains, and singular value decomposition. These domains generate sparse matrices. In this paper, we propose no-reference image quality assessment algorithms for images corrupted by blur and different compression algorithms using sparsity-based features computed from different domains and all features pooled by support vector regression. The proposed model has been tested on three standard image quality assessment datasets LIVE, CSIQ, and TID2013, and correlation with subjected human opinion scores has been presented along with comparative study with state-of-the-art quality measures. Experiments run on standard image quality databases show that the results obtained are outperforming the existing results.

Learning a Continuous-Time Streaming Video QoE Model

Over-the-top adaptive video streaming services are frequently impacted by fluctuating network conditions that can lead to rebuffering events (stalling events) and sudden bitrate changes. These events visually impact video consumers' quality of experience (QoE) and can lead to consumer churn. The development of models that can accurately predict viewers' instantaneous subjective QoE under such volatile network conditions could potentially enable the more efficient design of quality-control protocols for media-driven services, such as YouTube, Amazon, Netflix, and so on. However, most existing models only predict a single overall QoE score on a given video and are based on simple global video features, without accounting for relevant aspects of human perception and behavior. We have created a QoE evaluator, called the time-varying QoE Indexer, that accounts for interactions between stalling events, analyzes the spatial and temporal content of a video, predicts the perceptual video quality, models the state of the client-side data buffer, and consequently predicts continuous-time quality scores that agree quite well with human opinion scores. The new QoE predictor also embeds the impact of relevant human cognitive factors, such as memory and recency, and their complex interactions with the video content being viewed. We evaluated the proposed model on three different video databases and attained standout QoE prediction performance.

NIMA: Neural Image Assessment

Automatically learned quality assessment for images has recently become a hot topic due to its usefulness in a wide variety of applications such as evaluating image capture pipelines, storage techniques and sharing media. Despite the subjective nature of this problem, most existing methods only predict the mean opinion score provided by datasets such as AVA [1] and TID2013 [2]. Our approach differs from others in that we predict the distribution of human opinion scores using a convolutional neural network. Our architecture also has the advantage of being significantly simpler than other methods with comparable performance. Our proposed approach relies on the success (and retraining) of proven, state-of-the-art deep object recognition networks. Our resulting network can be used to not only score images reliably and with high correlation to human perception, but also to assist with adaptation and optimization of photo editing/enhancement algorithms in a photographic pipeline. All this is done without need for a "golden" reference image, consequently allowing for single-image, semantic- and perceptually-aware, no-reference quality assessment.

Biologically Inspired Blind Quality Assessment of Tone-Mapped Images

Currently, many tone mapping operators (TMOs) have been provided to compress high dynamic range images to low dynamic range (LDR) images for visualizing them on the common displays. Since quality degradation is inevitably induced by compression, how to evaluate the obtained LDR images is indeed a headache problem. Until now, only a few full reference (FR) image quality assessment metrics have been proposed. However, they highly depend on reference image and neglect human visual system characteristics, hindering the practical applications. In this paper, we propose an effective blind quality assessment method of tone-mapped image without access to reference image. Inspired by that the performance of existing TMOs largely depend on the brightness and chromatic and structural properties of a scene, we evaluate the perceptual quality from the perspective of color information processing in the brain. Specifically, motivated by the physiological and psychological evidence, we simulate the responses of single-opponent (SO) and double-opponent (DO) cells, which play an important role in the processing of the color information. To represent the textural information, we extract three features from gray-level co-occurrence matrix (GLCM) calculated from SO responses. Meanwhile, both GLCM and local binary pattern descriptor are employed to extract texture and structure in the responses of DO cells. All these extracted features and associated subjective ratings are learned to reveal the connection between feature space and human opinion score. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art blind quality assessment methods and is comparable with the popular FR methods on two recently published tone-mapped image databases.

A Haar wavelet-based perceptual similarity index for image quality assessment

In most practical situations, the compression or transmission of images and videos creates distortions that will eventually be perceived by a human observer. Vice versa, image and video restoration techniques, such as inpainting or denoising, aim to enhance the quality of experience of human viewers. Correctly assessing the similarity between an image and an undistorted reference image as subjectively experienced by a human viewer can thus lead to significant improvements in any transmission, compression, or restoration system. This paper introduces the Haar wavelet-based perceptual similarity index (HaarPSI), a novel and computationally inexpensive similarity measure for full reference image quality assessment. The HaarPSI utilizes the coefficients obtained from a Haar wavelet decomposition to assess local similarities between two images, as well as the relative importance of image areas. The consistency of the HaarPSI with the human quality of experience was validated on four large benchmark databases containing thousands of differently distorted images. On these databases, the HaarPSI achieves higher correlations with human opinion scores than state-of-the-art full reference similarity measures like the structural similarity index (SSIM), the feature similarity index (FSIM), and the visual saliency-based index (VSI). Along with the simple computational structure and the short execution time, these experimental results suggest a high applicability of the HaarPSI in real world tasks.

Subjective and objective quality assessment of degraded document images

Abstract The huge amount of degraded documents stored in libraries and archives around the world needs automatic procedures of enhancement, classification, transliteration, etc. While high-quality images of these documents are in general easy to be captured, the amount of damage these documents contain before imaging is unknown. It is highly desirable to measure the severity of degradation that each document image contains. The degradation assessment can be used in tuning parameters of processing algorithms, selecting the proper algorithm, finding damaged or exceptional documents, among other applications. In this paper, the first dataset of degraded document images along with the human opinion scores for each document image is introduced in order to evaluate the image quality assessment metrics on historical document images. In this research, human judgments on the overall quality of the document image are used instead of the previously used OCR performance. Also, we propose an objective no reference quality metric based on the statistics of the mean subtracted contrast normalized (MSCN) coefficients computed from segmented layers of each document image. The segmentation into four layers of foreground and background is done on the basis of an analysis of the log-Gabor filters. This segmentation is based on the assumption that the sensitivity of the human visual system (HVS) is different at the locations of text and non-text. Experimental results show that the proposed metric has comparable or better performance than the state-of-the-art metrics, while it has a moderate complexity. The developed dataset as well as the Matlab source code of the proposed metric is available at http://www.synchromedia.ca/system/files/VDIQA.zip .

Large-Scale Crowdsourced Study for Tone-Mapped HDR Pictures.

Measuring digital picture quality, as perceived by human observers, is increasingly important in many applications in which humans are the ultimate consumers of visual information. Standard dynamic range (SDR) images provide 8 b/color/pixel. High dynamic range (HDR) images, usually created from multiple exposures of the same scene, can provide 16 or 32 b/color/pixel, but need to be tonemapped to SDR for display on standard monitors. Multiexposure fusion (MEF) techniques bypass HDR creation by fusing an exposure stack directly to SDR images to achieve aesthetically pleasing luminance and color distributions. Many HDR and MEF databases have a relatively small number of images and human opinion scores, obtained under stringently controlled conditions, thereby limiting realistic viewing. Moreover, many of these databases are intended to compare tone-mapping algorithms, rather than being specialized for developing and comparing image quality assessment models. To overcome these challenges, we conducted a massively crowdsourced online subjective study. The primary contributions described in this paper are: 1) the new ESPL-LIVE HDR Image Database that we created containing diverse images obtained by tone-mapping operators and MEF algorithms, with and without post-processing; 2) a large-scale subjective study that we conducted using a crowdsourced platform to gather more than 300 000 opinion scores on 1811 images from over 5000 unique observers; and 3) a detailed study of the correlation performance of the state-of-the-art no-reference image quality assessment algorithms against human opinion scores of these images. The database is available at http://signal.ece.utexas.edu/%7Edebarati/HDRDatabase.zip.

No-reference image contrast measure using image statistics and random forest

Image quality assessment is very crucial for certain image processing applications. Images can be distorted from multiple sources like a fault in sensors, camera shake, poor lighting, over-exposure etc. All these distortions reduce the visual quality of images. Assessing the quality of images can be done with the use of a reference image of the same scene or without it. Without the use of reference image, quality assessment is a very difficult task. Machine learning approaches are very common in no-reference image quality assessment. No reference strategies are very useful if the type of distortion is known. Contrast assessment is a very important application in image processing as poor contrast images are difficult for automated image processing. In this paper, we propose a no-reference image quality measure for images with respect to contrast using random forest regression and validate the results using standard datasets. Experimental results on standard datasets show that the proposed method demonstrates promising results when compared to existing no-reference techniques and the proposed method shows high correlation values with human opinion scores.

BSD: Blind image quality assessment based on structural degradation

Research in biological vision and neurology has evidenced that there are separate mechanisms in human visual cortex to process the first- and second-order patterns. Image structures detected by a linear filter are the first-order patterns which describe luminance changes, while patterns that are invisible to linear filters are often referred as the second-order structures. In this paper, we propose a general-purpose blind image quality assessment (BIQA) method by taking account of both the first- and second-order image structures. Specifically, the Prewitt linear filters are used to extract first-order image structures and the local contrast normalization is employed to extract second-order image structures. Perceptual features are extracted from these two image structural maps and used as the input to a support vector regression to model the nonlinear relationship between feature space to human opinion score. Extensive experiments on five image databases manifest the outstanding performance of the proposed method compared to the relevant state-of-the-art BIQA methods.

A Completely Blind Video Integrity Oracle.

Considerable progress has been made toward developing still picture perceptual quality analyzers that do not require any reference picture and that are not trained on human opinion scores of distorted images. However, there do not yet exist any such completely blind video quality assessment (VQA) models. Here, we attempt to bridge this gap by developing a new VQA model called the video intrinsic integrity and distortion evaluation oracle (VIIDEO). The new model does not require the use of any additional information other than the video being quality evaluated. VIIDEO embodies models of intrinsic statistical regularities that are observed in natural vidoes, which are used to quantify disturbances introduced due to distortions. An algorithm derived from the VIIDEO model is thereby able to predict the quality of distorted videos without any external knowledge about the pristine source, anticipated distortions, or human judgments of video quality. Even with such a paucity of information, we are able to show that the VIIDEO algorithm performs much better than the legacy full reference quality measure MSE on the LIVE VQA database and delivers performance comparable with a leading human judgment trained blind VQA model. We believe that the VIIDEO algorithm is a significant step toward making real-time monitoring of completely blind video quality possible.

Massive Online Crowdsourced Study of Subjective and Objective Picture Quality.

Most publicly available image quality databases have been created under highly controlled conditions by introducing graded simulated distortions onto high-quality photographs. However, images captured using typical real-world mobile camera devices are usually afflicted by complex mixtures of multiple distortions, which are not necessarily well-modeled by the synthetic distortions found in existing databases. The originators of existing legacy databases usually conducted human psychometric studies to obtain statistically meaningful sets of human opinion scores on images in a stringently controlled visual environment, resulting in small data collections relative to other kinds of image analysis databases. Toward overcoming these limitations, we designed and created a new database that we call the LIVE In the Wild Image Quality Challenge Database, which contains widely diverse authentic image distortions on a large number of images captured using a representative variety of modern mobile devices. We also designed and implemented a new online crowdsourcing system, which we have used to conduct a very large-scale, multi-month image quality assessment (IQA) subjective study. Our database consists of over 350 000 opinion scores on 1162 images evaluated by over 8100 unique human observers. Despite the lack of control over the experimental environments of the numerous study participants, we demonstrate excellent internal consistency of the subjective data set. We also evaluate several top-performing blind IQA algorithms on it and present insights on how the mixtures of distortions challenge both end users as well as automatic perceptual quality prediction models. The new database is available for public use at http://live.ece.utexas.edu/research/ChallengeDB/index.html.