Human Perception Of Image Quality Research Articles

Learning to Assess Image Quality Like an Observer.

Human observers are the ultimate receivers and evaluators of the image visual information and have powerful perception ability of visual quality with short-term global perception and long-term regional observation. Thus, it is natural to design an image quality assessment (IQA) computational model to act like an observer for accurately predicting the human perception of image quality. Inspired by this, here, we propose a novel observer-like network (OLN) to perform IQA by jointly considering the global glimpsing information and local scanning information. Specifically, the OLN consists of a global distortion perception (GDP) module and a local distortion observation (LDO) module. The GDP module is designed to mimic the observer's global perception of image quality through performing classification of images' distortion categories and levels. Simultaneously, to simulate the human local observation behavior, the LDO module attempts to gather the long-term regional observation information of the distorted images by continuously tracing the human scanpath in the observer-like scanning manner. By leveraging the bilinear pooling layer to collaborate the short-term global perception with the long-term regional observation, our network precisely predicts the quality scores of distorted images, such as human observers. Comprehensive experiments on the public datasets powerfully demonstrate that the proposed OLN achieves state-of-the-art performance.

Federated learning based nonlinear two-stage framework for full-reference image quality assessment: An application for biometric

The non-linearity in medical image processing is a critical issue. Because the privacy of the medical image and loss of data is a major concern in recent years. Federated learning is a most advanced form of machine learning in which, rather than transmitting data to local server, a machine learning (ML) algorithm is installed to various devices to train on the information. The parameters from the separate modules will then be transferred to a master ML/ (deep learning) DL model for global training. The research of Image Quality Assessment (IQA) aims to simulate the process of human perception of image quality and construct an objective image quality model as consistent as possible with subjective assessment. The existing IQA methods can be roughly divided into traditional methods and deep learning methods. Traditional methods are knowledge-driven, using prior knowledge or assumptions about the human visual system (HVS) to heuristically design image quality index. Deep learning methods are data-driven, using a large amount of annotated data to learn the mapping from the image to its visual quality end-to-end. To effectively integrate traditional methods into deep networks and investigate the knowledge (model)-driven deep learning methods are the current mainstream trends in IQA research. In this paper, we take the contrary direction and improve traditional methods guided by the cue from deep learning methods. The main works include: 1. the employment of activation function ensure the nonlinear approximation ability of the neural network, here we first extend the two-stage framework widely used in the field of full-reference image quality assessment and propose a nonlinear two-stage framework. 2. Within this framework, we revisit the Edge Strength SIMlarity (ESSIM) algorithm that we previously published in IEEE Signal Processing Letters, and proposed the Nonlinear Edge Strength SIMlarity (NESSIM) algorithm. Experiments on public databases show that NESSIM can obtain good assessment results in traditional methods. • Improve traditional methods guided by the cue from deep learning methods. • Employment of activation function ensure the nonlinear approximation ability of the neural network. • Propose a nonlinear two-stage framework • Proposed the Nonlinear Edge Strength SIMlarity (NESSIM) algorithm.

The context effect for blind image quality assessment

Image quality assessment (IQA) is a process of visuo-cognitive, which is an essential stage in human interaction with the environment. The study of the context effect (Brown and Daniel, 1987) also shows that the evaluation results made by the human vision system (HVS) is related to the contrast between the distorted image and the background environment. However, the existing IQA methods carry out the quality evaluation that only depends on the distorted image itself and ignores the impact of environment to human perception. In this paper, we propose a novel blind image quality assessment(BIQA) based on the context effect. At first, we use a graphical model to describe how the context effect influences human perception of image quality. Based on the established graph, we construct the context relation between the distorted image and the background environment by the X. Han et al. (2015). Then the context features are extracted from the constructed relation, and the quality-related features are extracted by the fine-tuned neural network from the distorted image in pixel-wise. Finally, these features are concatenated to quantify image quality degradations and then regress to quality scores. In addition, the proposed method is adaptive to various deep neural networks. Experimental results show that the proposed method not only has the state-of-art performance on the synthetic distorted images, but also has a great improvement on the authentic distorted images.

Multi-layer noise reshaping and perceptual optimization for effective adversarial attack of images

Adversarial attack aims to fail the deep neural network by adding a small amount of perturbation to the input image, in which the attack success rate and resulting image quality are maximized under the lp norm perturbation constraint. However, the lp norm is not accurately correlated to human perception of image quality. Attack methods based on l0 norm constraint usually suffer from the high computational cost due to the iterative search for candidate pixels to modify. In this work, we explore how perceptual quality optimization can be incorporated into the adversarial attack design and propose a two-stage attack method to reshape the adversarial noise by an initial attack and optimize the visual quality of the attacked images without sacrificing the attack success rate. Specifically, we construct a visual attention network to generate a perceptual attention map to modulate the adversarial noise generated by a base attack method. The network is trained to maximize the visual quality in Structural Similarity Index Metric (SSIM) while achieving the same attack success rate. To improve the image perceptual quality further, we propose a fast search algorithm to perform an iterative block-wise pruning of the adversarial noise. We evaluate our method on the mini-ImageNet dataset against three different defense schemes. The results have demonstrated that our method can achieve better attack performance in image quality, attack success rate, and efficiency than the state-of-the-art attack methods.

Fine-Grained Image Quality Caption With Hierarchical Semantics Degradation.

Blind image quality assessment (BIQA), which is capable of precisely and automatically estimating human perceived image quality with no pristine image for comparison, attracts extensive attention and is of wide applications. Recently, many existing BIQA methods commonly represent image quality with a quantitative value, which is inconsistent with human cognition. Generally, human beings are good at perceiving image quality in terms of semantic description rather than quantitative value. Moreover, cognition is a needs-oriented task where humans are able to extract image contents with local to global semantics as they need. The mediocre quality value represents coarse or holistic image quality and fails to reflect degradation on hierarchical semantics. In this paper, to comply with human cognition, a novel quality caption model is inventively proposed to measure fine-grained image quality with hierarchical semantics degradation. Research on human visual system indicates there are hierarchy and reverse hierarchy correlations between hierarchical semantics. Meanwhile, empirical evidence shows that there are also bi-directional degradation dependencies between them. Thus, a novel bi-directional relationship-based network (BDRNet) is proposed for semantics degradation description, through adaptively exploring those correlations and degradation dependencies in a bi-directional manner. Extensive experiments demonstrate that our method outperforms the state-of-the-arts in terms of both evaluation performance and generalization ability.

An Effective Multi-Task Two-Stage Network with the Cross-Scale Training Strategy for Multi-Scale Image Super Resolution

Convolutional neural networks and the per-pixel loss function have shown their potential to be the best combination for super-resolving severely degraded images. However, there are still challenges, such as the massive number of parameters requiring prohibitive memory and vast computing and storage resources as well as time-consuming training and testing. What is more, the per-pixel loss measured by L2 and the Peak Signal-to-Noise Ratio do not correlate well with human perception of image quality, since L2 simply does not capture the intricate characteristics of human visual systems. To address these issues, we propose an effective two-stage hourglass network with multi-task co-optimization, which enables the entire network to focus on training and testing time and inherent image patterns such as local luminance, contrast, structure and data distribution. Moreover, to avoid overwhelming memory overheads, our model is capable of performing real-time single image multi-scale super-resolution, so it is memory-friendly, meaning that memory space is utilized efficiently. In addition, in order to best use the underlying structure and perception of image quality and the intermediate estimates during the inference process, we introduce a cross-scale training strategy with 2×, 3× and 4× image super-resolution. This effective multi-task two-stage network with the cross-scale strategy for multi-scale image super-resolution is named EMTCM. Quantitative and qualitative experiment results show that the proposed EMTCM network outperforms state-of-the-art methods in recovering high-quality images.

No-reference image quality assessment for contrast-changed images via a semi-supervised robust PCA model

Contrast plays an important role in human perception of image quality. In this paper, we propose a metric for no-reference quality assessment of contrast-changed images by using a novel semi-supervised robust PCA, which can realize feature selection and denoising simultaneously, guided by the available supervisory information. To select features adaptively, the information-oriented features (e.g. entropy and natural scene statistics) and appearance-oriented features (e.g. colorfulness) are adopted. The proposed model is formulated as a constraint optimization problem, which is further casted to a convex problem and solved via augmented Lagrangian multiplier method. Extensive experimental results on CCID2014, CSIQ, SIQAD and TID2013 databases show that the proposed semi-supervised image quality metric based on robust PCA (SIQMR) provides a more accurate prediction than other metrics on the human perception of contrast variations.

Perceptual Image Quality Assessment for Various Viewing Conditions and Display Systems

From complete darkness to direct sunlight, real-world displays operate in various viewing conditions often resulting in a non-optimal viewing experience. Most existing Image Quality Assessment (IQA) methods, however, assume ideal environments and displays, and thus cannot be used when viewing conditions differ from the standard. In this paper, we investigate the influence of ambient illumination level and display luminance on human perception of image quality. We conduct a psychophysical study to collect a novel dataset of over 10000 image quality preference judgments performed in illumination conditions ranging from 0 lux to 20000 lux. We also propose a perceptual IQA framework that allows most existing image quality metrics (IQM) to accurately predict image quality for a wide range of illumination conditions and display parameters. Our analysis demonstrates strong correlation between human IQA and the predictions of our proposed framework combined with multiple prominent IQMs and across a wide range of luminance values.

No-reference image quality assessment based on quality patches in real time

Image quality assessment methods quantify the quality of an image that is highly correlated with human-perceived image quality. In this paper, the image quality is represented by the image patches that are selected by applying the spatial correlation of pixel-pairs. The quality image patches are selected from the database LIVE2, and quality feature detectors are learned by running the FastICA algorithm on the patches. Then being quality features, the independent component coefficients are found out from each quality image patch. A Hash lookup table is built by the binarization of the independent component coefficients of quality image patches, and the Hash table can be indexed by the binary code of the independent component coefficients of a tested image. The results of proposed approach were compared with results from other methods of image quality assessment and with the subjective image quality evaluated scores. And the experimental results expressed that the proposed metric method of no-reference image quality assessment could accurately measure the distorted degree of images. The Pearson linear correlation coefficient (PCC) achieves a high value 0.949 for the accuracy of evaluating results. The Spearman order correlation (SRC) achieves a high value 0.996 for the monotonicity of evaluating results. And the root mean square error (RMSE) is 5.917 for the subjective consistency of evaluating results. Extra evaluating aerial images, the proposed method obtained the result that the foggy weather led to the worst quality.

Modeling Human Perception of Image Quality.

Humans can determine image quality instantly and intuitively, but the mechanism of human perception of image quality is unknown. The purpose of this work was to identify the most important quantitative metrics responsible for the human perception of digital image quality. Digital images from two different datasets-CT tomography (MedSet) and scenic photographs of trees (TreeSet)-were presented in random pairs to unbiased human viewers. The observers were then asked to select the best-quality image from each image pair. The resulting human-perceived image quality (HPIQ) ranks were obtained from these pairwise comparisons with two different ranking approaches. Using various digital image quality metrics reported in the literature, we built two models to predict the observed HPIQ rankings, and to identify the most important HPIQ predictors. Evaluating the quality of our HPIQ models as the fraction of falsely predicted pairwise comparisons (inverted image pairs), we obtained 70-71% of correct HPIQ predictions for the first, and 73-76%for the second approach. Taking into account that 10-14% of inverted pairs were already present in the original rankings, limitations of the models, and only a few principal HPIQ predictors used, we find this result very satisfactory. We obtained a small set of most significant quantitative image metrics associated with the human perception of image quality. This can be used for automatic image quality ranking, machine learning, and quality-improvement algorithms.

Blind image quality prediction by exploiting multi-level deep representations

Blind image quality assessment (BIQA) aims at precisely estimating human perceived image quality with no access to a reference. Recently, several attempts have been made to develop BIQA methods based on deep neural networks (DNNs). Although these methods obtained promising performance, they have some limitations: (1) their DNN models are actually ”shallow” in term of depth; and (2) these methods typically use the output of the last layer in the DNN model as the feature representation for quality prediction. Since the representation depth has been demonstrated beneficial for various vision tasks, it is significant to explore very deep networks for learning BIQA models. Besides, the information in the last layer may unduly generalize over local artifacts which are highly related to quality degradation. On the contrary, intermediate layers may be sensitive to local degradations but will not capture high-level semantics. Thus, reasoning at multiple levels of representation is necessary in the IQA task. In this paper, we propose to extract multi-level representations from a very deep DNN model for learning an effective BIQA model, and consequently present a simple but extraordinarily effective BIQA framework, codenamed BLINDER (BLind Image quality predictioN via multi-level DEep Representations). Thorough experiments have been conducted on five standard databases, which show that a significant improvement can be achieved by adopting multi-level deep representations. Besides, BLINDER considerably outperforms previous state-of-the-art BIQA methods for authentically distorted images.

PATCH-IQ: A patch based learning framework for blind image quality assessment

Most well-known blind image quality assessment (BIQA) models usually follow a two-stage framework whereby various types of features are first extracted and used as an input to a regressor. The regression algorithm is used to model human perceptual measures based on a training set of distorted images. However, this approach requires an intensive training phase to optimise the regression parameters. In this paper, we overcome this limitation by proposing an alternative BIQA model that predicts image quality using nearest neighbour methods which have virtually zero training cost. The model, termed PATCH based blind Image Quality assessment (PATCH-IQ), has a learning framework that operates at the patch level. This enables PATCH-IQ to provide not only a global image quality estimation but also a local image quality estimation. Based on the assumption that the perceived quality of a distorted image will be best predicted by features drawn from images with the same distortion class, PATCH-IQ also introduces a distortion identification stage in its framework. This enables PATCH-IQ to identify the distortion affecting the image, a property that can be useful for further local processing stages. PATCH-IQ is evaluated on the standard IQA databases, and the provided scores are highly correlated to human perception of image quality. It also delivers competitive prediction accuracy and computational performance in relationship to other state-of-the-art BIQA models.

DeepSim: Deep similarity for image quality assessment

This paper studies one interesting problem: how does the deep neural network (DNN) architecture affect the image quality assessment (IQA) performance? In order to find the answer, we propose a novel full-reference IQA framework, codenamed deep similarity (DeepSim). In DeepSim, we first measure the local similarities between the features (produced by a DNN model) of the test image and those of the reference image; Afterwards, the local quality indices are gradually pooled together to estimate the overall quality score. In addition, various factors that may affect the IQA performance are investigated. Thorough experiments conducted on standard databases show that: (1) DeepSim can accurately predict human perceived image quality and outperforms previous state-of-the-art; (2) mid-level representations are most effective for quality prediction; and (3) preprocessing, the restricted linear units and max-pooling operations are beneficial for the IQA performance.

Nonparametric Quality Assessment of Natural Images

In this article, the authors explore an alternative way to perform no-reference image quality assessment (NR-IQA). Following a feature extraction stage in which spatial domain statistics are utilized as features, a two-stage nonparametric NR-IQA framework is proposed. This approach requires no training phase, and it enables prediction of the image distortion type as well as local regions' quality, which is not available in most current algorithms. Experimental results on IQA databases show that the proposed framework achieves high correlation to human perception of image quality and delivers competitive performance to state-of-the-art NR-IQA algorithms.

No reference image quality assessment using sparse feature representation in two dimensions spatial correlation

Universal no reference image quality assessment is attracting significant attention in the fields of image processing, visual and machine learning. This study presents a novel method to evaluate the image quality from human subjective scores of the training samples by regressing. The primary characteristic of this novel method is the learning dictionary derived from extracting features of two dimensional spatial correlations of the sample images. Each atom in the dictionary includes 10 elements. They are DMOS (differential mean opinion score), three extracted features and their corresponding image structure patches and PCMs (pixel correlative matrix). The three extracted features are the standard deviation, the gray scale deviation and the distribution width. During the quality assessment, a distorted image is transformed into an image with structural information and partitioned into patches. The patch with the largest feature value is selected and represented sparsely in the learning dictionary. Afterwards the image quality index is obtained by quantifying the sparse representation coefficients, DMOS values and feature values. Comparing with other image quality assessment models, the proposed NSRCIQ method is simple and effective. The resulted IQA scores have not only comparable accuracy, but also high linearity to human perception of image quality. Moreover, the algorithm can be implemented in real-time.

Biologically inspired image quality assessment

Image quality assessment (IQA) aims at developing computational models that can precisely and automatically estimate human perceived image quality. To date, various IQA methods have been proposed to mimic the processing of the human visual system, with limited success. Here, we present a novel IQA approach named biologically inspired feature similarity (BIFS), which is demonstrated to be highly consistent with the human perception. In the proposed approach, biologically inspired features (BIFs) of the test image and the relevant reference image are first extracted. Afterwards, local similarities between the reference BIFs and the distorted ones are calculated and then combined to obtain a final quality index. Thorough experiments on a number of IQA databases demonstrate that the proposed method is highly effective and robust, and outperform state-of-the-art FR-IQA methods across various datasets.

Face Detection on Distorted Images Augmented by Perceptual Quality-Aware Features

Motivated by the proliferation of low-cost digital cameras in mobile devices being deployed in automated surveillance networks, we study the interaction between perceptual image quality and a classic computer vision task of face detection. We quantify the degradation in performance of a popular and effective face detector when human-perceived image quality is degraded by distortions commonly occurring in capture, storage, and transmission of facial images, including noise, blur, and compression. It is observed that, within a certain range of perceived image quality, a modest increase in image quality can drastically improve face detection performance. These results can be used to guide resource or bandwidth allocation in acquisition or communication/delivery systems that are associated with face detection tasks. A new set of features, called qualHOG, are proposed for robust facedetection that augments face-indicative Histogram of Oriented Gradients (HOG) features with perceptual quality-aware spatial Natural Scene Statistics (NSS) features. Face detectors trained on these new features provide statistically significant improvement in tolerance to image distortions over a strong baseline. Distortiondependent and distortion-unaware variants of the face detectors are proposed and evaluated on a large database of face images representing a wide range of distortions. A biased variant of the training algorithm is also proposed that further enhances the robustness of these face detectors. To facilitate this research, we created a new distorted face database (DFD), containing face and non-face patches from images impaired by a variety of common distortion types and levels. This new data set and relevant code are available for download and further experimentation at www.live.ece.utexas.edu/research/Quality/index.htm.

No-Reference Quality Assessment of Contrast-Distorted Images Based on Natural Scene Statistics

Contrast distortion is often a determining factor in human perception of image quality, but little investigation has been dedicated to quality assessment of contrast-distorted images without assuming the availability of a perfect-quality reference image. In this letter, we propose a simple but effective method for no-reference quality assessment of contrast distorted images based on the principle of natural scene statistics (NSS). A large scale image database is employed to build NSS models based on moment and entropy features. The quality of a contrast-distorted image is then evaluated based on its unnaturalness characterized by the degree of deviation from the NSS models. Support vector regression (SVR) is employed to predict human mean opinion score (MOS) from multiple NSS features as the input. Experiments based on three publicly available databases demonstrate the promising performance of the proposed method.

Subjective evaluation of image quality measures for white noise and Gaussian blur-distorted images

Image quality assessment has diverse applications. A number of image quality measures (IQMs) are proposed, but none is proved to be true representative of human perception of image quality. We have subjectively investigated spectral distance-based and human visual system (HVS)-based IQMs for their effectiveness in representing the human perception for images corrupted with white noise and Gaussian blur. Two sets of 160 images with various degrees of white noise and Gaussian blur are subjectively evaluated by 50 human subjects, resulting in 16 000 human judgments. On the basis of evaluations, image-independent human perception values are calculated. The perception values are plotted against spectral distance-based and HVS-based IQMs. The performance of quality measures is determined by graphical observations and polynomial curve fitting, resulting in best performance by HVS absolute norm and block spectral phase-magnitude error for white noise and Gaussian blur distortions, respectively. It is also observed that the performances of various quality measures differ for different noise distortions, suggesting the use of different quality measures for different noise types rather than a single universal IQM.

A New Image Quality Metric using System Visual Human Characteristics

Several human visual system (HVS) based quality metrics have been developed in recent years to measure the quality distortions caused by digital image coding techniques. Because of the complicated nature of the HVS characteristics, these metrics do not provide acceptable correlation with perceptual evaluations of the distortion. Recent studies by the Visual Quality Experts Group (VQEG) also show that the current HVS-based techniques do not provide a clear advantage over a mathematically defined technique such as mean square error (MSE). Therefore, this paper proposes a new quality metric based on the dynamic characteristics and on the just-noticeable-difference threshold characteristics introduced by Weber's law: the relative weighted Peak to Signal Noise Ratio. Our experiments on various image distortion types indicate that our metric performs significantly better than the widely used distortion metric Mean Squared Error. We should that the rwPSNR correlates with human perception of image quality.