IQA Model Research Articles

Black-box adversarial attacks against image quality assessment models

The problem of No-Reference Image Quality Assessment (NR-IQA) is to predict the perceptual quality of an image in line with its subjective evaluation. However, the vulnerabilities of NR-IQA models to the adversarial attacks have not been thoroughly studied for model refinement. This paper aims to investigate the potential loopholes of NR-IQA models via black-box adversarial attacks. Specifically, we first formulate the attack problem as maximizing the deviation between the estimated quality scores of original and perturbed images, while restricting the perturbed image distortions for visual quality preservation. Under such formulation, we then design a Bi-directional loss function to mislead the estimated quality scores of adversarial examples towards an opposite direction with maximum deviation. On this basis, we finally develop an efficient and effective black-box attack method for NR-IQA models based on a random search paradigm. Comprehensive experiments on three benchmark datasets show that all evaluated NR-IQA models are significantly vulnerable to the proposed attack method. After being attacked, the average change rates in terms of two well-known IQA performance metrics achieved by victim models reach 97% and 101%, respectively. In addition, our attack method also outperforms a newly introduced black-box attack approach on IQA models. We also observe that the generated perturbations are not transferable, which points out a new research direction in NR-IQA community. The source code is available at https://github.com/GZHU-DVL/AttackIQA.

Toward a blind image quality evaluator in the wild by learning beyond human opinion scores

Nowadays, most existing blind image quality assessment (BIQA) models inthewild heavily rely on human ratings, which are extraordinarily labor-expensive to collect. Here, we propose an opinion−free BIQA method that learns from multiple annotators to assess the perceptual quality of images captured in the wild. Specifically, we first synthesize distorted images based on the pristine counterparts. We then randomly assemble a set of image pairs from the synthetic images, and use a group of IQA models to assign pseudo-binary labels for each pair indicating which image has higher quality as the supervisory signal. Based on the newly established pseudo-labeled dataset, we train a deep neural network (DNN)-based BIQA model to rank the perceptual quality, optimized for consistency with the binary rank labels. Since there exists domain shift, e.g., distortion shift and content shift, between the synthetic and in-the-wild images, we leverage two ways to alleviate this issue. First, the simulated distortions should be similar to authentic distortions as much as possible. Second, an unsupervised domain adaptation (UDA) module is further applied to encourage learning domain-invariant features between two domains. Extensive experiments demonstrate the effectiveness of our proposed opinion−free BIQA model, yielding SOTA performance in terms of correlation with human opinion scores, as well as gMAD competition. Our code is available at: https://github.com/wangzhihua520/OF_BIQA.

Deep Ordinal Regression Framework for No-Reference Image Quality Assessment

Due to the rapid development of deep learning techniques, no-reference image quality assessment (NR-IQA) has achieved significant improvement. NR-IQA aims to predict a real-valued variable for image quality, using the image in question as the sole input. Existing deep learning-based NR-IQA models are formulated as a regression problem and trained by minimising the mean squared error. The error measurement does not consider the relative ordering between different ratings on the quality scale, which consequently affects the efficacy of the model. To account for this problem, we reformulate NR-IQA learning as an ordinal regression problem and propose a simple yet effective framework using deep convolutional neural networks (DCNN) and Transformers. NR-IQA learning is achieved by a deep ordinal loss and using a soft ordinal inference to transform the predicted probabilities to a continuous variable for image quality. Experimental results demonstrate the superiority of our proposed NR-IQA model based on deep ordinal regression. In addition, this framework can be easily extended with various DCNN architectures to build advanced IQA models.

CARNet: Context-Aware Residual Learning for JPEG-LS Compressed Remote Sensing Image Restoration

JPEG-LS (a lossless (LS) compression standard developed by the Joint Photographic Expert Group) compressed image restoration is a significant problem in remote sensing applications. It faces the following two challenges: first, bridging small pixel-value gaps from wide numerical ranges; and second, removing banding artifacts in the condition of lacking available context information. As far as we know, there is currently no research dealing with the above issues. Hence, we develop this initial line of work on JPEG-LS compressed remote sensing image restoration. We propose a novel CNN model called CARNet. Its core idea is a context-aware residual learning mechanism. Specifically, it realizes residual learning for accurate restoration by adopting a scale-invariant baseline. It enables large receptive fields for banding artifact removal through a context-aware scheme. Additionally, it eases the information flow among stages by utilizing a prior-guided feature-fusion mechanism. Alternatively, we design novel R IQA models to provide a better restoration performance assessment for our study by utilizing gradient priors of JPEG-LS banding artifacts. Furthermore, we prepare a new dataset of JPEG-LS compressed remote sensing images to supplement existing benchmark data. Experiments show that our method sets the state-of-the-art for JPEG-LS compressed remote sensing image restoration.

Systemic distortion analysis with deep distortion directed image quality assessment models

An image processing system may bring distortions to the images going through it due to its inherent attributes or defects. Identifying the distortion types in a system is important to solve the system defects and improve its service quality. In this paper, we propose a novel algorithm to analyse the systemic distortions with multiple deep distortion directed image quality assessment models. Specifically, We pre-train several image quality assessment models that target to different distortions respectively. We use multiple images as anchor to test the system. We use the pre-trained IQA models to evaluate the anchor images that have gone through the system, and explore the features about the relationship between distortion types and quality scores. To analyse multiple distortions in a system, we embed dictionary learning into the multi-label classification framework. The experimental results demonstrate that our algorithm has good effectiveness, robustness and extensibility.

Image Quality Assessment Using Contrastive Learning.

We consider the problem of obtaining image quality representations in a self-supervised manner. We use prediction of distortion type and degree as an auxiliary task to learn features from an unlabeled image dataset containing a mixture of synthetic and realistic distortions. We then train a deep Convolutional Neural Network (CNN) using a contrastive pairwise objective to solve the auxiliary problem. We refer to the proposed training framework and resulting deep IQA model as the CONTRastive Image QUality Evaluator (CONTRIQUE). During evaluation, the CNN weights are frozen and a linear regressor maps the learned representations to quality scores in a No-Reference (NR) setting. We show through extensive experiments that CONTRIQUE achieves competitive performance when compared to state-of-the-art NR image quality models, even without any additional fine-tuning of the CNN backbone. The learned representations are highly robust and generalize well across images afflicted by either synthetic or authentic distortions. Our results suggest that powerful quality representations with perceptual relevance can be obtained without requiring large labeled subjective image quality datasets. The implementations used in this paper are available at https://github.com/pavancm/CONTRIQUE.

SPIQ: A Self-Supervised Pre-Trained Model for Image Quality Assessment

Blind image quality assessment (BIQA) has witnessed a flourishing progress due to the rapid advances in deep learning technique. The vast majority of prior BIQA methods try to leverage models pre-trained on ImageNet to mitigate the data shortage problem. These well-trained models, however, can be sub-optimal when applied to BIQA task that varies considerably from the image classification domain. To address this issue, we make the first attempt to leverage the plentiful unlabeled data to conduct self-supervised pre-training for BIQA task. Based on the distorted images generated from the high-quality samples using the designed distortion augmentation strategy, the proposed pre-training is implemented by a feature representation prediction task. Specifically, patch-wise feature representations corresponding to a certain grid are integrated to make prediction for the representation of the patch below it. The prediction quality is then evaluated using a contrastive loss to capture quality-aware information for BIQA task. Experimental results conducted on KADID-10 k and KonIQ-10 k databases demonstrate that the learned pre-trained model can significantly benefit the existing learning based IQA models.

An efficient approach for no-reference image quality assessment based on statistical texture and structural features

This paper presents a rotation-invariant and computationally efficient no-reference image quality assessment (NR-IQA) model. It estimates the image quality based on texture and structural information associated with the images. The human visual system (HVS) uses perceptual features such as texture and structure as primary information to understand the visual scene and the image content. Moreover, the texture and structural information capture the loss of naturalness due to distortions in the image. Therefore, in this work, the important texture features are extracted using the local binary patterns (LBP).The modified LBP, also known as the hyper-smoothing LBP (H-LBP) and Laplacian of H-LBP (LH-LBP), represents the image structure. Further, the image quality prediction model computes the quality of the image based on the statistical feature measures of the texture and structural information. In the proposed approach, the image quality prediction model uses support vector regression (SVR) to measure image quality. Various experimentations are carried out on the LIVE and TID2013 database to test the effectiveness of the proposed NR-IQA model. The performance metrics such as Spearman rank-ordered correlation coefficient, Pearson linear correlation coefficient, and root mean square error is computed to show the efficiency of the presented approach. The experimental results illustrate a high correlation between the predicted quality score and the human visual perceptions. It is also found to be competitive with the best-performing full-reference and no-reference IQA models.

CT image quality assessment based on prior information of pre-restored images

We propose a CT IQA strategy based on the prior information of pre-restored images (PR-IQA) to improve the performance of IQA models. We propose a CNN-based no-reference CT IQA strategy using the prior information of image quality features in the image restoration algorithm, which is combined with the original distorted image information into the two CNNs through the pre-restored image and the residual image. Multi-information fusion was used to improve the feature extraction ability and prediction performance of CNN. We built a CT IQA dataset based on spiral CT data published by Mayo Clinic. The performance of PR- IQA was evaluated by calculating the quantitative metrics and statistical tests. The influence of different hyperparameter settings for PR-IQA was analyzed. We then compared PR-IQA with the BASELINE model based on the single CNN to evaluate the original distorted image without reference image and other eight IQA algorithms. The comparative experiment results showed that the PR-IQA model based on the prior information of 3 different image restoration algorithms (BF, NLM and BM3D) was better than all the tested IQA algorithms. Compared with the BASELINE method, the proposed method showed significantly improved performance, and the mean PLCC was increased by 12.56% and SROCC by 19.95%, and RMSE was decreased by 22.77%. The proposed PR-IQA method can make full use of the prior information of the image restoration algorithm to effectively predict the quality of CT images.

A comprehensive assessment of content-based image retrieval using selected full reference image quality assessment algorithms

In the area of full-reference (FR) objective ‘image quality assessment’ (IQA), a huge amount of improvement has been done in the last few years that can calculate image quality, consistently. Contrarily, query-based image/video databases and search engines retrieve related data using ‘ranking and indexing’ depending on stored content. It is also to be noted that both techniques use feature extraction to achieve their goal. The efficiency of seven selected FR-IQA schemes is described in this article for the retrieval of an image signal. Extensive tests are done on two freely accessible databases. The comparison results express that mean-structural-similarity-index-measure (MSSIM), and feature-similarity-index-measure (FSIM) offer superior outcome than other IQA models. Our assessment outcome and the related thought will be very much useful for the researchers to understand the latest application areas of the IQA model in the area of image searching and retrieval.

Blind Omnidirectional Image Quality Assessment Based on Structure and Natural Features

Omnidirectional image quality assessment (OIQA) is a hot research topic in image processing. Besides, IQA is also very important in the field of instrumentation and measurement. In this article, we propose a new no-reference OIQA model by analyzing the relationship between image features and perceived quality. A gradient weighted local phrase quantization map is built to complement the gradient map to model the structure degradation of omnidirectional images caused by various distortions. Considering the large amount of information contained in the omnidirectional image, we extract the luminance features, global entropy instead of local entropy, and color features, which are ignored in most previous works to estimate the naturalness quality of the omnidirectional image. Finally, support vector regression is utilized to train and test our model based on the image features and human subjective quality assessment scores. The experiments on two databases show that our model outperforms the state-of-the-art IQA models and correlates well with subjective scores in the two databases.

Visual Quality Assessment for Perceptually Encrypted Light Field Images

Perceptual encryption has received widespread attention as a technology for protecting multimedia visual information. In multimedia applications, perceptual encryption only protects a portion of the content without previewing all the information. At present, perceptual encryption is mostly oriented to conventional plain images while few methods aim at visual security measures for perceptually encrypted images. Existing solutions usually adopt well-known quality assessment metrics to measure the visual quality of encrypted images. However, they often exhibit undesired behavior on perceptually encrypted images with low quality. As a typical representation of three-dimensional scenes, light field images record the intensity and direction of light during propagation which is distinct from 2-D images. In this paper, we construct a perceptually encrypted light field image database (PE-LFID) for quality assessment based on 14 reference light field plaintext images. For each scene, we employ four encryption methods, each of which has six levels. Additionally, a novel visual security evaluation method based on PE-LFID is proposed by taking into account the local and global features of the light field images. First, we use a multi-threshold edge detection method to obtain the edge similarity in the spatial domain of the light field image. Afterwards, the epipolar plane image (EPI) generated from the angular domain of the light field image is used to calculate the gradient magnitude similarity, which is expressed as a global feature. Furthermore, the final quality prediction score is calculated by adaptively weighting between local and global features. We conduct extensive experiments on the proposed PE-LFID to assess the performance of classical and state-of-the-art IQA models. The experimental results demonstrate the effectiveness of the proposed method for visual security evaluation of perceptually encrypted light field images, as well as the scalability of PE-LFID.

Quality Prediction on Deep Generative Images.

In recent years, deep neural networks have been utilized in a wide variety of applications including image generation. In particular, generative adversarial networks (GANs) are able to produce highly realistic pictures as part of tasks such as image compression. As with standard compression, it is desirable to be able to automatically assess the perceptual quality of generative images to monitor and control the encode process. However, existing image quality algorithms are ineffective on GAN generated content, especially on textured regions and at high compressions. Here we propose a new "naturalness"-based image quality predictor for generative images. Our new GAN picture quality predictor is built using a multi-stage parallel boosting system based on structural similarity features and measurements of statistical similarity. To enable model development and testing, we also constructed a subjective GAN image quality database containing (distorted) GAN images and collected human opinions of them. Our experimental results indicate that our proposed GAN IQA model delivers superior quality predictions on the generative image datasets, as well as on traditional image quality datasets.

Study of 3D Virtual Reality Picture Quality

Virtual Reality (VR) and its applications have attracted significant and increasing attention. However, the requirements of much larger file sizes, different storage formats, and immersive viewing conditions pose significant challenges to the goals of acquiring, transmitting, compressing and displaying high quality VR content. Towards meeting these challenges, it is important to be able to understand the distortions that arise and that can affect the perceived quality of displayed VR content. It is also important to develop ways to automatically predict VR picture quality. Meeting these challenges requires basic tools in the form of large, representative subjective VR quality databases on which VR quality models can be developed and which can be used to benchmark VR quality prediction algorithms. Towards making progress in this direction, here we present the results of an immersive 3D subjective image quality assessment study. In the study, 450 distorted images obtained from 15 pristine 3D VR images modified by 6 types of distortion of varying severities were evaluated by 42 subjects in a controlled VR setting. Both the subject ratings as well as eye tracking data were recorded and made available as part of the new database, in hopes that the relationships between gaze direction and perceived quality might be better understood. We also evaluated several publicly available IQA models on the new database, and also report a statistical evaluation of the performances of the compared IQA models.

Multivariate Statistical Approach to Image Quality Tasks

Many existing Natural Scene Statistics-based no reference image quality assessment (NR IQA) algorithms employ univariate parametric distributions to capture the statistical inconsistencies of bandpass distorted image coefficients. Here we propose a multivariate model of natural image coefficients expressed in the bandpass spatial domain that has the potential to capture higher-order correlations that may be induced by the presence of distortions. We analyze how the parameters of the multivariate model are affected by different distortion types, and we show their ability to capture distortion-sensitive image quality information. We also demonstrate the violation of Gaussianity assumptions that occur when locally estimating the energies of distorted image coefficients. Thus we propose a generalized Gaussian-based local contrast estimator as a way to implement non-linear local gain control, that facilitates the accurate modeling of both pristine and distorted images. We integrate the novel approach of generalized contrast normalization with multivariate modeling of bandpass image coefficients into a holistic NR IQA model, which we refer to as multivariate generalized contrast normalization (MVGCN). We demonstrate the improved performance of MVGCN on quality relevant tasks on multiple imaging modalities, including visible light image quality prediction and task success prediction on distorted X-ray images.

An Enhanced LSDBIQ Algorithm for Full Reference Image Quality Assessment for Multi Distorted Images

Image processing is an emerging technology as image is used in various fields like medical and education. Images may corrupt due to the various categories of noises. Image quality reduces because of the image acquisition or transmission. Noise reduction is the main focus to retain the quality of the image. For the removal of this noise, there are various techniques and filters. Before applying further processing on the image, noise should be removed from the image. In this paper we deal with with a practical and effectual IQA model, called LSDBIQ (local standard deviation based image quality). This metric is examined on a well known database MDID (multi distorted image dataset). Exploratory results manifest that this metric perform better than alternative techniques for the assessment of image quality and have very low computational complexity.

No-Reference Quality Assessment of Tone-Mapped HDR Pictures.

Being able to automatically predict digital picture quality, as perceived by human observers, has become important in many applications where humans are the ultimate consumers of displayed visual information. Standard dynamic range (SDR) images provide 8 b/color/pixel. High dynamic range (HDR) images, which are usually created from multiple exposures of the same scene, can provide 16 or 32 b/color/pixel, but must be tonemapped to SDR for display on standard monitors. Multi-exposure fusion techniques bypass HDR creation, by fusing the exposure stack directly to SDR format while aiming for aesthetically pleasing luminance and color distributions. Here, we describe a new no-reference image quality assessment (NR IQA) model for HDR pictures that is based on standard measurements of the bandpass and on newly conceived differential natural scene statistics (NSS) of HDR pictures. We derive an algorithm from the model which we call the HDR IMAGE GRADient-based Evaluator. NSS models have previously been used to devise NR IQA models that effectively predict the subjective quality of SDR images, but they perform significantly worse on tonemapped HDR content. Toward ameliorating this we make here the following contributions: 1) we design HDR picture NR IQA models and algorithms using both standard space-domain NSS features as well as novel HDR-specific gradient-based features that significantly elevate prediction performance; 2) we validate the proposed models on a large-scale crowdsourced HDR image database; and 3) we demonstrate that the proposed models also perform well on legacy natural SDR images. The software is available at: http://live.ece.utexas.edu/research/Quality/higradeRelease.zip.

Perceptual Reduced-Reference Visual Quality Assessment for Contrast Alteration

In image/video systems, contrast adjustment which manages to enhance visual quality is nowadays an important research topic. Yet very limited struggles have been made to the exploration of visual quality assessment for contrast adjustment. To tackle the issue, this paper proposes a novel reduced-reference (RR) quality metric with the integration of bottom-up and top-down strategies. The former one stems from the recently revealed free energy principle that tells that the human visual system seeks to comprehend an input image via uncertainty removal, while the latter one is toward using the symmetric Kullback–Leibler divergence to compare the histogram of the contrast-altered image with that of the pristine image. The bottom-up and top-down strategies are lastly incorporated to derive the RR contrast-altered image quality measure. A comparison using numerous existing IQA models is carried out on five contrast related databases/subsets in CID2013, CCID2014, CSIQ, TID2008, and TID2013, and experimental results validate the superiority of the proposed technique.

A No Reference Image Quality Assessment Metric Based on Visual Perception

Nowadays, how to evaluate image quality reasonably is a basic and challenging problem. In view of the present no reference evaluation methods, they cannot reflect the human visual perception of image quality accurately. In this paper, we propose an efficient general-purpose no reference image quality assessment (NRIQA) method based on visual perception, and effectively integrates human visual characteristics into the NRIQA fields. First, a novel algorithm for salient region extraction is presented. Two characteristics graphs of texture and edging of the original image are added to the Itti model. Due to the normalized luminance coefficients of natural images obey the generalized Gauss probability distribution, we utilize this characteristic to extract statistical features in the regions of interest (ROI) and regions of non-interest respectively. Then, the extracted features are fused to be an input to establish the support vector regression (SVR) model. Finally, the IQA model obtained by training is used to predict the quality of the image. Experimental results show that this method has good predictive ability, and the evaluation effect is better than existing classical algorithms. Moreover, the predicted results are more consistent with human subjective perception, which can accurately reflect the human visual perception to image quality.

Local Standard Deviation Based Image Quality Metrics for JPEG Compressed Images

In this paper, we address the Full-Reference (FR) Image Quality Metric (IQM) to assess the quality of JPEG-coded images and we present a new effective and efficient IQA model, called Local Standard Deviation Based Image Quality (LSDBIQ). The approach is based on the comparison of the local standard deviation of two images. The proposed metrics is tested on four well-known databases available in the literature (TID2013, TID2008, LIVE and CSIQ). Experimental results show that the proposed metrics outperforms other models for the assessment of image quality and have very low computational complexity.