Full-reference Image Quality Assessment Methods Research Articles

Hierarchical Degradation-aware Network for Full-Reference Image Quality Assessment

Full-Reference Image Quality Assessment (FR-IQA) algorithms excel in evaluating perceptual distortions by comparing reference and distorted images. However, as the severity and quantity of distortions in datasets increase, existing FR-IQA methods struggle to capture complex nonlinear perceptual features. This limitation results in reduced adaptability and inaccurate assessments for images with more severe or multiple distortions. Recognizing the importance of understanding image degradation mechanisms, we propose a novel hierarchical degradation-aware network (HDaN) method. First, by exploring the degradation mechanisms from the reference image to the distorted image, our degradation network matches distortions that align more closely with the human visual system (HVS). Next, we design a convertor to project the matched features into multiple spaces, creating multidimensional feature representations that more comprehensively capture the complexity of image distortions rather than being confined to a single feature space. Then, we calculate a similarity matrix between the distorted and mapped features, selecting the most similar (top-k) features for merging. Finally, a regression network maps the merged features to quality scores, providing the final quality prediction. The experimental results demonstrate that our proposed HDaN method outperforms traditional deep learning-based FR-IQA methods. Specifically, the HDaN shows higher PLCC and SROCC metrics on benchmark datasets, significantly improving over existing methods. Moreover, the method exhibits better adaptability to images with varying degrees and types of distortions, thereby greatly enhancing the overall performance of IQA.

Low-dose computed tomography perceptual image quality assessment

In computed tomography (CT) imaging, optimizing the balance between radiation dose and image quality is crucial due to the potentially harmful effects of radiation on patients. Although subjective assessments by radiologists are considered the gold standard in medical imaging, these evaluations can be time-consuming and costly. Thus, objective methods, such as the peak signal-to-noise ratio and structural similarity index measure, are often employed as alternatives. However, these metrics, initially developed for natural images, may not fully encapsulate the radiologists’ assessment process. Consequently, interest in developing deep learning-based image quality assessment (IQA) methods that more closely align with radiologists’ perceptions is growing. A significant barrier to this development has been the absence of open-source datasets and benchmark models specific to CT IQA. Addressing these challenges, we organized the Low-dose Computed Tomography Perceptual Image Quality Assessment Challenge in conjunction with the Medical Image Computing and Computer Assisted Intervention 2023. This event introduced the first open-source CT IQA dataset, consisting of 1,000 CT images of various quality, annotated with radiologists’ assessment scores. As a benchmark, this challenge offers a comprehensive analysis of six submitted methods, providing valuable insight into their performance. This paper presents a summary of these methods and insights. This challenge underscores the potential for developing no-reference IQA methods that could exceed the capabilities of full-reference IQA methods, making a significant contribution to the research community with this novel dataset. The dataset is accessible at https://zenodo.org/records/7833096.

Full-reference calibration-free image quality assessment

Objective Image Quality Assessment (IQA) methods often lack of linearity of their quality estimates with respect to scores expressed by human subjects and therefore IQA metrics undergo a calibration process based on subjective quality examples. However, example-based training presents a challenge in terms of generalization hampering result comparison across different applications and operative conditions. In this paper, new Full Reference (FR) techniques, providing estimates linearly correlated with human scores without using calibration are introduced. We show that on natural images, application of estimation theory and psychophysical principles to images degraded by Gaussian blur leads to a so-called canonical IQA method, whose estimates are linearly correlated to both the subjective scores and the viewing distance. Then, we show that any mainstream IQA methods can be reconducted to the canonical method by converting its metric based on a unique specimen image. The proposed scheme is extended to wide classes of degraded images, e.g. noisy and compressed images. The resulting calibration-free FR IQA methods allows for comparability and interoperability across different imaging systems and on different viewing distances. A comparison of their statistical performance with respect to state-of-the-art calibration prone methods is finally provided, showing that the presented model is a valid alternative to the final 5-parameter calibration step of IQA methods, and the two parameters of the model have a clear operational meaning and are simply determined in practical applications. The enhanced performance are achieved across multiple viewing distance databases by independently realigning the blur values associated with each distance.

A Full-Reference Image Quality Assessment Method via Deep Meta-Learning and Conformer

A Full-Reference Image Quality Assessment Method via Deep Meta-Learning and Conformer

Full-Reference Image Quality Assessment via Low-Level and High-Level Feature Fusion

We propose a full-reference image quality assessment (FR-IQA) method by incorporating low-level and high-level image features. First, in contrast to the preexisting deep IQA methods, which only use the features extracted by the deep network, we not only use the image gradient to replace the low-level features in the first two stages of the deep network, but also combine them with the middle-stage features of the deep network to construct the new low-level features. The deep features of shallow layers contain some unwanted noise and further result in a decline in IQA performance. Second, we combine the global features extracted by the self-attention-based model with the semantic features extracted by the convolutional neural network to form the high-level features. Instead of directly using the self-attention-based model trained on the classification task, we first train a no-reference (NR) IQA regression model on a larger dataset and then use the global features of this NR-IQA model. The self-attention-based model can capture the internal connections of the image and is more effective in extracting global information due to its larger perceptual field. In the final pooling stage, we combine the average pooling and the standard deviation pooling to obtain the dispersion and concentration of the similarity maps for a more comprehensive description of quality. Experiments show that our FR-IQA method is able to obtain competitive results on three standard IQA datasets.

Applicability Evaluation of Full-Reference Image Quality Assessment Methods for Computed Tomography Images

Image quality assessments (IQA) are an important task for providing appropriate medical care. Full-reference IQA (FR-IQA) methods, such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), are often used to evaluate imaging conditions, reconstruction conditions, and image processing algorithms, including noise reduction and super-resolution technology. However, these IQA methods may be inapplicable for medical images because they were designed for natural images. Therefore, this study aimed to investigate the correlation between objective assessment by some FR-IQA methods and human subjective assessment for computed tomography (CT) images. For evaluation, 210 distorted images were created from six original images using two types of degradation: noise and blur. We employed nine widely used FR-IQA methods for natural images: PSNR, SSIM, feature similarity (FSIM), information fidelity criterion (IFC), visual information fidelity (VIF), noise quality measure (NQM), visual signal-to-noise ratio (VSNR), multi-scale SSIM (MSSSIM), and information content-weighted SSIM (IWSSIM). Six observers performed subjective assessments using the double stimulus continuous quality scale (DSCQS) method. The performance of IQA methods was quantified using Pearson’s linear correlation coefficient (PLCC), Spearman rank order correlation coefficient (SROCC), and root-mean-square error (RMSE). Nine FR-IQA methods developed for natural images were all strongly correlated with the subjective assessment (PLCC and SROCC > 0.8), indicating that these methods can apply to CT images. Particularly, VIF had the best values for all three items, PLCC, SROCC, and RMSE. These results suggest that VIF provides the most accurate alternative measure to subjective assessments for CT images.

Full-Reference Image Quality Assessment Based on Multi-Channel Visual Information Fusion

This study focuses on improving the objective alignment of image quality assessment (IQA) algorithms with human visual perception. Existing methodologies, predominantly those based on the Laplacian of Gaussian (LoG) filter, often neglect the impact of color channels on human visual perception. Consequently, we propose a full-reference IQA method that integrates multi-channel visual information in color images. The methodology begins with converting red, green, blue (RGB) images into the luminance (L), red–green opponent color channel (M), blue–yellow opponent color channel (N) or LMN color space. Subsequently, the LoG filter is separately applied to the L, M, and N channels. The convoluted components are then fused to generate a contrast similarity map using the root-mean-square method, while the chromaticity similarity map is derived from the color channels. Finally, multi-channel LoG filtering, contrast, and chromaticity image features are connected. The standard deviation method is then used for sum pooling to create a full-reference IQA computational method. To validate the proposed method, distorted images from four widely used image databases were tested. The evaluation, based on four criteria, focused on the method’s prediction accuracy, computational complexity, and generalizability. The Pearson linear correlation coefficient (PLCC) values, recorded from the databases, ranged from 0.8822 (TID2013) to 0.9754 (LIVE). Similarly, the Spearman rank-order correlation coefficient (SROCC) values spanned from 0.8606 (TID2013) to 0.9798 (LIVE). In comparison to existing methods, the proposed IQA method exhibited superior visual correlation prediction accuracy, indicating its promising potential in the field of image quality assessment.

A method to pan-sharpen SAR amplitude data via electromagnetic reflectance similarities

ABSTRACT For a long time, the idea of fusing synthetic radar imagery (SAR) and higher-resolution panchromatic imagery, with the aim of increasing the spatial resolution of radar imagery, has been conceived as impossible. The challenge is due to the different natures of microwave radiation and visible radiation, where the former is the spectrum of SAR imagery, and the latter is of panchromatic imagery. Average reflection amplitude, which constitutes pixel values on both panchromatic and SAR imagery, is taken as a common framework around which this study proposes a method that enables the translation of an array on a panchromatic spectrum into an array of C-band microwave spectrum in order to achieve a higher spatial resolution. In this regard, open-access Worldview-2 and Worldview-3 panchromatic and Sentinel-1 (amplitude-only) images are used. The outputs are qualified using spatial correlation coefficient (SCC), net histogram curve, and R2 difference, and Erreur relative globale adimensionnelle de synthese (ERGAS) as full-reference image quality assessment methods in various scenarios. The results are tenable enough to consider this method for ordinary earth observation and monitoring objectives.

Dual-Channel Multi-Task CNN for No-Reference Screen Content Image Quality Assessment

Nowadays the problem of image quality assessment (IQA) for screen content images (SCIs) has become a research hotspot as they are ubiquitous in multimedia applications. Although the quality assessment of natural images (NIs) has been continuously developed in the past few decades, few NI-oriented IQA methods can be directly applied on SCIs due to different visual characteristics between them. In this paper, we present a no-reference quality prediction approach considering the content information of SCIs, which is based on dual-channel multi-task convolutional neural network. First, we segment a SCI into small patches and classify them as the textual patches and the pictorial patches. Then, we devise a novel dual-channel convolutional neural network (CNN) to predict the quality of textual patches and pictorial patches. Finally, we propose an effective adaptive weighting strategy for quality score aggregation. The proposed CNN is built on an end-to-end multi-task learning framework, which assists the SCI quality prediction task through the histogram of oriented gradient (HOG) feature prediction task to learn a better mapping between the input patch and its quality score. The adaptive weighting strategy further improves the representation ability of each SCI patch. Experimental results on two largest SCI-oriented databases demonstrate that the proposed method outperforms most of the state-of-the-art no-reference IQA methods and the full-reference IQA methods.

Screen content image quality assessment using distortion-based directional edge and gradient similarity maps

With the prevalent use of applications like Facebook, Twitter, and remote control applications, assessing the quality of the screen content images (SCIs) has become one of the critical fields of researches in image processing. In this paper, we develop a novel full-reference image quality assessment (IQA) method, called distortion-based directional edge and gradient similarity maps (DDEGSM) method, to evaluate the quality of SCIs by efficiently incorporating the effect of two challenging distortion types: contrast change (CC) and color saturation change (CSC). The proposed DDEGSM method has four main steps. First, we form an edge similarity map by extracting two edge and gradient features using the image’s gradient in 12 directions and the Laplacian of Gaussian filter. Next, we use the difference of Gaussian filter to weigh the strength of each pixel in the edge similarity map. Then, we examine the distorted image using two efficient approaches to find out whether the image contains the CC or CSC distortion. Finally, we calculate the quality score of the input SCI from the distortion-based feature maps using a pooling strategy. Our extensive experiments on three commonly-used SCI datasets indicate the proposed method is superior to the state-of-the-art full-reference IQA methods and is more consistent with the subjective assessments.

An optimized CNN-based quality assessment model for screen content image

Most existing convolutional neural network (CNN) based models designed for natural image quality assessment (IQA) employ image patches as training samples for data augmentation, and obtain final quality score by averaging all predicted scores of image patches. This brings two problems when applying these methods for screen content image (SCI) quality assessment. Firstly, SCI contains more complex content compared to natural image. As a result, qualities of SCI patches are different, and the subjective differential mean opinion score (DMOS) is not appropriate as qualities of all image patches. Secondly, the average score of image patches does not represent the quality of entire SCI since the human visual system (HVS) is sensitive to image patches containing texture and edge information. In this paper, we propose a novel quadratic optimized model based on the deep convolutional neural network (QODCNN) for full-reference (FR) and no-reference (NR) SCI quality assessment to overcome these two problems. The contribution of our algorithm can be concluded as follows: 1) Considering the characteristics of SCIs, a valid network architecture is designed for both NR and FR visual quality evaluation of SCIs, which makes the networks learn the feature differences for FR-IQA; 2) with the consideration of correlation between local quality and DMOS, a training data selection method is proposed to fine-tune the pre-trained model with valid SCI patches; 3) an adaptive pooling approach is employed to fuse patch quality to obtain image quality, owns strong noise robust and effects on both FR and NR IQA. Experimental results verify that our model outperforms both current no-reference and full-reference image quality assessment methods on the benchmark screen content image quality assessment database (SIQAD). Cross-database evaluation shows high generalization ability and high effectiveness of our model.

Active Fine-Tuning From gMAD Examples Improves Blind Image Quality Assessment.

The research in image quality assessment (IQA) has a long history, and significant progress has been made by leveraging recent advances in deep neural networks (DNNs). Despite high correlation numbers on existing IQA datasets, DNN-based models may be easily falsified in the group maximum differentiation (gMAD) competition. Here we show that gMAD examples can be used to improve blind IQA (BIQA) methods. Specifically, we first pre-train a DNN-based BIQA model using multiple noisy annotators, and fine-tune it on multiple synthetically distorted images, resulting in a "top-performing" baseline model. We then seek pairs of images by comparing the baseline model with a set of full-reference IQA methods in gMAD. The spotted gMAD examples are most likely to reveal the weaknesses of the baseline, and suggest potential ways for refinement. We query human quality annotations for the selected images in a well-controlled laboratory environment, and further fine-tune the baseline on the combination of human-rated images from gMAD and existing databases. This process may be iterated, enabling active fine-tuning from gMAD examples for BIQA. We demonstrate the feasibility of our active learning scheme on a large-scale unlabeled image set, and show that the fine-tuned quality model achieves improved generalizability in gMAD, without destroying performance on previously seen databases.

A Combined Full-Reference Image Quality Assessment Method Based on Convolutional Activation Maps

The goal of full-reference image quality assessment (FR-IQA) is to predict the perceptual quality of an image as perceived by human observers using its pristine (distortion free) reference counterpart. In this study, we explore a novel, combined approach which predicts the perceptual quality of a distorted image by compiling a feature vector from convolutional activation maps. More specifically, a reference-distorted image pair is run through a pretrained convolutional neural network and the activation maps are compared with a traditional image similarity metric. Subsequently, the resulting feature vector is mapped onto perceptual quality scores with the help of a trained support vector regressor. A detailed parameter study is also presented in which the design choices of the proposed method is explained. Furthermore, we study the relationship between the amount of training images and the prediction performance. Specifically, it is demonstrated that the proposed method can be trained with a small amount of data to reach high prediction performance. Our best proposal—called ActMapFeat—is compared to the state-of-the-art on six publicly available benchmark IQA databases, such as KADID-10k, TID2013, TID2008, MDID, CSIQ, and VCL-FER. Specifically, our method is able to significantly outperform the state-of-the-art on these benchmark databases.

Quality Assessment of 3D Printed Surfaces Using Combined Metrics Based on Mutual Structural Similarity Approach Correlated with Subjective Aesthetic Evaluation

Quality assessment of the 3D printed surfaces is one of the crucial issues related to fast prototyping and manufacturing of individual parts and objects using the fused deposition modeling, especially in small series production. As some corrections of minor defects may be conducted during the printing process or just after the manufacturing, an automatic quality assessment of object’s surfaces is highly demanded, preferably well correlated with subjective quality perception, considering aesthetic aspects. On the other hand, the presence of some greater and more dense distortions may indicate a reduced mechanical strength. In such cases, the manufacturing process should be interrupted to save time, energy, and the filament. This paper focuses on the possibility of using some general-purpose full-reference image quality assessment methods for the quality assessment of the 3D printed surfaces. As the direct application of an individual (elementary) metric does not provide high correlation with the subjective perception of surface quality, some modifications of similarity-based methods have been proposed utilizing the calculation of the average mutual similarity, making it possible to use full-reference metrics without the perfect quality reference images, as well as the combination of individual metrics, leading to a significant increase of correlation with subjective scores calculated for a specially prepared dataset.

Image Quality Assessment by Combining Masking Texture and Perceptual Color Difference Model

Objective image quality assessment (IQA) models have been developed by effective features to imitate the characteristics of human visual system (HVS). Actually, HVS is extremely sensitive to color degradation and complex texture changes. In this paper, we firstly reveal that many existing full reference image quality assessment (FR-IQA) methods can hardly measure the image quality with contrast and masking texture changes. To solve this problem, considering texture masking effect, we proposed a novel FR-IQA method, called Texture and Color Quality Index (TCQI). The proposed method considers both in the masking effect texture and color visual perceptual threshold, which adopts three kinds of features to reflect masking texture, color difference and structural information. Furthermore, random forest (RF) is used to address the drawbacks of existing pooling technologies. Compared with other traditional learning-based tools (support vector regression and neural network), RF can achieve the better prediction performance. Experiments conducted on five large-scale databases demonstrate that our approach is highly consistent with subjective perception, outperforms twelve the state-of-the-art IQA models in terms of prediction accuracy and keeps a moderate computational complexity. The cross database validation also validates our approach achieves the ability to maintain high robustness.

Screen content image quality assessment based on convolutional neural networks

Screen content image (SCI) is a composite image including textual and pictorial regions resulting in many difficulties in image quality assessment (IQA). Large SCIs are divided into image patches to increase training samples for CNN training of IQA model, and this brings two problems: (1) local quality of each image patch is not equal to subjective differential mean opinion score (DMOS) of an entire image; (2) importance of different image patches is not same for quality assessment. In this paper, we propose a novel no-reference (NR) IQA model based on the convolutional neural network (CNN) for assessing the perceptual quality of SCIs. Our model conducts two designs solving problems which benefits from two strategies. For the first strategy, to imitate full-reference (FR) CNN-based model behavior, a CNN-based model is designed for both FR and NR IQA, and performance of NR-IQA part improves when the image patch scores predicted by FR-IQA part are adopted as the ground-truth to train NR-IQA part. For the second strategy, image patch qualities of one entire SCI are fused to obtain the SCI quality with an adaptive weighting method taking account the effect of the different image patch contents. Experimental results verify that our model outperforms all test NR IQA methods and most FR IQA methods on the screen content image quality assessment database (SIQAD). On the cross-database evaluation, the proposed method outperforms the existing NR IQA method in terms of at least 2.4 percent in PLCC and 2.8 percent in SRCC, which shows high generalization ability and high effectiveness of our model.

A Full-Reference Image Quality Assessment for Multiply Distorted Image based on Visual Mutual Information

Abstract A full-reference image quality assessment (FR-IQA) method for multi-distortion based on visual mutual information (MD-IQA) is proposed to solve the problem that the existing FR-IQA methods are mostly applicable to single-distorted images, but the assessment result for multiply distorted images is not ideal. First, the reference image and the distorted image are preprocessed by steerable pyramid decomposition and contrast sensitivity function (CSF). Next, a Gaussian scale mixture (GSM) model and an image distorted model are respectively constructed for the reference images and the distorted images. Then, visual distorted models are constructed both for the reference images and the distorted images. Finally, the mutual information between the processed reference image and the distorted image is calculated to obtain the full-reference quality assessment index for multiply distorted images. The experimental results show that the proposed method has higher accuracy and better performance for multiply distorted images.

Adaptation of Full-Reference Image Quality Assessment Methods for Automatic Visual Evaluation of the Surface Quality of 3D Prints

Automatic visual quality assessment of the 3D printed surfaces is currently one of the most demanding challenges in additive manufacturing. Regardless of the applications of the computer vision for the 3D printing process monitoring purposes, a reliable surface quality evaluation during manufacturing may introduce brand new possibilities. The detection of some distortions and their automatic evaluation can be helpful when deciding to stop the process to save time, energy, and filament. In some cases, some further corrections can also be made for relatively small distortions. Since many general-purpose image quality assessment methods have been proposed in recent years, their applications for the quality evaluation in the additive manufacturing are investigated. As most of the metrics are full-reference and require the availability of the original perfect quality image, their direct application is not possible. Therefore, their adaptation is described in the paper together with experimental verification of classification results obtained using various metrics.

Blind Noisy Image Quality Assessment Using Sub-Band Kurtosis.

Noise that afflicts natural images, regardless of the source, generally disturbs the perception of image quality by introducing a high-frequency random element that, when severe, can mask image content. Except at very low levels, where it may play a purpose, it is annoying. There exist significant statistical differences between distortion-free natural images and noisy images that become evident upon comparing the empirical probability distribution histograms of their discrete wavelet transform (DWT) coefficients. The DWT coefficients of low- or no-noise natural images have leptokurtic, peaky distributions with heavy tails; while noisy images tend to be platykurtic with less peaky distributions and shallower tails. The sample kurtosis is a natural measure of the peakedness and tail weight of the distributions of random variables. Here, we study the efficacy of the sample kurtosis of image wavelet coefficients as a feature driving, an extreme learning machine which learns to map kurtosis values into perceptual quality scores. The model is trained and tested on five types of noisy images, including additive white Gaussian noise, additive Gaussian color noise, impulse noise, masked noise, and high-frequency noise from the LIVE, CSIQ, TID2008, and TID2013 image quality databases. The experimental results show that the trained model has better quality evaluation performance on noisy images than existing blind noise assessment models, while also outperforming general-purpose blind and full-reference image quality assessment methods.

Image Quality Assessment by Considering Multiscale and Multidirectional Visibility Differences in Shearlet Domain

Conventional objective image assessment metrics, such as mean squared error and peak signal-to-noise ratio, which only calculates pixel-based differences between the original and the degraded images, are not in agreement with the human vision. In this paper, we present an improved objective full-reference image quality assessment method, called the multiscale and multidirectional visibility differences (MMVD) predictor. The proposed MMVD metric considers multiscale and multidirectional visibility differences in the domain of the discrete nonseparable shearlet transform, which emulates the multichannel structure of information processing of the human vision system. In the process of constructing the visual just noticeable difference threshold in the shearlet domain, the contrast sensitivity function and the visual masking effect which are important properties of the human visual perception are considered simultaneously to approximate the sensitivities of human visual responses. Both contrast masking and entropy masking are considered to tackle the visual masking issue. All subbands of the shearlet transform are evaluated, and perceptual errors of subbands are pooled together to yield the objective quality index of a distorted image. The extensive validation experiments are conducted on five public image databases, namely, TID2008, TID2013, CSIQ, IVC, and LIVE. The experimental results demonstrate the proposed method is well coherent with human perception and has better performance compared to the several state-of-the-art image quality metrics.