Full-reference Image Quality Assessment Methods Research Articles

A Full-Reference Image Quality Assessment Method via Deep Meta-Learning and Conformer

A Full-Reference Image Quality Assessment Method via Deep Meta-Learning and Conformer

Full-Reference Image Quality Assessment via Low-Level and High-Level Feature Fusion

We propose a full-reference image quality assessment (FR-IQA) method by incorporating low-level and high-level image features. First, in contrast to the preexisting deep IQA methods, which only use the features extracted by the deep network, we not only use the image gradient to replace the low-level features in the first two stages of the deep network, but also combine them with the middle-stage features of the deep network to construct the new low-level features. The deep features of shallow layers contain some unwanted noise and further result in a decline in IQA performance. Second, we combine the global features extracted by the self-attention-based model with the semantic features extracted by the convolutional neural network to form the high-level features. Instead of directly using the self-attention-based model trained on the classification task, we first train a no-reference (NR) IQA regression model on a larger dataset and then use the global features of this NR-IQA model. The self-attention-based model can capture the internal connections of the image and is more effective in extracting global information due to its larger perceptual field. In the final pooling stage, we combine the average pooling and the standard deviation pooling to obtain the dispersion and concentration of the similarity maps for a more comprehensive description of quality. Experiments show that our FR-IQA method is able to obtain competitive results on three standard IQA datasets.

Applicability Evaluation of Full-Reference Image Quality Assessment Methods for Computed Tomography Images

Image quality assessments (IQA) are an important task for providing appropriate medical care. Full-reference IQA (FR-IQA) methods, such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), are often used to evaluate imaging conditions, reconstruction conditions, and image processing algorithms, including noise reduction and super-resolution technology. However, these IQA methods may be inapplicable for medical images because they were designed for natural images. Therefore, this study aimed to investigate the correlation between objective assessment by some FR-IQA methods and human subjective assessment for computed tomography (CT) images. For evaluation, 210 distorted images were created from six original images using two types of degradation: noise and blur. We employed nine widely used FR-IQA methods for natural images: PSNR, SSIM, feature similarity (FSIM), information fidelity criterion (IFC), visual information fidelity (VIF), noise quality measure (NQM), visual signal-to-noise ratio (VSNR), multi-scale SSIM (MSSSIM), and information content-weighted SSIM (IWSSIM). Six observers performed subjective assessments using the double stimulus continuous quality scale (DSCQS) method. The performance of IQA methods was quantified using Pearson’s linear correlation coefficient (PLCC), Spearman rank order correlation coefficient (SROCC), and root-mean-square error (RMSE). Nine FR-IQA methods developed for natural images were all strongly correlated with the subjective assessment (PLCC and SROCC > 0.8), indicating that these methods can apply to CT images. Particularly, VIF had the best values for all three items, PLCC, SROCC, and RMSE. These results suggest that VIF provides the most accurate alternative measure to subjective assessments for CT images.

Full-Reference Image Quality Assessment Based on Multi-Channel Visual Information Fusion

This study focuses on improving the objective alignment of image quality assessment (IQA) algorithms with human visual perception. Existing methodologies, predominantly those based on the Laplacian of Gaussian (LoG) filter, often neglect the impact of color channels on human visual perception. Consequently, we propose a full-reference IQA method that integrates multi-channel visual information in color images. The methodology begins with converting red, green, blue (RGB) images into the luminance (L), red–green opponent color channel (M), blue–yellow opponent color channel (N) or LMN color space. Subsequently, the LoG filter is separately applied to the L, M, and N channels. The convoluted components are then fused to generate a contrast similarity map using the root-mean-square method, while the chromaticity similarity map is derived from the color channels. Finally, multi-channel LoG filtering, contrast, and chromaticity image features are connected. The standard deviation method is then used for sum pooling to create a full-reference IQA computational method. To validate the proposed method, distorted images from four widely used image databases were tested. The evaluation, based on four criteria, focused on the method’s prediction accuracy, computational complexity, and generalizability. The Pearson linear correlation coefficient (PLCC) values, recorded from the databases, ranged from 0.8822 (TID2013) to 0.9754 (LIVE). Similarly, the Spearman rank-order correlation coefficient (SROCC) values spanned from 0.8606 (TID2013) to 0.9798 (LIVE). In comparison to existing methods, the proposed IQA method exhibited superior visual correlation prediction accuracy, indicating its promising potential in the field of image quality assessment.

A method to pan-sharpen SAR amplitude data via electromagnetic reflectance similarities

ABSTRACT For a long time, the idea of fusing synthetic radar imagery (SAR) and higher-resolution panchromatic imagery, with the aim of increasing the spatial resolution of radar imagery, has been conceived as impossible. The challenge is due to the different natures of microwave radiation and visible radiation, where the former is the spectrum of SAR imagery, and the latter is of panchromatic imagery. Average reflection amplitude, which constitutes pixel values on both panchromatic and SAR imagery, is taken as a common framework around which this study proposes a method that enables the translation of an array on a panchromatic spectrum into an array of C-band microwave spectrum in order to achieve a higher spatial resolution. In this regard, open-access Worldview-2 and Worldview-3 panchromatic and Sentinel-1 (amplitude-only) images are used. The outputs are qualified using spatial correlation coefficient (SCC), net histogram curve, and R2 difference, and Erreur relative globale adimensionnelle de synthese (ERGAS) as full-reference image quality assessment methods in various scenarios. The results are tenable enough to consider this method for ordinary earth observation and monitoring objectives.

Dual-Channel Multi-Task CNN for No-Reference Screen Content Image Quality Assessment

Nowadays the problem of image quality assessment (IQA) for screen content images (SCIs) has become a research hotspot as they are ubiquitous in multimedia applications. Although the quality assessment of natural images (NIs) has been continuously developed in the past few decades, few NI-oriented IQA methods can be directly applied on SCIs due to different visual characteristics between them. In this paper, we present a no-reference quality prediction approach considering the content information of SCIs, which is based on dual-channel multi-task convolutional neural network. First, we segment a SCI into small patches and classify them as the textual patches and the pictorial patches. Then, we devise a novel dual-channel convolutional neural network (CNN) to predict the quality of textual patches and pictorial patches. Finally, we propose an effective adaptive weighting strategy for quality score aggregation. The proposed CNN is built on an end-to-end multi-task learning framework, which assists the SCI quality prediction task through the histogram of oriented gradient (HOG) feature prediction task to learn a better mapping between the input patch and its quality score. The adaptive weighting strategy further improves the representation ability of each SCI patch. Experimental results on two largest SCI-oriented databases demonstrate that the proposed method outperforms most of the state-of-the-art no-reference IQA methods and the full-reference IQA methods.

Screen content image quality assessment using distortion-based directional edge and gradient similarity maps

With the prevalent use of applications like Facebook, Twitter, and remote control applications, assessing the quality of the screen content images (SCIs) has become one of the critical fields of researches in image processing. In this paper, we develop a novel full-reference image quality assessment (IQA) method, called distortion-based directional edge and gradient similarity maps (DDEGSM) method, to evaluate the quality of SCIs by efficiently incorporating the effect of two challenging distortion types: contrast change (CC) and color saturation change (CSC). The proposed DDEGSM method has four main steps. First, we form an edge similarity map by extracting two edge and gradient features using the image’s gradient in 12 directions and the Laplacian of Gaussian filter. Next, we use the difference of Gaussian filter to weigh the strength of each pixel in the edge similarity map. Then, we examine the distorted image using two efficient approaches to find out whether the image contains the CC or CSC distortion. Finally, we calculate the quality score of the input SCI from the distortion-based feature maps using a pooling strategy. Our extensive experiments on three commonly-used SCI datasets indicate the proposed method is superior to the state-of-the-art full-reference IQA methods and is more consistent with the subjective assessments.

An optimized CNN-based quality assessment model for screen content image

Most existing convolutional neural network (CNN) based models designed for natural image quality assessment (IQA) employ image patches as training samples for data augmentation, and obtain final quality score by averaging all predicted scores of image patches. This brings two problems when applying these methods for screen content image (SCI) quality assessment. Firstly, SCI contains more complex content compared to natural image. As a result, qualities of SCI patches are different, and the subjective differential mean opinion score (DMOS) is not appropriate as qualities of all image patches. Secondly, the average score of image patches does not represent the quality of entire SCI since the human visual system (HVS) is sensitive to image patches containing texture and edge information. In this paper, we propose a novel quadratic optimized model based on the deep convolutional neural network (QODCNN) for full-reference (FR) and no-reference (NR) SCI quality assessment to overcome these two problems. The contribution of our algorithm can be concluded as follows: 1) Considering the characteristics of SCIs, a valid network architecture is designed for both NR and FR visual quality evaluation of SCIs, which makes the networks learn the feature differences for FR-IQA; 2) with the consideration of correlation between local quality and DMOS, a training data selection method is proposed to fine-tune the pre-trained model with valid SCI patches; 3) an adaptive pooling approach is employed to fuse patch quality to obtain image quality, owns strong noise robust and effects on both FR and NR IQA. Experimental results verify that our model outperforms both current no-reference and full-reference image quality assessment methods on the benchmark screen content image quality assessment database (SIQAD). Cross-database evaluation shows high generalization ability and high effectiveness of our model.

Active Fine-Tuning From gMAD Examples Improves Blind Image Quality Assessment.

The research in image quality assessment (IQA) has a long history, and significant progress has been made by leveraging recent advances in deep neural networks (DNNs). Despite high correlation numbers on existing IQA datasets, DNN-based models may be easily falsified in the group maximum differentiation (gMAD) competition. Here we show that gMAD examples can be used to improve blind IQA (BIQA) methods. Specifically, we first pre-train a DNN-based BIQA model using multiple noisy annotators, and fine-tune it on multiple synthetically distorted images, resulting in a "top-performing" baseline model. We then seek pairs of images by comparing the baseline model with a set of full-reference IQA methods in gMAD. The spotted gMAD examples are most likely to reveal the weaknesses of the baseline, and suggest potential ways for refinement. We query human quality annotations for the selected images in a well-controlled laboratory environment, and further fine-tune the baseline on the combination of human-rated images from gMAD and existing databases. This process may be iterated, enabling active fine-tuning from gMAD examples for BIQA. We demonstrate the feasibility of our active learning scheme on a large-scale unlabeled image set, and show that the fine-tuned quality model achieves improved generalizability in gMAD, without destroying performance on previously seen databases.

A Combined Full-Reference Image Quality Assessment Method Based on Convolutional Activation Maps

The goal of full-reference image quality assessment (FR-IQA) is to predict the perceptual quality of an image as perceived by human observers using its pristine (distortion free) reference counterpart. In this study, we explore a novel, combined approach which predicts the perceptual quality of a distorted image by compiling a feature vector from convolutional activation maps. More specifically, a reference-distorted image pair is run through a pretrained convolutional neural network and the activation maps are compared with a traditional image similarity metric. Subsequently, the resulting feature vector is mapped onto perceptual quality scores with the help of a trained support vector regressor. A detailed parameter study is also presented in which the design choices of the proposed method is explained. Furthermore, we study the relationship between the amount of training images and the prediction performance. Specifically, it is demonstrated that the proposed method can be trained with a small amount of data to reach high prediction performance. Our best proposal—called ActMapFeat—is compared to the state-of-the-art on six publicly available benchmark IQA databases, such as KADID-10k, TID2013, TID2008, MDID, CSIQ, and VCL-FER. Specifically, our method is able to significantly outperform the state-of-the-art on these benchmark databases.

Quality Assessment of 3D Printed Surfaces Using Combined Metrics Based on Mutual Structural Similarity Approach Correlated with Subjective Aesthetic Evaluation

Quality assessment of the 3D printed surfaces is one of the crucial issues related to fast prototyping and manufacturing of individual parts and objects using the fused deposition modeling, especially in small series production. As some corrections of minor defects may be conducted during the printing process or just after the manufacturing, an automatic quality assessment of object’s surfaces is highly demanded, preferably well correlated with subjective quality perception, considering aesthetic aspects. On the other hand, the presence of some greater and more dense distortions may indicate a reduced mechanical strength. In such cases, the manufacturing process should be interrupted to save time, energy, and the filament. This paper focuses on the possibility of using some general-purpose full-reference image quality assessment methods for the quality assessment of the 3D printed surfaces. As the direct application of an individual (elementary) metric does not provide high correlation with the subjective perception of surface quality, some modifications of similarity-based methods have been proposed utilizing the calculation of the average mutual similarity, making it possible to use full-reference metrics without the perfect quality reference images, as well as the combination of individual metrics, leading to a significant increase of correlation with subjective scores calculated for a specially prepared dataset.

Image Quality Assessment by Combining Masking Texture and Perceptual Color Difference Model

Objective image quality assessment (IQA) models have been developed by effective features to imitate the characteristics of human visual system (HVS). Actually, HVS is extremely sensitive to color degradation and complex texture changes. In this paper, we firstly reveal that many existing full reference image quality assessment (FR-IQA) methods can hardly measure the image quality with contrast and masking texture changes. To solve this problem, considering texture masking effect, we proposed a novel FR-IQA method, called Texture and Color Quality Index (TCQI). The proposed method considers both in the masking effect texture and color visual perceptual threshold, which adopts three kinds of features to reflect masking texture, color difference and structural information. Furthermore, random forest (RF) is used to address the drawbacks of existing pooling technologies. Compared with other traditional learning-based tools (support vector regression and neural network), RF can achieve the better prediction performance. Experiments conducted on five large-scale databases demonstrate that our approach is highly consistent with subjective perception, outperforms twelve the state-of-the-art IQA models in terms of prediction accuracy and keeps a moderate computational complexity. The cross database validation also validates our approach achieves the ability to maintain high robustness.

Screen content image quality assessment based on convolutional neural networks

Screen content image (SCI) is a composite image including textual and pictorial regions resulting in many difficulties in image quality assessment (IQA). Large SCIs are divided into image patches to increase training samples for CNN training of IQA model, and this brings two problems: (1) local quality of each image patch is not equal to subjective differential mean opinion score (DMOS) of an entire image; (2) importance of different image patches is not same for quality assessment. In this paper, we propose a novel no-reference (NR) IQA model based on the convolutional neural network (CNN) for assessing the perceptual quality of SCIs. Our model conducts two designs solving problems which benefits from two strategies. For the first strategy, to imitate full-reference (FR) CNN-based model behavior, a CNN-based model is designed for both FR and NR IQA, and performance of NR-IQA part improves when the image patch scores predicted by FR-IQA part are adopted as the ground-truth to train NR-IQA part. For the second strategy, image patch qualities of one entire SCI are fused to obtain the SCI quality with an adaptive weighting method taking account the effect of the different image patch contents. Experimental results verify that our model outperforms all test NR IQA methods and most FR IQA methods on the screen content image quality assessment database (SIQAD). On the cross-database evaluation, the proposed method outperforms the existing NR IQA method in terms of at least 2.4 percent in PLCC and 2.8 percent in SRCC, which shows high generalization ability and high effectiveness of our model.

A Full-Reference Image Quality Assessment for Multiply Distorted Image based on Visual Mutual Information

Abstract A full-reference image quality assessment (FR-IQA) method for multi-distortion based on visual mutual information (MD-IQA) is proposed to solve the problem that the existing FR-IQA methods are mostly applicable to single-distorted images, but the assessment result for multiply distorted images is not ideal. First, the reference image and the distorted image are preprocessed by steerable pyramid decomposition and contrast sensitivity function (CSF). Next, a Gaussian scale mixture (GSM) model and an image distorted model are respectively constructed for the reference images and the distorted images. Then, visual distorted models are constructed both for the reference images and the distorted images. Finally, the mutual information between the processed reference image and the distorted image is calculated to obtain the full-reference quality assessment index for multiply distorted images. The experimental results show that the proposed method has higher accuracy and better performance for multiply distorted images.

Adaptation of Full-Reference Image Quality Assessment Methods for Automatic Visual Evaluation of the Surface Quality of 3D Prints

Automatic visual quality assessment of the 3D printed surfaces is currently one of the most demanding challenges in additive manufacturing. Regardless of the applications of the computer vision for the 3D printing process monitoring purposes, a reliable surface quality evaluation during manufacturing may introduce brand new possibilities. The detection of some distortions and their automatic evaluation can be helpful when deciding to stop the process to save time, energy, and filament. In some cases, some further corrections can also be made for relatively small distortions. Since many general-purpose image quality assessment methods have been proposed in recent years, their applications for the quality evaluation in the additive manufacturing are investigated. As most of the metrics are full-reference and require the availability of the original perfect quality image, their direct application is not possible. Therefore, their adaptation is described in the paper together with experimental verification of classification results obtained using various metrics.

Blind Noisy Image Quality Assessment Using Sub-Band Kurtosis.

Noise that afflicts natural images, regardless of the source, generally disturbs the perception of image quality by introducing a high-frequency random element that, when severe, can mask image content. Except at very low levels, where it may play a purpose, it is annoying. There exist significant statistical differences between distortion-free natural images and noisy images that become evident upon comparing the empirical probability distribution histograms of their discrete wavelet transform (DWT) coefficients. The DWT coefficients of low- or no-noise natural images have leptokurtic, peaky distributions with heavy tails; while noisy images tend to be platykurtic with less peaky distributions and shallower tails. The sample kurtosis is a natural measure of the peakedness and tail weight of the distributions of random variables. Here, we study the efficacy of the sample kurtosis of image wavelet coefficients as a feature driving, an extreme learning machine which learns to map kurtosis values into perceptual quality scores. The model is trained and tested on five types of noisy images, including additive white Gaussian noise, additive Gaussian color noise, impulse noise, masked noise, and high-frequency noise from the LIVE, CSIQ, TID2008, and TID2013 image quality databases. The experimental results show that the trained model has better quality evaluation performance on noisy images than existing blind noise assessment models, while also outperforming general-purpose blind and full-reference image quality assessment methods.

Image Quality Assessment by Considering Multiscale and Multidirectional Visibility Differences in Shearlet Domain

Conventional objective image assessment metrics, such as mean squared error and peak signal-to-noise ratio, which only calculates pixel-based differences between the original and the degraded images, are not in agreement with the human vision. In this paper, we present an improved objective full-reference image quality assessment method, called the multiscale and multidirectional visibility differences (MMVD) predictor. The proposed MMVD metric considers multiscale and multidirectional visibility differences in the domain of the discrete nonseparable shearlet transform, which emulates the multichannel structure of information processing of the human vision system. In the process of constructing the visual just noticeable difference threshold in the shearlet domain, the contrast sensitivity function and the visual masking effect which are important properties of the human visual perception are considered simultaneously to approximate the sensitivities of human visual responses. Both contrast masking and entropy masking are considered to tackle the visual masking issue. All subbands of the shearlet transform are evaluated, and perceptual errors of subbands are pooled together to yield the objective quality index of a distorted image. The extensive validation experiments are conducted on five public image databases, namely, TID2008, TID2013, CSIQ, IVC, and LIVE. The experimental results demonstrate the proposed method is well coherent with human perception and has better performance compared to the several state-of-the-art image quality metrics.

Blind Image Quality Assessment of Natural Scenes Based on Entropy Differences in the DCT Domain.

Blind/no-reference image quality assessment is performed to accurately evaluate the perceptual quality of a distorted image without prior information from a reference image. In this paper, an effective blind image quality assessment approach based on entropy differences in the discrete cosine transform domain for natural images is proposed. Information entropy is an effective measure of the amount of information in an image. We find the discrete cosine transform coefficient distribution of distorted natural images shows a pulse-shape phenomenon, which directly affects the differences of entropy. Then, a Weibull model is used to fit the distributions of natural and distorted images. This is because the Weibull model sufficiently approximates the pulse-shape phenomenon as well as the sharp-peak and heavy-tail phenomena of natural scene statistics rules. Four features that are related to entropy differences and human visual system are extracted from the Weibull model for three scaling images. Image quality is assessed by the support vector regression method based on the extracted features. This blind Weibull statistics algorithm is thoroughly evaluated using three widely used databases: LIVE, TID2008, and CSIQ. The experimental results show that the performance of the proposed blind Weibull statistics method is highly consistent with that of human visual perception and greater than that of the state-of-the-art blind and full-reference image quality assessment methods in most cases.

RVSIM: a feature similarity method for full-reference image quality assessment

Image quality assessment is an important topic in the field of digital image processing. In this study, a full-reference image quality assessment method called Riesz transform and Visual contrast sensitivity-based feature SIMilarity index (RVSIM) is proposed. More precisely, a Log-Gabor filter is first used to decompose reference and distorted images, and Riesz transform is performed on the decomposed images on the basis of monogenic signal theory. Then, the monogenic signal similarity matrix is obtained by calculating the similarity of the local amplitude/phase/direction characteristics of monogenic signal. Next, we weight the summation of these characteristics with visual contrast sensitivity. Since the first-order Riesz transform cannot clearly express the corners and intersection points in the image, we calculate the gradient magnitude similarity between the reference and distorted images as a feature, which is combined with monogenic signal similarity to obtain a local quality map. Finally, we conduct the monogenic phase congruency using the Riesz transform feature matrix from the reference image and utilize it as a weighted function to derive the similarity index. Extensive experiments on five benchmark IQA databases, namely, LIVE, CSIQ, TID2008, TID2013, and Waterloo Exploration, indicate that RVSIM is a robust IQA method.

基于人眼视觉特性的深度学习全参考图像质量评价方法

Since the current image quality assessment methods are generally based on hand-crafted features, it is difficult to automatically and effectively extract image features that conform to the human visual system. Inspired by human visual characteristics, a new method of full-reference image quality assessment was proposed by this paper which was based on convolutional neural network (DeepFR). According to this method, the DeepFR model of convolutional neural network was designed which was based on the understanding of the dataset by itself using the human visual system to weight the sensitivity of the gradient, and the visual gradient perception map was extracted that was consistent with human visual characteristics. The experimental results show that the DeepFR model is superior to the current full-reference image quality assessment methods, its prediction score and subjective quality evaluation have good accuracy and consistency.