Full-reference Image Quality Assessment Research Articles

Multi-gene Genetic Programming based Predictive Models for Full-reference Image Quality Assessment

Abstract Many objective quality metrics for assessing the visual quality of images have been developed during the last decade. A simple way to fine tune the efficiency of assessment is through permutation and combination of these metrics. The goal of this fusion approach is to take advantage of the metrics utilized and minimize the influence of their drawbacks. In this paper, a symbolic regression technique using an evolutionary algorithm known as multi-gene genetic programming (MGGP) is applied for predicting subject scores of images in datasets using a combination of objective scores of a set of image quality metrics (IQM). By learning from image datasets, the MGGP algorithm can determine appropriate image quality metrics, from 21 metrics utilized, whose objective scores employed as predictors in the symbolic regression model, by optimizing simultaneously two competing objectives of model ‘goodness of fit’ to data and model ‘complexity’. Six large image databases (namely LIVE, CSIQ, TID2008, TID2013, IVC and MDID) that are available in public domain are used for learning and testing the predictive models, according the k-fold-cross-validation and the cross dataset strategies. The proposed approach is compared against state-of-the-art objective image quality assessment approaches. Results of comparison reveal that the proposed approach outperforms other state-of-the-art recently developed fusion approaches.

Structural similarity index (SSIM) revisited: A data-driven approach

Several contemporaneous image processing and computer vision systems rely upon the full-reference image quality assessment (IQA) measures. The single-scale structural similarity index (SS-SSIM) is one of the most popular measures, and it owes its success to the mathematical simplicity, low computational complexity, and implicit incorporation of Human Visual System’s (HVS) characteristics. In this paper, we revise the original parameters of SSIM and its multi-scale counterpart (MS-SSIM) to increase their correlation with subjective evaluation. More specifically, we exploit the evolutionary computation and the swarm intelligence methods on five popular IQA databases, two of which are dedicated distance-changed databases, to determine the best combination of parameters efficiently. Simultaneously, we explore the effect of different scale selection approaches in the context of SS-SSIM. The experimental results show that with a proper fine-tuning (1) the performance of SS-SSIM and MS-SSIM can be improved, in average terms, by 8% and by 3%, respectively, (2) the SS-SSIM after the so-called standard scale selection achieves similar performance as if applying computationally more expensive state-of-the-art scale selection methods or MS-SSIM; moreover, (3) there is evidence that the parameters learned on a given database can be successfully transferred to other (previously unseen) databases; finally, (4) we propose a new set of reference parameters for SSIM’s variants and provide their interpretation.

Full-Reference Quality Assessment for Screen Content Images Based on the Concept of Global-Guidance and Local-Adjustment

Benefiting from the development of multimedia communication terminals, the visual content presented to people on mobile devices is no longer a single form, but contains text, natural images, and other computer-generated graphics, which is called screen content images (SCIs). Inspired by the different visual stimuli that text and images bring to human eyes and the concept of global-guidance and local-adjustment, we design a novel full-reference image quality assessment (IQA) model using the structural features of the text, the perceptual features of pictures, and a score integration model (SPSIM) to evaluate SCIs quality. Firstly, we split the SCIs into textual and pictorial regions through a fully convolutional network (FCN) to conduct separate analyses. For textual regions, we take advantage of narrow edge extensions and high edge steps as structural features to compute the textual score. For pictorial regions, we extract the just noticeable difference (JND) features, which measure the human eye's ability to detect distortion as perceptual features to calculate the pictorial score. Finally, an innovative score integration method based on the global-guidance and local-adjustment is designed to better analyze the relationship between the above regional scores and the whole global SCIs score. Abundant experiments in SCIs databases have shown that the SPSIM model can achieve better consistency with the human eyes system (HVS) in predicting the visual quality of SCIs.

Unified quality assessment of natural and screen content images via adaptive weighting on double scales

In multimedia communication systems, digital images may contain various visual contents, among which the natural scene images (NSIs) and screen content images (SCIs) are two important and common types. The existing full-reference image quality assessment (IQA) metrics are designed for only one type of images, but cannot precisely perceive the visual quality of another type. It is still unclear what the different characteristics are between NSIs and SCIs resulting in this failure. Inspired by some psychological studies, we figure out that it is due to the different structural scale levels between NSIs and SCIs. Given this observation, this paper introduces the gradient degradation of Gaussians (GDoG) to analyze the images’ structural scale level, proposes a fast unified IQA index for both NSIs and SCIs by incorporating an adaptive weighting strategy on double scales. Experimental results conducted on several databases verify the effectiveness and efficiency of the proposed unified IQA index for both types of images, also demonstrate that the adaptive weighting strategy based on GDoG can improve the existing models for cross-content-type images.

Evaluation of quality measures for color quantization

Visual quality evaluation is one of the challenging basic problems in image processing. It also plays a central role in the shaping, implementation, optimization, and testing of many methods. The existing image quality assessment methods focused on images corrupted by common degradation types while little attention was paid to color quantization. This in spite there is a wide range of applications requiring color quantization assessment being used as a preprocessing step when color-based tasks are more efficiently accomplished on a reduced number of colors. In this paper, we propose and carry-out a quantitative performance evaluation of nine well-known and commonly used full-reference image quality assessment measures. The evaluation is done by using two publicly available and subjectively rated image quality databases for color quantization degradation and by considering suitable combinations or subparts of them. The results indicate the quality measures that have closer performances in terms of their correlation to the subjective human rating and show that the evaluation of the statistical performance of the quality measures for color quantization is significantly impacted by the selected image quality database while maintaining a similar trend on each database. The detected strong similarity both on individual databases and on databases obtained by integration provides the ability to validate the integration process and to consider the quantitative performance evaluation on each database as an indicator for performance on the other databases. The experimental results are useful to address the choice of suitable quality measures for color quantization and to improve their future employment.

Extended Damage Detection and Identification in Aircraft Structure Based on Multifrequency Eddy Current Method and Mutual Image Similarity Assessment.

In this paper, a novel approach to Non-Destructive Testing (NDT) of defective materials for the aircraft industry is proposed, which utilizes an approach based on multifrequency and spectrogram eddy current method combined with an image analysis method previously applied for general-purpose full-reference image quality assessment (FR IQA). The proposed defect identification method is based on the use of the modified SSIM4 image quality metric. The developed method was thoroughly tested for various locations, sizes, and configurations of defects in the examined structure. Its application makes it possible to not only determine the presence of cracks but also estimate their size.

An efficient approach for no-reference image quality assessment based on statistical texture and structural features

This paper presents a rotation-invariant and computationally efficient no-reference image quality assessment (NR-IQA) model. It estimates the image quality based on texture and structural information associated with the images. The human visual system (HVS) uses perceptual features such as texture and structure as primary information to understand the visual scene and the image content. Moreover, the texture and structural information capture the loss of naturalness due to distortions in the image. Therefore, in this work, the important texture features are extracted using the local binary patterns (LBP).The modified LBP, also known as the hyper-smoothing LBP (H-LBP) and Laplacian of H-LBP (LH-LBP), represents the image structure. Further, the image quality prediction model computes the quality of the image based on the statistical feature measures of the texture and structural information. In the proposed approach, the image quality prediction model uses support vector regression (SVR) to measure image quality. Various experimentations are carried out on the LIVE and TID2013 database to test the effectiveness of the proposed NR-IQA model. The performance metrics such as Spearman rank-ordered correlation coefficient, Pearson linear correlation coefficient, and root mean square error is computed to show the efficiency of the presented approach. The experimental results illustrate a high correlation between the predicted quality score and the human visual perceptions. It is also found to be competitive with the best-performing full-reference and no-reference IQA models.

A No-Reference Image Quality Assessment Metric for Wood Images

Image Quality Assessment (IQA) is a vital element in improving the efficiency of an automatic recognition system of various wood species. There is a need to develop a No-Reference IQA (NR-IQA) system as a perfect and distortion free wood images may be impossible to be acquired in the dusty environment in timber factories. To the best of our knowledge, there is no NR-IQA developed for wood images specifically. Therefore, a Gray Level Co-Occurrence Matrix (GLCM) and Gabor features-based NR-IQA (GGNR-IQA) metric is proposed to assess the quality of wood images. The proposed metric is developed by training the support vector machine regression with GLCM and Gabor features calculated for wood images together with scores obtained from subjective evaluation. The proposed IQA metric is compared with a widely used NR-IQA metric, Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) and Full Reference-IQA (FR-IQA) metrics. Results shows that the proposed NR-IQA metric outperforms the BRISQUE and the FR-IQA metrics. Moreover, the proposed NR-IQA metric is beneficial in wood industry as a distortion free reference image is not needed to evaluate the wood images.

A Novel Just-Noticeable-Difference-Based Saliency-Channel Attention Residual Network for Full-Reference Image Quality Predictions

Recently, due to the strength of deep convolutional neural networks (CNN), many CNN-based image quality assessment (IQA) models have been studied. However, previous CNN-based IQA models likely have yet to utilize the characteristics of the human visual system (HVS) fully for IQA problems when they simply entrust everything to the CNN, expecting it to learn from a training dataset. Therefore, the performance capabilities of such deep-learning-based methods are somewhat saturated. However, in this article, we propose a novel saliency-channel attention residual network based on the just-noticeable-difference (JND) concept for full-reference image quality assessments (FR-IQA). It is referred to as JND-SalCAR and shows significant improvements in large IQA datasets with various types of distortion. The proposed JND-SalCAR effectively learns how to incorporate human psychophysical characteristics, such as visual saliency and JND, into image quality predictions. In the proposed network, a SalCAR block is devised so that perceptually important features can be extracted with the help of saliency-based spatial attention and channel attention schemes. In addition, a saliency map serves as a guideline for predicting a patch weight map in order to afford stable training of end-to-end optimization for the JND-SalCAR. To the best of our knowledge, our work presents the first HVS-inspired trainable FR-IQA network that considers both visual saliency and the JND characteristics of the HVS. When the visual saliency map and the JND probability map are explicitly given as priors, they can be usefully combined to predict IQA scores rated by humans more precisely, eventually leading to performance improvements and faster convergence. The experimental results show that the proposed JND-SalCAR significantly outperforms all recent state-of-the-art FR-IQA methods on large IQA datasets in terms of the Spearman rank order coefficient (SRCC) and the Pearson linear correlation coefficient (PLCC).

An Imaged Based Method for Universal Performance Evaluation of Electrical Impedance Tomography Systems.

This paper describes a simple and reproducible method for universal evaluation of the performance of electrical impedance tomography (EIT) systems using reconstructed images. To address the issues where common electrical parameters are not directly related to the quality of EIT images, based on objective full reference (FR) image quality assessment, the method provides a visually distinguishable hot colormap and two new FR metrics, the global and the more specific 'region of interest'. A passive 16 electrode EIT system using an application specific integrated circuit front-end was used to evaluate the proposed method. The measured results show, both visually and in terms of the proposed FR metrics, the impact on recorded EIT images with different design parameters and non-idealities. The paper also compares the image results of a passive electrode system with a matched 'single variable' active electrode system and demonstrates the merit of an active electrode system for noise interference. A figure of merit based on the FR metrics is proposed.

Blind 360-degree image quality assessment via saliency-guided convolution neural network

With the rapid development of virtual reality (VR) technologies, quality assessment of 360-degree images has become increasingly urgent. Unlike the traditional 2D images, the distortion is not evenly distributed in the 360-degree images, e.g., the projection distortion in the polar region of the Equirectangular projection (ERP) is more serious than that in other regions. Thus, the traditional 2D quality model cannot be directly applied to 360-degree images. In this paper, we propose a saliency-guided CNN model for blind 360-degree image quality assessment (SG360BIQA), which is mainly composed of a saliency prediction network (SP-Net) and a feature extraction network (F-Net). By training the whole network with the two sub-networks together, more discriminant features can be extracted and the mapping from feature representations to quality scores can be established more accurately. Besides, due to the lack of sufficient large 360-degree images database, we use a pre-trained network model instead of the initial random parameter model to overcome the limitation. Experimental results on two public 360-IQA databases demonstrate that our proposed model outperforms state-of-the-art full-reference and no-reference IQA metrics in terms of generalization ability and evaluation accuracy.

CNN Based No‐Reference HDR Image Quality Assessment

Motivated by the problems of nonuniversality and over-reliance on the original reference image in High dynamic range (HDR) Image quality assessment (IQA), a convolutional neural network-based algorithm for no-reference HDR image quality assessment is proposed. The Salience detection by self-resemblance (SDSR) algorithm which extracts the salient regions of the HDR image, is used to simulate the human visual attention mechanism. Then a visual quality perception network for training quality prediction models is designed according to the visual characteristics of luminance and contrast sensitivity. And this network consists of an Error estimation network (Error-net), a Perceptual resistance network (PR-net) and a mixing function. The experimental results indicate that the method proposed has high consistency with subjective perception, and the value of assessment metrics Spearman rank-order correlation coefficient (SROCC), Pearson product-moment correlation coefficient (PLCC) and Root mean square error (RMSE) correspondingly reaches 0.941, 0.910 and 8.176 as well. It is comparable with classic full-reference HDR IQA methods.

An optimized CNN-based quality assessment model for screen content image

Most existing convolutional neural network (CNN) based models designed for natural image quality assessment (IQA) employ image patches as training samples for data augmentation, and obtain final quality score by averaging all predicted scores of image patches. This brings two problems when applying these methods for screen content image (SCI) quality assessment. Firstly, SCI contains more complex content compared to natural image. As a result, qualities of SCI patches are different, and the subjective differential mean opinion score (DMOS) is not appropriate as qualities of all image patches. Secondly, the average score of image patches does not represent the quality of entire SCI since the human visual system (HVS) is sensitive to image patches containing texture and edge information. In this paper, we propose a novel quadratic optimized model based on the deep convolutional neural network (QODCNN) for full-reference (FR) and no-reference (NR) SCI quality assessment to overcome these two problems. The contribution of our algorithm can be concluded as follows: 1) Considering the characteristics of SCIs, a valid network architecture is designed for both NR and FR visual quality evaluation of SCIs, which makes the networks learn the feature differences for FR-IQA; 2) with the consideration of correlation between local quality and DMOS, a training data selection method is proposed to fine-tune the pre-trained model with valid SCI patches; 3) an adaptive pooling approach is employed to fuse patch quality to obtain image quality, owns strong noise robust and effects on both FR and NR IQA. Experimental results verify that our model outperforms both current no-reference and full-reference image quality assessment methods on the benchmark screen content image quality assessment database (SIQAD). Cross-database evaluation shows high generalization ability and high effectiveness of our model.

A comprehensive assessment of content-based image retrieval using selected full reference image quality assessment algorithms

In the area of full-reference (FR) objective ‘image quality assessment’ (IQA), a huge amount of improvement has been done in the last few years that can calculate image quality, consistently. Contrarily, query-based image/video databases and search engines retrieve related data using ‘ranking and indexing’ depending on stored content. It is also to be noted that both techniques use feature extraction to achieve their goal. The efficiency of seven selected FR-IQA schemes is described in this article for the retrieval of an image signal. Extensive tests are done on two freely accessible databases. The comparison results express that mean-structural-similarity-index-measure (MSSIM), and feature-similarity-index-measure (FSIM) offer superior outcome than other IQA models. Our assessment outcome and the related thought will be very much useful for the researchers to understand the latest application areas of the IQA model in the area of image searching and retrieval.

Image denoising based on BCOLTA: Dataset and study

Robot deburring is an effective method for improving the surface quality of the high-voltage copper contact. The first step of robot deburring is to acquire the burr images. We propose a new burr mathematical model and build a real burr image dataset for burr image denoising. In order to improve burr image denoising effects of the high-voltage copper contact, this study proposes an online burr image denoising algorithm, that is, block cosparsity overcomplete learning transform algorithm (BCOLTA). The penalty term and the condition number are affected by the burr parameter. The clustering and transform alternate minimisation algorithms are adopted to achieve lower computational cost and better denoising effect. In addition, BCOLTA also has a good adaptibility to inherent noise images, especially in Gaussian noise. Compared with other traditional and deep learning algorithms by no reference and full reference image quality assessment methods, BCOLTA has state-of-the-art denoising effects and computational complexity on dealing with burr images. This research will play an important role in the intelligent manufacturing field.

Comparison of Full-Reference Image Quality Models for Optimization of Image Processing Systems.

The performance of objective image quality assessment (IQA) models has been evaluated primarily by comparing model predictions to human quality judgments. Perceptual datasets gathered for this purpose have provided useful benchmarks for improving IQA methods, but their heavy use creates a risk of overfitting. Here, we perform a large-scale comparison of IQA models in terms of their use as objectives for the optimization of image processing algorithms. Specifically, we use eleven full-reference IQA models to train deep neural networks for four low-level vision tasks: denoising, deblurring, super-resolution, and compression. Subjective testing on the optimized images allows us to rank the competing models in terms of their perceptual performance, elucidate their relative advantages and disadvantages in these tasks, and propose a set of desirable properties for incorporation into future IQA models.

Automatic Object Removal With Obstructed Façades Completion Using Semantic Segmentation and Generative Adversarial Inpainting

Automatic object removal with obstructed facades completion in the urban environment is essential for many applications such as scene restoration, environmental impact assessment, and urban mapping. However, the previous object removal typically requires a user to manually create a mask around unwanted objects and obtain background facade information in advance, which would be labor-intensive when implementing multitasking projects. Moreover, accurately detecting objects to be removed in the cityscape and inpainting the static obstructed building facade to obtain plausible images are the main challenges for this objective. To overcome these difficulties, this study addresses the object removal with the facade inpainting problem from the following two aspects. First, we proposed an image-based cityscape elimination method for automatic object removal and facade inpainting by applying semantic segmentation to detect several classes, including pedestrians, riders, vegetation, and cars, as well as using generative adversarial networks (GANs) for filling detected regions by background textures and patching information from street-level imagery. Second, we proposed a workflow to filter unoccluded building facades from street view images automatically and tailored a dataset for the GAN-based image inpainting model with original and mask images. Furthermore, several full-reference image quality assessment (IQA) metrics are introduced to evaluate the generated image quality. Validation results demonstrated the feasibility and effectiveness of our proposed method, and the synthetic image is visually realistic and semantically consistent.

Evaluating Accuracy of Algorithms Providing Subsurface Properties Using Full-Reference Image Quality Assessment

Evaluating Accuracy of Algorithms Providing Subsurface Properties Using Full-Reference Image Quality Assessment

On Numerical Implementation of the Laplace Equation-Based Image Inpainting

In the current study, we focus on the development of a numerical algorithm for the Laplace equation-based image inpainting problem. A software program to implement the proposed algorithm is also developed. In experiments, the proposed method recovered corrupted images effectively. It can also remove defects as well as unnecessary objects successfully. The program provides an effective, convenient, and easy solution to perform the image inpainting in practical tasks. It also provides a useful feature to assess image quality after processing with both full reference and blind image quality assessment.

Learning to predict the quality of distorted-then-compressed images via a deep neural network

Being captured by amateur photographers, reciprocally propagated through multimedia pipelines, and compressed with different levels, real-world images usually suffer from a wide variety of hybrid distortions. Faced with this scenario, full-reference (FR) image quality assessment (IQA) algorithms can not deliver promising predictions due to the inferior references. Meanwhile, existing no-reference (NR) IQA algorithms remain limited in their efficacy to deal with different distortion types. To address this obstacle, we explore a NR-IQA metric by predicting the perceptual quality of distorted-then-compressed images using a deep neural network (DNN). First, we propose a novel two-stream DNN to handle both authentic distortions and synthetic compressions and adopt effective strategies to pre-train the two branches of the network. Specifically, we transfer the knowledge learned from in-the-wild images to account for authentic distortions by utilizing a pre-trained deep convolutional neural network (CNN) to provide meaningful initializations. Meanwhile, we build a CNN for synthetic compressions and pre-train it on a dataset including synthetic compressed images. Subsequently, we bilinearly pool these two sets of features as the image representation. The overall network is fine-tuned on an elaborately-designed auxiliary dataset, which is annotated by a reliable objective quality metric. Furthermore, we integrate the output of the authentic-distortion-aware branch with that of the overall network following a two-step prediction manner to boost the prediction performance, which can be applied in the distorted-then-compressed scenario when the reference image is available. Extensive experimental results on several databases especially on the LIVE Wild Compressed Picture Quality Database show that the proposed method achieves state-of-the-art performance with good generalizability and moderate computational complexity.