Image Quality Assessment Metrics Research Articles

Enhanced Solar Coronal Imaging: A GAN Approach with Fused Attention and Perceptual Quality Enhancement

The activity of the solar corona has a significant impact on all aspects of human life. People typically use images obtained from astronomical telescopes to observe coronal activities, among which the Atmospheric Imaging Assembly (AIA) of the Solar Dynamics Observatory (SDO) is particularly widely used. However, due to resolution limitations, we have begun to study the application of generative adversarial network super-resolution techniques to enhance the image data quality for a clearer observation of the fine structures and dynamic processes in the solar atmosphere, which improves the prediction accuracy of solar activities. We aligned SDO/AIA images with images from the High-Resolution Coronal Imager (Hi-C) to create a dataset. This research proposes a new super-resolution method named SAFCSRGAN, which includes a spatial attention module that incorporates channel information, allowing the network model to better capture the corona’s features. A Charbonnier loss function was introduced to enhance the perceptual quality of the super-resolution images. Compared to the original method using ESRGAN, our method achieved an 11.9% increase in Peak Signal-to-Noise Ratio (PSNR) and a 4.8% increase in Structural Similarity (SSIM). Additionally, we introduced two perceptual image quality assessment metrics, the Natural Image Quality Evaluator (NIQE) and Learned Perceptual Image Patch Similarity (LPIPS), which improved perceptual quality by 10.8% and 1.3%, respectively. Finally, our experiments demonstrated that our improved model surpasses other models in restoring the details of coronal images.

No-Reference Hyperspectral Image Quality Assessment via Ranking Feature Learning

In hyperspectral image (HSI) reconstruction tasks, due to the lack of ground truth in real imaging processes, models are usually trained and validated on simulation datasets and then tested on real measurements captured by real HSI imaging systems. However, due to the gap between the simulation imaging process and the real imaging process, the best model validated on the simulation dataset may fail on real measurements. To obtain the best model for the real-world task, it is crucial to design a suitable no-reference HSI quality assessment metric to reflect the reconstruction performance of different models. In this paper, we propose a novel no-reference HSI quality assessment metric via ranking feature learning (R-NHSIQA), which calculates the Wasserstein distance between the distribution of the deep features of the reconstructed HSIs and the benchmark distribution. Additionally, by introducing the spectral self-attention mechanism, we propose a Spectral Transformer (S-Transformer) to extract the spatial-spectral representative deep features of HSIs. Furthermore, to extract quality-sensitive deep features, we use quality ranking as a pre-training task to enhance the representation capability of the S-Transformer. Finally, we introduce the Wasserstein distance to measure the distance between the distribution of the deep features and the benchmark distribution, improving the assessment capacity of our method, even with non-overlapping distributions. The experimental results demonstrate that the proposed metric yields consistent results with multiple full-reference image quality assessment (FR-IQA) metrics, validating the idea that the proposed metric can serve as a substitute for FR-IQA metrics in real-world tasks.

Blind cartoon image quality assessment based on local structure and chromatic statistics

With the help of computer-assisted systems, cartoon synthesis has become convenient and efficient by reusing existing cartoon materials. However, quality evaluation of the obtained cartoon image still relies on labor-intensive subjective judgment. This accordingly raises an urgent demand for effective quality evaluation methods to automatically select a cartoon image of high quality from a set of candidates with different parameter settings. In this paper, a new blind image quality assessment metric is developed for evaluating the perceptual quality of cartoon images by considering structure and chromatic distortions. The extracted gradient-based local structure features and multiscale chromatic statistical features are integrated into one representation for an overall perceptual quality prediction. Experimental results on two benchmark cartoon image datasets, i.e., NBU-CIQAD and HFUT-CID, indicate that the proposed metric outperforms both the state-of-the-art blind quality evaluation methods designed for natural images or synthetic images.

High-Quality Synthesized Face Sketch Using Generative Reference Prior

Face sketch synthesis (FSS) is considered as an image-to-image translation problem, where a face sketch is generated from an input face photo. FSS plays a vital role in video/image surveillance-based law enforcement. In this paper, motivated by the recent success of generative adversarial networks (GAN), we consider conditional GAN (cGAN) to approach the problem of face sketch synthesis. However, despite the powerful cGAN model’s ability to generate fine textures, low-quality inputs characterized by the facial sketches drawn by artists cannot offer realistic and faithful details and have unknown degradation due to the drawing process, while high-quality references are inaccessible or even unexistent. In this context, we propose an approach based on Generative Reference Prior (GRP) to improve the synthesized face sketch perception. Our proposed model, that we call cGAN-GRP, leverages diverse and rich priors encapsulated in a pre-trained face GAN for generating high-quality facial sketch synthesis. Extensive experiments on publicly available face databases using facial sketch recognition rate and image quality assessment metrics as criteria demonstrate the effectiveness of our proposed model compared to several state-of-the-art methods.

DICAM: Deep Inception and Channel-wise Attention Modules for underwater image enhancement

In underwater environments, imaging devices suffer from water turbidity, attenuation of lights, scattering, and particles, leading to low quality, poor contrast, and biased color images. This has led to great challenges for underwater condition monitoring and inspection using conventional vision techniques. In recent years, underwater image enhancement has attracted increasing attention due to its critical role in improving the performance of current computer vision tasks in underwater object detection and segmentation. As existing methods, built mainly from natural scenes, have performance limitations in improving the color richness and distributions we propose a novel deep learning-based approach namely Deep Inception and Channel-wise Attention Modules (DICAM) to enhance the quality, contrast, and color cast of the hazy underwater images. The proposed DICAM model enhances the quality of underwater images, considering both the proportional degradations and non-uniform color cast. Extensive experiments on two publicly available underwater image enhancement datasets have verified the superiority of our proposed model compared with several state-of-the-art conventional and deep learning-based methods in terms of full-reference and reference-free image quality assessment metrics. The source code of our DICAM model is available at https://github.com/hfarhaditolie/DICAM.

Image Quality Assessment of Multi-Satellite Pan-Sharpening Approach: A Case Study using Sentinel-2 Synthetic Panchromatic Image and Landsat-8

INTRODUCTION: The satellite's physical and technical capabilities limit high spectral and spatial resolution image acquisition. In Remote Sensing (RS), when high spatial and spectral resolution data is essential for specific Geographic Information System (GIS) applications, Pan Sharpening (PanS) becomes imperative in obtaining such data. 
 OBJECTIVES: Study aims to enhance the spatial resolution of the multispectral Landsat-8 (L8) images using a synthetic panchromatic band generated by averaging four fine-resolution bands in the Sentinel-2 (S2) images. 
 METHODS: Evaluation of the proposed multi-satellite PanS approach, three different PanS techniques, Smoothed Filter Intensity Modulation (SFIM), Gram-Schmidt (GS), and High Pass Filter Additive (HPFA) are used for two different study areas. The techniques' effectiveness was evaluated using well-known Image Quality Assessment Metrics (IQAM) such as Root Mean Square Error (RMSE), Correlation Coefficient (CC), Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS), and Relative Average Spectral Error (RASE). This study leveraged the GEE platform for datasets and implementation.
 RESULTS: The promising values were provided by the GS technique, followed by the SFIM technique, whereas the HPFA technique produced the lowest quantitative result. 
 CONCLUSION: In this study, the spectral bands of the MS image’s performance show apparent variation with respect to that of the different PanS techniques used.

Channel-wise attention enhanced and structural similarity constrained cycleGAN for effective synthetic CT generation from head and neck MRI images

BackgroundMagnetic resonance imaging (MRI) plays an increasingly important role in radiotherapy, enhancing the accuracy of target and organs at risk delineation, but the absence of electron density information limits its further clinical application. Therefore, the aim of this study is to develop and evaluate a novel unsupervised network (cycleSimulationGAN) for unpaired MR-to-CT synthesis.MethodsThe proposed cycleSimulationGAN in this work integrates contour consistency loss function and channel-wise attention mechanism to synthesize high-quality CT-like images. Specially, the proposed cycleSimulationGAN constrains the structural similarity between the synthetic and input images for better structural retention characteristics. Additionally, we propose to equip a novel channel-wise attention mechanism based on the traditional generator of GAN to enhance the feature representation capability of deep network and extract more effective features. The mean absolute error (MAE) of Hounsfield Units (HU), peak signal-to-noise ratio (PSNR), root-mean-square error (RMSE) and structural similarity index (SSIM) were calculated between synthetic CT (sCT) and ground truth (GT) CT images to quantify the overall sCT performance.ResultsOne hundred and sixty nasopharyngeal carcinoma (NPC) patients who underwent volumetric-modulated arc radiotherapy (VMAT) were enrolled in this study. The generated sCT of our method were more consistent with the GT compared with other methods in terms of visual inspection. The average MAE, RMSE, PSNR, and SSIM calculated over twenty patients were 61.88 ± 1.42, 116.85 ± 3.42, 36.23 ± 0.52 and 0.985 ± 0.002 for the proposed method. The four image quality assessment metrics were significantly improved by our approach compared to conventional cycleGAN, the proposed cycleSimulationGAN produces significantly better synthetic results except for SSIM in bone.ConclusionsWe developed a novel cycleSimulationGAN model that can effectively create sCT images, making them comparable to GT images, which could potentially benefit the MRI-based treatment planning.

Comparative Analysis of Image Quality Assessment Metrics: MSE, PSNR, SSIM and FSIM

Comparative Analysis of Image Quality Assessment Metrics: MSE, PSNR, SSIM and FSIM

Pansharpening Low-Altitude Multispectral Images of Potato Plants Using a Generative Adversarial Network

Image preprocessing and fusion are commonly used for enhancing remote-sensing images, but the resulting images often lack useful spatial features. As the majority of research on image fusion has concentrated on the satellite domain, the image-fusion task for Unmanned Aerial Vehicle (UAV) images has received minimal attention. This study investigated an image-improvement strategy by integrating image preprocessing and fusion tasks for UAV images. The goal is to improve spatial details and avoid color distortion in fused images. Techniques such as image denoising, sharpening, and Contrast Limited Adaptive Histogram Equalization (CLAHE) were used in the preprocessing step. The unsharp mask algorithm was used for image sharpening. Wiener and total variation denoising methods were used for image denoising. The image-fusion process was conducted in two steps: (1) fusing the spectral bands into one multispectral image and (2) pansharpening the panchromatic and multispectral images using the PanColorGAN model. The effectiveness of the proposed approach was evaluated using quantitative and qualitative assessment techniques, including no-reference image quality assessment (NR-IQA) metrics. In this experiment, the unsharp mask algorithm noticeably improved the spatial details of the pansharpened images. No preprocessing algorithm dramatically improved the color quality of the enhanced images. The proposed fusion approach improved the images without importing unnecessary blurring and color distortion issues.

WaterPairs: a paired dataset for underwater image enhancement and underwater object detection

Due to its importance in marine engineering and aquatic robotics, underwater image enhancement works as a preprocessing step to improve the performance of high-level vision tasks such as underwater object detection and recognition. Although several studies exhibit that underwater image enhancement algorithms can boost the detection accuracy of detectors, no work has focused on studying the relationship between these two tasks. This is mainly because current underwater datasets lack either bounding box annotations or high-quality reference images, based on which detection accuracy or image quality assessment metrics are calculated. To examine how underwater image enhancement methods affect underwater object detection tasks, we provide a large-scale underwater object detection dataset with both bounding box annotations and high-quality reference images, namely, the WaterPairs dataset. The WaterPairs dataset offers a platform for researchers to comprehensively study the influence of underwater image enhancement algorithms on underwater object detection tasks. We will release our dataset at https://github.com/IanDragon/WaterPairs once this paper is accepted.

An Long Short-Term Memory Model with Multi-Scale Context Fusion and Attention for Radar Echo Extrapolation

Precipitation nowcasting is critical for areas such as agriculture, water resource management, urban drainage systems, transport and disaster preparedness. In recent years, methods such as convolutional recurrent neural networks (ConvRNN) in deep learning techniques have been used to solve this task. Despite the effective improvement in forecasting quality, there are still problems with blurred and distorted prediction images, as well as difficulties in effectively forecasting high echo regions. To solve the above problems, this article presents a spatio-temporal long–short-term memory network model in view of multi-scale context fusion and attention mechanisms. This method fully extracts the short-term context information of different scales of radar image through the multi-scale context fusion module. The attention module broadens the time perception domain of the prediction unit so that the model perceives more historical time dynamics. Using the Hong Kong region weather radar data as a sample, the results of the experimental comparative analysis show that the spatio-temporal long and short-term memory network in view of multi-scale context fusion and attention mechanism achieves better prediction performance. Our model is effective in improving both image quality and meteorological assessment metrics with higher accuracy and more details.

Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Images

Computer graphics images (CGIs) are artificially generated by means of computer programs and are widely perceived under various scenarios, such as games, streaming media, etc. In practice, the quality of CGIs consistently suffers from poor rendering during production, inevitable compression artifacts during the transmission of multimedia applications, and low aesthetic quality resulting from poor composition and design. However, few works have been dedicated to dealing with the challenge of computer graphics image quality assessment (CGIQA). Most image quality assessment (IQA) metrics are developed for natural scene images (NSIs) and validated on databases consisting of NSIs with synthetic distortions, which are not suitable for in-the-wild CGIs. To bridge the gap between evaluating the quality of NSIs and CGIs, we construct a large-scale in-the-wild CGIQA database consisting of 6,000 CGIs (CGIQA-6k) and carry out the subjective experiment in a well-controlled laboratory environment to obtain the accurate perceptual ratings of the CGIs. Then, we propose an effective deep learning–based no-reference (NR) IQA model by utilizing both distortion and aesthetic quality representation. Experimental results show that the proposed method outperforms all other state-of-the-art NR IQA methods on the constructed CGIQA-6k database and other CGIQA-related databases. The database is released athttps://github.com/zzc-1998/CGIQA6K.

IMPACT OF UAV AND SENTINEL-2A IMAGERY FUSION ON VEGETATION INDICES PERFORMANCE

Abstract. Image fusion techniques can improve the quality of remote sensing images by combining high spatial resolution images with low spectral resolution images. This enhancement of the images may impact the performance of various vegetation indices (VI’s). This study investigates the impact of image fusion on the quality of vegetation indices by fusing UAV (Unmanned Aerial Vehicle) bands with Sentinel-2A images using Principal Component Analysis (PCA) and Brovery Transform (BT) fusion techniques.The fused images were used to calculate the Normalized Difference Vegetation Index, Normalized Difference Red Edge, Green Red Vegetation Index, and Normalized Difference Water Index. To assess the performance of the fused images, several image quality assessment metrics were used, including Root Mean Square Error (RMSE), Entropy, etc...The results showed that image fusion techniques can improve the quality of images which is important to assess crop health. The PCA image fusion technique showed higher quality than the BT technique. The PCA fused images had lower RMSE, ERGAS, and Entropy Difference and higher UIQI, CC, and SSIM values than the original images. Moreover, the fused images produced higher VIs values than the Sentinel-2A images.Finally, scatter plots were created to compare the correlation between the VIs calculated from the original and fused images. The results showed a strong correlation between the VIs calculated from the Sentinel-2A and fused images, indicating that the fused images can accurately estimate vegetation health parameters. Overall, this study demonstrates the potential of image fusion techniques to improve the quality of VI’s for monitoring vegetation health.

A feasibility analysis of image approximation with image quality assessments

AbstractWith technological developments, the resolution of the display systems is increasing, causing image data to increase. Several technologies are aiding in optimizing these data during transmission and storage while maintaining quality. Still, the internal data transmission for accessing the memory sub‐system during reading and writing cycles consumes significant power while requiring large memory for storage, which is inefficient, especially for portable displays. Hence, this work investigates the feasibility of reducing data through approximation for images in different formats and resolutions for human and computer vision based applications. Different image quality assessment metrics are utilized for performance evaluations with optimum image quality assessment selection according to application requirements. In addition, the feasibility analysis is conducted for three different applications as examples of computer vision. The experimental analysis highlights theoretical and industrial importance by showing that image data can be reduced from 25.0% to 62.5%, while satisfying requirements for HVS‐based and computer vision‐based applications.

A Motion Deblurring Network for Enhancing UAV Image Quality in Bridge Inspection

Unmanned aerial vehicles (UAVs) have been increasingly utilized for facility safety inspections due to their superior safety, cost effectiveness, and inspection accuracy compared to traditional manpower-based methods. High-resolution images captured by UAVs directly contribute to identifying and quantifying structural defects on facility exteriors, making image quality a critical factor in achieving accurate results. However, motion blur induced by external factors such as vibration, low light conditions, and wind during UAV operation significantly degrades image quality, leading to inaccurate defect detection and quantification. To address this issue, this research proposes a deblurring network using a Generative Adversarial Network (GAN) to eliminate the motion blur effect in UAV images. The GAN-based motion deblur network represents an image inpainting method that leverages generative models to correct blurry artifacts, thereby generating clear images. Unlike previous studies, this proposed approach incorporates deblur and blur learning modules to realistically generate blur images required for training the generative models. The UAV images processed using the motion deblur network are evaluated using a quality assessment method based on local blur map and other well-known image quality assessment (IQA) metrics. Moreover, in the experiment of crack detection utilizing the object detection system, improved detection results are observed when using enhanced images. Overall, this research contributes to improving the quality and accuracy of facility safety inspections conducted with UAV-based inspections by effectively addressing the challenges associated with motion blur effects in UAV-captured images.

Attentive Deep Image Quality Assessment for Omnidirectional Stitching

Omnidirectional images or videos are commonly generated via the stitching of multiple images or videos, and the quality of omnidirectional stitching strongly influences the quality of experience (QoE) of the generated scenes. Although there were many studies research the omnidirectional image quality assessment (IQA), the evaluation of the omnidirectional stitching quality has not been sufficiently explored. In this paper, we focus on the IQA for the omnidirectional stitching of dual fisheye images. We first establish an omnidirectional stitching image quality assessment (OSIQA) database, which includes 300 distorted images and 300 corresponding reference images generated from 12 raw scenes. The database contains a variety of distortion types caused by omnidirectional stitching, including color distortion, geometric distortion, blur distortion, and ghosting distortion, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">etc.</i> A subjective quality assessment study is conducted on the database and human opinion scores are collected for the distorted omnidirectional images. We then devise a deep learning based objective IQA metric termed Attentive Multi-channel IQA Net. In particular, we extend hyper-ResNet by developing a subnetwork for spatial attention and propose a spatial regularization item. Experimental results show that our proposed FR and NR models achieve the best performance compared with the state-of-the-art FR and NR IQA metrics on the OSIQA database. The OSIQA database as well as the proposed Attentive Multi-channel IQA Net will be released to facilitate future research.

PVBLiF: A Pseudo Video-Based Blind Quality Assessment Metric for Light Field Image

Going beyond traditional 2D imaging is not only an emerging trend of imaging technology, but also the key to a more immersive user experience. Light Field Image (LFI) is a typical high-dimensional imaging format, and the quality evaluation of which is very challenging but necessary. In this paper, we propose a novel Pseudo Video-based Blind quality assessment metric for Light Field image (PVBLiF). In contrast to most previous Light Field Image Quality Assessment (LF-IQA) metrics, in which different types of 2D representations derived from LFI are used for quality assessment indirectly, our metric exploits a more intuitive 3D representation, named Pseudo Video Block Sequence (PVBS), to evaluate the perceptual quality of LFI. For this purpose, we first divide the LFI into a massive number of non-overlapping PVBSs, which simultaneously contain spatial and angular information of LFI. Then, we propose a novel network (named PVBSNet) based on Convolutional Neural Networks (CNNs) to extract the spatio-angular features of PVBS and further evaluate the PVBS quality. The proposed PVBSNet consists of four stages: multi-information division, intra-feature extraction, cross-feature fusion, and quality regression. Finally, a Saliency- and Variance-guided Pooling (SVPooling) method is presented to integrate all the PVBS quality into the overall quality of LFI. The proposed PVBLiF metric has been extensively evaluated on three widely-used LFI datasets: Win5-LID, NBU-LF1.0, and SHU. Experimental results demonstrate that our proposed PVBLiF metric outperforms state-of-the-art metrics and is capable of highly approximating the performance of human observers. The source code of PVBLiF is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ZhengyuZhang96/PVBLiF</uri> .

Analysis of Image Quality Assessment Methods and Metrics: A Comprehensive Review

The evaluation of metrics, the effects of machine learning, and the use of preprocessing techniques for image enhancement constitute the essence of image quality. Through a thorough investigation, this study seeks to identify the core components of image quality. First, various measures for gauging image quality are compared to see how well they work. These metrics offer precise evaluations of aspects of a picture such as sharpness, contrast, and color integrity. The effect of machine learning techniques on the quality of images is then examined. The study investigates the use of machine learning approaches to improve image quality by training models on substantial datasets and tailoring them for certain tasks. In addition, the function of preprocessing methods in picture enhancement is investigated. Before further processing or analysis, the image quality is improved using various techniques, including noise reduction, image denoising, and local entropy filtering. The findings of this work offer important new perspectives on the assessment of image quality measurements, the potential of machine learning for image enhancement, and the significance of preprocessing methods in producing higher image quality.

Raindrop-Removal Image Translation Using Target-Mask Network with Attention Module

Image processing plays a crucial role in improving the performance of models in various fields such as autonomous driving, surveillance cameras, and multimedia. However, capturing ideal images under favorable lighting conditions is not always feasible, particularly in challenging weather conditions such as rain, fog, or snow, which can impede object recognition. This study aims to address this issue by focusing on generating clean images by restoring raindrop-deteriorated images. Our proposed model comprises a raindrop-mask network and a raindrop-removal network. The raindrop-mask network is based on U-Net architecture, which learns the location, shape, and brightness of raindrops. The rain-removal network is a generative adversarial network based on U-Net and comprises two attention modules: the raindrop-mask module and the residual convolution block module. These modules are employed to locate raindrop areas and restore the affected regions. Multiple loss functions are utilized to enhance model performance. The image-quality assessment metrics of proposed method, such as SSIM, PSNR, CEIQ, NIQE, FID, and LPIPS scores, are 0.832, 26.165, 3.351, 2.224, 20.837, and 0.059, respectively. Comparative evaluations against state-of-the-art models demonstrate the superiority of our proposed model based on qualitative and quantitative results.

TPP: Deep learning based threshold post-processing multi-focus image fusion method

How to accurately determine focused and defocused domains is a difficult problem in the field of multi-focused image fusion. To solve this problem, we propose a new deep learning method called threshold post processing(TPP). The TPP network model contains three architecturally different channels, and three different weighted maps can be obtained for the same input. The output of the model is fused and thresholded to obtain a final decision map, which is then used to complete the image fusion task. The decision map represents the focus condition of the source image at a pixel level. This allows TPP to retain focused regions of the source image more accurately than other existing deep learning methods. The multi-channel network model combined with the thresholding process optimizes the final decision map. It avoids the drawback of other methods that cause inaccurate discrimination of the focused area due to a single network channel and therefore cannot be corrected. Subsequent experimental results show that the images generated by TPP maintain a high level under the various image quality assessment metrics measures.