Image Quality Assessment Metrics Research Articles

DICAM: Deep Inception and Channel-wise Attention Modules for underwater image enhancement

In underwater environments, imaging devices suffer from water turbidity, attenuation of lights, scattering, and particles, leading to low quality, poor contrast, and biased color images. This has led to great challenges for underwater condition monitoring and inspection using conventional vision techniques. In recent years, underwater image enhancement has attracted increasing attention due to its critical role in improving the performance of current computer vision tasks in underwater object detection and segmentation. As existing methods, built mainly from natural scenes, have performance limitations in improving the color richness and distributions we propose a novel deep learning-based approach namely Deep Inception and Channel-wise Attention Modules (DICAM) to enhance the quality, contrast, and color cast of the hazy underwater images. The proposed DICAM model enhances the quality of underwater images, considering both the proportional degradations and non-uniform color cast. Extensive experiments on two publicly available underwater image enhancement datasets have verified the superiority of our proposed model compared with several state-of-the-art conventional and deep learning-based methods in terms of full-reference and reference-free image quality assessment metrics. The source code of our DICAM model is available at https://github.com/hfarhaditolie/DICAM.

Image Quality Assessment of Multi-Satellite Pan-Sharpening Approach: A Case Study using Sentinel-2 Synthetic Panchromatic Image and Landsat-8

INTRODUCTION: The satellite's physical and technical capabilities limit high spectral and spatial resolution image acquisition. In Remote Sensing (RS), when high spatial and spectral resolution data is essential for specific Geographic Information System (GIS) applications, Pan Sharpening (PanS) becomes imperative in obtaining such data. 
 OBJECTIVES: Study aims to enhance the spatial resolution of the multispectral Landsat-8 (L8) images using a synthetic panchromatic band generated by averaging four fine-resolution bands in the Sentinel-2 (S2) images. 
 METHODS: Evaluation of the proposed multi-satellite PanS approach, three different PanS techniques, Smoothed Filter Intensity Modulation (SFIM), Gram-Schmidt (GS), and High Pass Filter Additive (HPFA) are used for two different study areas. The techniques' effectiveness was evaluated using well-known Image Quality Assessment Metrics (IQAM) such as Root Mean Square Error (RMSE), Correlation Coefficient (CC), Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS), and Relative Average Spectral Error (RASE). This study leveraged the GEE platform for datasets and implementation.
 RESULTS: The promising values were provided by the GS technique, followed by the SFIM technique, whereas the HPFA technique produced the lowest quantitative result. 
 CONCLUSION: In this study, the spectral bands of the MS image’s performance show apparent variation with respect to that of the different PanS techniques used.

Channel-wise attention enhanced and structural similarity constrained cycleGAN for effective synthetic CT generation from head and neck MRI images

BackgroundMagnetic resonance imaging (MRI) plays an increasingly important role in radiotherapy, enhancing the accuracy of target and organs at risk delineation, but the absence of electron density information limits its further clinical application. Therefore, the aim of this study is to develop and evaluate a novel unsupervised network (cycleSimulationGAN) for unpaired MR-to-CT synthesis.MethodsThe proposed cycleSimulationGAN in this work integrates contour consistency loss function and channel-wise attention mechanism to synthesize high-quality CT-like images. Specially, the proposed cycleSimulationGAN constrains the structural similarity between the synthetic and input images for better structural retention characteristics. Additionally, we propose to equip a novel channel-wise attention mechanism based on the traditional generator of GAN to enhance the feature representation capability of deep network and extract more effective features. The mean absolute error (MAE) of Hounsfield Units (HU), peak signal-to-noise ratio (PSNR), root-mean-square error (RMSE) and structural similarity index (SSIM) were calculated between synthetic CT (sCT) and ground truth (GT) CT images to quantify the overall sCT performance.ResultsOne hundred and sixty nasopharyngeal carcinoma (NPC) patients who underwent volumetric-modulated arc radiotherapy (VMAT) were enrolled in this study. The generated sCT of our method were more consistent with the GT compared with other methods in terms of visual inspection. The average MAE, RMSE, PSNR, and SSIM calculated over twenty patients were 61.88 ± 1.42, 116.85 ± 3.42, 36.23 ± 0.52 and 0.985 ± 0.002 for the proposed method. The four image quality assessment metrics were significantly improved by our approach compared to conventional cycleGAN, the proposed cycleSimulationGAN produces significantly better synthetic results except for SSIM in bone.ConclusionsWe developed a novel cycleSimulationGAN model that can effectively create sCT images, making them comparable to GT images, which could potentially benefit the MRI-based treatment planning.

Comparative Analysis of Image Quality Assessment Metrics: MSE, PSNR, SSIM and FSIM

Comparative Analysis of Image Quality Assessment Metrics: MSE, PSNR, SSIM and FSIM

Pansharpening Low-Altitude Multispectral Images of Potato Plants Using a Generative Adversarial Network

Image preprocessing and fusion are commonly used for enhancing remote-sensing images, but the resulting images often lack useful spatial features. As the majority of research on image fusion has concentrated on the satellite domain, the image-fusion task for Unmanned Aerial Vehicle (UAV) images has received minimal attention. This study investigated an image-improvement strategy by integrating image preprocessing and fusion tasks for UAV images. The goal is to improve spatial details and avoid color distortion in fused images. Techniques such as image denoising, sharpening, and Contrast Limited Adaptive Histogram Equalization (CLAHE) were used in the preprocessing step. The unsharp mask algorithm was used for image sharpening. Wiener and total variation denoising methods were used for image denoising. The image-fusion process was conducted in two steps: (1) fusing the spectral bands into one multispectral image and (2) pansharpening the panchromatic and multispectral images using the PanColorGAN model. The effectiveness of the proposed approach was evaluated using quantitative and qualitative assessment techniques, including no-reference image quality assessment (NR-IQA) metrics. In this experiment, the unsharp mask algorithm noticeably improved the spatial details of the pansharpened images. No preprocessing algorithm dramatically improved the color quality of the enhanced images. The proposed fusion approach improved the images without importing unnecessary blurring and color distortion issues.

WaterPairs: a paired dataset for underwater image enhancement and underwater object detection

Due to its importance in marine engineering and aquatic robotics, underwater image enhancement works as a preprocessing step to improve the performance of high-level vision tasks such as underwater object detection and recognition. Although several studies exhibit that underwater image enhancement algorithms can boost the detection accuracy of detectors, no work has focused on studying the relationship between these two tasks. This is mainly because current underwater datasets lack either bounding box annotations or high-quality reference images, based on which detection accuracy or image quality assessment metrics are calculated. To examine how underwater image enhancement methods affect underwater object detection tasks, we provide a large-scale underwater object detection dataset with both bounding box annotations and high-quality reference images, namely, the WaterPairs dataset. The WaterPairs dataset offers a platform for researchers to comprehensively study the influence of underwater image enhancement algorithms on underwater object detection tasks. We will release our dataset at https://github.com/IanDragon/WaterPairs once this paper is accepted.

An Long Short-Term Memory Model with Multi-Scale Context Fusion and Attention for Radar Echo Extrapolation

Precipitation nowcasting is critical for areas such as agriculture, water resource management, urban drainage systems, transport and disaster preparedness. In recent years, methods such as convolutional recurrent neural networks (ConvRNN) in deep learning techniques have been used to solve this task. Despite the effective improvement in forecasting quality, there are still problems with blurred and distorted prediction images, as well as difficulties in effectively forecasting high echo regions. To solve the above problems, this article presents a spatio-temporal long–short-term memory network model in view of multi-scale context fusion and attention mechanisms. This method fully extracts the short-term context information of different scales of radar image through the multi-scale context fusion module. The attention module broadens the time perception domain of the prediction unit so that the model perceives more historical time dynamics. Using the Hong Kong region weather radar data as a sample, the results of the experimental comparative analysis show that the spatio-temporal long and short-term memory network in view of multi-scale context fusion and attention mechanism achieves better prediction performance. Our model is effective in improving both image quality and meteorological assessment metrics with higher accuracy and more details.

Hybrid Simulated Annealing‐Evaporation Rate‐Based Water Cycle Algorithm Application for Medical Image Enhancement

Visualizing medical images is difficult due to artifacts, poor local contrast, low soft tissue contrast, excessive noise levels, and a wide dynamic range. This has created a serious problem for physicians, resulting in unapproachable and inaccurate disease diagnoses. To circumvent this problem, this research proposes a medical image enhancement (MIE) approach based on the hybrid simulated annealing‐evaporation rate‐based water cycle algorithm (SA‐ERWCA). The ERWCA enhances distorted medical images by finding the best optimal solution for the transformation parameters according to the fitness function. The fitness function consists of three objective measurements, namely, entropy, number of edges, and sum of edge intensities. Due to its high potential for finding a globally optimal solution, SA is applied solely to determine the optimized initial population of the ERWCA, thereby enhancing its convergence characteristics. To simply put, the main objective of SA is to avoid premature convergence of the ERWCA. Thus, blending SA and ERWCA produces better‐quality medical images. The performance of the proposed algorithm was compared to histogram equalization (HE), low contrast stretching (LCS), contrast‐limited adaptive histogram equalization (CLAHE), particle swarm optimization (PSO), accelerated PSO (APSO), and the water cycle algorithm (WCA). Along with the objective function fitness, seven full reference (FR) image quality assessment (IQA) metrics were implemented to evaluate image quality and compare performance. The findings of the present study showed that the proposed strategy outperforms all of the compared methods in terms of objective function fitness and perceptual visual IQA metrics such as the Harr wavelet‐based perceptual similarity index (HaarPSI), visual signal‐to‐noise ratio (VSNR), and information content weighted structural similarity index measure (IW‐SSIM). The suggested technique also exhibited better stability and faster convergence time to the optimum solution. Furthermore, the proposed approach outperformed others by avoiding premature convergence to the best solution and providing excellent optimization results with acceptable computational efficiency.

Magnetic resonance and computer tomography image fusion using novel weight maps obtained by using median and guided filters

An attempt is made in this paper to diagnose brain-related diseases like sarcoma, fatal stroke disease, cerebral disease, and Alzheimer's disease by using saliency information from magnetic resonance and computed tomography source images. The saliency information for each source image is computed using guided and median filters. The obtained saliency maps are used for computing the weight maps of each source image by using image statistics. The obtained weights are used to fuse the approximate and detailed layers of the source images by using the weighted average fusion technique. The proposed algorithm is simulated in MATLAB for various benchmark data sets of brains taken from the brain atlas provided by Harvard Medical School, available at https://www.med.harvard.edu/aanlib. In order to test the efficacy of the novel method, comparative analysis is performed in terms of image quality assessment metrics like mean, mutual information, average gradient, standard deviation, spatial frequency, etc. From the analysis, this paper concludes that the proposed algorithm improved gradient information in the fused image by 35.7%, entropy by 5.7%, spatial frequency by 32.7%, edge strength by 14.5%, and minimized the information loss by 43.6%. Therefore, the novel method of weight map computation produces a detailed and noise-free image, which is helpful for better diagnosis in clinical applications.

Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Images

Computer graphics images (CGIs) are artificially generated by means of computer programs and are widely perceived under various scenarios, such as games, streaming media, etc. In practice, the quality of CGIs consistently suffers from poor rendering during production, inevitable compression artifacts during the transmission of multimedia applications, and low aesthetic quality resulting from poor composition and design. However, few works have been dedicated to dealing with the challenge of computer graphics image quality assessment (CGIQA). Most image quality assessment (IQA) metrics are developed for natural scene images (NSIs) and validated on databases consisting of NSIs with synthetic distortions, which are not suitable for in-the-wild CGIs. To bridge the gap between evaluating the quality of NSIs and CGIs, we construct a large-scale in-the-wild CGIQA database consisting of 6,000 CGIs (CGIQA-6k) and carry out the subjective experiment in a well-controlled laboratory environment to obtain the accurate perceptual ratings of the CGIs. Then, we propose an effective deep learning–based no-reference (NR) IQA model by utilizing both distortion and aesthetic quality representation. Experimental results show that the proposed method outperforms all other state-of-the-art NR IQA methods on the constructed CGIQA-6k database and other CGIQA-related databases. The database is released athttps://github.com/zzc-1998/CGIQA6K.

IMPACT OF UAV AND SENTINEL-2A IMAGERY FUSION ON VEGETATION INDICES PERFORMANCE

Abstract. Image fusion techniques can improve the quality of remote sensing images by combining high spatial resolution images with low spectral resolution images. This enhancement of the images may impact the performance of various vegetation indices (VI’s). This study investigates the impact of image fusion on the quality of vegetation indices by fusing UAV (Unmanned Aerial Vehicle) bands with Sentinel-2A images using Principal Component Analysis (PCA) and Brovery Transform (BT) fusion techniques.The fused images were used to calculate the Normalized Difference Vegetation Index, Normalized Difference Red Edge, Green Red Vegetation Index, and Normalized Difference Water Index. To assess the performance of the fused images, several image quality assessment metrics were used, including Root Mean Square Error (RMSE), Entropy, etc...The results showed that image fusion techniques can improve the quality of images which is important to assess crop health. The PCA image fusion technique showed higher quality than the BT technique. The PCA fused images had lower RMSE, ERGAS, and Entropy Difference and higher UIQI, CC, and SSIM values than the original images. Moreover, the fused images produced higher VIs values than the Sentinel-2A images.Finally, scatter plots were created to compare the correlation between the VIs calculated from the original and fused images. The results showed a strong correlation between the VIs calculated from the Sentinel-2A and fused images, indicating that the fused images can accurately estimate vegetation health parameters. Overall, this study demonstrates the potential of image fusion techniques to improve the quality of VI’s for monitoring vegetation health.

A feasibility analysis of image approximation with image quality assessments

AbstractWith technological developments, the resolution of the display systems is increasing, causing image data to increase. Several technologies are aiding in optimizing these data during transmission and storage while maintaining quality. Still, the internal data transmission for accessing the memory sub‐system during reading and writing cycles consumes significant power while requiring large memory for storage, which is inefficient, especially for portable displays. Hence, this work investigates the feasibility of reducing data through approximation for images in different formats and resolutions for human and computer vision based applications. Different image quality assessment metrics are utilized for performance evaluations with optimum image quality assessment selection according to application requirements. In addition, the feasibility analysis is conducted for three different applications as examples of computer vision. The experimental analysis highlights theoretical and industrial importance by showing that image data can be reduced from 25.0% to 62.5%, while satisfying requirements for HVS‐based and computer vision‐based applications.

A Motion Deblurring Network for Enhancing UAV Image Quality in Bridge Inspection

Unmanned aerial vehicles (UAVs) have been increasingly utilized for facility safety inspections due to their superior safety, cost effectiveness, and inspection accuracy compared to traditional manpower-based methods. High-resolution images captured by UAVs directly contribute to identifying and quantifying structural defects on facility exteriors, making image quality a critical factor in achieving accurate results. However, motion blur induced by external factors such as vibration, low light conditions, and wind during UAV operation significantly degrades image quality, leading to inaccurate defect detection and quantification. To address this issue, this research proposes a deblurring network using a Generative Adversarial Network (GAN) to eliminate the motion blur effect in UAV images. The GAN-based motion deblur network represents an image inpainting method that leverages generative models to correct blurry artifacts, thereby generating clear images. Unlike previous studies, this proposed approach incorporates deblur and blur learning modules to realistically generate blur images required for training the generative models. The UAV images processed using the motion deblur network are evaluated using a quality assessment method based on local blur map and other well-known image quality assessment (IQA) metrics. Moreover, in the experiment of crack detection utilizing the object detection system, improved detection results are observed when using enhanced images. Overall, this research contributes to improving the quality and accuracy of facility safety inspections conducted with UAV-based inspections by effectively addressing the challenges associated with motion blur effects in UAV-captured images.

Attentive Deep Image Quality Assessment for Omnidirectional Stitching

Omnidirectional images or videos are commonly generated via the stitching of multiple images or videos, and the quality of omnidirectional stitching strongly influences the quality of experience (QoE) of the generated scenes. Although there were many studies research the omnidirectional image quality assessment (IQA), the evaluation of the omnidirectional stitching quality has not been sufficiently explored. In this paper, we focus on the IQA for the omnidirectional stitching of dual fisheye images. We first establish an omnidirectional stitching image quality assessment (OSIQA) database, which includes 300 distorted images and 300 corresponding reference images generated from 12 raw scenes. The database contains a variety of distortion types caused by omnidirectional stitching, including color distortion, geometric distortion, blur distortion, and ghosting distortion, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">etc.</i> A subjective quality assessment study is conducted on the database and human opinion scores are collected for the distorted omnidirectional images. We then devise a deep learning based objective IQA metric termed Attentive Multi-channel IQA Net. In particular, we extend hyper-ResNet by developing a subnetwork for spatial attention and propose a spatial regularization item. Experimental results show that our proposed FR and NR models achieve the best performance compared with the state-of-the-art FR and NR IQA metrics on the OSIQA database. The OSIQA database as well as the proposed Attentive Multi-channel IQA Net will be released to facilitate future research.

PVBLiF: A Pseudo Video-Based Blind Quality Assessment Metric for Light Field Image

Going beyond traditional 2D imaging is not only an emerging trend of imaging technology, but also the key to a more immersive user experience. Light Field Image (LFI) is a typical high-dimensional imaging format, and the quality evaluation of which is very challenging but necessary. In this paper, we propose a novel Pseudo Video-based Blind quality assessment metric for Light Field image (PVBLiF). In contrast to most previous Light Field Image Quality Assessment (LF-IQA) metrics, in which different types of 2D representations derived from LFI are used for quality assessment indirectly, our metric exploits a more intuitive 3D representation, named Pseudo Video Block Sequence (PVBS), to evaluate the perceptual quality of LFI. For this purpose, we first divide the LFI into a massive number of non-overlapping PVBSs, which simultaneously contain spatial and angular information of LFI. Then, we propose a novel network (named PVBSNet) based on Convolutional Neural Networks (CNNs) to extract the spatio-angular features of PVBS and further evaluate the PVBS quality. The proposed PVBSNet consists of four stages: multi-information division, intra-feature extraction, cross-feature fusion, and quality regression. Finally, a Saliency- and Variance-guided Pooling (SVPooling) method is presented to integrate all the PVBS quality into the overall quality of LFI. The proposed PVBLiF metric has been extensively evaluated on three widely-used LFI datasets: Win5-LID, NBU-LF1.0, and SHU. Experimental results demonstrate that our proposed PVBLiF metric outperforms state-of-the-art metrics and is capable of highly approximating the performance of human observers. The source code of PVBLiF is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ZhengyuZhang96/PVBLiF</uri> .

Analysis of Image Quality Assessment Methods and Metrics: A Comprehensive Review

The evaluation of metrics, the effects of machine learning, and the use of preprocessing techniques for image enhancement constitute the essence of image quality. Through a thorough investigation, this study seeks to identify the core components of image quality. First, various measures for gauging image quality are compared to see how well they work. These metrics offer precise evaluations of aspects of a picture such as sharpness, contrast, and color integrity. The effect of machine learning techniques on the quality of images is then examined. The study investigates the use of machine learning approaches to improve image quality by training models on substantial datasets and tailoring them for certain tasks. In addition, the function of preprocessing methods in picture enhancement is investigated. Before further processing or analysis, the image quality is improved using various techniques, including noise reduction, image denoising, and local entropy filtering. The findings of this work offer important new perspectives on the assessment of image quality measurements, the potential of machine learning for image enhancement, and the significance of preprocessing methods in producing higher image quality.

Raindrop-Removal Image Translation Using Target-Mask Network with Attention Module

Image processing plays a crucial role in improving the performance of models in various fields such as autonomous driving, surveillance cameras, and multimedia. However, capturing ideal images under favorable lighting conditions is not always feasible, particularly in challenging weather conditions such as rain, fog, or snow, which can impede object recognition. This study aims to address this issue by focusing on generating clean images by restoring raindrop-deteriorated images. Our proposed model comprises a raindrop-mask network and a raindrop-removal network. The raindrop-mask network is based on U-Net architecture, which learns the location, shape, and brightness of raindrops. The rain-removal network is a generative adversarial network based on U-Net and comprises two attention modules: the raindrop-mask module and the residual convolution block module. These modules are employed to locate raindrop areas and restore the affected regions. Multiple loss functions are utilized to enhance model performance. The image-quality assessment metrics of proposed method, such as SSIM, PSNR, CEIQ, NIQE, FID, and LPIPS scores, are 0.832, 26.165, 3.351, 2.224, 20.837, and 0.059, respectively. Comparative evaluations against state-of-the-art models demonstrate the superiority of our proposed model based on qualitative and quantitative results.

TPP: Deep learning based threshold post-processing multi-focus image fusion method

How to accurately determine focused and defocused domains is a difficult problem in the field of multi-focused image fusion. To solve this problem, we propose a new deep learning method called threshold post processing(TPP). The TPP network model contains three architecturally different channels, and three different weighted maps can be obtained for the same input. The output of the model is fused and thresholded to obtain a final decision map, which is then used to complete the image fusion task. The decision map represents the focus condition of the source image at a pixel level. This allows TPP to retain focused regions of the source image more accurately than other existing deep learning methods. The multi-channel network model combined with the thresholding process optimizes the final decision map. It avoids the drawback of other methods that cause inaccurate discrimination of the focused area due to a single network channel and therefore cannot be corrected. Subsequent experimental results show that the images generated by TPP maintain a high level under the various image quality assessment metrics measures.

22‐4: Invited Paper: Deep Learning‐based Image Deblurring for Display Vision Inspection

Image quality acts as a major factor in determining its performance in a vision inspection task. The moire pattern caused by frequency aliasing severely degrades the visual quality in display devices, where such high‐quality images are required. To remove these quality undermining patterns, the images are acquired by intentional defocusing. Then, to restore the details lost during the image acquisition, image deblurring is used. The existing deblurring methods fail to output satisfactory results for low contrast Mura images. To solve this problem, we present a novel approach using a generalized Gaussian kernel for real‐world vision inspection tasks. We evaluated the performance and experimented under different settings to validate the robustness of the proposed method. The performance for the proposed method has improved in no‐reference image quality assessment metrics.

Bridge the gap between full-reference and no-reference: A totally full-reference induced blind image quality assessment via deep neural networks

Blind image quality assessment (BIQA) is of fundamental importance in low-level computer vision community. Increasing interest has been drawn in exploiting deep neural networks for BIQA. Despite of the notable success achieved, there is a broad consensus that training deep convolutional neural networks (DCNN) heavily relies on massive annotated data. Unfortunately, BIQA is typically a small sample problem, resulting the generalization ability of BIQA severely restricted. In order to improve the accuracy and generalization ability of BIQA metrics, this work proposed a totally opinion-unaware BIQA in which no subjective annotations are involved in the training stage. Multiple full-reference image quality assessment (FR-IQA) metrics are employed to label the distorted image as a substitution of subjective quality annotation. A deep neural network (DNN) is trained to blindly predict the multiple FR-IQA score in absence of corresponding pristine image. In the end, a self-supervised FR-IQA score aggregator implemented by adversarial auto-encoder pools the predictions of multiple FR-IQA scores into the final quality predicting score. Even though none of subjective scores are involved in the training stage, experimental results indicate that our proposed full reference induced BIQA framework is as competitive as state-of-the-art BIQA metrics.