Image Quality Assessment Metrics Research Articles

MRI Super-Resolution Analysis via MRISR: Deep Learning for Low-Field Imaging

Abstract: This paper presents a novel MRI super-resolution analysis model, MRISR. Through the utilization of generative adversarial networks for the estimation of degradation kernels and the injection of noise, we have constructed a comprehensive dataset of high-quality paired high- and low-resolution MRI images. The MRISR model seamlessly integrates VMamba and Transformer technologies, demonstrating superior performance across various no-reference image quality assessment metrics compared with existing methodologies. It effectively reconstructs high-resolution MRI images while meticulously preserving intricate texture details, achieving a fourfold enhancement in resolution. This research endeavor represents a significant advancement in the field of MRI super-resolution analysis, contributing a cost-effective solution for rapid MRI technology that holds immense promise for widespread adoption in clinical diagnostic applications.

Perceptual metric for face image quality with pixel-level interpretability

This paper tackles the shortcomings of image evaluation metrics in evaluating facial image quality. Conventional metrics do neither accurately reflect the unique attributes of facial images nor correspond with human visual perception. To address these issues, we introduce a novel metric designed specifically for faces, utilizing a learning-based adversarial framework. This framework comprises a generator for simulating face restoration and a discriminator for quality evaluation. Drawing inspiration from facial neuroscience studies, our metric emphasizes the importance of primary facial features, acknowledging that minor changes in the eyes, nose, and mouth can significantly impact perception. Another key limitation of existing image evaluation metrics is their focus on numerical values at the image level, without providing insight into how different areas of the image contribute to the overall assessment. Our proposed metric offers interpretability regarding how each region of the image is evaluated. Comprehensive experimental results confirm that our face-specific metric surpasses traditional general image quality assessment metrics for facial images, including both full-reference and no-reference methods. The code and models are available at https://github.com/AIM-SKKU/IFQA.

Overview of High-Dynamic-Range Image Quality Assessment.

In recent years, the High-Dynamic-Range (HDR) image has gained widespread popularity across various domains, such as the security, multimedia, and biomedical fields, owing to its ability to deliver an authentic visual experience. However, the extensive dynamic range and rich detail in HDR images present challenges in assessing their quality. Therefore, current efforts involve constructing subjective databases and proposing objective quality assessment metrics to achieve an efficient HDR Image Quality Assessment (IQA). Recognizing the absence of a systematic overview of these approaches, this paper provides a comprehensive survey of both subjective and objective HDR IQA methods. Specifically, we review 7 subjective HDR IQA databases and 12 objective HDR IQA metrics. In addition, we conduct a statistical analysis of 9 IQA algorithms, incorporating 3 perceptual mapping functions. Our findings highlight two main areas for improvement. Firstly, the size and diversity of HDR IQA subjective databases should be significantly increased, encompassing a broader range of distortion types. Secondly, objective quality assessment algorithms need to identify more generalizable perceptual mapping approaches and feature extraction methods to enhance their robustness and applicability. Furthermore, this paper aims to serve as a valuable resource for researchers by discussing the limitations of current methodologies and potential research directions in the future.

Enhanced plasmonic scattering imaging via deep learning-based super-resolution reconstruction for exosome imaging.

Exosome analysis plays pivotal roles in various physiological and pathological processes. Plasmonic scattering microscopy (PSM) has proven to be an excellent label-free imaging platform for exosome detection. However, accurately detecting images scattered from exosomes remains a challenging task due to noise interference. Herein, we proposed an image processing strategy based on a new blind super-resolution deep learning neural network, named ESRGAN-SE, to improve the resolution of exosome PSI images. This model can obtain super-resolution reconstructed images without increasing experimental complexity. The trained model can directly generate high-resolution plasma scattering images from low-resolution images collected in experiments. The results of experiments involving the detection of light scattered by exosomes showed that the proposed super-resolution detection method has strong generalizability and robustness. Moreover, ESRGAN-SE achieved excellent results of 35.52036, 0.09081, and 8.13176 in terms of three reference-free image quality assessment metrics, respectively. These results show that the proposed network can effectively reduce image information loss, enhance mutual information between pixels, and decrease feature differentiation. And, the single-image SNR evaluation score of 3.93078 also showed that the distinction between the target and the background was significant. The suggested model lays the foundation for a potentially successful approach to imaging analysis. This approach has the potential to greatly improve the accuracy and efficiency of exosome analysis, leading to more accurate cancer diagnosis and potentially improving patient outcomes.

Full-reference calibration-free image quality assessment

Objective Image Quality Assessment (IQA) methods often lack of linearity of their quality estimates with respect to scores expressed by human subjects and therefore IQA metrics undergo a calibration process based on subjective quality examples. However, example-based training presents a challenge in terms of generalization hampering result comparison across different applications and operative conditions. In this paper, new Full Reference (FR) techniques, providing estimates linearly correlated with human scores without using calibration are introduced. We show that on natural images, application of estimation theory and psychophysical principles to images degraded by Gaussian blur leads to a so-called canonical IQA method, whose estimates are linearly correlated to both the subjective scores and the viewing distance. Then, we show that any mainstream IQA methods can be reconducted to the canonical method by converting its metric based on a unique specimen image. The proposed scheme is extended to wide classes of degraded images, e.g. noisy and compressed images. The resulting calibration-free FR IQA methods allows for comparability and interoperability across different imaging systems and on different viewing distances. A comparison of their statistical performance with respect to state-of-the-art calibration prone methods is finally provided, showing that the presented model is a valid alternative to the final 5-parameter calibration step of IQA methods, and the two parameters of the model have a clear operational meaning and are simply determined in practical applications. The enhanced performance are achieved across multiple viewing distance databases by independently realigning the blur values associated with each distance.

ConIQA: A deep learning method for perceptual image quality assessment with limited data

Effectively assessing the realism and naturalness of images in virtual (VR) and augmented (AR) reality applications requires Full Reference Image Quality Assessment (FR-IQA) metrics that closely align with human perception. Deep learning-based IQAs that are trained on human-labeled data have recently shown promise in generic computer vision tasks. However, their performance decreases in applications where perfect matches between the reference and the distorted images should not be expected, or whenever distortion patterns are restricted to specific domains. Tackling this issue necessitates training a task-specific neural network, yet generating human-labeled FR-IQAs is costly, and deep learning typically demands substantial labeled data. To address these challenges, we developed ConIQA, a deep learning-based IQA that leverages consistency training and a novel data augmentation method to learn from both labeled and unlabeled data. This makes ConIQA well-suited for contexts with scarce labeled data. To validate ConIQA, we considered the example application of Computer-Generated Holography (CGH) where specific artifacts such as ringing, speckle, and quantization errors routinely occur, yet are not explicitly accounted for by existing IQAs. We developed a new dataset, HQA1k, that comprises 1000 natural images each paired with an image rendered using various popular CGH algorithms, and quality-rated by thirteen human participants. Our results show that ConIQA achieves superior Pearson (0.98), Spearman (0.965), and Kendall’s tau (0.86) correlations over fifteen FR-IQA metrics by up to 5%, showcasing significant improvements in aligning with human perception on the HQA1k dataset.

Hyperspectral image restoration using noise gradient and dual priors under mixed noise conditions

AbstractImages obtained from hyperspectral sensors provide information about the target area that extends beyond the visible portions of the electromagnetic spectrum. However, due to sensor limitations and imperfections during the image acquisition and transmission phases, noise is introduced into the acquired image, which can have a negative impact on downstream analyses such as classification, target tracking, and spectral unmixing. Noise in hyperspectral images (HSI) is modelled as a combination from several sources, including Gaussian/impulse noise, stripes, and deadlines. An HSI restoration method for such a mixed noise model is proposed. First, a joint optimisation framework is proposed for recovering hyperspectral data corrupted by mixed Gaussian‐impulse noise by estimating both the clean data as well as the sparse/impulse noise levels. Second, a hyper‐Laplacian prior is used along both the spatial and spectral dimensions to express sparsity in clean image gradients. Third, to model the sparse nature of impulse noise, an ℓ1 − norm over the impulse noise gradient is used. Because the proposed methodology employs two distinct priors, the authors refer to it as the hyperspectral dual prior (HySpDualP) denoiser. To the best of authors' knowledge, this joint optimisation framework is the first attempt in this direction. To handle the non‐smooth and non‐convex nature of the general ℓp − norm‐based regularisation term, a generalised shrinkage/thresholding (GST) solver is employed. Finally, an efficient split‐Bregman approach is used to solve the resulting optimisation problem. Experimental results on synthetic data and real HSI datacube obtained from hyperspectral sensors demonstrate that the authors’ proposed model outperforms state‐of‐the‐art methods, both visually and in terms of various image quality assessment metrics.

Enhancing pavement crack segmentation via semantic diffusion synthesis model for strategic road assessment

Computer-aided deep learning has significantly advanced road crack segmentation. However, supervised models face challenges due to limited annotated images. There is also a lack of emphasis on deriving pavement condition indices from predicted masks. This article introduces a novel semantic diffusion synthesis model that creates synthetic crack images from segmentation masks. The model is optimized in terms of architectural complexity, noise schedules, and condition scaling. The optimal architecture outperforms state-of-the-art semantic synthesis models across multiple benchmark datasets, demonstrating superior image quality assessment metrics. The synthetic frames augment these datasets, resulting in segmentation models with significantly improved efficiency. This approach enhances results without extensive data collection or annotation, addressing a key challenge in engineering. Finally, a refined pavement condition index has been developed for automated end-to-end defect detection systems, promoting more effective maintenance planning.

Performance comparison of decision-based median filtering for brain MR images with high multiplicative noise

Digital images, particularly medical images, are prone to various noise attacks, such as additive, multiplicative, etc. Salt and pepper noise (SPN) is a type of multiplicative noise that degrades image quality and complicates identification, potentially leading to inaccurate diagnoses. Removing SPN is crucial yet challenging as noise removal can also eliminate important edge details. This study addresses the challenge of removing SPN without losing edge details. Existing SPN filtering algorithms, such as Average Filter (AF), Median Filter (MF), Switched MF (SMF), Adaptive MF (AMF), and Decision-based MF (DMF), perform poorly under high SPN attacks. We propose two effective SPN filters, CA_SPN1 and CA_SPN2, under the framework of Cellular Automata (CA) and MF. CA_SPN1 excels at high SPN levels, while CA_SPN2, a combination of CA_SPN1 and AMF, is effective across all SPN levels. Quality evaluations using Image Quality Assessment (IQA) metrics – Peak Signal to Noise Ratio (PSNR), Structural Similarity Index (SSIM), Figure of Merit (FOM), and Perceptual Similarity (PSIM) – demonstrate that the proposed algorithms restore images more effectively while preserving edge details compared to state-of-the-art algorithms. Furthermore, experimental results confirm the superior restoration quality of the proposed filters, particularly under high SPN conditions.

Self-organized underwater image enhancement

Underwater images captured in diverse underwater scenes exhibit varying types and degrees of degradation, including color deviations, low contrast, blurry details, etc. Single image enhancement methods tend to insufficiently address the diverse degradation issues, resulting in inappropriate results that do not align well with human visual perception or underwater color prior. To overcome these deficiencies, we develop a novel reinforcement learning framework that selects a sequence of image enhancement methods and configures their parameters in a self-organized manner for the purpose of underwater image enhancement. In contrast to end-to-end deep learning-based black-box mechanisms, the novel framework operates in a white-box fashion where the mechanisms for the method selection and parameter configuration are transparent. Furthermore, our framework incorporates the human visual perception and the underwater color prior into non-reference score increments for rewarding the underwater image enhancement. This breaks through the training limit imposed by volunteer-selected enhanced images as references. Comprehensive qualitative and quantitative experiments ultimately demonstrate that our framework outperforms nine state-of-the-art underwater image enhancement methods in terms of visual quality, and achieves better performance in five underwater image quality assessment metrics on three underwater image datasets. We release our code at https://gitee.com/wanghaoupc/Self_organized_UIE.

Occupancy Estimation from Blurred Video: A Multifaceted Approach with Privacy Consideration

Building occupancy information is significant for a variety of reasons, from allocation of resources in smart buildings to responding during emergency situations. As most people spend more than 90% of their time indoors, a comfortable indoor environment is crucial. To ensure comfort, traditional HVAC systems condition rooms assuming maximum occupancy, accounting for more than 50% of buildings’ energy budgets in the US. Occupancy level is a key factor in ensuring energy efficiency, as occupancy-controlled HVAC systems can reduce energy waste by conditioning rooms based on actual usage. Numerous studies have focused on developing occupancy estimation models leveraging existing sensors, with camera-based methods gaining popularity due to their high precision and widespread availability. However, the main concern with using cameras for occupancy estimation is the potential violation of occupants’ privacy. Unlike previous video-/image-based occupancy estimation methods, we addressed the issue of occupants’ privacy in this work by proposing and investigating both motion-based and motion-independent occupancy counting methods on intentionally blurred video frames. Our proposed approach included the development of a motion-based technique that inherently preserves privacy, as well as motion-independent techniques such as detection-based and density-estimation-based methods. To improve the accuracy of the motion-independent approaches, we utilized deblurring methods: an iterative statistical technique and a deep-learning-based method. Furthermore, we conducted an analysis of the privacy implications of our motion-independent occupancy counting system by comparing the original, blurred, and deblurred frames using different image quality assessment metrics. This analysis provided insights into the trade-off between occupancy estimation accuracy and the preservation of occupants’ visual privacy. The combination of iterative statistical deblurring and density estimation achieved a 16.29% counting error, outperforming our other proposed approaches while preserving occupants’ visual privacy to a certain extent. Our multifaceted approach aims to contribute to the field of occupancy estimation by proposing a solution that seeks to balance the trade-off between accuracy and privacy. While further research is needed to fully address this complex issue, our work provides insights and a step towards a more privacy-aware occupancy estimation system.

Underwater Image Enhancement Algorithm Based on Adversarial Training

Ocean observation is the first step in the development of the ocean, whose abundant resources and strategic significance are attracting increasing attention. Observation methods based on visual sensor networks have received great attention from researchers due to their visualization capability and high information capacity. However, below the sea surface, objective factors such as blurriness, turbulence, and underwater color casting can cause image distortion and affect the acquisition of images. In this paper, the enhancement of underwater images is tackled using an adversarial learning-based approach. First, pre-processing is applied to address the significant color casting in the dataset, thus enhancing feature learning for subsequent style transfer. Then, corresponding improvements are made to a generative adversarial network’s structure and loss functions to better restore the features of the network output. Finally, evaluations and comparisons are performed using underwater image quality assessment metrics and several public datasets. Through multidimensional experiments, the proposed algorithm is shown to exhibit excellent performance in both subjective and objective evaluation metrics compared to state-of-the-art algorithms, as well as in practical visual applications.

Analyzing the Quality of Distorted Images by the Normalized Mutual Information Measure

This research explores how different types of distorting algorithms impact the Full-Reference image quality assessment, particularly when subjective quality evaluations are incorporated. We draw upon the TID2013 database, which contains 3000 images distorted by 24 distinct algorithms, in conjunction with Mean Opinion Scores (MOS) for quality ratings. We compare the results of Normalized Mutual Information (NMI) for image quality score with W2, based on Weibull distribution, the common PSNR similarity measure and MOS. We advocate for integrating of NMI into the repertoire of image quality assessment metrics.

Selection of suitable fusional band combination from Sentinel-2A and UAV imagery for agricultural applications

ABSTRACT This study explores the fusion of Sentinel-2A (S2) and different bands of UAV imagery (Blue, Green, Red and Panchromatic (PAN)) to enhance spatial and spectral information for crop health monitoring. It assesses the quality of fused images, accuracy of the classified fused images and statistical analysis between the fused and S2 vegetation indices. Various fusion techniques are examined to select an optimal combination of bands. The quality of the fused images is evaluated using image quality assessment metrics and classification employs the Random Forest algorithm, assessing accuracy with metrics like F-score and Kappa Coefficient. Statistical analysis involves comparing vegetation indices from fused and S2 imagery. Notably, the fusion of UAV Green and Red bands with S2 imagery, using BT and PCA techniques, emerges as an effective combination for plant-level agricultural health monitoring. This research contributes to advancing precision agriculture techniques by leveraging multispectral imaging fusion for enhanced crop monitoring and management.

Image inpainting based on GAN-driven structure- and texture-aware learning with application to object removal

In this paper, a novel deep learning-based image inpainting framework consisting of restoring image structure and reconstructing image details from corrupted images is proposed. Most image inpainting methods in the literature aim at restoring image details, outlines, and colors, simultaneously, which may suffer from blurring, deformation, and unreasonable content recovery due to interference among various information. To solve these problems, a two-stage image inpainting deep neural network based on GAN (generative adversarial network) architecture is proposed. The proposed inpainting framework consists of two modules: (1) the first stage, called the structure-aware learning stage, aims at learning a GAN-based structure restoration network, focusing on recovering the low-frequency image component, including colors and outlines of the missing regions of the input corrupted image; and (2) the second stage, called the texture-aware learning stage, aims at learning a GAN-based detail refinement network, focusing on rebuilding the high-frequency image details and texture information. In particular, we also propose to remove details from the training images to better train the structure restoration network to avoid inadequate image structure recovery induced by richer image textures, where the detail reconstruction task is left to the second stage. This strategy achieves to balance the workload between the two stages and the image quality can be progressively enhanced through the two stages. Experimental results have shown that the proposed deep inpainting framework quantitatively and qualitatively achieves state-of-the-art performance on the well-known datasets, including the CelebA, Places2, and ImageNet datasets, compared with existing deep learning-based image inpainting approaches. More specifically, in terms of the two well-known image quality assessment metrics, PSNR (peak signal-to-noise ratio) and SSIM (structural similarity), the improvement percentage of the proposed method, compared with the baseline approach, respectively, ranges from 3.23 % to 11.12 %, and 1.95 % to 13.39 %. The improvements have been shown to stably and significantly outperform the compared state-of-the-art methods in most types of inpainting mask. We also show that the proposed method is applicable to image editing in object removal from a single image.

Enhanced Solar Coronal Imaging: A GAN Approach with Fused Attention and Perceptual Quality Enhancement

The activity of the solar corona has a significant impact on all aspects of human life. People typically use images obtained from astronomical telescopes to observe coronal activities, among which the Atmospheric Imaging Assembly (AIA) of the Solar Dynamics Observatory (SDO) is particularly widely used. However, due to resolution limitations, we have begun to study the application of generative adversarial network super-resolution techniques to enhance the image data quality for a clearer observation of the fine structures and dynamic processes in the solar atmosphere, which improves the prediction accuracy of solar activities. We aligned SDO/AIA images with images from the High-Resolution Coronal Imager (Hi-C) to create a dataset. This research proposes a new super-resolution method named SAFCSRGAN, which includes a spatial attention module that incorporates channel information, allowing the network model to better capture the corona’s features. A Charbonnier loss function was introduced to enhance the perceptual quality of the super-resolution images. Compared to the original method using ESRGAN, our method achieved an 11.9% increase in Peak Signal-to-Noise Ratio (PSNR) and a 4.8% increase in Structural Similarity (SSIM). Additionally, we introduced two perceptual image quality assessment metrics, the Natural Image Quality Evaluator (NIQE) and Learned Perceptual Image Patch Similarity (LPIPS), which improved perceptual quality by 10.8% and 1.3%, respectively. Finally, our experiments demonstrated that our improved model surpasses other models in restoring the details of coronal images.

No-Reference Hyperspectral Image Quality Assessment via Ranking Feature Learning

In hyperspectral image (HSI) reconstruction tasks, due to the lack of ground truth in real imaging processes, models are usually trained and validated on simulation datasets and then tested on real measurements captured by real HSI imaging systems. However, due to the gap between the simulation imaging process and the real imaging process, the best model validated on the simulation dataset may fail on real measurements. To obtain the best model for the real-world task, it is crucial to design a suitable no-reference HSI quality assessment metric to reflect the reconstruction performance of different models. In this paper, we propose a novel no-reference HSI quality assessment metric via ranking feature learning (R-NHSIQA), which calculates the Wasserstein distance between the distribution of the deep features of the reconstructed HSIs and the benchmark distribution. Additionally, by introducing the spectral self-attention mechanism, we propose a Spectral Transformer (S-Transformer) to extract the spatial-spectral representative deep features of HSIs. Furthermore, to extract quality-sensitive deep features, we use quality ranking as a pre-training task to enhance the representation capability of the S-Transformer. Finally, we introduce the Wasserstein distance to measure the distance between the distribution of the deep features and the benchmark distribution, improving the assessment capacity of our method, even with non-overlapping distributions. The experimental results demonstrate that the proposed metric yields consistent results with multiple full-reference image quality assessment (FR-IQA) metrics, validating the idea that the proposed metric can serve as a substitute for FR-IQA metrics in real-world tasks.

Noise reduction in low-dose positron emission tomography with adaptive parameter estimation in sinogram domain

Noise reduction in low-dose positron emission tomography (PET) is a well-researched topic aimed at reducing patient radiation doses and improving diagnosis. Software-based noise reduction mainly improves the contrast between regions by reducing the variation of the acquired image. However, it should be performed under appropriate parameters to reduce discrimination. We propose a method that derives optimal noise-reduction parameters using the multi-scale structural similarity index measure and visual information fidelity, which are metrics for image quality assessment. Simulation and experimental studies demonstrated the viability of the proposed algorithm. The contrast-to-noise ratio value of the denoised reconstruction slice, which was used as the optimal parameter, increased approximately three times compared to that of the low-dose slice while preserving the resolution. The results indicate that the proposed method successfully predicted the parameters according to the noise-reduction algorithm and PET system conditions in the sinogram domain. The proposed algorithm should help prevent misdiagnosis and provide standardized medical images for clinical application by performing appropriate noise reduction.

Blind cartoon image quality assessment based on local structure and chromatic statistics

With the help of computer-assisted systems, cartoon synthesis has become convenient and efficient by reusing existing cartoon materials. However, quality evaluation of the obtained cartoon image still relies on labor-intensive subjective judgment. This accordingly raises an urgent demand for effective quality evaluation methods to automatically select a cartoon image of high quality from a set of candidates with different parameter settings. In this paper, a new blind image quality assessment metric is developed for evaluating the perceptual quality of cartoon images by considering structure and chromatic distortions. The extracted gradient-based local structure features and multiscale chromatic statistical features are integrated into one representation for an overall perceptual quality prediction. Experimental results on two benchmark cartoon image datasets, i.e., NBU-CIQAD and HFUT-CID, indicate that the proposed metric outperforms both the state-of-the-art blind quality evaluation methods designed for natural images or synthetic images.

High-Quality Synthesized Face Sketch Using Generative Reference Prior

Face sketch synthesis (FSS) is considered as an image-to-image translation problem, where a face sketch is generated from an input face photo. FSS plays a vital role in video/image surveillance-based law enforcement. In this paper, motivated by the recent success of generative adversarial networks (GAN), we consider conditional GAN (cGAN) to approach the problem of face sketch synthesis. However, despite the powerful cGAN model’s ability to generate fine textures, low-quality inputs characterized by the facial sketches drawn by artists cannot offer realistic and faithful details and have unknown degradation due to the drawing process, while high-quality references are inaccessible or even unexistent. In this context, we propose an approach based on Generative Reference Prior (GRP) to improve the synthesized face sketch perception. Our proposed model, that we call cGAN-GRP, leverages diverse and rich priors encapsulated in a pre-trained face GAN for generating high-quality facial sketch synthesis. Extensive experiments on publicly available face databases using facial sketch recognition rate and image quality assessment metrics as criteria demonstrate the effectiveness of our proposed model compared to several state-of-the-art methods.