Image Quality Assessment Model Research Articles

Increasing the Robustness of Image Quality Assessment Models Through Adversarial Training

The adversarial robustness of image quality assessment (IQA) models to adversarial attacks is emerging as a critical issue. Adversarial training has been widely used to improve the robustness of neural networks to adversarial attacks, but little in-depth research has examined adversarial training as a way to improve IQA model robustness. This study introduces an enhanced adversarial training approach tailored to IQA models; it adjusts the perceptual quality scores of adversarial images during training to enhance the correlation between an IQA model’s quality and the subjective quality scores. We also propose a new method for comparing IQA model robustness by measuring the Integral Robustness Score; this method evaluates the IQA model resistance to a set of adversarial perturbations with different magnitudes. We used our adversarial training approach to increase the robustness of five IQA models. Additionally, we tested the robustness of adversarially trained IQA models to 16 adversarial attacks and conducted an empirical probabilistic estimation of this feature.

Automated deep learning approach for identifying etiologies of hypertrophic phenocopies by echocardiography

Abstract Background Left ventricular hypertrophy (LVH) is an indicator of poor prognosis across various cardiovascular diseases and is developed by various etiologies. Transthoracic echocardiography (TTE) is the most available cardiac imaging tool to detect LVH. However, it is still challenging to differentiate three non-ischemic cardiomyopathies which frequently presenting LVH: Fabry cardiomyopathy (FC), hypertrophic cardiomyopathy (HCM), and cardiac amyloidosis (CA) from morphology or cardiac function evaluated by TTE. Purpose We aimed to develop stepwise deep learning models to differentiate FC, HCM, and CA using radiomics extracted from TTE. Methods We conducted a retrospective cohort study between 2011 and 2022 in Taiwan. Patients with left ventricular (LV) mass index≥95g/m² in women or ≥115g/m² in men, or the maximal LV wall thickness≥13mm on echocardiography were enrolled. The TTE automatic processing models included four steps: image quality assessment, view classification, auto-segmentation of interventricular septum (IVS), and dynamic radiomics analysis. One echocardiographic study was scanned per patient, and each study contains multiple cardiac cycles. One sample was extracted from each cycle and all samples were evaluated by image quality assessment and view classification model that based on VGG16. Samples with apical four (A4C) and apical five chamber (A5C) views were retained for following steps. Manual contouring of the septal region served as the ground truth for auto-segmentation based on U-Net. The last step was the extraction of dynamic radiomic features from IVS and the dynamic radiomic features were analyzed using an LSTM model to differentiate three hypertrophic cardiomyopathies. Results There were 103 patients with FC, 96 patients with HCM, and 49 patients with CA in this study. Patients with CA were older and had lower LV ejection fraction (EF) than those with FC or HCM (age: 65.8±9.7 vs. 59.3±11.1 vs. 58.6±14.2 years, P=0.002; LVEF: 56.6±10.8 vs. 61.9±8.8 vs. 64.5±8.3%, P<0.001). In addition, patient with FC had larger LV mass index than those with HCM or CA. (Fig. 2A) The details of the performances of four stepwise models were provided in Figure 2 (B-F). The image quality assessment and view classification models achieved accuracies of 91.7% and 95.2%, respectively. The auto-segmentation had the median Dice score as 0.88 for contouring IVS. The results indicated that deep learning algorithms had great ability to recognize different views with proper image quality, and could contour the IVS adequately. The accuracy of dynamic radiomics-based LSTM model to differentiate three hypertrophic phenocopies was 74.5%. Conclusion We developed an automatic process with four stepwise models for analyzing the radiomics of IVS on TTE to differentiate FC, HCM and CA with the modest performance by LSTM model. The developed automatic models may provide potential implications to identify various etiologies LVH in the future.Figure 1-Workflow of this studyFigure 2-Performances of stepwise models

Multi‐domain pseudo‐reference quality evaluation for infrared and visible image fusion

AbstractInfrared and visible image fusion involves merging the advantages of infrared and visible images to generate a composite image that encompasses thermal radiation as well as intricate texture details. Infrared and visible image fusion has garnered increasing attention, with numerous fusion methods proposed. However, how to fairly perceive the performance of fused image remains a contentious topic. This paper is dedicated to solving this problem from two perspectives (e.g., subjective and objective aspects). Firstly, an infrared and visible fusion image quality assessment dataset was constructed, including 60 pairs of infrared and visible images captured in various scenes, along with 540 fusion images with different types and degrees of distortions. Additionally, a subjective evaluation dataset of 16,200 subjective scores by 30 participants was further provided for the fused image. Secondly, to overcome the challenging assessment for infrared and visible fusion images without a real reference image, an interesting multi‐domain pseudo‐reference image quality assessment model (MPIQAM) is proposed, by comprehensively considering the thermal radiation information distortion, texture information distortion, and overall naturalness of the fused image. The proposed MPIQAM was compared with 18 mainstream objective metrics, and the experimental findings showcased a commendable level of competitiveness.

Panoramic image quality assessment based on multi-viewport adaptive fusion

Existing panoramic image quality assessment models are relatively independent when extracting local features of each viewport, resulting in high computational complexity and difficulty in characterizing the correlation between viewports using an end-to-end fusion model. To address the above problems, a quality assessment method based on feature sharing and multi-viewport adaptive fusion is proposed. Using a shared backbone network, the viewport segmentation and calculation tasks that are independent of each other in the existing method are converted to the feature domain, so that the local features of the entire image can be extracted with only one feedforward calculation. On this basis, a feature domain viewport segmentation method based on spherical uniform sampling is introduced to ensure that the pixel density of the observation space and the representation space is consistent, and semantic information is used to guide the adaptive fusion of local quality features of each viewport. The linear correlation coefficient and rank correlation coefficient on the CVIQ and OIQA datasets are both above 0.96, which is the best compared with the existing mainstream evaluation methods. Compared with the traditional evaluation method SSIM, its average linear correlation coefficient and average rank correlation coefficient on the two datasets are improved by 9.52%and respectively 8.69%; compared with the latest evaluation method MPFIQA, its average linear correlation coefficient and average rank correlation coefficient are improved by 1.71%and respectively 1.44%.

Black-box adversarial attacks against image quality assessment models

The problem of No-Reference Image Quality Assessment (NR-IQA) is to predict the perceptual quality of an image in line with its subjective evaluation. However, the vulnerabilities of NR-IQA models to the adversarial attacks have not been thoroughly studied for model refinement. This paper aims to investigate the potential loopholes of NR-IQA models via black-box adversarial attacks. Specifically, we first formulate the attack problem as maximizing the deviation between the estimated quality scores of original and perturbed images, while restricting the perturbed image distortions for visual quality preservation. Under such formulation, we then design a Bi-directional loss function to mislead the estimated quality scores of adversarial examples towards an opposite direction with maximum deviation. On this basis, we finally develop an efficient and effective black-box attack method for NR-IQA models based on a random search paradigm. Comprehensive experiments on three benchmark datasets show that all evaluated NR-IQA models are significantly vulnerable to the proposed attack method. After being attacked, the average change rates in terms of two well-known IQA performance metrics achieved by victim models reach 97% and 101%, respectively. In addition, our attack method also outperforms a newly introduced black-box attack approach on IQA models. We also observe that the generated perturbations are not transferable, which points out a new research direction in NR-IQA community. The source code is available at https://github.com/GZHU-DVL/AttackIQA.

A multi-task deep learning approach for real-time view classification and quality assessment of echocardiographic images

High-quality standard views in two-dimensional echocardiography are essential for accurate cardiovascular disease diagnosis and treatment decisions. However, the quality of echocardiographic images is highly dependent on the practitioner’s experience. Ensuring timely quality control of echocardiographic images in the clinical setting remains a significant challenge. In this study, we aimed to propose new quality assessment criteria and develop a multi-task deep learning model for real-time multi-view classification and image quality assessment (six standard views and “others”). A total of 170,311 echocardiographic images collected between 2015 and 2022 were utilized to develop and evaluate the model. On the test set, the model achieved an overall classification accuracy of 97.8% (95%CI 97.7–98.0) and a mean absolute error of 6.54 (95%CI 6.43–6.66). A single-frame inference time of 2.8 ms was achieved, meeting real-time requirements. We also analyzed pre-stored images from three distinct groups of echocardiographers (junior, senior, and expert) to evaluate the clinical feasibility of the model. Our multi-task model can provide objective, reproducible, and clinically significant view quality assessment results for echocardiographic images, potentially optimizing the clinical image acquisition process and improving AI-assisted diagnosis accuracy.

High Prevalence of Artifacts in Optical Coherence Tomography With Adequate Signal Strength.

This study aims to investigate the prevalence of artifacts in optical coherence tomography (OCT) images with acceptable signal strength and evaluate the performance of supervised deep learning models in improving OCT image quality assessment. We conducted a retrospective study on 4555 OCT images from 546 patients, with each image having an acceptable signal strength (≥6). A comprehensive analysis of prevalent OCT artifacts was performed, and five pretrained convolutional neural network models were trained and tested to infer images based on quality. Our results showed a high prevalence of artifacts in OCT images with acceptable signal strength. Approximately 21% of images were labeled as nonacceptable quality. The EfficientNetV2 model demonstrated superior performance in classifying OCT image quality, achieving an area under the receiver operating characteristic curve of 0.950 ± 0.007 and an area under the precision recall curve of 0.985 ± 0.002. The findings highlight the limitations of relying solely on signal strength for OCT image quality assessment and the potential of deep learning models in accurately classifying image quality. Application of the deep learning-based OCT image quality assessment models may improve the OCT image data quality for both clinical applications and research.

Rethinking the Effectiveness of Objective Evaluation Metrics in Multi-Focus Image Fusion: A Statistic-Based Approach.

As an effective technique to extend the depth-of-field (DOF) of optical lenses, multi-focus image fusion has recently become an active topic in image processing community. However, a major problem remaining unsolved in this field is the lack of universal criteria in selecting objective evaluation metrics. Consequently, the metrics utilized in different studies often vary significantly, leading to high difficulties in achieving unbiased evaluation. To address this problem, this paper proposes a statistic-based approach for verifying the effectiveness of objective metrics in multi-focus image fusion. The core idea is to adopt statistical correlation measures to evaluate the performance consistency between a certain fusion metric and some popular full-reference image quality assessment models. In addition, a convolutional neural network (CNN)-based fusion metric is presented to measure the similarity between the source images and the fused image based on the semantic features at multiple abstraction levels. A comparative study is conducted to evaluate 20 existing fusion metrics using the proposed statistic-based approach on a large-scale, realistic and with-ground-truth multi-focus image fusion dataset recently released. Experimental results demonstrate the feasibility of the proposed approach in evaluating the effectiveness of objective metrics and the advantage of our CNN-based metric.

RAD-IQMRI: A benchmark for MRI image quality assessment

Magnetic resonance imaging (MRI) is susceptible to visual artifacts that can degrade the perceptual image quality, potentially leading to inaccurate or inefficient diagnoses in clinical practice. It is critical to evaluate the perceptual image quality and build this technique into clinical solutions. In a previous study, an MRI database was created for image quality assessment (IQA), where various types of MRI artifacts with different degrees of degradation were simulated. Application specialists assessed the image quality; however, radiologists’ perception of MRI image quality remains unknown. To make IQA clinically relevant, in this paper we conduct a new subjective experiment where 13 radiologists rated the quality of images contained in the MRI database. Based on this subjective IQA benchmark named RAD-IQMRI, we evaluate the performance of state-of-the-art objective IQA models, providing insights into their application for MRI image quality assessment in clinical settings.

Stacked deformable convolution network with weighted non-local attention and branch residual connection for image quality assessment

Convolutional neural networks are data-driven Image Quality Assessment (IQA) models based on the convolutional operation in a rectangular window. Deformable convolutions with learnable receptive fields can efficiently extract structural features of irregular objects in an image. By superimposing deformable convolutions, a plug-and-play module is designed to obtain information for irregular geometric shapes. To selectively fuse shallow and deep features, we propose a weighted non-local attention (WNLA) module with the input and output of self-attention in a weighted manner. This paper proposes a dual branch residual full-reference IQA network that combines weighted non-local attention and stacked deformable convolution. The proposed network was trained on PIPAL dataset and tested on LIVE and TID2013. The cross-dataset evaluation shows that the network has a competitive generalization ability. Ablation experiments indicate that the proposed modules can effectively improve the performance of the network. Comparative experiments reveal that our network is superior to existing excellent networks. The codes for training, test and visualization are available at: https://github.com/Pengchang-haha/SDCN.git.

Chest CT-IQA: A multi-task model for chest CT image quality assessment and classification

In recent years, especially during the COVID-19 pandemic, a large number of Computerized Tomography (CT) images are produced every day for the purpose of inspecting lung diseases. However, the diagnosis accuracy depends on the quality of CT imaging and low quality images may greatly affect clinical diagnosis, resulting in misdiagnosis. It is difficult to effectively rate the quality of massive CT images. To solve the above problems, we first constructed a dataset of 800 CT volumes for chest CT image quality assessment. Then we propose a multi-task model for chest CT image quality assessment and classification. This model can automatically classify CT image sequences of different visual inspection windows, and automatically estimate CT image quality score, to match the visual score from clinicians. The experimental results show that the window classification accuracy and the dose exposure classification accuracy of our model can reach 0.8375 and 0.8813 respectively. The Pearson Linear Correlation Coefficient (PLCC) and Root Mean Square Error (RMSE) between the model prediction results and the two radiologist’s annotation average result reached 0.3288 and 1.9264. It shows that our model has a potential to mimic quality evaluation of experts.

Hardware Implementation of a Deep Learning-based Model for Image Quality Assessment

Hardware Implementation of a Deep Learning-based Model for Image Quality Assessment

Shift-insensitive perceptual feature of quadratic sum of gradient magnitude and LoG signals for image quality assessment and image classification

Most existing full-reference (FR) Image quality assessment (IQA) models work in the premise of that the two images should be well registered. Shifting an image would lead to an inaccurate evaluation of image quality, because small spatial shifts are far less noticeable than structural distortion for human observers. To this regard, we propose to study an IQA feature that is shift-insensitive to the basic primitive structure of images, i.e., image edge. According to previous studies, the image gradient magnitude (GM) and the Laplacian of Gaussian (LoG) operator that depict the edge profiles of natural images are highly efficient structural features in IQA tasks. In this paper, we find that the Quadratic sum of the normalized GM and the LoG signals (QGL) has excellent shift-insensitive property in representing image edges after theoretically solving the selection problem of a ratio parameter to balance the GM and LoG signals. Based on the proposed QGL feature, two FR-IQA models can be built directly by measuring the similarity map with mean and standard deviation pooling strategies, named mQGL and sQGL, respectively. Experimental results show that the proposed sQGL and mQGL work robustly on four benchmark IQA databases, and QGL-based models show great shift-insensitive property to spatial translation and image rotation while judging the image quality. In addition, we explore the feasibility of combining QGL feature with deep neural networks, and verify that it can help to promote image pattern recognition in texture classification tasks.

HDRC: a subjective quality assessment database for compressed high dynamic range image

Evaluating the quality of high dynamic range (HDR) images has emerged as a challenging and contemporary topic with the proliferation of HDR content. In this work, a new HDR compression (HDRC) database is proposed, aiming to provide a benchmark for the development of full-reference HDR image quality assessment (IQA) algorithms when facing the latest HDR compression distortions. In particular, the proposed HDRC database is the first HDR-IQA database to incorporate Versatile Video Coding (VVC) compression distortions, closely associated with practical application scenarios. Furthermore, the proposed HDRC database is currently the largest HDR-IQA database, including 80 reference images and 400 distorted images. Extensive experiments are conducted by studying the performance compared to existing HDR-IQA databases when evaluating three HDR-specific IQA models and nine IQA models prevalent for low dynamic range (LDR) content, revealing the challenges the proposed HDRC database brings. The results indicate that the existing IQA models demonstrate noticeable decreases in accuracy when assessing new compression distortions, underscoring the need for the development of novel HDR-IQA models. Consequently, the suggested HDRC database can serve as a potential database for HDR-IQA research, fostering a comprehensive exploration of the associated fields.

McmIQA: Multi-Module Collaborative Model for No-Reference Image Quality Assessment

No reference image quality assessment is a technique that uses computers to simulate the human visual system and automatically evaluate the perceived quality of images. In recent years, with the widespread success of deep learning in the field of computer vision, many end-to-end image quality assessment algorithms based on deep learning have emerged. However, unlike other computer vision tasks that focus on image content, an excellent image quality assessment model should simultaneously consider distortions in the image and comprehensively evaluate their relationships. Motivated by this, we propose a Multi-module Collaborative Model for Image Quality Assessment (McmIQA). The image quality assessment is divided into three subtasks: distortion perception, content recognition, and correlation mapping. And specific modules are constructed for each subtask: the distortion perception module, the content recognition module, and the correlation mapping module. Specifically, we apply two contrastive learning frameworks on two constructed datasets to train the distortion perception module and the content recognition module to extract two types of features from the image. Subsequently, using these extracted features as input, we employ a ranking loss to train the correlation mapping module to predict image quality on image quality assessment datasets. Extensive experiments conducted on seven relevant datasets demonstrated that the proposed method achieves state-of-the-art performance in both synthetic distortion and natural distortion image quality assessment tasks.

Robust deep learning for eye fundus images: Bridging real and synthetic data for enhancing generalization

Deep learning applications for assessing medical images are limited because the datasets are often small and imbalanced. The use of synthetic data has been proposed in the literature, but neither a robust comparison of the different methods nor generalizability has been reported. Our approach integrates a retinal image quality assessment model and StyleGAN2 architecture to enhance Age-related Macular Degeneration (AMD) detection capabilities and improve generalizability. This work compares ten different Generative Adversarial Network (GAN) architectures to generate synthetic eye-fundus images with and without AMD. We combined subsets of three public databases (iChallenge-AMD, ODIR-2019, and RIADD) to form a single training and test set. We employed the STARE dataset for external validation, ensuring a comprehensive assessment of the proposed approach. The results show that StyleGAN2 reached the lowest Fréchet Inception Distance (166.17), and clinicians could not accurately differentiate between real and synthetic images. ResNet-18 architecture obtained the best performance with 85% accuracy and outperformed the two human experts (80% and 75%) in detecting AMD fundus images. The accuracy rates were 82.8% for the test set and 81.3% for the STARE dataset, demonstrating the model’s generalizability. The proposed methodology for synthetic medical image generation has been validated for robustness and accuracy, with free access to its code for further research and development in this field.

Perceptual Quality Assessment of Omnidirectional Images: A Benchmark and Computational Model

Compared with traditional 2D images, omnidirectional images (also referred to as 360 ∘ images) have more complicated perceptual characteristics due to the particularities of imaging and display. How humans perceive omnidirectional images in an immersive environment and form the immersive quality of experience are important problems. Thus, it is crucial to measure the quality of omnidirectional images under different viewing conditions, which suffer from realistic distortions. In this article, we build a large-scale subjective assessment database for omnidirectional images and carry out a comprehensive psychophysical experiment to study the relationships between different factors (viewing conditions and viewing behaviors) and the perceptual quality of omnidirectional images. In addition, we collect both subjective ratings and head movement data. A thorough analysis of the collected subjective data is also provided, where we make several interesting findings. Moreover, with the proposed database, we propose a novel transformer-based omnidirectional image quality assessment model. To be consistent with the human viewing process, viewing conditions and behaviors are naturally incorporated into the proposed model. Specifically, the proposed model mainly consists of three parts: viewport sequence generation, multi-scale feature extraction, and perceptual quality prediction. Extensive experimental results conducted on the proposed database demonstrate the effectiveness of the proposed method over existing image quality assessment methods.

DRAC 2022: A public benchmark for diabetic retinopathy analysis on ultra-wide optical coherence tomography angiography images

We described a challenge named "DRAC - Diabetic Retinopathy Analysis Challenge" in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022). Within this challenge, we provided the DRAC datset, an ultra-wide optical coherence tomography angiography (UW-OCTA) dataset (1,103 images), addressing three primary clinical tasks: diabetic retinopathy (DR) lesion segmentation, image quality assessment, and DR grading. The scientific community responded positively to the challenge, with 11, 12, and 13 teams submitting different solutions for these three tasks, respectively. This paper presents a concise summary and analysis of the top-performing solutions and results across all challenge tasks. These solutions could provide practical guidance for developing accurate classification and segmentation models for image quality assessment and DR diagnosis using UW-OCTA images, potentially improving the diagnostic capabilities of healthcare professionals. The dataset has been released to support the development of computer-aided diagnostic systems for DR evaluation.

2AFC Prompting of Large Multimodal Models for Image Quality Assessment

2AFC Prompting of Large Multimodal Models for Image Quality Assessment

Image Quality Assessment: Measuring Perceptual Degradation via Distribution Measures in Deep Feature Spaces.

This study aims to develop advanced and training-free full-reference image quality assessment (FR-IQA) models based on deep neural networks. Specifically, we investigate measures that allow us to perceptually compare deep network features and reveal their underlying factors. We find that distribution measures enjoy advanced perceptual awareness and test the Wasserstein distance (WSD), Jensen-Shannon divergence (JSD), and symmetric Kullback-Leibler divergence (SKLD) measures when comparing deep features acquired from various pretrained deep networks, including the Visual Geometry Group (VGG) network, SqueezeNet, MobileNet, and EfficientNet. The proposed FR-IQA models exhibit superior alignment with subjective human evaluations across diverse image quality assessment (IQA) datasets without training, demonstrating the advanced perceptual relevance of distribution measures when comparing deep network features. Additionally, we explore the applicability of deep distribution measures in image super-resolution enhancement tasks, highlighting their potential for guiding perceptual enhancements. The code is available on website. (https://github.com/Buka-Xing/Deep-network-based-distribution-measures-for-full-reference-image-quality-assessment).