Image Quality Assessment Model Research Articles

Hardware Implementation of a Deep Learning-based Model for Image Quality Assessment

Hardware Implementation of a Deep Learning-based Model for Image Quality Assessment

Stacked deformable convolution network with weighted non-local attention and branch residual connection for image quality assessment

Convolutional neural networks are data-driven Image Quality Assessment (IQA) models based on the convolutional operation in a rectangular window. Deformable convolutions with learnable receptive fields can efficiently extract structural features of irregular objects in an image. By superimposing deformable convolutions, a plug-and-play module is designed to obtain information for irregular geometric shapes. To selectively fuse shallow and deep features, we propose a weighted non-local attention (WNLA) module with the input and output of self-attention in a weighted manner. This paper proposes a dual branch residual full-reference IQA network that combines weighted non-local attention and stacked deformable convolution. The proposed network was trained on PIPAL dataset and tested on LIVE and TID2013. The cross-dataset evaluation shows that the network has a competitive generalization ability. Ablation experiments indicate that the proposed modules can effectively improve the performance of the network. Comparative experiments reveal that our network is superior to existing excellent networks. The codes for training, test and visualization are available at: https://github.com/Pengchang-haha/SDCN.git.

Shift-insensitive perceptual feature of quadratic sum of gradient magnitude and LoG signals for image quality assessment and image classification

Most existing full-reference (FR) Image quality assessment (IQA) models work in the premise of that the two images should be well registered. Shifting an image would lead to an inaccurate evaluation of image quality, because small spatial shifts are far less noticeable than structural distortion for human observers. To this regard, we propose to study an IQA feature that is shift-insensitive to the basic primitive structure of images, i.e., image edge. According to previous studies, the image gradient magnitude (GM) and the Laplacian of Gaussian (LoG) operator that depict the edge profiles of natural images are highly efficient structural features in IQA tasks. In this paper, we find that the Quadratic sum of the normalized GM and the LoG signals (QGL) has excellent shift-insensitive property in representing image edges after theoretically solving the selection problem of a ratio parameter to balance the GM and LoG signals. Based on the proposed QGL feature, two FR-IQA models can be built directly by measuring the similarity map with mean and standard deviation pooling strategies, named mQGL and sQGL, respectively. Experimental results show that the proposed sQGL and mQGL work robustly on four benchmark IQA databases, and QGL-based models show great shift-insensitive property to spatial translation and image rotation while judging the image quality. In addition, we explore the feasibility of combining QGL feature with deep neural networks, and verify that it can help to promote image pattern recognition in texture classification tasks.

HDRC: a subjective quality assessment database for compressed high dynamic range image

AbstractEvaluating the quality of high dynamic range (HDR) images has emerged as a challenging and contemporary topic with the proliferation of HDR content. In this work, a new HDR compression (HDRC) database is proposed, aiming to provide a benchmark for the development of full-reference HDR image quality assessment (IQA) algorithms when facing the latest HDR compression distortions. In particular, the proposed HDRC database is the first HDR-IQA database to incorporate Versatile Video Coding (VVC) compression distortions, closely associated with practical application scenarios. Furthermore, the proposed HDRC database is currently the largest HDR-IQA database, including 80 reference images and 400 distorted images. Extensive experiments are conducted by studying the performance compared to existing HDR-IQA databases when evaluating three HDR-specific IQA models and nine IQA models prevalent for low dynamic range (LDR) content, revealing the challenges the proposed HDRC database brings. The results indicate that the existing IQA models demonstrate noticeable decreases in accuracy when assessing new compression distortions, underscoring the need for the development of novel HDR-IQA models. Consequently, the suggested HDRC database can serve as a potential database for HDR-IQA research, fostering a comprehensive exploration of the associated fields.

McmIQA: Multi-Module Collaborative Model for No-Reference Image Quality Assessment

No reference image quality assessment is a technique that uses computers to simulate the human visual system and automatically evaluate the perceived quality of images. In recent years, with the widespread success of deep learning in the field of computer vision, many end-to-end image quality assessment algorithms based on deep learning have emerged. However, unlike other computer vision tasks that focus on image content, an excellent image quality assessment model should simultaneously consider distortions in the image and comprehensively evaluate their relationships. Motivated by this, we propose a Multi-module Collaborative Model for Image Quality Assessment (McmIQA). The image quality assessment is divided into three subtasks: distortion perception, content recognition, and correlation mapping. And specific modules are constructed for each subtask: the distortion perception module, the content recognition module, and the correlation mapping module. Specifically, we apply two contrastive learning frameworks on two constructed datasets to train the distortion perception module and the content recognition module to extract two types of features from the image. Subsequently, using these extracted features as input, we employ a ranking loss to train the correlation mapping module to predict image quality on image quality assessment datasets. Extensive experiments conducted on seven relevant datasets demonstrated that the proposed method achieves state-of-the-art performance in both synthetic distortion and natural distortion image quality assessment tasks.

Robust deep learning for eye fundus images: Bridging real and synthetic data for enhancing generalization

Deep learning applications for assessing medical images are limited because the datasets are often small and imbalanced. The use of synthetic data has been proposed in the literature, but neither a robust comparison of the different methods nor generalizability has been reported. Our approach integrates a retinal image quality assessment model and StyleGAN2 architecture to enhance Age-related Macular Degeneration (AMD) detection capabilities and improve generalizability. This work compares ten different Generative Adversarial Network (GAN) architectures to generate synthetic eye-fundus images with and without AMD. We combined subsets of three public databases (iChallenge-AMD, ODIR-2019, and RIADD) to form a single training and test set. We employed the STARE dataset for external validation, ensuring a comprehensive assessment of the proposed approach. The results show that StyleGAN2 reached the lowest Fréchet Inception Distance (166.17), and clinicians could not accurately differentiate between real and synthetic images. ResNet-18 architecture obtained the best performance with 85% accuracy and outperformed the two human experts (80% and 75%) in detecting AMD fundus images. The accuracy rates were 82.8% for the test set and 81.3% for the STARE dataset, demonstrating the model’s generalizability. The proposed methodology for synthetic medical image generation has been validated for robustness and accuracy, with free access to its code for further research and development in this field.

Perceptual Quality Assessment of Omnidirectional Images: A Benchmark and Computational Model

Compared with traditional 2D images, omnidirectional images (also referred to as 360 ∘ images) have more complicated perceptual characteristics due to the particularities of imaging and display. How humans perceive omnidirectional images in an immersive environment and form the immersive quality of experience are important problems. Thus, it is crucial to measure the quality of omnidirectional images under different viewing conditions, which suffer from realistic distortions. In this article, we build a large-scale subjective assessment database for omnidirectional images and carry out a comprehensive psychophysical experiment to study the relationships between different factors (viewing conditions and viewing behaviors) and the perceptual quality of omnidirectional images. In addition, we collect both subjective ratings and head movement data. A thorough analysis of the collected subjective data is also provided, where we make several interesting findings. Moreover, with the proposed database, we propose a novel transformer-based omnidirectional image quality assessment model. To be consistent with the human viewing process, viewing conditions and behaviors are naturally incorporated into the proposed model. Specifically, the proposed model mainly consists of three parts: viewport sequence generation, multi-scale feature extraction, and perceptual quality prediction. Extensive experimental results conducted on the proposed database demonstrate the effectiveness of the proposed method over existing image quality assessment methods.

DRAC 2022: A public benchmark for diabetic retinopathy analysis on ultra-wide optical coherence tomography angiography images

We described a challenge named "DRAC - Diabetic Retinopathy Analysis Challenge" in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022). Within this challenge, we provided the DRAC datset, an ultra-wide optical coherence tomography angiography (UW-OCTA) dataset (1,103 images), addressing three primary clinical tasks: diabetic retinopathy (DR) lesion segmentation, image quality assessment, and DR grading. The scientific community responded positively to the challenge, with 11, 12, and 13 teams submitting different solutions for these three tasks, respectively. This paper presents a concise summary and analysis of the top-performing solutions and results across all challenge tasks. These solutions could provide practical guidance for developing accurate classification and segmentation models for image quality assessment and DR diagnosis using UW-OCTA images, potentially improving the diagnostic capabilities of healthcare professionals. The dataset has been released to support the development of computer-aided diagnostic systems for DR evaluation.

Image Quality Assessment: Measuring Perceptual Degradation via Distribution Measures in Deep Feature Spaces.

This study aims to develop advanced and training-free full-reference image quality assessment (FR-IQA) models based on deep neural networks. Specifically, we investigate measures that allow us to perceptually compare deep network features and reveal their underlying factors. We find that distribution measures enjoy advanced perceptual awareness and test the Wasserstein distance (WSD), Jensen-Shannon divergence (JSD), and symmetric Kullback-Leibler divergence (SKLD) measures when comparing deep features acquired from various pretrained deep networks, including the Visual Geometry Group (VGG) network, SqueezeNet, MobileNet, and EfficientNet. The proposed FR-IQA models exhibit superior alignment with subjective human evaluations across diverse image quality assessment (IQA) datasets without training, demonstrating the advanced perceptual relevance of distribution measures when comparing deep network features. Additionally, we explore the applicability of deep distribution measures in image super-resolution enhancement tasks, highlighting their potential for guiding perceptual enhancements. The code is available on website. (https://github.com/Buka-Xing/Deep-network-based-distribution-measures-for-full-reference-image-quality-assessment).

Rethinking the Effectiveness of Objective Evaluation Metrics in Multi-focus Image Fusion: A Statistic-based Approach.

As an effective technique to extend the depth-of-field (DOF) of optical lenses, multi-focus image fusion has recently become an active topic in image processing community. However, a major problem remaining unsolved in this field is the lack of universal criteria in selecting objective evaluation metrics. Consequently, the metrics utilized in different studies often vary significantly, leading to high difficulties in achieving unbiased evaluation. To address this problem, this paper proposes a statistic-based approach for verifying the effectiveness of objective metrics in multi-focus image fusion. The core idea is to adopt statistical correlation measures to evaluate the performance consistency between a certain fusion metric and some popular full-reference image quality assessment models. In addition, a convolutional neural network (CNN)-based fusion metric is presented to measure the similarity between the source images and the fused image based on the semantic features at multiple abstraction levels. A comparative study is conducted to evaluate 20 existing fusion metrics using the proposed statistic-based approach on a large-scale, realistic and with-ground-truth multi-focus image fusion dataset recently released. Experimental results demonstrate the feasibility of the proposed approach in evaluating the effectiveness of objective metrics and the advantage of our CNN-based metric. Resources will be released at https://github.com/yuliu316316.

A no-reference image quality assessment model based on neighborhood component analysis and Gaussian process

In this paper, a human visual system (HVS) based no-reference image quality assessment model (NR-IQA) is proposed. This is an objective NR-IQA model that uses perceptual features such as structure and orientation to compute the loss of naturalness in the image. An adaptive feature extraction process is modeled to capture the underlying significant features of the images along with the noise. Moreover, the optimal subset of features is derived using the neighborhood component analysis and further used as inputs to the Gaussian process for regression. The various experimentations are carried out on LIVE, TID2008, and TID2013 databases to test the effectiveness of the proposed approach. It is observed that the presented model exhibits a competitive performance in comparison with the existing IQA approaches. Moreover, the computation complexity and run-time show the effectuality of the proposed approach. Further, the experiments show that the predicted score matches with human perceptions. Thus, the proposed model is more accurate, less complex, independent of distortions, and well-suited for real-time applications.

Xception-based architecture with cross-sampled training for Image Quality Assessment on KonIQ-10k

Image quality assessment is a~crucial task in various fields such as digital photography, online content creation, and automated quality control, as it ensures an optimal visual experience and aids in maintaining consistent standards. In this paper, we propose an efficient method for training image quality assessment models on the KonIQ-10k dataset. Our novel approach utilizes a~dual-Xception architecture that analyzes both the image content and additional image parameters, outperforming traditional single convolutional models. We introduce cross-sampling methods with random draw sampling of instances from majority classes, effectively enhancing prediction quality in the Mean Opinion Score (MOS) ranges that are underrepresented in the database. This methodology allows us to achieve near state-of-the-art results with limited computing costs and resources. Most importantly, our predictions across the entire spectrum of MOS values maintain consistent quality. Because of using a~novel and highly effective method for image sampling, we achieved these results with much lower computational cost, making our approach the most effective way of MOS estimation on the KonIQ-10k database.

Subjective and Objective Quality Assessment for in-the-Wild Computer Graphics Images

Computer graphics images (CGIs) are artificially generated by means of computer programs and are widely perceived under various scenarios, such as games, streaming media, etc. In practice, the quality of CGIs consistently suffers from poor rendering during production, inevitable compression artifacts during the transmission of multimedia applications, and low aesthetic quality resulting from poor composition and design. However, few works have been dedicated to dealing with the challenge of computer graphics image quality assessment (CGIQA). Most image quality assessment (IQA) metrics are developed for natural scene images (NSIs) and validated on databases consisting of NSIs with synthetic distortions, which are not suitable for in-the-wild CGIs. To bridge the gap between evaluating the quality of NSIs and CGIs, we construct a large-scale in-the-wild CGIQA database consisting of 6,000 CGIs (CGIQA-6k) and carry out the subjective experiment in a well-controlled laboratory environment to obtain the accurate perceptual ratings of the CGIs. Then, we propose an effective deep learning–based no-reference (NR) IQA model by utilizing both distortion and aesthetic quality representation. Experimental results show that the proposed method outperforms all other state-of-the-art NR IQA methods on the constructed CGIQA-6k database and other CGIQA-related databases. The database is released athttps://github.com/zzc-1998/CGIQA6K.

Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Training

Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, we propose a novel BIQA model for in-the-wild images by addressing two critical problems in this field: how to learn better quality-aware feature representation, and how to solve the problem of insufficient training samples in terms of their content and distortion diversity. Considering that perceptual visual quality is affected by both low-level visual features (e.g. distortions) and high-level semantic information (e.g. content), we first propose a staircase structure to hierarchically integrate the features from intermediate layers into the final feature representation, which enables the model to make full use of visual information from low-level to high-level. Then an iterative mixed database training (IMDT) strategy is proposed to train the BIQA model on multiple databases simultaneously, so the model can benefit from the increase in both training samples and image content and distortion diversity and can learn a more general feature representation. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model shows an excellent performance in the cross-database evaluation experiments, which further demonstrates that the learned feature representation is robust to images with diverse distortions and content. The code is available at https://github.com/sunwei925/StairIQA.

Blind Image Quality Assessment for Pathological Microscopic Image under Screen and Immersion Scenarios.

The high-quality pathological microscopic images are essential for physicians or pathologists to make a correct diagnosis. Image quality assessment (IQA) can quantify the visual distortion degree of images and guide the imaging system to improve image quality, thus raising the quality of pathological microscopic images. Current IQA methods are not ideal for pathological microscopy images due to their specificity. In this paper, we present deep learning-based blind image quality assessment model with saliency block and patch block for pathological microscopic images. The saliency block and patch block can handle the local and global distortions, respectively. To better capture the area of interest of pathologists when viewing pathological images, the saliency block is fine-tuned by eye movement data of pathologists. The patch block can capture lots of global information strongly related to image quality via the interaction between different image patches from different positions. The performance of the developed model is validated by the home-made Pathological Microscopic Image Quality Database under Screen and Immersion Scenarios (PMIQD-SIS) and cross-validated by the five public datasets. The results of ablation experiments demonstrate the contribution of the added blocks. The dataset and the corresponding code are publicly available at: https://github.com/mikugyf/PMIQD-SIS.

Attention-Aware Patch-Based CNN for Blind 360-Degree Image Quality Assessment.

An attention-aware patch-based deep-learning model for a blind 360-degree image quality assessment (360-IQA) is introduced in this paper. It employs spatial attention mechanisms to focus on spatially significant features, in addition to short skip connections to align them. A long skip connection is adopted to allow features from the earliest layers to be used at the final level. Patches are properly sampled on the sphere to correspond to the viewports displayed to the user using head-mounted displays. The sampling incorporates the relevance of patches by considering (i) the exploration behavior and (ii) a latitude-based selection. An adaptive strategy is applied to improve the pooling of local patch qualities to global image quality. This includes an outlier score rejection step relying on the standard deviation of the obtained scores to consider the agreement, as well as a saliency to weigh them based on their visual significance. Experiments on available 360-IQA databases show that our model outperforms the state of the art in terms of accuracy and generalization ability. This is valid for general deep-learning-based models, multichannel models, and natural scene statistic-based models. Furthermore, when compared to multichannel models, the computational complexity is significantly reduced. Finally, an extensive ablation study gives insights into the efficacy of each component of the proposed model.

Blind Quality Assessment of Images Containing Objects of Interest.

To monitor objects of interest, such as wildlife and people, image-capturing devices are used to collect a large number of images with and without objects of interest. As we are recording valuable information about the behavior and activity of objects, the quality of images containing objects of interest should be better than that of images without objects of interest, even if the former exhibits more severe distortion than the latter. However, according to current methods, quality assessments produce the opposite results. In this study, we propose an end-to-end model, named DETR-IQA (detection transformer image quality assessment), which extends the capability to perform object detection and blind image quality assessment (IQA) simultaneously by adding IQA heads comprising simple multi-layer perceptrons at the top of the DETRs (detection transformers) decoder. Using IQA heads, DETR-IQA carried out blind IQAs based on the weighted fusion of the distortion degree of the region of objects of interest and the other regions of the image; the predicted quality score of images containing objects of interest was generally greater than that of images without objects of interest. Currently, the subjective quality score of all public datasets is in accordance with the distortion of images and does not consider objects of interest. We manually extracted the images in which the five predefined classes of objects were the main contents of the largest authentic distortion dataset, KonIQ-10k, which was used as the experimental dataset. The experimental results show that with slight degradation in object detection performance and simple IQA heads, the values of PLCC and SRCC were 0.785 and 0.727, respectively, and exceeded those of some deep learning-based IQA models that are specially designed for only performing IQA. With the negligible increase in the computation and complexity of object detection and without a decrease in inference speeds, DETR-IQA can perform object detection and IQA via multi-tasking and substantially reduce the workload.

Konx: cross-resolution image quality assessment

Scale-invariance is an open problem in many computer vision subfields. For example, object labels should remain constant across scales, yet model predictions diverge in many cases. This problem gets harder for tasks where the ground-truth labels change with the presentation scale. In image quality assessment (IQA), down-sampling attenuates impairments, e.g., blurs or compression artifacts, which can positively affect the impression evoked in subjective studies. To accurately predict perceptual image quality, cross-resolution IQA methods must therefore account for resolution-dependent discrepancies induced by model inadequacies as well as for the perceptual label shifts in the ground truth. We present the first study of its kind that disentangles and examines the two issues separately via KonX, a novel, carefully crafted cross-resolution IQA database. This paper contributes the following: 1. Through KonX, we provide empirical evidence of label shifts caused by changes in the presentation resolution. 2. We show that objective IQA methods have a scale bias, which reduces their predictive performance. 3. We propose a multi-scale and multi-column deep neural network architecture that improves performance over previous state-of-the-art IQA models for this task. We thus both raise and address a novel research problem in image quality assessment.

Unsupervised blind image quality assessment via joint spatial and transform features

A novel unsupervised blind image quality assessment (BIQA) method, which requires no mean opinion scores for model training is presented in this paper. The method employs joint spatial and transform features as quality degradation metrics, specifically, phase congruency, gradient magnitude (GM), and GM and Laplacian of Gaussian response and local normalized coefficient are extracted as spatial features, and Karhunen–Loéve transform coefficient and discrete cosine transform coefficient are modeled as transform features. Both spatial and transform features are well analyzed to remove the redundancy, and then fitted to the multivariate Gaussian model for no-reference image quality assessment. Extensive experiments conducted on seven IQA databases demonstrate the superiority of the proposed method over the state-of-the-art both supervised and unsupervised BIQA methods.

Phase congruency based on derivatives of circular symmetric Gaussian function: an efficient feature map for image quality assessment

Image quality assessment (IQA) has become a hot issue in the area of image processing, which aims to evaluate image quality automatically by a metric being consistent with subjective evaluation. The first stage of conventional IQA model design is the quality-aware feature selection. Taking advantages of early visual feature, Phase congruency (PC) operates in frequency domain to measure local structures such as edges, corners, lines, etc., by computing the local amplitudes and local energies in multiple scales. Conventional local PC features are calculated with log-Gabor-based filtrations in several orientations, and usually combined with other features for IQA model design. Generally, a directional filter is sensitive to the changes on specific direction, and insensitive to other directions. This leads to multi-directional calculation and much time consumption in practical applications. Recently, researchers suggested that spatially circular symmetric filters, such as gradient magnitude (GM) and Laplacian of Gaussian (LoG), are highly efficient quality features and hence have been widely used in various IQA model designs. With the odd-symmetric and even-symmetric properties of GM and LoG operators, the two features are a suitable pair for PC compositions and can be computed uniformly by a Gaussian function with one scale factor. In this regard, we propose to combine GM and LoG signals to construct a new PC model with non-directional property. With ability to catch different types of distortions, the proposed PC feature can be promoted to a full-reference IQA model simply with average pooling or standard deviation pooling, and shows state-of-the-art performance compared with existing methods. Furthermore, our proposed PC algorithm can take the place of conventional PC component in well-known FSIM metric, which achieves improved performance and spends less in computation cost. This study suggests that the proposed circular symmetric PC feature is a highly efficient quality feature and can be exclusively used in IQA model designs.