Image Quality Assessment Methods Research Articles

Is the development of objective image quality assessment methods keeping pace with technological developments?

Is the development of objective image quality assessment methods keeping pace with technological developments?

No reference image quality assessment using gabor convolutional neural networks

With the emergence of image capturing devices and increase in usage of internet, massive volume of network data was occupied with digital images. For efficient data transmission, the image may undergo several processing units in from the point it captures to the display/storage device. It may result in loss of the perceptual image quality. Therefore, it is necessary to estimate the image quality to measure the quality of experience. It was found that the convolutional neural networks serve as potential tool for effective feature extraction in many image processing applications. Particularly, with the first layer as Gabor filters, the robustness of the network can be reinforced with learnable Gabor parameters. This paper proposes Gabor Convolutional Neural Network method for No-Reference Image Quality Assessment. Their well-defined spatial structured filters are promising in extracting quality features from the local patches and maps them to perceptual quality scores. Our proposed architecture was tested over synthetic and authentic databases such as LIVE, TID2013, CSIQ, LIVE-MD, MDID2016, LIVE Wild and KoNiQ-10k. The proposed approach was also tested on the Waterloo 3D phase-II database, which contains high-resolution images of both the eyes individually with their respective DMOS scores. The proposed approach out performs over LIVE-MD and LIVE Wild and competes with existing algorithm over other databases.

Omnidirectional image quality assessment with local–global vision transformers

With the rising popularity of omnidirectional images (ODIs) in virtual reality applications, the need for specialized image quality assessment (IQA) methods becomes increasingly critical. Traditional IQA approaches, designed for rectilinear images, often fail to evaluate ODIs accurately due to their 360-degree scene representation. Addressing this, we introduce the Local–Global Transformer for 360-degree Image Quality Assessment (LGT360IQ). This novel framework features dual branches tailored to mimic top-down and bottom-up visual attention mechanisms, adapted for the spherical characteristics of ODIs. The local branch processes tangent viewports from salient regions within the equirectangular projection image, extracting detailed features for granular quality assessment. In parallel, the global branch utilizes a task-dependent token sampling strategy for holistic image feature processing and quality score prediction. This integrated approach combines local and global information, offering an effective IQA method for ODIs. Our extensive evaluation across three benchmark ODI datasets, CVIQ, OIQA, and ODI, demonstrates LGT360IQ superior performance and establishes its role in advancing the field of IQA for omnidirectional images.

A Systematic Survey of Deep Learning-Based Single-Image Super-Resolution

Single-image super-resolution (SISR) is an important task in image processing, which aims to enhance the resolution of imaging systems. Recently, SISR has made a huge leap and has achieved promising results with the help of deep learning (DL). In this survey, we give an overview of DL-based SISR methods and group them according to their design targets. Specifically, we first introduce the problem definition, research background, and the significance of SISR. Secondly, we introduce some related works, including benchmark datasets, upsampling methods, optimization objectives, and image quality assessment methods. Thirdly, we provide a detailed investigation of SISR and give some domain-specific applications of it. Fourthly, we present the reconstruction results of some classic SISR methods to intuitively know their performance. Finally, we discuss some issues that still exist in SISR and summarize some new trends and future directions. This is an exhaustive survey of SISR, which can help researchers better understand SISR and inspire more exciting research in this field. An investigation project for SISR is provided at https://github.com/CV-JunchengLi/SISR-Survey.

Semi-TSGAN: Semi-Supervised Learning for Highlight Removal Based on Teacher-Student Generative Adversarial Network.

Despite recent notable advancements in highlight image restoration techniques, the dearth of annotated data and the lightweight deployment of highlight removal networks pose significant impediments to further advancements in the field. In this paper, to the best of our knowledge, we first propose a semi-supervised learning paradigm for highlight removal, merging the fusion version of a teacher-student model and a generative adversarial network, featuring a lightweight network architecture. Initially, we establish a dependable repository to house optimal predictions as pseudo ground truth through empirical analyses guided by the most reliable No-Reference Image Quality Assessment (NR-IQA) method. This method serves to assess rigorously the quality of model predictions. Subsequently, addressing concerns regarding confirmation bias, we integrate contrastive regularization into the framework to curtail the risk of overfitting on inaccurate labels. Finally, we introduce a comprehensive feature aggregation module and an extensive attention mechanism within the generative network, considering a balance between network performance and computational efficiency. Our experimental evaluations encompass comprehensive assessments on both full-reference and non-reference highlight benchmarks. The results demonstrate conclusively the substantive quantitative and qualitative enhancements achieved by our proposed algorithm in comparison to state-of-the-art methodologies.

G2NPAN: GAN-guided nuance perceptual attention network for multimodal medical fusion image quality assessment.

Multimodal medical fusion images (MMFI) are formed by fusing medical images of two or more modalities with the aim of displaying as much valuable information as possible in a single image. However, due to the different strategies of various fusion algorithms, the quality of the generated fused images is uneven. Thus, an effective blind image quality assessment (BIQA) method is urgently required. The challenge of MMFI quality assessment is to enable the network to perceive the nuances between fused images of different qualities, and the key point for the success of BIQA is the availability of valid reference information. To this end, this work proposes a generative adversarial network (GAN) -guided nuance perceptual attention network (G2NPAN) to implement BIQA for MMFI. Specifically, we achieve the blind evaluation style via the design of a GAN and develop a Unique Feature Warehouse module to learn the effective features of fused images from the pixel level. The redesigned loss function guides the network to perceive the image quality. In the end, the class activation mapping supervised quality assessment network is employed to obtain the MMFI quality score. Extensive experiments and validation have been conducted in a database of medical fusion images, and the proposed method is superior to the state-of-the-art BIQA method.

Review of Image Quality Assessment Methods for Compressed Images.

The compression of images for efficient storage and transmission is crucial in handling large data volumes. Lossy image compression reduces storage needs but introduces perceptible distortions affected by content, compression levels, and display environments. Each compression method generates specific visual anomalies like blocking, blurring, or color shifts. Standardizing efficient lossy compression necessitates evaluating perceptual quality. Objective measurements offer speed and cost efficiency, while subjective assessments, despite their cost and time implications, remain the gold standard. This paper delves into essential research queries to achieve visually lossless images. The paper describes the influence of compression on image quality, appropriate objective image quality metrics (IQMs), and the effectiveness of subjective assessment methods. It also provides an overview of the existing literature, surveys, and subjective and objective image quality assessment (IQA) methods. Our aim is to offer insights, identify challenges in existing methodologies, and assist researchers in selecting the most effective assessment approach for their needs.

Hierarchical prior-guided quality assessment method for underwater images

Many underwater image enhancement (UIE) algorithms have been proposed to improve the quality of the underwater image. However, the lack of effective objective image quality assessment (IQA) methods has hindered the further development of UIE algorithms. Unlike traditional distorted images, underwater images usually contain natural distortions such as color cast and fog effects induced by light absorption and scattering. In addition, due to the complexity of underwater scenes, the enhanced results generated using existing UIE algorithms may have artificial distortions such as under-enhancement, over-enhancement, and detail loss. Thus, it is difficult for traditional IQA methods to comprehensively capture quality-related features in complicated underwater conditions. Based on this, this paper proposes a new hierarchical prior-guided quality assessment method for underwater images with the aid of deep learning technology, which first decomposes the underwater image into the base layer and detail layer. Then, the two layers are taken as input of feature extraction network for extracting the luminance information and texture information of the image, respectively. Meanwhile, two pseudo-reference images are used to guide model training so as to obtain luminance and texture features more accurately and efficiently. Finally, the two types of features are fused to obtain the quality score of the underwater image. Extensive experiments have been conducted on two publicly available underwater image datasets, and the experimental results show that the proposed method outperforms other underwater image quality assessment methods.

Enhancing Retinal Fundus Image Quality Assessment With Swin-Transformer-Based Learning Across Multiple Color-Spaces.

The assessment of retinal image (RI) quality holds significant importance in both clinical trials and large datasets, because suboptimal images can potentially conceal early signs of diseases, thereby resulting in inaccurate medical diagnoses. This study aims to develop an automatic method for Retinal Image Quality Assessment (RIQA) that incorporates visual explanations, aiming to comprehensively evaluate the quality of retinal fundus images (RIs). We developed an automatic RIQA system, named Swin-MCSFNet, utilizing 28,792 RIs from the EyeQ dataset, as well as 2000 images from the EyePACS dataset and an additional 1,000 images from the OIA-ODIR dataset. After preprocessing, including cropping black regions, data augmentation, and normalization, a Swin-MCSFNet classifier based on the Swin-Transformer for multiple color-space fusion was proposed to grade the quality of RIs. The generalizability of Swin-MCSFNet was validated across multiple data centers. Additionally, for enhanced interpretability, a Score-CAM-generated heatmap was applied to provide visual explanations. Experimental results reveal that the proposed Swin-MCSFNet achieves promising performance, yielding a micro-receiver operating characteristic (ROC) of 0.93 and ROC scores of 0.96, 0.81, and 0.96 for the "Good," "Usable," and "Reject" categories, respectively. These scores underscore the accuracy of RIQA based on Swin-MCSF in distinguishing among the three categories. Furthermore, heatmaps generated across different RIQA classification scores and various color spaces suggest that regions in the retinal images from multiple color spaces contribute significantly to the decision-making process of the Swin-MCSFNet classifier. Our study demonstrates that the proposed Swin-MCSFNet outperforms other methods in experiments conducted on multiple datasets, as evidenced by the superior performance metrics and insightful Score-CAM heatmaps. This study constructs a new retinal image quality evaluation system, which will contribute to the subsequent research of retinal images.

Joint Distortion Restoration and Quality Feature Learning for No-reference Image Quality Assessment

No-reference image quality assessment (NR-IQA) methods, inspired by the free energy principle, improve the accuracy of image quality prediction by simulating the human brain’s repair process for distorted images. However, existing methods use separate optimization schemes for distortion restoration and quality prediction, which undermines the accurate mapping of feature representations to quality scores. To address this issue, we propose a joint restoration and quality feature learning NR-IQA (RQFL-IQA) method to jointly tackle distortion image restoration and quality prediction within a unified framework. To accurately establish the quality reconstruction relationship between distorted and restored images, a hybrid loss function based on pixel-wise and structure-wise representations is used to improve the restoration capability of the image restoration network. The proposed RQFL-IQA exploits rich labels, including restored images and quality scores, to enable the model to learn more discriminative features and establish a more accurate mapping from feature representation to quality scores. In addition, to avoid the impact of poor restoration on quality prediction, we propose a module with a cleaning function to reweight the fusion of restored and primitive features to achieve more perceptual consistency in feature fusion. Experimental results on public IQA datasets show that the proposed RQFL-IQA is superior over existing methods.

An image quality assessment method based on edge extraction and singular value for blurriness

The automatic assessment of perceived image quality is crucial in the field of image processing. To achieve this idea, we propose an image quality assessment (IQA) method for blurriness. The features of gradient and singular value were extracted in this method instead of the single feature in the traditional IQA algorithms. According to the insufficient size of existing public image quality assessment datasets to support deep learning, machine learning was introduced to fuse the features of multiple domains, and a new no-reference (NR) IQA method for blurriness denoted Feature fusion IQA(Ffu-IQA) was proposed. The Ffu-IQA uses a probabilistic model to estimate the probability of each edge detection blur in the image, and then uses machine learning to aggregate the probability information to obtain the edge quality score. After that uses the singular value obtained by singular value decomposition of the image matrix to calculate the singular value score. Finally, machine learning pooling is used to obtain the true quality score. Ffu-IQA achieves PLCC scores of 0.9570 and 0.9616 on CSIQ and TID2013, respectively, and SROCC scores of 0.9380 and 0.9531, which are better than most traditional image quality assessment methods for blurriness.

Perceptual Quality Assessment of Omnidirectional Images: A Benchmark and Computational Model

Compared with traditional 2D images, omnidirectional images (also referred to as 360 ∘ images) have more complicated perceptual characteristics due to the particularities of imaging and display. How humans perceive omnidirectional images in an immersive environment and form the immersive quality of experience are important problems. Thus, it is crucial to measure the quality of omnidirectional images under different viewing conditions, which suffer from realistic distortions. In this article, we build a large-scale subjective assessment database for omnidirectional images and carry out a comprehensive psychophysical experiment to study the relationships between different factors (viewing conditions and viewing behaviors) and the perceptual quality of omnidirectional images. In addition, we collect both subjective ratings and head movement data. A thorough analysis of the collected subjective data is also provided, where we make several interesting findings. Moreover, with the proposed database, we propose a novel transformer-based omnidirectional image quality assessment model. To be consistent with the human viewing process, viewing conditions and behaviors are naturally incorporated into the proposed model. Specifically, the proposed model mainly consists of three parts: viewport sequence generation, multi-scale feature extraction, and perceptual quality prediction. Extensive experimental results conducted on the proposed database demonstrate the effectiveness of the proposed method over existing image quality assessment methods.

A deep learning-based automatic image quality assessment method for respiratory phase on computed tomography chest images.

Computed tomography (CT) chest scans have become commonly used in clinical diagnosis. Image quality assessment (IQA) for CT images plays an important role in CT examination. It is worth noting that IQA is still a manual and subjective process, and even experienced radiologists make mistakes due to human limitations (fatigue, perceptual biases, and cognitive biases). There are also kinds of biases because of poor consensus among radiologists. Excellent IQA methods can reliably give an objective evaluation result and also reduce the workload of radiologists. This study proposes a deep learning (DL)-based automatic IQA method, to assess whether the image quality of respiratory phase on CT chest images are optimal or not, so that the CT chest images can be used in the patient's physical condition assessment. This retrospective study analysed 212 patients' chest CT images, with 188 patients allocated to a training set (150 patients), validation set (18 patients), and a test set (20 patients). The remaining 24 patients were used for the observer study. Data augmentation methods were applied to address the problem of insufficient data. The DL-based IQA method combines image selection, tracheal carina segmentation, and bronchial beam detection. To automatically select the CT image containing the tracheal carina, an image selection model was employed. Afterward, the area-based approach and score-based approach were proposed and used to further optimize the tracheal carina segmentation and bronchial beam detection results, respectively. Finally, the score about the image quality of the patient's respiratory phase images given by the DL-based automatic IQA method was compared with the mean opinion score (MOS) given in the observer study, in which four blinded experienced radiologists took part. The DL-based automatic IQA method achieved good performance in assessing the image quality of the respiratory phase images. For the CT sequence of the same patient, the DL-based IQA method had an accuracy of 92% in the assessment score, while the radiologists had an accuracy of 88%. The Kappa value of the assessment score between the DL-based IQA method and radiologists was 0.75, with a sensitivity of 85%, specificity of 91%, positive predictive value (PPV) of 92%, negative predictive value (NPV) of 93%, and accuracy of 88%. This study develops and validates a DL-based automatic IQA method for the respiratory phase on CT chest images. The performance of this method surpassed that of the experienced radiologists on the independent test set used in this study. In clinical practice, it is possible to reduce the workload of radiologists and minimize errors caused by human limitations.

A Full-Reference Image Quality Assessment Method via Deep Meta-Learning and Conformer

A Full-Reference Image Quality Assessment Method via Deep Meta-Learning and Conformer

No‐reference image quality assessment via a dual‐branch residual network

AbstractThe No‐Reference Image Quality Assessment (NR‐IQA) method can predict quality scores of distorted images without the reference image. However, due to the variability in both color and structure of images, existing NR‐IQA methods struggle to accurately predict quality scores of distorted images. Therefore, an NR‐IQA method based on a Dual‐Branch Residual Network (DBRIQA) for evaluating the quality scores of color‐distorted images is proposed. First, guided filtering is applied to the hue images in the HSV space to extract distortion information from the image's color. Then, due to significant differences in distorted images, enhancements are made to the traditional residual blocks, forming a feature extraction module that captures multi‐scale features from the image. In order to capture the global relationships in the distorted image, a Global‐Level Attention Block (GLAB) is introduced, facilitating the interaction of information among the extracted features. Experiments were conducted across four publicly available IQA datasets, including LIVE, CSIQ, TID2008, and TID2013. The experimental results demonstrate that the proposed method exhibits strong performance and generalization capabilities in predicting image quality compared to peer methods.

Inverse synthetic aperture radar image quality assessment based on PSO-BP neural network

ABSTRACT Based on particle swarm optimization-back propagation (PSO-BP) neural network, a novel inverse synthetic aperture radar (ISAR) image quality assessment (IQA) method is proposed to address the challenge of selecting high-quality ISAR images from a large number of images. This method incorporates six assessment metrics: energy gradient (EG), effective area ratio of image (EARI), image contrast (IC), image entropy (IE), equivalent number of looks (ENL), and signal-to-noise ratio (SNR) to evaluate the quality of ISAR images. The PSO-BP neural network is employed to fit the relationship between these metrics and ISAR image quality. Experimental results show that the new method is effective in ISAR-IQA under different imaging conditions and various attitudes, and closely matches the subjective evaluation of experts.

Auxiliary Information Guided Self-attention for Image Quality Assessment

Image quality assessment (IQA) is an important problem in computer vision with many applications. We propose a transformer-based multi-task learning framework for the IQA task. Two subtasks: constructing an auxiliary information error map and completing image quality prediction, are jointly optimized using a shared feature extractor. We use visual transformers (ViT) as a feature extractor for feature extraction and guide ViT to focus on image quality-related features by building auxiliary information error map subtask. In particular, we propose a fusion network that includes a channel focus module. Unlike the fusion methods commonly used in previous IQA methods, we use the fusion network, including the channel attention module, to fuse the auxiliary information error map features with the image features, which facilitates the model to mine the image quality features for more accurate image quality assessment. And by jointly optimizing the two subtasks, ViT focuses more on extracting image quality features and building a more precise mapping from feature representation to quality score. With slight adjustments to the model, our approach can be used in both no-reference (NR) and full-reference (FR) IQA environments. We evaluate the proposed method in multiple IQA databases, showing better performance than state-of-the-art FR and NR IQA methods.

A Mixed Reality Design System for Interior Renovation: Inpainting with 360-Degree Live Streaming and Generative Adversarial Networks after Removal

In contemporary society, “Indoor Generation” is becoming increasingly prevalent, and spending long periods of time indoors affects well-being. Therefore, it is essential to research biophilic indoor environments and their impact on occupants. When it comes to existing building stocks, which hold significant social, economic, and environmental value, renovation should be considered before new construction. Providing swift feedback in the early stages of renovation can help stakeholders achieve consensus. Additionally, understanding proposed plans can greatly enhance the design of indoor environments. This paper presents a real-time system for architectural designers and stakeholders that integrates mixed reality (MR), diminished reality (DR), and generative adversarial networks (GANs). The system enables the generation of interior renovation drawings based on user preferences and designer styles via GANs. The system’s seamless integration of MR, DR, and GANs provides a unique and innovative approach to interior renovation design. MR and DR technologies then transform these 2D drawings into immersive experiences that help stakeholders evaluate and understand renovation proposals. In addition, we assess the quality of GAN-generated images using full-reference image quality assessment (FR-IQA) methods. The evaluation results indicate that most images demonstrate moderate quality. Almost all objects in the GAN-generated images can be identified by their names and purposes without any ambiguity or confusion. This demonstrates the system’s effectiveness in producing viable renovation visualizations. This research emphasizes the system’s role in enhancing feedback efficiency during renovation design, enabling stakeholders to fully evaluate and understand proposed renovations.

Task-Specific Normalization for Continual Learning of Blind Image Quality Models.

In this paper, we present a simple yet effective continual learning method for blind image quality assessment (BIQA) with improved quality prediction accuracy, plasticity-stability trade-off, and task-order/-length robustness. The key step in our approach is to freeze all convolution filters of a pre-trained deep neural network (DNN) for an explicit promise of stability, and learn task-specific normalization parameters for plasticity. We assign each new IQA dataset (i.e., task) a prediction head, and load the corresponding normalization parameters to produce a quality score. The final quality estimate is computed by a weighted summation of predictions from all heads with a lightweight K -means gating mechanism. Extensive experiments on six IQA datasets demonstrate the advantages of the proposed method in comparison to previous training techniques for BIQA.

Review of the Image Quality Assessment Methods for the Compressed Images

Review of the Image Quality Assessment Methods for the Compressed Images