Traditional Image Quality Assessment Research Articles

Omnidirectional image quality assessment with local–global vision transformers

With the rising popularity of omnidirectional images (ODIs) in virtual reality applications, the need for specialized image quality assessment (IQA) methods becomes increasingly critical. Traditional IQA approaches, designed for rectilinear images, often fail to evaluate ODIs accurately due to their 360-degree scene representation. Addressing this, we introduce the Local–Global Transformer for 360-degree Image Quality Assessment (LGT360IQ). This novel framework features dual branches tailored to mimic top-down and bottom-up visual attention mechanisms, adapted for the spherical characteristics of ODIs. The local branch processes tangent viewports from salient regions within the equirectangular projection image, extracting detailed features for granular quality assessment. In parallel, the global branch utilizes a task-dependent token sampling strategy for holistic image feature processing and quality score prediction. This integrated approach combines local and global information, offering an effective IQA method for ODIs. Our extensive evaluation across three benchmark ODI datasets, CVIQ, OIQA, and ODI, demonstrates LGT360IQ superior performance and establishes its role in advancing the field of IQA for omnidirectional images.

Hierarchical prior-guided quality assessment method for underwater images

Many underwater image enhancement (UIE) algorithms have been proposed to improve the quality of the underwater image. However, the lack of effective objective image quality assessment (IQA) methods has hindered the further development of UIE algorithms. Unlike traditional distorted images, underwater images usually contain natural distortions such as color cast and fog effects induced by light absorption and scattering. In addition, due to the complexity of underwater scenes, the enhanced results generated using existing UIE algorithms may have artificial distortions such as under-enhancement, over-enhancement, and detail loss. Thus, it is difficult for traditional IQA methods to comprehensively capture quality-related features in complicated underwater conditions. Based on this, this paper proposes a new hierarchical prior-guided quality assessment method for underwater images with the aid of deep learning technology, which first decomposes the underwater image into the base layer and detail layer. Then, the two layers are taken as input of feature extraction network for extracting the luminance information and texture information of the image, respectively. Meanwhile, two pseudo-reference images are used to guide model training so as to obtain luminance and texture features more accurately and efficiently. Finally, the two types of features are fused to obtain the quality score of the underwater image. Extensive experiments have been conducted on two publicly available underwater image datasets, and the experimental results show that the proposed method outperforms other underwater image quality assessment methods.

Automated Web-based Software for CT Quality Control Testing of Low-contrast Detectability using Model Observers.

The Channelized Hotelling observer (CHO) is well correlated with human observer performance in many CT detection/classification tasks but has not been widely adopted in routine CT quality control and performance evaluation, mainly because of the lack of an easily available, efficient, and validated software tool. We developed a highly automated solution - CT image quality evaluation and Protocol Optimization (CTPro), a web-based software platform that includes CHO and other traditional image quality assessment tools such as modulation transfer function and noise power spectrum. This tool can allow easy access to the CHO for both the research and clinical community and enable efficient, accurate image quality evaluation without the need of installing additional software. Its application was demonstrated by comparing the low-contrast detectability on a clinical photon-counting-detector (PCD)-CT with a traditional energy-integrating-detector (EID)-CT, which showed UHR-T3D had 6.2% higher d' than EID-CT with IR (p = 0.047) and 4.1% lower d' without IR (p = 0.122).

An image quality assessment method based on edge extraction and singular value for blurriness

The automatic assessment of perceived image quality is crucial in the field of image processing. To achieve this idea, we propose an image quality assessment (IQA) method for blurriness. The features of gradient and singular value were extracted in this method instead of the single feature in the traditional IQA algorithms. According to the insufficient size of existing public image quality assessment datasets to support deep learning, machine learning was introduced to fuse the features of multiple domains, and a new no-reference (NR) IQA method for blurriness denoted Feature fusion IQA(Ffu-IQA) was proposed. The Ffu-IQA uses a probabilistic model to estimate the probability of each edge detection blur in the image, and then uses machine learning to aggregate the probability information to obtain the edge quality score. After that uses the singular value obtained by singular value decomposition of the image matrix to calculate the singular value score. Finally, machine learning pooling is used to obtain the true quality score. Ffu-IQA achieves PLCC scores of 0.9570 and 0.9616 on CSIQ and TID2013, respectively, and SROCC scores of 0.9380 and 0.9531, which are better than most traditional image quality assessment methods for blurriness.

No-Reference Quality Assessment of Extended Target Adaptive Optics Images Using Deep Neural Network.

This paper proposes a supervised deep neural network model for accomplishing highly efficient image quality assessment (IQA) for adaptive optics (AO) images. The AO imaging systems based on ground-based telescopes suffer from residual atmospheric turbulence, tracking error, and photoelectric noise, which can lead to varying degrees of image degradation, making image processing challenging. Currently, assessing the quality and selecting frames of AO images depend on either traditional IQA methods or manual evaluation by experienced researchers, neither of which is entirely reliable. The proposed network is trained by leveraging the similarity between the point spread function (PSF) of the degraded image and the Airy spot as its supervised training instead of relying on the features of the degraded image itself as a quality label. This approach is reflective of the relationship between the degradation factors of the AO imaging process and the image quality and does not require the analysis of the image's specific feature or degradation model. The simulation test data show a Spearman's rank correlation coefficient (SRCC) of 0.97, and our method was also validated using actual acquired AO images. The experimental results indicate that our method is more accurate in evaluating AO image quality compared to traditional IQA methods.

Image Restoration Quality Assessment Based on Regional Differential Information Entropy.

With the development of image recovery models, especially those based on adversarial and perceptual losses, the detailed texture portions of images are being recovered more naturally. However, these restored images are similar but not identical in detail texture to their reference images. With traditional image quality assessment methods, results with better subjective perceived quality often score lower in objective scoring. Assessment methods suffer from subjective and objective inconsistencies. This paper proposes a regional differential information entropy (RDIE) method for image quality assessment to address this problem. This approach allows better assessment of similar but not identical textural details and achieves good agreement with perceived quality. Neural networks are used to reshape the process of calculating information entropy, improving the speed and efficiency of the operation. Experiments conducted with this study's image quality assessment dataset and the PIPAL dataset show that the proposed RDIE method yields a high degree of agreement with people's average opinion scores compared with other image quality assessment metrics, proving that RDIE can better quantify the perceived quality of images.

A large-scale image database for benchmarking mobile camera quality and NR-IQA algorithms

Camera-oriented quality assessment (CQA) differs from traditional image quality assessment (IQA) in that “distorted” images are straight out of real devices instead of various types and levels of artificial operations. However, despite its value for both customers and manufacturers, academic and industrial fields, there are few CQA databases created years ago. To reflect recent mobile imaging advancements, we present a new massive Phone Camera Benchmarking database (PCB2021) in this paper. In PCB2021, 40 modern phone units featuring photography are simultaneously compared in 182 scenes for a total of 7280 images, which can be classified into six categories (sub-datasets) based on different focal lengths and user cases: main camera (107), ultra-wide (20), 2×, 3×, 5× zoom lenses (26, 8, 10) and night-mode (11). The shooting process begins from 7:00 am to 11:00 pm and lasts more than a month. In the subjective study, to overcome the high-resolution-induced overall quality evaluation difficulty, five image attributes: exposure/contrast, color, sharpness, graininess, artifacts are assessed separately on each dataset. To reduce ranking complexity for large-scale cameras, a dynamic anchor ruler method is proposed to obtain quality orders efficiently. With the constructed PCB2021, we further evaluate 15 mainstream no-reference (NR) IQA algorithms. The finding is that for zoomed images, sharpness metrics can achieve Spearman correlation coefficients above 0.8, while for the subtle main camera and night-mode images, performances of all fifteen algorithms drop down quickly, i.e. 0.1-0.2 for the former and ∼0.5 for the latter. The entire database, expert rankings and algorithm performance reports will be freely available on request.

CSPP-IQA: a multi-scale spatial pyramid pooling-based approach for blind image quality assessment.

The traditional image quality assessment (IQA) methods are usually based on convolutional neural networks (CNNs). For these IQA methods using CNNs, limited by the feature size of the fully connected layer, the input image needs be tailored to a pre-defined size, which usually results in destroying the original structure and content of the input image and thus reduces the accuracy of the quality assessment. In this paper, a blind image quality assessment method (named CSPP-IQA), which is based on multi-scale spatial pyramid pooling, is proposed. CSPP-IQA allows inputting the original image when assessing the image quality without any image adjustment. Moreover, by facilitating the convolutional block attention module and image understanding module, CSPP-IQA achieved better accuracy, generalization and efficiency than traditional IQA methods. The result of experiments running on real-scene IQA datasets in this study verified the effectiveness and efficiency of CSPP-IQA.

Evaluating the Stability of Deep Image Quality Assessment with Respect to Image Scaling

Image quality assessment (IQA) is a fundamental metric for image processing tasks (e.g., compression). With full-reference IQAs, traditional IQAs, such as PSNR and SSIM, have been used. Recently, IQAs based on deep neural networks (deep IQAs), such as LPIPS and DISTS, have also been used. It is known that image scaling is inconsistent among deep IQAs, as some perform down-scaling as pre-processing, whereas others instead use the original image size. In this paper, we show that the image scale is an influential factor that affects deep IQA performance. We comprehensively evaluate four deep IQAs on the same five datasets, and the experimental results show that image scale significantly influences IQA performance. We found that the most appropriate image scale is often neither the default nor the original size, and the choice differs depending on the methods and datasets used. We visualized the stability and found that PieAPP is the most stable among the four deep IQAs.

Dual-quality map based no reference image quality assessment using deformable convolution

Traditional feature-based no-reference image quality assessment (NR-IQA) technologies rely heavily on the correlation between the handcrafted features and distortions, and often fail to predict perceptual quality because of the diversity of image distortion types. End-to-end deep learning-based metrics are driven mainly by human-labeled data, which are prohibitively labor-expensive to collect. An NR-IQA method that imitates the hierarchical and inferential perceptual behavior of the human vision system (HVS) is proposed to address the challenges of distortion diversity and lack of labeled data. In the proposed method, first, a dual-quality map is generated for use as an intermediate target, which contains multiscale features that are highly related to image quality. To capture human eye-sensitive distortion information in distorted images, a dual attention mechanism is utilized in the quality map generation network. Then, a regression network is designed for mapping the intermediate target to a quality score. The deformable convolution which has adaptive receptive field is introduced to extract multiple scale features from the dual-quality map. Experiments on eight benchmark databases show the superiority of the proposed method over other state-of-the-art NR-IQA metrics.

Automated Image Super Resolution with the Aid of Activation Function Optimized Deep CNN and Adaptive Wavelet Lifting Approach

Diverse image super-resolution (SR) techniques have been implemented to reconstruct the high-resolution (HR) images from input images through lower spatial resolutions. However, the evaluation of the perceptual quality of SR images remains an important and complex research problem. This paper proposes a new image SR model with the intention of attaining maximum Peak Signal-to-Noise Ratio (PSNR). The conversion of low-resolution (LR) images from the HR images is performed by bicubic interpolation-based downsampling and upsampling. Then, the four sub-bands of LR and HR images are generated by the novel Adaptive Wavelet Lifting approach, in which the filter modes are optimized using the proposed SA-CBO. From this technique, LR wavelet sub-bands (LRSB) for LR images and HR wavelet sub-bands (HRSB) for HR images are formed. With the help of the LRSB and HRSB images, the residual images are formed by the adoption of the optimized Activation function and optimized hidden neurons in a deep convolutional neural network (CNN). The improvement in both the adaptive wavelet lifting approach and deep CNN is made by the self-adaptive-colliding bodies optimization (SA-CBO). Finally, the inverse adaptive wavelet lifting approach is used to produce the final SR image. Experimental results on publicly available SR image quality databases confirm the effectiveness and generalization ability of the proposed method compared with the traditional image quality assessment algorithms.

GMANet: Gradient Mask Attention Network for Finding Clearest Human Fecal Microscopic Image in Autofocus Process

The intelligent recognition of formed elements in microscopic images is a research hotspot. Whether the microscopic image is clear or blurred is the key factor affecting the recognition accuracy. Microscopic images of human feces contain numerous items, such as undigested food, epithelium, bacteria and other formed elements, leading to a complex image composition. Consequently, traditional image quality assessment (IQA) methods cannot accurately assess the quality of fecal microscopic images or even identify the clearest image in the autofocus process. In response to this difficulty, we propose a blind IQA method based on a deep convolutional neural network (CNN), namely GMANet. The gradient information of the microscopic image is introduced into a low-level convolutional layer of the CNN as a mask attention mechanism to force high-level features to pay more attention to sharp regions. Experimental results show that the proposed network has good consistency with human visual properties and can accurately identify the clearest microscopic image in the autofocus process. Our proposed model, trained on fecal microscopic images, can be directly applied to the autofocus process of leucorrhea and blood samples without additional transfer learning. Our study is valuable for the autofocus task of microscopic images with complex compositions.

Real‐world super‐resolution of face‐images from surveillance cameras

Most existing face image Super-Resolution (SR) methods assume that the Low-Resolution (LR) images were artificially downsampled from High-Resolution (HR) images with bicubic interpolation. This operation changes the natural image characteristics and reduces noise. Hence, SR methods trained on such data most often fail to produce good results when applied to real LR images. To solve this problem, a novel framework for the generation of realistic LR/HR training pairs is proposed. The framework estimates realistic blur kernels, noise distributions, and JPEG compression artifacts to generate LR images with similar image characteristics as the ones in the source domain. This allows to train an SR model using high-quality face images as Ground-Truth (GT). For better perceptual quality, a Generative Adversarial Network (GAN) based SR model is used, where the commonly used VGG-loss [1] is exchanged with LPIPS-loss [2]. Experimental results on both real and artificially corrupted face images show that our method results in more detailed reconstructions with less noise compared to the existing State-of-the-Art (SoTA) methods. In addition, it is shown that the traditional non-reference Image Quality Assessment (IQA) methods fail to capture this improvement and demonstrate that the more recent NIMA metric [3] correlates better with human perception via Mean Opinion Rank (MOR).

No-reference light field image quality assessment based on depth, structural and angular information

Light field imaging can capture more abundant scene information, including angular and spatial information, compared with traditional imaging technologies. However, light field image (LFI) processing will inevitably introduce distortion which is different from traditional imaging process. Therefore, LFI quality assessment has become one of important research issues on LFI processing. In this paper, we propose a depth, structural and angular information based no-reference LFI quality assessment method. Firstly, the horizontal and vertical mean difference images (MDIs) are defined to integrate the difference images of LFI's sub-aperture images (SAIs) and reflect the horizontal and vertical depth and structural information of LFI. Considering the multi-channel characteristics of human visual system, an effective feature extraction scheme with Curvelet decomposition is designed for MDIs and SAIs of the distorted LFI. As one of the representations of LFI, epipolar plane images (EPIs) contain LFI's angular and depth information, which reflect the angular consistency of LFI more intuitively. Therefore, an algorithm, namely local maximum similarity index statistics, is designed to extract the directional features from horizontal and vertical EPIs. Finally, the quality of distorted LFI is predicted by pooling these extracted features. Experimental results on three LFI datasets show that the proposed method can achieve better performance, compared with the representative traditional image quality assessment methods as well as the state-of-the-art LFI quality assessment methods.

Perceptual Quality Assessment for Asymmetrically Distorted Stereoscopic Video by Temporal Binocular Rivalry

In this paper, we propose a two-stage weighting based perceptual quality assessment framework for asymmetrically distorted stereoscopic video (SV) sequences by temporal binocular rivalry. Firstly, a traditional 2D image quality assessment (IQA) method is employed to measure spatial distortion, and the temporal distortion is evaluated by the magnitude differences between motion vectors of distorted and reference video frames. Secondly, the structural strength (SS) computed by gradient map and the motion energy (ME) computed by frame difference map are used to estimate the intensity of visual stimulus in spatial and temporal domain respectively. Then, SS and ME are considered as the importance indexes to combine the quality scores of spatial and temporal distortion to estimate perceived distortion of single-view video sequences, which is denoted as the first-stage weighting. Finally, considering that the difference of intensity of visual stimulus between two eyes results in binocular rivalry, a novel temporal binocular rivalry inspired weighting method is designed to integrate the quality scores of left- and right-views for the final visual quality prediction of SV sequences, which is denoted as the second-stage weighting. Experimental results on Waterloo-IVC SV quality databases show that several specific examples of 2D-IQA methods within the proposed framework can obtain highly competitive performance over other existing ones.

Quality Evaluation for Image Retargeting With Instance Semantics

To meet the ever-increasing demand for devices with diversified displays, image retargeting has become a prevalent technique for adaptive image resizing. In practice, the retargeting operation inevitably causes impairments in the images; thus, image retargeting quality assessment (IRQA) is urgently needed and, can be used to guide algorithm optimization, selection and design. Unlike traditional image quality assessment, image retargeting introduces geometric distortions, which typically affect high-level image semantics. With this motivation, this paper presents a quality evaluation model for image retargeting based on INstance SEMantics (INSEM). Considering that the human visual system (HVS) perceives images highly dependent on apprehensible areas and that impairments in image retargeting mainly degrade the salient instances, an image instance is utilized as the basic semantic unit, and a top-down method is devised to extract instance-level semantic features for IRQA. In addition, taking into account the influence of semantic categories on the perception of retargeting quality, we further propose Semantic-based self-adaptive pooling (SSAP) to integrate instance-based semantic features. Finally, global features are incorporated to generate quality scores that are more consistent with people's perceptions. Extensive experiments and comparisons of three public databases, in terms of both intradatabase and cross-database settings, demonstrate the superiority of the proposed metric over state-of-the-art methods.

Subjective and objective quality assessment for image restoration: A critical survey

Image restoration is the process of restoring latent clear images from degraded images, and it has received substantial attention in various restoration scenarios recently. Contrary to the considerable progress of image restoration algorithms, the quality evaluation for image restoration falls behind, which may hinder the further development of advanced image restoration techniques. So far, only several objective quality metrics have been proposed to evaluate the quality of restored images for specific restoration scenarios, and little work has been dedicated to the applications of image restoration quality metrics. Besides, the performance of these metrics remains an open problem, and an obvious disadvantage of these metrics is that their robustness is quite weak. To bridge this gap, we present this survey paper. First, the difference between the traditional image quality assessment and quality assessment for image restoration was analyzed profoundly. Then, for different image restoration scenarios, this paper provides (i) a comprehensive description of the existing subjective quality databases, (ii) a thorough insight into existing objective quality metrics and (iii) two applications of image restoration quality metrics, namely, parameter selection and benchmarking image restoration algorithms. After that, the experimental results and a detailed performance analysis on (i) quality assessment of restored images, (ii) benchmarking image restoration algorithms and (iii) time complexity are given based on public restored image databases. Finally, the paper outlines the challenges and future trends of subjective and objective quality assessment for image restoration in various aspects.

Unified non-uniform scale adaptive sampling model for quality assessment of natural scene and screen content images

Among many multimedia scenarios, digital images are mainly composed of natural scene images (NSIs) and screen content images (SCIs). NSIs are always captured by cameras from the real world while SCIs are typically the mixture of pictures and text. Traditional full-reference image quality assessment (IQA) metrics are designed for natural scene images, and they cannot work well on SCIs, because they do not take account of characteristics of different visual content/regions between NSIs and SCIs. Inspired by early eye movement study that human visual system samples the world non-completely or non-uniformly, we observe that human brain perceive different image content at different scales. Given these discoveries, we propose a non-uniform scale adaptive sampling model (NUSM) which dedicates to preprocess the input images for IQA methods, enabling traditional NSI IQA models to predict the quality of SCIs effectively. The proposed sampling model applies different sample scales into different regions to generate pre-sampled image pairs, then send to existing IQAs to evaluate the quality of images. The experimental results demonstrate that our model has achieved a promising performance on both NSIs and SCIs.

Locally and multiply distorted image quality assessment via multi-stage CNNs

The majority of existing objective Image Quality Assessment (IQA) methods are designed specifically for singly and globally distorted images, which are incapable of dealing with locally and multiply distorted images effectively. On the one hand, artificially extracted features in traditional IQA methods are insufficient to represent quality variations in locally and multiply distorted images. On the other hand, the IQA methods suitable for both locally and multiply distorted images are scarce. In view of this, an IQA method based on multi-stage deep Convolutional Neural Networks (CNNs) is proposed for locally and multiply distorted images in this paper. The method adopts a three-stage strategy, which are distortion classification, quality prediction of single distortion and comprehensive assessment, respectively. Firstly, three datasets of locally, multiply and singly distorted images are designed and established. Secondly, a local and multiple distortion classifier, a distortion type classifier and prediction models of single distortions are obtained based on CNN models in their corresponding stages. Thirdly, the predicted results of single distortions are weighted by the output confidence probability of the classifiers, thus obtaining the final comprehensive quality. Experimental results verified the advantages of the proposed method in measuring the quality of locally and multiply distorted images.

Research on the parallelization of image quality analysis algorithm based on deep learning

Image quality assessment is an indispensable in computer vision applications, such as image classification, image parsing. With the development of Internet, image data acquisition becomes more conveniently. However, image distortion is inevitable due to imperfect image acquisition system, image transmission medium and image recording equipment. Traditional image quality assessment algorithms only focus on low-level visual features such as color or texture, which could not encode high-level features effectively. CNN-based methods have shown satisfactory results in image quality assessment. However, existing methods have problems such as incomplete feature extraction, partial image block distortion, and inability to determine scores. So in this paper, we propose a novel framework for image quality assessment based on deep learning. We incorporate both low-level visual features and high-level semantic features to better describe images. And image quality is analyzed in a parallel processing mode. Experiments are conducted on LIVE and TID2008 datasets demonstrate the proposed model can predict the quality of the distorted image well, and both SROCC and PLCC can reach 0.92 or higher.