Image Quality Assessment Model Research Articles

Image quality assessment based on the perceived structural similarity index of an image.

Image quality assessment (IQA) has a very important role and wide applications in image acquisition, storage, transmission and processing. In designing IQA models, human visual system (HVS) characteristics introduced play an important role in improving their performances. In this paper, combining image distortion characteristics with HVS characteristics, based on the structure similarity index (SSIM) model, a novel IQA model based on the perceived structure similarity index (PSIM) of image is proposed. In the method, first, a perception model for HVS perceiving real images is proposed, combining the contrast sensitivity, frequency sensitivity, luminance nonlinearity and masking characteristics of HVS; then, in order to simulate HVS perceiving real image, the real images are processed with the proposed perception model, to eliminate their visual redundancy, thus, the perceived images are obtained; finally, based on the idea and modeling method of SSIM, combining with the features of perceived image, a novel IQA model, namely PSIM, is proposed. Further, in order to illustrate the performance of PSIM, 5335 distorted images with 41 distortion types in four image databases (TID2013, CSIQ, LIVE and CID) are used to simulate from three aspects: overall IQA of each database, IQA for each distortion type of images, and IQA for special distortion types of images. Further, according to the comprehensive benefit of precision, generalization performance and complexity, their IQA results are compared with those of 12 existing IQA models. The experimental results show that the accuracy (PLCC) of PSIM is 9.91% higher than that of SSIM in four databases, on average; and its performance is better than that of 12 existing IQA models. Synthesizing experimental results and theoretical analysis, it is showed that the proposed PSIM model is an effective and excellent IQA model.

Blind Image Quality Assessment via Cross-View Consistency

Image quality assessment (IQA) is very important for both end-users and service-providers since a high-quality image can significantly improve the user's quality of experience (QoE). Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, a BIQA model is proposed that consists of a desirable self-supervised feature learning approach to mitigate the data shortage problem and learn comprehensive feature representations, and a self-attention-based feature fusion module to introduce self-attention mechanism. We develop the image quality assessment model under the framework of contrastive learning with multi views. Since human visual system perceives signals through multiple channels, the most important visual information should exist among all views of the channels. So we design the cross-view consistent information mining (CVC-IM) module to extract compact mutual information between different views. Color information and pseudo-reference image (PRI) of different distortion types are employed to formulate rich feature embeddings and preserve the quality-aware fidelity of learned representations. We employ the Transformer as the self-attention-based architecture to integrate feature embeddings. Extensive experiments show that our model achieves remarkable image quality assessment results on in-the-wild IQA datasets.

LIQA: Lifelong Blind Image Quality Assessment

The image distortions are complex and dynamically changing in the real-world scenario, due to the fast development of the image processing system. The blind image quality assessment (BIQA) models may encounter the challenge of processing images with distortion types never seen before deployment. However, existing BIQA models generally cannot evolve with unseen distortion types adaptively, which greatly limits the deployment and application of BIQA models in real-world scenarios. To address this problem, we propose a novel Lifelong blind Image Quality Assessment (LIQA) approach, targeting to achieve the lifelong learning of BIQA. Without accessing to previous training data, our proposed LIQA can not only learn new knowledge, but also mitigate the catastrophic forgetting of learned knowledge. Specifically, we adopt the Split-and-Merge distillation strategy to train a single-head network that makes task-agnostic predictions. In the split stage, we first employ a distortion-specific generator to generate pseudo features of each previously seen distortion. Then, we utilize an auxiliary multi-head regression network to keep the response of each distortion. In the merge stage, we replay the pseudo features and use the pseudo labels generated by the auxiliary multi-head network to distill the knowledge of the multiple heads, which can build the final regression single head. Extensive experiments demonstrate that LIQA can perform well in handling both inner-dataset distortion shift and cross-dataset distortion shift. More importantly, our model can achieve stable performance even if the task sequences are long.

Deep Blind Image Quality Assessment Powered by Online Hard Example Mining

Recently, blind image quality assessment (BIQA) models based on deep neural networks (DNNs) have achieved impressive performance on existing datasets. However, due to the intrinsic imbalance property of the training set, not all distortions or images are handled equally well. Online hard example mining (OHEM) is a promising way to alleviate this issue. Inspired by the recent finding that network pruning disproportionately hampers the model's memorization of a tractable subset, atypical, low-quality, long-tailed samples, that are hard-to-memorize during training and easily “forgotten” during pruning, we propose an effective “plug-and-play” OHEM pipeline, especially for generalizable deep BIQA. Specifically, we train two parallel weight-sharing branches simultaneously, where one is full model and other is a “self-competitor” generated from the full model online by network pruning. Then, we leverage the prediction disagreement between the full model and its pruned variant (i.e., the self-competitor) to expose easily “forgettable” samples, which are therefore regarded as the hard ones. We then enforce the prediction consistency between the full model and its pruned variant to implicitly put more focus on these hard samples, which benefits the full model to recover forgettable information introduced by pruning. Extensive experiments across multiple datasets and BIQA models demonstrate that the proposed OHEM can improve the model performance and generalizability as measured by correlation numbers and group maximum differentiation (gMAD) competition. Our code are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/wangzhihua520/IQA_with_OHEM</uri>

Degraded Reference Image Quality Assessment.

In practical media distribution systems, visual content usually undergoes multiple stages of quality degradation along the delivery chain, but the pristine source content is rarely available at most quality monitoring points along the chain to serve as a reference for quality assessment. As a result, full-reference (FR) and reduced-reference (RR) image quality assessment (IQA) methods are generally infeasible. Although no-reference (NR) methods are readily applicable, their performance is often not reliable. On the other hand, intermediate references of degraded quality are often available, e.g., at the input of video transcoders, but how to make the best use of them in proper ways has not been deeply investigated. Here we make one of the first attempts to establish a new paradigm named degraded-reference IQA (DR IQA). Specifically, by using a two-stage distortion pipeline we lay out the architectures of DR IQA and introduce a 6-bit code to denote the choices of configurations. We construct the first large-scale databases dedicated to DR IQA and will make them publicly available. We make novel observations on distortion behavior in multi-stage distortion pipelines by comprehensively analyzing five multiple distortion combinations. Based on these observations, we develop novel DR IQA models and make extensive comparisons with a series of baseline models derived from top-performing FR and NR models. The results suggest that DR IQA may offer significant performance improvement in multiple distortion environments, thereby establishing DR IQA as a valid IQA paradigm that is worth further exploration.

Blind Dehazed Image Quality Assessment: A Deep CNN-Based Approach

Research on image dehazing has made the need for a suitable dehazed image quality assessment (DIQA) method even more urgent. The performance of existing DIQA methods heavily relies on handcrafted haze-related features. Since hazy images with uneven haze density distributions will result in uneven quality distributions after dehazing, the manually extracted feature expression is neither accurate nor robust. In this paper, we design a deep CNN-based DIQA method without a handcrafted feature requirement. Specifically, we propose a blind dehazed image quality assessment model (BDQM), which consists of three components: image preprocessing, a haze-related feature extraction network (HFNet), and an improved regression network (IRNet). In HFNet, we design a perceptual information enhancement (PIE) module to learn powerful feature representations and enhance network capability according to channel attention, multiscale convolution and residual concatenation. IRNet aims to aggregate all patch information for the quality prediction of the whole image, where the effect of inhomogeneous distortion from the dehazing procedure is attenuated via a specifically designed patch attention (PA) mechanism. Experimental results on benchmark datasets demonstrate the effectiveness and superiority of the proposed network architecture over state-of-the-art methods.

Conv-Former: A Novel Network Combining Convolution and Self-Attention for Image Quality Assessment.

To address the challenge of no-reference image quality assessment (NR-IQA) for authentically and synthetically distorted images, we propose a novel network called the Combining Convolution and Self-Attention for Image Quality Assessment network (Conv-Former). Our model uses a multi-stage transformer architecture similar to that of ResNet-50 to represent appropriate perceptual mechanisms in image quality assessment (IQA) to build an accurate IQA model. We employ adaptive learnable position embedding to handle images with arbitrary resolution. We propose a new transformer block (TB) by taking advantage of transformers to capture long-range dependencies, and of local information perception (LIP) to model local features for enhanced representation learning. The module increases the model's understanding of the image content. Dual path pooling (DPP) is used to keep more contextual image quality information in feature downsampling. Experimental results verify that Conv-Former not only outperforms the state-of-the-art methods on authentic image databases, but also achieves competing performances on synthetic image databases which demonstrate the strong fitting performance and generalization capability of our proposed model.

Automatic No-Reference kidney tissue whole slide image quality assessment based on composite fusion models

The visual quality of digitized whole slide images (WSIs) for kidney tissue not only affects the diagnosis and subsequent treatment, but also has a decisive impact on the accuracy of multiclass segmentation, classification and object detection during computer intelligent analysis. Currently, pathologists usually assess image quality through eye screening, which greatly relies on the pathologist experience and brings about subjectivity and non-repeatability issues. In this paper, we develop a no-reference image quality assessment framework including a fused CNN classification module, a quality score conversion module and a comprehensive quality prediction module, which automatically classifies WSIs of kidney tissue into four quality levels: excellent, good, average, and poor, and calculates a rough quality score. The original image and the regions of interest are combined and fused to comprehensively evaluate the quality of a WSI through multiple factors instead of a simple deep learning network. Extensive experiments conducted on our in-house dataset confirm that our proposed framework obtains satisfactory results with an accuracy of 90.05%, surpassing the performance of the typical image quality assessment models, and achieves the level of junior pathologist. Therefore, our proposed method can be embedded into a computer assisted diagnosis system to help pathologists in analysis of histopathological images and judgment of reliability of the images. The source code and trained models will be available at https://github.com/kidneyPathology/WSIQA.

Blind Image Quality Assessment with Deep Learning: A Replicability Study and Its Reproducibility in Lifelogging

The wide availability and small size of different types of sensors have allowed for the acquisition of a huge amount of data about a person’s life in real time. With these data, usually denoted as lifelog data, we can analyze and understand personal experiences and behaviors. Most of the lifelog research has explored the use of visual data. However, a considerable amount of these images or videos are affected by different types of degradation or noise due to the non-controlled acquisition process. Image Quality Assessment can plays an essential role in lifelog research to deal with these data. We present in this paper a twofold study on the topic of blind image quality assessment. On the one hand, we explore the replication of the training process of a state-of-the-art deep learning model for blind image quality assessment in the wild. On the other hand, we present evidence that blind image quality assessment is an important pre-processing step to be further explored in the context of information retrieval in lifelogging applications. We consider that our efforts have been successful in the replication of the model training process, achieving similar results of inference when compared to the original version, while acknowledging a fair number of assumptions that we had to consider. Moreover, these assumptions motivated an extensive additional analysis that led to significant insights on the influence of both batch size and loss functions when training deep learning models in this context. We include preliminary results of the replicated model on a lifelogging dataset, as a potential reproducibility aspect to be considered.

Hybrid No-Reference Quality Assessment for Surveillance Images

Intelligent video surveillance (IVS) technology is widely used in various security systems. However, quality degradation in surveillance images (SIs) may affect its performance on vision-based tasks, leading to the difficulties in the IVS system extracting valid information from SIs. In this paper, we propose a hybrid no-reference image quality assessment (NR IQA) model for SIs that can help to identify undesired distortions and provide useful guidelines for IVS technology. Specifically, we first extract two main types of quality-aware features: the low-level visual features related to various distortions, and the high-level semantic information, which is extracted by a state-of-the-art (SOTA) vision transformer backbone. Then, we fuse these two kinds of features into the final quality-aware feature vector, which is mapped into the quality index through the feature regression module. Our experimental results on two surveillance content quality databases demonstrate that the proposed model achieves the best performance compared to the SOTA on NR IQA metrics.

A brain-inspired computational model for extremely few reference image quality assessment

In the study of brain science, the free energy principle and attention perception mechanism have been the two of the most critical findings during the past few decades, arousing a wide range of attention and valuable applications from the research fields of image and video processing, computer vision, etc. Motivated by the aforementioned two important findings, we in this paper develop a brain-inspired computational model for extremely few reference image quality assessment (IQA), dubbed as BCM. The proposed BCM implements with the two main steps. First, we combine free energy principle and sparse perception mechanism to achieve the goal of only using extremely few reference for assessing the image quality. Second, we further introduce the attention perception mechanism to boost the assessment performance by improving the sparse perception mechanism mentioned above. Based on the most commonly used image quality database, it was found that our proposed model has derived higher performance than the peer extremely few reference IQA models and competitive performance as compared with the benchmark full reference IQA models.

Systemic distortion analysis with deep distortion directed image quality assessment models

An image processing system may bring distortions to the images going through it due to its inherent attributes or defects. Identifying the distortion types in a system is important to solve the system defects and improve its service quality. In this paper, we propose a novel algorithm to analyse the systemic distortions with multiple deep distortion directed image quality assessment models. Specifically, We pre-train several image quality assessment models that target to different distortions respectively. We use multiple images as anchor to test the system. We use the pre-trained IQA models to evaluate the anchor images that have gone through the system, and explore the features about the relationship between distortion types and quality scores. To analyse multiple distortions in a system, we embed dictionary learning into the multi-label classification framework. The experimental results demonstrate that our algorithm has good effectiveness, robustness and extensibility.

DMvLNet: deep multiview learning network for blindly accessing image quality

In image quality assessment (IQA), the quality of an image is essentially dependent on both semantic contents and image distortions. Conventional nonreference IQA (NR-IQA) methods either encode the semantic contents only or combine them in a straightforward manner without exploring their intrinsic correlation. A unified deep multiview learning network (DMvLNet) is proposed to combine both semantic contents and image distortions for NR-IQA. Using ResNet50 as the backbone, we design IQA-aware semantic features instead of directly using a recognition-specific CNN model for IQA, as in many existing methods. To this end, a high-level feature refining module built on deep layers is developed for generating refined high-level feature maps. Meanwhile, shallow layers are employed for low-level descriptions. Thus, both the refined high-level and low-level features are used for characterizing semantic contents and image distortions, respectively. Furthermore, the refined semantic features and distortion features are simultaneously integrated into DMvLNet by exploring their mutual correlation, leading to the projected multiview embedding passing through the subsequent transformer network for image quality prediction. Extensive experiments conducted on both our assembled large-scale dataset and seven public benchmark datasets demonstrate the superiority of DMvLNet over the state-of-the-art methods.

ARET-IQA: An Aspect-Ratio-Embedded Transformer for Image Quality Assessment

Image quality assessment (IQA) aims to automatically evaluate image perceptual quality by simulating the human visual system, which is an important research topic in the field of image processing and computer vision. Although existing deep-learning-based IQA models have achieved significant success, these IQA models usually require input images with a fixed size, which varies the perceptual quality of images. To this end, this paper proposes an aspect-ratio-embedded Transformer-based image quality assessment method, which can implant the adaptive aspect ratios of input images into the multihead self-attention module of the Swin Transformer. In this way, the proposed IQA model can not only relieve the variety of perceptual quality caused by size changes in input images but also leverage more global content correlations to infer image perceptual quality. Furthermore, to comprehensively capture the impact of low-level and high-level features on image quality, the proposed IQA model combines the output features of multistage Transformer blocks for jointly inferring image quality. Experimental results on multiple IQA databases show that the proposed IQA method is superior to state-of-the-art methods for assessing image technical and aesthetic quality.

Unsupervised blind image quality assessment based on joint structure and natural scene statistics features

Compared with the widely used supervised blind image quality assessment (BIQA) models, unsupervised BIQA models require little prior knowledge for calculating the objective quality scores of distorted images. In this paper, we propose an unsupervised BIQA method that aims to achieve both good performance and generalization capability with low computational complexity. Carefully selected and extensive structure and natural scene statistics (NSS) features can better represent image quality. First, we employ phase congruency (PC) and finely selected gradient magnitude map and Laplacian of Gaussian response (GM-LOG) features to represent image structure information. Second, we calculate the local mean-subtracted and contrast-normalized (MSCN) coefficients and the Karhunen–Loéve transform (KLT) coefficients to represent the naturalness of the distorted images. Last, multivariate Gaussian (MVG) model with joint features extracted from both the pristine images and the distorted images is adopted to calculate the objective image quality. Extensive experiments conducted on nine IQA databases demonstrate that the proposed method achieves better performance than the state-of-the-art BIQA methods.

Super Resolution Image Visual Quality Assessment Based on Feature Optimization.

Most existing no-referenced image quality assessment (NR-IQA) algorithms need to extract features first and then predict image quality. However, only a small number of features work in the model, and the rest will degrade the model performance. Consequently, an NR-IQA framework based on feature optimization is proposed to solve this problem and apply to the SR-IQA field. In this study, we designed a feature engineering method to solve this problem. Specifically, the features associate with the SR images were first collected and aggregated. Furthermore, several advanced feature selection algorithms were used to sort the feature sets according to their importance, and the importance matrix of features is obtained. Then, we examined the linear relationship between the number of features and Pearson linear correlation coefficient (PLCC) to determine the optimal number of features and the optimal feature selection algorithm, so as to obtain the optimal model. The results showed that the image quality scores predicted by the optimal model are in good agreement with the human subjective scores. Adopting the proposed feature optimization framework, we can effectively reduce the number of features in the model and obtain better performance. The experimental results indicated that SR image quality can be accurately predicted using only a small part of image features. In summary, we proposed a feature optimization framework to solve the current problem of irrelevant features in SR-IQA, and an SR image quality assessment model was proposed consequently.

Adaptive total variation based autofocusing strategy in ptychography

In ptychography, the axial diffraction distance between specimen and detector is extremely important since the precision of this a prior parameter can strongly affect the quality of reconstruction. However, in experiments, applying a simple evaluation model for image quality assessment (IQA) will result in low accuracy with various specimens in the autofocusing methods. In this paper, we propose an autofocusing strategy with adaptive Lp-norm based total variation (TV) model, which achieves high accuracy of the convergence. It has been verified in optical experiments that our method is applicable to various types of specimens. Besides, the method can work efficiently and stably with poorly estimated specimen-detector distance.

Fine-Grained Image Quality Assessment: A Revisit and Further Thinking

Image quality assessment (IQA) plays a central role in many image processing algorithms and systems. Although many popular IQA models achieves high performance on existing released databases, they are still not well accepted in practical applications due to the not-always satisfactory accuracy on real-world data and situations. In this paper, we revisit the IQA research, and point out an ignored but interesting problem in IQA: the coarse-grained (i.e., when quality variation is sufficiently big, as the setting of most IQA databases up to date) statistical results evaluated on existing databases mask the fine-grained differentiation. Accordingly, we present a survey on image quality assessment from a new perspective: fine-grained image quality assessment (FG-IQA). Recent FG-IQA research on five major kinds of images is introduced, and some popular IQA methods are analyzed from FG-IQA perspective. The potential problems for current IQA research based on existing coarse-grained databases are analyzed and the necessity of more FG-IQA research is justified. Finally, we discuss some challenges and possible directions for future works in FG-IQA.

Deep CNN Model for Non-Screen Content and Screen Content Image Quality Assessment

In the current world, user experience in various platforms matters a lot for different organizations. But providing a better experience can be challenging if the multimedia content on online platforms is having different kinds of distortions which impact the overall experience of the user. There can be various reasons behind distortions such as compression or minimal lighting condition while taking photos. In this work, a deep CNN-based Non-Screen Content and Screen Content NR-IQA framework is proposed which solves this issue in a more effective way. The framework is known as DNSSCIQ. Two different architectures are proposed based upon the input image type whether the input is a screen content or non-screen content image. This work attempts to solve this by evaluating the quality of such images.

ASSP: An adaptive sample statistics-based pooling for full-reference image quality assessment

Most full-reference image quality assessment (IQA) models first compute local quality scores and then pool them into an overall score. In this paper, we develop an innovative pooling strategy based on sample statistics to adaptively make the IQA more consistent with human visual assessment. The innovation of this work is threefold. First, we identify that standard sample statistics and robust sample statistics could provide complementary information about the degree of degradation in distorted images. Second, an effective IQA metric is proposed by adaptively integrating robust sample statistics and standard sample statistics via excess kurtosis. Third, instead of using the statistics directly, we adjust them by taking into account the global change of image gradients to avoid exaggerating the degradation degree. Experiments conducted on five well-known IQA databases demonstrate the effectiveness of the proposed pooling strategy in terms of high prediction accuracy and monotonicity.