Blind Image Quality Assessment Model Research Articles

Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Training

Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, we propose a novel BIQA model for in-the-wild images by addressing two critical problems in this field: how to learn better quality-aware feature representation, and how to solve the problem of insufficient training samples in terms of their content and distortion diversity. Considering that perceptual visual quality is affected by both low-level visual features (e.g. distortions) and high-level semantic information (e.g. content), we first propose a staircase structure to hierarchically integrate the features from intermediate layers into the final feature representation, which enables the model to make full use of visual information from low-level to high-level. Then an iterative mixed database training (IMDT) strategy is proposed to train the BIQA model on multiple databases simultaneously, so the model can benefit from the increase in both training samples and image content and distortion diversity and can learn a more general feature representation. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model shows an excellent performance in the cross-database evaluation experiments, which further demonstrates that the learned feature representation is robust to images with diverse distortions and content. The code is available at https://github.com/sunwei925/StairIQA.

Blind Image Quality Assessment for Pathological Microscopic Image under Screen and Immersion Scenarios.

The high-quality pathological microscopic images are essential for physicians or pathologists to make a correct diagnosis. Image quality assessment (IQA) can quantify the visual distortion degree of images and guide the imaging system to improve image quality, thus raising the quality of pathological microscopic images. Current IQA methods are not ideal for pathological microscopy images due to their specificity. In this paper, we present deep learning-based blind image quality assessment model with saliency block and patch block for pathological microscopic images. The saliency block and patch block can handle the local and global distortions, respectively. To better capture the area of interest of pathologists when viewing pathological images, the saliency block is fine-tuned by eye movement data of pathologists. The patch block can capture lots of global information strongly related to image quality via the interaction between different image patches from different positions. The performance of the developed model is validated by the home-made Pathological Microscopic Image Quality Database under Screen and Immersion Scenarios (PMIQD-SIS) and cross-validated by the five public datasets. The results of ablation experiments demonstrate the contribution of the added blocks. The dataset and the corresponding code are publicly available at: https://github.com/mikugyf/PMIQD-SIS.

Deep Neural Network for Blind Visual Quality Assessment of 4K Content

The 4K content can deliver a more immersive visual experience to consumers due to the huge improvement in spatial resolution. However, the high spatial resolution brings a great challenge for video transmission and storage. Therefore, it is necessary to compress or downscale the 4K content before transmitting it to end-users. Existing blind image quality assessment (BIQA) methods are not suitable for 4K contents due to the high spatial resolution and specific distortions caused by upscaling methods. In this paper, we propose a deep learning-based BIQA model for 4K content, aiming to recognize true and pseudo 4K content and meanwhile evaluate their visual quality. Considering the characteristic that high spatial resolution can represent more abundant high-frequency information, we first propose a Grey-level Co-occurrence Matrix (GLCM) based texture complexity measure to select three representative image patches from a 4K image, which can reduce the computational complexity and is proven to be very effective for the overall quality prediction. Then, we extract various visual features from the intermediate layers of the convolutional neural network (CNN) and integrate them into the quality-aware feature representation. Finally, two multilayer perception (MLP) networks are utilized to map the quality-aware features into the class probability and the quality score of each patch respectively. The overall quality index is obtained through averaging the results of all patches. The proposed model is trained via the multi-task learning manner and the uncertainty principle is introduced to balance the losses of the classification and regression tasks. The experimental results show that the proposed model outperforms all compared BIQA metrics on four 4K content quality assessment databases.

Toward a blind image quality evaluator in the wild by learning beyond human opinion scores

Nowadays, most existing blind image quality assessment (BIQA) models inthewild heavily rely on human ratings, which are extraordinarily labor-expensive to collect. Here, we propose an opinion−free BIQA method that learns from multiple annotators to assess the perceptual quality of images captured in the wild. Specifically, we first synthesize distorted images based on the pristine counterparts. We then randomly assemble a set of image pairs from the synthetic images, and use a group of IQA models to assign pseudo-binary labels for each pair indicating which image has higher quality as the supervisory signal. Based on the newly established pseudo-labeled dataset, we train a deep neural network (DNN)-based BIQA model to rank the perceptual quality, optimized for consistency with the binary rank labels. Since there exists domain shift, e.g., distortion shift and content shift, between the synthetic and in-the-wild images, we leverage two ways to alleviate this issue. First, the simulated distortions should be similar to authentic distortions as much as possible. Second, an unsupervised domain adaptation (UDA) module is further applied to encourage learning domain-invariant features between two domains. Extensive experiments demonstrate the effectiveness of our proposed opinion−free BIQA model, yielding SOTA performance in terms of correlation with human opinion scores, as well as gMAD competition. Our code is available at: https://github.com/wangzhihua520/OF_BIQA.

LIQA: Lifelong Blind Image Quality Assessment

The image distortions are complex and dynamically changing in the real-world scenario, due to the fast development of the image processing system. The blind image quality assessment (BIQA) models may encounter the challenge of processing images with distortion types never seen before deployment. However, existing BIQA models generally cannot evolve with unseen distortion types adaptively, which greatly limits the deployment and application of BIQA models in real-world scenarios. To address this problem, we propose a novel Lifelong blind Image Quality Assessment (LIQA) approach, targeting to achieve the lifelong learning of BIQA. Without accessing to previous training data, our proposed LIQA can not only learn new knowledge, but also mitigate the catastrophic forgetting of learned knowledge. Specifically, we adopt the Split-and-Merge distillation strategy to train a single-head network that makes task-agnostic predictions. In the split stage, we first employ a distortion-specific generator to generate pseudo features of each previously seen distortion. Then, we utilize an auxiliary multi-head regression network to keep the response of each distortion. In the merge stage, we replay the pseudo features and use the pseudo labels generated by the auxiliary multi-head network to distill the knowledge of the multiple heads, which can build the final regression single head. Extensive experiments demonstrate that LIQA can perform well in handling both inner-dataset distortion shift and cross-dataset distortion shift. More importantly, our model can achieve stable performance even if the task sequences are long.

Blind Image Quality Assessment via Cross-View Consistency

Image quality assessment (IQA) is very important for both end-users and service-providers since a high-quality image can significantly improve the user's quality of experience (QoE). Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, a BIQA model is proposed that consists of a desirable self-supervised feature learning approach to mitigate the data shortage problem and learn comprehensive feature representations, and a self-attention-based feature fusion module to introduce self-attention mechanism. We develop the image quality assessment model under the framework of contrastive learning with multi views. Since human visual system perceives signals through multiple channels, the most important visual information should exist among all views of the channels. So we design the cross-view consistent information mining (CVC-IM) module to extract compact mutual information between different views. Color information and pseudo-reference image (PRI) of different distortion types are employed to formulate rich feature embeddings and preserve the quality-aware fidelity of learned representations. We employ the Transformer as the self-attention-based architecture to integrate feature embeddings. Extensive experiments show that our model achieves remarkable image quality assessment results on in-the-wild IQA datasets.

Deep Blind Image Quality Assessment Powered by Online Hard Example Mining

Recently, blind image quality assessment (BIQA) models based on deep neural networks (DNNs) have achieved impressive performance on existing datasets. However, due to the intrinsic imbalance property of the training set, not all distortions or images are handled equally well. Online hard example mining (OHEM) is a promising way to alleviate this issue. Inspired by the recent finding that network pruning disproportionately hampers the model's memorization of a tractable subset, atypical, low-quality, long-tailed samples, that are hard-to-memorize during training and easily “forgotten” during pruning, we propose an effective “plug-and-play” OHEM pipeline, especially for generalizable deep BIQA. Specifically, we train two parallel weight-sharing branches simultaneously, where one is full model and other is a “self-competitor” generated from the full model online by network pruning. Then, we leverage the prediction disagreement between the full model and its pruned variant (i.e., the self-competitor) to expose easily “forgettable” samples, which are therefore regarded as the hard ones. We then enforce the prediction consistency between the full model and its pruned variant to implicitly put more focus on these hard samples, which benefits the full model to recover forgettable information introduced by pruning. Extensive experiments across multiple datasets and BIQA models demonstrate that the proposed OHEM can improve the model performance and generalizability as measured by correlation numbers and group maximum differentiation (gMAD) competition. Our code are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/wangzhihua520/IQA_with_OHEM</uri>

Explainable and Generalizable Blind Image Quality Assessment via Semantic Attribute Reasoning

Blind image quality assessment (BIQA) that can directly evaluate image quality without perfect-quality reference has been a long-standing research topic. Although the existing BIQA models have achieved very encouraging performance, the lack of explainability and generalization ability limits their real-world applications to a great extent. People usually assess image quality according to semantic attributes, e.g., brightness, color, contrast, noise and sharpness. Furthermore, judgment on image quality is also impacted by the scene presented in the image. Therefore, the inherent relationship between semantic attributes and scenes is crucial for image quality assessment, which has rarely been explored yet. With this motivation, this paper presents a Semantic Attribute Reasoning based image QUality Evaluator (SARQUE). Specifically, we propose a two-stream network to predict semantic attributes and scene categories from distorted images. To investigate the inherent relationship between the semantic attributes and scene category, a semantic reasoning module is further proposed based on the graph convolution network (GCN), producing the final quality score. Extensive experiments conducted on five in-the-wild image quality databases demonstrate the superiority of the proposed SARQUE model over the state-of-the-arts. Furthermore, the proposed model features better explainability and generalization ability due to the use of semantic attributes.

ASCAM-Former: Blind image quality assessment based on adaptive spatial & channel attention merging transformer and image to patch weights sharing

Blind Image Quality Assessment (BIQA) is a challenging, unsolved research topic which is crucial for analyzing, understanding, and improving visual experience. Recently, transformer-based BIQA models are drawing increasing attention due to their powerful capacity in modeling global dependencies amongst tokens. However, existing works tend to apply self-attention mechanism for exploring the spatial dependencies whilst neglecting the impact of channel-wise self-attention. In this paper, we explore the feasibility of incorporating attention mechanism in a channel-wise manner for BIQA. By systematically studying the interactions between channel-wise and spatial-wise attention, an adaptive spatial and channel attention merging Transformer (ASCAM-Former) is then proposed for aggregating both the spatial-wise and channel-wise attention information. In addition, to accommodate IQA datasets containing both image and patch quality labels, an image to patch weights sharing (I2PWS) scheme is designed to take advantage of local quality learning tasks for reinforcing the learning of global quality, and vice versa. The experimental results indicate that channel-wise attention mechanism is as competitive as spatial-wise for IQA tasks, and the proposed ASCAM-Former yield accurate prediction on both authentically and synthetically distorted image quality datasets.

Unsupervised blind image quality assessment based on joint structure and natural scene statistics features

Compared with the widely used supervised blind image quality assessment (BIQA) models, unsupervised BIQA models require little prior knowledge for calculating the objective quality scores of distorted images. In this paper, we propose an unsupervised BIQA method that aims to achieve both good performance and generalization capability with low computational complexity. Carefully selected and extensive structure and natural scene statistics (NSS) features can better represent image quality. First, we employ phase congruency (PC) and finely selected gradient magnitude map and Laplacian of Gaussian response (GM-LOG) features to represent image structure information. Second, we calculate the local mean-subtracted and contrast-normalized (MSCN) coefficients and the Karhunen–Loéve transform (KLT) coefficients to represent the naturalness of the distorted images. Last, multivariate Gaussian (MVG) model with joint features extracted from both the pristine images and the distorted images is adopted to calculate the objective image quality. Extensive experiments conducted on nine IQA databases demonstrate that the proposed method achieves better performance than the state-of-the-art BIQA methods.

Blind quality assessment of authentically distorted images.

Blind image quality assessment (BIQA) of authentically distorted images is a challenging problem due to the lack of a reference image and the coexistence of blends of distortions with unknown characteristics. In this article, we present a convolutional neural network based BIQA model. It encodes the input image into multi-level features to estimate the perceptual quality score. The proposed model is designed to predict the image quality score but is trained for jointly treating the image quality assessment as a classification, regression, and pairwise ranking problem. Experimental results on three different datasets of authentically distorted images show that the proposed method achieves comparable results with state-of-the-art methods in intra-dataset experiments and is more effective in cross-dataset experiments.

Blind Image Quality Assessment Using a CNN and Edge Distortion

The present paper introduces a Convolutional Neural Network (CNN) for the assessment of image quality without a reference image, which comes under the category of Blind Image Quality Assessment models. Edge distortions in the image are characterized as input feature vectors. This approach is in justification of the fact that subjective assessment focusses on image features that emanate from the edges and the boundaries present in the image. The earlier methods were found to use complex transformations on the image to extract the features before training or as a part of the training. The present work uses Prewitt kernel approach to extract the horizontal and vertical edge maps of the training images. These maps are then input to a simple CNN for extracting higher level features using non-linear transformations. The resultant features are mapped to image quality score by regression. The network uses Spatial Pyramid Pooling (SPP) layer to accommodate input images of varying sizes. The present proposed model was tested on popular datasets used in the domain of Image Quality Assessment (IQA). The experimental results have shown that the model competes with the earlier proposed models with simplicity of feature extraction and involvement of minimal complexity.

BGT: A blind image quality evaluator via gradient and texture statistical features

Blind image quality assessment (BIQA) aims to design a model that can accurately evaluate the quality of the distorted image without any information about its reference image. Previous studies have shown that gradients and textures of image is widely used in image quality evaluation tasks. However, few studies used the joint statistics of gradient and texture information to evaluate image quality. Considering the visual perception characteristics of the human visual system, we develop a novel general-purpose BIQA model via two sets of complementary perception features. Specifically, the joint statistical histograms of gradient and texture are extracted as the first set of features, and the second set of features is extracted using the local binary pattern (LBP) operator. After extracting two groups of complementary quality-aware features, the feature vectors are sent to the support vector regression machine to establish the nonlinear relationship between quality-aware features and quality scores. A large number of experiments on seven large benchmark databases show that the proposed BIQA model has higher accuracy, better generalization properties and lower computational complexity than the relevant state-of-the-art BIQA metrics.

Uncertainty-Aware Blind Image Quality Assessment in the Laboratory and Wild.

Performance of blind image quality assessment (BIQA) models has been significantly boosted by end-to-end optimization of feature engineering and quality regression. Nevertheless, due to the distributional shift between images simulated in the laboratory and captured in the wild, models trained on databases with synthetic distortions remain particularly weak at handling realistic distortions (and vice versa). To confront the cross-distortion-scenario challenge, we develop a unified BIQA model and an approach of training it for both synthetic and realistic distortions. We first sample pairs of images from individual IQA databases, and compute a probability that the first image of each pair is of higher quality. We then employ the fidelity loss to optimize a deep neural network for BIQA over a large number of such image pairs. We also explicitly enforce a hinge constraint to regularize uncertainty estimation during optimization. Extensive experiments on six IQA databases show the promise of the learned method in blindly assessing image quality in the laboratory and wild. In addition, we demonstrate the universality of the proposed training strategy by using it to improve existing BIQA models.

Blind Image Quality Assessment by Visual Neuron Matrix

Modeling of the human visual system (HVS) is considered to be the most appropriate method to evaluate the image quality. The natural image statistics (NIS) can not only reflect certain response characteristics of the visual system, but also provide accurate quantitative descriptions of the HVS. Inspired by this fact, a visual model, which is named as visual neuron matrix (VNM), is built by a statistical learning process on natural image samples. On the basis of the VNM, a blind image quality assessment (BIQA) model is proposed. First, the VNM is obtained by independent component analysis, which can simulate the visual neurons in the cerebral cortex as well as extract the visual features from test images. Then, a regression model (neural network or support vector machine) is used to associate the visual features with the quality score of a test image. The experimental results show that the proposed method is more competitive and efficient than the state-of-the-art BIQA models.

Active Fine-Tuning From gMAD Examples Improves Blind Image Quality Assessment.

The research in image quality assessment (IQA) has a long history, and significant progress has been made by leveraging recent advances in deep neural networks (DNNs). Despite high correlation numbers on existing IQA datasets, DNN-based models may be easily falsified in the group maximum differentiation (gMAD) competition. Here we show that gMAD examples can be used to improve blind IQA (BIQA) methods. Specifically, we first pre-train a DNN-based BIQA model using multiple noisy annotators, and fine-tune it on multiple synthetically distorted images, resulting in a "top-performing" baseline model. We then seek pairs of images by comparing the baseline model with a set of full-reference IQA methods in gMAD. The spotted gMAD examples are most likely to reveal the weaknesses of the baseline, and suggest potential ways for refinement. We query human quality annotations for the selected images in a well-controlled laboratory environment, and further fine-tune the baseline on the combination of human-rated images from gMAD and existing databases. This process may be iterated, enabling active fine-tuning from gMAD examples for BIQA. We demonstrate the feasibility of our active learning scheme on a large-scale unlabeled image set, and show that the fine-tuned quality model achieves improved generalizability in gMAD, without destroying performance on previously seen databases.

Subjective and Objective De-Raining Quality Assessment Towards Authentic Rain Image

Images acquired by outdoor vision systems easily suffer poor visibility and annoying interference due to the rainy weather, which brings great challenge for accurately understanding and describing the visual contents. Recent researches have devoted great efforts on the task of rain removal for improving the image visibility. However, there is very few exploration about the quality assessment of de-rained image, even it is crucial for accurately measuring the performance of various de-raining algorithms. In this paper, we first create a de-raining quality assessment (DQA) database that collects 206 authentic rain images and their de-rained versions produced by 6 representative single image rain removal algorithms. Then, a subjective study is conducted on our DQA database, which collects the subject-rated scores of all de-rained images. To quantitatively measure the quality of de-rained image with non-uniform artifacts, we propose a bi-directional feature embedding network (B-FEN) which integrates the features of global perception and local difference together. Experiments confirm that the proposed method significantly outperforms many existing universal blind image quality assessment models. To help the research towards perceptually preferred de-raining algorithm, we will publicly release our DQA database and B-FEN source code on https://github.com/wqb-uestc .

BM-IQE: An Image Quality Evaluator with Block-Matching for Both Real-Life Scenes and Remote Sensing Scenes.

Like natural images, remote sensing scene images; of which the quality represents the imaging performance of the remote sensor, also suffer from the degradation caused by imaging system. However, current methods measuring the imaging performance in engineering applications require for particular image patterns and lack generality. Therefore, a more universal approach is demanded to assess the imaging performance of remote sensor without constraints of land cover. Due to the fact that existing general-purpose blind image quality assessment (BIQA) methods cannot obtain satisfying results on remote sensing scene images; in this work, we propose a BIQA model of improved performance for natural images as well as remote sensing scene images namely BM-IQE. We employ a novel block-matching strategy called Structural Similarity Block-Matching (SSIM-BM) to match and group similar image patches. In this way, the potential local information among different patches can get expressed; thus, the validity of natural scene statistics (NSS) feature modeling is enhanced. At the same time, we introduce several features to better characterize and express remote sensing images. The NSS features are extracted from each group and the feature vectors are then fitted to a multivariate Gaussian (MVG) model. This MVG model is therefore used against a reference MVG model learned from a corpus of high-quality natural images to produce a basic quality estimation of each patch (centroid of each group). The further quality estimation of each patch is obtained by weighting averaging of its similar patches’ basic quality estimations. The overall quality score of the test image is then computed through average pooling of the patch estimations. Extensive experiments demonstrate that the proposed BM-IQE method can not only outperforms other BIQA methods on remote sensing scene image datasets but also achieve competitive performance on general-purpose natural image datasets as compared to existing state-of-the-art FR/NR-IQA methods.

End-to-End Blind Image Quality Prediction With Cascaded Deep Neural Network

The deep convolutional neural network (CNN) has achieved great success in image recognition. Many image quality assessment (IQA) methods directly use recognition-oriented CNN for quality prediction. However, the properties of IQA task is different from image recognition task. Image recognition should be sensitive to visual content and robust to distortion, while IQA should be sensitive to both distortion and visual content. In this paper, an IQA-oriented CNN method is developed for blind IQA (BIQA), which can efficiently represent the quality degradation. CNN is large-data driven, while the sizes of existing IQA databases are too small for CNN optimization. Thus, a large IQA dataset is firstly established, which includes more than one million distorted images (each image is assigned with a quality score as its substitute of Mean Opinion Score (MOS), abbreviated as pseudo-MOS). Next, inspired by the hierarchical perception mechanism (from local structure to global semantics) in human visual system, a novel IQA-orientated CNN method is designed, in which the hierarchical degradation is considered. Finally, by jointly optimizing the multilevel feature extraction, hierarchical degradation concatenation (HDC) and quality prediction in an end-to-end framework, the Cascaded CNN with HDC (named as CaHDC) is introduced. Experiments on the benchmark IQA databases demonstrate the superiority of CaHDC compared with existing BIQA methods. Meanwhile, the CaHDC (with about 0.73M parameters) is lightweight comparing to other CNN-based BIQA models, which can be easily realized in the microprocessing system. The dataset and source code of the proposed method are available at https://web.xidian.edu.cn/wjj/paper.html .

Blind quality assessment of gamut-mapped images via local and global statistical analysis

Gamut mapping is a key technology to achieve high-quality cross-media color reproduction. To optimize a gamut mapping algorithm, an important step is to conduct an accurate evaluation of its psycho-visual performance. This paper presents an objective blind image quality assessment (BIQA) metric for gamut-mapped images based on natural scene statistics. Considering both the local and global aspects of distortions in gamut-mapped images, two categories of statistics are analyzed. Specifically, the local statistical features are used to portray structural and color distortions and features extracted from global statistics are utilized to characterize the naturalness of image. The proposed metric does not need ground truth quality scores for training, thus it is ”completely” blind. Experimental results on three gamut mapping databases demonstrate that our method outperforms the state-of-the-art general-purpose BIQA models. To further validate its effectiveness, the proposed metric is applied for benchmarking GMAs as an application and achieves encouraging performance.