Existing Image Quality Assessment Research Articles

Perceptual Quality Assessment of Omnidirectional Images: A Benchmark and Computational Model

Compared with traditional 2D images, omnidirectional images (also referred to as 360 ∘ images) have more complicated perceptual characteristics due to the particularities of imaging and display. How humans perceive omnidirectional images in an immersive environment and form the immersive quality of experience are important problems. Thus, it is crucial to measure the quality of omnidirectional images under different viewing conditions, which suffer from realistic distortions. In this article, we build a large-scale subjective assessment database for omnidirectional images and carry out a comprehensive psychophysical experiment to study the relationships between different factors (viewing conditions and viewing behaviors) and the perceptual quality of omnidirectional images. In addition, we collect both subjective ratings and head movement data. A thorough analysis of the collected subjective data is also provided, where we make several interesting findings. Moreover, with the proposed database, we propose a novel transformer-based omnidirectional image quality assessment model. To be consistent with the human viewing process, viewing conditions and behaviors are naturally incorporated into the proposed model. Specifically, the proposed model mainly consists of three parts: viewport sequence generation, multi-scale feature extraction, and perceptual quality prediction. Extensive experimental results conducted on the proposed database demonstrate the effectiveness of the proposed method over existing image quality assessment methods.

A brain-inspired quality assessment model for sonar image super-resolution

Sonar sensors are vital in the marine industry for detecting underwater targets in challenging conditions. The imaging distance and image resolution are negatively correlated due to the propagation characteristics of sound waves in water. Although Super-Resolution (SR) techniques alleviate this limitation, they introduce complex distortions that may not fit the desired utility of reconstructed sonar images. Quantifying image quality is essential, yet existing Image Quality Assessment (IQA) algorithms fail to simultaneously consider reconstruction distortions and the sonar image task background. Furthermore, the scarcity of sonar images poses challenges for deep-learning-based algorithms. To address these issues, we propose a brain-inspired model for Super-Resolution reconstructed Sonar Image Quality Assessment (SRSIQA) based on transfer learning. On the one hand, we adopt an effective feature extractor to extract multi-level features that align with the human brain recognition process. On the other hand, we develop an SEA-Block to implement feature weight adjustment and multi-level feature scale matching, given that some transferred features do not fit the IQA task well. Experimental results demonstrate the superiority of the proposed method.

Underwater image quality assessment method based on color space multi-feature fusion

The complexity and challenging underwater environment leading to degradation in underwater image. Measuring the quality of underwater image is a significant step for the subsequent image processing step. Existing Image Quality Assessment (IQA) methods do not fully consider the characteristics of degradation in underwater images, which limits their performance in underwater image assessment. To address this problem, an Underwater IQA (UIQA) method based on color space multi-feature fusion is proposed to focus on underwater image. The proposed method converts underwater images from RGB color space to CIELab color space, which has a higher correlation to human subjective perception of underwater visual quality. The proposed method extract histogram features, morphological features, and moment statistics from luminance and color components and concatenate the features to obtain fusion features to better quantify the degradation in underwater image quality. After features extraction, support vector regression(SVR) is employed to learn the relationship between fusion features and image quality scores, and gain the quality prediction model. Experimental results on the SAUD dataset and UIED dataset show that our proposed method can perform well in underwater image quality assessment. The performance comparisons on LIVE dataset, TID2013 dataset,LIVEMD dataset,LIVEC dataset and SIQAD dataset demonstrate the applicability of the proposed method.

A Soft-Reference Breast Ultrasound Image Quality Assessment Method That Considers the Local Lesion Area.

The quality of breast ultrasound images has a significant impact on the accuracy of disease diagnosis. Existing image quality assessment (IQA) methods usually use pixel-level feature statistical methods or end-to-end deep learning methods, which focus on the global image quality but ignore the image quality of the lesion region. However, in clinical practice, doctors' evaluation of ultrasound image quality relies more on the local area of the lesion, which determines the diagnostic value of ultrasound images. In this study, a global-local integrated IQA framework for breast ultrasound images was proposed to learn doctors' clinical evaluation standards. In this study, 1285 breast ultrasound images were collected and scored by experienced doctors. After being classified as either images with lesions or images without lesions, they were evaluated using soft-reference IQA or bilinear CNN IQA, respectively. Experiments showed that for ultrasound images with lesions, our proposed soft-reference IQA achieved PLCC 0.8418 with doctors' annotation, while the existing end-to-end deep learning method that did not consider the local lesion features only achieved PLCC 0.6606. Due to the accuracy improvement for the images with lesions, our proposed global-local integrated IQA framework had better performance in the IQA task than the existing end-to-end deep learning method, with PLCC improving from 0.8306 to 0.8851.

Age-specific perceptual image quality assessment

With the development of image-based applications, assessing the quality of images has become increasingly important. Although our perception of image quality changes as we age, most existing image quality assessment (IQA) metrics make simplifying assumptions about the age of observers, thus limiting their use for age-specific applications. In this work, we propose a personalized IQA metric to assess the perceived image quality of observers from different age groups. Firstly, we apply an age simulation algorithm to compute how an observer with a particular age would perceive a given image. More specifically, we process the input image according to an age-specific contrast sensitivity function (CSF), which predicts the reduction of contrast visibility associated with the aging eye. We combine age simulation with existing IQA metrics to calculate the age-specific perceived image quality score. To validate the effectiveness of our combined model, we conducted a psychophysical experiment in a controlled laboratory environment with young (18-31 y.o.), middle-aged (32-52 y.o.), and older (53+ y.o.) adults, measuring their image quality preferences for 84 test images. Our analysis shows that the predictions by our age-specific IQA metric are well correlated with the collected subjective IQA results from our psychophysical experiment.

No-Reference Image Quality Assessment based on Quality Awareness Feature and Multi-task Training

The existing image quality assessment (IQA) datasets have a small number of samples. Some methods based on transfer learning or data augmentation cannot make good use of image quality-related features. A No Reference (NR)-IQA method based on multi-task training and quality awareness is proposed. First, single or multiple distortion types and levels are imposed on the original image, and different strategies are used to augment different types of distortion datasets. With the idea of weak supervision, we use the Full Reference (FR)-IQA methods to obtain the pseudo-score label of the generated image. Then, we combine the classification information of the distortion type, level, and the information of the image quality score. The ResNet50 network is trained in the pre-train stage on the augmented dataset to obtain more quality-aware pre-training weights. Finally, the fine-tuning stage training is performed on the target IQA dataset using the quality-aware weights to predicate the final prediction score. Various experiments designed on the synthetic distortions and authentic distortions datasets (LIVE, CSIQ, TID2013, LIVEC, KonIQ-10K) prove that the proposed method can utilize the image quality-related features better than the method using only single-task training. The extracted quality-aware features improve the accuracy of the model.

Blind image quality assessment based on progressive multi-task learning

Due to the lack of adequate training data and sufficient mining of prior knowledge related to perceived quality, most existing image quality assessment (IQA) methods show limited generalization performance. In this paper, we study the prior knowledge from the factors affecting perceived quality, and introduce a novel progressive multi-task learning based blind IQA method. Inspired by the definition of IQA: human comprehensive perception for degradation of image content, we firstly deconstruct IQA into three elements, i.e., image content, pattern of degradation, and intensity of degradation. Based on these elements, we design the corresponding auxiliary tasks for instructing the network to learn IQA. By statistical analysis on a great deal of data, we find that there is progressive relevance among the four tasks. Furthermore, we mathematically derive that introducing the progressive relevance into a multi-task learning network can effectively constrain the hypothesis space of the main task. Under the guidance of the derivation, we propose an end-to-end IQA framework based on progressive multi-task learning. Experimental results demonstrate the excellent generalization capability of the proposed method, which achieves state-of-the-art performance against these existing IQA methods.

Blind Quality Evaluator of Tone-Mapped HDR and Multi-Exposure Fused Images for Electronic Display

The problem of reproducing high dynamic range (HDR) images on electronic display and photography with restricted dynamic range has gained a lot of interest in the consumer electronics community. There exist various approaches to this issue, e.g., tone mapping operators (TMOs) and multi-exposure fusion algorithms (MEFs). Many existing image quality assessment (IQA) methods have been proposed to compare images of quality degradation generated by TMOs/MEFs. Although promising performances have been achieved, they seldom consider local specific artifacts difference (i.e., abnormal exposure and color cast) related with the TMOs/MEFs. To address this limitation, this paper proposes a Blind Quality Evaluator of Tone-Mapped HDR and Multi-Exposure Fused Images (BQE-TM/MEFI). First, two purpose-designed segment models are utilized to distinguish well-exposedness dense patches (WEDPes) and non-WEDPes, color cast patches (CCPes) and non-CCPes respectively. Second, multiple quality-perception features are extracted to measure local artifacts: 1) structure and sharpness features from WEDPes, 2) saturation features from non-CCPes, and 3) edge structure features. Then, three new low-complexity regional features (over-exposure ratio, entropy and color confidence index) are calculated based on over-exposure segmentation model. Finally, all extracted features are aggregated into a machine-learning regression model to pool a quality score. The simplicity and good performance of the proposed method makes it suitable for electronic displays and other consumer electronics.

Evaluation of quality measures for color quantization

Visual quality evaluation is one of the challenging basic problems in image processing. It also plays a central role in the shaping, implementation, optimization, and testing of many methods. The existing image quality assessment methods focused on images corrupted by common degradation types while little attention was paid to color quantization. This in spite there is a wide range of applications requiring color quantization assessment being used as a preprocessing step when color-based tasks are more efficiently accomplished on a reduced number of colors. In this paper, we propose and carry-out a quantitative performance evaluation of nine well-known and commonly used full-reference image quality assessment measures. The evaluation is done by using two publicly available and subjectively rated image quality databases for color quantization degradation and by considering suitable combinations or subparts of them. The results indicate the quality measures that have closer performances in terms of their correlation to the subjective human rating and show that the evaluation of the statistical performance of the quality measures for color quantization is significantly impacted by the selected image quality database while maintaining a similar trend on each database. The detected strong similarity both on individual databases and on databases obtained by integration provides the ability to validate the integration process and to consider the quantitative performance evaluation on each database as an indicator for performance on the other databases. The experimental results are useful to address the choice of suitable quality measures for color quantization and to improve their future employment.

Contrast-distorted image quality assessment based on curvelet domain features

Contrast is one of the most popular forms of distortion. Recently, the existing image quality assessment algorithms (IQAs) works focusing on distorted images by compression, noise and blurring. Reduced-reference image quality metric for contrast-changed images (RIQMC) and no reference-image quality assessment (NR-IQA) for contrast-distorted images (NR-IQA-CDI) have been created for CDI. NR-IQA-CDI showed poor performance in two out of three image databases, where the pearson correlation coefficient (PLCC) were only 0.5739 and 0.7623 in TID2013 and CSIQ database, respectively. Spatial domain features are the basis of NR-IQA-CDI architecture. Therefore, in this paper, the spatial domain features are complementary with curvelet domain features, in order to take advantage of the potent properties of the curvelet in extracting information from images such as multiscale and multidirectional. The experimental outcome rely on K-fold cross validation (K ranged 2-10) and statistical test showed that the performance of NR-IQA-CDI rely on curvelet domain features (NR-IQA-CDI-CvT) significantly surpasses those which are rely on five spatial domain features.

MP2020: Visual quality assessment database for macro photography images

With the development of mobile phone camera technology, mobile phones can take a large number of macro photography images that previously could only be taken by professional cameras. Therefore, it is of great significance to study the quality of macro photography images. For this reason, a macro photography image visual quality evaluation database is established and it is named as MP2020. The database contains 100 reference images and 800 distorted images of four distortion types, including 200 distorted images of JPEG 2000, 200 distorted images of JPEG, 200 distorted images of white noise, and 200 distorted images of Gaussian blur. The DMOS values in the database were calculated from 48000 data which are provided by 60 subjects. Ten classical image quality assessment algorithms were tested on the MP2020 database. The experimental results show that the existing image quality assessment algorithms, which are widely used, are not applicable to the macro photography images. Therefore, MP2020 would contribute to the improvement of existing algorithms and the development of new algorithms. MP2020 has been uploaded to GitHub for download.

Perceptual Quality Evaluation of Hazy Natural Images

Haze is an intrusion element that disrupts color fidelity and contrast of outdoor natural images, affecting their perceptual quality. The differential characteristics of hazy images compared to other natural images restrict the generalization of existing image quality assessment (IQA) algorithms. At the same time, efficient IQA algorithms for predicting the perceptual quality of naturally hazed images have not been proposed in the literature due to lack of a relevant dataset. To address this, we build the IIT-JMU Hazy Image Dataset comprising of 1000 high-definition hazy natural images consisting of diverse categories such as landscape, forests, roads, seascapes, and cityscapes, along with their subjective quality ratings. We present an analysis of existing natural-scene-statistics-based IQA algorithms on hazy natural images. In this article, we propose a convolutional-neural-network-based quality assessment algorithm for hazy natural images along with an IQA metric called deep learning-based haze perceptual quality evaluator (DLHPQE). The proposed DLHPQE efficiently predicts the perceptual quality of hazy natural images without a reference. Our results demonstrate that the DLHPQE outperforms existing state-of-the-art no-reference IQAs in terms of several performance parameters such as Pearson linear correlation coefficient, Spearman rank-order correlation coefficient, Kendall's rank-order correlation coefficient, and root-mean-square error.

MMMNet: An End-to-End Multi-Task Deep Convolution Neural Network With Multi-Scale and Multi-Hierarchy Fusion for Blind Image Quality Assessment

As the evaluation of image quality depends on the human visual system (HVS), many existing image quality assessment (IQA) methods focus on modeling the HVS to account for subjective perception. The visual attention of the HVS makes humans more sensitive to distortion on the attended regions than on regions which are not the focus of attention. Therefore, we propose an end-to-end multi-task deep convolution neural network with multi-scale and multi-hierarchy fusion (MMMNet), in which the IQA and saliency subtasks are jointly optimized to improve saliency-guided IQA performance. Particularly, the incorporation of saliency information is achieved by fusing saliency features with IQA features hierarchically to progressively improve the IQA features over network depth. A multi-scale feature extraction module (MSFE) is proposed to provide effective saliency features for the IQA network. Based on the saliency fusion, MMMNet introduces an auxiliary saliency task, achieving the multi-task learning to improve the generalization of the IQA task. Experimental results show that MMMNet achieves state-of-the-art performance and strong generalization ability on IQA databases.

Image Data Assessment Approach for Deep Learning-Based Metal Surface Defect-Detection Systems

The current trend in automated optical inspection (AOI) systems employs deep learning models to detect defects on a metal surface. The setback of deep learning models is that they are time-consuming because the images obtained after every lighting adjustment must be used to train the deep learning models again and confirm whether the detection results have improved. To save the time spent using datasets to train deep networks, we proposed a comprehensive assessment score that combines defect visibility, visibility distribution, and overexposure based on the operation principles of convolution neural networks. It can be used to assess whether the training image dataset can improve the defect detection rate of the deep learning model such as You Only Look Once (YOLO), Single Shot MultiBox Detector (SSD), and Faster Region-based Convolutional Neural Network (Faster R-CNN) without training defect image datasets. We collected all of the weight combinations with the right prediction results and used linear regression to obtain the optimal weight coefficients. We found that visibility and overexposure had a greater impact on the comprehensive assessment score. We compared the proposed approach with existing image quality assessment methods, including mean absolute error (MAE), peak signal-to-noise ratio (PSNR), universal image quality index (UIQI), natural image quality evaluator (NIQE), perception-based quality evaluator (PIQE), and blind/referenceless image spatial quality evaluator (BRISQUE). The experiment results indicated that our proposed comprehensive assessment score is more correlated to the F2-score of the detection models than the IQA methods by the verification methods of Spearman Rank Correlation Coefficient (SRCC), Pearson Correlation, and Kendall Correlation. Thus, referring to this index during the collection of image data and choosing datasets with the highest score to train the model will produce better detection accuracy.

Real distorted images quality assessment based on multi-layer visual perception mechanism and high-level semantics

Most of the existing image quality assessment (IQA) methods are directed to artificially synthesized distorted images, in which the types and characteristics of distortion are different from those in the real world. In view of the fact that the existing non-reference IQA methods can not accurately evaluate the quality of the real distortion image, combined with the theoretical analysis of multi-layer visual perception mechanism, we propose a real image distortion IQA method based on image underlying features and high-level semantics. Considering non-linear hierarchical structure of human visual perception, firstly, k-means clustering algorithm is performed according to the underlying feature indexs of the image so that the used image database can be divided into several groups, which aims to improve the accuracy of predicted quality score. Secondly, the deep convolutional neural network (DCNN) is used to extract the first-grade high-level semantic features in each group. Then, second-grade high-level semantic features that can provide better representation of image features are obtained by performing multiple statistical functions on first-grade high-level semantics. Besides, we establish an effective high-capacity regressor with high-level semantics and subjective mean opinion scores (MOS) values of the human eyes. The experimental results show that the proposed model on the KonIQ-10 k image database can predict the quality score effectively and achieve a high consistency with the corresponding MOS value, which is helpful for the subsequent image enhancement.

Deep quality assessment toward defogged aerial images

In order to improve the visual appearance of defogged of aerial images, in this work, a novel defogging algorithm based on conditional generative adversarial network is proposed. More specifically, the training process is carried out through an end-to-end trainable deep neural network. In detail, we upgrade the traditional adversarial loss function by incorporating an L1-regularized gradient to encode a rich set of detailed visual information inside each aerial image. In practice, to our best knowledge, existing image quality assessment algorithms might have deviation and supersaturation distortion on aerial images. To alleviate this problem, we leverage a random forest classification model to learn the mapping relationship between aerial image features and the quality ranking results. Subsequently, we transform the objective of defogged image quality assessment into a classification problem. Comprehensive experimental results on our compiled fogged aerial images quality data set have clearly demonstrated the effectiveness of our proposed algorithm.

Perceptual Image Quality Assessment for Various Viewing Conditions and Display Systems

From complete darkness to direct sunlight, real-world displays operate in various viewing conditions often resulting in a non-optimal viewing experience. Most existing Image Quality Assessment (IQA) methods, however, assume ideal environments and displays, and thus cannot be used when viewing conditions differ from the standard. In this paper, we investigate the influence of ambient illumination level and display luminance on human perception of image quality. We conduct a psychophysical study to collect a novel dataset of over 10000 image quality preference judgments performed in illumination conditions ranging from 0 lux to 20000 lux. We also propose a perceptual IQA framework that allows most existing image quality metrics (IQM) to accurately predict image quality for a wide range of illumination conditions and display parameters. Our analysis demonstrates strong correlation between human IQA and the predictions of our proposed framework combined with multiple prominent IQMs and across a wide range of luminance values.

Full Reference Image Quality Assessment by Considering Intra-Block Structure and Inter-Block Texture

The purpose of the image quality assessment is to evaluate image quality consistently with human’s subjective evaluation. In image quality assessment, many structural features have been used to measure the quality degradation of an image. However, most of the existing image quality assessment methods are block-based, which ignores the features among neighboring blocks. In this paper, we argue that the human visual system perceives distortions not only depends on local structural (intra-block structure) distortions, but also relates to the structural distortions of their neighborhoods (inter-block texture). Based on this insight, we propose a novel image quality assessment method, called the diffusion speed structure similarity (DSSIM), by considering both intra-block structure and inter-block texture. Specifically, to characterize the inter-block texture, we devise a novel visual feature based on the image diffusion speed. To measure the changes of the intra-block structure, we adopt the image gradient magnitude. Furthermore, to differentiate the importance of a local region, we devise a weighting function based on the image diffusion speed. Extensive experimental results on six benchmark databases demonstrate that our proposed method yields a better performance than the state-of-the-art methods.

Blind Quality Metric of DIBR-Synthesized Images in the Discrete Wavelet Transform Domain

Free viewpoint video (FVV) has received considerable attention owing to its widespread applications in several areas such as immersive entertainment, remote surveillance and distanced education. Since FVV images are synthesized via a depth image-based rendering (DIBR) procedure in the "blind" environment (without reference images), a real-time and reliable blind quality assessment metric is urgently required. However, the existing image quality assessment metrics are insensitive to the geometric distortions engendered by DIBR. In this research, a novel blind method of DIBR-synthesized images is proposed based on measuring geometric distortion, global sharpness and image complexity. First, a DIBR-synthesized image is decomposed into wavelet subbands by using discrete wavelet transform. Then, the Canny operator is employed to detect the edges of the binarized low-frequency subband and high-frequency subbands. The edge similarities between the binarized low-frequency subband and high-frequency subbands are further computed to quantify geometric distortions in DIBR-synthesized images. Second, the log-energies of wavelet subbands are calculated to evaluate global sharpness in DIBR-synthesized images. Third, a hybrid filter combining the autoregressive and bilateral filters is adopted to compute image complexity. Finally, the overall quality score is derived to normalize geometric distortion and global sharpness by the image complexity. Experiments show that our proposed quality method is superior to the competing reference-free state-of-the-art DIBR-synthesized image quality models.

Quality assessment of images with multiple distortions based on phase congruency and gradient magnitude

In image communication systems, images are often contaminated by multiple types of distortions. However, most existing image quality assessment (IQA) methods mainly focused on a single type of distortions. In this paper, we proposed a no-reference (NR) IQA method for images with multiple distortions. Image distortions not only destroy the intensity of low-level image features, but also alter their distribution, to both of which the human vision system (HVS) is sensitive. Based on these observations, low-level features are represented by phase congruency (PC) which is consistent with human perception. The distribution of low-level features is extracted using local binary pattern (LBP) in PC domain at multiple scales, which can effectively characterize the impact of multiple distortions on images. Given that PC is contrast invariant while the contrast does affect perceptual image quality of the HVS, image gradient magnitude (GM) is employed as a weighting factor for LBP histogram creation. Finally a support vector regression model is trained to map the gradient-weighted LBP histograms in PC domain at multi-scale to quality scores. Experimental results on two benchmark databases demonstrate that the proposed method achieves high consistency with subjective perception and performs better than other state-of-the-art full-reference (FF) and NR IQA methods.