Natural Scene Statistics Features Research Articles

Toward A No-reference Omnidirectional Image Quality Evaluation by Using Multi-perceptual Features

Compared to ordinary images, omnidirectional image (OI) usually has a broader view and a higher resolution, and image quality assessment (IQA) can help people to understand and improve their visual experience. However, the current IQA works cannot achieve good performance. To address this, we proposed a novel visual perception-based no-reference/blind omnidirectional image quality assessment (NR/B-OIQA) model. The gradient-based global structural features and gray-level co-occurrence matrix-based local structural features are combined together to highlight the rich quality-aware structural information. And a novel steganalysis real model-based color descriptor is extracted to reflect the color information that ignored in most IQA models. With a multi-scale visual perception, we take image entropy and the natural scene statistics features to convey the high-level semantics and quantify the unnaturalness of omnidirectional images. Finally, we apply support vector regression to predict the objective quality value based on the subjective scores and extracted all features. Experiments are conducted on OIQA and CVIQD2018 Databases, and the results illustrate that our model has more reliable performance and stronger competitiveness and receives better conformity with the subjective values.

A CNN-based no reference image quality metric exploiting content saliency

Assessing the quality of images is a challenging task. To achieve this goal, images must be evaluated by a pool of subjects following a well-defined protocol or an objective quality metric must be defined. In this work, an objective quality metric based on deep neural network is proposed. The metric takes into account the human vision system by computing the saliency map and natural scene statistics features of the image under test. The neural network is composed by two modules: the convolutional layers and the regression units. The first one is trained by using preprocessed distorted images. The feature weights of the first module are smoothed by exploiting the estimated saliency map. The latter module is fit with the ground truth quality scores of the input image and the scaled feature weights obtained from first module by using visual sensitivity factor of image obtained using natural scene statistics features. The performances of the proposed metric have been evaluated by using four datasets: LIVEIQA, TID2013, CSIQ, and KADID10K. The achieved results show the effectiveness of the proposed system in closely matching the predicted quality scores with the ground truth ones.

Unsupervised blind image quality assessment based on joint structure and natural scene statistics features

Compared with the widely used supervised blind image quality assessment (BIQA) models, unsupervised BIQA models require little prior knowledge for calculating the objective quality scores of distorted images. In this paper, we propose an unsupervised BIQA method that aims to achieve both good performance and generalization capability with low computational complexity. Carefully selected and extensive structure and natural scene statistics (NSS) features can better represent image quality. First, we employ phase congruency (PC) and finely selected gradient magnitude map and Laplacian of Gaussian response (GM-LOG) features to represent image structure information. Second, we calculate the local mean-subtracted and contrast-normalized (MSCN) coefficients and the Karhunen–Loéve transform (KLT) coefficients to represent the naturalness of the distorted images. Last, multivariate Gaussian (MVG) model with joint features extracted from both the pristine images and the distorted images is adopted to calculate the objective image quality. Extensive experiments conducted on nine IQA databases demonstrate that the proposed method achieves better performance than the state-of-the-art BIQA methods.

Multitopic underwater image quality assessment with visual attention factors

Statistical information on underwater images related to subjective quality perception differs based on the environment. We initially propose whitish, yellowish, greenish, bluish topics for underwater images obtained by applying an integrated color distribution, that is, images under the same topic have the same hue. Inspired by the local multiscale, multidirectional statistical characteristics of the structure, contrast, and color of underwater images, we constructed visual attention factor sets for the above four typical underwater environment topics. Furthermore, a multitopic underwater image quality evaluation (MTUIQE) model is proposed to measure mix distortions in different underwater images. The method of measuring image quality based on image topic is proven to be more effective than directly fitting all the natural scene statistics features to the mean opinion scores. Experiments were conducted to show that the proposed MTUIQE approach is superior to existing objective underwater image quality evaluation methods. Compared with six commonly used blind image quality evaluation methods, the Pearson linear correlation coefficient and Spearman rank-order correlation coefficient are increased by more than 10%, which means that the quality assessment model adapted to specific degradation topics is more consistent with subjective underwater image quality perception, and it can effectively evaluate the performance of underwater image enhancement and restoration methods.

Blind Image Quality Measurement via Data-Driven Transform-Based Feature Enhancement

Blind image quality measurement (BIQM) attempts to measure image perceptual quality in the absence of reference ones. Based on the types of features, BIQM methods can be divided into two categories: hand-crafted features and learning-based features. In this paper, we present a data-driven transform based feature enhancement approach for BIQM that combines the portability of hand-crafted features with the high performance of learning-based features. Specifically, we first extract the image structural features from Karhunen-Loéve transform (KLT), phase congruency (PC) and gradient magnitude (GM) coefficients, and then natural scene statistics (NSS) feature from the local normalized coefficient. Then, we utilize KLT as features enhancement process to enhance the structural and NSS features, with Weibull distribution and generalized Gaussian distribution (GGD) modeling the distributions of transform coefficient in all frequency bands as quality-aware features. Finally, support vector regression is adopted to map the features to subjective scores. The proposed method has been compared favorably with state-of-the-art BIQM methods on seven widely utilized databases on both artificial and authentic distortion types and achieves highly competitive performance.

Blind Image Quality Assessment by Natural Scene Statistics and Perceptual Characteristics

Opinion-unaware blind image quality assessment (OU BIQA) refers to establishing a blind quality prediction model without using the expensive subjective quality scores, which is a highly promising direction in the BIQA research. In this article, we focus on OU BIQA and propose a novel OU BIQA method. Specifically, in our proposed method, we deeply investigate the natural scene statistics (NSS) and the perceptual characteristics of the human brain for visual perception. Accordingly, a set of quality-aware NSS and perceptual characteristics-related features are designed to characterize the image quality effectively. For inferring the image quality, we learn a pristine multivariate Gaussian (MVG) model on a collection of pristine images, which serves as the reference information for quality evaluation. At last, the quality of a new given image is defined by measuring the divergence between its MVG model and the learned pristine MVG model. Thorough experiments performed on seven popular image databases demonstrate that the proposed OU BIQA method delivers superior performance to the state-of-the-art OU BIQA methods. The Matlab source code of the proposed method will be made publicly available at https://github.com/YT2015?tab=;repositories.

Natural scene statistics model independent no-reference image quality assessment using patch based discrete cosine transform

Most of no-reference image quality assessment (NR-IQA) techniques reported in literature have utilized transform coefficients, which are modeled using curve fitting to extract features based on natural scene statistics (NSS). The performance of NR-IQA techniques that utilize curve-fitting suffers from degradation in performance because the distribution of curve fitted NSS features deviate from the statistical distribution of a distorted image. Although deep convolutional neural networks (DCNNs) have been used for NR-IQA that are NSS model-independent but their performance is dependent upon the size of training data. The available datasets for NR-IQA are small, therefore data augmentation is used that affects the performance of DCNN based NR-IQA techniques and is also computationally expensive. This work proposes a new patch-based NR-IQA technique, which utilizes features extracted from discrete cosine transform coefficients. The proposed technique is curve fitting independent and helps in avoiding errors in the statistical distribution of NSS features. It relies on global statistics to estimate image quality based on local patches, which allow us to decompose the statistics of images. The proposed technique divides the image into patches and extracts nine handcrafted features i.e., entropy, mean, variance, skewness, kurtosis, mobility, band power, energy, complexity, and peak to peak value. The extracted features are used with a support vector regression model to predict the image quality score. The experimental results have shown that the proposed technique is database and image content-independent. It shows better performance over a majority of distortion types and on images taken in real-time.

No-reference stereoscopic images quality assessment method based on monocular superpixel visual features and binocular visual features

No-reference quality assessment of images has received considerable attention. However, the accuracy of such assessment remains questionable because of its weak biological basis. In this paper, we propose a novel quality assessment model based on the superpixel index and biological binocular mechanisms. The technical contributions of our model are the introduction of local monocular superpixel features and three global binocular visual features. We utilize monocular superpixel segmentation to extract two types of entropies as the local visual features for accurate quality-aware feature extraction. In addition, natural scene statistics features are extracted from the binocular visual information to complement the local monocular features and quantify the naturalness of the stereoscopic images. Finally, a regression model is learned to evaluate the quality of the stereoscopic images. Experimental results from three popular databases demonstrate that the proposed model has a more reliable performance than earlier models in terms of prediction accuracy and generalizability.

BM-IQE: An Image Quality Evaluator with Block-Matching for Both Real-Life Scenes and Remote Sensing Scenes.

Like natural images, remote sensing scene images; of which the quality represents the imaging performance of the remote sensor, also suffer from the degradation caused by imaging system. However, current methods measuring the imaging performance in engineering applications require for particular image patterns and lack generality. Therefore, a more universal approach is demanded to assess the imaging performance of remote sensor without constraints of land cover. Due to the fact that existing general-purpose blind image quality assessment (BIQA) methods cannot obtain satisfying results on remote sensing scene images; in this work, we propose a BIQA model of improved performance for natural images as well as remote sensing scene images namely BM-IQE. We employ a novel block-matching strategy called Structural Similarity Block-Matching (SSIM-BM) to match and group similar image patches. In this way, the potential local information among different patches can get expressed; thus, the validity of natural scene statistics (NSS) feature modeling is enhanced. At the same time, we introduce several features to better characterize and express remote sensing images. The NSS features are extracted from each group and the feature vectors are then fitted to a multivariate Gaussian (MVG) model. This MVG model is therefore used against a reference MVG model learned from a corpus of high-quality natural images to produce a basic quality estimation of each patch (centroid of each group). The further quality estimation of each patch is obtained by weighting averaging of its similar patches’ basic quality estimations. The overall quality score of the test image is then computed through average pooling of the patch estimations. Extensive experiments demonstrate that the proposed BM-IQE method can not only outperforms other BIQA methods on remote sensing scene image datasets but also achieve competitive performance on general-purpose natural image datasets as compared to existing state-of-the-art FR/NR-IQA methods.

Naturalness-Aware Deep No-Reference Image Quality Assessment

No-reference image quality assessment (NR-IQA) is a non-trivial task, because it is hard to find a pristine counterpart for an image in real applications, such as image selection, high quality image recommendation, etc. In recent years, deep learning-based NR-IQA methods emerged and achieved better performance than previous methods. In this paper, we present a novel deep neural networks-based multi-task learning approach for NR-IQA. Our proposed network is designed by a multi-task learning manner that consists of two tasks, namely, natural scene statistics (NSS) features prediction task and the quality score prediction task. NSS features prediction is an auxiliary task, which helps the quality score prediction task to learn better mapping between the input image and its quality score. The main contribution of this work is to integrate the NSS features prediction task to the deep learning-based image quality prediction task to improve the representation ability and generalization ability. To the best of our knowledge, it is the first attempt. We conduct the same database validation and cross database validation experiments on LIVE 1 , TID2013 2 , CSIQ 3 , LIVE multiply distorted image quality database (LIVE MD) 4 , CID2013 5 , and LIVE in the wild image quality challenge (LIVE challenge) 6 databases to verify the superiority and generalization ability of the proposed method. Experimental results confirm the superior performance of our method on the same database validation; our method especially achieves 0.984 and 0.986 on the LIVE image quality assessment database in terms of the Pearson linear correlation coefficient (PLCC) and Spearman rank-order correlation coefficient (SROCC), respectively. Also, experimental results from cross database validation verify the strong generalization ability of our method. Specifically, our method gains significant improvement up to 21.8% on unseen distortion types.

An effective general-purpose NR-IQA model using natural scene statistics (NSS) of the luminance relative order

Blind/no-reference image quality assessment (NR-IQA) aims to assess the quality of an image without any reference image. In this paper, we propose an effective and efficient general-purpose NR-IQA model using natural scene statistics (NSS) of the luminance relative order, based on the observation that the variation of the marginal distribution of the relative order coefficients effectively reflect the degree of warping caused by different types of image distortions. In the literature, gradient-relevant methods have had a big success in full-reference (FR) IQA and reduced-reference (RR) IQA. Inspired by these, we extend it to NR-IQA in this paper. Notice that the NSS-based models usually extract their features derived from the spatial, wavelet, DCT and spectral domain etc. Unlike these metrics, the proposed method firstly extracts 32 natural scene statistics features of the luminance relative order, obtained from the log histograms of log horizontal, vertical, main-diagonal and secondary-diagonal derivatives, along with kurtosis, variance, differential entropy and entropy at two scales. Then a mapping is learned to predict the quality score using a support vector regression. The experimental results on several benchmark databases showed that the proposed method is comparable with the state-of-the-art methods and has a relatively low complexity.

Opinion-Unaware Blind Quality Assessment of Multiply and Singly Distorted Images via Distortion Parameter Estimation.

Over the past couple of decades, numerous image quality assessment (IQA) algorithms have been developed to estimate the quality of images that contain a single type of distortion. Although in practice, images can be contaminated by multiple distortions, previous research on quality assessment of multiply-distorted images is very limited. In this paper, we propose an efficient algorithm to blindly assess the quality of both multiply and singly distorted images based on predicting the distortion parameters using a bag of natural scene statistics (NSS) features. Our method, called MUltiply- and Singlydistorted Image QUality Estimator (MUSIQUE), operates via three main stages. In the first stage, a two-layer classification model is employed to identify the distortion types (i.e., Gaussian blur, JPEG compression, and white noise) that may exist in an image. In the second stage, specific regression models are employed to predict the three distortion parameters (i.e., σG for Gaussian blur, Q for JPEG compression, and σN for white noise) by learning the different NSS features for different distortion types and combinations. In the final stage, the three estimated distortion parameter values are mapped and combined into an overall quality estimate based on quality-mapping curves and the most-apparent-distortion strategy. Experimental results tested on three multiply-distorted and seven singly-distorted image quality databases demonstrate that the proposed MUSIQUE algorithm can achieve better/competitive performance as compared to other state-of-the-art FR/NR IQA algorithms.

Blind image quality assessment utilising local mean eigenvalues

Eigenvalues are intrinsic and representative values of a square matrix. They have thus been used in many image processing areas due to their important application value, but not in the image quality assessment (IQA) field. In this Letter, the authors study the correlation between local mean eigenvalues (LMEs) and perceptual quality of images, and demonstrate the applicability of LMEs in IQA. The LMEs are related to structural complexity of images. The LMEs and natural scene statistics features are utilised for a sparse dictionary learning. Experimental results conducted on promising IQA databases show their method's superiority in comparison with top-performing blind IQA metrics.

Blind image quality assessment in multiple bandpass and redundancy domains

Blind image quality assessment (BIQA) aims to automatically evaluate image quality without any prior knowledge of reference images and distortion types. Most of existing BIQA methods use certain probability distribution models to capture the natural scene statistics features of images in bandpass domains which can be viewed as a process of removing redundancy. There also exist methods which apply NSS features in redundancy domain. In this paper, we propose a novel method that employs both bandpass and redundancy domains to acquire the complementary features in multiple color spaces. Furthermore, hierarchical feature extraction strategy is adopted to make the image representation more powerful. Then we stack them as a multi-channel feature maps group, and use Gaussian mixture model to fit them. Finally, Fisher Vectors are used to encode them and a support vector regression model is trained as the quality predictor. Extensive experiments on four commonly evaluated image quality assessment benchmark databases show the proposed method is very competitive against other BIQA methods and has good generalization ability.

No-Reference Stereoimage Quality Assessment for Multimedia Analysis Towards Internet-of-Things

With continuous progress of Internet of Things, multimedia analysis in it has attracted more and more attention. Specially, stereoscopic display technology plays an important role in the multimedia analysis processing. In the Internet of Things system, the quality of stereoscopic image will be reduced in the transmission process. In this mode, it will have a great impact on multimedia analysis to judge whether the quality of stereoscopic image meets the requirements. In this paper, a new no-reference stereoscopic image quality assessment model for multimedia analysis towards Internet of Things is built, which is based on a deep learning model to learn from the class labels and image representations. In our framework, images are represented by natural scene statistics features that are extracted from discrete cosine transform domain, and a regression model is employed to shine upon the quality from the feature vector. The training process of the proposed model contains an unsupervised pretraining phase and a supervised fine-tuning phase, enabling it to generalize over the whole distortion types and severity. The proposed model greatly shows the correlation with subjective assessment as demonstrated by experiments on the LIVE 3-D Image Quality Database and IVC 3-D Image Quality Database.

Blind Tone-Mapped Image Quality Assessment Based on Brightest/Darkest Regions, Naturalness and Aesthetics

A tone-mapped image (TMI) converted from its high-dynamic-range image (HDRI) tends to appear overexposed in its brightest regions or underexposed in its darkest regions, resulting in inevitable loss of details and impaired naturalness and aesthetics. To address this issue, this paper proposes a novel blind TMI quality assessment (BTMIQA) method for HDRIs used in different dynamic range displays. The brightest and darkest regions of the TMI are first defined and segmented, and their local detail features are used in combination with the global detail feature of the TMI to evaluate the detail distortion. The natural scene statistics features of the luminance and yellow channels of the TMI are then used to evaluate the luminance naturalness and chrominance naturalness, respectively. Subsequently, to predict the aesthetics of the TMI, a series of colorfulness features are extracted and concatenated in a final feature vector, which is used to evaluate the TMI quality by random forest regression. Experiments are performed on the TMI database and ESPL-LIVE high-dynamic-range database, and the results show the effectiveness of the proposed method. Compared with the existing TMI quality assessment methods, the proposed BTMIQA method is more consistent with human visual perception.

No-reference image quality assessment with center-surround based natural scene statistics

In this paper, we propose an efficient no-reference image quality assessment (NR-IQA) method dubbed Center-Surround based Blind Image Quality Assessment (CS-BIQA). Our proposed method employs the Difference of Gaussian (DoG) model to decompose images into several frequency bands, considering the center-surround effect and multi-channel attribute of human visual system (HVS). The integrated natural scene statistics (NSS) features can be further derived from all DoG bands. After that, regression models between the integrated features and associated subjective assessment scores are learned on the training dataset. Subsequently, the learned models are used to predict the quality scores of test images. The main contribution of this paper is twofold. Firstly, the empirical distributions of DoG bands of images are proven to be a Gaussian-like distribution. And thus, the NSS features can be employed to represent the perceptual quality of images. Secondly, different types of distortions are observed to affect different frequency components of images. So, the integrated features extracted from multi-frequency bands are employed in CS-BIQA to achieve stronger distinguishable capability of image quality. Excessive experiments are conducted to indicate that our proposed CS-BIQA metric can represent the perceptual characteristics of HVS. The results on popular IQA databases demonstrate that the CS-BIQA metric is competitive with the state-of-the-art relevant IQA metrics. Furthermore, our proposed method has very low computational complexity, making it more suitable for real-time applications.

Blind image quality assessment based on aesthetic and statistical quality-aware features

The main goal of image quality assessment (IQA) methods is the emulation of human perceptual image quality judgments. Therefore, the correlation between objective scores of these methods with human perceptual scores is considered as their performance metric. Human judgment of the image quality implicitly includes many factors when assessing perceptual image qualities such as aesthetics, semantics, context, and various types of visual distortions. The main idea of this paper is to use a host of features that are commonly employed in image aesthetics assessment in order to improve blind image quality assessment (BIQA) methods accuracy. We propose an approach that enriches the features of BIQA methods by integrating a host of aesthetics image features with the features of natural image statistics derived from multiple domains. The proposed features have been used for augmenting five different state-of-the-art BIQA methods, which use statistical natural scene statistics features. Experiments were performed on seven benchmark image quality databases. The experimental results showed significant improvement of the accuracy of the methods.

Bayesian depth estimation from monocular natural images.

Estimating an accurate and naturalistic dense depth map from a single monocular photographic image is a difficult problem. Nevertheless, human observers have little difficulty understanding the depth structure implied by photographs. Two-dimensional (2D) images of the real-world environment contain significant statistical information regarding the three-dimensional (3D) structure of the world that the vision system likely exploits to compute perceived depth, monocularly as well as binocularly. Toward understanding how this might be accomplished, we propose a Bayesian model of monocular depth computation that recovers detailed 3D scene structures by extracting reliable, robust, depth-sensitive statistical features from single natural images. These features are derived using well-accepted univariate natural scene statistics (NSS) models and recent bivariate/correlation NSS models that describe the relationships between 2D photographic images and their associated depth maps. This is accomplished by building a dictionary of canonical local depth patterns from which NSS features are extracted as prior information. The dictionary is used to create a multivariate Gaussian mixture (MGM) likelihood model that associates local image features with depth patterns. A simple Bayesian predictor is then used to form spatial depth estimates. The depth results produced by the model, despite its simplicity, correlate well with ground-truth depths measured by a current-generation terrestrial light detection and ranging (LIDAR) scanner. Such a strong form of statistical depth information could be used by the visual system when creating overall estimated depth maps incorporating stereopsis, accommodation, and other conditions. Indeed, even in isolation, the Bayesian predictor delivers depth estimates that are competitive with state-of-the-art "computer vision" methods that utilize highly engineered image features and sophisticated machine learning algorithms.

No-Reference Quality Assessment of Tone-Mapped HDR Pictures.

Being able to automatically predict digital picture quality, as perceived by human observers, has become important in many applications where humans are the ultimate consumers of displayed visual information. Standard dynamic range (SDR) images provide 8 b/color/pixel. High dynamic range (HDR) images, which are usually created from multiple exposures of the same scene, can provide 16 or 32 b/color/pixel, but must be tonemapped to SDR for display on standard monitors. Multi-exposure fusion techniques bypass HDR creation, by fusing the exposure stack directly to SDR format while aiming for aesthetically pleasing luminance and color distributions. Here, we describe a new no-reference image quality assessment (NR IQA) model for HDR pictures that is based on standard measurements of the bandpass and on newly conceived differential natural scene statistics (NSS) of HDR pictures. We derive an algorithm from the model which we call the HDR IMAGE GRADient-based Evaluator. NSS models have previously been used to devise NR IQA models that effectively predict the subjective quality of SDR images, but they perform significantly worse on tonemapped HDR content. Toward ameliorating this we make here the following contributions: 1) we design HDR picture NR IQA models and algorithms using both standard space-domain NSS features as well as novel HDR-specific gradient-based features that significantly elevate prediction performance; 2) we validate the proposed models on a large-scale crowdsourced HDR image database; and 3) we demonstrate that the proposed models also perform well on legacy natural SDR images. The software is available at: http://live.ece.utexas.edu/research/Quality/higradeRelease.zip.