Blind Quality Research Articles

No-reference quality assessment of tone-mapped image using clustering perception and salient regions detection

In order to display a high dynamic range (HDR) image on a standard display device, we need to compress an HDR image into a tone-mapped image (TMI) using the tone-mapped operator (TMO). In order to evaluate the performance of different TMOs, a blind quality evaluation method of TMI is proposed. First, the TMI is divided into bright, dark and middle regions by clustering algorithm, and the Y channel is extracted in these three regions respectively to calculate the goodness of Gaussian fitness of the histogram. Second, the TMI is decomposed by tucker decomposition, and the texture features are extracted on the first sub-band. After that, the non-negative matrix is applied to calculate the salient regions of TMI, and features in the salient regions are extracted. Then, for the sake of comprehensive evaluation the quality of TMI, the naturalness feature is extracted in the gray scale domain and colorfulness features are extracted in HSI color space. Finally, the final score quality is predicted using support vector regression model training. Experimental results on TMID and ESPL-LIVE HDR database proved the proposed method is very effective and superior to the existing blind TMI quality assessment methods.

Learning Graph Convolutional Network for Blind Mesh Visual Quality Assessment

This paper proposes a new method for blind mesh visual quality assessment (MVQA) based on a graph convolutional network. For that, we address the node classification problem to predict the perceived visual quality. First, two matrices representing the 3D mesh are considered: a graph adjacency matrix and a feature matrix. Both matrices are used as input to a shallow graph convolutional network. The network consists of two convolutional layers followed by a max-pooling layer to provide the final feature representation. With this structure, the Softmax classifier predicts the quality score category without the reference mesh's availability. Experiments are conducted on four publicly available databases constructed explicitly for the mesh quality assessment task. We investigate several perceptual and visual features to select the most effective combination. Comparisons with the state-of-the-art alternative methods show the effectiveness of the proposed framework.

Pseudoreference Subaperture Images and Microlens Image-Based Blind Light Field Image Quality Measurement

Light field image (LFI) records the radiation intensity and direction information of light in scene, offering powerful capabilities for computer vision. It is found that the distortion caused by LFI processing is obviously reflected in the two representations of LFI, that is, subaperture images (SAIs) and microlens images (MLI). Specifically, on the one hand, pseudoreference SAI (PSAI) of a distorted SAI, which is rendered with the adjacent distorted SAIs, can be used to measure the distortion of the involved SAIs. On the other hand, macropixel blocks (MPBs) of MLI can be adopted to measure LFI global angular distortion. Therefore, a PSAIs- and MLI-based blind LFI quality measurement (PM-BLFIQM) method is proposed in this article. First, the local light field (LLF) is defined to generate PSAI of the central distorted SAI in each LLF of LFI, and then a pseudo-full-reference quality measurement metric based on PSAIs is presented. Second, to measure the angular consistency of the distorted LFI, an MLI-based feature extraction scheme is proposed for MPBs and their mean removal version by using discrete cosine transform and singular value decomposition. Finally, the quality of the distorted LFI is predicted with the features extracted from LLFs and MLI. Experimental results demonstrate that both PSAI and MPB are effective for measuring the quality of distorted LFI. In addition, the results on three LFI databases show that the proposed method achieves higher Spearman rank-order correlation coefficient (SROCC) and Pearson linear correlation coefficient (PLCC) values, so it outperforms the state-of-the-art methods in terms of consistency with human visual perception.

RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content

Blind or no-reference video quality assessment of user-generated content (UGC) has become a trending, challenging, heretofore unsolved problem. Accurate and efficient video quality predictors suitable for this content are thus in great demand to achieve more intelligent analysis and processing of UGC videos. Previous studies have shown that natural scene statistics and deep learning features are both sufficient to capture spatial distortions, which contribute to a significant aspect of UGC video quality issues. However, these models are either incapable or inefficient for predicting the quality of complex and diverse UGC videos in practical applications. Here we introduce an effective and efficient video quality model for UGC content, which we dub the Rapid and Accurate Video Quality Evaluator (RAPIQUE), which we show performs comparably to state-of-the-art (SOTA) models but with orders-of-magnitude faster runtime. RAPIQUE combines and leverages the advantages of both quality-aware scene statistics features and semantics-aware deep convolutional features, allowing us to design the first general and efficient spatial and temporal (space-time) bandpass statistics model for video quality modeling. Our experimental results on recent large-scale UGC video quality databases show that RAPIQUE delivers top performances on all the datasets at a considerably lower computational expense. We hope this work promotes and inspires further efforts towards practical modeling of video quality problems for potential real-time and low-latency applications. To promote public usage, an implementation of RAPIQUE has been made freely available online: \url{https://github.com/vztu/RAPIQUE}.

UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content.

Recent years have witnessed an explosion of user-generated content (UGC) videos shared and streamed over the Internet, thanks to the evolution of affordable and reliable consumer capture devices, and the tremendous popularity of social media platforms. Accordingly, there is a great need for accurate video quality assessment (VQA) models for UGC/consumer videos to monitor, control, and optimize this vast content. Blind quality prediction of in-the-wild videos is quite challenging, since the quality degradations of UGC videos are unpredictable, complicated, and often commingled. Here we contribute to advancing the UGC-VQA problem by conducting a comprehensive evaluation of leading no-reference/blind VQA (BVQA) features and models on a fixed evaluation architecture, yielding new empirical insights on both subjective video quality studies and objective VQA model design. By employing a feature selection strategy on top of efficient BVQA models, we are able to extract 60 out of 763 statistical features used in existing methods to create a new fusion-based model, which we dub the VIDeo quality EVALuator (VIDEVAL), that effectively balances the trade-off between VQA performance and efficiency. Our experimental results show that VIDEVAL achieves state-of-the-art performance at considerably lower computational cost than other leading models. Our study protocol also defines a reliable benchmark for the UGC-VQA problem, which we believe will facilitate further research on deep learning-based VQA modeling, as well as perceptually-optimized efficient UGC video processing, transcoding, and streaming. To promote reproducible research and public evaluation, an implementation of VIDEVAL has been made available online: https://github.com/vztu/VIDEVAL.

Viewport Perception Based Blind Stereoscopic Omnidirectional Image Quality Assessment

Compared with traditional 2D images, stereoscopic omnidirectional images (SOIs) usually have more complex perceptual factors due to the particularities of imaging and display, making the objective quality assessment of SOIs challenging. In this paper, we construct a large and diverse subjective SOIs database named as NBU-SOID for further research demand. And then, we propose a viewport perception based blind SOIs quality assessment (VP-BSOIQA) method by considering the impacts of viewport, user behavior and stereoscopic perception on human visual system, which is mainly composed of binocular perception model (BPM) and omnidirectional perception model (OPM). In the BPM, a binocular combination perception map is generated by the dimension reduction of stereopair and the weighting of binocular energy to reflect the binocular masking effect. In the OPM, several viewports are first created to ensure the consistency of evaluation objects. Then, the intra-viewport and inter-viewport weighting factors are designed with the common influences of visual attention and peripheral vision sensitivity to aggregate the novel multi-orientation structural features extracted from all potential viewports. Experimental results on the NBU-SOID and SOLID databases demonstrate that BPM and OPM can be robustly combined with the existing 2D image quality assessment (IQA) methods, thus averagely achieving 10.2% and 12.2% performance gain in terms of SRCC, respectively. In addition, the proposed VP-BSOIQA method outperforms the state-of-the-art blind IQA methods in predicting the quality of SOIs.

Multi-layer fusion network for blind stereoscopic 3D visual quality prediction

Stereoscopic 3D (S3D) visual quality prediction (VQP) is used to predict human perception of visual quality for S3D images accurately and automatically. Unlike that of 2D VQP, the quality prediction of S3D images is more difficult owing to complex binocular vision mechanisms. In this study, inspired by the binocular fusion and competition of the binocular visual system (BVS), we designed a blind deep visual quality predictor for S3D images. The proposed predictor is a multi-layer fusion network that fuses different levels of features. The left- and right-view sub-networks use the same structure and parameters. The weights and qualities for the left- and right-view patches of S3D images can be predicted. Furthermore, training patches with more saliency information can improve the accuracy of prediction results, which also make the predictor more robust. The LIVE 3D Phase I and II datasets were used to evaluate the proposed predictor. The results demonstrate that the performance of the proposed predictor surpasses most existing predictors on both asymmetrically and symmetrically distorted S3D images.

Noise reduction for sonar images by statistical analysis and fields of experts

Sonar images are usually suffering from speckle noise which results in poor visual quality. In order to improve the sonar imaging quality, removing or reducing these speckle noises is a very important and arduous task. In this paper, the imaging principle and noise characteristics of the side-scan sonar (SSS) are analyzed, and five typical probability distribution functions are used to fit the seabed reverberation. Through experiment comparison, the Gamma distribution is selected to simulate the noise of the SSS image caused by the reverberation. Simultaneously, the fields of experts denoising algorithm based on the Gamma distribution (Gamma FoE) is proposed for SSS image denoising. In order to perceive and measure the denoising effect better, evaluation indexes of Fast Noise Variance Estimation (FNVE, an image noise estimation method) and Blind Referenceless Image Spatial Quality Evaluator (BRISQUE, an image quality evaluation method) are selected for image quality perception. The final results of the SSS image denoise experiment show that the Gamma FoE denoise algorithm has a better effect on SSS image denoise application than other denoise algorithms.

Blind Quality Assessment of Screen Content Images Via Macro-Micro Modeling of Tensor Domain Dictionary

Screen content images (SCIs) have been rapidly and widely applied in interactive multimedia applications. The problem of quality assessment for SCIs is an interesting research topic. Most of the existing methods use subjective and independent features in gray domain to predict the image quality, which cannot comprehensively characterize the image properties or lack unified mathematical explanation for SCIs. To address these problems, we propose a novel blind quality assessment method based on macro-micro modeling of tensor domain dictionary for SCIs in this article. In the proposed method, the tensor decomposition is explored first to avoid the loss of color information, and then a target dictionary is learned more effectively with the principal components. Furthermore, a macro-micro model is established to characterize the micro and macro features in the target dictionary space, which can provide a systematic mathematical interpretation for feature extraction. For the micro features, a log-normal pooling scheme is designed to enhance the effectiveness of feature aggregation by analyzing the particularity of the statistical distribution of sparse codes. Additionally, the statistical properties are mainly discussed and studied based on the Bernoulli law of large numbers, and then a reliable macro feature is generated to describe the relationship between the statistical distribution and quality degradation of SCIs. Experimental results determined by using three public SCI databases show that the proposed method can perform better than relevant existing methods in the prediction of the visual quality of SCIs, especially in terms of the generalization for distortion type and interpretability for feature generation.

Blind quality assessment of omnidirectional videos using spatio-temporal convolutional neural networks

Panoramic videos are different from 2D videos in the spherical viewing mode. Normally, 2D videos are viewed by consumers on flat screen televisions, while panoramic videos need specific head-mounted displays to create virtual reality experience. Therefore, it is meaningful to study a quality evaluation model suitable for panoramic videos, which will bring a better quality of experience in the field of visual consumption. Unfortunately, to the best of our knowledge, there are few omnidirectional video quality assessments especially in no-reference quality evaluation modes. Besides, the existing 2D quality evaluation methods cannot well evaluate panoramic content quality. Based on above motivation and the vacancy of related research, we propose a blind deep learning-driven quality evaluation framework for panoramic videos to address the correlative issues. From the perspective of the viewport format, we eliminate the sampling distortion of panoramic videos in equirectangular projection (ERP) format via cubemap projection (CMP) format projection. In addition, we use a two-stream convolutional network to fully extract the intra-frame and inter-frame (defined as spatio-temporal) information for more comprehensive modeling of panoramic video features. Experimental results show that our method is superior to the existing full-reference and no-reference methods in solving the task of panoramic video quality assessment.

No Reference Quality Assessment for Screen Content Images Using Stacked Autoencoders in Pictorial and Textual Regions.

Recently, the visual quality evaluation of screen content images (SCIs) has become an important and timely emerging research theme. This article presents an effective and novel blind quality evaluation metric for SCIs by using stacked autoencoders (SAE) based on pictorial and textual regions. Since the SCI consists of not only the pictorial area but also the textual area, the human visual system (HVS) is not equally sensitive to their different distortion types. First, the textual and pictorial regions can be obtained by dividing an input SCI via an SCI segmentation metric. Next, we extract quality-aware features from the textual region and pictorial region, respectively. Then, two different SAEs are trained via an unsupervised approach for quality-aware features that are extracted from these two regions. After the training procedure of the SAEs, the quality-aware features can evolve into more discriminative and meaningful features. Subsequently, the evolved features and their corresponding subjective scores are input into two regressors for training. Each regressor can obtain one output predictive score. Finally, the final perceptual quality score of a test SCI is computed by these two predicted scores via a weighted model. Experimental results on two public SCI-oriented databases have revealed that the proposed scheme can compare favorably with the existing blind image quality assessment metrics.

Blind Quality Evaluation for Screen Content Images Based on Regionalized Structural Features

Currently, screen content images (SCIs) are widely used in our modern society. However, since SCIs have distinctly different properties compared to natural images, traditional quality assessment methods of natural images cannot precisely evaluate the quality of SCIs. Thus, we propose a blind quality evaluation method for SCIs based on regionalized structural features that are closely relevant to the intrinsic quality of SCIs. Firstly, the features of textual and pictorial regions of SCIs are extracted separately. For textual regions, since they contain noticeable structural information, we propose improved histograms of oriented gradients extracted from multi-order derivatives as structural features. For pictorial regions, since human vision is sensitive to texture information and luminance variation, we adopt texture as the structural feature; meanwhile, luminance is used as the auxiliary feature. The local derivative pattern and the shearlet local binary pattern are used to extract texture in the spatial and shearlet domains, respectively. Secondly, to derive the quality of textual and pictorial regions, two mapping functions are respectively trained from their features to subjective values. Finally, an activity weighting strategy is proposed to combine the quality of textual and pictorial regions. Experimental results show that the proposed method achieves better performance than the state-of-the-art methods.

The subcortical basis of outcome and cognitive impairment in TBI: A longitudinal cohort study.

To understand how, biologically, the acute event of traumatic brain injury gives rise to a long-term disease, we address the relationship between evolving cortical and subcortical brain damage and measures of functional outcome and cognitive functioning at 6 months after injury. For this longitudinal analysis, clinical and MRI data were collected in a tertiary neurointensive care setting in a continuous sample of 157 patients surviving moderate to severe traumatic brain injury between 2000 and 2018. For each patient, we collected T1- and T2-weighted MRI data acutely and at the 6-month follow-up, as well as acute measures of injury severity (Glasgow Coma Scale), follow-up measures of functional impairment (Glasgow Outcome Scale-extended), and, in a subset of patients, neuropsychological measures of attention, executive functions, and episodic memory. In the final cohort of 113 subcortical and 92 cortical datasets that survived (blind) quality control, extensive atrophy was observed over the first 6 months after injury across the brain. However, only atrophy within subcortical regions, particularly in the left thalamus, was associated with functional outcome and neuropsychological measures of attention, executive functions, and episodic memory. Furthermore, when brought together in an analytical model, longitudinal brain measurements could distinguish good from bad outcome with 90% accuracy, whereas acute brain and clinical measurements alone could achieve only 20% accuracy. Despite great injury heterogeneity, secondary thalamic pathology is a measurable minimum common denominator mechanism directly relating biology to clinical measures of outcome and cognitive functioning, potentially linking the acute event and the longer-term disease of traumatic brain injury.

Opinion-unaware blind picture quality measurement using deep encoder–decoder architecture

Recently, deep-learning-based blind picture quality measurement (BPQM) metrics have gained significant attention. However, training a robust deep BPQM metric remains a difficult and challenging task because of the limited number of subject-rated training samples. State-of-the-art full-reference (FR) picture quality measurement (PQM) metrics are in good agreement with human subjective quality scores. Therefore, they can be employed to approximate human subjective quality scores to train BPQM metrics. Inspired by this, we propose a deep encoder–decoder architecture (DEDA) for opinion-unaware (OU) BPQM that does not require human-labeled distorted samples for training. In the training procedure, to avoid overfitting and to ensure the independency of the training and testing samples, we first construct 6,000 distorted pictures and use their objective quality/similarity maps obtained using a high-performance FR-PQM metric for distorted pictures as training labels. Subsequently, an end-to-end map between the distorted pictures and their objective quality/similarity maps (labels) is learned, represented as the DEDA that takes the distorted picture as the input and outputs its predicted quality/similarity map. In the DEDA, the pyramid supervision training strategy is used, which applies supervised learning over three scale layers to efficiently optimize the parameters. In the testing procedure, the quality/similarity maps of the testing samples that can help localize distortions can be predicted with the trained DEDA architecture. The predicted quality/similarity maps are then gradually pooled together to obtain the overall objective quality scores. Comparative experiments on three publicly available standard PQM datasets demonstrate that our proposed DEDA metric is in good agreement with subjective assessment compared to previous state-of-the-art OU-BPQM metrics.

Blind Image Quality Assessment by Natural Scene Statistics and Perceptual Characteristics

Opinion-unaware blind image quality assessment (OU BIQA) refers to establishing a blind quality prediction model without using the expensive subjective quality scores, which is a highly promising direction in the BIQA research. In this article, we focus on OU BIQA and propose a novel OU BIQA method. Specifically, in our proposed method, we deeply investigate the natural scene statistics (NSS) and the perceptual characteristics of the human brain for visual perception. Accordingly, a set of quality-aware NSS and perceptual characteristics-related features are designed to characterize the image quality effectively. For inferring the image quality, we learn a pristine multivariate Gaussian (MVG) model on a collection of pristine images, which serves as the reference information for quality evaluation. At last, the quality of a new given image is defined by measuring the divergence between its MVG model and the learned pristine MVG model. Thorough experiments performed on seven popular image databases demonstrate that the proposed OU BIQA method delivers superior performance to the state-of-the-art OU BIQA methods. The Matlab source code of the proposed method will be made publicly available at https://github.com/YT2015?tab=;repositories.

Blind quality assessment for multiply distorted stereoscopic images towards IoT-based 3D capture systems

Empowered by 5G mobile communication networks, multimedia processing has been considered as a very promising application of Internet-of-Things (IoT). Stereoscopic image quality assessment (SIQA), as an important part of 3D capture system, can be embedded in the cloud or fog servers to automatically monitor the perceptual quality of the collected stereoscopic images. In this paper, a novel blind image quality assessment method towards IoT-based 3D capture systems is developed for multiply-distorted stereoscopic images (MDSIs), in which five complementary channels, including left view, right view, cyclopean map, summation map and difference map, are jointly considered in dictionary learning for characterizing the monocular receptive field (MRF) and binocular receptive field (BRF) properties of the visual cortex in response to MDSIs. Additionally, the high order statistics scheme is adopted by utilizing the statistical differences between the codebook and images to ensure the stable and robust quality prediction performance for MDSIs. The proposed method shows competitive prediction performances on four benchmark databases compared with the existing SIQA metrics.

Blind Tone-Mapped Image Quality Assessment Based on Regional Sparse Response and Aesthetics.

High dynamic range (HDR) images give a strong disposition to capture all parts of natural scene information due to their wider brightness range than traditional low dynamic range (LDR) images. However, to visualize HDR images on common LDR displays, tone mapping operations (TMOs) are extra required, which inevitably lead to visual quality degradation, especially in the bright and dark regions. To evaluate the performance of different TMOs accurately, this paper proposes a blind tone-mapped image quality assessment method based on regional sparse response and aesthetics (RSRA-BTMI) by considering the influences of detail information and color on the human visual system. Specifically, for the detail loss in a tone-mapped image (TMI), multi-dictionaries are first designed for different brightness regions and whole TMI. Then regional sparse atoms aggregated by local entropy and global reconstruction residuals are presented to characterize the regional and global detail distortion in TMI, respectively. Besides, a few efficient aesthetic features are extracted to measure the color unnaturalness of TMI. Finally, all extracted features are linked with relevant subjective scores to conduct quality regression via random forest. Experimental results on the ESPL-LIVE HDR database demonstrate that the proposed RSRA-BTMI method is superior to the existing state-of-the-art blind TMI quality assessment methods.

DrlNet: Blind object proposal quality assessment with discriminative response learning

Object proposal quality assessment without ground truth as reference is a challenging task. Some existing methods measure the quality with hand-crafted metrics for subjective metrics, such as objectness and foreground confidence. Recently, deep learning is adopted for direct assessment for quantifiable metric, such as Intersection over Union (IoU). However, we find that IoU, the commonly used quality metric, is far from fully describing the quality of an object proposal. Proposals with the same IoU score may carry totally different amount of discriminative attribute. We introduce a new metric named Discriminative Information Richness (DIR) to characterize the discriminative degree of the given object proposal. DIR is derived from the response intensity of the projected deep feature maps, whose high correlation response indicates the discriminative regions. Besides, we design a convolutional neural network named DrlNet to simultaneously predict IoU scores and perceive the richness of the identification information. DrlNet is defined as a multi-metric joint deep regression network for both spatial covering prediction and discriminative information richness perception. Compared with the solely IoU based models, DrlNet can provide more comprehensive quality assessment. We perform comprehensive experiments on both PASCAL VOC dataset and COCO dataset. The experimental results show that our DrlNet performs well on both proposal selection and object detection tasks. Particularly, experimental results on COCO dataset demonstrate the good generalization ability of the proposed model.

Comparison between Digital Tone-Mapping Operators and a Focal-Plane Pixel-Parallel Circuit

In a previous work, we proposed a color extension of an existing focal-plane tone-mapping operator (FPTMO) and introduced circuit modifications that led to smaller sensing array area and simpler off-chip white-balance operations. The proposed FPTMO chip was fabricated and its experimental test setup is under development. In the present work, we systematically compare the previously proposed FPTMOs with conventional digital tone-mapping operators (DTMOs), both in terms of overall image quality and execution time. To assess the image quality, we use the Tone-Mapped image Quality Index (TMQI) metric, the Blind Tone-Mapped Quality Index (BTMQI) metric, and an image colorfulness metric. By statistically comparing the results given by both metrics, we verify that the proposed FPTMOs achieve results similar to those obtained with DTMOs, occasionally outperforming them. To compare the tone-mapping operators execution times, we perform a detailed complexity analysis. Results show that system-level software FPTMO descriptions (FPTMO functionality described in software) rank among the fastest DTMOs and that the hardware FPTMO (described in software) speed-up depends on the target frame rate. For 30-frames-per-second color image generation, the fastest FPTMO is 15 times faster than the DTMO with highest TMQI and BTMQI metric scores, and 170 times faster than the DTMO with best colorfulness metric score.

Corrections to “Blind quality assessment for image superresolution using deep two-stream convolutional networks”

Corrections to “Blind quality assessment for image superresolution using deep two-stream convolutional networks”