Perceptual Quality Assessment Research Articles

Color distortion and edge feature for perceptual quality assessment

The color distortion effect has an important impact on the perceived quality, which is ignored in previous related works. Unified with the color distortion outcome and edge similarity, a new full-reference image quality assessment was proposed named the gradient similarity-based distorted pixel and deformed color measure (GDCM). The components RGB of the color image are converted into image coded in YIQ color space. Then, Ruderman operators and the gradient images are calculated from the Y component. I and Q elements are used to identify the color distortion. Finally, the previous results are combined to compute the ultimate measure. Experimental results on databases illustrate that the GDCM performs very well.

A content-oriented no-reference perceptual video quality assessment method for computer graphics animation videos

In this paper, we propose a content-oriented no-reference (NR) perceptual video quality assessment (VQA) method for computer graphics (CG) animation videos. First, we extract features in terms of spatiotemporal information and its visual perception from the videos as inputs of our proposed artificial neural network-based VQA model. Second, to facilitate the video quality evaluation, we apply a convolutional neural network (CNN) in the VQA model to generate weight factors for the input features adaptively according to the different types of CG content in videos. Third, we build a subjective CG video quality database for validation of VQA metrics. Experiments demonstrated that our method achieved superior performance in terms of evaluating the quality of CG animation videos. Both the code and proposed database are publicly available at https://github.com/WeizhiXian/CGVQA. The corresponding newly established database is available at https:// pan.baidu.com/s/1_P2ZNrLzJwZfG6xa6tKnDQ (password: cgvq).

Perceptual Quality Assessment of Omnidirectional Images

Omnidirectional images, also called 360◦images, have attracted extensive attention in recent years, due to the rapid development of virtual reality (VR) technologies. During omnidirectional image processing including capture, transmission, consumption, and so on, measuring the perceptual quality of omnidirectional images is highly desired, since it plays a great role in guaranteeing the immersive quality of experience (IQoE). In this paper, we conduct a comprehensive study on the perceptual quality of omnidirectional images from both subjective and objective perspectives. Specifically, we construct the largest so far subjective omnidirectional image quality database, where we consider several key influential elements, i.e., realistic non-uniform distortion, viewing condition, and viewing behavior, from the user view. In addition to subjective quality scores, we also record head and eye movement data. Besides, we make the first attempt by using the proposed database to train a convolutional neural network (CNN) for blind omnidirectional image quality assessment. To be consistent with the human viewing behavior in the VR device, we extract viewports from each omnidirectional image and incorporate the user viewing conditions naturally in the proposed model. The proposed model is composed of two parts, including a multi-scale CNN-based feature extraction module and a perceptual quality prediction module. The feature extraction module is used to incorporate the multi-scale features, and the perceptual quality prediction module is designed to regress them to perceived quality scores. The experimental results on our database verify that the proposed model achieves the competing performance compared with the state-of-the-art methods.

Perceptual quality assessment of panoramic stitched contents for immersive applications: a prospective survey

The recent advancements in the field of Virtual Reality (VR) and Augmented Reality (AR) have a substantial impact on modern day technology by digitizing each and everything related to human life and open the doors to the next generation Software Technology (Soft Tech). VR and AR technology provide astonishing immersive contents with the help of high quality stitched panoramic contents and 360° imagery that widely used in the education, gaming, entertainment, and production sector. The immersive quality of VR and AR contents are greatly dependent on the perceptual quality of panoramic or 360° images, in fact a minor visual distortion can significantly degrade the overall quality. Thus, to ensure the quality of constructed panoramic contents for VR and AR applications, numerous Stitched Image Quality Assessment (SIQA) methods have been proposed to assess the quality of panoramic contents before using in VR and AR. In this survey, we provide a detailed overview of the SIQA literature and exclusively focus on objective SIQA methods presented till date. For better understanding, the objective SIQA methods are classified into two classes namely Full-Reference SIQA and No-Reference SIQA approaches. Each class is further categorized into traditional and deep learning-based methods and examined their performance for SIQA task. Further, we shortlist the publicly available benchmark SIQA datasets and evaluation metrices used for quality assessment of panoramic contents. In last, we highlight the current challenges in this area based on the existing SIQA methods and suggest future research directions that need to be target for further improvement in SIQA domain.

Editorial: Computational Neuroscience for Perceptual Quality Assessment

Editorial: Computational Neuroscience for Perceptual Quality Assessment

Face anti-spoofing detection based on multi-scale image quality assessment

This article introduces an effective face PAD algorithm based on multiscale perceptual image quality assessment features. Unique hand-crafted texture features extracted from face images are exploited for spoofing detection. The proposed features are classified into three major models: generalized Gaussian density-based, asymmetric generalized Gaussian density-based, and top gradient similarity deviation features. In light of the essential attributes of these models, a total of 21 multiscale features are acquired for classification, which is performed through a support vector machine (SVM). Extensive experiments on five benchmark databases, CASIA, Replay-Attack, UVAD, OULU-NPU, and SiW along with our new dataset demonstrated the effectiveness of the proposed framework. Experimental results indicated that our face PAD algorithm produced satisfactory detection accuracy on the tested datasets based on both intra-dataset and cross-dataset protocols. While outperforming a number of traditional face PAD methods, the proposed scheme achieved comparable results with many state-of-the-art deep learning-based networks. The introduction of the image quality assessment features with multiscale analysis into face PAD is promising for detection accuracy improvement.

Perceptual Quality Assessment of HEVC and VVC Standards for 8K Video

With the growing data consumption of emerging video applications and users’ requirement for higher resolutions, up to 8K, a huge effort has been made in video compression technologies. Recently, versatile video coding (VVC) has been standardized by the moving picture expert group (MPEG), providing a significant improvement in compression performance over its predecessor high efficiency video coding (HEVC). In this paper, we provide a comparative subjective quality evaluation between VVC and HEVC standards for 8K resolution videos. In addition, we evaluate the perceived quality improvement offered by 8K over UHD 4K resolution. The compression performance of both VVC and HEVC standards has been conducted in random access (RA) coding configuration, using their respective reference software, VVC test model (VTM-11) and HEVC test model (HM-16.20). Objective measurements, using PSNR, MS-SSIM and VMAF metrics have shown that the bitrate gains offered by VVC over HEVC for 8K video content are around 31%, 26% and 35%, respectively. Subjectively, VVC offers an average of around 41% of bitrate reduction over HEVC for the same visual quality. A compression gain of 50% has been reached for some tested video sequences regarding a Student’s t-test analysis. In addition, for most tested scenes, a significant visual difference between uncompressed 4K and 8K has been noticed.

Deep ensembling for perceptual image quality assessment

Deep ensembling for perceptual image quality assessment

Cross-Domain Feature Similarity Guided Blind Image Quality Assessment.

This work proposes an end-to-end cross-domain feature similarity guided deep neural network for perceptual quality assessment. Our proposed blind image quality assessment approach is based on the observation that features similarity across different domains (e.g., Semantic Recognition and Quality Prediction) is well correlated with the subjective quality annotations. Such phenomenon is validated by thoroughly analyze the intrinsic interaction between an object recognition task and a quality prediction task in terms of characteristics of the human visual system. Based on the observation, we designed an explicable and self-contained cross-domain feature similarity guided BIQA framework. Experimental results on both authentical and synthetic image quality databases demonstrate the superiority of our approach, as compared to the state-of-the-art models.

Exploring Contrast Multi-Attribute Representation With Deep Network for No-Reference Perceptual Quality Assessment

Aiming at the effectiveness of contrast feature design, we proposed a promising novel non-reference quality assessment approach in exploring Attribute-Based representation. The method generates three perceptual attribute categories tailored to contrast. The first is semantic attribute derived from deep convolutional neural network, which implements adaptive contrast prediction relevant to scenario content. Second, for perceiving Spatial channel attribute, the global and local features generated by dark channel map through the designed dual convolution structures. Third, for statistical attribute, we assume the enhanced image as “reference” and calculate the structural similarity with pristine image, and the entropy and histogram metrics are also employed to assist learning. After that, for maximizing utilization, the features are embedded and integrated hierarchically to translate into objective score. In addition, a medium-scale contrast distortion database is established to support further research, which is more challenging than existing datasets because of the sufficient content and sophisticated changes. We demonstrate the availability of structures quantitatively and verify the rationality of hypothesis. Extensive experiments reveal that the proposed method outperforms advanced methods and achieves the state-of-the-art on the created database and CSIQ, TID2013, CCID2014.

Screen Content Video Quality Assessment Model Using Hybrid Spatiotemporal Features.

In this paper, a full-reference video quality assessment (VQA) model is designed for the perceptual quality assessment of the screen content videos (SCVs), called the hybrid spatiotemporal feature-based model (HSFM). The SCVs are of hybrid structure including screen and natural scenes, which are perceived by the human visual system (HVS) with different visual effects. With this consideration, the three dimensional Laplacian of Gaussian (3D-LOG) filter and three dimensional Natural Scene Statistics (3D-NSS) are exploited to extract the screen and natural spatiotemporal features, based on the reference and distorted SCV sequences separately. The similarities of these extracted features are then computed independently, followed by generating the distorted screen and natural quality scores for screen and natural scenes. After that, an adaptive screen and natural quality fusion scheme through the local video activity is developed to combine them for arriving at the final VQA score of the distorted SCV under evaluation. The experimental results on the Screen Content Video Database (SCVD) and Compressed Screen Content Video Quality (CSCVQ) databases have shown that the proposed HSFM is more in line with the perceptual quality assessment of the SCVs perceived by the HVS, compared with a variety of classic and latest IQA/VQA models.

Visual Perceptual Quality Assessment Based on Blind Machine Learning Techniques.

This paper presents the construction of a new objective method for estimation of visual perceiving quality. The proposal provides an assessment of image quality without the need for a reference image or a specific distortion assumption. Two main processes have been used to build our models: The first one uses deep learning with a convolutional neural network process, without any preprocessing. The second objective visual quality is computed by pooling several image features extracted from different concepts: the natural scene statistic in the spatial domain, the gradient magnitude, the Laplacian of Gaussian, as well as the spectral and spatial entropies. The features extracted from the image file are used as the input of machine learning techniques to build the models that are used to estimate the visual quality level of any image. For the machine learning training phase, two main processes are proposed: The first proposed process consists of a direct learning using all the selected features in only one training phase, named direct learning blind visual quality assessment . The second process is an indirect learning and consists of two training phases, named indirect learning blind visual quality assessment . This second process includes an additional phase of construction of intermediary metrics used for the construction of the prediction model. The produced models are evaluated on many benchmarks image databases as , , and in the wild image quality challenge. The experimental results demonstrate that the proposed models produce the best visual perception quality prediction, compared to the state-of-the-art models. The proposed models have been implemented on an platform to demonstrate the feasibility of integrating the proposed solution on an image sensor.

Perceptual Quality Assessment of Low-light Image Enhancement

Low-light image enhancement algorithms (LIEA) can light up images captured in dark or back-lighting conditions. However, LIEA may introduce various distortions such as structure damage, color shift, and noise into the enhanced images. Despite various LIEAs proposed in the literature, few efforts have been made to study the quality evaluation of low-light enhancement. In this article, we make one of the first attempts to investigate the quality assessment problem of low-light image enhancement. To facilitate the study of objective image quality assessment (IQA), we first build a large-scale low-light image enhancement quality (LIEQ) database. The LIEQ database includes 1,000 light-enhanced images, which are generated from 100 low-light images using 10 LIEAs. Rather than evaluating the quality of light-enhanced images directly, which is more difficult, we propose to use the multi-exposure fused (MEF) image and stack-based high dynamic range (HDR) image as a reference and evaluate the quality of low-light enhancement following a full-reference (FR) quality assessment routine. We observe that distortions introduced in low-light enhancement are significantly different from distortions considered in traditional image IQA databases that are well-studied, and the current state-of-the-art FR IQA models are also not suitable for evaluating their quality. Therefore, we propose a new FR low-light image enhancement quality assessment (LIEQA) index by evaluating the image quality from four aspects: luminance enhancement, color rendition, noise evaluation, and structure preserving, which have captured the most key aspects of low-light enhancement. Experimental results on the LIEQ database show that the proposed LIEQA index outperforms the state-of-the-art FR IQA models. LIEQA can act as an evaluator for various low-light enhancement algorithms and systems. To the best of our knowledge, this article is the first of its kind comprehensive low-light image enhancement quality assessment study.

Perceptual quality assessment of nighttime video

The quality of nighttime videos is important for consumer photography and monitoring of video clearness. However, little work has been done on the study of nighttime video quality assessment. In this paper, to the best of our knowledge, we explore the study on the nighttime video quality assessment for the first time. First, we build a real-world nighttime video quality assessment database (NVQA) containing 200 videos with abundant content and diverse distortion. Additionally, we carry out subjective tests to rate all nighttime videos in the NVQA database. Thereafter, we proposed a blind nighttime video quality assessment model based on feature fusion and conducted experiments to evaluate the performance and efficiency of our proposed model. The experiment results demonstrate that our model outperforms most traditional methods.

GSW: A High Performance IQA Index Based on Global and Saliency Window Similarity

Perceptual image quality assessment (IQA) attracts significant attention in recent years. It is proved that both global score and an image’s visual saliency (VS) are consistent with subjective evaluation. The global quality score reflects the consistency of the overall structure between two images, and VS map contributes complementarily in evaluating perceptual quality. This paper presents an effective IQA index based on global and saliency window similarity, namely GSW. It chooses VS map as an image feature and uses a special strategy to draw the saliency window with largest significancy. Meanwhile, the background area is taken into account to guarantee the robustness of the quality score. Experimental results on four most widely used databases verify that, compared with state-of-the-art IQA methods, GSW performs consistently well can provide more accurate quality prediction with a low computational complexity.

Comprehensive performance analysis of objective quality metrics for digital holography

Objective quality assessment of digital holograms has proven to be a challenging task. While prediction of perceptual quality of the recorded 3D content from the holographic wavefield is an open problem; perceptual quality assessment from content after rendering, requires a time-consuming rendering step and a multitude of possible viewports. In this research, we use 96 Fourier holograms of the recently released HoloDB database to evaluate the performance of well-known and state-of-the-art image quality metrics on digital holograms. We compare the reference holograms with their distorted versions: (i) before rendering on the real and imaginary parts of the quantized complex-wavefield, (ii) after converting Fourier to Fresnel holograms, (iii) after rendering, on the quantized amplitude of the reconstructed data, and (iv) after subsequently removing speckle noise using a Wiener filter. For every experimental track, the quality metric predictions are compared to the Mean Opinion Scores (MOS) gathered on a 2D screen, light field display and a holographic display. Additionally, a statistical analysis of the results and a discussion on the performance of the metrics are presented. The tests demonstrate that while for each test track a few quality metrics present a highly correlated performance compared to the multiple sets of available MOS, none of them demonstrates a consistently high-performance across all four test-tracks.

Perceptual Quality Assessment of Compressed Vibrotactile Signals Through Comparative Judgment.

In this article, we present a comprehensive scheme for the quality assessment of compressed vibrotactile signals with human assessors. Inspired by the multiple stimulus test with hidden reference and anchors (MUSHRA) from the audio domain, we designed a method in which each compressed signal is compared to its original signal and rated on a numerical scale. For each signal tested, the hidden reference and two anchor signals are used to validate the results and provide assessor screening criteria. Differing from previous approaches, our method is hierarchically structured and strictly timed in a sequential manner to avoid experimental confounds and provide precise psychophysical assessments. We validated our method in an experiment with 20 human participants in which we compared two state-of-the-art lossy codecs. The results show that, with our approach, the performance of different codecs can be compared effectively. Furthermore, the method also provides a measure of subjective quality at different data compression rates. The proposed procedure can be easily adapted to evaluate other vibrotactile codecs.

No-Reference Stereoscopic Image Quality Assessment Based on Binocular Statistical Features and Machine Learning

Learning a deep structure representation for complex information networks is a vital research area, and assessing the quality of stereoscopic images or videos is challenging due to complex 3D quality factors. In this paper, we explore how to extract effective features to enhance the prediction accuracy of perceptual quality assessment. Inspired by the structure representation of the human visual system and the machine learning technique, we propose a no-reference quality assessment scheme for stereoscopic images. More specifically, the statistical features of the gradient magnitude and Laplacian of Gaussian responses are extracted to form binocular quality-predictive features. After feature extraction, these features of distorted stereoscopic image and its human perceptual score are used to construct a statistical regression model with the machine learning technique. Experimental results on the benchmark databases show that the proposed model generates image quality prediction well correlated with the human visual perception and delivers highly competitive performance with the typical and representative methods. The proposed scheme can be further applied to the real-world applications on video broadcasting and 3D multimedia industry.

Video Compression With CNN-Based Postprocessing

In recent years, video compression techniques have been significantly challenged by the rapidly increased demands associated with high quality and immersive video content. Among various compression tools, post-processing can be applied on reconstructed video content to mitigate visible compression artefacts and to enhance overall perceptual quality. Inspired by advances in deep learning, we propose a new CNN-based post-processing approach, which has been integrated with two state-of-the-art coding standards, VVC and AV1. The results show consistent coding gains on all tested sequences at various spatial resolutions, with average bit rate savings of 4.0% and 5.8% against original VVC and AV1 respectively (based on the assessment of PSNR). This network has also been trained with perceptually inspired loss functions, which have further improved reconstruction quality based on perceptual quality assessment (VMAF), with average coding gains of 13.9% over VVC and 10.5% against AV1.

Perceptual Quality Assessment of Omnidirectional Images as Moving Camera Videos.

Omnidirectional images (also referred to as static 360 ° panoramas) impose viewing conditions much different from those of regular 2D images. How do humans perceive image distortions in immersive virtual reality (VR) environments is an important problem which receives less attention. We argue that, apart from the distorted panorama itself, two types of VR viewing conditions are crucial in determining the viewing behaviors of users and the perceived quality of the panorama: the starting point and the exploration time. We first carry out a psychophysical experiment to investigate the interplay among the VR viewing conditions, the user viewing behaviors, and the perceived quality of 360 ° images. Then, we provide a thorough analysis of the collected human data, leading to several interesting findings. Moreover, we propose a computational framework for objective quality assessment of 360 ° images, embodying viewing conditions and behaviors in a delightful way. Specifically, we first transform an omnidirectional image to several video representations using different user viewing behaviors under different viewing conditions. We then leverage advanced 2D full-reference video quality models to compute the perceived quality. We construct a set of specific quality measures within the proposed framework, and demonstrate their promises on three VR quality databases.