Perceptual Quality Assessment Research Articles

Comparative Evaluation of User Perceived Quality Assessment of Design Strategies for HTTP-based Adaptive Streaming

HTTP-based Adaptive Streaming (HAS) is the dominant Internet video streaming application. One specific HAS approach, Dynamic Adaptive Streaming over HTTP (DASH), is of particular interest, as it is a widely deployed, standardized implementation. Prior academic research has focused on networking and protocol issues, and has contributed an accepted understanding of the performance and possible performance issues in large deployment scenarios. Our work extends the current understanding of HAS by focusing directly on the impacts of choice of the video quality adaptation algorithm on end-user perceived quality. In congested network scenarios, the details of the adaptation algorithm determine the amount of bandwidth consumed by the application as well as the quality of the rendered video stream. HAS will lead to user-perceived changes in video quality due to intentional changes in quality video segments, or unintentional perceived quality impairments caused by video decoder artifacts such as pixelation, stutters, or short or long stalls in the rendered video when the playback buffer becomes empty. The HAS adaptation algorithm attempts to find the optimal solution to mitigate the conflict between avoiding buffer stalls and maximizing video quality. In this article, we present results from a user study that was designed to provide insights into “best practice guidelines” for a HAS adaptation algorithm. Our findings suggest that a buffer-based strategy might provide a better experience under higher network impairment conditions. For the two network scenarios considered, the buffer-based strategy is effective in avoiding stalls but does so at the cost of reduced video quality. However, the buffer-based strategy does yield a lower number of quality switches as a result of infrequent bitrate adaptations. Participants in buffer-based strategy do notice the drop in video quality causing a decrease in perceived QoE, but the perceived levels of video quality, viewer frustration, and opinions of video clarity and distortion are significantly worse due to artifacts such as stalls in capacity-based strategy. The capacity-based strategy tries to provide the highest video quality possible but produces many more artifacts during playback. The results suggest that player video quality has more of an impact on perceived quality when stalls are infrequent. The study methodology also contributes a unique method for gathering continuous quantitative subjective measure of user perceived quality using a Wii remote.

Validation of algorithmic CT image quality metrics with preferences of radiologists

Automated assessment of perceptual image quality on clinical Computed Tomography (CT) data by computer algorithms has the potential to greatly facilitate data-driven monitoring and optimization of CT image acquisition protocols. The application of these techniques in clinical operation requires the knowledge of how the output of the computer algorithms corresponds to clinical expectations. This study addressed the need to validate algorithmic image quality measurements on clinical CT images with preferences of radiologists and determine the clinically acceptable range of algorithmic measurements for abdominal CT examinations. Algorithmic measurements of image quality metrics (organ HU, noise magnitude, and clarity) were performed on a clinical CT image dataset with supplemental measures of noise power spectrum from phantom images using techniques developed previously. The algorithmic measurements were compared to clinical expectations of image quality in an observer study with seven radiologists. Sets of CT liver images were selected from the dataset where images in the same set varied in terms of one metric at a time. These sets of images were shown via a web interface to one observer at a time. First, the observer rank ordered the CT images in a set according to his/her preference for the varying metric. The observer then selected his/her preferred acceptable range of the metric within the ranked images. The agreement between algorithmic and observer rankings of image quality were investigated and the clinically acceptable image quality in terms of algorithmic measurements were determined. The overall rank-order agreements between algorithmic and observer assessments were 0.90, 0.98, and 1.00 for noise magnitude, liver parenchyma HU, and clarity, respectively. The results indicate a strong agreement between the algorithmic and observer assessments of image quality. Clinically acceptable thresholds (median) of algorithmic metric values were (17.8, 32.6) HU for noise magnitude, (92.1, 131.9) for liver parenchyma HU, and (0.47, 0.52) for clarity. The observer study results indicated that these algorithms can robustly assess the perceptual quality of clinical CT images in an automated fashion. Clinically acceptable ranges of algorithmic measurements were determined. The correspondence of these image quality assessment algorithms to clinical expectations paves the way toward establishing diagnostic reference levels in terms of clinically acceptable perceptual image quality and data-driven optimization of CT image acquisition protocols.

PeQASO: Perceptual Quality Assessment of Streamed Videos Using Optical Flow Features

In this paper, we introduce PeQASO, a perceptual quality assessment framework for streamed videos using optical flow features. This approach is a reduced-reference pixel-based and relies only on the deviation of the optical flow of the corrupted frames. This technique compares an optical flow descriptor from the received frame against the descriptor obtained from the anchor frame. This approach is suitable for videos with complex motion patterns. Our technique does not make any assumptions on the coding conditions, network loss patterns or error concealment techniques. In this paper, we consider both sources of artifacts and distortions in streaming, including compression artifacts. We validate our proposed metric by testing it on a variety of distorted sequences from three proposed and commonly utilized video quality assessment databases. Our results show that our metric estimates the perceptual quality at the sequence level accurately. We report the correlation coefficients with the differential mean opinion scores reported in the database. For compression artifacts, the results show Spearman’s and Pearson’s correlations of 0.96 and 0.94 for all the tested sequences, respectively. For channel-induced distortion, the results show Spearman’s and Pearson’s correlations of 0.88 and 0.89, respectively. For all other distortions, the average Spearman’s and Pearson’s correlations are 0.82 and 0.79, respectively.

Correlation of Algorithmic and Visual Assessment of Lesion Detection in Clinical Images

Clinically-relevant quantitative measures of task-based image quality play key roles in effective optimization of medical imaging systems. Conventional phantom-based measures do not adequately reflect the real-world image quality of clinical Computed Tomography (CT) series which is most relevant for diagnostic decision-making. The assessment of detectability index which incorporates measurements of essential image quality metrics on patient CT images can overcome this limitation. Our current investigation extends and validates the technique on standard-of-care clinical cases. We obtained a clinical CT image dataset from an Institutional Review Board-approved prospective study on colorectal adenocarcinoma patients for detecting hepatic metastasis. For this study, both perceptual image quality and lesion detection performance of same-patient CT image series with standard and low dose acquisitions in the same breath hold and four processing algorithms applied to each acquisition were assessed and ranked by expert radiologists. The clinical CT image dataset was processed using the previously validated method to estimate a detectability index for each known lesion size in the size distribution of hepatic lesions relevant for the imaging task and for each slice of a CT series. We then combined these lesion-size-specific and slice-specific detectability indexes with the size distribution of hepatic lesions relevant for the imaging task to compute an effective detectability index for a clinical CT imaging condition of a patient. The assessed effective detectability indexes were used to rank task-based image quality of different imaging conditions on the same patient for all patients. We compared the assessments to those by expert radiologists in the prospective study in terms of rank order agreement between the rankings of algorithmic and visual assessment of lesion detection and perceptual quality. Our investigation indicated that algorithmic assessment of lesion detection and perceptual quality can predict observer assessment for detecting hepatic metastasis. The algorithmic and visual assessment of lesion detection and perceptual quality are strongly correlated using both the Kendall's Tau and Spearman's Rho methods (perfect agreement has value 1): for assessment of lesion detection, 95% of the patients have rank correlation coefficients values exceeding 0.87 and 0.94, respectively, and for assessment of perceptual quality, 0.85 and 0.94, respectively. This study used algorithmic detectability index to assess task-based image equality for detecting hepatic lesions and validated it against observer rankings on standard-of-care clinical CT cases. Our study indicates that detectability index provides a robust reflection of overall image quality for detecting hepatic lesions under clinical CT imaging conditions. This demonstrates the concept of utilizing the measure to quantitatively assess the quality of the information content that different imaging conditions can provide for the same clinical imaging task, which enables targeted optimization of clinical CT systems to minimize clinical and patient risks.

Depth perceptual quality assessment for symmetrically and asymmetrically distorted stereoscopic 3D videos

Depth perception is one of the most important features provided by stereoscopic 3D videos, which is also the major difference to 2D videos. Its quality, called depth perceptual quality or depth quality, is one of the key factors that directly affects the quality of experience of 3D videos. Since 3D video shall be compressed, transmitted and reconstructed in 3D video system, distortion introduced from compression and processing affects the video quality as well as the depth perceptual quality. However, understandings of how the distortions affect the depth perception and how to evaluate the depth perceptual quality of 3D video remain limited. In this paper, subjective experiments are firstly conducted for investigating the compression and processing distortions’ impacts on the perceived depth quality of 3D videos. The subjective experiments over the datasets show that the loss of video details can cause the degradation of the depth perceptual quality in monocular and binocular perspectives. In addition, the relationships between video quality and depth quality for both symmetric and asymmetric distorted 3D stereoscopic videos are analyzed. Both the subjective video quality and depth quality scores are released and available for public. Secondly, an objective depth quality assessment algorithm is proposed for measuring the depth perceptual quality degradation of symmetrically and asymmetrically distorted stereoscopic 3D videos. The experimental results demonstrate that the proposed metric has better consistency with the subjective scores of human vision system compared with the state-of-the-art schemes.

Perceptual stereoscopic images quality assessment method based on visual adaptable characteristics

The quality assessment of stereoscopic images plays an important role in various three-dimensional (3-D) areas and encounters more problems than two-dimensional (2-D) image quality assessment. We propose a perceptual full-reference quality assessment model by considering human adaptable double visual channel. Human binocular combination characteristics and depth perception are both considered in this model, and an adaptable gain-control model is adopted to assign appropriate weights to the information of human visual channels. Experimental results indicate that the proposed algorithm can serve as an efficient predictive image quality feature, which delivers not only highly competitive prediction accuracy but also moderate computational complexity.

Perceived Quality Evaluation with the Use of Extended Reality

AbstractIf designers want to communicate quality aspects of the product, there is a need to bring these characteristics into the measurable space of perceived quality (PQ) attributes. To illustrate the solution for designers' dilemma of the “best design choice” in this study we applied the PQ attributes importance ranking (PQAIR) method, with the example of a bread toaster. We choose for evaluation three PQ attributes which can significantly influence visual quality of a product: Gap, Flush and Parallelism. We performed the experiment measuring subjective preferences over the toaster designs of two respondent's groups - “Designers” and “Customers.” We used sequentially: (i) web-survey (still images); (ii) desktop system; and (iii) fully immersive head-mounted display system (Virtual Reality).Consequently, we conducted a post-experiment survey regarding subjective preferences, related to the PQ communication channels that have been implemented during the study. Our results indicate advantages and drawbacks for each PQ communication method that we applied in this experiment and encourage further research in the area of products' perceived quality assessment

Uniform Color Space-Based High Dynamic Range Video Compression

Recently, there has been a significant progress in the research and development of the high dynamic range (HDR) video technology and the state-of-the-art video pipelines are able to offer a higher bit depth support to capture, store, encode, and display HDR video content. In this paper, we introduce a novel HDR video compression algorithm, which uses a perceptually uniform color opponent space, a novel perceptual transfer function to encode the dynamic range of the scene, and a novel error minimization scheme for accurate chroma reproduction. The proposed algorithm was objectively and subjectively evaluated against four state-of-the-art algorithms. The objective evaluation was conducted across a set of 39 HDR video sequences, using the latest x265 10-bit video codec along with several perceptual and structural quality assessment metrics at 11 different quality levels. Furthermore, a rating-based subjective evaluation ( $n=40$ ) was conducted with six sequences at two different output bitrates. Results suggest that the proposed algorithm exhibits the lowest coding error amongst the five algorithms evaluated. Additionally, the rate–distortion characteristics suggest that the proposed algorithm outperforms the existing state-of-the-art at bitrates ≥ 0.4 bits/pixel.

An End-to-End Perceptual Quality Assessment Method via Score Distribution Prediction

Image quality assessment (IQA) has become a rapidly growing field of technology as it automatically predicts the perceptual quality, which is of vital importance for consumer-centric services. However, most existing IQA algorithms focus on predicting the mean opinion score regardless of the inevitable opinion diversity. To address this shortcoming, in this paper, we propose to predict the distribution of opinion scores via an end-to-end convolutional neural network. The network is based on a pre-trained ResNet with 50 layers and a novel Statistical Region-of-Interest (ROI) Pooling layer is introduced for lower model complexity, which enables effective training with few datum. Meanwhile, instead of using traditional mean-square-error as loss function, our model is trained with cross-entropy loss, which is more suitable for probability distribution learning. Extensive experiments have been carried out on ESPL-LIVE HDR datasets with highly diverse opinion scores. It is shown that the statistical ROI Pooling is more efficient than traditional ROI Pooling layers and classical dimensionality reduction of principle component analysis. And the proposed algorithm achieves superior performance than state-of-the-art label distribution learning methods in terms of six representative evaluation metrics.

Perceptual quality assessment of video using machine learning algorithm

User experience has become the most reliable and trustworthy source for service providers to assess system performance. To fulfill customer requirements, service providers require an efficient quality of experience (QoE) estimation model. QoE is a subjective metric that deals with user perception and can vary dramatically due to various factors such as emotions, the degree of annoyance, past experience, and aesthetics. Moreover, subjective QoE evaluation is expensive and time-consuming because of human participation. Therefore, a model is required to objectively measure QoE with reasonable accuracy. In the context of service and network providers fulfilling user requirements, with a reasonable quality of service (QoS), needs to be provided to the relevant services/applications. However, QoS parameters do not reflect subjective opinion of the user accurately. Therefore, it is necessary to compute the mapping or correlation between QoS and QoE. The mapping may help service providers to understand the behavior of the overall network on user experience and efficiently manage the network resources. In this paper, a new feature number of displayed frames impacted (NoDFI) along with a machine learning based model is presented to compute a correlation between QoS and QoE. A publically available dataset is used to represent the correlation between objective QoS parameters and subjective QoE metric. The result of experiments showed that proposed feature NoDFI proved to be a valuable addition when compared with previously proposed models.

No-reference perceptual quality assessment of stereoscopic images based on binocular visual characteristics

Perceptual quality assessment of distorted stereoscopic images has attracted considerable attention and become an important yet challenging task in 3D multimedia applications. In this paper, we propose a blind/no-reference stereoscopic 3D image quality assessment scheme that utilizes binocular visual characteristics. The design of this scheme is motivated by studies on the perception of distorted stereoscopic images. The major technical contribution of this paper is the introduction of binocular features and binocular energy for stereoscopic image quality prediction. To be more specific, the proposed scheme extracts the local magnitude, local phase and visual saliency from a stereopair as binocular features. In addition, the binocular energy responses are also obtained as quality-predictive features. After feature extraction, a statistical regression model is used to map the features of distorted stereoscopic image to its subjective quality score. The effectiveness of the designed metric is verified on the publicly available 3D image quality assessment databases. Experimental results show that the proposed scheme achieves superiority over other related algorithms in terms of consistent alignment with subjective assessment of stereoscopic 3D images.

Perceptual Quality Assessment of Pan-Sharpened Images

Pan-sharpening (PS) is a method of fusing the spatial details of a high-resolution panchromatic (PAN) image with the spectral information of a low-resolution multi-spectral (MS) image. Visual inspection is a crucial step in the evaluation of fused products whose subjectivity renders the assessment of pansharpened data a challenging problem. Most previous research on the development of PS algorithms has only superficially addressed the issue of qualitative evaluation, generally by depicting visual representations of the fused images. Hence, it is highly desirable to be able to predict pan-sharpened image quality automatically and accurately, as it would be perceived and reported by human viewers. Such a method is indispensable for the correct evaluation of PS techniques that produce images for visual applications such as Google Earth and Microsoft Bing. Here, we propose a new image quality assessment (IQA) measure that supports the visual qualitative analysis of pansharpened outcomes by using the statistics of natural images, commonly referred to as natural scene statistics (NSS), to extract statistical regularities from PS images. Importantly, NSS are measurably modified by the presence of distortions. We analyze six PS methods in the presence of two common distortions, blur and white noise, on PAN images. Furthermore, we conducted a human study on the subjective quality of pristine and degraded PS images and created a completely blind (opinion-unaware) fused image quality analyzer. In addition, we propose an opinion-aware fused image quality analyzer, whose predictions with respect to human perceptual evaluations of pansharpened images are highly correlated.

Toward a Quality Predictor for Stereoscopic Images via Analysis of Human Binocular Visual Perception

Perceptual stereoscopic image quality assessment (SIQA) has become a challenge research problem due to the poor understanding of human binocular visual characteristics. For the task of SIQA, an intuitive idea is to develop effective models on the basis of the image content and depth perception. In this paper, we propose a full-reference objective quality evaluator for stereoscopic images by simulating binocular behaviors of the human visual system (HVS): Binocular interaction and depth perception. This model is based on a cyclopean image from a novel binocular combination model as image content quality description and a depth binocular combination model from a depth synthesized procedure as depth perception description. The final quality score of the distorted stereoscopic images is calculated by integrating the above two perception indicators. The experimental results on two stereoscopic image quality databases demonstrate that our proposed metric works efficiently for both symmetric and asymmetric distortions and achieves high consistent alignment with subjective observations.

A No-Reference Image Quality Assessment Metric by Multiple Characteristics of Light Field Images

Evaluation of light field image (LFI), especially micro-lens camera light field (LF), is a new and challenging work. The development of image quality assessment (IQA) metric of LFIs relies on the subjective quality assessment database. In this paper, we establish a perceptual quality assessment dataset consisting of 240 distorted images from 8 source images with five distortion types. Furthermore, a no-reference IQA metric is proposed by combining 2D and 3D characteristics of LFI with the Support Vector Regression (SVR) model. The performance of the proposed metric is demonstrated by comparing with some classical full reference IQA metrics both on the presented dataset and a third-party dataset. The experiment results show that our method has a better performance than others.

Dynamical stochastic resonance for non‐uniform illumination image enhancement

Images taken under poor illumination conditions have low contrast and dark tones. General dark image enhancement algorithms cannot effectively enhance these images without introducing over-enhancement, detail loss, and noise amplification. In this study, a simple and fast enhancement technique of non-uniform illumination images is proposed based on dynamical stochastic resonance (DSR). The low-contrast images are enhanced through the nonlinear iteration by solving monostable Langevin equation. Iteration parameters are dynamically adjusted according to the intensity distribution of the original images, which ensure the balance of visibility and naturalness in the entire areas. A threshold is defined to automatically confirm the optimal outputs. The enhanced image is obtained by fusing the DSR result, original component, and illumination compensation component. The computational time, no-reference perceptual quality assessment, and lightness order error are measured to evaluate the performance of experimental results. The subjective and objective comparison with state-of-the-art methods shows that our method performs well to enhance the non-uniform illumination images with a low-computational complexity.

Perceptual quality assessment of stereoscopic images based on local and global visual characteristics

The quality assessment of stereoscopic images has attracted considerable attention and become an important issue in 3D multimedia applications. The 3D image quality assessment (IQA) encounters many challenges and simple extension of the 2D quality metrics to the 3D case is not satisfying. In this paper, we propose a new perceptual quality assessment scheme for stereoscopic 3D images by considering the local and global visual characteristics. The design of this scheme is motivated by studies on the perception of distorted stereoscopic images. To be more specific, after the log-Gabor filter processing, the local amplitude and phase from the left and right views of the reference and distorted 3D images are utilized as features in local quality evaluation. Meanwhile, the global structure changes of the left and right views are also incorporated into the final quality pooling. The overall 3D quality score is obtained by combining the local and global quality indexes together. The effectiveness of the designed metric is verified on publicly available 3D image quality assessment databases. Experimental results show that the proposed scheme exhibits better performance than other related algorithms in terms of consistency with subjective assessment of stereoscopic 3D images.

Analysis of Structural Characteristics for Quality Assessment of Multiply Distorted Images

Perceptual image quality assessment (IQA) plays an important role in numerous applications, including image restoration, compression, enhancement, and others. Although many works have been conducted on individually distorted IQA problems and have achieved encouraging results, few studies have been conducted on multiple distorted (MD) IQA problems. Thus, limited progress has been made. In this paper, we propose a novel no reference image quality assessment (NR-IQA) method, named improved multiscale local binary pattern (IMLBP), for addressing multiply distorted IQA problems. The image structures are sensitive to image distortions, which motivates us to utilize the structural characteristics for overall image quality prediction. We improved the local binary pattern (LBP) by considering the human visual mechanism to better extract the structural information. The IMLBP contains two parts, the LBP and the radius difference LBP (DLBP). The DLBP reflects the values’ changes in the radial direction. Specifically, when the radius value is small, the proposed descriptor is computed to represent microstructural information. Conversely, it represents macrostructural information when the radius becomes large. Moreover, to better mimick the human visual mechanism, the IMLBP is computed with the multiscale strategy and the operation is based on a patch unit whose size is proportional to the radius value. The frequency histogram of feature maps is transformed to feature vectors. Subsequently, a predictable function trained by the support vector regression is used to infer the overall quality score. Experimental results show that the proposed method outperforms most state-of-the-art IQA metrics on publicly available multiply distorted image databases.

GPU Acceleration of the Most Apparent Distortion Image Quality Assessment Algorithm

The primary function of multimedia systems is to seamlessly transform and display content to users while maintaining the perception of acceptable quality. For images and videos, perceptual quality assessment algorithms play an important role in determining what is acceptable quality and what is unacceptable from a human visual perspective. As modern image quality assessment (IQA) algorithms gain widespread adoption, it is important to achieve a balance between their computational efficiency and their quality prediction accuracy. One way to improve computational performance to meet real-time constraints is to use simplistic models of visual perception, but such an approach has a serious drawback in terms of poor-quality predictions and limited robustness to changing distortions and viewing conditions. In this paper, we investigate the advantages and potential bottlenecks of implementing a best-in-class IQA algorithm, Most Apparent Distortion, on graphics processing units (GPUs). Our results suggest that an understanding of the GPU and CPU architectures, combined with detailed knowledge of the IQA algorithm, can lead to non-trivial speedups without compromising prediction accuracy. A single-GPU and a multi-GPU implementation showed a 24× and a 33× speedup, respectively, over the baseline CPU implementation. A bottleneck analysis revealed the kernels with the highest runtimes, and a microarchitectural analysis illustrated the underlying reasons for the high runtimes of these kernels. Programs written with optimizations such as blocking that map well to CPU memory hierarchies do not map well to the GPU’s memory hierarchy. While compute unified device architecture (CUDA) is convenient to use and is powerful in facilitating general purpose GPU (GPGPU) programming, knowledge of how a program interacts with the underlying hardware is essential for understanding performance bottlenecks and resolving them.

Perceptual image quality assessment through spectral analysis of error representations

In this paper, we analyze the statistics of error signals to assess the perceived quality of images. Specifically, we focus on the magnitude spectrum of error images obtained from the difference of reference and distorted images. Analyzing spectral statistics over grayscale images partially models interference in spatial harmonic distortion exhibited by the visual system but it overlooks color information, selective and hierarchical nature of visual system. To overcome these shortcomings, we introduce an image quality assessment algorithm based on the Spectral Understanding of Multi-scale and Multi-channel Error Representations, denoted as SUMMER. We validate the quality assessment performance over 3 databases with around 30 distortion types. These distortion types are grouped into 7 main categories as compression artifact, image noise, color artifact, communication error, blur, global and local distortions. In total, we benchmark the performance of 17 algorithms along with the proposed algorithm using 5 performance metrics that measure linearity, monotonicity, accuracy, and consistency. In addition to experiments with standard performance metrics, we analyze the distribution of objective and subjective scores with histogram difference metrics and scatter plots. Moreover, we analyze the classification performance of quality assessment algorithms along with their statistical significance tests. Based on our experiments, SUMMER significantly outperforms majority of the compared methods in all benchmark categories.

Blind quality assessment of three-dimensional images using joint statistics of binocular rivalry and orientation-tuned responses

Perceptual quality assessment of a three-dimensional (3-D) image is one of the most important tasks in various applications such as 3-D image coding, processing, enhancement, and monitoring system. But objective quality assessment of 3-D images is still a challenging task. Especially, blind quality assessment of 3-D images encounters an arduous challenge due to lack of prior information about the original images. To solve this problem, we propose a blind 3-D images’ quality evaluator by simulating binocular rivalry and orientation responses of the human visual system. As a main technical contribution of this research, both the low- and high-level binocular rivalry responses (BRR), as well as binocular orientation-tuned (BOT) responses, are considered for blind quality assessment of 3-D images. Specifically, the self-similarity of the BRR and BOT responses is extracted from the distorted 3-D images, which will change in the presence of distortions. Subsequently, all quality-aware features are mapped to subjective quality scores of the distorted 3-D images via using support vector regression. The performance of our algorithm is evaluated over two popular LIVE 3-D phase I and phase II databases and shown to be competitive with the state-of-the-art algorithms.