Human Perception Of Quality Research Articles

Objective Video Quality Assessment Method for Object Recognition Tasks

In the field of video quality assessment for object recognition tasks, accurately predicting the impact of different quality factors on recognition algorithms remains a significant challenge. Our study introduces a novel evaluation framework designed to address this gap by focussing on machine vision rather than human perceptual quality metrics. We used advanced machine learning models and custom Video Quality Indicators to enhance the predictive accuracy of object recognition performance under various conditions. Our results indicate a model performance, achieving a mean square error (MSE) of 672.4 and a correlation coefficient of 0.77, which underscores the effectiveness of our approach in real-world scenarios. These findings highlight not only the robustness of our methodology but also its potential applicability in critical areas such as surveillance and telemedicine.

Subjective and Objective Quality Assessment of Rendered Human Avatar Videos in Virtual Reality.

We study the visual quality judgments of human subjects on digital human avatars (sometimes referred to as "holograms" in the parlance of virtual reality [VR] and augmented reality [AR] systems) that have been subjected to distortions. We also study the ability of video quality models to predict human judgments. As streaming human avatar videos in VR or AR become increasingly common, the need for more advanced human avatar video compression protocols will be required to address the tradeoffs between faithfully transmitting high-quality visual representations while adjusting to changeable bandwidth scenarios. During transmission over the internet, the perceived quality of compressed human avatar videos can be severely impaired by visual artifacts. To optimize trade-offs between perceptual quality and data volume in practical workflows, video quality assessment (VQA) models are essential tools. However, there are very few VQA algorithms developed specifically to analyze human body avatar videos, due, at least in part, to the dearth of appropriate and comprehensive datasets of adequate size. Towards filling this gap, we introduce the LIVE-Meta Rendered Human Avatar VQA Database, which contains 720 human avatar videos processed using 20 different combinations of encoding parameters, labeled by corresponding human perceptual quality judgments that were collected in six degrees of freedom VR headsets. To demonstrate the usefulness of this new and unique video resource, we use it to study and compare the performances of a variety of state-of-the-art Full Reference and No Reference video quality prediction models, including a new model called HoloQA. As a service to the research community, we publicly releases the metadata of the new database at https://live.ece.utexas.edu/research/LIVE-Meta-rendered-human-avatar/index.html.

HDR or SDR? A Subjective and Objective Study of Scaled and Compressed Videos.

We conducted a large-scale study of human perceptual quality judgments of High Dynamic Range (HDR) and Standard Dynamic Range (SDR) videos subjected to scaling and compression levels and viewed on three different display devices. While conventional expectations are that HDR quality is better than SDR quality, we have found subject preference of HDR versus SDR depends heavily on the display device, as well as on resolution scaling and bitrate. To study this question, we collected more than 23,000 quality ratings from 67 volunteers who watched 356 videos on OLED, QLED, and LCD televisions, and among many other findings, observed that HDR videos were often rated as lower quality than SDR videos at lower bitrates, particularly when viewed on LCD and QLED displays. Since it is of interest to be able to measure the quality of videos under these scenarios, e.g. to inform decisions regarding scaling, compression, and SDR vs HDR, we tested several well-known full-reference and no-reference video quality models on the new database. Towards advancing progress on this problem, we also developed a novel no-reference model called HDRPatchMAX, that uses a contrast-based analysis of classical and bit-depth features to predict quality more accurately than existing metrics.

No-Reference Quality Assessment of Authentically Distorted Images Based on Local and Global Features.

With the development of digital imaging techniques, image quality assessment methods are receiving more attention in the literature. Since distortion-free versions of camera images in many practical, everyday applications are not available, the need for effective no-reference image quality assessment algorithms is growing. Therefore, this paper introduces a novel no-reference image quality assessment algorithm for the objective evaluation of authentically distorted images. Specifically, we apply a broad spectrum of local and global feature vectors to characterize the variety of authentic distortions. Among the employed local features, the statistics of popular local feature descriptors, such as SURF, FAST, BRISK, or KAZE, are proposed for NR-IQA; other features are also introduced to boost the performances of local features. The proposed method was compared to 12 other state-of-the-art algorithms on popular and accepted benchmark datasets containing RGB images with authentic distortions (CLIVE, KonIQ-10k, and SPAQ). The introduced algorithm significantly outperforms the state-of-the-art in terms of correlation with human perceptual quality ratings.

An Effective Foveated 360° Image Assessment Based on Graph Convolution Network

Virtual reality (VR) has been adopted in various fields such as entertainment, education, healthcare, and the military, due to its ability to provide an immersive experience to users. However, 360° images, one of the main components in VR systems, have bulky sizes and thus require effective transmitting and rendering solutions. One of the potential solutions is to use foveated technologies, that take advantage of the foveation feature of the human eyes. Foveated technologies can significantly reduce the data required for transmission and computation complexity in rendering. However, understanding the impact of foveated 360° images on human quality perception is still limited. This paper addresses the above problems by proposing an accurate machine-learning-based quality assessment model for foveated 360° images. The proposed model is proven to outperform the three cutting-edge machine-learning-based models, which apply deep learning techniques and 25 traditional-metric-based models (or analytical-function-based-models), which utilize analytical functions. It is also expected that our model helps to evaluate and improve 360° content streaming and rendering solutions to further reduce data sizes while ensuring user experience. Also, this model could be used as a building block to construct quality assessment methods for 360° videos, that are reserved for our future work. The source code is available at https://github.com/telagment/FoVGCN.

Human Perceptual Quality Driven Underwater Image Enhancement Framework

Underwater images suffer from severe color casts, low contrast, and blurriness, which greatly degrade the visibility and color fidelity of underwater images. Recently, numerous underwater image enhancement (UIE) algorithms have been proposed. Existing synthetic datasets-based deep learning methods employ synthetic datasets to train UIE models. However, there is a gap between synthetic datasets and real underwater images, leading to poor generalization of synthetic datasets-based UIE methods. Besides, existing real datasets-based deep learning methods largely focus on minimizing the mean squared reconstruction error between UIE results and corresponding ground-truth on the real datasets, but do not take human visual perception into account. Thus, although they achieve high PSNR between UIE results and corresponding ground-truth obtained by user study on the real datasets, they often achieve unsatisfactory perceptual quality. To address these problems, we propose a Human Perceptual Quality Driven Underwater Image Enhancement Framework (HPQ-UIEF) to achieve better results in human perceptual quality and maintain satisfactory PSNR, which is trained on a real underwater enhancement quality assessment database (UEQAB). Specifically, an Underwater Image Quality Assessment Network (UIQAN) for UIE images is first proposed to assist UIE task, in which a novel depth map prior spatial attention block (DPPAB) is embedded into UIQAN. The DPPAB can adaptively recalibrate the quality-aware feature maps and model human visual attention in a data-driven manner. Then, the UIE model is proposed, in which the UIQAN is introduced as the loss function to optimize our UIE model in the direction of perceptual metrics. Moreover, since the confidence map acquired by UIQAN can effectively reflect the sensitivity of human perceptual of local area in an UIE image, the confidence map is introduced to our UIEF to help our UIEF to perceive the perceptually important regions. Thus, the confidence map is down-sampled and then concatenated into the decoder module of the UIE model, which can further improve the perceptual quality of the UIE results. Extensive experimental results show that the proposed HPQ-UIEF outperforms state-of-the-art UIE methods qualitatively and quantitatively.

ENTROPY CODING FOR H.264 VIDEO COMPRESSION

H.264/AVC standard of the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group. The H.264/AVC standardization is now achieving signicant improvement in rate-distortion efciency for digital systems worldwide. This paper provides an overview of the technical features of H.264/AVC, describing its baseline proles, motion estimation for inter, and intra prediction. To improve the human perceptual quality of videos deblocking lter and perceptual quality assessments are described in detail.

Enhancing the correlation between the quality and intelligibility objective metrics with the subjective scores by shallow feed forward neural network for time–frequency masking speech separation algorithms

Multiple objective metrics are in use by the researchers to evaluate the performance of separation systems. In this paper, we investigate the correlation of different state-of-the-art widely used objective evaluation metrics with the subjective evaluation for different time–frequency (TF) masking based binaural speech separation algorithms and find the metrics which correlate best with the human perception of quality and intelligibility for such algorithms. We separate a speech source from a speech mixture by a speech separation algorithm, and evaluate its quality and intelligibility by using the objective evaluation metrics. Then we carry out the subjective listening test of this estimated speech source on a large number of participants. For each algorithm, we repeat this process for 10 mixtures and then find the correlation between the objective and average subjective score of all the participants by Pearson’s correlation coefficient and calculate the statistical significance of these results. We also rank the separation algorithms both by the subjective and the objective testing. The results show that none of the existing objective metrics for judging the quality and intelligibility of the speech separated by TF masking based binaural source separation algorithms correlates well with the human perception. We then use shallow feed forward (FF) classification and regression neural networks (NNs) to enhance this correlation, enabling almost 87% and 88% classification accuracy for the quality and intelligibility objective metrics respectively by using the classification NN, and 98% correlation with the average subjective quality scores by using the regression NN.

An inspired haxby brain perceptual model for facial images quality assessment

Face is the most important and most popular biometric used in many identification and verification systems. In these systems, for reducing recognition error rate, the quality of input images need to be as high as possible. Face Image Compliancy verification (FICV) is one of the most essential methods for this purpose. In this research, a brain functionality inspired model is presented for FICV using Haxby model, which is a face visual perception consistent model containing three bilateral areas for three different functionalities. As a result, contribution of this work is presenting a new model, based on human brain functionality, improving the compliancy verification of face images in FICV context. Perceptual understanding of an image is the motivation of most of the quality assessment methods, i.e., the human quality perception is considered as a gold standard and a perfect reference for recognition and quality assessment. The model presented in this work aims to make the operational process of face image quality assessment system closer to the performance of a human expert. Three basic modules have been introduced. Face structural information, for initial information encoding, is simulated by an extended Viola-Jones model. Face image quality assessment is presented by International Civil Aviation Organization (ICAO), in ICAO (ISO / IEC19794 -11) requirements’ compliancy assessment document. Like Haxby model, perception is performed through two distinct functional and neurological pathways, using Hierarchical Maximum pooling (HMAX) and Convolutional Deep Belief Networks (CDBN). Information storing and fetching for training are similar to their corresponding modules in brain. For simulating the brain decision making, the final results of two separate paths are integrated by weighting sum operator. Nine ISO / ICAO requirements were used for testing the model. The simulation results, using AR and PUT databases, shows improvements in six requirements using the proposed method, in comparison with the FICV benchmark.

Dual-Stream Interactive Networks for No-Reference Stereoscopic Image Quality Assessment.

The goal of objective stereoscopic image quality assessment (SIQA) is to predict the human perceptual quality of stereoscopic/3D images automatically and accurately. Compared with traditional 2D image quality assessment, the quality assessment of stereoscopic images is more challenging because of complex binocular vision mechanisms and multiple quality dimensions. In this paper, inspired by the hierarchical dual-stream interactive nature of the human visual system, we propose a stereoscopic image quality assessment network (StereoQA-Net) for no-reference stereoscopic image quality assessment. The proposed StereoQA-Net is an end-to-end dual-stream interactive network containing left and right view sub-networks, where the interaction of the two sub-networks exists in multiple layers. We evaluate our method on the LIVE stereoscopic image quality databases. The experimental results show that our proposed StereoQA-Net outperforms state-of-the-art algorithms on both symmetrically and asymmetrically distorted stereoscopic image pairs of various distortion types. In a more general case, the proposed StereoQA-Net can effectively predict the perceptual quality of local regions. In addition, cross-dataset experiments also demonstrate the generalization ability of our algorithm.

Impacts of Retina-Related Zones on Quality Perception of Omnidirectional Image

Virtual Reality (VR), which brings immersive experiences to viewers, has been gaining popularity in recent years. A key feature in VR systems is the use of omnidirectional content, which provides 360-degree views of scenes. In this work, we study the human quality perception of omnidirectional images, focusing on different zones surrounding the foveation point. For that purpose, an extensive subjective experiment is carried out to assess the perceptual quality of omnidirectional images with non-uniform quality. Through experimental results, the impacts of different zones are analyzed. Moreover, nineteen objective quality metrics, including foveal quality metrics, are evaluated using our database. It is quantitatively shown that the zones corresponding to the fovea and parafovea of human eyes are extremely important for quality perception, while the impacts of the other zones corresponding to the perifovea and periphery are small. Besides, the investigated metrics are found to be not effective enough to reflect the quality perceived by viewers.

Evaluating Multiexposure Fusion Using Image Information

Multiexposure fusion (MEF) refers to image fusion methods that capture high dynamic range natural scenes from a set of low dynamic range camera images. In this letter, we study the problem of designing quality assessment (QA) algorithms to estimate the perceptual quality of images generated by different MEF algorithms. We develop our quality index by evaluating individual quality maps between a given fused test image and individual over-/underexposed images at multiple scales and orientations and then combining these maps across the over-/underexposed images. Our approach works on the premise that the true undistorted reference is contained across the over-/underexposed source images. We identify this true reference based on the notion of perceived image information using natural scene statistical models. It is shown that our approach outperforms the state of the art QA algorithms in terms of correlation with human perception of quality on a publicly available MEF database.

Perceptual references for independent dimensions of speech quality as measured by electroencephalography

Quality of transmitted speech can be decomposed into three perceptual dimensions: Noisiness, coloration and discontinuity (Wältermann, Dimension-based quality modeling of transmitted speech. Springer, Berlin. doi:10.1007/978-3-642-35019-1, 2013). The purpose of the present study was to explore whether degradation of speech quality on each perceptual dimension affected the morphological and temporal characteristics of the P300 event-related brain potential (ERP) component. The P300 is composed of two subcomponents, P3a and P3b, which served as neurophysiological indicators of distinct processes in human quality perception (Polich, Neuropsychology of P300. Oxford University Press, Oxford. doi:10.1093/oxfordhb/9780195374148.013.0089, 2012; Raake and Egger, Quality and quality of experience. Springer International Publishing, Cham, pp. 11–33, 2014): While the earlier P3a reflects attentional processing after the occurrence of novel sensory events, the later P3b is associated with memory operations following the detection of task-relevant stimuli. Electroencephalography (EEG) was used to record the electrical brain activity of subjects (N = 24) performing a three-stimulus oddball task. Degraded stimuli were generated from the audio recording of a spoken word. The analysis of P3a- and P3b-related activity at electrode positions Fz, Cz and Pz provided support for the existence of different perceptual references for quality-impaired vs. high-quality stimulus contexts as well as quality degradations on single perceptual dimensions.

A content recognizability measure for image quality assessment considering the high frequency attenuating distortions

The existing image quality assessment (IQA) techniques try to estimate image distortions regardless of their destructive effects on image contents. Analyzing the subjective scores of image quality databases shows that the worst opinions belong to distortions which make the images non-recognizable. In this paper, we investigate the effects of image contents clarity on human perception of quality. We found that among the several image distortions, the high frequency attenuating (HFA) ones are noticeable distortions which influence the image content recognizability, and accordingly the image understandability. To evaluate the severity of HFA distortions, we employ the non-negative matrix factorization (NMF), which has the ability to part-based image decomposition. When this decomposition is applied on an image, the resulting factors provide the latent information regarding the image parts. We employ the statistical characteristics of NMF factors to quantify the influence of HFA distortions on image contents. Our experiments performed on popular image quality databases show that the accuracy of our proposed measure is promising.

Interview

Professor Mohamed Deriche of King Fahd University, Saudi Arabia, talks to us about the work behind the paper ‘Quantifying blur in colour images using higher order singular values’, page 1755 Professor Mohamed Deriche Digital images and videos are becoming an essential part of our quality of experience, and they provide a source of information for various social and economic aspects of society. Every minute, more than 3 million videos are viewed on YouTube, over 500,000 photos are posted online, and more than 20 million messages are exchanged on WhatsApp, many of which contain images. Unfortunately, with this substantial amount of internet traffic, and with a lack of control of content, the quality of images and videos posted suffers the most. During different processing stages (acquisition, transmission, compression, etc.), various kinds of artefacts are introduced. Among these, blur is the most commonly observed distortion, which appears due to the limitations of acquisition equipment (like out-of-of focus camera lens, low-lighting conditions, relative movement) and different processing stages before final viewing by the user. More importantly, blur affects edge information which is key to human perception of quality. In this Letter, we have introduced a no-reference blur assessment technique for colour images. The spatial and inter-channel correlations in colour images are exploited to quantify blur efficiently, rather than using the traditional luminance component only, or the individual colour channels in existing techniques. A colour image is considered as a 3rd-order tensor and is decomposed into different 2D matrices or unfoldings. The higher-order singular values are calculated for these unfoldings using conventional SVD. We show that the singular values in these unfoldings follow an exponentially decreasing curve, whereas the degree of exponent varies with the amount of blur, and hence is used as an objective quality score. The proposed blur metric was tested on distorted images from various publicly available databases, and the results validate its superiority and effectiveness compared to state-of-the-art blur assessment methods. The proposed technique could be embedded in camera sensors to provide photo-after-blur removal, and could be used in multimedia applications for best quality of experience delivery of videos to the end users. We also mentioned above adding the proposed work as a feature in different applications and systems providing multimedia content online. Quality of experience in the multimedia industry is driving technology to its best. The move towards more virtual reality applications and systems will substantially benefit from validated research work in objectively quantifying different manipulation effects on original content. Here, we are working with blur which is an important component in the spectrum of distortions, but the work can be extended to quantifying enhancement (such as retouching in the entertainment industry), or even expanding it to related applications such as image forging, watermarking, content, authentication, etc. This DSP research group within the EE Department at KFUPM is actively involved in developing new algorithms and systems for different applications in multimedia. These include: image and video quality assessment, compression, content retrieval, transmission, biometrics, crowd control, sign language recognition, etc. The work presented here has been developed in collaboration with Prof. Azeddine Beghdadi, the director of the L2TI Research Lab. at Univ. Paris 13. Special thanks and acknowledgements go to my student, Mr. Muhammad Qureshi, who implemented the algorithms developed here, tested these, and compared their performance to state-of-the-art techniques.

Learning a blind quality evaluation engine of screen content images

We in this paper investigate how to blindly predict the visual quality of a screen content image (SCI). With the popularity of multi-client and remote-controlling systems, SCIs and the relevant applications have been a hot research topic. In general, SCIs contain texts or graphics in cartoons, ebooks or captures of computer screens. As for blind quality assessment (QA) of natural scene images (NSIs), it has been well established since NSIs possess certain statistical properties. SCIs however do not have reliable statistic models so far and thus the associated blind QA task is hard to be addressed. Aiming at solving this problem, we first extract 13 perceptual-inspired features with the free energy based brain theory and structural degradation model. In order to avoid the overfitting and guarantee the independence of training and testing samples, we then collect 100,000 images and use their objective quality scores computed via a high-accuracy full-reference QA method for SCIs as labels, before learning a new blind quality measure from aforementioned 13 features to the objective quality score. Experimental results performed on a large-scale screen image quality assessment database (SIQAD) demonstrate that the proposed blind quality metric has a good correlation with human perception of quality, even superior to state-of-the-art full-, reduced- and no-reference QA algorithms.

A quantitative evaluation function for 3D tree-like structure segmentations in liver images

The analysis of vascular structure from volumetric datasets plays a crucial role in many medical applications. Many segmentation algorithms have been designed to extract the vessel features. However, to date, these algorithms have not been efficiently evaluated and a large-scale manual analysis is always impossible. In this paper, we propose a quantitative evaluation function based on connectivity, overlap volume ratio, skeleton coincidence and branches structure error to deal with this evaluation task. The function is applicable to 3D tree-like structure segmentations and does not depend on the segmentation algorithms used. The performances of this evaluation function are tested on real liver datasets. The results show that the values of evaluation function are very close to the human scoring value (standard value), and the average value of relative errors is only 7.3% over all the eight datasets, while other evaluation measurements are 20% or more. So, it provides the greatest correlation with human quality perception when compared with other evaluation measurements. Thus, it is the most suitable measure for the evaluation of 3D tree-like structure segmentations in liver images.

Novel Cross-Layer QoE-Aware Radio Resource Allocation Algorithms in Multiuser OFDMA Systems

The assurance of quality of experience (QoE) and provisioning of high throughput of the system represent the main goals of future wireless and mobile networks. This paper presents novel and practical cross-layer QoE-aware radio resource allocation (RRA) algorithms for the downlink of a heterogeneous orthogonal frequency division multiple access (OFDMA) system. The objective of the proposed algorithms is to assure the appropriate level of QoE for each user of the system by incorporating application-layer parameters and subjective human perception of quality into the RRA process. We propose two user-oriented joint subcarrier and power allocation algorithms with low complexity for real-time and interactive services. The first algorithm dynamically allocates resources by assuring the same level of QoE to all users of the system, whereas the second algorithm introduces the efficient trade-off between the user's QoE and the spectral efficiency of the system. By considering application-layer parameters and user's perception of quality, high users' QoE and explicit control of data rates can be achieved. The numerical results show that the proposed algorithms achieve significant increase in the level of QoE compared to previous works, a fair distribution of capacity among users and near to optimal solution of QoE for the OFDMA system.

The impact of network impairment on quality of experience (QoE) in H.265/HEVC video streaming

Users of modern portable consumer devices (smartphones, tablets etc.) expect ubiquitous delivery of high quality services, which fully utilise the capabilities of their devices. Video streaming is one of the most widely used yet challenging services for operators to deliver with assured service levels. This challenge is more apparent in wireless networks where bandwidth constraints and packet loss are common. The lower bandwidth requirements of High Efficiency Video Coding (HEVC) provide the potential to enable service providers to deliver high quality video streams in low-bandwidth networks; however, packet loss may result in greater damage in perceived quality given the higher compression ratio. This work considers the delivery of HEVC encoded video streams in impaired network environments and quantifies the effects of network impairment on HEVC video streaming from the perspective of the end user. HEVC encoded streams were transmitted over a test network with both wired and wireless segments that had imperfect communication channels subject to packet loss. Two different error concealment methods were employed to mitigate packet loss and overcome reference decoder robustness issues. The perceptual quality of received video was subjectively assessed by a panel of viewers. Existing subjective studies of HEVC quality have not considered the implications of network impairments. Analysis of results has quantified the effects of packet loss in HEVC on perceptual quality and provided valuable insight into the relative importance of the main factors observed to influence user perception in HEVC streaming. The outputs from this study show the relative importance and relationship between those factors that affect human perception of quality in impaired HEVC encoded video streams. The subjective analysis is supported by comparison with commonly used objective quality measurement techniques. Outputs from this work may be used in the development of quality of experience (QoE) oriented streaming applications for HEVC in loss prone networks.

Algorithmic Assessment of 3D Quality of Experience

The field of algorithmically assessing the 3D quality of experience is an extremely challenging one. In this paper a no-reference algorithm is proposed to assess the comfort associated with viewing stereo videos. The proposed measure of 3D quality of experience is shown to correlate well with human perception of quality, and extracts statistical features from disparity and disparity gradient maps. And the measure utilizes these spatial features along with motion compensated disparity differences to predict quality of experience.