Perceptual Metrics Research Articles

The Effect of Realistic Appearance of Virtual Characters in Immersive Environments - Does the Character's Personality Play a Role?

Virtual characters that appear almost photo-realistic have been shown to induce negative responses from viewers in traditional media, such as film and video games. This effect, described as the uncanny valley, is the reason why realism is often avoided when the aim is to create an appealing virtual character. In Virtual Reality, there have been few attempts to investigate this phenomenon and the implications of rendering virtual characters with high levels of realism on user enjoyment. In this paper, we conducted a large-scale experiment on over one thousand members of the public in order to gather information on how virtual characters are perceived in interactive virtual reality games. We were particularly interested in whether different render styles (realistic, cartoon, etc.) would directly influence appeal, or if a character's personality was the most important indicator of appeal. We used a number of perceptual metrics such as subjective ratings, proximity, and attribution bias in order to test our hypothesis. Our main result shows that affinity towards virtual characters is a complex interaction between the character's appearance and personality, and that realism is in fact a positive choice for virtual characters in virtual reality.

Human dissimilarity ratings of musical instrument timbre: A computational meta-analysis

Musical instrument timbre has been intensively investigated through dissimilarity rating tasks. It is now well known that audio descriptors such as attack time and spectral centroid, among others, account well for the dimensions of the timbre spaces underlying these dissimilarity ratings. Nevertheless, it remains very difficult to reproduce these perceptual judgments from distances computed on acoustical representations such as the waveform or the spectrogram. Interestingly, biologically inspired representations based on spectrotemporal modulation spectra such as spectrotemporal receptive fields (STRF) have been shown to be well-suited to reproduce human dissimilarity ratings (Patil et al., 2012). Here, we propose a meta-analysis of seven former studies on timbre spaces in light of these recently developed representations. We implemented a computational framework that optimizes the correlation between the perceptual results and distances obtained from a set of different acoustic representations, in particular through the STRF. We observed that distances computed from spectrotemporal modulation representations provide the best correlation with the perceptual results across the seven timbre spaces. Finally, we highlighted the parts of the representations contributing the most to the correlation suggesting new insights into the underlying perceptual metrics. [Supported by Canada Research Chair, NSERC (RGPIN-2015-05208, RGPAS-478121-15), (RGPIN-262808-2012), and EU MSCf (Project MIM, H2020-MSCA-IF-2014, GA no. 659).]

A perception based bit allocation for 3DTV broadcasting

Perceptually adaptive bit allocation plays vital roles in efficient rate control to achieve rate distortion optimization in terms of perceptual video coding. This paper proposes a perceptually improved bit allocation scheme by employing a more accurate distortion model inspired by a state-of-the-art perceptual video quality metric. The contributions of this paper mainly involve the following three aspects. Firstly, a perceptual rate-distortion model is developed to shape the framework of perceptual video coding. Secondly, based on the characteristics of the model, an adaptively frame-level bit allocation scheme is proposed using the rate distortion optimization technology. Meanwhile, the status of the encoder buffer is also taken into account to avoid overflow or underflow. Finally, a macroblock-level quantization parameter adjustment is presented to improve perceptual video quality. Experimental results demonstrate that the whole proposed bit allocation scheme achieves better perceptual rate-distortion performance and guarantees the accuracy of rate control well.

Proof of Concept of a Tracheoesophageal Voice Prosthesis Insufflator for Speech Production After Total Laryngectomy

Proof of Concept of a Tracheoesophageal Voice Prosthesis Insufflator for Speech Production After Total Laryngectomy

Perceptual Metrics Quality

A 3D mesh can be subjected to different types of operations, such as compression, watermarking etc. Such processes lead to geometric distortions compared to the original version. In this context, quantifying the resultant modifications to the original mesh and evaluating the perceptual quality of degraded meshes become a critical issue. The perceptual 3D meshes quality is central in several applications to preserve the visual appearance of these treatments. The used metrics results have to be well correlated to the visual perception of humans. Although there are objective metrics, they do not allow the prediction of the perceptual quality, and do not include the human visual system properties. In the current work, a comparative study between the perceptual quality assessment metrics for 3D meshes was conducted. The experimental study on subjective database published by LIRIS / EPFL was used to test and to validate the results of six metrics. The results established that the Mesh Structural Distortion Measure metric achieved superior results compared to the other metrics.

Sparsity-Based Image Error Concealment via Adaptive Dual Dictionary Learning and Regularization.

In this paper, we propose a novel sparsity-based image error concealment (EC) algorithm through adaptive dual dictionary learning and regularization. We define two feature spaces: the observed space and the latent space, corresponding to the available regions and the missing regions of image under test, respectively. We learn adaptive and complete dictionaries individually for each space, where the training data are collected via an adaptive template matching mechanism. Based on the piecewise stationarity of natural images, a local correlation model is learned to bridge the sparse representations of the aforementioned dual spaces, allowing us to transfer the knowledge of the available regions to the missing regions for EC purpose. Eventually, the EC task is formulated as a unified optimization problem, where the sparsity of both spaces and the learned correlation model are incorporated. Experimental results show that the proposed method outperforms the state-of-the-art techniques in terms of both objective and perceptual metrics.

Subjective and Objective Visual Quality Assessment of Textured 3D Meshes

Objective visual quality assessment of 3D models is a fundamental issue in computer graphics. Quality assessment metrics may allow a wide range of processes to be guided and evaluated, such as level of detail creation, compression, filtering, and so on. Most computer graphics assets are composed of geometric surfaces on which several texture images can be mapped to make the rendering more realistic. While some quality assessment metrics exist for geometric surfaces, almost no research has been conducted on the evaluation of texture-mapped 3D models. In this context, we present a new subjective study to evaluate the perceptual quality of textured meshes, based on a paired comparison protocol. We introduce both texture and geometry distortions on a set of 5 reference models to produce a database of 136 distorted models, evaluated using two rendering protocols. Based on analysis of the results, we propose two new metrics for visual quality assessment of textured mesh, as optimized linear combinations of accurate geometry and texture quality measurements. These proposed perceptual metrics outperform their counterparts in terms of correlation with human opinion. The database, along with the associated subjective scores, will be made publicly available online.

Adaptive transfer functions

Medical datasets are continuously increasing in size. Although larger models may be available for certain research purposes, in the common clinical practice the models are usually of up to $$512 \times 512 \times 2000$$512×512×2000 voxels. These resolutions exceed the capabilities of conventional GPUs, the ones usually found in the medical doctors' desktop PCs. Commercial solutions typically reduce the data by downsampling the dataset iteratively until it fits the available target specifications. The data loss reduces the visualization quality and this is not commonly compensated with other actions that might alleviate its effects. In this paper, we propose adaptive transfer functions, an algorithm that improves the transfer function in downsampled multiresolution models so that the quality of renderings is highly improved. The technique is simple and lightweight, and it is suitable, not only to visualize huge models that would not fit in a GPU, but also to render not-so-large models in mobile GPUs, which are less capable than their desktop counterparts. Moreover, it can also be used to accelerate rendering frame rates using lower levels of the multiresolution hierarchy while still maintaining high-quality results in a focus and context approach. We also show an evaluation of these results based on perceptual metrics.

Semi-Parametric Decolorization with Laplacian-based Perceptual Quality Metric

While the RGB2GRAY conversion with fixed parameters is a classical and widely used tool for image decolorization, recent studies showed that adapting weighting parameters in a two-order multivariance polynomial model has great potential to improve the conversion ability. In this paper, by viewing the two-order model as the sum of three subspaces, it is observed that the first subspace in the two-order model has the dominating importance and the second and the third subspace can be seen as refinement. Therefore, we present a semiparametric strategy to take advantage of both the RGB2GRAY and the two-order models. In the proposed method, the RGB2GRAY result on the first subspace is treated as an immediate grayed image, and then the parameters in the second and the third subspace are optimized. Experimental results show that the proposed approach is comparable to other state-of-the-art algorithms in both quantitative evaluation and visual quality, especially for images with abundant colors and patterns. This algorithm also exhibits good resistance to noise. In addition, instead of the color contrast preserving ratio using the first-order gradient for decolorization quality metric, the color contrast correlation preserving ratio utilizing the second-order gradient is calculated as a new perceptual quality metric.

On the Efficiency of Image Metrics for Evaluating the Visual Quality of 3D Models

3D meshes are deployed in a wide range of application processes (e.g., transmission, compression, simplification, watermarking and so on) which inevitably introduce geometric distortions that may alter the visual quality of the rendered data. Hence, efficient model-based perceptual metrics, operating on the geometry of the meshes being compared, have been recently introduced to control and predict these visual artifacts. However, since the 3D models are ultimately visualized on 2D screens, it seems legitimate to use images of the models (i.e., snapshots from different viewpoints) to evaluate their visual fidelity. In this work we investigate the use of image metrics to assess the visual quality of 3D models. For this goal, we conduct a wide-ranging study involving several 2D metrics, rendering algorithms, lighting conditions and pooling algorithms, as well as several mean opinion score databases. The collected data allow (1) to determine the best set of parameters to use for this image-based quality assessment approach and (2) to compare this approach to the best performing model-based metrics and determine for which use-case they are respectively adapted. We conclude by exploring several applications that illustrate the benefits of image-based quality assessment.

Object detection in video sequences by a temporal modular self-adaptive SOM

A video segmentation algorithm that takes advantage of using a background subtraction (BS) model with low learning rate (LLR) or a BS model with high learning rate (HLR) depending on the video scene dynamics is presented in this paper. These BS models are based on a neural network architecture, the self-organized map (SOM), and the algorithm is termed temporal modular self-adaptive SOM, TMSA_SOM. Depending on the type of scenario, the TMSA_SOM automatically classifies and processes each video into one of four different specialized modules based on an initial sequence analysis. This approach is convenient because unlike state-of-the-art (SoA) models, our proposed model solves different situations that may occur in the video scene (severe dynamic background, initial frames with dynamic objects, static background, stationary objects, etc.) with a specialized module. Furthermore, TMSA_SOM automatically identifies whether the scene has drastically changed (e.g., stationary objects of interest become dynamic or drastic illumination changes have occurred) and automatically detects when the scene has become stable again and uses this information to update the background model in a fast way. The proposed model was validated with three different video databases: Change Detection, BMC, and Wallflower. Findings showed a very competitive performance considering metrics commonly used in the literature to compare SoA models. TMSA_SOM also achieved the best results on two perceptual metrics, Ssim and D-Score, and obtained the best performance on the global quality measure, FSD (based on F-Measure, Ssim, and D-Score), demonstrating its robustness with different and complicated non-controlled scenarios. TMSA_SOM was also compared against SoA neural network approaches obtaining the best average performance on Re, Pr, and F-Measure.

An Objective Analysis Method for Perceptual Quality of a Virtual Bass System

Due to the physical size and frequency response constraints of miniaturized and flat panel loudspeakers, low frequency reproduction from these loudspeakers is generally limited and unsatisfactory. The virtual bass system (VBS) enhances the bass performance of such loudspeakers by tricking the human brain to perceive the fundamental frequency from its higher harmonics. Problematically, the additional harmonics from VBS can also introduce perceptual distortion and reduce the audio quality. Therefore, a reliable method to assess the quality of VBS-enhanced signals is necessary in the designing of VBS. Since subjective experiments are often time-consuming and may be inconsistent, it is desirable to develop an objective assessment method for VBS. Earlier studies only utilized some simple objective metrics, which generally do not consider the human auditory model and are unable to accurately predict the perceptual quality of VBS. In this paper, we introduce a perceptual quality-assessment method for VBS based on the model output variables (MOVs) of the ITU Recommendation ITU-R BS.1387. Suitable combinations of MOVs are selected to derive perceptual quality metrics that correlate closely to the subjective quality. A verification experiment is presented to justify the accuracy of the metrics.

Free-classification of perceptually similar speakers with dysarthria.

In this investigation, the construct of perceptual similarity was explored in the dysarthrias. Specifically, we employed an auditory free-classification task to determine whether listeners could cluster speakers by perceptual similarity, whether the clusters mapped to acoustic metrics, and whether the clusters were constrained by dysarthria subtype diagnosis. Twenty-three listeners blinded to speakers' medical and dysarthria subtype diagnoses participated. The task was to group together (drag and drop) the icons corresponding to 33 speakers with dysarthria on the basis of how similar they sounded. Cluster analysis and multidimensional scaling (MDS) modeled the perceptual dimensions underlying similarity. Acoustic metrics and perceptual judgments were used in correlation analyses to facilitate interpretation of the derived dimensions. Six clusters of similar-sounding speakers and 3 perceptual dimensions underlying similarity were revealed. The clusters of similar-sounding speakers were not constrained by dysarthria subtype diagnosis. The 3 perceptual dimensions revealed by MDS were correlated with metrics for articulation rate, intelligibility, and vocal quality, respectively. This study shows (a) feasibility of a free-classification approach for studying perceptual similarity in dysarthria, (b) correspondence between acoustic and perceptual metrics to clusters of similar-sounding speakers, and (c) similarity judgments transcended dysarthria subtype diagnosis.

Perceptual sharpness metric based on human visual system

Sharpness is an important quality attribute influencing image quality. Existing sharpness assessment models have not given sufficient consideration to the human visual system, especially the human luminance masking effect. On the basis of an isotropic local contrast model, an objective perceptual sharpness assessment model is proposed combined with the human luminance masking effect. Experimental results show that the proposed perceptual sharpness metric provides better predictions than four existing sharpness metrics and can evaluate image perceptual sharpness effectively.

Evidence for Hand-Size Constancy: The Dominant Hand as a Natural Perceptual Metric

The hand is a reliable and ecologically useful perceptual ruler that can be used to scale the sizes of close, manipulatable objects in the world in a manner similar to the way in which eye height is used to scale the heights of objects on the ground plane. Certain objects are perceived proportionally to the size of the hand, and as a result, changes in the relationship between the sizes of objects in the world and the size of the hand are attributed to changes in object size rather than hand size. To illustrate this notion, we provide evidence from several experiments showing that people perceive their dominant hand as less magnified than other body parts or objects when these items are subjected to the same degree of magnification. These findings suggest that the hand is perceived as having a more constant size and, consequently, can serve as a reliable metric with which to measure objects of commensurate size.

Methods for Compensating Contrast Effects in Information Visualization

AbstractColor, as one of the most effective visual variables, is used in many techniques to encode and group data points according to different features. Relations between features and groups appear as visual patterns in the visualization. However, optical illusions may bias the perception at the first level of the analysis process. For instance, in pixel‐based visualizations contrast effects make pixels appear brighter if surrounded by a darker area, which distorts the encoded metric quantity of the data points. Even if we are aware of these perceptual issues, our visual cognition system is not able to compensate these effects accurately. To overcome this limitation, we present a color optimization algorithm based on perceptual metrics and color perception models to reduce physiological contrast or color effects. We evaluate our technique with a user study and find that the technique doubles the accuracy of users comparing and estimating color encoded data values. Since the presented technique can be used in any application without adaption to the visualization itself, we are able to demonstrate its effectiveness on data visualizations in different domains.

Perceptual video quality assessment in H.264 video coding standard using objective modeling.

Since usage of digital video is wide spread nowadays, quality considerations have become essential, and industry demand for video quality measurement is rising. This proposal provides a method of perceptual quality assessment in H.264 standard encoder using objective modeling. For this purpose, quality impairments are calculated and a model is developed to compute the perceptual video quality metric based on no reference method. Because of the shuttle difference between the original video and the encoded video the quality of the encoded picture gets degraded, this quality difference is introduced by the encoding process like Intra and Inter prediction. The proposed model takes into account of the artifacts introduced by these spatial and temporal activities in the hybrid block based coding methods and an objective modeling of these artifacts into subjective quality estimation is proposed. The proposed model calculates the objective quality metric using subjective impairments; blockiness, blur and jerkiness compared to the existing bitrate only calculation defined in the ITU G 1070 model. The accuracy of the proposed perceptual video quality metrics is compared against popular full reference objective methods as defined by VQEG.

Wavelet domain dictionary learning-based single image superresolution

Recently sparse representations over learned dictionaries have been proven to be a very successful representation method for many image processing applications. This paper proposes a new approach for increasing the resolution from a single low-resolution image. This approach is based on learned dictionaries in the wavelet domain. The proposed method combines many desired properties of wavelet-based representations such as compactness, directionality and analysis in many scales with the flexibility of redundant sparse representations. Such an approach serves for two main purposes. First, it sparsifies the training set, and second, it allows the design of structured dictionaries. Structured dictionaries better capture intrinsic image characteristics. Furthermore, the design of multiple structured dictionaries serves to reduce the number of dictionary atoms and consequently reduces the computational complexity. Three couples of wavelet subband dictionaries are designed using the K-SVD algorithm: three for the low-resolution and three for the high-resolution wavelet subband images. The image patch size and dictionary redundancy issues are empirically investigated in this work. Extensive tests indicate that a patch size of \(6\times 6\) and a dictionary width of 216 is a good compromise between computational complexity and representation quality. The proposed algorithm is shown to be superior to the leading spatial domain sparse representation techniques both visually and quantitatively with an average PSNR increase of 1.71 dB as tested over the Kodak data set. This result is also validated in terms of SSIM as a perceptual quality metric. It is shown that the proposed approach better restores the lost high-frequency details in the three wavelet detail subbands. Furthermore, the proposed algorithm is shown to significantly reduce the dictionary learning and sparse coding computational complexity.

Image inpainting based on sparse representations with a perceptual metric

This paper presents an image inpainting method based on sparse representations optimized with respect to a perceptual metric. In the proposed method, the structural similarity (SSIM) index is utilized as a criterion to optimize the representation performance of image data. Specifically, the proposed method enables the formulation of two important procedures in the sparse representation problem, 'estimation of sparse representation coefficients’ and 'update of the dictionary’, based on the SSIM index. Then, using the generated dictionary, approximation of target patches including missing areas via the SSIM-based sparse representation becomes feasible. Consequently, image inpainting for which procedures are totally derived from the SSIM index is realized. Experimental results show that the proposed method enables successful inpainting of missing areas.

Parametric estimation of structural similarity degradation for video transmission over error‐prone networks

A parametric model to estimate the degradation of objective video quality over error-prone networks is proposed. The model estimates an expected quality degradation in terms of one of the most reliable perceptual quality metrics, structural similarities (SSIMs), for a given encoded video and network condition described by a packet loss rate. The simulation results demonstrate that the proposed model can estimate the expected SSIM degradation of H.264/ advanced video coding encoded videos with high accuracy.