While the RGB2GRAY conversion with fixed parameters is a classical and widely used tool for image decolorization, recent studies showed that adapting weighting parameters in a two-order multivariance polynomial model has great potential to improve the conversion ability. In this paper, by viewing the two-order model as the sum of three subspaces, it is observed that the first subspace in the two-order model has the dominating importance and the second and the third subspace can be seen as refinement. Therefore, we present a semiparametric strategy to take advantage of both the RGB2GRAY and the two-order models. In the proposed method, the RGB2GRAY result on the first subspace is treated as an immediate grayed image, and then the parameters in the second and the third subspace are optimized. Experimental results show that the proposed approach is comparable to other state-of-the-art algorithms in both quantitative evaluation and visual quality, especially for images with abundant colors and patterns. This algorithm also exhibits good resistance to noise. In addition, instead of the color contrast preserving ratio using the first-order gradient for decolorization quality metric, the color contrast correlation preserving ratio utilizing the second-order gradient is calculated as a new perceptual quality metric.

3D meshes are deployed in a wide range of application processes (e.g., transmission, compression, simplification, watermarking and so on) which inevitably introduce geometric distortions that may alter the visual quality of the rendered data. Hence, efficient model-based perceptual metrics, operating on the geometry of the meshes being compared, have been recently introduced to control and predict these visual artifacts. However, since the 3D models are ultimately visualized on 2D screens, it seems legitimate to use images of the models (i.e., snapshots from different viewpoints) to evaluate their visual fidelity. In this work we investigate the use of image metrics to assess the visual quality of 3D models. For this goal, we conduct a wide-ranging study involving several 2D metrics, rendering algorithms, lighting conditions and pooling algorithms, as well as several mean opinion score databases. The collected data allow (1) to determine the best set of parameters to use for this image-based quality assessment approach and (2) to compare this approach to the best performing model-based metrics and determine for which use-case they are respectively adapted. We conclude by exploring several applications that illustrate the benefits of image-based quality assessment.

A video segmentation algorithm that takes advantage of using a background subtraction (BS) model with low learning rate (LLR) or a BS model with high learning rate (HLR) depending on the video scene dynamics is presented in this paper. These BS models are based on a neural network architecture, the self-organized map (SOM), and the algorithm is termed temporal modular self-adaptive SOM, TMSA_SOM. Depending on the type of scenario, the TMSA_SOM automatically classifies and processes each video into one of four different specialized modules based on an initial sequence analysis. This approach is convenient because unlike state-of-the-art (SoA) models, our proposed model solves different situations that may occur in the video scene (severe dynamic background, initial frames with dynamic objects, static background, stationary objects, etc.) with a specialized module. Furthermore, TMSA_SOM automatically identifies whether the scene has drastically changed (e.g., stationary objects of interest become dynamic or drastic illumination changes have occurred) and automatically detects when the scene has become stable again and uses this information to update the background model in a fast way. The proposed model was validated with three different video databases: Change Detection, BMC, and Wallflower. Findings showed a very competitive performance considering metrics commonly used in the literature to compare SoA models. TMSA_SOM also achieved the best results on two perceptual metrics, Ssim and D-Score, and obtained the best performance on the global quality measure, FSD (based on F-Measure, Ssim, and D-Score), demonstrating its robustness with different and complicated non-controlled scenarios. TMSA_SOM was also compared against SoA neural network approaches obtaining the best average performance on Re, Pr, and F-Measure.

Due to the physical size and frequency response constraints of miniaturized and flat panel loudspeakers, low frequency reproduction from these loudspeakers is generally limited and unsatisfactory. The virtual bass system (VBS) enhances the bass performance of such loudspeakers by tricking the human brain to perceive the fundamental frequency from its higher harmonics. Problematically, the additional harmonics from VBS can also introduce perceptual distortion and reduce the audio quality. Therefore, a reliable method to assess the quality of VBS-enhanced signals is necessary in the designing of VBS. Since subjective experiments are often time-consuming and may be inconsistent, it is desirable to develop an objective assessment method for VBS. Earlier studies only utilized some simple objective metrics, which generally do not consider the human auditory model and are unable to accurately predict the perceptual quality of VBS. In this paper, we introduce a perceptual quality-assessment method for VBS based on the model output variables (MOVs) of the ITU Recommendation ITU-R BS.1387. Suitable combinations of MOVs are selected to derive perceptual quality metrics that correlate closely to the subjective quality. A verification experiment is presented to justify the accuracy of the metrics.

In this investigation, the construct of perceptual similarity was explored in the dysarthrias. Specifically, we employed an auditory free-classification task to determine whether listeners could cluster speakers by perceptual similarity, whether the clusters mapped to acoustic metrics, and whether the clusters were constrained by dysarthria subtype diagnosis. Twenty-three listeners blinded to speakers' medical and dysarthria subtype diagnoses participated. The task was to group together (drag and drop) the icons corresponding to 33 speakers with dysarthria on the basis of how similar they sounded. Cluster analysis and multidimensional scaling (MDS) modeled the perceptual dimensions underlying similarity. Acoustic metrics and perceptual judgments were used in correlation analyses to facilitate interpretation of the derived dimensions. Six clusters of similar-sounding speakers and 3 perceptual dimensions underlying similarity were revealed. The clusters of similar-sounding speakers were not constrained by dysarthria subtype diagnosis. The 3 perceptual dimensions revealed by MDS were correlated with metrics for articulation rate, intelligibility, and vocal quality, respectively. This study shows (a) feasibility of a free-classification approach for studying perceptual similarity in dysarthria, (b) correspondence between acoustic and perceptual metrics to clusters of similar-sounding speakers, and (c) similarity judgments transcended dysarthria subtype diagnosis.

Sharpness is an important quality attribute influencing image quality. Existing sharpness assessment models have not given sufficient consideration to the human visual system, especially the human luminance masking effect. On the basis of an isotropic local contrast model, an objective perceptual sharpness assessment model is proposed combined with the human luminance masking effect. Experimental results show that the proposed perceptual sharpness metric provides better predictions than four existing sharpness metrics and can evaluate image perceptual sharpness effectively.

The hand is a reliable and ecologically useful perceptual ruler that can be used to scale the sizes of close, manipulatable objects in the world in a manner similar to the way in which eye height is used to scale the heights of objects on the ground plane. Certain objects are perceived proportionally to the size of the hand, and as a result, changes in the relationship between the sizes of objects in the world and the size of the hand are attributed to changes in object size rather than hand size. To illustrate this notion, we provide evidence from several experiments showing that people perceive their dominant hand as less magnified than other body parts or objects when these items are subjected to the same degree of magnification. These findings suggest that the hand is perceived as having a more constant size and, consequently, can serve as a reliable metric with which to measure objects of commensurate size.

Abstract Color, as one of the most effective visual variables, is used in many techniques to encode and group data points according to different features. Relations between features and groups appear as visual patterns in the visualization. However, optical illusions may bias the perception at the first level of the analysis process. For instance, in pixel‐based visualizations contrast effects make pixels appear brighter if surrounded by a darker area, which distorts the encoded metric quantity of the data points. Even if we are aware of these perceptual issues, our visual cognition system is not able to compensate these effects accurately. To overcome this limitation, we present a color optimization algorithm based on perceptual metrics and color perception models to reduce physiological contrast or color effects. We evaluate our technique with a user study and find that the technique doubles the accuracy of users comparing and estimating color encoded data values. Since the presented technique can be used in any application without adaption to the visualization itself, we are able to demonstrate its effectiveness on data visualizations in different domains.

Recently sparse representations over learned dictionaries have been proven to be a very successful representation method for many image processing applications. This paper proposes a new approach for increasing the resolution from a single low-resolution image. This approach is based on learned dictionaries in the wavelet domain. The proposed method combines many desired properties of wavelet-based representations such as compactness, directionality and analysis in many scales with the flexibility of redundant sparse representations. Such an approach serves for two main purposes. First, it sparsifies the training set, and second, it allows the design of structured dictionaries. Structured dictionaries better capture intrinsic image characteristics. Furthermore, the design of multiple structured dictionaries serves to reduce the number of dictionary atoms and consequently reduces the computational complexity. Three couples of wavelet subband dictionaries are designed using the K-SVD algorithm: three for the low-resolution and three for the high-resolution wavelet subband images. The image patch size and dictionary redundancy issues are empirically investigated in this work. Extensive tests indicate that a patch size of \(6\times 6\) and a dictionary width of 216 is a good compromise between computational complexity and representation quality. The proposed algorithm is shown to be superior to the leading spatial domain sparse representation techniques both visually and quantitatively with an average PSNR increase of 1.71 dB as tested over the Kodak data set. This result is also validated in terms of SSIM as a perceptual quality metric. It is shown that the proposed approach better restores the lost high-frequency details in the three wavelet detail subbands. Furthermore, the proposed algorithm is shown to significantly reduce the dictionary learning and sparse coding computational complexity.

This paper presents an image inpainting method based on sparse representations optimized with respect to a perceptual metric. In the proposed method, the structural similarity (SSIM) index is utilized as a criterion to optimize the representation performance of image data. Specifically, the proposed method enables the formulation of two important procedures in the sparse representation problem, 'estimation of sparse representation coefficients’ and 'update of the dictionary’, based on the SSIM index. Then, using the generated dictionary, approximation of target patches including missing areas via the SSIM-based sparse representation becomes feasible. Consequently, image inpainting for which procedures are totally derived from the SSIM index is realized. Experimental results show that the proposed method enables successful inpainting of missing areas.

Abstract Almost all mesh processing procedures cause some more or less visible changes in the appearance of objects represented by polygonal meshes. In many cases, such as mesh watermarking, simplification or lossy compression, the objective is to make the change in appearance negligible, or as small as possible, given some other constraints. Measuring the amount of distortion requires taking into account the final purpose of the data. In many applications, the final consumer of the data is a human observer, and therefore the perceptibility of the introduced appearance change by a human observer should be the criterion that is taken into account when designing and configuring the processing algorithms. In this review, we discuss the existing comparison metrics for static and dynamic (animated) triangle meshes. We describe the concepts used in perception‐oriented metrics used for 2D image comparison, and we show how these concepts are employed in existing 3D mesh metrics. We describe the character of subjective data used for evaluation of mesh metrics and provide comparison results identifying the advantages and drawbacks of each method. Finally, we also discuss employing the perception‐correlated metrics in perception‐oriented mesh processing algorithms.

Centrally-mediated sensory information processing is impacted with increased alcohol consumption in college-aged individuals

We present an information-theoretic method to measure the similarity between a given set of observed, real-world data and visual simulation technique for aggregate crowd motions of a complex system consisting of many individual agents. This metric uses a two-step process to quantify a simulator's ability to reproduce the collective behaviors of the whole system, as observed in the recorded real-world data. First, Bayesian inference is used to estimate the simulation states which best correspond to the observed data, then a maximum likelihood estimator is used to approximate the prediction errors. This process is iterated using the EM-algorithm to produce a robust, statistical estimate of the magnitude of the prediction error as measured by its entropy (smaller is better). This metric serves as a simulator-to-data similarity measurement. We evaluated the metric in terms of robustness to sensor noise, consistency across different datasets and simulation methods, and correlation to perceptual metrics.

Natural scenes have an intricate statistical structure, including correlations of low and high orders that covary in a complex way. Thus, while the function of the visual system is best understood in the context of its natural inputs, it can be difficult to move from experiments that study responses to naturalistic inputs, to computational models that analyze how these responses arise. This motivated us to develop stimulus sets that abstract the statistics in natural scenes, and enable testing their effects individually and in combination. To reduce the dimensionality of the problem, we focus on binarized images. Following the findings of Tkacik et al. (2010), we restrict consideration to configurations of pixels in 2x2 neighborhoods, as this typifies the informative local image statistics. The 16 possible configurations within these neighborhoods are completely described by 10 image statistics, which thus form the coordinates of a perceptual space. We use a 4-AFC segmentation task to characterize human visual sensitivity to these 10 image statistics, alone and in combination. Results show that sensitivity to individual statistics is highly consistent across N=12 subjects (including naïve and experienced observers), as is sensitivity to pairwise interactions (N=6). In 4 subjects, we determined the perceptual metric in the entire 10-dimensional space. The metric is very close (~5% RMSE) to Euclidean. Moreover, the deviations from a Euclidean metric, though small, are conserved across subjects. These deviations are of two types: (i) an asymmetry between sensitivity to positive and negative variations of even-order statistics, and (ii) a unique interaction between specific pairwise statistics. (i) may reflect interactions across spatial scale, and (ii) may reflect a special role for corners. In sum, the informative statistics of natural images map out an orderly perceptual space, with simple, conserved rules for how individual statistics of low and high order combine. Meeting abstract presented at VSS 2012

Local image statistics have a perceptual metric that is nearly Euclidean

Delivering high perceptual quality video over wireless channels is challenging due to the changing channel quality and the variations in the importance of one source packet to the next for the end-user's perceptual experience. Leveraging perceptual metrics in concert with link adaptation to maximize perceptual quality and satisfy real-time delay constraints is largely unexplored. We introduce an APP/MAC/PHY cross-layer architecture that enables optimizing perceptual quality for delay-constrained scalable video transmission. We propose an online QoS-to-QoE mapping technique to quantify the loss visibility of packets from each video layer using the ACK history and perceptual metrics. At the PHY layer, we develop a link adaptation technique that uses the QoS-to-QoE mapping to provide perceptually-optimized unequal error protection per layer according to packet loss visibility. At the APP layer, the source rate is adapted by selecting the set of temporal and quality layers to be transmitted based on the channel statistics, source rates, and playback buffer state. The proposed cross-layer optimization framework allows the channel to adapt at a faster time scale than the video codec. Furthermore, it provides a tradeoff between playback buffer occupancy and perceptual quality. We show that the proposed architecture prevents playback buffer starvation, provides immunity against short-term channel fluctuations, regulates the buffer size, and achieves a 30% increase in video capacity versus throughput-optimal link adaptation.

Abstract The field of computational photography, and in particular the design and implementation of coded apertures, has yielded impressive results in the last years. In this paper we introduce perceptually optimized coded apertures for defocused deblurring. We obtain near‐optimal apertures by means of optimization, with a novel evaluation function that includes two existing image quality perceptual metrics. These metrics favour results where errors in the final deblurred images will not be perceived by a human observer. Our work improves the results obtained with a similar approach that only takes into account the L2 metric in the evaluation function.

Binaural impulse response measurements have been made at multiple locations within 20 unoccupied elementary school classrooms in a public school district in Nebraska, USA. Assorted objective metrics have been calculated from these binaural impulse responses, including speech transmission index, distortion of frequency-smoothed magnitude, interaural cross-correlations, and interaural level differences. This presentation highlights the results of these measurements within each classroom and between classrooms. These metrics have been correlated to student scores on standardized achievement tests, obtained as averages for each classroom. One interesting finding is that the distortion of frequency-smoothed magnitude was found to be significantly related to student achievement scores in the language subject areas, even though classroom reverberation times were not, due to the limited range of reverberation times across this investigation.

Objective quality assessment plays a very important role in the video applications, as they promise the means to evaluate the performance of acquisition, display, coding and communication systems. Objective video quality assessment still has a long way to go before it reaches the level of success. Perception video sequences quality metrics are of great potential benefit to the video industry. Many researchers have focused on developing digital video sequences quality metrics which produce results that accurately emulate subjective responses. However, to be widely applicable a metric must also work over a wide range of quality, and be useful for in-service quality monitoring. In this paper, we propose novel quality metrics for video sequences. The temporal correlations of video frames and the visual interest feature are considered in this method. Meanwhile the metrics are capable of capturing spatial distortions in video sequences. And they can quantify the spatial distortions and differentiate the type of distortion. Furthermore the metrics correlate well with the subjective quality measures because perception distortions of human have been taken into consideration. Experimental results show that our perceptual quality metrics performs better than the existing methods.

Graphics on mobile devices is becoming popular because untethered computing is convenient and makes workers more productive. Mobile displays have a wide range of resolutions that affect the scene Level-of-Detail (LoD) that users can perceive: smaller displays show less detail, therefore lower resolution meshes and textures are acceptable. Mobile devices frequently have limited battery energy, low memory and disk space. To minimize wasting limited system resources, the authors render mobile graphics scenes at the lowest LoD at which users do not perceive distortion due to simplification. This is called LoD the Point of Imperceptibility (PoI). Increasing the mesh or texture resolution beyond the PoI wastes valuable system resources without increasing perceivable visual realism. The authors propose a perceptual metric that can easily be evaluated to identify the LoD corresponding to a target mobile display’s PoI and accounts for object geometry, lighting and shading. Previous work did not directly compute changes in the PoI due to target screen resolution. The perceptual metric generates a screen-dependent Pareto distribution with a knee point that corresponds to the PoI. We employ wavelets for simplification, which gives direct access to the mesh undulation frequency that we then use to parameterize the CSF curve.