Probability Integral Transformation Research Articles

Density Forecast Evaluations via a Simulation-Based Dynamic Probability Integral Transformation*

Abstract This paper presents simulation-based density forecast evaluation methods using particle filters. The simulation-based dynamic probability integral transformation or log-likelihood evaluation method is combined with the existing density forecast evaluation methods. This methodology is applicable to various density forecast models, such as log stochastic volatility models and affine jump diffusion (AJD) models, for which the probability integral transform or likelihood computation is difficult due to the presence of latent stochastic volatilities. This methodology is applied to the daily S&P 500 stock index. The empirical results show that the AJD models with jumps perform the best for out-of-sample evaluations.

Improved parameter estimation of Time Dependent Kernel Density by using Artificial Neural Networks

Time Dependent Kernel Density Estimation (TDKDE) used in modelling time-varying phenomenon requires two input parameters known as bandwidth and discount to perform. A Maximum Likelihood Estimation (MLE) procedure is commonly used to estimate these parameters in a set of data but this method has a weakness; it may not produce stable kernel estimates. In this article, a novel estimation procedure is developed using Artificial Neural Networks which eliminates this inherent issue. Moreover, evaluating the performance of the kernel estimation in terms of the uniformity of Probability Integral Transform (PIT) shows a significant improvement using the proposed method. A real-life application of TDKDE parameter estimation on NASDQ stock returns validates the flawless performance of the new technique.

Photometric redshift estimation via deep learning

Context. The need to analyze the available large synoptic multi-band surveys drives the development of new data-analysis methods. Photometric redshift estimation is one field of application where such new methods improved the results, substantially. Up to now, the vast majority of applied redshift estimation methods have utilized photometric features. Aims. We aim to develop a method to derive probabilistic photometric redshift directly from multi-band imaging data, rendering pre-classification of objects and feature extraction obsolete. Methods. A modified version of a deep convolutional network was combined with a mixture density network. The estimates are expressed as Gaussian mixture models representing the probability density functions (PDFs) in the redshift space. In addition to the traditional scores, the continuous ranked probability score (CRPS) and the probability integral transform (PIT) were applied as performance criteria. We have adopted a feature based random forest and a plain mixture density network to compare performances on experiments with data from SDSS (DR9). Results. We show that the proposed method is able to predict redshift PDFs independently from the type of source, for example galaxies, quasars or stars. Thereby the prediction performance is better than both presented reference methods and is comparable to results from the literature. Conclusions. The presented method is extremely general and allows us to solve of any kind of probabilistic regression problems based on imaging data, for example estimating metallicity or star formation rate of galaxies. This kind of methodology is tremendously important for the next generation of surveys.

A class of uniform tests for goodness-of-fit of the multivariate $$L_p$$Lp-norm spherical distributions and the $$l_p$$lp-norm symmetric distributions

In this paper we employ the conditional probability integral transformation (CPIT) method to transform a d-dimensional sample from two classes of generalized multivariate distributions into a uniform sample in the unit interval $$(0,\,1)$$ or in the unit hypercube $$[0,\,1]^{d-1}$$ ( $$d\ge 2$$ ). A class of existing uniform statistics are adopted to test the uniformity of the transformed sample. Monte Carlo studies are carried out to demonstrate the performance of the tests in controlling type I error rates and power against a selected group of alternative distributions. It is concluded that the proposed tests have satisfactory empirical performance and the CPIT method in this paper can serve as a general way to construct goodness-of-fit tests for many generalized multivariate distributions.

Multivariate global sensitivity analysis for dynamic models based on energy distance

In this paper, a new kind of multivariate global sensitivity index based on energy distance is proposed. The covariance decomposition based index has been widely used for multivariate global sensitivity analysis. However, it just considers the variance of multivariate model output and ignores the correlation between different outputs. The proposed index considers the whole probability distribution of dynamic output based on characteristic function and contains more information of uncertainty than the covariance decomposition based index. The multivariate probability integral transformation based index is an extension of the popularly used moment-independent sensitivity analysis index. Although it considers the whole probability distribution of dynamic output, it is difficult to estimate the joint cumulative distribution function of dynamic output. The proposed sensitivity index can be easily estimated, especially for models with high dimensional outputs. Compared to the classic sensitivity indices, the proposed sensitivity index can be easily used for dynamic systems and obtain reasonable results. An efficient method based on the idea of the given-data method is used to estimate the proposed sensitivity index with only one set of input-output samples. The numerical and engineering examples are employed to compare the proposed index and the covariance decomposition based index. The results show that the input variables may have different effect on the whole probability distribution and variance of dynamic model output since the proposed index and the covariance decomposition based index measure the effects of input variables on the whole distribution and variance of model output separately.

Prequential omnibus goodness-of-fit tests for stochastic processes: A numerical study

ABSTRACTThis article is a contribution to the study of an omnibus goodness-of-fit (Gof) test based on Rosenblatt Probability Integral Transform (RPIT) within Dawid's prequential framework. This Gof test is easy to use since it has a common test statistic (with apparently the same asymptotic distribution) for a wide range of stochastic models. Intensive Monte-Carlo simulations are presented to investigate the behavior of this test for several stochastic models: renewal, autoregressive (AR, ARMA, ARCH, GARCH) and Poisson processes, generalized linear models... These simulations suggest that the RPIT test could be used to test the fit of a wide range of stochastic models but it may be not powerful when compared to Gof tests specifically designed for the tested processes. It is also conjectured that this test is still appropriate for testing the Gof of any discrete-time stochastic process provided that efficient estimators are used.

A Density‐Ratio Model of Crop Yield Distributions

AbstractThis paper proposes a density ratio estimator of crop yield distributions, wherein the number of observations for individual distributions is often quite small. The density ratio approach models individual densities as distortions from a common baseline density. We introduce a probability integral transformation to the density ratio method that simplifies the modeling of distortion functions. We further present an implementation approach based on the Poisson regression, which facilitates model estimation and diagnostics. Monte Carlo simulations demonstrate good finite sample performance of the proposed method. We apply this method to estimate the corn yield distributions of ninety‐nine Iowa counties, and to calculate crop insurance premiums. Lastly, we illustrate that we can employ the proposed method to effectively identify profitable insurance policies.

Independent Component Analysis via Distance Covariance

ABSTRACT This article introduces a novel statistical framework for independent component analysis (ICA) of multivariate data. We propose methodology for estimating mutually independent components, and a versatile resampling-based procedure for inference, including misspecification testing. Independent components are estimated by combining a nonparametric probability integral transformation with a generalized nonparametric whitening method based on distance covariance that simultaneously minimizes all forms of dependence among the components. We prove the consistency of our estimator under minimal regularity conditions and detail conditions for consistency under model misspecification, all while placing assumptions on the observations directly, not on the latent components. U statistics of certain Euclidean distances between sample elements are combined to construct a test statistic for mutually independent components. The proposed measures and tests are based on both necessary and sufficient conditions for mutual independence. We demonstrate the improvements of the proposed method over several competing methods in simulation studies, and we apply the proposed ICA approach to two real examples and contrast it with principal component analysis.

Validating Dynamic Engineering Models Under Uncertainty

Validating dynamic engineering models is critically important in practical applications by assessing the agreement between simulation results and experimental observations. Though significant progresses have been made, the existing metrics lack the capability of managing uncertainty in both simulations and experiments. In addition, it is challenging to validate a dynamic model aggregately over both the time domain and a model input space with data at multiple validation sites. To overcome these difficulties, this paper presents an area-based metric to systemically handle uncertainty and validate computational models for dynamic systems over an input space by simultaneously integrating the information from multiple validation sites. To manage the complexity associated with a high-dimensional data space, eigenanalysis is performed for the time series data from simulations at each validation site to extract the important features. A truncated Karhunen–Loève (KL) expansion is then constructed to represent the responses of dynamic systems, resulting in a set of uncorrelated random coefficients with unit variance. With the development of a hierarchical data-fusion strategy, probability integral transform (PIT) is then employed to pool all the resulting random coefficients from multiple validation sites across the input space into a single aggregated metric. The dynamic model is thus validated by calculating the cumulative area difference of the cumulative density functions. The proposed model validation metric for dynamic systems is illustrated with a mathematical example, a supported beam problem with stochastic loads, and real data from the vehicle occupant-restraint system.

The Use of Conditional Probability Integral Transformation Method for Testing Accelerated Failure Time Models

This paper suggests the use of the conditional probability integral transformation (CPIT) method as a goodness of fit (GOF) technique in the field of accelerated life testing (ALT), specifically for validating the underlying distributional assumption in accelerated failure time (AFT) model. The method is based on transforming the data into independent and identically distributed (i.i.d) Uniform (0, 1) random variables and then applying the modified Watson statistic to test the uniformity of the transformed random variables. This technique is used to validate each of the exponential, Weibull and lognormal distributions' assumptions in AFT model under constant stress and complete sampling. The performance of the CPIT method is investigated via a simulation study. It is concluded that this method performs well in case of exponential and lognormal distributions. Finally, a real life example is provided to illustrate the application of the proposed procedure.

Evaluating predictive count data distributions in retail sales forecasting

Massive increases in computing power and new database architectures allow data to be stored and processed at finer and finer granularities, yielding count data time series with lower and lower counts. These series can no longer be dealt with using the approximative methods that are appropriate for continuous probability distributions. In addition, it is not sufficient to calculate point forecasts alone: we need to forecast entire (discrete) predictive distributions, particularly for supply chain forecasting and inventory control, but also for other planning processes. However, tools that are suitable for evaluating the quality of discrete predictive distributions are not commonly used in sales forecasting. We explore classical point forecast accuracy measures, explain why measures such as MAD, MASE and wMAPE are inherently unsuitable for count data, and use the randomized Probability Integral Transform (PIT) and proper scoring rules to compare the performances of multiple causal and noncausal forecasting models on two datasets of daily retail sales.

A new kind of sensitivity index for multivariate output

Mathematical and computational models with correlated multivariate output are commonly used for risk assessment and decision support in engineering. Traditional methods for sensitivity analysis of the model with scalar output fail to provide satisfactory results for this multivariate case. In this work, we introduce a new sensitivity index which looks at the influence of input uncertainty on the entire distribution of the multivariate output without reference to a specific moment of the output. The definition of the new index is based on the multivariate probability integral transformation (PIT), which can take into account both of the uncertainties and the correlations among multivariate output. The mathematical properties of the proposed sensitivity index are discussed and its differences with the sensitivity indices previously introduced in the literature are highlighted. Two numerical examples and a rotating shaft model of an aircraft wing are employed to illustrate the validity and potential benefits of the new sensitivity index.

A data-driven smooth test of symmetry

In this paper we propose a data driven smooth test of symmetry. We first transform the raw data via the probability integral transformation according to a symmetrized empirical distribution, and show that under the null hypothesis of symmetry, the transformed data has a limiting uniform distribution, reducing testing for symmetry to testing for uniformity. Employing Neyman’s smooth test of uniformity, we show that only odd-ordered orthogonal moments of the transformed data are required in constructing the test statistic. We present a standardized smooth test that is distribution-free asymptotically and derive the asymptotic behavior of the test and establish its consistency. Extension to dependent data case is discussed. We investigate the finite sample performance of the proposed tests on both homogeneous and mixed distributions (with unobserved heterogeneity). An empirical application on testing symmetry of wage adjustment process, based on heterogeneous wage contracts with different durations, is provided.

Bayesian Analysis of the Censored Regression Model with an AEPD Error Term

We present the censored regression model with the error term following the asymmetric exponential power distribution. We propose three Markov chain Monte Carlo (MCMC) algorithms: the first one uses the probability integral transformation; the second one uses a combination of the probability integral transformation and random walk draws; while the third one uses random walk draws. Using simulated data we compare the performance of the three MCMC algorithms. Then we compare the posterior means, or Bayes estimates, with maximum likelihood estimates. We estimate the stock option portion of executive compensation as an example of the empirical application.

Bayesian Analysis of the Censored Regression Model with an AEPD Error Term

We present the censored regression model with the error term following the asymmetric exponential power distribution. We propose three Markov chain Monte Carlo (MCMC) algorithms: the first one uses the probability integral transformation; the second one uses a combination of the probability integral transformation and random walk draws; while the third one uses random walk draws. Using simulated data we compare the performance of the three MCMC algorithms. Then we compare the posterior means, or Bayes estimates, with maximum likelihood estimates. We estimate the stock option portion of executive compensation as an example of the empirical application.

On Families of Distributions with Shape Parameters

SummaryUnivariate continuous distributions are one of the fundamental components on which statistical modelling, ancient and modern, frequentist and Bayesian, multi‐dimensional and complex, is based. In this article, I review and compare some of the main general techniques for providing families of typically unimodal distributions on with one or two, or possibly even three, shape parameters, controlling skewness and/or tailweight, in addition to their all‐important location and scale parameters. One important and useful family is comprised of the ‘skew‐symmetric’ distributions brought to prominence by Azzalini. As these are covered in considerable detail elsewhere in the literature, I focus more on their complements and competitors. Principal among these are distributions formed by transforming random variables, by what I call ‘transformation of scale’—including two‐piece distributions—and by probability integral transformation of non‐uniform random variables. I also treat briefly the issues of multi‐variate extension, of distributions on subsets of and of distributions on the circle. The review and comparison is not comprehensive, necessarily being selective and therefore somewhat personal. © 2014 The Authors. International Statistical Review © 2014 International Statistical Institute

Diagnostic tools for bivariate accelerated life regression models.

We propose graphical diagnostic tools to assess the fit of a bivariate accelerated lifetime regression model. Using univariate residuals for each response measurement in a pair, we assess their dependence structure via the bivariate probability integral transformation of univariate residuals, which we call V-residuals. To reduce the computational burden associated with plots of V-residuals, as well as some uncertainty associated with parameter estimation, we next develop K-residuals. We also devise adjusted V- and K-residuals to account for right censoring of any response. Via simulation studies, we examine the statistical behaviour of Q-Q plots of the estimated K-residuals, and demonstrate the potential of these plots to identify an appropriate choice of frailty distribution in bivariate modeling. We apply our proposed diagnostic tools to the Diabetic Retinopathy Study and assess the goodness of fit of various models fitted to these study data using different choices of frailty distributions combined with several baseline survivor functions.

Reliable probabilistic forecasts from an ensemble reservoir inflow forecasting system

AbstractThis paper describes a probabilistic reservoir inflow forecasting system that explicitly attempts to sample from major sources of uncertainty in the modeling chain. Uncertainty in hydrologic forecasts arises due to errors in the hydrologic models themselves, their parameterizations, and in the initial and boundary conditions (e.g., meteorological observations or forecasts) used to drive the forecasts. The Member‐to‐Member (M2M) ensemble presented herein uses individual members of a numerical weather model ensemble to drive two different distributed hydrologic models, each of which is calibrated using three different objective functions. An ensemble of deterministic hydrologic states is generated by spinning up the daily simulated state using each model and parameterization. To produce probabilistic forecasts, uncertainty models are used to fit probability distribution functions (PDF) to the bias‐corrected ensemble. The parameters of the distribution are estimated based on statistical properties of the ensemble and past verifying observations. The uncertainty model is able to produce reliable probability forecasts by matching the shape of the PDF to the shape of the empirical distribution of forecast errors. This shape is found to vary seasonally in the case‐study watershed. We present an “intelligent” adaptation to a Probability Integral Transform (PIT)‐based probability calibration scheme that relabels raw cumulative probabilities into calibrated cumulative probabilities based on recent past forecast performance. As expected, the intelligent scheme, which applies calibration corrections only when probability forecasts are deemed sufficiently unreliable, improves reliability without the inflation of ignorance exhibited in certain cases by the original PIT‐based scheme.

New validation metrics for models with multiple correlated responses

Validating models with correlated multivariate outputs involves the comparison of multiple stochastic quantities. Considering both uncertainty and correlations among multiple responses from model and physical observations imposes challenges. Existing marginal comparison methods and the hypothesis testing-based methods either ignore correlations among responses or only reach Boolean conclusions (yes or no) without accounting for the amount of discrepancy between a model and the underlying reality. A new validation metric is needed to quantitatively characterize the overall agreement of multiple responses considering correlations among responses and uncertainty in both model predictions and physical observations. In this paper, by extending the concept of “area metric” and the “u-pooling method” developed for validating a single response, we propose new model validation metrics for validating correlated multiple responses using the multivariate probability integral transformation (PIT). One new metric is the PIT area metric for validating multi-responses at a single validation site. The other is the t-pooling metric that allows for pooling observations of multiple responses observed at multiple validation sites to assess the global predictive capability. The proposed metrics have many favorable properties that are well suited for validation assessment of models with correlated responses. The two metrics are examined and compared with the direct area metric and the marginal u-pooling method respectively through numerical case studies and an engineering example to illustrate their validity and potential benefits.

Probabilistic Forecasting

A probabilistic forecast takes the form of a predictive probability distribution over future quantities or events of interest. Probabilistic forecasting aims to maximize the sharpness of the predictive distributions, subject to calibration, on the basis of the available information set. We formalize and study notions of calibration in a prediction space setting. In practice, probabilistic calibration can be checked by examining probability integral transform (PIT) histograms. Proper scoring rules such as the logarithmic score and the continuous ranked probability score serve to assess calibration and sharpness simultaneously. As a special case, consistent scoring functions provide decision-theoretically coherent tools for evaluating point forecasts. We emphasize methodological links to parametric and nonparametric distributional regression techniques, which attempt to model and to estimate conditional distribution functions; we use the context of statistically postprocessed ensemble forecasts in numerical weather prediction as an example. Throughout, we illustrate concepts and methodologies in data examples.