Nonparametric Importance Sampling Research Articles

Nonparametric importance sampling for wind turbine reliability analysis with stochastic computer models

Using aeroelastic stochastic simulations, this study presents an importance sampling method for assessing wind turbine reliability. As the size of modern wind turbines gets larger, structural reliability analysis becomes more important to prevent any catastrophic failures. At the design stage, operational data do not exist or are scarce. Therefore, aeroelastic simulation is often employed for reliability analysis. Importance sampling is one of the powerful variance reduction techniques to mitigate computational burden in stochastic simulations. In the literature, wind turbine reliability assessment with importance sampling has been studied with a single variable, wind speed. However, other atmospheric stability conditions also impose substantial stress on the turbine structure. Moreover, each environmental factor’s effect on the turbine’s load response depends on other factors. This study investigates how multiple environmental factors collectively affect the turbine reliability. Specifically, we devise a new nonparametric importance sampling method that can quantify the contributions of each environmental factor and its interactions with other factors, while avoiding computational problems and data sparsity issue arising in rare event simulation. Our wind turbine case study and numerical examples demonstrate the advantage of the proposed approach.

A new maximum entropy-based importance sampling for reliability analysis

Importance sampling can be highly efficient if a good importance sampling density is constructed. Although the parametric sampling densities centered on the design points are often good choices, the determination of the design points can be a difficult and inefficient task itself, especially for problems with multiple design points, or highly nonlinear limit state functions. This paper introduces a nonparametric importance sampling method based on the Markov chain simulation and maximum-entropy density estimation (MEDE). In the proposed method, Markov chain simulation is utilized to generate samples that distribute asymptotically to the optimal importance sampling density. A nonparametric estimation of the optimal importance sampling density is then obtained using the MEDE technique. The conventional MEDE method is difficult for multi-dimensional problems as it needs to solve a set of simultaneous nonlinear integral equations. This paper developed a new MEDE technique for multivariate dataset. The method starts with using histogram to approximate a density. The multi-dimensional histogram is converted into a series of one-dimensional conditional PDFs in each dimension and the density is reconstructed by means of orthogonal expansion. Thus, the solution of MEDE is converted to a set of coefficients of the Legendre polynomials. The new importance sampling method is illustrated and compared with the classical kernel-based importance sampling using a number of numerical and structural examples.

Estimation of a launch vehicle stage fallout zone with parametric and non-parametric importance sampling algorithms in presence of uncertain input distributions

The estimation of launch vehicle fall back safety zone is a crucial problem in space application since the consequence of a mistake may be dramatic for the population. It consists in estimating the probability that the distance between the launcher stage fall-back position calculated, with a trajectory simulation code, and the predicted one is lower than a given threshold. This probability of having a distance that exceeds the critical limit is of course low and may hardly be estimated with crude Monte Carlo methods. One proposes in this paper adaptive importance sampling algorithms when some parameters of input probability density suffer from uncertainty. This situation is a difficult issue for computational reasons since a complete importance sampling procedure is necessary to estimate the rare event probability associated to each combination of input density parameters. It is no more the case with the proposed parametric and non-parametric approaches based on the definition of a more adapted initial sampling density.

Generative modeling of convolutional neural networks

The convolutional neural networks (CNNs) have proven to be a powerful tool for discriminative learning. Recently researchers have also started to show interest in the generative aspects of CNNs in order to gain a deeper understanding of what they have learned and how to further improve them. This paper investigates generative modeling of CNNs. The main contributions include: (1) We construct a generative model for the CNN in the form of exponential tilting of a reference distribution. (2) We propose a generative gradient for pre-training CNNs by a non-parametric importance sampling scheme, which is fundamentally different from the commonly used discriminative gradient, and yet has the same computational architecture and cost as the latter. (3) We propose a generative visualization method for the CNNs by sampling from an explicit parametric image distribution. The proposed visualization method can directly draw synthetic samples for any given node in a trained CNN by the Hamiltonian Monte Carlo (HMC) algorithm, without resorting to any extra hold-out images. Experiments on the challenging ImageNet benchmark show that the proposed generative gradient pre-training consistently helps improve the performances of CNNs, and the proposed generative visualization method generates meaningful and varied samples of synthetic images from a large-scale deep CNN.

Quantifying Precipitation Uncertainty for Land Data Assimilation Applications

Abstract Ensemble-based data assimilation techniques are often applied to land surface models in order to estimate components of terrestrial water and energy balance. Precipitation forcing uncertainty is the principal source of spread among the ensembles that is required for utilizing information in observations to correct model priors. Precipitation fields may have both position and magnitude errors. However, current uncertainty characterizations of precipitation forcing in land data assimilation systems often do no more than applying multiplicative errors to precipitation fields. In this paper, an ensemble-based Bayesian method for characterization of uncertainties associated with precipitation retrievals from spaceborne instruments is introduced. This method is used to produce stochastic replicates of precipitation fields that are conditioned on precipitation observations. Unlike previous studies, the error likelihood is derived using an archive of historical measurements. The ensemble replicates are generated using a stochastic method, and they are intermittent in space and time. The replicates are first projected in a low-dimension subspace using a problem-specific set of attributes. The attributes are derived using a dimensionality reduction scheme that takes advantage of singular value decomposition. A nonparametric importance sampling technique is formulated in terms of the attribute vectors to solve the Bayesian sampling problem. Examples are presented using retrievals from operational passive microwave instruments, and performance of the method is assessed using ground validation measurements from a surface weather radar network. Results indicate that this ensemble characterization approach provides a useful description of precipitation uncertainties with a posterior ensemble that is narrower in distribution than its prior while containing both precipitation position and magnitude errors.

Importance Sampling in Stochastic Programming: A Markov Chain Monte Carlo Approach

Stochastic programming models are large-scale optimization problems that are used to facilitate decision making under uncertainty. Optimization algorithms for such problems need to evaluate the expected future costs of current decisions, often referred to as the recourse function. In practice, this calculation is computationally difficult as it requires the evaluation of a multidimensional integral whose integrand is an optimization problem. In turn, the recourse function has to be estimated using techniques such as scenario trees or Monte Carlo methods, both of which require numerous functional evaluations to produce accurate results for large-scale problems with multiple periods and high-dimensional uncertainty. In this work, we introduce an importance sampling framework for stochastic programming that can produce accurate estimates of the recourse function using a small number of samples. Our framework combines Markov chain Monte Carlo methods with kernel density estimation algorithms to build a nonparametric importance sampling distribution, which can then be used to produce a lower-variance estimate of the recourse function. We demonstrate the increased accuracy and efficiency of our approach using variants of well-known multistage stochastic programming problems. Our numerical results show that our framework produces more accurate estimates of the optimal value of stochastic programming models, especially for problems with moderate variance, multimodal, or rare-event distributions.

Nonparametric particle filtering and smoothing with quasi-Monte Carlo sampling

Sequential Monte Carlo methods (also known as particle filters and smoothers) are used for filtering and smoothing in general state-space models. These methods are based on importance sampling. In practice, it is often difficult to find a suitable proposal which allows effective importance sampling. This article develops an original particle filter and an original particle smoother which employ nonparametric importance sampling. The basic idea is to use a nonparametric estimate of the marginally optimal proposal. The proposed algorithms provide a better approximation of the filtering and smoothing distributions than standard methods. The methods’ advantage is most distinct in severely nonlinear situations. In contrast to most existing methods, they allow the use of quasi-Monte Carlo (QMC) sampling. In addition, they do not suffer from weight degeneration rendering a resampling step unnecessary. For the estimation of model parameters, an efficient on-line maximum-likelihood (ML) estimation technique is proposed which is also based on nonparametric approximations. All suggested algorithms have almost linear complexity for low-dimensional state-spaces. This is an advantage over standard smoothing and ML procedures. Particularly, all existing sequential Monte Carlo methods that incorporate QMC sampling have quadratic complexity. As an application, stochastic volatility estimation for high-frequency financial data is considered, which is of great importance in practice. The computer code is partly available as supplemental material.

Non-parametric partial importance sampling for financial derivative pricing

Importance sampling is a promising variance reduction technique for Monte Carlo simulation-based derivative pricing. Existing importance sampling methods are based on a parametric choice of the proposal. This article proposes an algorithm that estimates the optimal proposal non-parametrically using a multivariate frequency polygon estimator. In contrast to parametric methods, non-parametric estimation allows for close approximation of the optimal proposal. Standard non-parametric importance sampling is inefficient for high-dimensional problems. We solve this issue by applying the procedure to a low-dimensional subspace, which is identified through principal component analysis and the concept of the effective dimension. The mean square error properties of the algorithm are investigated and its asymptotic optimality is shown. Quasi-Monte Carlo is used for further improvement of the method. It is easy to implement, particularly it does not require any analytical computation, and it is computationally very efficient. We demonstrate through path-dependent and multi-asset option pricing problems that the algorithm leads to significant efficiency gains compared with other algorithms in the literature.

Computationally Efficient Nonparametric Importance Sampling

The variance reduction established by importance sampling strongly depends on the choice of the importance sampling distribution. A good choice is often hard to achieve especially for high-dimensional integration problems. Nonparametric estimation of the optimal importance sampling distribution (known as “nonparametric importance sampling”) is a reasonable alternative to parametric approaches. In this article, nonparametric variants of both the self-normalized and the unnormalized importance sampling estimator are proposed and investigated. A common critique of nonparametric importance sampling is the increased computational burden compared with parametric methods. We solve this problem to a large degree by utilizing the linear blend frequency polygon estimator instead of a kernel estimator. Mean square error convergence properties are investigated, leading to recommendations for the efficient application of nonparametric importance sampling. Particularly, we show that nonparametric importance sampling asymptotically attains optimal importance sampling variance. The efficiency of nonparametric importance sampling algorithms relies heavily on the computational efficiency of the nonparametric estimator used. The linear blend frequency polygon outperforms kernel estimators in terms of certain criteria such as efficient sampling and evaluation. Furthermore, it is compatible with the inversion method for sample generation. This allows one to combine nonparametric importance sampling with other variance reduction techniques such as stratified sampling. Empirical evidence for the usefulness of the suggested algorithms is obtained by means of three benchmark integration problems. We show empirically that these methods may work in higher dimensions, at least up to dimension eight. As an application, we estimate the distribution of the queue length of a spam filter queuing system based on real data.