Approximate Probability Density Research Articles

Approximate probability density function for nonlinear surging in irregular following seas

Approximate probability density function for nonlinear surging in irregular following seas

Mixtures of multivariate Gaussians

This article discusses the approximation of probability densities by mixtures of Gaussian densities. The Kullback-Leibler divergence is used as a measure between densities, followed by applications of the EM algorithm. The conditions under which we study these questions are motivated by approximations introduced in non-linear Kalman-type filtering.

Exponential expansions for approximation of probability distributions

AbstractThis work analyses expansions of exponential form for approximating probability density functions, through the utilization of diverse orthogonal polynomial bases. Notably, exponential expansions ensure the maintenance of positive probabilities regardless of the degree of skewness and kurtosis inherent in the true density function. In particular, we introduce novel findings concerning the convergence of this series towards the true density function, employing mathematical tools of functional statistics. In particular, we show that the exponential expansion is a Fourier series of the true probability with respect to a given orthonormal basis of the so called Bayesian Hilbert space. Furthermore, we present a numerical technique for estimating the coefficients of the expansion, based on the first n exact moments of the corresponding true distribution. Finally, we provide numerical examples that effectively demonstrate the efficiency and straightforward implementability of our proposed approach.

Response of Gaussian white noise excited oscillators with inertia nonlinearity based on the RBFNN method

Although stochastic averaging methods have proven effective in solving the responses of nonlinear oscillators with a strong stiffness term under broadband noise excitations, these methods appear to be ineffective when dealing with oscillators that have a strong inertial nonlinearity term (also known as coordinate-dependent mass) or multiple potential wells. To address this limitation, a radial basis function neural network (RBFNN) algorithm is applied to predict the responses of oscillators with both a strong inertia nonlinearity term and multiple potential wells. The well-known Gaussian functions are chosen as radial basis functions in the model. Then, the approximate stationary probability density function (PDF) is expressed as the sum of Gaussian basis functions (GBFs) with weights. The squared error of the approximate solution for the Fokker-Plank-Kolmogorov (FPK) function is minimized using the Lagrange multiplier method to determine optimal weight coefficients. Three examples are presented to demonstrate how inertia nonlinearity terms and potential wells affect the responses. The mean square errors between Monte Carlo simulations (MCS) and RBFNN predictions are provided. The results indicate that RBFNN predictions align perfectly with those obtained from MCS.

Choice of Approximation Bases Used in Computational Functional Algorithms for Approximating Probability Densities on the Basis of Given Sample

Choice of Approximation Bases Used in Computational Functional Algorithms for Approximating Probability Densities on the Basis of Given Sample

A method for assessing the characteristics of the virtual machine migration process, taking into account the type of migrations and applications

Dynamic resource allocation in cloud computing is a critical aspect. It is necessary for computing resources scaling to maintain application performance, reduce costs and ensure the reliability of IT systems. Dynamic resource allocation is possible using open virtual machines (VM), but live migration is a relatively expensive and resource-intensive operation. A number of VM migration algorithms have been proposed, each with different performance characteristics depending on the state of the host system, network, and the VM workload. Recently, a large number of works have emerged that have developed models for assessing the effectiveness of of VM migration, but many of them do not provide a standard of satisfactory prediction accuracy and are built for a single migration algorithm, which limits the use of data models. This work uses a statistical approach to estimate the total migration time and VM downtime based on the approximation of probability densities by Gram-Charlier and Laguerre Series. The proposed method, in comparison with the known ones, allows one to answer the question of what is the probability of total migration time and VM downtime for a given migration type and VM application. The analysis of total migration time and VM downtime is based on the Virtual Machine Live Migration Dataset includes more than 40 000 virtual machine migration records with five different live migration algorithms: post-copy migration, pre-copy migration and its modifications: CPU throttling delta compression, and data compression. The dataset includes approximately 8000 migration records of each migration algorithm with 9 types of workloads. Analysis of the results allows us to conclude that the type of application significantly influences both the shape of the empirical distribution (histograms) of total migration time and its characteristics, therefore it is advisable to obtain an analytical expression for the distribution law of the total migration time and VM downtime taking into account these circumstances. The results of the experiments shows that this method does not depend on the migration algorithm and the type of working application. The use of the Laguerre series can be recommended as giving more reliable results compared to Gram-Charlier series. A method for estimating total migration time and VM downtime can be integrated into a management system to select the best monitoring window for servers and evaluate service-level agreement terms.

A scaled kernel density estimation prior for dynamic borrowing of historical information with application to clinical trial design.

Incorporating historical data into a current data analysis can improve estimation of parameters shared across both datasets and increase the power to detect associations of interest while reducing the time and cost of new data collection. Several methods for prior distribution elicitation have been introduced to allow for the data-driven borrowing of historical information within a Bayesian analysis of the current data. We propose scaled Gaussian kernel density estimation (SGKDE) prior distributions as potentially more flexible alternatives. SGKDE priors directly use posterior samples collected from a historical data analysis to approximate probability density functions, whose variances depend on the degree of similarity between the historical and current datasets, which are used as prior distributions in the current data analysis. We compare the performances of the SGKDE priors with some existing approaches using a simulation study. Data from a recently completed phase III clinical trial of a maternal vaccine for respiratory syncytial virus are used to further explore the properties of SGKDE priors when designing a new clinical trial while incorporating historical data. Overall, both studies suggest that the new approach results in improved parameter estimation and power in the current data analysis compared to the considered existing methods.

Data-based approach for time-correlated closures of turbulence models.

Developed turbulent motion of fluid still lacks an analytical description despite more than a century of active research. Nowadays, phenomenological ideas are widely used in practical applications, such as small-scale closures for numerical simulations of turbulent flows. In the present paper, we use a shell model of turbulence to construct a closure intended to have a solid theoretical background and to capture intrinsic probabilistic features of turbulence. Shell models of turbulence are dynamical deterministic systems used to model energy cascade and other key aspects of the Navier-Stokes such as intermittency. We rescale the variables of the Sabra model in a way which leads to hidden symmetries and universal distributions. We then use such fine distributions to write closures, i.e., missing expressions for some of the Sabra variables. Our closures rely on approximating probability density functions using a Gaussian mixture model, which makes them probabilistic by nature and allows us to write time-correlated closures. We also provide a framework where other machine learning tools can be employed with reduced black-box aspects.

USE OF ARTIFICAL INTELLIGENCE MODELS IN ACCEPTANCE TESTS

This paper describes the application of fuzzy perсeptron in acceptance testing problems. A study on the use of various fuzzy functions in solving the problem of approximation of probability densities was carried out. Experiments were conducted on time series with preset distribution laws, evaluating approximation quality using standard deviation in the series of 29 tests. As a result of the research, using Gaussian fuzzy function, the mean value of standard deviation equal to 0.00145 determined which confirms good approximative ability of the fuzzy perсeptron. The system was tested on the acceptance test data of the gas compressor engine. In this case, approximation quality was assessed using the Kolmogorov-Smirnov agreement criterion. Analysis results confirm the high approximative ability of the fuzzy perсeptron, emphasizing its applicability in real acceptance tests.

Stability and Convergence of Solutions to Stochastic Inverse Problems Using Approximate Probability Densities

Stability and Convergence of Solutions to Stochastic Inverse Problems Using Approximate Probability Densities

Probabilistic assessment of seismic earth pressure against backfills with a nonlinear failure criterion

A novel approach for the probabilistic assessment of seismic earth pressure against nonlinear backfills is introduced in this study. To implement reliability-based design for the retaining structures under seismic loading condition, nonlinear upper bound analysis is adopted to obtain the seismic earth pressure through optimization procedure. Firstly, rigorous analytical function is formulated to reveal the normal-shear stress information along the failure surface in soil with nonlinear failure criterion. Subsequently, a kinematic equation is put forward to estimate the seismic earth pressures by balancing the external work rate and the internal energy dissipation rate. The solutions are verified by the Discontinuity Layout Optimization numerical modellings based on the derived stress data from the proposed method. To consider the probability analysis of nonlinear backfill properties, the moment method is presented by combining the adaptive dimension decomposition with the direct integral method. According to the estimated first four statistical moments, the approximated probability density function of the performance function is determined contently. Finally, the failure probability of seismic earth pressure is calculated based on the proposed moment method. Two numerical examples demonstrate that the moment method can be adapted to the characteristics of nonlinear backfills and further improve the accuracy of reliability estimation by introducing higher-order moments analytically.

A computationally efficient Bayesian approach to full‐waveform inversion

AbstractConventional methods solve the full‐waveform inversion making use of gradient‐based algorithms to minimize an error function, which commonly measure the Euclidean distance between observed and predicted waveforms. This deterministic approach only provides a ‘best‐fitting’ model and cannot account for the uncertainties affecting the predicted solution. Local methods are also usually prone to get trapped into local minima of the error function. On the other hand, casting this inverse problem into a probabilistic framework has to deal with the formidable computational effort of the Bayesian approach when applied to non‐linear problems with expensive forward evaluations and large model spaces. We present a gradient‐based Markov Chain Monte Carlo full‐waveform inversion in which the posterior sampling is accelerated by compressing the data and model spaces through the discrete cosine transform, and by also defining a proposal that is a local, Gaussian approximation of the target posterior probability density. This proposal is constructed using the local Hessian and gradient informations of the log posterior, which are made computationally manageable thanks to the compression of the data and model spaces. We demonstrate the applicability of the approach by performing two synthetic inversion tests on portions of the Marmousi and BP acoustic model. In these examples, the forward modelling is performed using Devito, a finite difference domain‐specific language that solves the discretized wave equation on a Cartesian grid. For both examples, the results obtained by the implemented method are also validated against those obtained using a classic deterministic approach. Our tests illustrate the efficiency of the proposed probabilistic method, which seems quite robust against cycle‐skipping issues and also characterized by a computational cost comparable to that of the local inversion. The outcomes of the proposed probabilistic inversion can also play the role of starting models for a subsequent local inversion step aimed at improving the spatial resolution of the probabilistic result, which was limited by the model compression.

Variational Inference via Rényi Bound Optimization and Multiple-Source Adaptation.

Variational inference provides a way to approximate probability densities through optimization. It does so by optimizing an upper or a lower bound of the likelihood of the observed data (the evidence). The classic variational inference approach suggests maximizing the Evidence Lower Bound (ELBO). Recent studies proposed to optimize the variational Rényi bound (VR) and the χ upper bound. However, these estimates, which are based on the Monte Carlo (MC) approximation, either underestimate the bound or exhibit a high variance. In this work, we introduce a new upper bound, termed the Variational Rényi Log Upper bound (VRLU), which is based on the existing VR bound. In contrast to the existing VR bound, the MC approximation of the VRLU bound maintains the upper bound property. Furthermore, we devise a (sandwiched) upper-lower bound variational inference method, termed the Variational Rényi Sandwich (VRS), to jointly optimize the upper and lower bounds. We present a set of experiments, designed to evaluate the new VRLU bound and to compare the VRS method with the classic Variational Autoencoder (VAE) and the VR methods. Next, we apply the VRS approximation to the Multiple-Source Adaptation problem (MSA). MSA is a real-world scenario where data are collected from multiple sources that differ from one another by their probability distribution over the input space. The main aim is to combine fairly accurate predictive models from these sources and create an accurate model for new, mixed target domains. However, many domain adaptation methods assume prior knowledge of the data distribution in the source domains. In this work, we apply the suggested VRS density estimate to the Multiple-Source Adaptation problem (MSA) and show, both theoretically and empirically, that it provides tighter error bounds and improved performance, compared to leading MSA methods.

Robust scalable initialization for Bayesian variational inference with multi-modal Laplace approximations

Predictive modeling typically relies on Bayesian model calibration to provide uncertainty quantification. Variational inference utilizing fully independent (“mean-field”) Gaussian distributions are often used as approximate probability density functions. This simplification is attractive since the number of variational parameters grows only linearly with the number of unknown model parameters. However, the resulting diagonal covariance structure and unimodal behavior can be too restrictive to provide useful approximations of intractable Bayesian posteriors that exhibit highly non-Gaussian behavior, including multimodality. High-fidelity surrogate posteriors for these problems can be obtained by considering the family of Gaussian mixtures. Gaussian mixtures are capable of capturing multiple modes and approximating any distribution to an arbitrary degree of accuracy, while maintaining some analytical tractability. Unfortunately, variational inference using Gaussian mixtures with full-covariance structures suffers from a quadratic growth in variational parameters with the number of model parameters. The existence of multiple local minima due to strong nonconvex trends in the loss functions often associated with variational inference present additional complications, These challenges motivate the need for robust initialization procedures to improve the performance and computational scalability of variational inference with mixture models.In this work, we propose a method for constructing an initial Gaussian mixture model approximation that can be used to warm-start the iterative solvers for variational inference. The procedure begins with a global optimization stage in model parameter space. In this step, local gradient-based optimization, globalized through multistart, is used to determine a set of local maxima, which we take to approximate the mixture component centers. Around each mode, a local Gaussian approximation is constructed via the Laplace approximation. Finally, the mixture weights are determined through constrained least squares regression. The robustness and scalability of the proposed methodology is demonstrated through application to an ensemble of synthetic tests using high-dimensional, multimodal probability density functions. The practical aspects of the approach are demonstrated with inversion problems in structural dynamics.

Bayesian inversion with α-stable priors

We propose using Lévy α-stable distributions to construct priors for Bayesian inverse problems. The construction is based on Markov fields with stable-distributed increments. Special cases include the Cauchy and Gaussian distributions, with stability indices α = 1, and α = 2, respectively. Our target is to show that these priors provide a rich class of priors for modeling rough features. The main technical issue is that the α-stable probability density functions lack closed-form expressions, and this limits their applicability. For practical purposes, we need to approximate probability density functions through numerical integration or series expansions. For Bayesian inversion, the currently available approximation methods are either too time-consuming or do not function within the range of stability and radius arguments. To address the issue, we propose a new hybrid approximation method for symmetric univariate and bivariate α-stable distributions that is both fast to evaluate and accurate enough from a practical viewpoint. In the numerical implementation of α-stable random field priors, we use the constructed approximation method. We show how the constructed priors can be used to solve specific Bayesian inverse problems, such as the deconvolution problem and the inversion of a function governed by an elliptic partial differential equation. We also demonstrate hierarchical α-stable priors in the one-dimensional deconvolution problem. For all numerical examples, we use maximum a posteriori estimation. To that end, we exploit the limited-memory BFGS and its bounded variant for the estimator.

Data-driven probability density forecast for stochastic dynamical systems

In this paper, a data-driven non-parametric approach is presented for forecasting the probability density evolution of stochastic dynamical systems. The method is based on stochastic Koopman operator and extended dynamic mode decomposition (EDMD). To approximate the finite-dimensional eigendecomposition of the stochastic Koopman operator, EDMD is applied to the training data set sampled from the stationary distribution of the underlying stochastic dynamical system. The family of the Koopman operators form a semigroup, which is generated by the infinitesimal generator of the stochastic dynamical system. A significant connection between the generator and Fokker-Planck operator is leveraged to construct an orthonormal basis of a weighted Hilbert space, enabling a spectral decomposition of the probability density function to be performed in this space. This approach is a data-driven method and used to predict the probability density evolution and real-time moment estimation. In the limit of the large number of snapshots and observables, the data-driven probability density approximation converges to the Galerkin projection of the semigroup solution of Fokker-Planck equation on a basis adapted to an invariant measure. This method shares similarities with diffusion forecasting but is capable of producing more accurate probability density forecasts than diffusion forecasting. A few numerical examples are presented to illustrate the performance of the proposed data-driven probability density forecast.

Higher-order averaging procedure for performance analysis of a mono-stable Duffing piezoelectric energy harvesting system under white noise excitation

This work is concerned with the energy harvesting of mono-stable Duffing oscillators with piezoelectric coupling under white noise excitation. This is the first time that the higher-order averaging procedure is extended to find the first-, second- and third-order approximate stationary joint probability density functions of amplitude and full phase by solving the corresponding Fokker-Planck-Kolmogorov (FPK) equation. Then the mean-square electric voltage and mean output power are derived through the approximate relation between the electric voltage and the mechanical states. The analytical results are verified by Monte Carlo simulation. The effects of system parameters, such as excitation intensity, mechanical damping term, nonlinear stiffness coefficients, and the resistance ratio of the resistive–capacitive circuit on the mean-square displacement and mean output power are discussed in detail. It has been shown that the averaging procedure of the averaging method of coefficients in FPK equation can be considered as a particular case of the proposed higher-order averaging procedure and the higher-order approximate solutions are much more accurate than the one obtained by the averaging method of coefficients in FPK equation.© 2022 The Authors. Published by Elsevier Ltd.

Further exploration into the valid regions of Gram–Charlier densities

Density estimation plays an important and fundamental role in finance, pattern recognition, machine learning, and statistics. Based on derivatives of a Gaussian density, a Gram–Charlier series presents an infinite expansion. Its truncated series is often used in many fields to approximate probability density functions. Although the expansions are convenient, there are constrained regions on the value of the cumulants (or moments) that admit a valid (nonnegative) probability density function. Lin and Zhang (2022)’s paper focuses on Gram–Charlier densities to show how the valid region of higher cumulants can be numerically implemented by semidefinite programming, which ensures that a series truncated at a cumulant of an arbitrary even order represents a valid probability density. This paper is the further exploration into the same problem. First, we use the representation theorem of such polynomials as sum of squares on the Gram–Charlier density to show how to develop the corresponding convex optimization problem for its valid region. Second, we provide the valid skewness–kurtosis regions of Gram–Charlier densities only up to the sixteenth-order because the semidefinite programming fails to calculate these regions when the order is above that. Third, we explore the valid region of the fourth-order Gram–Charlier defined on an arbitrary finite domain [−q,q] but not the field R of real numbers. Our analysis proves that the ranges of skewness and kurtosis can be broadened with the finite domains, which earn a wider application. Furthermore, the impact of the length of finite domains 2q on valid regions has been shown.

Fitting stochastic epidemic models to gene genealogies using linear noise approximation.

Phylodynamics is a set of population genetics tools that aim at reconstructing demographic history of a population based on molecular sequences of individuals sampled from the population of interest. One important task in phylodynamics is to estimate changes in (effective) population size. When applied to infectious disease sequences such estimation of population size trajectories can provide information about changes in the number of infections. To model changes in the number of infected individuals, current phylodynamic methods use non-parametric approaches (e.g., Bayesian curve-fitting based on change-point models or Gaussian process priors), parametric approaches (e.g., based on differential equations), and stochastic modeling in conjunction with likelihood-free Bayesian methods. The first class of methods yields results that are hard to interpret epidemiologically. The second class of methods provides estimates of important epidemiological parameters, such as infection and removal/recovery rates, but ignores variation in the dynamics of infectious disease spread. The third class of methods is the most advantageous statistically, but relies on computationally intensive particle filtering techniques that limits its applications. We propose a Bayesian model that combines phylodynamic inference and stochastic epidemic models, and achieves computational tractability by using a linear noise approximation (LNA) - a technique that allows us to approximate probability densities of stochastic epidemic model trajectories. LNA opens the door for using modern Markov chain Monte Carlo tools to approximate the joint posterior distribution of the disease transmission parameters and of high dimensional vectors describing unobserved changes in the stochastic epidemic model compartment sizes (e.g., numbers of infectious and susceptible individuals). In a simulation study, we show that our method can successfully recover parameters of stochastic epidemic models. We apply our estimation technique to Ebola genealogies estimated using viral genetic data from the 2014 epidemic in Sierra Leone and Liberia.

AdaAnn: ADAPTIVE ANNEALING SCHEDULER FOR PROBABILITY DENSITY APPROXIMATION

Approximating probability distributions can be a challenging task, particularly when they are supported over regions of high geometrical complexity or exhibit multiple modes. Annealing can be used to facilitate this task which is often combined with constant a priori selected increments in inverse temperature. However, using constant increments limits the computational efficiency due to the inability to adapt to situations where smooth changes in the annealed density could be handled equally well with larger increments. We introduce AdaAnn, an adaptive annealing scheduler that automatically adjusts the temperature increments based on the expected change in the Kullback-Leibler divergence between two distributions with a sufficiently close annealing temperature. AdaAnn is easy to implement and can be integrated into existing sampling approaches such as normalizing flows for variational inference and Markov chain Monte Carlo. We demonstrate the computational efficiency of the AdaAnn scheduler for variational inference with normalizing flows on a number of examples, including posterior estimation of parameters for dynamical systems and probability density approximation in multimodal and high-dimensional settings.