Multimodal Probability Density Functions Research Articles

Robust scalable initialization for Bayesian variational inference with multi-modal Laplace approximations

Predictive modeling typically relies on Bayesian model calibration to provide uncertainty quantification. Variational inference utilizing fully independent (“mean-field”) Gaussian distributions are often used as approximate probability density functions. This simplification is attractive since the number of variational parameters grows only linearly with the number of unknown model parameters. However, the resulting diagonal covariance structure and unimodal behavior can be too restrictive to provide useful approximations of intractable Bayesian posteriors that exhibit highly non-Gaussian behavior, including multimodality. High-fidelity surrogate posteriors for these problems can be obtained by considering the family of Gaussian mixtures. Gaussian mixtures are capable of capturing multiple modes and approximating any distribution to an arbitrary degree of accuracy, while maintaining some analytical tractability. Unfortunately, variational inference using Gaussian mixtures with full-covariance structures suffers from a quadratic growth in variational parameters with the number of model parameters. The existence of multiple local minima due to strong nonconvex trends in the loss functions often associated with variational inference present additional complications, These challenges motivate the need for robust initialization procedures to improve the performance and computational scalability of variational inference with mixture models.In this work, we propose a method for constructing an initial Gaussian mixture model approximation that can be used to warm-start the iterative solvers for variational inference. The procedure begins with a global optimization stage in model parameter space. In this step, local gradient-based optimization, globalized through multistart, is used to determine a set of local maxima, which we take to approximate the mixture component centers. Around each mode, a local Gaussian approximation is constructed via the Laplace approximation. Finally, the mixture weights are determined through constrained least squares regression. The robustness and scalability of the proposed methodology is demonstrated through application to an ensemble of synthetic tests using high-dimensional, multimodal probability density functions. The practical aspects of the approach are demonstrated with inversion problems in structural dynamics.

On the effect of flow regime and pore structure on the flow signatures in porous media

In this study, lattice Boltzmann method (LBM) is utilized for three-dimensional simulation of fluid flow through two porous structures, consisting of grains with the same diameter: (i) a homogeneous porous domain, in which the grains are placed with a simple cubic packing configuration, and (ii) a randomly packed porous domain. An ultra-fine mesh size is considered to perform the simulations in three orders of magnitude of Reynolds number (Re), covering laminar to turbulent flow regimes, and capture different flow signatures. Pore velocity fields are derived, and their sample probability density functions (PDF) are analyzed vs time to investigate the dynamics of the flow. The analysis of the PDFs clearly shows that stagnant zones play a significant role in the formation of the pore flow fields, manifested by multimodal PDFs, and the distribution of the velocities in porous media at various Re cannot be characterized by a single PDF model regardless of the pore structure. While the velocities at the stagnant regions and in the vicinity of the solid boundaries are primarily affected by the viscous forces and exhibit a power-law PDF at different Re, the velocities in the main (preferential) flow pathways away from the boundaries are shown to be influenced by the inertial forces, hence having an exponential PDF when Re is low. At high Re, however, depending on the tortuosity of the porous structure, the velocities may exhibit an exponential or even Laplace PDF.

Examination of probability distribution of mixture fraction in LES/FDF modelling of a turbulent partially premixed jet flame

An accurate prediction of the probability density function (PDF) of the mixture fraction is crucial to the prediction of combustion since mixing plays an important role in turbulent non-premixed and partially premixed flames. This work provides an assessment of the large-eddy simulation (LES)/filtered density function (FDF) method for the prediction of the PDF of the mixture fraction. The advantage of the LES/FDF method is that it provides the full predictions of the statistical distribution of scalars including the mixture fraction. The predictive accuracy of the method for the PDF is yet to be fully validated. Assessing the prediction of the PDF of the mixture fraction, a conserved scalar, is an important starting point. The Sydney/Sandia inhomogeneous inlet jet flame is used as a test case. A quick comparison shows that the LES/FDF predicted PDF shapes of the mixture fraction deviate significantly from the commonly presumed Beta-PDF as well as from the experimental data in the flame. To examine the source of the discrepancy, we clarify the different PDF definitions used in the comparison among the predictions, measurements, and presumed shape PDFs. The discrepancy observed from the comparison is largely reconciled by clarifying the difference between the PDFs that are examined. The PDF of the resolved mixture fraction is shown to be close to the Beta-PDF in both the measurements and predictions, while the PDF directly deduced from the LES/FDF particles deviates significantly from the Beta-PDF. A multimodal PDF analysis and a pseudo convergence analysis are conducted to provide plausible evidence to support the predicted multimodal PDF shapes. The sub-filter scale FDF is shown to be close to the Beta-PDF too through the construction of a synthesised PDF, which supports the common presumed Beta-PDF assumption used in the presumed PDF methods when combined with LES.

A new maximum entropy method for estimation of multimodal probability density function

The high-precision estimation of a multimodal probability density function is a difficult problem in many engineering fields. We propose a new method to improve the estimation accuracy based on the fractional moment-based maximum entropy method with a nonlinear transformation and a multi-peak recognition method. For the translation parameters in the nonlinear transformation, three approaches, such as sample-based least square polynomial fitting technique, sample-based kernel density estimation and classical maximum entropy method, are presented to determine the parameters. By adjusting the translation parameter, the valley of the probability density function curve can be translated to the position with a larger slope, and the distance between adjacent peaks is enlarged to avoid the wrong fitting form of the probability density function curve with multiple peaks. After the parameters of the transformation are obtained, the fractional moment-based maximum entropy method is applied to predict the probability density function of the transformed performance function. Two numerical examples are used to verify the accuracy and stability of the proposed method. Two engineering examples are introduced to illustrate the applicability and efficiency of the proposed method in the real-life engineering setting. It is concluded that the proposed method uses fewer moments with less additional calculation costs, and has good computational efficiency and applicability for the modeling of multimodal probability density functions, compared with the classical fractional moment-based maximum entropy method.

Reliability analysis of structures with multimodal distributions based on direct probability integral method

For practical structures, some input random variables follow multimodal distributions, and conventional reliability analysis methods may result in large computational errors. In this paper, a novel direct probability integral method (DPIM), which decouples governing equation of structure and the probability density integral equation (PDIE), is proposed to address the static and dynamic reliability assessment of structures involving random variables with multimodal distributions. Firstly, the multimodal probability density functions (PDFs) of input random variables are established by the Gaussian mixture model. Then, using numerical integration and smoothing technique of Dirac delta function, the PDFs of structural responses with multimodal random variables are achieved by solving the PDIE, in which three numerical integration algorithms, namely, the Quasi-Monte Carlo approach, the sparse grid approach, and Generalized F-discrepancy-based point selection approach are employed. Further, the reliability of static structure can be readily obtained by integrating the PDF of response function, while the dynamic reliability can be evaluated by DPIM combining with extreme value distribution of stochastic process. Finally, several examples demonstrate the superiority of DPIM, and the Generalized F-discrepancy-based point selection approach has the highest accuracy and efficiency for solving PDIE. The characteristics of uncertainty propagation in structures involving multimodal distributions of input random variables are revealed.

Spatiotemporal Learning of Directional Uncertainty in Urban Environments With Kernel Recurrent Mixture Density Networks

Autonomous vehicles operating in urban environments need to deal with an abundance of other dynamic objects, such as pedestrians and vehicles. This requires the development of predictive models that capture the complexity and long-term patterns of motion in the environment. We approach this problem by modeling movement directions of a typical object in the environment over time. We propose a method to build a continuous map to capture spatiotemporal movements. At a given coordinate in space and time, our method provides a multi-modal probability density function over the possible directions an object can move in. We achieve this by projecting data into a high-dimensional feature space using sparse approximate kernels, and passing the highdimensional features through a long short-term memory (LSTM) network. A mixture density network (MDN) with von Mises distributions is then trained on the hidden representations of the LSTM. Once trained, the outputs of the MDN can be used to construct the probability distribution over movement directions. The model is continuous in both spatial and temporal domains, as we can query the change of directional uncertainty at arbitrary resolution in space and time. We conduct experiments on three simulated and three real-world benchmark datasets with pedestrians and vehicles, and demonstrate that both the continuous nature and the addition of temporal patterns improves performance.

Non-Gaussian statistics in global atmospheric dynamics: a study with a 10 240-member ensemble Kalman filter using an intermediate atmospheric general circulation model

Abstract. We previously performed local ensemble transform Kalman filter (LETKF) experiments with up to 10 240 ensemble members using an intermediate atmospheric general circulation model (AGCM). While the previous study focused on the impact of localization on the analysis accuracy, the present study focuses on the probability density functions (PDFs) represented by the 10 240-member ensemble. The 10 240-member ensemble can resolve the detailed structures of the PDFs and indicates that non-Gaussianity is caused in those PDFs by multimodality and outliers. The results show that the spatial patterns of the analysis errors are similar to those of non-Gaussianity. While the outliers appear randomly, large multimodality corresponds well with large analysis error, mainly in the tropical regions and storm track regions where highly nonlinear processes appear frequently. Therefore, we further investigate the life cycle of multimodal PDFs, and show that they are mainly generated by the on–off switch of convective parameterization in the tropical regions and by the instability associated with advection in the storm track regions. Sensitivity to the ensemble size suggests that approximately 1000 ensemble members are necessary in the intermediate AGCM-LETKF system to represent the detailed structures of non-Gaussian PDFs such as skewness and kurtosis; the higher-order non-Gaussian statistics are more vulnerable to the sampling errors due to a smaller ensemble size.

A new uncertainty propagation method considering multimodal probability density functions

In practical engineering applications, random variables may follow multimodal distributions with multiple modes in the probability density functions, such as the structural fatigue stress of a steel bridge carrying both highway and railway traffic and the vibratory load of a blade subject to stochastic dynamic excitations, etc. Traditional uncertainty propagation methods are mainly used to treat random variables with only unimodal probability density functions, which, therefore, tend to result in large computational errors when multimodal probability density functions are involved. In this paper, an uncertainty propagation method is developed for problems in which multimodal probability density functions are involved. Firstly, the multimodal probability density functions of input random variables are established using the Gaussian mixture model. Secondly, the uncertainties of the input random variables are propagated to the response function through an integration of the sparse grid numerical method and maximum entropy method. Finally, the convergence mechanism is developed to improve the uncertainty propagation accuracy step by step. Two numerical examples and one engineering application are studied to demonstrate the effectiveness of the proposed method.

A high-precision probabilistic uncertainty propagation method for problems involving multimodal distributions

Abstract In practical engineering applications, random variables may follow multimodal distributions with multiple modes in the probability density functions, such as the structural fatigue stress of a steel bridge carrying both highway and railway traffic and the vibratory load of a blade subject to stochastic dynamic excitations, etc. Traditional probabilistic uncertainty propagation methods are mainly used to treat random variables with only unimodal distributions, which, therefore, tend to result in large computational errors when multimodal distributions are involved. In this paper, a high-precision probabilistic uncertainty propagation method is proposed for problems involving multimodal distributions. Firstly, the multimodal probability density functions of input random variables are constructed based on the Gaussian mixture model. Secondly, the high-order moments of the response function are calculated using the univariate dimension reduction method, based on which the input uncertainty is effectively propagated. Thirdly, the probability density function of the response is estimated using the maximum entropy method. Finally, a convergence mechanism is formulated to help ensure the uncertainty propagation accuracy. Two mathematical problems and two truss structures are investigated to demonstrate the effectiveness of the proposed method.

Sequential Outlier Detection Based on Incremental Decision Trees

We introduce an online outlier detection algorithm to detect outliers in a sequentially observed data stream. For this purpose, we use a two-stage filtering and hedging approach. In the first stage, we construct a multimodal probability density function to model the normal samples. In the second stage, given a new observation, we label it as an anomaly if the value of aforementioned density function is below a specified threshold at the newly observed point. In order to construct our multimodal density function, we use an incremental decision tree to construct a set of subspaces of the observation space. We train a single component density function of the exponential family using the observations, which fall inside each subspace represented on the tree. These single component density functions are then adaptively combined to produce our multimodal density function, which is shown to achieve the performance of the best convex combination of the density functions defined on the subspaces. As we observe more samples, our tree grows and produces more subspaces. As a result, our modeling power increases in time, while mitigating overfitting issues. In order to choose our threshold level to label the observations, we use an adaptive thresholding scheme. We show that our adaptive threshold level achieves the performance of the optimal prefixed threshold level, which knows the observation labels in hindsight. Our algorithm provides significant performance improvements over the state of the art in our wide set of experiments involving both synthetic as well as real data.

Open and closed-loop stochastic dynamics of a class of nonlinear chemical processes with multiplicative noise

The open and closed-loop global state probability density function behavior of a class of single-state nonlinear chemical processes subjected to additive and multiplicative white noises is characterized with analytic formulae through Fokker Planck (FP) theory, in terms of: (i) stationary state probability density function (PDF), (ii) PDF evolution along deterministic, diffusion, and escape time scales, (iii) conditions for PDF metastability along escape time scale, and (iv) dependency of PDF motion on deterministic dynamics. Comparing with noise additivity, multiplicativeness can yield similar or substantially different open-loop PDF evolution behavior. The application of control to an open-loop (possibly fragile and metastable) multimodal PDF yields a closed-loop robust monomodal PDF with mode regulation capability. The developments and findings are illustrated with numerical simulations of FP’s PDE equation.

New Families of Skewed Higher-Order Kernel Estimators to Solve the BSS/ICA Problem for Multimodal Sources Mixtures.

This letter suggests two new types of asymmetrical higher-order kernels (HOK) that are generated using the orthogonal polynomials Laguerre (positive or right skew) and Bessel (negative or left skew). These skewed HOK are implemented in the blind source separation/independent component analysis (BSS/ICA) algorithm. The tests for these proposed HOK are accomplished using three scenarios to simulate a real environment using actual sound sources, an environment of mixtures of multimodal fast-changing probability density function (pdf) sources that represent a challenge to the symmetrical HOK, and an environment of an adverse case (near gaussian). The separation is performed by minimizing the mutual information (MI) among the mixed sources. The performance of the skewed kernels is compared to the performance of the standard kernels such as Epanechnikov, bisquare, trisquare, and gaussian and the performance of the symmetrical HOK generated using the polynomials Chebyshev1, Chebyshev2, Gegenbauer, Jacobi, and Legendre to the tenth order. The gaussian HOK are generated using the Hermite polynomial and the Wand and Schucany procedure. The comparison among the 96 kernels is based on the average intersymbol interference ratio (AISIR) and the time needed to complete the separation. In terms of AISIR, the skewed kernels' performance is better than that of the standard kernels and rivals most of the symmetrical kernels' performance. The importance of these new skewed HOK is manifested in the environment of the multimodal pdf mixtures. In such an environment, the skewed HOK come in first place compared with the symmetrical HOK. These new families can substitute for symmetrical HOKs in such applications.

Uncertainty Quantification of geochemical and mechanical compaction in layered sedimentary basins

In this work we propose an Uncertainty Quantification methodology for sedimentary basins evolution under mechanical and geochemical compaction processes, which we model as a coupled, time-dependent, non-linear, monodimensional (depth-only) system of PDEs with uncertain parameters. While in previous works (Formaggia et al. 2013, Porta et al. 2014) we assumed a simplified depositional history with only one material, in this work we consider multi-layered basins, in which each layer is characterized by a different material, and hence by different properties. This setting requires several improvements with respect to our earlier works, both concerning the deterministic solver and the stochastic discretization. On the deterministic side, we replace the previous fixed-point iterative solver with a more efficient Newton solver at each step of the time-discretization. On the stochastic side, the multi-layered structure gives rise to discontinuities in the dependence of the state variables on the uncertain parameters, that need an appropriate treatment for surrogate modeling techniques, such as sparse grids, to be effective. We propose an innovative methodology to this end which relies on a change of coordinate system to align the discontinuities of the target function within the random parameter space. The reference coordinate system is built upon exploiting physical features of the problem at hand. We employ the fact that the locations of the material interfaces display a smooth dependence on the random parameters and are therefore amenable to sparse grid polynomial approximations. We showcase the capabilities of our numerical methodologies through two synthetic test cases. In particular, we show that our methodology reproduces with high accuracy multi-modal probability density functions displayed by target state variables (e.g., porosity).

A Unified tool to estimate Distances, Ages, and Masses (UniDAM) from spectrophotometric data

Galactic archaeology - the study of the formation and evolution of the Milky Way by reconstructing its past from its current constituents - requires precise and accurate knowledge of stellar parameters for as many stars as possible. To achieve this a number of large spectroscopic surveys have been undertaken and are still ongoing. So far consortia carrying out the different spectroscopic surveys have used different tools to determine stellar parameters of stars from their derived effective temperatures (Teff), surface gravities (log g) and metallicities ([Fe/H]) possibly combined with photometric, astrometric, interferometric or asteroseismic information. Here we aim to homogenise the stellar characterisation by applying a unified tool to a large set of publicly available spectrophotometric data. We use spectroscopic data from a variety of large surveys combined with infra-red photometry from 2MASS and AllWISE and compare these in a Bayesian manner with PARSEC isochrones to derive probability density functions (PDFs) for stellar masses, ages and distances. We treat PDFs of pre-helium-core burning, helium-core burning and post helium-core burning solutions as well as different peaks in multi-modal PDFs (i.e. each unimodal sub-PDF) of the different evolutionary phases separately. For over 2.5 million stars we report mass, age and distance estimate for each evolutionary phase and unimodal sub-PDF. We report Gaussian, skewed Gaussian, truncated Gaussian, modified truncated exponential distribution or truncated Student's t-distribution functions to represent each sub-PDF, allowing to reconstruct detailed PDFs. Comparisons with stellar parameter estimates from the literature show good agreement within uncertainties. We present UniDAM - the unified tool applicable to spectrophotometric data of different surveys to obtain a homogenised set of stellar parameters.

Inverse trishear modeling of bedding dip data using Markov chain Monte Carlo methods

We present a method for fitting trishear models to surface profile data, by restoring bedding dip data and inverting for model parameters using a Markov chain Monte Carlo method. Trishear is a widely-used kinematic model for fault-propagation folds. It lacks an analytic solution, but a variety of data inversion techniques can be used to fit trishear models to data. Where the geometry of an entire folded bed is known, models can be tested by restoring the bed to its pre-folding orientation. When data include bedding attitudes, however, previous approaches have relied on computationally-intensive forward modeling. This paper presents an equation for the rate of change of dip in the trishear zone, which can be used to restore dips directly to their pre-folding values. The resulting error can be used to calculate a probability for each model, which allows solution by Markov chain Monte Carlo methods and inversion of datasets that combine dips and contact locations. These methods are tested using synthetic and real datasets. Results are used to approximate multimodal probability density functions and to estimate uncertainty in model parameters. The relative value of dips and contacts in constraining parameters and the effects of uncertainty in the data are investigated.

A Bank of Maximum <italic>A Posteriori</italic> (MAP) Estimators for Target Tracking

Nonlinear estimation problems, such as range-only and bearing-only target tracking, are often addressed using linearized estimators, e.g., the extended Kalman filter (EKF). These estimators generally suffer from linearization errors as well as the inability to track multimodal probability density functions. In this paper, we propose a bank of batch maximum a posteriori (MAP) estimators as a general estimation framework that provides relinearization of the entire state trajectory, multihypothesis tracking, and an efficient hypothesis generation scheme. Each estimator in the bank is initialized using a locally optimal state estimate for the current time step. Every time a new measurement becomes available, we relax the original batch-MAP problem and solve it incrementally. More specifically, we convert the relaxed one-step-ahead cost function into polynomial or rational form and compute all the local minima analytically. These local minima generate highly probable hypotheses for the target's trajectory and hence greatly improve the quality of the overall MAP estimate. Additionally, pruning of least probable hypotheses and marginalization of old states are employed to control the computational cost. Monte Carlo simulation and real-world experimental results show that the proposed approach significantly outperforms the standard EKF, the batch-MAP estimator, and the particle filter.

Estimating probability distributions of solar irradiance

In the presence of clouds the ability to calculate instantaneous spectral irradiance values is limited by the ability to acquire appropriate input parameters for radiative transfer solvers. However, the knowledge of the statistical characteristics of spectral irradiance as a function of season and time of the day is relevant for solar energy and health applications. For this purpose a method to derive the wavelength dependent probability density functions (PDFs) and its seasonal site variability is presented. In contrast to the UVB range, the derived PDFS at three stations in Europe (Bilthoven, Garmisch-Partenkirchen and Thessaloniki) show only minor wavelength dependence above 315 nm. But there are major differences of the PDFs that are attributed to the site specific cloud climatology at these stations. Furthermore the results suggest that the previously described relationship between air mass and bimodality is the consequence of seasonal cloud variations. For Thessaloniki it is shown that the pyranometer sample spread around the cloudless value is proportional to the secant of the solar zenith angle and therefore scales according to air mass. Cloud amount observations are utilized to associate the local maxima of the multimodal PDFs with rough cloudiness states confirming the already established interpretation of broadband data for spectral data as well. As one application example the likelihood of irradiance enhancements over the clear sky case due to clouds is assessed.

Cross-entropy-based adaptive importance sampling using Gaussian mixture

Structural reliability analysis frequently requires the use of sampling-based methods, particularly for the situation where the failure domain in the random variable space is complex. One of the most efficient and widely utilized methods to use in such a situation is importance sampling. Recently, an adaptive importance sampling method was proposed to find a near-optimal importance sampling density by minimizing Kullback–Leibler cross entropy, i.e. a measure of the difference between the absolute best sampling density and the one being used for the importance sampling. In this paper, the adaptive importance sampling approach is further developed by incorporating a nonparametric multimodal probability density function model called the Gaussian mixture as the importance sampling density. This model is used to fit the complex shape of the absolute best sampling density functions including those with multiple important regions. An efficient procedure is developed to update the Gaussian mixture model toward a near-optimal density using a small size of pre-samples. The proposed method needs only a few steps to achieve a near-optimal sampling density, and shows significant improvement in efficiency and accuracy for a variety of component and system reliability problems. The method requires far less samples than both crude Monte Carlo simulation and the cross-entropy-based adaptive importance sampling method employing a unimodal density function; thus achieving relatively small values of the coefficient of variation efficiently. The computational efficiency and accuracy of the proposed method are not hampered by the probability level, dimension of random variable space, and curvatures of limit-state function. Moreover, the distribution model parameters of the Gaussian densities in the obtained near-optimal density help identify important areas in the random variable space and their relative importance.

Belief functions induced by multimodal probability density functions, an application to the search and rescue problem

In this paper, we propose a new method to generate a continuous belief functions from a multimodal probability distribution function defined over a continuous domain. We generalize Smets' approach in the sense that focal elements of the resulting continuous belief function can be disjoint sets of the extended real space of dimension n. We then derive the continuous belief function from multimodal probability density functions using the least commitment principle. We illustrate the approach on two examples of probability density functions (unimodal and multimodal). On a case study of Search And Rescue (SAR), we extend the traditional probabilistic framework of search theory to continuous belief functions theory. We propose a new optimization criterion to allocate the search effort as well as a new rule to update the information about the lost object location in this latter framework. We finally compare the allocation of the search effort using this alternative uncertainty representation to the traditional probabilistic representation.

Monitoring-Based Fatigue Reliability Assessment of Steel Bridges: Analytical Model and Application

A fatigue reliability model which integrates the probability distribution of hot spot stress range with a continuous probabilistic formulation of Miner's damage cumulative rule is developed for fatigue life and reliability evaluation of steel bridges with long-term monitoring data. By considering both the nominal stress obtained by measurements and the corresponding stress concentration factor (SCF) as random variables, a probabilistic model of the hot spot stress is formulated with the use of the S-N curve and the Miner's rule, which is then used to evaluate the fatigue life and failure probability with the aid of structural reliability theory. The proposed method is illustrated using the long-term strain monitoring data from the instrumented Tsing Ma Bridge. A standard daily stress spectrum accounting for highway traffic, railway traffic, and typhoon effects is derived by use of the monitoring data. Then global and local finite element models (FEMs) of the bridge are developed for numerically calculating the SCFs at fatigue-susceptible locations, while the stochastic characteristics of SCF for a typical welded T-joint are obtained by full-scale model experiments of a railway beam section of the bridge. A multimodal probability density function (PDF) of the stress range is derived from the monitoring data using the finite mixed Weibull distributions in conjunction with a hybrid parameter estimation algorithm. The failure probability and reliability index versus fatigue life are achieved from the obtained joint PDF of the hot spot stress in terms of the nominal stress and SCF random variables.