Conditional Probability Density Research Articles

Compressed Decentralized Learning of Conditional Mean Embedding Operators in Reproducing Kernel Hilbert Spaces

Conditional mean embedding (CME) operators encode conditional probability densities within Reproducing Kernel Hilbert Space (RKHS). In this paper, we present a decentralized algorithm for a collection of agents to cooperatively approximate CME over a network. Communication constraints limit the agents from sending all data to their neighbors; we only allow sparse representations of covariance operators to be exchanged among agents, compositions of which defines CME. Using a coherence-based compression scheme, we present a consensus-type algorithm that preserves the average of the approximations of the covariance operators across the network. We theoretically prove that the iterative dynamics in RKHS is stable. We then empirically study our algorithm to estimate CMEs to learn spectra of Koopman operators for Markovian dynamical systems and to execute approximate value iteration for Markov decision processes (MDPs).

The Mori–Zwanzig formulation of deep learning

We develop a new formulation of deep learning based on the Mori–Zwanzig (MZ) formalism of irreversible statistical mechanics. The new formulation is built upon the well-known duality between deep neural networks and discrete dynamical systems, and it allows us to directly propagate quantities of interest (conditional expectations and probability density functions) forward and backward through the network by means of exact linear operator equations. Such new equations can be used as a starting point to develop new effective parameterizations of deep neural networks and provide a new framework to study deep learning via operator-theoretic methods. The proposed MZ formulation of deep learning naturally introduces a new concept, i.e., the memory of the neural network, which plays a fundamental role in low-dimensional modeling and parameterization. By using the theory of contraction mappings, we develop sufficient conditions for the memory of the neural network to decay with the number of layers. This allows us to rigorously transform deep networks into shallow ones, e.g., by reducing the number of neurons per layer (using projection operators), or by reducing the total number of layers (using the decay property of the memory operator).

A Statistical Dependence Framework Based on a Multivariate Normal Copula Function and Stochastic Differential Equations for Multivariate Data in Forestry

Stochastic differential equations and Copula theories are important topics that have many advantages for applications in almost every discipline. Many studies in forestry collect longitudinal, multi-dimensional, and discrete data for which the amount of measurement of individual variables does not match. For example, during sampling experiments, the diameters of all trees, the heights of approximately 10% of the trees, and the tree crown base height and crown width for a significantly smaller number of trees are measured. In this study, for estimating five-dimensional dependencies, we used a normal copula approach, where the dynamics of individual tree variables (diameter, potentially available area, height, crown base height, and crown width) are described by a stochastic differential equation with mixed-effect parameters. The approximate maximum likelihood method was used to obtain parameter estimates of the presented stochastic differential equations, and the normal copula dependence parameters were estimated using the pseudo-maximum likelihood method. This study introduced the normalized multi-dimensional interaction information index based on differential entropy to capture dependencies between state variables. Using conditional copula-type probability density functions, the exact form equations defining the links among the diameter, potentially available area, height, crown base height, and crown width were derived. All results were implemented in the symbolic algebra system MAPLE.

Asynchronous Spatiotemporal Spike Metric for Event Cameras.

Event cameras as bioinspired vision sensors have shown great advantages in high dynamic range and high temporal resolution in vision tasks. Asynchronous spikes from event cameras can be depicted using the marked spatiotemporal point processes (MSTPPs). However, how to measure the distance between asynchronous spikes in the MSTPPs still remains an open issue. To address this problem, we propose a general asynchronous spatiotemporal spike metric considering both spatiotemporal structural properties and polarity attributes for event cameras. Technically, the conditional probability density function is first introduced to describe the spatiotemporal distribution and polarity prior in the MSTPPs. Besides, a spatiotemporal Gaussian kernel is defined to capture the spatiotemporal structure, which transforms discrete spikes into the continuous function in a reproducing kernel Hilbert space (RKHS). Finally, the distance between asynchronous spikes can be quantified by the inner product in the RKHS. The experimental results demonstrate that the proposed approach outperforms the state-of-the-art methods and achieves significant improvement in computational efficiency. Especially, it is able to better depict the changes involving spatiotemporal structural properties and polarity attributes.

Data Assimilation Networks

AbstractData Assimilation aims at estimating the posterior conditional probability density functions based on error statistics of the noisy observations and the dynamical system. State of the art methods are sub‐optimal due to the common use of Gaussian error statistics and the linearization of the non‐linear dynamics. To achieve a good performance, these methods often require case‐by‐case fine‐tuning by using explicit regularization techniques such as inflation and localization. In this paper, we propose a fully data driven deep learning framework generalizing recurrent Elman networks and data assimilation algorithms. Our approach approximates a sequence of prior and posterior densities conditioned on noisy observations using a log‐likelihood cost function. By construction our approach can then be used for general nonlinear dynamics and non‐Gaussian densities. As a first step, we evaluate the performance of the proposed approach by using fully and partially observed Lorenz‐95 system in which the outputs of the recurrent network are fitted to Gaussian densities. We numerically show that our approach, without using any explicit regularization technique, achieves comparable performance to the state‐of‐the‐art methods, IEnKF‐Q and LETKF, across various ensemble size.

Фильтрация сигналов при скачкообразных мультипликативных помехах

The problem of constructing the approximately optimal Bayesian algorithm of filtering the output signal for a linear stochastic dynamical system is solved. Non-linear mixture of the output signal and of the multiplicative interference was measured. This interference is a continuous-valued random process with the unknown distribution law within the given limits of [c(1), c(2)], c(1) > 0, c(2) > 0, and in the frequency range of [0, ∆ω]. The filtering algorithm is synthesized by the approximately optimal signal estimation method based on the theory of systems with random jump structure and the method of two-time parametric approximation of the phase coordinates probability distribution. The method consists in approximate replacement of the unknown probability densities of phase coordinates by the known distribution laws with the unknown mathematical expectations and covariances determined as a result of solving the problem. The proposed approximate-optimal filtering algorithm is based on replacement of the continuous-valued multiplicative interference by a random jump process, i.e., Markov chain with two states c(1), c(2) and equal intensities of transitions from one state to another q′ = 2∆ω. Conditional probability densities of the output signal for fixed states of the Markov chain c(1), c(2) are approximated by gamma distributions that depend on mathematical expectations and variances of the x(t) signal at the fixed c(1), c(2) and measurements x(t) in a mixture with the Markov jump interference. An example of constructing an algorithm for evaluating the distance from an aircraft to the object based on object irradiance measurement by the infrared direction finder was considered. Irradiance was equal to the radiation strength ratio to the square of the distance. Radiation strength, i.e., multiplicative interference, was the random process with unknown continuous distribution in a limited range. The approximate-optimal filter was constructed for distance evaluation by replacing this process with a two-state Markov chain and approximating the distance probability density by the Rayleigh distribution

The mechanical behaviors of random curved fiber networks by numerical simulations

Random curved fiber networks (RCFNs) are ubiquitous in biological materials and artificial composites, attracting much attention because of their unique mechanical properties and functionalities. Based on numerical simulations, the effects of fiber morphology, fiber aspect ratio and relative density on mechanical properties of two dimensional (2D) RCFNs are investigated in terms of the elastic modulus, yield strength as well as energy ratio of stretching to bending deformation. Based on the conditional probability density function (CPDF), a general theoretical model is extended for calculating the crosslink density of fiber networks, which elucidates the independence of crosslink density on fiber shape in RCFNs. According to numerical results, the elastic modulus and yield strength decrease with rising fiber curvature and are enhanced by increasing of aspect ratio and relative density. The influence of fiber aspect ratio in RCFNs exhibits thresholds, beyond which the mechanical properties will reach upper limits for constant curvature. The scaled yield strength to modulus of RCFNs versus curvature is found to collapse into a nonlinear curve regardless of aspect ratios. The energy ratio is influenced by fiber curvature more vigorously which is explained by deformations of curved segments and load paths deviations. This paper is expected to contribute to deep understanding of relationships between overall mechanical properties and structural parameters of RCFNs.

Time-variant reliability global sensitivity analysis with single-loop Kriging model combined with importance sampling

This paper develops a novel failure probability-based global sensitivity index by introducing the Bayes formula into the moment-independent global sensitivity index to approximate the effect of input random variables or stochastic processes on the time-variant reliability. The proposed global sensitivity index can estimate the effect of uncertain inputs on the time-variant reliability by comparing the difference between the unconditional probability density function of input variables and the conditional probability density function in failure state of input variables. Furthermore, a single-loop active learning Kriging method combined with metamodel-based importance sampling is employed to improve the computational efficiency. The accuracy of the results obtained by Kriging model is verified by the reference results provided by the Monte Carlo simulation. Four examples are investigated to demonstrate the significance of the proposed failure probability-based global sensitivity index and the effectiveness of the computational method.

Proximal Decoding for LDPC Codes

This paper presents a novel optimization-based decoding algorithm for LDPC codes. The proposed decoding algorithm is based on a proximal gradient method for solving an approximate maximum a posteriori (MAP) decoding problem. The key idea of the proposed algorithm is the use of a code-constraint polynomial to penalize a vector far from a codeword as a regularizer in the approximate MAP objective function. A code proximal operator is naturally derived from a code-constraint polynomial. The proposed algorithm, called proximal decoding, can be described by a simple recursive formula consisting of the gradient descent step for a negative log-likelihood function corresponding to the channel conditional probability density function and the code proximal operation regarding the code-constraint polynomial. Proximal decoding is experimentally shown to be applicable to several non-trivial channel models such as LDPC-coded massive MIMO channels, correlated Gaussian noise channels, and nonlinear vector channels. In particular, in MIMO channels, proximal decoding outperforms known massive MIMO detection algorithms, such as an MMSE detector with belief propagation decoding. The simple optimization-based formulation of proximal decoding allows a way for developing novel signal processing algorithms involving LDPC codes.

Two-stage conditional density estimation based on Bernstein polynomials

Two-stage conditional probability density function estimators are proposed and studied. Specifically, the Nadaraya-Watson (NW) and local linear (LL) conditional distribution function estimators have been smoothed using Bernstein polynomials in the first stage. Second, the proposed estimators are obtained by differentiating NW and LL estimators. The asymptotic properties of these estimators are established such as asymptotic bias, variance, and normality. Finally, a simulation study is carried out to assess the relative advantage of our estimators compared to other competitors. In addition, the well-known Old Faithful Geyser data is analyzed using the proposed estimators.

A Bivariate Extension to Exponentiated Inverse Flexible Weibull Distribution: Shock Model, Features, and Inference to Model Asymmetric Data

The primary objective of this article was to introduce a new probabilistic model for the discussion and analysis of random covariates. The introduced model was derived based on the Marshall–Olkin shock model. After proposing the mathematical form of the new bivariate model, some of its distributional properties, including joint probability distribution, joint reliability distribution, joint reversed (hazard) rate distribution, marginal probability density function, conditional probability density function, moments, and distributions for both Y=max{X1,X2} and W=min{X1,X2}, were investigated. This novel model can be applied to discuss and evaluate symmetric and asymmetric data under various kinds of dispersion. Moreover, it can be used as a probability approach to analyze different shapes of hazard rates. The maximum likelihood approach was utilized for estimating the parameters of the bivariate model. A simulation study was carried out to assess the performance of the parameters, and it was noted that the maximum likelihood technique can be used to generate consistent estimators. Finally, two real datasets were analyzed to illustrate the notability of the novel bivariate distribution, and it was found that the suggested distribution provided a better fit than the competitive bivariate models.

Efficient conditional probability theorem and importance sampling-based methods for global reliability sensitivity analysis

Global reliability sensitivity (GRS) analysis can measure the effect of random inputs on failure probability (FP). To efficiently solve GRS, two conditional probability theorem (CPT)-based methods are proposed by combining adaptive Kriging (AK) with importance sampling (IS) (CPT-AK-IS) and combining AK with Meta-IS (CPT-AK-Meta-IS) respectively. Firstly, differentiation approximation and CPT are used to convert the estimation of conditional probability density function (PDF), which is required by the existing Bayes theorem-based methods, into that of a series of probabilities. Secondly, GRS can be directly estimated by the failure samples of IS, while the existing Bayes theorem methods based on IS need to transform the failure samples of IS into those of original PDF. Both the first and second strategies can reduce the computational complexity of solving GRS. Thirdly, by selecting a suitable differentiation interval with a proposed adaptive strategy, the estimation of a series of probabilities can be accurately completed as a byproduct of one IS based simulation for solving FP without additional computational cost. Finally, by introducing AK into IS and Meta-IS, it can reduce the number of evaluating performance function and the size of candidate sample pool simultaneously. These novelties are sufficiently verified by the presented examples.

Analysis of transition path ensemble in the exactly solvable models via overdamped langevin equation

Transition of a system between two states is an important but difficult problem in natural science. In this article we study the transition problem in the framework of transition path ensemble. Using the overdamped Langevin method, we introduce the path integral formulation of the transition probability and obtain the equation for the minimum action path in the transition path space. For the effective sampling in the transition path ensemble, we derive a conditional overdamped Langevin equation. In two exactly solvable models, the free particle system and the harmonic system, we present the expression of the conditional probability density and the explicit solutions for the conditional Langevin equation and the minimum action path. The analytic results demonstrate the consistence of the conditional Langevin equation with the desired probability distribution in the transition. It is confirmed that the conditional Langevin equation is an effective tool to sample the transition path ensemble, and the stationary action principle actually leads to the most probable path.

Pressure effects on soot formation and evolution in turbulent jet flames

In this study, two series of pressurized turbulent jet sooting flames at 1, 3, and 5 bar with either fixed jet velocity or fixed Reynolds number are simulated to study the pressure effects on soot formation and evolution. Through a radiation flamelet progress variable approach with a conditional soot subfilter probability density function (PDF) model to consider the turbulence–chemistry–soot interactions, quantitatively good agreements are achieved for soot volume fraction (SVF) predictions compared with the experimental data, regardless different turbulent intensities and residence times. SVF source terms are then discussed to show the pressure effects on nucleation, condensation, surface growth, and oxidation at different axial positions in these flames. It is found that surface growth and oxidation increase by about three orders of magnitude from 1 to 5 bar, while nucleation and condensation only increase within one order of magnitude. The stronger SVF scaling on pressure than measured data is found to be attributed to the inaccurate surface growth and oxidation scaling on pressure. Further analysis indicates that (i) the uncertainty of C2H2 prediction at elevated pressures is likely a major reason for the too strong surface growth scaling; and (ii) taking account of pressure effects in the conditional subfilter PDF modeling for turbulence–soot–chemistry interactions is likely a key to improve oxidation prediction. The results in this study open up the possibilities for improving future turbulent sooting flame modeling by improving C2H2 chemistry and turbulence–chemistry–soot modeling at elevated pressures.

A Multirobot Person Search System for Finding Multiple Dynamic Users in Human-Centered Environments.

Multirobot coordination for finding multiple users in an environment can be used in numerous robotic applications, including search and rescue, surveillance/monitoring, and activities of daily living assistance. Existing approaches have limited coordination between robots when generating team plans or do not consider user location probability within these plans. This results in long searches and robots potentially revisiting the same locations in succession. In this article, we present a novel multirobot person search system to generate search plans for multirobot teams to find multiple dynamic users before a deadline. Our approach is unique in that it simultaneously considers the search actions of all robots and user location probabilities when generating team plans, where user location probabilities are represented as conditional spatial-temporal probability density functions. We model this multirobot person search problem as a two-stage optimization problem to maximize the expected number of users found before the deadline. Stage 1 solves the action selection problem to determine a set of team actions, and the second stage solves the action allocation problem to distribute these actions amongst the robots. Namely, in stage 1, a novel conditional multiperiod multiknapsack problem is modeled as a min-flow graph solved sequentially by the Bellman-Ford shortest path algorithm. Stage 2 is a variant of the min-max multitraveling salesperson problem which models the environment topology as a search region network and search times selected by the previous stage. This stage is solved by a novel fuzzy clustering method. Numerous experiments comparing our proposed method to other existing approaches with varying environment sizes, search durations, and the number of users showed that our approach was able to find more target users before a defined deadline.

Function applicability to class conditional probability density of hourly rainfall

暴雨强度公式在水文、气象、工程设计等各领域都是非常关注的问题,而常用降雨概率分布函数的适用性欠缺,理论分布函数一直处于争鸣之中。从逐时降雨概率密度函数的适用性分析入手,有利于发现普适且恰当的理论密度函数。本文从我国暴雨洪涝灾害易发区中沿30°N选取4个经纬度长方形区域(雅安附近、鄂西南、江汉平原南部、杭州湾西),并在其南、北各选一对比分区(海南岛、郑州),对6个分区内降雨资料直接采用全样本,统计逐时降雨的三类概率密度经验函数,对照这些函数的特性,从理论上分析了众多分布函数的适用性,筛选适用函数并进行拟合试验,优选出理论密度函数。研究结果表明:三参广义伽玛函数拟合误差最小,而两参广义正态函数更恰当、被首推为理论密度函数;拟合参数寻优时的目标函数综合了乘性与加性误差模型,能使拟合曲线兼顾头尾;本研究有别于极值降雨概率分布中仅采用极少部分样本的方法,采用降雨概率密度方法替代传统的年极值法,使重现期计算更准确有效,能提高暴雨强度公式的科学性,拟合的高精度与函数的普适性有望解决降雨概率分布模型统一的问题。;Due to the deficiency of international-unified theoretical function for rainfall probability distribution, it is necessary to analyze the applicability, appropriateness and universality of commonly used probability density function(PDF). On the basis of summarizing the characteristics of three kinds of class conditional probability density of hourly rainfall, the applicability of 20 kinds of functions is theoretically analyzed and compared. The generalized Gamma distribution(GΓD), generalized normal distribution(GND) and Weibull distributions are selected as reference functions, and the genetic algorithm and genetic algorithms are used to carry out fitting experiments. When the fitting deviation is equivalent to the observation error, a necessary condition for the appropriateness of the theoretical distribution function is used for discriminant analysis. The fitting results show that the Weibull distribution performs worst, the GΓD has the smallest fitting deviation, and the GND performs best. Therefore the GND is recommended for theoretical distribution fitting. In addition, the genetic algorithm can directly obtain the optimal approximate solution through multi-objective parameter optimization, which can simplify the derivation and calculation steps. The multiplicative and additive deviation models are both used as objective functions, which can take the beginning and the end of fitting curves, as well as the absolute and relative deviations into account.

Discriminant analysis of environmental data based on zero inflated spatial auto-beta models

In this paper, a supervised classification problem of classifying feature observation into one of two populations is considered. Classical supervised classification in this work is expanded for non-Gaussian spatial data specified by zero inflated auto-beta models. A classification rule based on the Bayes discriminant function (BDF) that uses conditional probability density functions in the expression is proposed. This rule is applied to real data set for identifying sea bottom type by using Black carrageen concentration data over the southeastern Baltic Sea. Different feature observation models are chosen and classification results are compared using a hold-out error rate measure.

Probabilistic Interval Predictor Based on Dissimilarity Functions

This work presents a new methodology to obtain probabilistic interval predictions of a dynamical system. The proposed strategy uses stored past system measurements to estimate the future evolution of the system. The method relies on the use of dissimilarity functions to estimate the conditional probability density function of the outputs. A family of empirical probability density functions, parameterized by means of two scalars, is introduced. It is shown that the proposed family encompasses the multivariable normal probability density function as a particular case. We show that the presented approach constitutes a generalization of classical estimation methods. A validation scheme is used to tune the two parameters on which the methodology relies. In order to prove the effectiveness of the presented methodology, some numerical examples and comparisons are provided.

Non-parametric data-driven background modelling using conditional probabilities

Background modelling is one of the main challenges in particle physics data analysis. Commonly employed strategies include the use of simulated events of the background processes, and the fitting of parametric background models to the observed data. However, reliable simulations are not always available or may be extremely costly to produce. As a result, in many cases uncertainties associated with the accuracy or sample size of the simulation are the limiting factor in the analysis sensitivity. At the same time, parametric models are limited by the a priori unknown functional form and parameter values of the background distribution. These issues become ever more pressing when large datasets become available, as it is already the case at the CERN Large Hadron Collider, and when studying exclusive signatures involving hadronic backgrounds.A widely applicable approach for non-parametric data-driven background modelling is proposed, which addresses these issues for a broad class of searches and measurements. It relies on a relaxed version of the event selection to estimate conditional probability density functions and two different techniques are discussed for its realisation. The first relies on ancestral sampling and uses data from a relaxed event selection to estimate a graph of conditional probability density functions of the variables used in the analysis, while accounting for significant correlations. A background model is then generated from events drawn from this graph, on which the full event selection is applied. In the second, a novel generative adversarial network is trained to estimate the joint probability density function of the variables used in the analysis. The training is performed on a relaxed event selection, which excludes the signal region, and the network is conditioned on a blinding variable. Subsequently, the conditional probability density function is interpolated into the signal region to model the background. The application of each method on a benchmark analysis and on ensemble tests is presented in detail, and the performance is discussed.

Missing data imputation using utility-based regression and sampling approaches

Data are often missing not at random (MNAR) in scientific experiments. We treat the MNAR problem as an imbalanced learning task. Standard predictive error measures of regression (e.g., mean squared error) are not suitable for imbalanced learning problems, such as in clinical trials where extreme values tend to be MNAR. We investigate hybrid imbalanced learning approaches that combine utility-based regression (UBR) with synthetic minority oversampling technique for regression (SMOTER) in cross-sectional trial settings. UBR optimizes the product of the conditional probability density (estimated by quantile regression forests) and a utility function which takes the relevance of the target variable value and the prediction error into account. SMOTER oversamples the relevant rare cases. Simulations show that the proposed method provides plausible predictions and reduces the bias for realistic missing data scenarios when compared with standard approaches like random forests and multiple imputation (systematic bias is observed in those methods, i.e., a tendency to underestimate the mean and standard deviation given the presence of MNAR in the area of high values of the target variable). The proposed method is implemented in a real dataset from an antidepressant clinical trial, and similar pattern of the systematic bias from commonly used methods is observed in the real data compare to the proposed method. Therefore, we encourage the integration of utility-based learning strategies for handling of missing data in the analysis of clinical trials.