Unknown Probability Density Research Articles

Deep learning probability flows and entropy production rates in active matter

Active matter systems, from self-propelled colloids to motile bacteria, are characterized by the conversion of free energy into useful work at the microscopic scale. They involve physics beyond the reach of equilibrium statistical mechanics, and a persistent challenge has been to understand the nature of their nonequilibrium states. The entropy production rate and the probability current provide quantitative ways to do so by measuring the breakdown of time-reversal symmetry. Yet, their efficient computation has remained elusive, as they depend on the system's unknown and high-dimensional probability density. Here, building upon recent advances in generative modeling, we develop a deep learning framework to estimate the score of this density. We show that the score, together with the microscopic equations of motion, gives access to the entropy production rate, the probability current, and their decomposition into local contributions from individual particles. To represent the score, we introduce a spatially local transformer network architecture that learns high-order interactions between particles while respecting their underlying permutation symmetry. We demonstrate the broad utility and scalability of the method by applying it to several high-dimensional systems of active particles undergoing motility-induced phase separation (MIPS). We show that a single network trained on a system of 4,096 particles at one packing fraction can generalize to other regions of the phase diagram, including to systems with as many as 32,768 particles. We use this observation to quantify the spatial structure of the departure from equilibrium in MIPS as a function of the number of particles and the packing fraction.

Probabilistic neural networks for incremental learning over time-varying streaming data with application to air pollution monitoring

This paper proposes a novel algorithm for incremental learning over streaming data in a non-stationary environment. The idea refers to the applicability of Probabilistic Neural Networks (PNNs), commonly used as a fast and robust method for solving classification problems in a stationary scenario. It is well known that PNNs have solid mathematical foundations but they fail in the non-stationary scenario and are not able to deal with concept drift. In this paper, the method based on the orthogonal series expansions of unknown probability densities is proposed. This approach significantly extends the applicability of the PNNs. The nonparametric procedures called Incremental Probabilistic Neural Networks (IPNNs) for tracking drifting probability densities or conditional class densities, in the case of classification, are presented. We prove their convergence as the stream data size is growing to infinity. More precisely, we show convergence in probability and with probability one as the sample size tends to infinity. Compared to the existing literature, almost exclusively based on various heuristics, the proposed method is mathematically justified and has solid theoretical foundations. Simulations performed on synthetic and air pollution data confirm the effectiveness of the algorithm.

A Markov Chain Genetic Algorithm Approach for Non-Parametric Posterior Distribution Sampling of Regression Parameters

This paper proposes a genetic algorithm-based Markov Chain approach that can be used for non-parametric estimation of regression coefficients and their statistical confidence bounds. The proposed approach can generate samples from an unknown probability density function if a formal functional form of its likelihood is known. The approach is tested in the non-parametric estimation of regression coefficients, where the least-square minimizing function is considered the maximum likelihood of a multivariate distribution. This approach has an advantage over traditional Markov Chain Monte Carlo methods because it is proven to converge and generate unbiased samples computationally efficiently.

A novel maximum entropy method based on the B-spline theory and the low-discrepancy sequence for complex probability distribution reconstruction

The maximum entropy method (MEM) is a powerful tool for the recovery of unknown probability density functions (PDF) and has growing popularity in the reliability analysis community. However, MEM may be inaccurate for PDFs with a complex shape (e. g. multiple modals or a long tail), influencing the accuracy of the reliability analysis greatly. To overcome this deficiency, this study proposes a novel MEM paradigm based on the B-spline theory and the low-discrepancy sequence. Firstly, to enhance the performance of MEM for complex PDFs, the B-spline functions are used to construct the MEM PDF. Correspondingly, the iteration formulation is derived for the undetermined parameter estimation of the B-spline-based MEM PDF based on the closed solution for minimizing the Kullback-Leibler divergence. Then, we adopt the low-discrepancy sequence to calculate the objective function of minimizing the Kullback-Leibler divergence efficiently. Compared with MEM and other moment-based reliability analysis methods, the proposed method does not require the statistical moments, and integrates the advantages of the B-spline theory and MEM. To illustrate the benefits of our method, five examples are analyzed and compared with some classical reliability analysis methods.

Optimal Bandwidth Selection Methods with Application to Wind Speed Distribution

Accurate estimation of the unknown probability density functions of critical variables, such as wind speed—which plays a pivotal role in harnessing clean energy—is essential for various scientific and practical applications. This research conducts a comprehensive comparative analysis of seven distinct bandwidth calculation techniques across various normal distributions, using simulation as the evaluation method in the context of Kernel Density Estimation (KDE). This analysis includes the calculation of the optimal bandwidth and assessment of the performance of these methods with respect to Mean Squared Error (MSE), bias, and the optimal bandwidth value. The findings reveal that among the various bandwidth methods evaluated, the Bandwidth bandwidth-based Cross-Validation (BCV), especially for small sample sizes, consistently provides the closest result to the optimal bandwidth across most of the applied normal distributions. These results provide valuable insights into the selection of optimal bandwidths for accurate and reliable density estimation in the context of normal distributions. Another key aspect of this work is the extension of these methods to wind speed data in a specific region. Monthly wind speed kernel density estimates obtained using all seven bandwidth selection techniques show that Smoothed Cross-Validation (SCV) is suited for this type of real-world data.

A novel variational robust filter with Gaussian mixture model for unknown non-Gaussian noises

In numerous practical applications, the presence of unpredictable abnormal behaviors often leads to random measurement outliers, causing the random measurement noises are no longer Gaussian but unknown non-Gaussian distributions, and it is intractable to obtain accurate a priori information about the noises. To address this challenge and enable nonlinear state estimation under such measurement noises, we propose a novel robust filter based on Gaussian mixture model (GMM). Firstly, a GMM is introduced to model the unknown non-Gaussian measurement likelihood probability density function (ML-PDF). Without relying on any prior noise statistics, the GMM can accurately model the ML-PDF utilizing a set of model parameters adaptively learned by the variational Bayesian (VB) method. Then, considering the high computational burden associated with the existing variational iterative process, we develop an improved VB method, which independently calculates the posterior distributions of state and unknown model parameters, thereby improving the computational efficiency. Finally, a new robust filter is derived by integrating the GMM with the improved VB approach, The effectiveness and superiority of the filter are demonstrated through both numerical simulations and real-world data analysis.

Signal detection for FTN system under unknown noise model with imperfect CSI

The problem of signal detection for faster-than-Nyquist (FTN) signaling system with imperfect channel state information (CSI) is studied. Also, the non-Gaussian noise in real applications is considered. A general model, Gaussian mixture noise (GMN) model, is applied to describe the unknown probability density function (PDF) of the non-Gaussian noise. After that, a framework over variational inference is constructed to simplify the problem through finding an auxiliary PDF with factorization form. Based on the factor graph over the signal, joint response and the parameters of GMN, an iterative solution is obtained. Simulations demonstrate the effectiveness of the proposed approach.

Augmented Complex Minimization of Error Entropy With Fiducial Points for Power System Frequency Estimation

Frequency estimation plays an important role in ensuring the stability of the power system. The augmented complex minimum error entropy (ACMEE) algorithm has achieved better performance in frequency estimation under impulsive noise environments. However, it has the problem of unknown error probability density function (PDF) in frequency tracking, the improved algorithms called the complex minimization of error entropy with fiducial points (CMEEF) and the augmented CMEEF (ACMEEF) are proposed in this paper. The ACMEEF algorithm based on widely linear modeling uses second-order statistical information to process the non-circular signal through the complex voltage signal. At the same time, it incorporates a location-sensitive cost into minimum error entropy (MEE) to automatically locate the peak of the error PDF at the origin, which can accurately estimate the grid frequency under the unbalanced system and interference of noise. The stability of the proposed algorithm and the computational complexity are analyzed. Finally, the enhanced frequency estimation performance of the proposed ACMEEF algorithm is verified through simulation results of different synthetic signals and real-world data.

Superpolynomial quantum-classical separation for density modeling

Density modelling is the task of learning an unknown probability density function from samples, and is one of the central problems of unsupervised machine learning. In this work, we show that there exists a density modelling problem for which fault-tolerant quantum computers can offer a super-polynomial advantage over classical learning algorithms, given standard cryptographic assumptions. Along the way, we provide a variety of additional results and insights, of potential interest for proving future distribution learning separations between quantum and classical learning algorithms. Specifically, we (a) provide an overview of the relationships between hardness results in supervised learning and distribution learning, and (b) show that any weak pseudo-random function can be used to construct a classically hard density modelling problem. The latter result opens up the possibility of proving quantum-classical separations for density modelling based on weaker assumptions than those necessary for pseudo-random functions.

Efficient functional estimation and the super-oracle phenomenon

We consider the estimation of two-sample integral functionals, of the type that occur naturally, for example, when the object of interest is a divergence between unknown probability densities. Our first main result is that, in wide generality, a weighted nearest neighbour estimator is efficient, in the sense of achieving the local asymptotic minimax lower bound. Moreover, we also prove a corresponding central limit theorem, which facilitates the construction of asymptotically valid confidence intervals for the functional, having asymptotically minimal width. One interesting consequence of our results is the discovery that, for certain functionals, the worst-case performance of our estimator may improve on that of the natural ‘oracle’ estimator, which itself can be optimal in the related problem where the data consist of the values of the unknown densities at the observations.

Фильтрация сигналов при скачкообразных мультипликативных помехах

The problem of constructing the approximately optimal Bayesian algorithm of filtering the output signal for a linear stochastic dynamical system is solved. Non-linear mixture of the output signal and of the multiplicative interference was measured. This interference is a continuous-valued random process with the unknown distribution law within the given limits of [c(1), c(2)], c(1) > 0, c(2) > 0, and in the frequency range of [0, ∆ω]. The filtering algorithm is synthesized by the approximately optimal signal estimation method based on the theory of systems with random jump structure and the method of two-time parametric approximation of the phase coordinates probability distribution. The method consists in approximate replacement of the unknown probability densities of phase coordinates by the known distribution laws with the unknown mathematical expectations and covariances determined as a result of solving the problem. The proposed approximate-optimal filtering algorithm is based on replacement of the continuous-valued multiplicative interference by a random jump process, i.e., Markov chain with two states c(1), c(2) and equal intensities of transitions from one state to another q′ = 2∆ω. Conditional probability densities of the output signal for fixed states of the Markov chain c(1), c(2) are approximated by gamma distributions that depend on mathematical expectations and variances of the x(t) signal at the fixed c(1), c(2) and measurements x(t) in a mixture with the Markov jump interference. An example of constructing an algorithm for evaluating the distance from an aircraft to the object based on object irradiance measurement by the infrared direction finder was considered. Irradiance was equal to the radiation strength ratio to the square of the distance. Radiation strength, i.e., multiplicative interference, was the random process with unknown continuous distribution in a limited range. The approximate-optimal filter was constructed for distance evaluation by replacing this process with a two-state Markov chain and approximating the distance probability density by the Rayleigh distribution

Convergence of Policy Gradient Methods for Nash Equilibria in General-sum Stochastic Games

We study Nash equilibria learning of a general-sum stochastic game with an unknown transition probability density function. Agents take actions at the current environment state and their joint action influences the transition of the environment state and their immediate rewards. Each agent only observes the environment state and its own immediate reward and is unknown about the actions or immediate rewards of others. We introduce the concept of weighted asymptotic Nash equilibrium with probability 1 and design a two-loop algorithm by the equivalence of Nash equilibrium and variational inequality problems. In the outer loop, we sequentially update a constructed strongly monotone variational inequality by updating a proximal parameter while employing a single-call extra-gradient algorithm in the inner loop for solving the constructed variational inequality. We show that if the associated Minty variational inequality has a solution, then the designed algorithm converges to the k1/2 -weighted asymptotic Nash equilibrium.

A MOLLIFIER APPROACH TO THE DECONVOLUTION OF PROBABILITY DENSITIES

We use mollification to regularize the problem of deconvolution of random variables. This regularization method offers a unifying and generalizing framework in order to compare the benefits of various filter-type techniques like deconvolution kernels, Tikhonov, or spectral cutoff methods. In particular, the mollifier approach allows to relax some restrictive assumptions required for the deconvolution kernels, and has better stabilizing properties compared with spectral cutoff or Tikhonov. We show that this approach achieves optimal rates of convergence for both finitely and infinitely smoothing convolution operators under Besov and Sobolev smoothness assumptions on the unknown probability density. The qualification can be arbitrarily high depending on the choice of the mollifier function. We propose an adaptive choice of the regularization parameter using the Lepskiĭ method, and we provide simulations to compare the finite sample properties of our estimator with respect to the well-known regularization methods.

How much BiGAN and CycleGAN-learned hidden features are effective for COVID-19 detection from CT images? A comparative study

Bidirectional generative adversarial networks (BiGANs) and cycle generative adversarial networks (CycleGANs) are two emerging machine learning models that, up to now, have been used as generative models, i.e., to generate output data sampled from a target probability distribution. However, these models are also equipped with encoding modules, which, after weakly supervised training, could be, in principle, exploited for the extraction of hidden features from the input data. At the present time, how these extracted features could be effectively exploited for classification tasks is still an unexplored field. Hence, motivated by this consideration, in this paper, we develop and numerically test the performance of a novel inference engine that relies on the exploitation of BiGAN and CycleGAN-learned hidden features for the detection of COVID-19 disease from other lung diseases in computer tomography (CT) scans. In this respect, the main contributions of the paper are twofold. First, we develop a kernel density estimation (KDE)-based inference method, which, in the training phase, leverages the hidden features extracted by BiGANs and CycleGANs for estimating the (a priori unknown) probability density function (PDF) of the CT scans of COVID-19 patients and, then, in the inference phase, uses it as a target COVID-PDF for the detection of COVID diseases. As a second major contribution, we numerically evaluate and compare the classification accuracies of the implemented BiGAN and CycleGAN models against the ones of some state-of-the-art methods, which rely on the unsupervised training of convolutional autoencoders (CAEs) for attaining feature extraction. The performance comparisons are carried out by considering a spectrum of different training loss functions and distance metrics. The obtained classification accuracies of the proposed CycleGAN-based (resp., BiGAN-based) models outperform the corresponding ones of the considered benchmark CAE-based models of about 16% (resp., 14%).

Uncertainty Evaluation of Stochastic Structural Response with Correlated Random Variables

It has been realized that the influence of system parameter uncertainties may be very significant, even dominant, in stochastic response evaluation. Nevertheless, in reality, this evaluation process may be difficult to conduct due to these parameter variables (viz. structural property parameters, such as stiffness, damping, and strength, and excitation characteristics parameters, such as frequency content and duration) that are usually correlated with each other. Therefore, this study devotes to develop a method for evaluating stochastic response uncertainty involving correlated system parameter variables. In this method, the evaluation expression for the mean and standard deviation of the maximum response including uncertainty parameter variables are provided first; subsequently, a third-moment pseudo-correlation normal transformation is able to be performed for converting the correlated and non-normal system parameter variables with unknown joint probability density function (PDF) or marginal PDF into the mutually independent standard normal ones; ultimately, a point estimate procedure (PEP) based on univariate dimension reduction integration can be carried out for evaluating the structural stochastic response including uncertainty system parameters. Several numerical examples with an engineering background involving correlated system parameter variables are analyzed and discussed under stochastic excitation, and their results are compared with those yielded by Monte Carlo simulation (MCS) so as to demonstrate the effectiveness of the approach proposed. It indicated that the method proposed, in this study, provides an effective path to deal with uncertainty evaluation of stochastic structural response involving correlated random variables.

An Optimal Transport Approach for Selecting a Representative Subsample with Application in Efficient Kernel Density Estimation

Subsampling methods aim to select a subsample as a surrogate for the observed sample. Such methods have been used pervasively in large-scale data analytics, active learning, and privacy-preserving analysis in recent decades. Instead of model-based methods, in this article, we study model-free subsampling methods, which aim to identify a subsample, that is, not confined by model assumptions. Existing model-free subsampling methods are usually built upon clustering techniques or kernel tricks. Most of these methods suffer from either a large computational burden or a theoretical weakness. In particular, the theoretical weakness is that the empirical distribution of the selected subsample may not necessarily converge to the population distribution. Such computational and theoretical limitations hinder the broad applicability of model-free subsampling methods in practice. We propose a novel model-free subsampling method by using optimal transport techniques. Moreover, we develop an efficient subsampling algorithm, that is, adaptive to the unknown probability density function. Theoretically, we show the selected subsample can be used for efficient density estimation by deriving the convergence rate for the proposed subsample kernel density estimator. We also provide the optimal bandwidth for the proposed estimator. Numerical studies on synthetic and real-world datasets demonstrate the performance of the proposed method is superior.

Smoothing Splines Approximation Using Hilbert Curve Basis Selection

Smoothing splines have been used pervasively in nonparametric regressions. However, the computational burden of smoothing splines is significant when the sample size n is large. When the number of predictors , the computational cost for smoothing splines is at the order of using the standard approach. Many methods have been developed to approximate smoothing spline estimators by using q basis functions instead of n ones, resulting in a computational cost of the order . These methods are called the basis selection methods. Despite algorithmic benefits, most of the basis selection methods require the assumption that the sample is uniformly distributed on a hypercube. These methods may have deteriorating performance when such an assumption is not met. To overcome the obstacle, we develop an efficient algorithm that is adaptive to the unknown probability density function of the predictors. Theoretically, we show the proposed estimator has the same convergence rate as the full-basis estimator when q is roughly at the order of , where and are some constants depend on the type of the spline. Numerical studies on various synthetic datasets demonstrate the superior performance of the proposed estimator in comparison with mainstream competitors. Supplementary files for this article are available online.

IMM Robust Cardinality Balance Multi‐Bernoulli Filter for Multiple Maneuvering Target Tracking with Interval Measurement

This paper presents a novel Interacting multi-model (IMM) Robust Cardinality balance multi-target multi-Bernoulli (R-CBMeMBer) filter to solve the maneuvering target tracking problem in the case of interval measurement, unknown detection probability and unknown clutter density. In essence, IMM R-CBMeMBer filter is an extended application of R-CBMeMBer filter. In the IMM R-CBMeMBer filter, the target state is first extended to distinguish clutter from the real target. The detection probability and model probability of the target can be adaptively updated. Then, generalized likelihood function and IMM algorithm are introduced to interactively predict and update the state of the target in the IMM R-CBMeMBer filtering process. In addition, a particle application of the IMM R-CBMeMBer filter is given, and a numerical experiment is designed under nonlinear conditions. Meanwhile, Doppler information of the target is employed to estimate the velocity of each maneuvering target. Numerical experiments also verify that the IMM R-CBMeMBer filter can effectively estimate the target position, target velocity, target detection probability and clutter number in the condition of unknown detection probability, unknown clutter rate and interval measurement.

Derivation of Equations for a Size Distribution of Spherical Particles in Non-Transparent Materials

This paper presents a new proposition on how to derive mathematical formulas that describe an unknown Probability Density Function (PDF3) of the spherical radii (r3) of particles randomly placed in non-transparent materials. We have presented two attempts here, both of which are based on data collected from a random planar cross-section passed through space containing three-dimensional nodules. The first attempt uses a Probability Density Function (PDF2) the form of which is experimentally obtained on the basis of a set containing two-dimensional radii (r2). These radii are produced by an intersection of the space by a random plane. In turn, the second solution also uses an experimentally obtained Probability Density Function (PDF1). But the form of PDF1 has been created on the basis of a set containing chord lengths collected from a cross-section.The most important finding presented in this paper is the conclusion that if the PDF1 has proportional scopes, the PDF3 must have a constant value in these scopes. This fact allows stating that there are no nodules in the sample space that have particular radii belonging to the proportional ranges the PDF1.

Adaptive nonparametric estimation of a component density in a two-class mixture model

A two-class mixture model, where the density of one of the components is known, is considered. We address the issue of the nonparametric adaptive estimation of the unknown probability density of the second component. We propose a randomly weighted kernel estimator with a fully data-driven bandwidth selection method, in the spirit of the Goldenshluger and Lepski method. An oracle-type inequality for the pointwise quadratic risk is derived as well as convergence rates over Hölder smoothness classes. The theoretical results are illustrated by numerical simulations.