High-dimensional Probability Research Articles

Accurate estimation of the normalized mutual information of multidimensional data.

While the linear Pearson correlation coefficient represents a well-established normalized measure to quantify the inter-relation of two stochastic variables X and Y, it fails for multidimensional variables, such as Cartesian coordinates. Avoiding any assumption about the underlying data, the mutual information I(X, Y) does account for multidimensional correlations. However, unlike the normalized Pearson correlation, it has no upper bound (I ∈ [0, ∞)), i.e., it is not clear if say, I = 0.4 corresponds to a low or a high correlation. Moreover, the mutual information (MI) involves the estimation of high-dimensional probability densities (e.g., six-dimensional for Cartesian coordinates), which requires a k nearest-neighbor algorithm, such as the estimator by Kraskov et al. [Phys. Rev. E 69, 066138 (2004)]. As existing methods to normalize the MI cannot be used in connection with this estimator, a new approach is presented, which uses an entropy estimation method that is invariant under variable transformations. The algorithm is numerically efficient and does not require more effort than the calculation of the (un-normalized) MI. After validating the method by applying it to various toy models, the normalized MI between the Cα-coordinates of T4 lysozyme is considered and compared to a correlation analysis of inter-residue contacts.

Hilbert Curve Projection Distance for Distribution Comparison.

Distribution comparison plays a central role in many machine learning tasks like data classification and generative modeling. In this study, we propose a novel metric, called Hilbert curve projection (HCP) distance, to measure the distance between two probability distributions with low complexity. In particular, we first project two high-dimensional probability distributions using Hilbert curve to obtain a coupling between them, and then calculate the transport distance between these two distributions in the original space, according to the coupling. We show that HCP distance is a proper metric and is well-defined for probability measures with bounded supports. Furthermore, we demonstrate that the modified empirical HCP distance with the Lp cost in the d-dimensional space converges to its population counterpart at a rate of no more than O(n-1/2max{d,p}). To suppress the curse-of-dimensionality, we also develop two variants of the HCP distance using (learnable) subspace projections. Experiments on both synthetic and real-world data show that our HCP distance works as an effective surrogate of the Wasserstein distance with low complexity and overcomes the drawbacks of the sliced Wasserstein distance.

Quantum circuits for discrete graphical models

Graphical models are useful tools for describing structured high-dimensional probability distributions. The development of efficient algorithms for generating samples thereof remains an active research topic. In this work, we provide a quantum algorithm that enables the generation of unbiased and independent samples from general discrete graphical models. To this end, we identify a coherent embedding of the graphical model based on a repeat-until-success sampling scheme which clearly identifies whether a drawn sample represents the underlying distribution. Furthermore, we show that the success probability for finding a valid sample can be lower bounded with a quantity that depends on the number of maximal cliques and the model parameter norm. Moreover, we rigorously proof that the quantum embedding conserves the key property of graphical models, i.e., factorization over the cliques of the underlying conditional independence structure. The quantum sampling algorithm also allows for maximum likelihood learning as well as maximum a posteriori state approximation for the graphical model. Finally, the proposed quantum method shows potential to realize interesting problems on near-term quantum processors. In fact, illustrative experiments demonstrate that our method can carry out sampling and parameter learning not only with idealized simulations of quantum computers but also on actual quantum hardware solely supported by simple readout error mitigation.

Optimal allocation of renewable energy systems in a weak distribution network

In the context of a real electrical distribution network in Puerto Carreño, Colombia, this paper addresses the challenge of enhancing power supply reliability through the strategic integration of Renewable Energy Sources with Hydrogen Energy Systems. The planning problem is reformulated as a Mixed-Integer Quadratically Constrained Programming problem, enabling the use of efficient solvers with global optimal solution. Considering the extensive curtailment options and numerous allocation components, the optimization problem naturally scales up. To address this computational intensity, representative scenarios are derived from the original high-dimensional probability distribution employing the Gaussian Mixture Model. These scenarios are used in conjunction with the proposed optimization framework to achieve cost-effective power grids while reducing fossil fuel dependency.

Deep learning probability flows and entropy production rates in active matter

Active matter systems, from self-propelled colloids to motile bacteria, are characterized by the conversion of free energy into useful work at the microscopic scale. They involve physics beyond the reach of equilibrium statistical mechanics, and a persistent challenge has been to understand the nature of their nonequilibrium states. The entropy production rate and the probability current provide quantitative ways to do so by measuring the breakdown of time-reversal symmetry. Yet, their efficient computation has remained elusive, as they depend on the system's unknown and high-dimensional probability density. Here, building upon recent advances in generative modeling, we develop a deep learning framework to estimate the score of this density. We show that the score, together with the microscopic equations of motion, gives access to the entropy production rate, the probability current, and their decomposition into local contributions from individual particles. To represent the score, we introduce a spatially local transformer network architecture that learns high-order interactions between particles while respecting their underlying permutation symmetry. We demonstrate the broad utility and scalability of the method by applying it to several high-dimensional systems of active particles undergoing motility-induced phase separation (MIPS). We show that a single network trained on a system of 4,096 particles at one packing fraction can generalize to other regions of the phase diagram, including to systems with as many as 32,768 particles. We use this observation to quantify the spatial structure of the departure from equilibrium in MIPS as a function of the number of particles and the packing fraction.

Generative deep learning for data generation in natural hazard analysis: motivations, advances, challenges, and opportunities

Data mining and analysis are critical for preventing or mitigating natural hazards. However, data availability in natural hazard analysis is experiencing unprecedented challenges due to economic, technical, and environmental constraints. Recently, generative deep learning has become an increasingly attractive solution to these challenges, which can augment, impute, or synthesize data based on these learned complex, high-dimensional probability distributions of data. Over the last several years, much research has demonstrated the remarkable capabilities of generative deep learning for addressing data-related problems in natural hazards analysis. Data processed by deep generative models can be utilized to describe the evolution or occurrence of natural hazards and contribute to subsequent natural hazard modeling. Here we present a comprehensive review concerning generative deep learning for data generation in natural hazard analysis. (1) We summarized the limitations associated with data availability in natural hazards analysis and identified the fundamental motivations for employing generative deep learning as a critical response to these challenges. (2) We discuss several deep generative models that have been applied to overcome the problems caused by limited data availability in natural hazards analysis. (3) We analyze advances in utilizing generative deep learning for data generation in natural hazard analysis. (4) We discuss challenges associated with leveraging generative deep learning in natural hazard analysis. (5) We explore further opportunities for leveraging generative deep learning in natural hazard analysis. This comprehensive review provides a detailed roadmap for scholars interested in applying generative models for data generation in natural hazard analysis.

Quantifying Unbiased Conformational Ensembles from Biased Simulations Using ShapeGMM.

Quantifying the conformational ensembles of biomolecules is fundamental to describing mechanisms of processes such as protein folding, interconversion between folded states, ligand binding, and allosteric regulation. Accurate quantification of these ensembles remains a challenge for conventional molecular simulations of all but the simplest molecules due to insufficient sampling. Enhanced sampling approaches, such as metadynamics, were designed to overcome this challenge; however, the nonuniform frame weights that result from many of these approaches present an additional challenge to ensemble quantification techniques such as Markov State Modeling or structural clustering. Here, we present rigorous inclusion of nonuniform frame weights into a structural clustering method entitled shapeGMM. The result of frame-weighted shapeGMM is a high dimensional probability density and generative model for the unbiased system from which we can compute important thermodynamic properties such as relative free energies and configurational entropy. The accuracy of this approach is demonstrated by the quantitative agreement between GMMs computed by Hamiltonian reweighting and direct simulation of a coarse-grained helix model system. Furthermore, the relative free energy computed from a shapeGMM probability density of alanine dipeptide reweighted from a metadynamics simulation quantitatively reproduces the underlying free energy in the basins. Finally, the method identifies hidden structures along the actin globular to filamentous-like structural transition from a metadynamics simulation on a linear discriminant analysis coordinate trained on GMM states, illustrating how structural clustering of biased data can lead to biophysical insight. Combined, these results demonstrate that frame-weighted shapeGMM is a powerful approach to quantifying biomolecular ensembles from biased simulations.

Meta-model based sequential importance sampling method for structural reliability analysis under high dimensional small failure probability

Reliability analysis poses a significant challenge for complex structures with stringent reliability requirements. While Sequential Importance Sampling (SIS) and Subset Simulation (SUS) have proven highly effective in addressing high-dimensional problems with small failure probabilities, the computational burden of mechanical simulations remains substantial due to the time-consuming nature of numerical simulation processes. Consequently, this paper introduces a novel approach, denoted as AK-SIS, which combines SIS with Kriging metamodeling specifically designed to address computational challenges associated with small failure probabilities. The fundamental principle of this approach involves utilizing AK-MCS technology (Echard et al., 2011) [1] as a precursor to the SIS approach to initially generate metamodels. These metamodels are then employed in lieu of performance functions in subsequent steps, significantly reducing the number of function calls required to simulate complex engineering problems when applying SIS techniques directly. By inheriting the advantages of SIS, AK-SIS has demonstrated its suitability for reliability analysis in scenarios involving high-dimensional spaces and small fault probabilities. Furthermore, AK-SIS is not limited by the shape of the failure domain, eliminates the need to solve the design point, and is particularly well-suited for analyzing reliability in cases of discontinuous failure domains, multiple failure domains, as well as complex failure domains and rare events. The efficacy of AK-SIS is substantiated through rigorous evaluation encompassing nonlinear, high-dimensional examples, and an engineering application. These empirical validations collectively contribute to a robust methodological framework for reliability analysis of intricate structures characterized by stringent reliability requirements.

High-dimensional and permutation invariant anomaly detection

Methods for anomaly detection of new physics processes are often limited to low-dimensional spaces due to the difficulty of learning high-dimensional probability densities. Particularly at the constituent level, incorporating desirable properties such as permutation invariance and variable-length inputs becomes difficult within popular density estimation methods. In this work, we introduce a permutation-invariant density estimator for particle physics data based on diffusion models, specifically designed to handle variable-length inputs. We demonstrate the efficacy of our methodology by utilizing the learned density as a permutation-invariant anomaly detection score, effectively identifying jets with low likelihood under the background-only hypothesis. To validate our density estimation method, we investigate the ratio of learned densities and compare to those obtained by a supervised classification algorithm.

Importance sampling enhanced by adaptive two-stage Kriging model and active subspace for analyzing rare probability with high dimensional input vector

For the high dimensional and small failure probability events widely existing in engineering, structural reliability analysis method still faces challenges at present. Since active subspace (AS) method can effectively reduce the dimension of input vector space, and metamodel-based importance sampling (IS) can efficiently estimate small failure probability, this paper combines AS with IS to propose an adaptive two-stage Kriging model method (AS-IS-AK2) for rare failure event with high dimensional problem. The proposed AS-IS-AK2 includes two stages. In the first stage, a new learning function is proposed to construct the Kriging model for roughly estimating the performance function values of all IS samples, and the gradient information is provided as a by-product for AS. In the second stage, an adaptive Kriging method is constructed in the reduced dimension space to accurately judge the states of all IS samples. Since AS reduces the dimension of input vector space, and IS reduces the required size of candidate sample pool for obtaining the convergent failure probability estimation, the computational complexity is reduced in reconstructing the Kriging model to judge the states of IS samples. The results of three examples verify the advantage of the proposed method for high dimensional problem with small failure probability.

Visualizing high-dimensional loss landscapes with Hessian directions

Analyzing the geometric properties of high-dimensional loss functions, such as local curvature and the existence of other optima around a certain point in loss space, can help provide a better understanding of the interplay between neural-network structure, implementation attributes, and learning performance. In this paper, we combine concepts from high-dimensional probability and differential geometry to study how curvature properties in lower-dimensional loss representations depend on those in the original loss space. We show that saddle points in the original space are rarely correctly identified as such in the expected lower-dimensional representations if random projections are used. The principal curvature in the expected lower-dimensional representation is proportional to the mean curvature in the original loss space. Hence, the mean curvature in the original loss space determines if saddle points appear, on average, as either minima, maxima, or almost flat regions. We use the connection between expected curvature in random projections and mean curvature in the original space (i.e. the normalized Hessian trace) to compute Hutchinson-type trace estimates without calculating Hessian-vector products as in the original Hutchinson method. Because random projections are not suitable for correctly identifying saddle information, we propose to study projections along the dominant Hessian directions that are associated with the largest and smallest principal curvatures. We connect our findings to the ongoing debate on loss landscape flatness and generalizability. Finally, for different common image classifiers and a function approximator, we show and compare random and Hessian projections of loss landscapes with up to approximately 7×106 parameters.

Times Square Sampling: An Adaptive Algorithm for Free Energy Estimation

Estimating free energy differences, an important problem in computational drug discovery and in a wide range of other application areas, commonly involves a computationally intensive process of sampling a family of high-dimensional probability distributions and a procedure for computing estimates based on those samples. The variance of the free energy estimate of interest typically depends strongly on how the total computational resources available for sampling are divided among the distributions, but determining an efficient allocation is difficult without sampling the distributions. Here we introduce the Times Square sampling algorithm, a novel on-the-fly estimation method that dynamically allocates resources in such a way as to significantly accelerate the estimation of free energies and other observables, while providing rigorous convergence guarantees for the estimators. We also show that it is possible, surprisingly, for on-the-fly free energy estimation to achieve lower asymptotic variance than the maximum-likelihood estimator MBAR, raising the prospect that on-the-fly estimation could reduce variance in a variety of other statistical applications. Supplementary materials for this article are available online.

Learning lattice quantum field theories with equivariant continuous flows

We propose a novel machine learning method for sampling from the high-dimensional probability distributions of Lattice Field Theories, which is based on a single neural ODE layer and incorporates the full symmetries of the problem. We test our model on the \phi^4ϕ4 theory, showing that it systematically outperforms previously proposed flow-based methods in sampling efficiency, and the improvement is especially pronounced for larger lattices. Furthermore, we demonstrate that our model can learn a continuous family of theories at once, and the results of learning can be transferred to larger lattices. Such generalizations further accentuate the advantages of machine learning methods.

Adaptive active subspace-based metamodeling for high-dimensional reliability analysis

To address the challenges of reliability analysis in high-dimensional probability spaces, this paper proposes a new metamodeling method that couples active subspace, heteroscedastic Gaussian process, and active learning. The active subspace is leveraged to identify low-dimensional salient features of a high-dimensional computational model. A surrogate computational model is built in the low-dimensional feature space by a heteroscedastic Gaussian process. Active learning adaptively guides the surrogate model training toward the critical region that significantly contributes to the failure probability. A critical trait of the proposed method is that the three main ingredients–active subspace, heteroscedastic Gaussian process, and active learning–are coupled to adaptively optimize the feature space mapping in conjunction with the surrogate modeling. This coupling empowers the proposed method to accurately solve nontrivial high-dimensional reliability problems via low-dimensional surrogate modeling. Finally, numerical examples of a high-dimensional nonlinear function and structural engineering applications are investigated to verify the performance of the proposed method.

Bidirectionally self-normalizing neural networks

The problem of vanishing and exploding gradients has been a long-standing obstacle that hinders the effective training of neural networks. Despite various tricks and techniques that have been employed to alleviate the problem in practice, there still lacks satisfactory theories or provable solutions. In this paper, we address the problem from the perspective of high-dimensional probability theory. We provide a rigorous result that shows, under mild conditions, how the vanishing/exploding gradients problem disappears with high probability if the neural networks have sufficient width. Our main idea is to constrain both forward and backward signal propagation in a nonlinear neural network through a new class of activation functions, namely Gaussian–Poincaré normalized functions, and orthogonal weight matrices. Experiments on both synthetic and real-world data validate our theory and confirm its effectiveness on very deep neural networks when applied in practice.

Matrix deviation inequality for ℓp-norm

Motivated by the general matrix deviation inequality for i.i.d. ensemble Gaussian matrix [R. Vershynin, High-Dimensional Probability: An Introduction with Applications in Data Science, Cambridge Series in Statistical and Probabilistic Mathematics (Cambridge University Press, 2018), doi:10.1017/9781108231596 of Theorem 11.1.5], we show that this property holds for the [Formula: see text]-norm with [Formula: see text] and i.i.d. ensemble sub-Gaussian matrices, i.e. random matrices with i.i.d. mean-zero, unit variance, sub-Gaussian entries. As a consequence of our result, we establish the Johnson–Lindenstrauss lemma from [Formula: see text]-space to [Formula: see text]-space for all i.i.d. ensemble sub-Gaussian matrices.

Detecting human interventions by spatial dependence of extreme water levels using a high dimensional conditional probability approach over the Pearl River Delta

The Pearl River Delta (PRD) is one of the largest urbanized areas in both area and population and also suffers increasing flooding hazards in the context of climate change and anthropogenic interventions. The co-occurrence of multiple upstream flooding drivers (compound flooding) over different upstream channels often exacerbates stronger impacts on downstream water levels compared with their isolated occurrence. It is particularly important to derive more reliable risk estimations of extreme water levels in downstream channels of a nested river system by considering the spatial dependence between multiple upstream floods and downstream water levels. In this study, we present an approach, termed the high-dimensional conditional probability (HDCP) approach, to capture the probabilistic dependence behavior of upstream floods and downstream water level events at the PRD, and estimate the extreme water levels' likelihood and event-impacted areas under the multiple upstream flooding. The PRD, as one of the most vulnerable regions to extreme floods in China, was used as a case study to test the applicability of the HDCP approach. First, the HDCP is used to model the conditional dependence of upstream floods and downstream water levels based on the canonical vine copula. Then, different flood scenarios are investigated by estimating the occurrence likelihood of downstream water level events and their spatial patterns. The final part identifies the changes in the occurrence likelihood variability of extreme water level events after strong human interventions. Results indicate that the upstream flooding over the Xijiang River dominates the occurrence likelihood of extreme water level events over the PRD, and the Beijiang River plays a secondary role. After strong human interventions, the effect of the Beijiang River has been strengthened, while the effect of the Xijaing River has been weakened. The proposed HDCP approach is not only applicable to the PRD area dominated by multiple upstream rivers, as mentioned in this study, but it is also promising in detecting the spatial occurrence likelihood of other hydrometeorological variables in other areas with different climates and topographical conditions, and formulating regional development strategies for policymakers in water resources and disaster drought risk managements against single/compound extreme events.

On the phase transition of [formula omitted]-out-of-[formula omitted] systems with applications to optimal maintenance

An effective method for improving the reliability of a technical system is to build redundancy into it. A commonly used redundancy structure in reliability engineering is the k-out-of-n structure. A k-out-of-n structure, consisting of n components, operates if and only if at least k components operate. This paper studies a high-dimension feature of k-out-of-n systems consisting of independent and identical components. This phenomenon, commonly known as a phase transition in high-dimensional probability, shows that in a system consisting of a large number of components, if the components operate independently with common reliability, then the system reliability undergoes a ‘sharp transition’. That is, in a narrow interval, the value of the system reliability rapidly declines. We first consider the systems whose components have common static reliability p∈(0,1). Then we focus on k-out-of-n systems with time-dependent reliability. As an application of the results, we employ the phase transition property to propose optimal age-based maintenance for the system.

Testing the boundaries: Normalizing Flows for higher dimensional data sets

Normalizing Flows (NFs) are emerging as a powerful class of generative models, as they not only allow for efficient sampling, but also deliver, by construction, density estimation. They are of great potential usage in High Energy Physics (HEP), where complex high dimensional data and probability distributions are everyday’s meal. However, in order to fully leverage the potential of NFs it is crucial to explore their robustness as data dimensionality increases. Thus, in this contribution, we discuss the performances of some of the most popular types of NFs on the market, on some toy data sets with increasing number of dimensions.

Editorial: Tensor numerical methods and their application in scientific computing and data science

Editorial: Tensor numerical methods and their application in scientific computing and data science