Low-dimensional Distributions Research Articles

More General Wall Pressure Spectra Models: Combining Feature Engineering with Evolutionary Algorithms

This paper presents an improved mathematical expression for semi-empirical wall pressure spectra modeling based on gene expression programming (GEP). The main focus of this work is to obtain a model that applies to a wide range of cases in terms of parameters and the source of data. The dataset comprises flat plate and airfoil cases with adverse and favorable pressure gradients at various Reynolds numbers. First, a characterization of the dataset is performed to understand the low-dimensional distribution of parameters. Then, a feature importance study is conducted to choose the most suitable model input variables from the exhaustive list of nondimensional parameters. The GEP algorithm is modified to ensure that trained models adhere to the basic structure of previously published semi-empirical models. Following training on the diverse database, the new model is compared against existing, best-performing empirical models to quantify the performance improvements. The models are tested on cases with completely different configurations and parameter ranges, unseen during training, and maintain their superior performance. Finally, a comparison is made between models developed with GEP and neural networks in terms of their efficacy, complexity, and interpretability.

Registration of Deformed Tissue: A GNN-VAE Approach With Data Assimilation for Sim-to-Real Transfer

In image-guided surgery, deformation of soft tissues can cause substantial errors in targeting internal targets, since deformation can affect the translation of preoperative image-based surgical plans during surgery. Having a realistic tissue deformation simulator could enhance the accuracy of internal targets localization by giving an accurate estimation of the deformation applied to a preoperative model of the organ. A key challenge is to address the sim-to-real gap between the simulator and the actual intraoperative behavior of the tissue. The sim-to-real transfer challenge is addressed by formulating the problem as a probabilistic inference over a low-dimensional representation of deformed objects. The proposed method utilizes a generative variational autoencoder structure based on graph neural networks (GNN-VAE) to generate a probabilistic low-dimensional representation of the outputs of a physics-based simulator. To match simulation data to real data, the resultant low-dimensional distribution (i.e., prior distribution) is updated iteratively using an ensemble smoother with multiple data assimilation. The advantages of the proposed method are first, it only uses simulation data for training the GNN-VAE, and no retraining of GNN-VAE is required intraoperatively, second, it does not require estimating the mechanical properties of the tissue it is simulating, and third, is able to work with any physic-based simulator. The proposed framework was verified both in experimental and simulation studies and showed it can reduce the registration error in tissue deformation.

A Point Cloud Classification Method and Its Applications Based on Multi-Head Self-Attention

<p>In the monitoring the safety status of hazardous chemical warehouses by three-dimensional re-construction of deep camera point clouds, there are classification difficulties such as large space, sparse distribution of point clouds in cargo images, and similar distribution in low dimensions. Based on the above problem, a point cloud recognition method based on multi-head attention mechanism is proposed. The algorithm first normalizes the distribution of the point cloud data set through the affine transformation algorithm to solve the problem of sparse distribution. Then, the high-dimensional feature map is obtained by fusing the data down-sampling and curve feature aggregation algorithms to solve the problem of low-dimensional distribution approximation. The feature map is then encoded using a multi-head self-attention encoder to obtain features under different heads, which are then merged into a feature map. Finally, a multi-layer fully connected neural network is used as the decoder to decode the feature map into the final object classification. Comparative experiments were performed on the ModelNet40 dataset and the self-built dataset of warehouse goods, and the results showed that the accuracy of this paper was improved by 0.5% to 7.8% compared with that of other classification algorithms.</p> <p> </p>

Probabilistic neural data fusion for learning from an arbitrary number of multi-fidelity data sets

In many applications in engineering and sciences analysts have simultaneous access to multiple data sources. In such cases, the overall cost of acquiring information can be reduced via data fusion or multi-fidelity (MF) modeling where one leverages inexpensive low-fidelity (LF) sources to reduce the reliance on expensive high-fidelity (HF) data. In this paper, we employ neural networks (NNs) for data fusion in scenarios where data is very scarce and obtained from an arbitrary number of sources with varying levels of fidelity and cost. We introduce a unique NN architecture that converts MF modeling into a nonlinear manifold learning problem. Our NN architecture inversely learns non-trivial (e.g., non-additive and non-hierarchical) biases of the LF sources in an interpretable and visualizable manifold where each data source is encoded via a low-dimensional distribution. This probabilistic manifold quantifies model form uncertainties such that LF sources with small bias are encoded close to the HF source. Additionally, we endow the output of our NN with a parametric distribution not only to quantify aleatoric uncertainties, but also to reformulate the network’s loss function based on strictly proper scoring rules which improve robustness and accuracy on unseen HF data. Through a set of analytic and engineering examples, we demonstrate that our approach provides a high predictive power while quantifying various sources of uncertainty. Our codes and examples can be accessed via GitLab.

Multi-Dimensional Information Alignment in Different Modalities for Generalized Zero-Shot and Few-Shot Learning

Generalized zero-shot learning (GZSL) aims to solve the category recognition tasks for unseen categories under the setting that training samples only contain seen classes while unseen classes are not available. This research is vital as there are always existing new categories and large amounts of unlabeled data in realistic scenarios. Previous work for GZSL usually maps the visual information of the visible classes and the semantic description of the invisible classes into the identical embedding space to bridge the gap between the disjointed visible and invisible classes, while ignoring the intrinsic features of visual images, which are sufficiently discriminative to classify themselves. To better use discriminative information from visual classes for GZSL, we propose the n-CADA-VAE. In our approach, we map the visual feature of seen classes to a high-dimensional distribution while mapping the semantic description of unseen classes to a low-dimensional distribution under the same latent embedding space, thus projecting information of different modalities to corresponding space positions more accurately. We conducted extensive experiments on four benchmark datasets (CUB, SUN, AWA1, and AWA2). The results show our model’s superior performance in generalized zero-shot as well as few-shot learning.

Integral, mean and covariance of the simplex-truncated multivariate normal distribution.

Compositional data, which is data consisting of fractions or probabilities, is common in many fields including ecology, economics, physical science and political science. If these data would otherwise be normally distributed, their spread can be conveniently represented by a multivariate normal distribution truncated to the non-negative space under a unit simplex. Here this distribution is called the simplex-truncated multivariate normal distribution. For calculations on truncated distributions, it is often useful to obtain rapid estimates of their integral, mean and covariance; these quantities characterising the truncated distribution will generally possess different values to the corresponding non-truncated distribution. In this paper, three different approaches that can estimate the integral, mean and covariance of any simplex-truncated multivariate normal distribution are described and compared. These three approaches are (1) naive rejection sampling, (2) a method described by Gessner et al. that unifies subset simulation and the Holmes-Diaconis-Ross algorithm with an analytical version of elliptical slice sampling, and (3) a semi-analytical method that expresses the integral, mean and covariance in terms of integrals of hyperrectangularly-truncated multivariate normal distributions, the latter of which are readily computed in modern mathematical and statistical packages. Strong agreement is demonstrated between all three approaches, but the most computationally efficient approach depends strongly both on implementation details and the dimension of the simplex-truncated multivariate normal distribution. For computations in low-dimensional distributions, the semi-analytical method is fast and thus should be considered. As the dimension increases, the Gessner et al. method becomes the only practically efficient approach of the methods tested here.

Composite Likelihood Estimation of an Autoregressive Panel Ordered Probit Model with Random Effects

Modeling and estimating autocorrelated discrete data can be challenging. In this article, we use an autoregressive panel ordered probit model where the serial correlation in the discrete variable is driven by the autocorrelation in the latent variable. In such a nonlinear model, the presence of a lagged latent variable results in an intractable likelihood containing high-dimensional integrals. To tackle this problem, we use composite likelihoods that involve a much lower order of integration. However, parameter identification might potentially become problematic since the information employed in lower dimensional distributions may not be rich enough for identification. Therefore, we characterize types of composite likelihoods that are valid for this model and study conditions under which the parameters can be identified. Moreover, we provide consistency and asymptotic normality results for two different composite likelihood estimators and conduct Monte Carlo studies to assess their finite-sample performances. Finally, we apply our method to analyze corporate bond ratings. Supplementary materials for this article are available online.

M5 competition uncertainty: Overdispersion, distributional forecasting, GAMLSS, and beyond

The M5 competition uncertainty track aims for probabilistic forecasting of sales of thousands of Walmart retail goods. We show that the M5 competition data face strong overdispersion and sporadic demand, especially zero demand. We discuss modeling issues concerning adequate probabilistic forecasting of such count data processes. Unfortunately, the majority of popular prediction methods used in the M5 competition (e.g. lightgbm and xgboost GBMs) fail to address the data characteristics, due to the considered objective functions. Distributional forecasting provides a suitable modeling approach to overcome those problems. The GAMLSS framework allows for flexible probabilistic forecasting using low-dimensional distributions. We illustrate how the GAMLSS approach can be applied to M5 competition data by modeling the location and scale parameters of various distributions, e.g. the negative binomial distribution. Finally, we discuss software packages for distributional modeling and their drawbacks, like the R package gamlss with its package extensions, and (deep) distributional forecasting libraries such as TensorFlow Probability.

Improving generative adversarial networks with simple latent distributions

Generative Adversarial Networks (GANs) have drawn great attention recently since they are the powerful models to generate high-quality images. Although GANs have achieved great success, they usually suffer from unstable training and consequently may lead to the poor generations in some cases. Such drawback is argued mainly due to the difficulties in measuring the divergence between the highly complicated the real and fake data distributions, which are normally in the high-dimensional space. To tackle this problem, previous researchers attempt to search a proper divergence capable of measuring the departure of the complex distributions. In contrast, we attempt to alleviate this problem from a different perspective: while retaining the information as much as possible of the original high dimensional distributions, we learn and leverage an additional latent space where simple distributions are defined in a low-dimensional space; as a result, we can readily compute the distance between two simple distributions with an available divergence measurement. Concretely, to retain the data information, the mutual information is maximized between the variables for the high dimensional complex distributions and the low dimensional simple distributions. The departure of the resulting simple distributions are then measured in the original way of GANs. Additionally, for simplifying the optimization further, we optimize directly the lower bound for mutual information. Termed as SimpleGAN, we conduct the proposed approach over the several different baseline models, i.e., conventional GANs, DCGAN, WGAN-GP, WGAN-GP-res, and LSWGAN-GP on the benchmark CIFAR-10 and STL-10 datasets. SimpleGAN shows the obvious superiority on these baseline models. Furthermore, in comparison with the existing methods measuring directly the distribution departure in the high-dimensional space, our method clearly demonstrates its superiority. Finally, a series of experiments show the advantages of the proposed SimpleGAN.

Foundations of compositional models: inference

Compositional models, as an alternative to Bayesian networks, are assembled from a system of low-dimensional distributions. Thus the respective apparatus falls fully into probability theory. The present paper surveys the results supporting the design of computational procedures, without which the application of these models to practical problems would be impossible. The methods of inference cannot do without a possibility to focus on a part of the considered multidimensional model and to incorporate data describing the actual situation. Thus the paper shows how to compute marginals and conditionals of multidimensional models. Also, the paper briefly solves the problem of computing the effect of an intervention, in case the model is interpreted as a causal model.

Composite Likelihood Estimation of an Autoregressive Panel Probit Model with Random Effects

Modeling and estimating autocorrelated discrete data can be challenging. In this paper, we use an autoregressive panel probit model where the autocorrelation in the discrete variable is driven by the autocorrelation in the latent variable. In such a non-linear model, the autocorrelation in an unobserved variable results in an intractable likelihood containing high-dimensional integrals. To tackle this problem, we use composite likelihoods that involve much lower order of integration. However, parameter identification becomes problematic since the information employed in lower dimensional distributions may not be rich enough for identification. Therefore, we characterize types of composite likelihoods that are valid for this model and study conditions under which the parameters can be identified. Moreover, we provide consistency and asymptotic normality results of the pairwise composite likelihood estimator and conduct Monte Carlo studies to assess its finite-sample performances. Finally, we apply our method to analyze credit ratings. The results indicate a significant improvement in the estimated probabilities for rating transitions compared with static models.

Efficient importance sampling in low dimensions using affine arithmetic

Despite the development of sophisticated techniques such as sequential Monte Carlo (Del Moral et al. in J R Stat Soc Ser B 68(3):411–436, 2006), importance sampling (IS) remains an important Monte Carlo method for low dimensional target distributions (Chopin and Ridgway in Leave Pima Indians alone: binary regression as a benchmark for Bayesian computation, 32:64–87, 2017). This paper describes a new technique for constructing proposal distributions for IS, using affine arithmetic (de Figueiredo and Stolfi in Numer Algorithms 37(1–4):147–158, 2004). This work builds on the Moore rejection sampler (Sainudiin in Machine interval experiments, Cornell University, Ithaca, 2005; Sainudiin and York in Algorithms Mol Biol 4(1):1, 2009) to which we provide a comparison.

Semi-analytical expression of stochastic closed curve attractors in nonlinear dynamical systems under weak noise

In this paper, a simple but accurate semi-analytical method to approximate probability density function of stochastic closed curve attractors is proposed. The expression of distribution applies to systems with strong nonlinearities, while only weak noise condition is needed. With the understanding that additive noise does not change the longitudinal distribution of the attractors, the high-dimensional probability density distribution is decomposed into two low-dimensional distributions: the longitudinal and the transverse probability density distributions. The longitudinal distribution can be calculated from the deterministic systems, while the probability density in the transverse direction of the curve can be approximated by the stochastic sensitivity function method. The effectiveness of this approach is verified by comparing the expression of distribution with the results of Monte Carlo numerical simulations in several planar systems.

Effectiveness of the Euclidean distance in high dimensional spaces

This paper presents analysis of applicability and performance of the Euclidean distance in relation to the dimensionality of the space. The effect of dimensionality on the behavior of Euclidean distance is explored; Furthermore, it is shown that the minimum distance approaches the maximum distance under a broader set of conditions without requiring the calculation of variance of random variables. It is demonstrated that the minimum distance approaches the maximum distance even for some low dimensional distributions, such as normal distribution. Many proposed measures not based directly on Euclidean distance cannot enlarger the difference between closest point and farthest point. The analysis has been performed on a wide range of artificial and publicly available datasets. As the variables of different distributions have different convergence rates, the results should not be interpreted to mean that the Euclidean distance is not applicable. In fact, it is shown in experiments that the Euclidean distance is very useful in the noncentral t-distribution even for the dimensionality higher than 10,000. Furthermore, it is observed that the behavior of Euclidean distance becomes more useful with increased number of samples.

An algorithm for constructing high dimensional distributions from distributions of lower dimension

We propose a new sequential procedure for estimating multivariate distributions in cases when conventional maximum likelihood has too many parameters and is therefore inaccurate or non-operational. The procedure constructs a multivariate distribution and its pseudo-likelihood sequentially, in each step using lower-dimensional distributions with a small number of parameters. In an application, the procedure provides excellent fit when the dimension is moderate, and remains operational when the conventional method fails.

Principal component based diffeomorphic surface mapping.

We present a new diffeomorphic surface mapping algorithm under the framework of large deformation diffeomorphic metric mapping (LDDMM). Unlike existing LDDMM approaches, this new algorithm reduces the complexity of the estimation of diffeomorphic transformations by incorporating a shape prior in which a nonlinear diffeomorphic shape space is represented by a linear space of initial momenta of diffeomorphic geodesic flows from a fixed template. In addition, for the first time, the diffeomorphic mapping is formulated within a decision-theoretic scheme based on Bayesian modeling in which an empirical shape prior is characterized by a low dimensional Gaussian distribution on initial momentum. This is achieved using principal component analysis (PCA) to construct the eigenspace of the initial momentum. A likelihood function is formulated as the conditional probability of observing surfaces given any particular value of the initial momentum, which is modeled as a random field of vector-valued measures characterizing the geometry of surfaces. We define the diffeomorphic mapping as a problem that maximizes a posterior distribution of the initial momentum given observable surfaces over the eigenspace of the initial momentum. We demonstrate the stability of the initial momentum eigenspace when altering training samples using a bootstrapping method. We then validate the mapping accuracy and show robustness to outliers whose shape variation is not incorporated into the shape prior.

Sampling conformations in high dimensions using low-dimensional distribution functions.

We present an approximation to a molecule's N-dimensional conformational probability density function (pdf) in terms of marginal pdfs of highest order l, where l is much less than N. The approximation is constructed as a product of conditional pdfs derived by recursive application of the generalized Kirkwood superposition approximation. Furthermore, an algorithm is presented to sample conformations from the approximate full-dimensional pdf based upon all input marginal pdfs. The sampling algorithm is tested for three small molecule systems by using the algorithm to sample conformations at levels l=1, 2, or 3 and comparing the distributions of sampled conformations with those from the molecular dynamics (MD) simulations. The distributions of conformations sampled at third (l=3) order resemble the MD distributions rather well and significantly better than those sampled at second (l=2) or first (l=1) order. In addition to highlighting the importance of correlations among internal degrees of freedom, these results suggest that low-order correlations suffice to describe most of the conformational fluctuations of molecules in a thermal environment.

시점별 형상의 지역적 선형 사상을 통한 3차원 물체의 특성 분석

본 논문은 시점에 따른 형상의 변화를 이용하여 물체의 특성을 나타내는 기법을 제안한다. 구체적으로, 3차원 물체의 여러 시점별 형상을 추출한 후, 이를 지역적 선형 사상을 통해 차원 축소하여 저차원 분포를 생성하고, 이를 이용하여 물체의 특성을 나타낸다. 또한, 생성된 점집합들에 반복적 최근접점 기법 및 푸리에 변환을 적용하여 유사한 모델을 검색하는 기법과 그 결과를 제시한다. 제안하는 기법은 다양한 시점에서의 형상 자체만이 아니라 시점에 따른 형상의 변화도 물체의 특성을 표현한다는 것을 보여주며, 검색 등 물체 특성을 표현하는데 적용될 것으로 기대된다. This paper explores the possibility of describing objects from the change in the shape according to the change in viewpoint. Specifically, we sample the shapes from various viewpoints of a 3-D model, and apply dimension reduction by locally linear embedding. A low dimensional distribution of points are constructed, and characteristics of the object are described from this distribution. Also, we propose two 3-D retrieval methods by applying the iterative closest point algorithm, and by applying Fourier transform and measuring similarity by modified Housdorff distance, and present experimental results. The proposed method shows that the change of shape according to the change in viewpoint can describe the characteristics of an object.

Human Age Estimation With Regression on Discriminative Aging Manifold

Recently, extensive studies on human faces in the human-computer interaction (HCI) field reveal significant potentials for designing automatic age estimation systems via face image analysis. The success of such research may bring in many innovative HCI tools used for the applications of human-centered multimedia communication. Due to the temporal property of age progression, face images with aging features may display some sequential patterns with low-dimensional distributions. In this paper, we demonstrate that such aging patterns can be effectively extracted from a discriminant subspace learning algorithm and visualized as distinct manifold structures. Through the manifold method of analysis on face images, the dimensionality redundancy of the original image space can be significantly reduced with subspace learning. A multiple linear regression procedure, especially with a quadratic model function, can be facilitated by the low dimensionality to represent the manifold space embodying the discriminative property. Such a processing has been evaluated by extensive simulations and compared with the state-of-the-art methods. Experimental results on a large size aging database demonstrate the effectiveness and robustness of our proposed framework.

COMBINATION OF MULTIPLE CLASSIFIERS BY MINIMIZING THE UPPER BOUND OF BAYES ERROR RATE FOR UNCONSTRAINED HANDWRITTEN NUMERAL RECOGNITION

In order to raise a class discrimination power by the combination of multiple classifiers, the upper bound of Bayes error rate which is bounded by the conditional entropy of a class and decisions should be minimized. Based on the minimization of the upper bound of the Bayes error rate, Wang and Wong proposed only a tree dependence approximation scheme of a high-dimensional probability distribution composed of a class and patterns. This paper extends such a tree dependence approximation scheme to higher order dependency for improving the classification performance and thus optimally approximates the high-dimensional probability distribution with a product of low-dimensional distributions. And then, a new combination method by the proposed approximation scheme is presented and evaluated with classifiers recognizing unconstrained handwritten numerals.