Mixture Of Von Mises-Fisher Distributions Research Articles

Bridging spherical mixture distributions and word semantic knowledge for Neural Topic Modeling

Neural Topic Modeling has attracted significant attention from the Natural Language Processing community due to its black-box inference property and has made some progress. However, existing approaches have three main drawbacks: (1). hard to integrate external semantic knowledge to improve topic quality; (2). the posterior collapse issue of the Variational Auto-Encoder (VAE) harms the topic diversity; (3). cannot capture the semantic scopes of extracted topics. To address these limitations jointly, we model topics with a mixture of von Mises-Fisher distributions to incorporate word semantic knowledge and propose the Spherical Embedded Topic (SET) model based on the Wasserstein Auto-Encoder. Experiments have been conducted on three widely used text corpora, and the results show that SET obtains more coherent and diverse topics, outperforming the state-of-the-art approaches. Moreover, the ability of SET to capture semantic scopes of topics is also validated by our experiments.

Mixture of von Mises-Fisher distribution with sparse prototypes

Mixtures of von Mises-Fisher distributions can be used to cluster data on the unit hypersphere. This is particularly adapted for high-dimensional directional data such as texts. We propose in this article to estimate a von Mises mixture using a l1 penalized likelihood. This leads to sparse prototypes that improve clustering interpretability. We introduce an expectation–maximisation (EM) algorithm for this estimation and explore the trade-off between the sparsity term and the likelihood one with a path following algorithm. The model’s behaviour is studied on simulated data and, we show the advantages of the approach on real data benchmark. We also introduce a new data set on financial reports and exhibit the benefits of our method for exploratory analysis.

Estimation of the number of clusters on d-dimensional sphere

Spherical data is distributed on the sphere. The data appears in various fields such as meteorology, biology, and natural language processing. However, a method for analysis of spherical data does not develop enough yet. One of the important issues is an estimation of the number of clusters in spherical data. To address the issue, I propose a new method called the Spherical X-means (SX-means) that can estimate the number of clusters on d-dimensional sphere. The SX-means is the model-based method assuming that the data is generated from a mixture of von Mises-Fisher distributions. The present paper explains the proposed method and shows its performance of estimation of the number of clusters.

Clustering Directions Based on the Estimation of a Mixture of Von Mises-Fisher Distributions

Background:In the statistical analysis of directional data, the von Mises-Fisher distribution plays an important role to model unit vectors. The estimation of the parameters of a mixture of von Mises-Fisher distributions can be done through the Estimation-Maximization algorithm.Objective:In this paper we propose a dynamic clusters type algorithm based on the estimation of the parameters of a mixture of von Mises-Fisher distributions for clustering directions, and we compare this algorithm with the Estimation-Maximization algorithm. We also define the between-groups and within-groups variability measures to compare the solutions obtained with the algorithms through these measures.Results:The comparison of the clusters obtained with both algorithms is provided for a simulation study based on samples generated from a mixture of two Fisher distributions and for an illustrative example with spherical data.

Learning-based EM clustering for data on the unit hypersphere with application to exoplanet data

This study focuses on clustering algorithms for data on the unit hypersphere. This type of directional data lain on the surface of a unit hypersphere is used in geology, biology, meteorology, medicine and oceanography. The EM algorithm with mixtures of von Mises-Fisher distributions is often used for model-based clustering for data on the unit hypersphere. However, the EM algorithm is sensitive to initial values and outliers and a number of clusters must be assigned a priori. In this paper, we propose an effective approach, called a learning-based EM algorithm with von Mises-Fisher distributions, to cluster this type of hyper-spherical data. The proposed clustering method is robust to outliers, without the need for initialization, and automatically determines the number of clusters. Thus, it becomes a fully-unsupervised model-based clustering method for data on the unit hypersphere. Some numerical and real examples with comparisons are given to demonstrate the effectiveness and superiority of the proposed method. We also apply the proposed learning-based EM algorithm to cluster exoplanet data in extrasolar planets. The clustering results have several important implications for exoplanet data and allow an interpretation of exoplanet migration.

MovMF: AnRPackage for Fitting Mixtures of von Mises-Fisher Distributions

Finite mixtures of von Mises-Fisher distributions allow to apply model-based clustering methods to data which is of standardized length, i.e., all data points lie on the unit sphere. The R package movMF contains functionality to draw samples from finite mixtures of von Mises-Fisher distributions and to fit these models using the expectation-maximization algorithm for maximum likelihood estimation. Special features are the possibility to use sparse matrix representations for the input data, different variants of the expectation-maximization algorithm, different methods for determining the concentration parameters in the M-step and to impose constraints on the concentration parameters over the components. In this paper we describe the main fitting function of the package and illustrate its application. In addition we compare the clustering performance of finite mixtures of von Mises-Fisher distributions to spherical k-means. We also discuss the resolution of several numerical issues which occur for estimating the concentration parameters and for determining the normalizing constant of the von Mises-Fisher distribution.

Computational Representation of White Matter Fiber Orientations

We present a new methodology based on directional data clustering to represent white matter fiber orientations in magnetic resonance analyses for high angular resolution diffusion imaging. A probabilistic methodology is proposed for estimating intravoxel principal fiber directions, based on clustering directional data arising from orientation distribution function (ODF) profiles. ODF reconstructions are used to estimate intravoxel fiber directions using mixtures of von Mises-Fisher distributions. The method focuses on clustering data on the unit sphere, where complexity arises from representing ODF profiles as directional data. The proposed method is validated on synthetic simulations, as well as on a real data experiment. Based on experiments, we show that by clustering profile data using mixtures of von Mises-Fisher distributions it is possible to estimate multiple fiber configurations in a more robust manner than currently used approaches, without recourse to regularization or sharpening procedures. The method holds promise to support robust tractographic methodologies and to build realistic models of white matter tracts in the human brain.

Spatial Fading Correlation model using mixtures of Von Mises Fisher distributions

In this paper new expressions for the Spatial Fading Correlation (SFC) functions of Antenna Arrays (AA) in a 3-dimensional (3D) multipath channel are derived. In particular the Uniform Circular Array (UCA) antenna topology is considered. The derivation of the novel SFC function uses a Probability Density Function (PDF) originating from the field of directional statistics, the Von Mises Fisher (VMF) PDF. In particular the novel SFC function is based on the concept of mixture modeling and hence uses a mixture of VMF distributions. Since the SFC function is dependent on the Angle of Arrival (AoA) as well as the power of each cluster, the more appropriate power azimuth colatitude spectrum term has been used. The choice of distribution is validated with the use of Multiple Input Multiple Output (MIMO) experimental data that was obtained in an outdoor drive test campaign in Germany. A mixture can be composed of any number of clusters and this is mainly dependent on the clutter type encountered in the propagation environment. The parameters of the individual clusters within the mixture are derived and an estimation of those parameters is achieved using the soft-Expectation Maximization (EM) algorithm. The results indicate that the proposed model fits well with the MIMO data.

Three-dimensional channel modelling using spherical statistics for multiple-input multiple-output systems

Recently, the third generation partnership standards bodies (3GPP/3GPP2) have defined a two-dimensional channel model for multiple-input multiple-output (MIMO) systems, where the propagating plane waves are assumed to arrive only from the azimuthal direction and therefore not include the elevation domain. As a result of this assumption, the derived angle-of-arrival (AoA) distribution is characterised only by the azimuth direction of these waves. The AoA distribution of multipaths is implemented with a novel three-dimensional approach. The von Mises-Fisher (VMF) probability density function is used to describe their distribution within the propagation environment in both azimuth and co-latitude. More specifically, the proposed model uses a mixture of VMF distributions. A mixture can be composed of any number of clusters and this is clutter specific. The parameters of the individual cluster of scatterers within the mixture are derived and an estimation of those parameters is achieved using the spherical K-means algorithm and also the expectation maximisation algorithm. Statistical tests are provided to measure the goodness of fit of the proposed model. The results indicate that the proposed model fits well with MIMO experimental data obtained from a measurement campaign in Germany.

Mixture of Spherical Distributions for Single-View Relighting

We present a method for simultaneously estimating the illumination of a scene and the reflectance property of an object from single view images - a single image or a small number of images taken from the same viewpoint. We assume that the illumination consists of multiple point light sources and the shape of the object is known. First, we represent the illumination on the surface of a unit sphere as a finite mixture of von Mises-Fisher distributions based on a novel spherical specular reflection model that well approximates the Torrance-Sparrow reflection model. Next, we estimate the parameters of this mixture model including the number of its component distributions and the standard deviation of them, which correspond to the number of light sources and the surface roughness, respectively. Finally, using these results as the initial estimates, we iteratively refine the estimates based on the original Torrance-Sparrow reflection model. The final estimates can be used to relight single-view images such as altering the intensities and directions of the individual light sources. The proposed method provides a unified framework based on directional statistics for simultaneously estimating the intensities and directions of an unknown number of light sources as well as the specular reflection parameter of the object in the scene.

Frequency-sensitive competitive learning for scalable balanced clustering on high-dimensional hyperspheres.

Competitive learning mechanisms for clustering, in general, suffer from poor performance for very high-dimensional (>1000) data because of "curse of dimensionality" effects. In applications such as document clustering, it is customary to normalize the high-dimensional input vectors to unit length, and it is sometimes also desirable to obtain balanced clusters, i.e., clusters of comparable sizes. The spherical kmeans (spkmeans) algorithm, which normalizes the cluster centers as well as the inputs, has been successfully used to cluster normalized text documents in 2000+ dimensional space. Unfortunately, like regular kmeans and its soft expectation-maximization-based version, spkmeans tends to generate extremely imbalanced clusters in high-dimensional spaces when the desired number of clusters is large (tens or more). This paper first shows that the spkmeans algorithm can be derived from a certain maximum likelihood formulation using a mixture of von Mises-Fisher distributions as the generative model, and in fact, it can be considered as a batch-mode version of (normalized) competitive learning. The proposed generative model is then adapted in a principled way to yield three frequency-sensitive competitive learning variants that are applicable to static data and produced high-quality and well-balanced clusters for high-dimensional data. Like kmeans, each iteration is linear in the number of data points and in the number of clusters for all the three algorithms. A frequency-sensitive algorithm to cluster streaming data is also proposed. Experimental results on clustering of high-dimensional text data sets are provided to show the effectiveness and applicability of the proposed techniques. Index Terms-Balanced clustering, expectation maximization (EM), frequency-sensitive competitive learning (FSCL), high-dimensional clustering, kmeans, normalized data, scalable clustering, streaming data, text clustering.