Von Mises-Fisher Mixture Research Articles

Grouped Spherical Data Modeling Through Hierarchical Nonparametric Bayesian Models and Its Application to fMRI Data Analysis.

Recently, spherical data (i.e., L2 normalized vectors) modeling has become a promising research topic in various real-world applications (such as gene expression data analysis, document categorization, and gesture recognition). In this work, we propose a hierarchical nonparametric Bayesian model based on von Mises-Fisher (VMF) distributions for modeling spherical data that involve multiple groups, where each observation within a group is sampled from a VMF mixture model with an infinite number of components allowing them to be shared across groups. Our model is formulated by employing a hierarchical nonparametric Bayesian framework known as the hierarchical Pitman-Yor (HPY) process mixture model, which possesses a power-law nature over the distribution of the components and is particularly useful for data distributions with heavy tails and skewness. To learn the proposed HPY process mixture model with VMF distributions, we systematically develop a closed-form optimization algorithm based on variational Bayes (VB). The merits of the proposed hierarchical Bayesian nonparametric model for modeling grouped spherical data are demonstrated through experiments on both synthetic data and a real-world application about resting-state functional magnetic resonance imaging (fMRI) data analysis.

A deep clustering framework integrating pairwise constraints and a VMF mixture model

<abstract><p>We presented a novel deep generative clustering model called Variational Deep Embedding based on Pairwise constraints and the Von Mises-Fisher mixture model (VDEPV). VDEPV consists of fully connected neural networks capable of learning latent representations from raw data and accurately predicting cluster assignments. Under the assumption of a genuinely non-informative prior, VDEPV adopted a von Mises-Fisher mixture model to depict the hyperspherical interpretation of the data. We defined and established pairwise constraints by employing a random sample mining strategy and applying data augmentation techniques. These constraints enhanced the compactness of intra-cluster samples in the spherical embedding space while improving inter-cluster samples' separability. By minimizing Kullback-Leibler divergence, we formulated a clustering loss function based on pairwise constraints, which regularized the joint probability distribution of latent variables and cluster labels. Comparative experiments with other deep clustering methods demonstrated the excellent performance of VDEPV.</p></abstract>

Deep Clustering Analysis via Dual Variational Autoencoder With Spherical Latent Embeddings.

In recent years, clustering methods based on deep generative models have received great attention in various unsupervised applications, due to their capabilities for learning promising latent embeddings from original data. This article proposes a novel clustering method based on variational autoencoder (VAE) with spherical latent embeddings. The merits of our clustering method can be summarized as follows. First, instead of considering the Gaussian mixture model (GMM) as the prior over latent space as in a variety of existing VAE-based deep clustering methods, the von Mises-Fisher mixture model prior is deployed in our method, leading to spherical latent embeddings that can explicitly control the balance between the capacity of decoder and the utilization of latent embedding in a principled way. Second, a dual VAE structure is leveraged to impose the reconstruction constraint for the latent embedding and its corresponding noise counterpart, which embeds the input data into a hyperspherical latent space for clustering. Third, an augmented loss function is proposed to enhance the robustness of our model, which results in a self-supervised manner through the mutual guidance between the original data and the augmented ones. The effectiveness of the proposed deep generative clustering method is validated through comparisons with state-of-the-art deep clustering methods on benchmark datasets. The source code of the proposed model is available at https://github.com/fwt-team/DSVAE.

Cross-entropy-based directional importance sampling with von Mises-Fisher mixture model for reliability analysis

In structural reliability analysis, the estimation of rare failure probability is a significant challenge. Directional importance sampling may address this challenge when a quasi-optimal sampling density is chosen. Directional importance sampling involves sampling on the unit hypersphere and one-dimensional reliability analysis. This paper develops a cross-entropy-based directional importance sampling (CE-DIS) for reliability analysis. The von Mises-Fisher mixture (vMFM) model is selected as sampling density to generate direction vectors on the unit hypersphere. The updating rules based on the cross-entropy method are derived for the vMFM model to find a quasi-optimal sampling density. In addition, the DBSCAN algorithm is used to choose the number of distributions in the vMFM model in case of no prior information of failure domain, and the intermediate failure event is introduced in case of not enough samples to update the distribution parameters. Several benchmark numerical examples and engineering applications are used to test the performance of the proposed CE-DIS method. The results demonstrate that the CE-DIS method significantly improves the computational efficiency compared to original directional sampling and other CE-based importance sampling methods. In addition, the CE-DIS method can be applied to different types of reliability problems, e.g., multiple design points, multiple failure modes and rare failure probability.

Semantics-Aware Hidden Markov Model for Human Mobility

Understanding human mobility benefits numerous applications such as urban planning, traffic control, and city management. Previous work mainly focuses on modeling spatial and temporal patterns of human mobility. However, the semantics of trajectory are ignored, thus failing to model people's motivation behind mobility. In this paper, we propose a novel semantics-aware mobility model that captures human mobility motivation using large-scale semantic-rich spatial-temporal data from location-based social networks. In our system, we first develop a multimodal embedding method to project user, location, time, and activity on the same embedding space in an unsupervised way while preserving original trajectory semantics. Then, we use hidden Markov model to learn latent states and transitions between them in the embedding space, which is the location embedding vector, to jointly consider spatial, temporal, and user motivations. In order to tackle the sparsity of individual mobility data, we further propose a von Mises-Fisher mixture clustering for user grouping so as to learn a reliable and fine-grained model for groups of users sharing mobility similarity. We evaluate our proposed method on two large-scale real-world datasets, where we validate the ability of our method to produce high-quality mobility models. We also conduct extensive experiments on the specific task of location prediction. The results show that our model outperforms state-of-the-art mobility models with higher prediction accuracy and much higher efficiency.

Thirty biologically interpretable clusters of transcription factors distinguish cancer type

BackgroundTranscription factors are essential regulators of gene expression and play critical roles in development, differentiation, and in many cancers. To carry out their regulatory programs, they must cooperate in networks and bind simultaneously to sites in promoter or enhancer regions of genes. We hypothesize that the mRNA co-expression patterns of transcription factors can be used both to learn how they cooperate in networks and to distinguish between cancer types.ResultsWe recently developed a new algorithm, Thresher, that combines principal component analysis, outlier filtering, and von Mises-Fisher mixture models to cluster genes (in this case, transcription factors) based on expression, determining the optimal number of clusters in the process. We applied Thresher to the RNA-Seq expression data of 486 transcription factors from more than 10,000 samples of 33 kinds of cancer studied in The Cancer Genome Atlas (TCGA). We found that 30 clusters of transcription factors from a 29-dimensional principal component space were able to distinguish between most cancer types, and could separate tumor samples from normal controls. Moreover, each cluster of transcription factors could be either (i) linked to a tissue-specific expression pattern or (ii) associated with a fundamental biological process such as cell cycle, angiogenesis, apoptosis, or cytoskeleton. Clusters of the second type were more likely also to be associated with embryonically lethal mouse phenotypes.ConclusionsUsing our approach, we have shown that the mRNA expression patterns of transcription factors contain most of the information needed to distinguish different cancer types. The Thresher method is capable of discovering biologically interpretable clusters of genes. It can potentially be applied to other gene sets, such as signaling pathways, to decompose them into simpler, yet biologically meaningful, components.

Thresher: determining the number of clusters while removing outliers

BackgroundCluster analysis is the most common unsupervised method for finding hidden groups in data. Clustering presents two main challenges: (1) finding the optimal number of clusters, and (2) removing “outliers” among the objects being clustered. Few clustering algorithms currently deal directly with the outlier problem. Furthermore, existing methods for identifying the number of clusters still have some drawbacks. Thus, there is a need for a better algorithm to tackle both challenges.ResultsWe present a new approach, implemented in an R package called Thresher, to cluster objects in general datasets. Thresher combines ideas from principal component analysis, outlier filtering, and von Mises-Fisher mixture models in order to select the optimal number of clusters. We performed a large Monte Carlo simulation study to compare Thresher with other methods for detecting outliers and determining the number of clusters. We found that Thresher had good sensitivity and specificity for detecting and removing outliers. We also found that Thresher is the best method for estimating the optimal number of clusters when the number of objects being clustered is smaller than the number of variables used for clustering. Finally, we applied Thresher and eleven other methods to 25 sets of breast cancer data downloaded from the Gene Expression Omnibus; only Thresher consistently estimated the number of clusters to lie in the range of 4–7 that is consistent with the literature.ConclusionsThresher is effective at automatically detecting and removing outliers. By thus cleaning the data, it produces better estimates of the optimal number of clusters when there are more variables than objects. When we applied Thresher to a variety of breast cancer datasets, it produced estimates that were both self-consistent and consistent with the literature. We expect Thresher to be useful for studying a wide variety of biological datasets.

Cross-entropy-based adaptive importance sampling using von Mises-Fisher mixture for high dimensional reliability analysis

In order to address challenges in performing importance sampling in a high dimensional space of random variables, the paper develops a cross-entropy-based adaptive importance sampling technique that employs a von Mises-Fisher mixture as the sampling density model. By small-size pre-samplings, the proposed approach first finds a near-optimal sampling density by minimizing the Kullback–Leibler cross entropy between a von Mises-Fisher mixture model and the absolute best importance sampling density. To facilitate the minimization process, updating rules for parameters of the von Mises-Fisher mixture model are derived. Various practical issues associated with the updating rules are discussed and heuristic rules to improve the performance of the importance sampling are introduced. At the stage of final sampling, two slightly different sampling strategies are proposed to provide analysis options. Three numerical examples are investigated to test and demonstrate the proposed importance sampling method. The numerical examples show that the proposed approach, applicable to both component and system reliability problems, has superior performance for high dimensional reliability analysis problems with low failure probabilities.