Mises-Fisher Distribution Research Articles

Mises-Fisher similarity-based boosted additive angular margin loss for breast cancer classification

To enhance the accuracy of breast cancer diagnosis, current practices rely on biopsies and microscopic examinations. However, this approach is known for being time-consuming, tedious, and costly. While convolutional neural networks (CNNs) have shown promise for their efficiency and high accuracy, training them effectively becomes challenging in real-world learning scenarios such as class imbalance, small-scale datasets, and label noises. Angular margin-based softmax losses, which concentrate on the angle between features and classifiers embedded in cosine similarity at the classification layer, aim to regulate feature representation learning. Nevertheless, the cosine similarity’s lack of a heavy tail impedes its ability to compactly regulate intra-class feature distribution, limiting generalization performance. Moreover, these losses are constrained to target classes when margin penalties are applied, which may not always optimize effectiveness. Addressing these hurdles, we introduce an innovative approach termed MF-BAM (Mises-Fisher Similarity-based Boosted Additive Angular Margin Loss), which extends beyond traditional cosine similarity and is anchored in the von Mises-Fisher distribution. MF-BAM not only penalizes the angle between deep features and their corresponding target class weights but also considers angles between deep features and weights associated with non-target classes. Through extensive experimentation on the BreaKHis dataset, MF-BAM achieves outstanding accuracies of 99.92%, 99.96%, 100.00%, and 98.05% for magnification levels of ×40, ×100, ×200, and ×400, respectively. Furthermore, additional experiments conducted on the BACH dataset for breast cancer classification, as well as on the LFW and YTF datasets for face recognition, affirm the generalization capability of our proposed loss function.

Certifiably optimal rotation and pose estimation based on the Cayley map

We present novel, convex relaxations for rotation and pose estimation problems that can a posteriori guarantee global optimality for practical measurement noise levels. Some such relaxations exist in the literature for specific problem setups that assume the matrix von Mises-Fisher distribution (a.k.a., matrix Langevin distribution or chordal distance) for isotropic rotational uncertainty. However, another common way to represent uncertainty for rotations and poses is to define anisotropic noise in the associated Lie algebra. Starting from a noise model based on the Cayley map, we define our estimation problems, convert them to Quadratically Constrained Quadratic Programs (QCQPs), then relax them to Semidefinite Programs (SDPs), which can be solved using standard interior-point optimization methods; global optimality follows from Lagrangian strong duality. We first show how to carry out basic rotation and pose averaging. We then turn to the more complex problem of trajectory estimation, which involves many pose variables with both individual and inter-pose measurements (or motion priors). Our contribution is to formulate SDP relaxations for all these problems based on the Cayley map (including the identification of redundant constraints) and to show them working in practical settings. We hope our results can add to the catalogue of useful estimation problems whose solutions can be a posteriori guaranteed to be globally optimal.

Probabilistic Contrastive Learning for Long-Tailed Visual Recognition.

Long-tailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Such imbalance issue considerably impairs the performance of standard supervised learning algorithms, which are mainly designed for balanced training sets. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. However, the performance of supervised contrastive learning is plagued by an inherent challenge: it necessitates sufficiently large batches of training data to construct contrastive pairs that cover all categories, yet this requirement is difficult to meet in the context of class-imbalanced data. To overcome this obstacle, we propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space, and samples contrastive pairs accordingly. In fact, estimating the distributions of all classes using features in a small batch, particularly for imbalanced data, is not feasible. Our key idea is to introduce a reasonable and simple assumption that the normalized features in contrastive learning follow a mixture of von Mises-Fisher (vMF) distributions on unit space, which brings two-fold benefits. First, the distribution parameters can be estimated using only the first sample moment, which can be efficiently computed in an online manner across different batches. Second, based on the estimated distribution, the vMF distribution allows us to sample an infinite number of contrastive pairs and derive a closed form of the expected contrastive loss for efficient optimization. Other than long-tailed problems, ProCo can be directly applied to semi-supervised learning by generating pseudo-labels for unlabeled data, which can subsequently be utilized to estimate the distribution of the samples inversely. Theoretically, we analyze the error bound of ProCo. Empirically, extensive experimental results on supervised/semi-supervised visual recognition and object detection tasks demonstrate that ProCo consistently outperforms existing methods across various datasets.

Probabilistic PARAFAC2.

The Parallel Factor Analysis 2 (PARAFAC2) is a multimodal factor analysis model suitable for analyzing multi-way data when one of the modes has incomparable observation units, for example, because of differences in signal sampling or batch sizes. A fully probabilistic treatment of the PARAFAC2 is desirable to improve robustness to noise and provide a principled approach for determining the number of factors, but challenging because direct model fitting requires that factor loadings be decomposed into a shared matrix specifying how the components are consistently co-expressed across samples and sample-specific orthogonality-constrained component profiles. We develop two probabilistic formulations of the PARAFAC2 model along with variational Bayesian procedures for inference: In the first approach, the mean values of the factor loadings are orthogonal leading to closed form variational updates, and in the second, the factor loadings themselves are orthogonal using a matrix Von Mises-Fisher distribution. We contrast our probabilistic formulations to the conventional direct fitting algorithm based on maximum likelihood on synthetic data and real fluorescence spectroscopy and gas chromatography-mass spectrometry data showing that the probabilistic formulations are more robust to noise and model order misspecification. The probabilistic PARAFAC2, thus, forms a promising framework for modeling multi-way data accounting for uncertainty.

Bridging spherical mixture distributions and word semantic knowledge for Neural Topic Modeling

Neural Topic Modeling has attracted significant attention from the Natural Language Processing community due to its black-box inference property and has made some progress. However, existing approaches have three main drawbacks: (1). hard to integrate external semantic knowledge to improve topic quality; (2). the posterior collapse issue of the Variational Auto-Encoder (VAE) harms the topic diversity; (3). cannot capture the semantic scopes of extracted topics. To address these limitations jointly, we model topics with a mixture of von Mises-Fisher distributions to incorporate word semantic knowledge and propose the Spherical Embedded Topic (SET) model based on the Wasserstein Auto-Encoder. Experiments have been conducted on three widely used text corpora, and the results show that SET obtains more coherent and diverse topics, outperforming the state-of-the-art approaches. Moreover, the ability of SET to capture semantic scopes of topics is also validated by our experiments.

Deep probabilistic direction prediction in 3D with applications to directional dark matter detectors

We present the first method to probabilistically predict 3D direction in a deep neural network model. The probabilistic predictions are modeled as a heteroscedastic von Mises-Fisher distribution on the sphere S2 , giving a simple way to quantify aleatoric uncertainty. This approach generalizes the cosine distance loss which is a special case of our loss function when the uncertainty is assumed to be uniform across samples. We develop approximations required to make the likelihood function and gradient calculations stable. The method is applied to the task of predicting the 3D directions of electrons, the most complex signal in a class of experimental particle physics detectors designed to demonstrate the particle nature of dark matter and study solar neutrinos. Using simulated Monte Carlo data, the initial direction of recoiling electrons is inferred from their tortuous trajectories, as captured by the 3D detectors. For 40 keV electrons in a 70% He 30% CO2 gas mixture at STP, the new approach achieves a mean cosine distance of 0.104 (26∘) compared to 0.556 (64∘) achieved by a non-machine learning algorithm. We show that the model is well-calibrated and accuracy can be increased further by removing samples with high predicted uncertainty. This advancement in probabilistic 3D directional learning could increase the sensitivity of directional dark matter detectors.

On the magnitude of error in the determination of rotation axes

Rotation axes (together with rotation angles) are often used to describe crystal orientations and misorientations, and they are also needed to characterize some properties of crystalline materials. Since experimental orientation data are subject to errors, the directions of the axes obtained from such data are also inaccurate. A natural question arises: given the resolution of input rotations, what is the average error of the rotation axes? Assuming that rotation data characterized by a rotation angle ω deviate from the actual data by error rotations with fixed angle δ but which are otherwise random, the average error of the rotation axes of the data is expressed analytically as a function of ω and δ. A scheme for using this formula in practical cases when rotation errors δ follow the von Mises–Fisher distribution is also described. Finally, the impact of crystal symmetry on the determination of the average errors of the axis directions is discussed. The presented results are important for assessing the reliability of rotation axes in studies where the directions of crystal rotations play a role, e.g. in identifying deformation slip mechanisms.

On the directional nature of celestial object’s fall on the earth (Part 1: Distribution of fireball shower, meteor fall & crater on earth’s surface)

Abstract This paper investigates the directional distribution of extraterrestrial objects (meteors, fireballs, and craters) impacting Earth’s surface, and introduces a novel directional statistical mixture model to analyze their falls, validated through rigorous testing. First, we address whether these falls follow non-uniform directional patterns by explicitly employing directional statistical tools for analyzing such data. Using projection techniques for longitude and latitude and more importantly, a general spherical statistical approach, we statistically investigate the suitability of the von Mises distribution and its spherical version, the von Mises Fisher distribution, (a maximum entropy distribution for directional data). Moreover, leveraging extensive datasets encompassing meteor falls, fireball showers, and craters, we propose and validate a novel mixture von Mises-Fisher model for comprehensively analyzing extraterrestrial object falls. Our study reveals distinct statistical characteristics across datasets: fireball falls exhibit non-uniformity, while meteor craters suggest a potential for both uniform and von Mises distributions with a preference for the latter after further refinement. Meteor landings deviate from a single directional maximum entropic distribution; we demonstrate the effectiveness of an optimal 13-component mixture von Mises-Fisher distribution for accurate modeling. Similar analyses resulted in 3- and 6-component partitions for fireball and crater datasets. This research presents valuable insights into the spatial patterns and directional statistical distribution models governing extraterrestrial objects’ fall on Earth, useful for various future works.

Grouped Spherical Data Modeling Through Hierarchical Nonparametric Bayesian Models and Its Application to fMRI Data Analysis.

Recently, spherical data (i.e., L2 normalized vectors) modeling has become a promising research topic in various real-world applications (such as gene expression data analysis, document categorization, and gesture recognition). In this work, we propose a hierarchical nonparametric Bayesian model based on von Mises-Fisher (VMF) distributions for modeling spherical data that involve multiple groups, where each observation within a group is sampled from a VMF mixture model with an infinite number of components allowing them to be shared across groups. Our model is formulated by employing a hierarchical nonparametric Bayesian framework known as the hierarchical Pitman-Yor (HPY) process mixture model, which possesses a power-law nature over the distribution of the components and is particularly useful for data distributions with heavy tails and skewness. To learn the proposed HPY process mixture model with VMF distributions, we systematically develop a closed-form optimization algorithm based on variational Bayes (VB). The merits of the proposed hierarchical Bayesian nonparametric model for modeling grouped spherical data are demonstrated through experiments on both synthetic data and a real-world application about resting-state functional magnetic resonance imaging (fMRI) data analysis.

Learning Representations on the Unit Sphere: Investigating Angular Gaussian and Von Mises-Fisher Distributions for Online Continual Learning

We use the maximum a posteriori estimation principle for learning representations distributed on the unit sphere. We propose to use the angular Gaussian distribution, which corresponds to a Gaussian projected on the unit-sphere and derive the associated loss function. We also consider the von Mises-Fisher distribution, which is the conditional of a Gaussian in the unit-sphere. The learned representations are pushed toward fixed directions, which are the prior means of the Gaussians; allowing for a learning strategy that is resilient to data drift. This makes it suitable for online continual learning, which is the problem of training neural networks on a continuous data stream, where multiple classification tasks are presented sequentially so that data from past tasks are no longer accessible, and data from the current task can be seen only once. To address this challenging scenario, we propose a memory-based representation learning technique equipped with our new loss functions. Our approach does not require negative data or knowledge of task boundaries and performs well with smaller batch sizes while being computationally efficient. We demonstrate with extensive experiments that the proposed method outperforms the current state-of-the-art methods on both standard evaluation scenarios and realistic scenarios with blurry task boundaries. For reproducibility, we use the same training pipeline for every compared method and share the code at https://github.com/Nicolas1203/ocl-fd.

Data-Free Generalized Zero-Shot Learning

Deep learning models have the ability to extract rich knowledge from large-scale datasets. However, the sharing of data has become increasingly challenging due to concerns regarding data copyright and privacy. Consequently, this hampers the effective transfer of knowledge from existing data to novel downstream tasks and concepts. Zero-shot learning (ZSL) approaches aim to recognize new classes by transferring semantic knowledge learned from base classes. However, traditional generative ZSL methods often require access to real images from base classes and rely on manually annotated attributes, which presents challenges in terms of data restrictions and model scalability. To this end, this paper tackles a challenging and practical problem dubbed as data-free zero-shot learning (DFZSL), where only the CLIP-based base classes data pre-trained classifier is available for zero-shot classification. Specifically, we propose a generic framework for DFZSL, which consists of three main components. Firstly, to recover the virtual features of the base data, we model the CLIP features of base class images as samples from a von Mises-Fisher (vMF) distribution based on the pre-trained classifier. Secondly, we leverage the text features of CLIP as low-cost semantic information and propose a feature-language prompt tuning (FLPT) method to further align the virtual image features and textual features. Thirdly, we train a conditional generative model using the well-aligned virtual image features and corresponding semantic text features, enabling the generation of new classes features and achieve better zero-shot generalization. Our framework has been evaluated on five commonly used benchmarks for generalized ZSL, as well as 11 benchmarks for the base-to-new ZSL. The results demonstrate the superiority and effectiveness of our approach. Our code is available in https://github.com/ylong4/DFZSL.

Seismic moment tensor classification using elliptical distribution functions on the hypersphere

SUMMARY Discrimination of underground explosions from naturally occurring earthquakes and other anthropogenic sources is one of the fundamental challenges of nuclear explosion monitoring. In an operational setting, the number of events that can be thoroughly investigated by analysts is limited by available resources. The capability to rapidly screen out events that can be robustly identified as not being explosions is, therefore, of great potential benefit. Nevertheless, possible mis-classification of explosions as earthquakes currently limits the use of screening methods for verification of test-ban treaties. Moment tensors provide a physics-based classification tool for the characterization of different seismic sources and have enabled the advent of new techniques for discriminating between earthquakes and explosions. Following normalization and projection of their six-degree vectors onto the hypersphere, existing screening approaches use spherically symmetric metrics to determine whether any new moment tensor may have been an explosion. Here, we show that populations of moment tensors for both earthquakes and explosions are anisotropically distributed on the hypersphere. Distributions possessing elliptical symmetry, such as the scaled von Mises–Fisher distribution, therefore provide a better description of these populations than the existing spherically symmetric models. We describe a method that uses these elliptical distributions in combination with a Bayesian classifier to achieve successful classification rates of 99 per cent for explosions and 98 per cent for earthquakes using existing catalogues of events from the western United States. The 1983 May 5 Crowdie underground nuclear test and 2018 July 20 DAG-1 deep-borehole chemical explosion are the only two explosions out of 140 that are incorrectly classified. Application of the method to the 2006–2017 nuclear tests in the Democratic People’s Republic of Korea yields 100 per cent identification rates and we provide a simple routine MTid for general usage. The approach provides a means to rapidly assess the likelihood of an event being an explosion and can be built into monitoring workflows that rely on simultaneously assessing multiple different discrimination metrics.

Probabilistic rotation modeling based on directional mixture density networks

Predicting 3D rotations from a single image presents a significant challenge, primarily due to the inherent uncertainty arising from factors such as high symmetry, self-obscuration, and noise in the 3D environment. In this work, we propose a novel multimodal-based probabilistic model that integrates the matrix Fisher distribution and von Mises Fisher distribution into a mixture density network. Our model not only captures the inherent uncertainty of the object but also learns this uncertainty directly from the training data, thereby enhancing the robustness, flexibility, and efficiency of the model. To further refine the model's ability to handle ambiguities and recognize multiple distinct modes, we introduce a relaxed version of the winner-take-all loss function. This adaptation significantly improves the model's capability in accurately representing complex multimodal distributions. The performance of our model is rigorously assessed using two challenging datasets: Pascal3D+ and ModelNet10-SO(3). Extensive experimental analysis highlights the model's exceptional capability to fit complex multimodal distributions. Notably, when tested on the ModelNet10-SO(3) dataset, which is characterized by its ambiguity, and the more unequivocal Pascal3D+ dataset, our model outperforms the prevailing top baseline models by achieving accuracy improvements of 2.7% and 3.4%, respectively, at the minimum angle threshold. These results not only demonstrate our model's advanced capabilities in fitting complex distributions but also validate its effectiveness in accurately predicting 3D rotations in both ambiguous and unambiguous scenarios.

Deep Adaptive Graph Clustering via von Mises-Fisher Distributions

Graph clustering has been a hot research topic and is widely used in many fields, such as community detection in social networks. Lots of works combining auto-encoder and graph neural networks have been applied to clustering tasks by utilizing node attributes and graph structure. These works usually assumed the inherent parameters (i.e., size and variance) of different clusters in the latent embedding space are homogeneous, and hence the assigned probability is monotonous over the Euclidean distance between node embeddings and centroids. Unfortunately, this assumption usually does not hold since the size and concentration of different clusters can be quite different, which limits the clustering accuracy. In addition, the node embeddings in deep graph clustering methods are usually L2 normalized so that it lies on the surface of a unit hyper-sphere. To solve this problem, we proposed D eep A daptive G raph C lustering via von Mises-Fisher distributions, namely DAGC. DAGC assumes the node embeddings H can be drawn from a von Mises-Fisher distribution and each cluster k is associated with cluster inherent parameters ρ k which includes cluster center μ and cluster cohesion degree κ. Then we adopt an EM-like approach (i.e., 𝒫( H | ρ ) and 𝒫( ρ | H ), respectively) to learn the embedding and cluster inherent parameters alternately. Specifically, with the node embeddings, we proposed to update the cluster centers in an attraction-repulsion manner to make the cluster centers more separable. And given the cluster inherent parameters, a likelihood-based loss is proposed to make node embeddings more concentrated around cluster centers. Thus, DAGC can simultaneously improve the intra-cluster compactness and inter-cluster heterogeneity. Finally, extensive experiments conducted on four benchmark datasets have demonstrated that the proposed DAGC consistently outperforms the state-of-the-art methods, especially on imbalanced datasets.

Probabilistic Keyphrase Generation From Copy and Generating Spaces.

Keyphrase generation is one of the most fundamental tasks in natural language processing (NLP). Most existing works on keyphrase generation mainly focus on using holistic distribution to optimize the negative log-likelihood loss, but they do not directly manipulate the copy and generating spaces, which may reduce the generability of the decoder. Additionally, existing keyphrase models are either unable to determine the dynamic numbers of keyphrases or produce the number of keyphrases implicitly. In this article, we propose a probabilistic keyphrase generation model from copy and generating spaces. The proposed model is built upon the vanilla variational encoder-decoder (VED) framework. On top of VED, two separate latent variables are adopted to model the distribution of data within the latent copy and generating spaces, respectively. Specifically, we adopt a von Mises-Fisher (vMF) distribution to obtain a condensed variable for modifying the generating probability distribution over the predefined vocabulary. Meanwhile, we utilize a clustering module, which is designed to promote Gaussian Mixture learning and subsequently extract a latent variable for the copy probability distribution. Moreover, we utilize a natural property of the Gaussian mixture network and use the number of filtered components to determine the number of keyphrases. The approach is trained based on latent variable probabilistic modeling, neural variational inference, and self-supervised learning. Experiments on social media and scientific article datasets outperform the state-of-the-art baselines in generating accurate predictions and controllable keyphrase numbers.

Correlation Properties in Channels With von Mises-Fisher Distribution of Scatterers

Correlation Properties in Channels With von Mises-Fisher Distribution of Scatterers

Recognition of Occluded Goods under Prior Inference Based on Generative Adversarial Network.

Aiming at the recognition of intelligent retail dynamic visual container goods, two problems that lead to low recognition accuracy must be addressed; one is the lack of goods features caused by the occlusion of the hand, and the other is the high similarity of goods. Therefore, this study proposes an approach for occluding goods recognition based on a generative adversarial network combined with prior inference to address the two abovementioned problems. With DarkNet53 as the backbone network, semantic segmentation is used to locate the occluded part in the feature extraction network, and simultaneously, the YOLOX decoupling head is used to obtain the detection frame. Subsequently, a generative adversarial network under prior inference is used to restore and expand the features of the occluded parts, and a multi-scale spatial attention and effective channel attention weighted attention mechanism module is proposed to select fine-grained features of goods. Finally, a metric learning method based on von Mises-Fisher distribution is proposed to increase the class spacing of features to achieve the effect of feature distinction, whilst the distinguished features are utilized to recognize goods at a fine-grained level. The experimental data used in this study were all obtained from the self-made smart retail container dataset, which contains a total of 12 types of goods used for recognition and includes four couples of similar goods. Experimental results reveal that the peak signal-to-noise ratio and structural similarity under improved prior inference are 0.7743 and 0.0183 higher than those of the other models, respectively. Compared with other optimal models, mAP improves the recognition accuracy by 1.2% and the recognition accuracy by 2.82%. This study solves two problems: one is the occlusion caused by hands, and the other is the high similarity of goods, thus meeting the requirements of commodity recognition accuracy in the field of intelligent retail and exhibiting good application prospects.

Generation of random directions from the generalized von Mises–Fisher distribution

We introduce two algorithms for generating from the bimodal generalized von Mises-Fisher distribution on the sphere. This generalization of the von Mises-Fisher distribution is more flexible, in particular by allowing for multimodality, and it preserves many of the theoretical properties of the von Mises-Fisher. The first proposed generation algorithm is obtained from conditional simulation and acceptance-rejection. The second one is the Metropolis-Hastings with mixture of von Mises-Fisher as jumping or instrumental distribution. These two algorithms are compared through density estimations of generated polar angles.

Failure probability estimation through high-dimensional elliptical distribution modeling with multiple importance sampling

This paper addresses the challenge of performing importance sampling in high-dimensional space (several hundred inputs) in order to estimate the failure probability of a physical system subject to randomness. It is assumed that the failure domain defined in the input space can possibly include multiple failure regions. A new approach is developed to construct auxiliary importance sampling densities sequentially for each failure region identified as part of the failure domain. The search for failure regions is achieved through optimization. A stochastic decomposition of the elliptically distributed inputs is exploited in the structure of the auxiliary densities, which are expressed as the product of a parametric conditional distribution for the radial component, and a parametric von Mises–Fisher distribution for the directional vector. The failure probability is then estimated by multiple importance sampling with a mixture of the densities. To demonstrate the efficiency of the proposed method in high-dimensional space, several numerical examples are considered involving the multivariate Gaussian and Student distributions, which are commonly used elliptical distributions for input modeling. In comparison with other simulation methods, the numerical cost of the proposed approach is found to be quite low when the gradient of the performance function defining the failure domain is available.

Density estimation for spherical data using nonparametric mixtures

Nonparametric density estimation is studied for spherical data that may arise in many scientific and practical fields. In particular, nonparametric mixture models based on likelihood maximization are used. A nonparametric mixture has component distributions mixed together with a mixing distribution that is completely unspecified and needs to be determined from data. For mixture components, a two-parameter distribution family can be used, with one parameter as the mixing variable and the other to control the smoothness of the density estimator. For example, the popular von Mises-Fisher distributions can be readily used for this purpose. Numerical studies with various spherical data sets show that the resultant mixture-based density estimators are strong competitors with the best of the other density estimators.