Minimum Message Length Criterion Research Articles

Unsupervised Mixture Models on the Edge for Smart Energy Consumption Segmentation with Feature Saliency.

Smart meter datasets have recently transitioned from monthly intervals to one-second granularity, yielding invaluable insights for diverse metering functions. Clustering analysis, a fundamental data mining technique, is extensively applied to discern unique energy consumption patterns. However, the advent of high-resolution smart meter data brings forth formidable challenges, including non-Gaussian data distributions, unknown cluster counts, and varying feature importance within high-dimensional spaces. This article introduces an innovative learning framework integrating the expectation-maximization algorithm with the minimum message length criterion. This unified approach enables concurrent feature and model selection, finely tuned for the proposed bounded asymmetric generalized Gaussian mixture model with feature saliency. Our experiments aim to replicate an efficient smart meter data analysis scenario by incorporating three distinct feature extraction methods. We rigorously validate the clustering efficacy of our proposed algorithm against several state-of-the-art approaches, employing diverse performance metrics across synthetic and real smart meter datasets. The clusters that we identify effectively highlight variations in residential energy consumption, furnishing utility companies with actionable insights for targeted demand reduction efforts. Moreover, we demonstrate our method's robustness and real-world applicability by harnessing Concordia's High-Performance Computing infrastructure. This facilitates efficient energy pattern characterization, particularly within smart meter environments involving edge cloud computing. Finally, we emphasize that our proposed mixture model outperforms three other models in this paper's comparative study. We achieve superior performance compared to the non-bounded variant of the proposed mixture model by an average percentage improvement of 7.828%.

Data-Weighted Multivariate Generalized Gaussian Mixture Model: Application to Point Cloud Robust Registration.

In this paper, a weighted multivariate generalized Gaussian mixture model combined with stochastic optimization is proposed for point cloud registration. The mixture model parameters of the target scene and the scene to be registered are updated iteratively by the fixed point method under the framework of the EM algorithm, and the number of components is determined based on the minimum message length criterion (MML). The KL divergence between these two mixture models is utilized as the loss function for stochastic optimization to find the optimal parameters of the transformation model. The self-built point clouds are used to evaluate the performance of the proposed algorithm on rigid registration. Experiments demonstrate that the algorithm dramatically reduces the impact of noise and outliers and effectively extracts the key features of the data-intensive regions.

SSBM: A signed stochastic block model for multiple structure discovery in large-scale exploratory signed networks

Signed network structure discovery has received extensive attention and has become a research focus in the field of network science. However, most of the existing studies are focused on the networks with a single structure, e.g., community or bipartite, while ignoring multiple structures, e.g., the coexistence of community and bipartite structures. Furthermore, existing studies were faced with challenge regarding large-scale signed networks due to their high time complexity, especially when determining the number of clusters in the observed network without any prior knowledge. In view of this, we propose a mathematically principled method for signed network multiple structure discovery named the Signed Stochastic Block Model (SSBM). The SSBM can capture the multiple structures contained in signed networks, e.g., community, bipartite, and coexistence of them, by adopting a probabilistic model. Moreover, by integrating the minimum message length (MML) criterion and component-wise EM (CEM) algorithm, a scalable learning algorithm that has the ability of model selection is proposed to handle large-scale signed networks. By comparing state-of-the-art methods on synthetic and real-world signed networks, extensive experimental results demonstrate the effectiveness and efficiency of SSBM in discovering large-scale exploratory signed networks with multiple structures.

Sparse Count Data Clustering Using an Exponential Approximation to Generalized Dirichlet Multinomial Distributions.

Clustering frequency vectors is a challenging task on large data sets considering its high dimensionality and sparsity nature. Generalized Dirichlet multinomial (GDM) distribution is a competitive generative model for count data in terms of accuracy, yet its parameters estimation process is slow. The exponential-family approximation of the multivariate Polya distribution has shown to be efficient to train and cluster data directly, without dimensionality reduction. In this article, we derive an exponential-family approximation to the GDM distributions, and we call it (EGDM). A mixture model is developed based on the new member of the exponential-family of distributions, and its parameters are learned through the deterministic annealing expectation-maximization (DAEM) approach as a new clustering algorithm for count data. Moreover, we propose to estimate the optimal number of EGDM mixture components based on the minimum message length (MML) criterion. We have conducted a set of empirical experiments, concerning text, image, and video clustering, to evaluate the proposed approach performance. Results show that the new model attains a superior performance, and it is considerably faster than the corresponding method for GDM distributions.

Multivariate‐bounded Gaussian mixture model with minimum message length criterion for model selection

AbstractBounded support Gaussian mixture model (BGMM) has been proposed for data modelling as an alternative to unbounded support mixture models for the cases when the data lies in bounded support. In this paper, we propose applications of multivariate BGMM in data clustering for more insightful analysis of the model. We also propose minimum message length (MML) criterion for model selection in data clustering using multivariate BGMM. The presented model is applied to data clustering in several speech (TSP and Spoken Digits) and image databases (MNIST and Fashion MNIST). We also propose the application of BGMM in code‐book generation at feature extraction phase. Inspired by the success of bag of visual words approach in computer vision, it is also introduced in speech data representation and validated through experiments presented in this paper. For validation of model selection criterion, MML is applied to different medical, speech and image datasets. Experimental results obtained during the model selection through MML are further compared with seven different model selection criteria. The results presented in the paper demonstrate the effectiveness of BGMM for clustering speech and image databases, code‐book generation through clustering for feature representation and model selection.

Mixture‐based clustering for count data using approximated Fisher Scoring and Minorization–Maximization approaches

AbstractThe multinomial distribution has been widely used to model count data. To increase clustering efficiency, we use an approximation to the Fisher scoring algorithm, which is more robust regarding the choice of initial parameter values. Then, we use a novel approach to estimate the optimal number of components, based on minimum message length criterion. Moreover, we consider a generalization of the multinomial model obtained by introducing the Dirichlet as prior, yielding the Dirichlet Compound Multinomial (DCM). Even though DCM can address the burstiness phenomenon of count data, the presence of Gamma function in its density function usually leads to undesired complications. In this article, we use two alternative representations of DCM distribution to perform clustering based on finite mixture models, where the mixture parameters are estimated using the minorization–maximization framework. To evaluate and compare the performance of our proposed models, we have considered three challenging real‐world applications that involve high‐dimensional count vectors, namely, sentiment analysis, facial expression recognition, and human action recognition. The results show that the proposed algorithms increase the clustering efficiency of their respective models remarkably, and the best results are achieved by the second parametrization of DCM, which can accommodate over‐dispersed count data.

Heterogeneous Graphical Granger Causality by Minimum Message Length.

The heterogeneous graphical Granger model (HGGM) for causal inference among processes with distributions from an exponential family is efficient in scenarios when the number of time observations is much greater than the number of time series, normally by several orders of magnitude. However, in the case of “short” time series, the inference in HGGM often suffers from overestimation. To remedy this, we use the minimum message length principle (MML) to determinate the causal connections in the HGGM. The minimum message length as a Bayesian information-theoretic method for statistical model selection applies Occam’s razor in the following way: even when models are equal in their measure of fit-accuracy to the observed data, the one generating the most concise explanation of data is more likely to be correct. Based on the dispersion coefficient of the target time series and on the initial maximum likelihood estimates of the regression coefficients, we propose a minimum message length criterion to select the subset of causally connected time series with each target time series and derive its form for various exponential distributions. We propose two algorithms—the genetic-type algorithm (HMMLGA) and exHMML to find the subset. We demonstrated the superiority of both algorithms in synthetic experiments with respect to the comparison methods Lingam, HGGM and statistical framework Granger causality (SFGC). In the real data experiments, we used the methods to discriminate between pregnancy and labor phase using electrohysterogram data of Islandic mothers from Physionet databasis. We further analysed the Austrian climatological time measurements and their temporal interactions in rain and sunny days scenarios. In both experiments, the results of HMMLGA had the most realistic interpretation with respect to the comparison methods. We provide our code in Matlab. To our best knowledge, this is the first work using the MML principle for causal inference in HGGM.

Proportional data modeling via selection and estimation of a finite mixture of scaled Dirichlet distributions

AbstractThis paper proposes an unsupervised algorithm for learning a finite mixture of scaled Dirichlet distributions. Parameters estimation is based on the maximum likelihood approach, and the minimum message length (MML) criterion is proposed for selecting the optimal number of components. This research work is motivated by the flexibility issues of the Dirichlet distribution, the widely used model for multivariate proportional data, which has prompted a number of scholars to search for generalizations of the Dirichlet. By introducing the extra parameters of the scaled Dirichlet, several useful statistical models could be obtained. Experimental results are presented using both synthetic and real datasets. Moreover, challenging real‐world applications are empirically investigated to evaluate the efficiency of our proposed statistical framework.

A knowledge-driven way to interpret the isometric log-ratio transformation and mixture distributions of geochemical data

When extracting geo-information from geochemical data, it is essential to consider the compositional nature and geological signatures of the data. Isometric log-ratio transformation (ILR) produces an orthonormal basis of geochemical data and accounts for the compositional nature of the data. However, it still often remains difficult to interpret ILR-transformed variables because of the lack of a geological basis for a data-driven approach; therefore, it is necessary to find some geological knowledge-based criteria to help enable a more understandable interpretation following ILR transformation. Characterized by certain elements and elemental ratios, the chronological order of geological processes can be used to construct interpretable ILR transformation. This concept was applied to extract geo-information from stream sediment geochemical data in the Duolong mineral district, Tibet, China. Furthermore, the expectation-maximization (EM) algorithm modified by a minimum message length criterion (MML) was employed to investigate mixture distributions of the geochemical data. In the study area, mafic rocks with high concentrations of Cr and Ni were emplaced earlier than the porphyry Cu–Au deposits which have stable Cu/Au ratios. Based on these criteria and hierarchical cluster analysis, sequential binary partition was constructed among the Cu, Au, Cr, and Ni concentrations. The ILR-transformed variables follow either a bi-normal or tri-normal distribution, the subpopulations of which were interpreted as fingerprints inherited from mafic magmatic processes, Cu–Au hydrothermal systems, and later geological processes, respectively. The high-average subpopulation of ILR-transformed Cu corresponds to anomalies associated with the porphyry and epithermal Cu–Au systems, and two areas are predicted to have high Cu–Au mineralization potential. This study demonstrates that geological knowledge-driven ILR transformation is a promising way to efficiently extract geo-information from geochemical data.

Unsupervised learning of finite full covariance multivariate generalized Gaussian mixture models for human activity recognition

We propose in this paper to recognize human activities through an unsupervised learning of finite multivariate generalized Gaussian mixture model. We address an important cue in finite mixture model which is the estimation of the mixture model’s parameters for a full covariance matrix. We have developed a novel learning algorithm based on Fixed-point covariance matrix estimator combined with the Expectation-Maximization algorithm. Furthermore, we have proposed an appropriate minimum message length (MML) criterion to deal with model selection problem. We evaluated our proposed method on synthetic datasets and a challenging application namely : Human activity recognition from images and videos. The obtained resutls show clearly the merits of our proposed framework which has better capabilities with full covariance matrix when modeling correlated data.

Information-Theoretic Inference of an Optimal Dictionary of Protein Supersecondary Structures.

We recently developed an unsupervised Bayesian inference methodology to automatically infer a dictionary of protein supersecondary structures (Subramanian et al., IEEE data compression conference proceedings (DCC), 340-349, 2017). Specifically, this methodology uses the information-theoretic framework of minimum message length (MML) criterion for hypothesis selection (Wallace, Statistical and inductive inference by minimum message length, Springer Science & Business Media, New York, 2005). The best dictionary of supersecondary structures is the one that yields the most (lossless) compression on the source collection of folding patterns represented as tableaux (matrix representations that capture the essence of protein folding patterns (Lesk, J Mol Graph. 13:159-164, 1995). This book chapter outlines our MML methodology for inferring the supersecondary structure dictionary. The inferred dictionary is available at http://lcb.infotech.monash.edu.au/proteinConcepts/scop100/dictionary.html .

A novel statistical approach for clustering positive data based on finite inverted Beta-Liouville mixture models

Nowadays, a great number of positive data has been occurred naturally in many applications, however, it was not adequately analyzed. In this article, we propose a novel statistical approach for clustering multivariate positive data. Our approach is based on a finite mixture model of inverted Beta-Liouville (IBL) distributions, which is proper choice for modeling and analysis of positive vector data. We develop two different approaches to learn the proposed mixture model. Firstly, the maximum likelihood (ML) is utilized to estimate parameters of the finite inverted Beta-Liouville mixture model in which the right number of mixture components is determined according to the minimum message length (MML) criterion. Secondly, the variational Bayes (VB) is adopted to learn our model where the parameters and the number of mixture components can be determined simultaneously in a unified framework, without the requirement of using information criteria. We investigate the effectiveness of our model by conducting a series of experiments on both synthetic and real data sets.

Model selection and application to high-dimensional count data clustering

EDCM, the Exponential-family approximation to the Dirichlet Compound Multinomial (DCM), proposed by Elkan (2006), is an efficient statistical model for high-dimensional and sparse count data. EDCM models take into account the burstiness phenomenon correctly while being many times faster than DCM. This work proposes the use of Minimum Message Length (MML) criterion for determining the number of components that best describes the data with a finite EDCM mixture model. Parameters estimation is based on the previously proposed Deterministic Annealing Expectation-Maximization (DAEM) approach. The validation of the proposed unsupervised algorithm involves different real applications: text document modeling, topic novelty detection and hierarchical image clustering. A comparison with results obtained for other information-theory based selection criteria is provided.

Automatic Identification of Rock Fracture Sets Using Finite Mixture Models

The clustering and classification of fracture orientation data are crucial tasks in geotechnical engineering and rock engineering design. The explicit simulation of fracture orientations is always applied to compensate for the lack of direct measurements over the entire rock mass. In this study, a single step approach based on the theory of finite mixture models, where the component distributions are Fisher distributions, is proposed for automatic clustering and simulation of fracture orientation data. In the proposed workflow, the spherical K-means algorithm is applied to select the initial cluster centers, and the component-wise expectation–maximization algorithm using the minimum message length criterion is used to automatically determine the optimal number of fracture sets. An additional advantage of the proposed method is the representation of orientation data using a full sphere, instead of the conventional hemispherical characterization. The use of a full spherical representation effectively solves the issue of clustering for fractures with high dip angles. In addition, the calculation process of the mean direction is also simplified. The effectiveness of the model-based clustering method is tested with a complicated artificial data set and two real world data sets. Cluster validity is introduced to evaluate the clustering results. In addition, two other clustering algorithms are also presented for comparison. The results demonstrate that the proposed method can successfully detect the optimal number of clusters, and the parameters of the distributions are well estimated. In addition, the proposed method also exhibits good computational performance.

Anomaly Detection in IDSs by means of unsupervised greedy learning of finite mixture models

Nowadays, the worldwide Internet diffusion has made sharing information a piece of cake. This leads to the problem of protecting this huge amount of information (private data, commercial or strategic information) from those not authorized. A way for protecting it is the adoption of intrusion detection systems, which reveal whether an attacker is violating an information system. In this work, an algorithm for identifying anomalies on network traffic has been studied and developed. This is based on the unsupervised fitting of a set of network data by means of finite Gaussian mixtures models. Its key feature is the online selection of the number of mixture components together with the fitting parameter of each component. The best compromise between the description accuracy (many components) and the computational complexity (few components) is given by a derivation of the minimum message length criterion. The normal network behavior is assumed to be interpreted by the cluster with the highest covariance matrix, while the other smaller components are considered representing anomalies. We tested our technique with the well-known KDD99 Cup data set, in order to clearly compare our findings with the ones presenting the state of the art. Our results show the effectiveness of this approach, while encouraging for further improvements.

3D Human Pose Tracking Priors using Geodesic Mixture Models

We present a novel approach for learning a finite mixture model on a Riemannian manifold in which Euclidean metrics are not applicable and one needs to resort to geodesic distances consistent with the manifold geometry. For this purpose, we draw inspiration on a variant of the expectation-maximization algorithm, that uses a minimum message length criterion to automatically estimate the optimal number of components from multivariate data lying on an Euclidean space. In order to use this approach on Riemannian manifolds, we propose a formulation in which each component is defined on a different tangent space, thus avoiding the problems associated with the loss of accuracy produced when linearizing the manifold with a single tangent space. Our approach can be applied to any type of manifold for which it is possible to estimate its tangent space. Additionally, we consider using shrinkage covariance estimation to improve the robustness of the method, especially when dealing with very sparsely distributed samples. We evaluate the approach on a number of situations, going from data clustering on manifolds to combining pose and kinematics of articulated bodies for 3D human pose tracking. In all cases, we demonstrate remarkable improvement compared to several chosen baselines.

EM Algorithms for Weighted-Data Clustering with Application to Audio-Visual Scene Analysis.

Data clustering has received a lot of attention and numerous methods, algorithms and software packages are available. Among these techniques, parametric finite-mixture models play a central role due to their interesting mathematical properties and to the existence of maximum-likelihood estimators based on expectation-maximization (EM). In this paper we propose a new mixture model that associates a weight with each observed point. We introduce the weighted-data Gaussian mixture and we derive two EM algorithms. The first one considers a fixed weight for each observation. The second one treats each weight as a random variable following a gamma distribution. We propose a model selection method based on a minimum message length criterion, provide a weight initialization strategy, and validate the proposed algorithms by comparing them with several state of the art parametric and non-parametric clustering techniques. We also demonstrate the effectiveness and robustness of the proposed clustering technique in the presence of heterogeneous data, namely audio-visual scene analysis.

Minimum message length estimation of mixtures of multivariate Gaussian and von Mises-Fisher distributions

Mixture modelling involves explaining some observed evidence using a combination of probability distributions. The crux of the problem is the inference of an optimal number of mixture components and their corresponding parameters. This paper discusses unsupervised learning of mixture models using the Bayesian Minimum Message Length (MML) criterion. To demonstrate the effectiveness of search and inference of mixture parameters using the proposed approach, we select two key probability distributions, each handling fundamentally different types of data: the multivariate Gaussian distribution to address mixture modelling of data distributed in Euclidean space, and the multivariate von Mises-Fisher (vMF) distribution to address mixture modelling of directional data distributed on a unit hypersphere. The key contributions of this paper, in addition to the general search and inference methodology, include the derivation of MML expressions for encoding the data using multivariate Gaussian and von Mises-Fisher distributions, and the analytical derivation of the MML estimates of the parameters of the two distributions. Our approach is tested on simulated and real world data sets. For instance, we infer vMF mixtures that concisely explain experimentally determined three-dimensional protein conformations, providing an effective null model description of protein structures that is central to many inference problems in structural bioinformatics. The experimental results demonstrate that the performance of our proposed search and inference method along with the encoding schemes improve on the state of the art mixture modelling techniques.

On the Scalable Learning of Stochastic Blockmodel

Stochastic blockmodel (SBM) enables us to decompose and analyze an exploratory network without a priori knowledge about its intrinsic structure. However, the task of effectively and efficiently learning a SBM from a large-scale network is still challenging due to the high computational cost of its model selection and parameter estimation. To address this issue, we present a novel SBM learning algorithm referred to as BLOS (BLOckwise Sbm learning). Distinct from the literature, the model selection and parameter estimation of SBM are concurrently, rather than alternately, executed in BLOS by embedding the minimum message length criterion into a block-wise EM algorithm, which greatly reduces the time complexity of SBM learning without losing learning accuracy and modeling flexibility. Its effectiveness and efficiency have been tested through rigorous comparisons with the state-of-the-art methods on both synthetic and real-world networks.

Feature selection for clustering categorical data with an embedded modelling approach

AbstractResearch on the problem of feature selection for clustering continues to develop. This is a challenging task, mainly due to the absence of class labels to guide the search for relevant features. Categorical feature selection for clustering has rarely been addressed in the literature, with most of the proposed approaches having focused on numerical data. In this work, we propose an approach to simultaneously cluster categorical data and select a subset of relevant features. Our approach is based on a modification of a finite mixture model (of multinomial distributions), where a set of latent variables indicate the relevance of each feature. To estimate the model parameters, we implement a variant of the expectation‐maximization algorithm that simultaneously selects the subset of relevant features, using a minimum message length criterion. The proposed approach compares favourably with two baseline methods: a filter based on an entropy measure and a wrapper based on mutual information. The results obtained on synthetic data illustrate the ability of the proposed expectation‐maximization method to recover ground truth. An application to real data, referred to official statistics, shows its usefulness.