Minimum Description Length Principle Research Articles

An Estimator of Mutual Information and its Application to Independence Testing

This paper proposes a novel estimator of mutual information for discrete and continuous variables. The main feature of this estimator is that it is zero for a large sample size n if and only if the two variables are independent. The estimator can be used to construct several histograms, compute estimations of mutual information, and choose the maximum value. We prove that the number of histograms constructed has an upper bound of O(log n) and apply this fact to the search. We compare the performance of the proposed estimator with an estimator of the Hilbert-Schmidt independence criterion (HSIC), though the proposed method is based on the minimum description length (MDL) principle and the HSIC provides a statistical test. The proposed method completes the estimation in O(n log n) time, whereas the HSIC kernel computation requires O(n3) time. We also present examples in which the HSIC fails to detect independence but the proposed method successfully detects it.

Community Detection in Multi-Partite Multi-Relational Networks Based on Information Compression

Community detection in uni-partite single-relational networks which contain only one type of nodes and edges has been extensively studied in the past decade. However, many real-world systems are naturally described as multi-partite multi-relational networks which contain multiple types of nodes and edges. In this paper, we propose an information compression based method for detecting communities in such networks. Specifically, based on the minimum description length (MDL) principle, we propose a quality function for evaluating partitions of a multi-partite multi-relational network into communities, and develop a heuristic algorithm for optimizing the quality function. We demonstrate that our method outperforms the state-of-the-art techniques in both synthetic and real-world networks.

Major migration corridors of mesoscale ocean eddies in the South China Sea from 1992 to 2012

It has become a routine to automatically identify mesoscale ocean eddies in the world's oceans and reconstruct their trajectories from remote sensing data. However, the major migration pathways along which eddies mainly propagate are not clear, particularly in the South China Sea (SCS). This study utilized a trajectory partition-and-group method to quantitatively measure and group trajectories of eddies in the SCS from 1992 to 2012 to extract their major migration corridors. The trajectories were first simplified into segments using the minimum description length (MDL) principle, their origin and destination (OD) points, and their trajectory partitions (TP) between two consecutive tracking times, respectively. The MDL-, OD-, and TP-based segments were then respectively grouped into clusters using the density-based spatial clustering of applications with noise (DBSCAN) algorithm. Representative trajectories, i.e., the major migration corridors, were then extracted from different clusters. Results show that the MDL-based corridors are the most tenable in revealing the migration corridors of eddies in the SCS. The major MDL-based migration corridors of both anticyclonic and cyclonic eddy in the SCS mainly extend westward and show a meridional propagation toward the equator. The different major migration corridors in the northern, central, and southern SCS could be attributed to the various influences of seabed topography and/or background currents in these regions.

An efficient implementation of EMD algorithm for motif discovery in time series data

The extended motif discovery (EMD) algorithm, developed by Tanaka et al. (2005), is a well-known time series motif discovery algorithm which is based on minimum description length principle. One unique feature of the EMD algorithm is that it can determine the suitable length of the motif and hence does not require the length of the motif as a parameter supplied by user. Another interesting feature of the EMD is the lengths of each instances of a motif can be a bit different from each other and hence Tanaka et al. suggested that dynamic time warping (DTW) distance should be used to calculate the distances between the motif instances in this case. Due to the second feature, the EMD algorithm leads to a high computational complexity and is not easy to implement in practice. In this paper, we present an efficient implementation of the EMD algorithm in which we apply homothetic transformation to convert all pattern instances of different lengths into the same length so that we can easily calculate Euclidean distances between them. This modification accelerates the execution of the EMD remarkably and makes it easier to implement. Experimental results on seven real world time series datasets demonstrate the effectiveness of our EMD implementation method in time series motif discovery.

Kernel methods for transfer learning to avoid negative transfer

In the big data era, with the devolvement of information storage and processing capability of computers, new tasks become more and more complex and also with higher requirements. In the other side, out-of-date information of old tasks are abounded at will. Due to the considerable cost for classifying the emerged new tasks, transfer learning techniques are developed to extract useful knowledge from existing similar datasets, and a large number of research works have been published in recent years. However, an open problem in transfer learning is the negative transfer which happens due to different distributions among tasks. In this manuscript, we are targeting at proposing a kernel method to evaluate both the task relatedness and the instance similarities. The minimum description length principle MDLP is adopted which was proved effective for evaluating models in transfer learning scenario. Extensive experiments show the effectiveness of our algorithm in terms of the classification accuracy in real datasets.

Hybrid Genetic Algorithm for Medical Image Feature Extraction and Selection

For a hybrid medical image retrieval system, a genetic algorithm (GA) approach is presented for the selection of dimensionality reduced set of features. This system was developed in three phases. In first phase, three distinct algorithm are used to extract the vital features from the images. The algorithm devised for the extraction of the features are Texton based contour gradient extraction algorithm, Intrinsic pattern extraction algorithm and modified shift invariant feature transformation algorithm. In the second phase to identify the potential feature vector GA based feature selection is done, using a hybrid approach of “Branch and Bound Algorithm” and “Artificial Bee Colony Algorithm” using the breast cancer, Brain tumour and thyroid images. The Chi Square distance measurement is used to assess the similarity between query images and database images. A fitness function with respect Minimum description length principle were used as initial requirement for genetic algorithm. In the third phase to improve the performance of the hybrid content based medical image retrieval system diverse density based relevance feedback method is used. The term hybrid is used as this system can be used to retrieve any kind of medical image such as breast cancer, brain tumour, lung cancer, thyroid cancer and so on. This machine learning based feature selection method is used to reduce the existing system dimensionality problem. The experimental result shows that the GA driven image retrieval system selects optimal subset of feature to identify the right set of images.

Bayes Factors, relations to Minimum Description Length, and overlapping model classes

This article presents a non-technical perspective on two prominent methods for analyzing experimental data in order to select among model classes. Each class consists of model instances; each instance predicts a unique distribution of data outcomes. One method is Bayesian Model Selection (BMS), instantiated with the Bayes factor. The other is based on the Minimum Description Length principle (MDL), instantiated by a variant of Normalized Maximum Likelihood (NML): the variant is termed NML* and takes prior probabilities into account. The methods are closely related. The Bayes factor is a ratio of two values: V1 for model class M1, and V2 for M2. Each Vjis the sum over the instances of Mj,of thejoint probabilities (prior times likelihood) for the observed data, normalized by a sum of such sums for all possible data outcomes. NML* is qualitatively similar: The value it assigns to each class is the maximum over the instances in Miof the joint probability for the observed data normalized by a sum of such maxima for all possible data outcomes. The similarity of BMS to NML* is particularly close when model classes do not have instances that overlap, a way of comparing model classes that we advocate generally. These observations and suggestions are illustrated throughout with use of a simple example borrowed from Heck, Wagenmakers, and Morey (2015) in which the instances predict a binomial distribution of number of successes in N trials. The model classes posit the binomial probability of success to lie in various regions of the interval [0,1]. We illustrate the theory and the example not with equations but with tables coupled with simple arithmetic. Using the binomial example we carry out comparisons of BMS and NML* that do and do not involve model classes that overlap, and do and do not have uniform priors. When the classes do not overlap BMS and NML* produce qualitatively similar results.

Association Discovery in Two-View Data

Two-view datasets are datasets whose attributes are naturally split into two sets, each providing a different view on the same set of objects. We introduce the task of finding small and non-redundant sets of associations that describe how the two views are related. To achieve this, we propose a novel approach in which sets of rules are used to translate one view to the other and vice versa. Our models, dubbed translation tables , contain both unidirectional and bidirectional rules that span both views and provide lossless translation from either of the views to the opposite view. To be able to evaluate different translation tables and perform model selection, we present a score based on the Minimum Description Length (MDL) principle. Next, we introduce three Translator algorithms to find good models according to this score. The first algorithm is parameter-free and iteratively adds the rule that improves compression most. The other two algorithms use heuristics to achieve better trade-offs between runtime and compression. The empirical evaluation on real-world data demonstrates that only modest numbers of associations are needed to characterize the two-view structure present in the data, while the obtained translation rules are easily interpretable and provide insight into the data.

Boundary-to-Marker Evidence-Controlled Segmentation and MDL-Based Contour Inference for Overlapping Nuclei.

This paper presents a novel method for automated morphology delineation and analysis of cell nuclei in histopathology images. Combining the initial segmentation information and concavity measurement, the proposed method first segments clusters of nuclei into individual pieces, avoiding segmentation errors introduced by the scale-constrained Laplacian-of-Gaussian filtering. After that a nuclear boundary-to-marker evidence computing is introduced to delineate individual objects after the refined segmentation process. The obtained evidence set is then modeled by the periodic B-splines with the minimum description length principle, which achieves a practical compromise between the complexity of the nuclear structure and its coverage of the fluorescence signal to avoid the underfitting and overfitting results. The algorithm is computationally efficient and has been tested on the synthetic database as well as 45 real histopathology images. By comparing the proposed method with several state-of-the-art methods, experimental results show the superior recognition performance of our method and indicate the potential applications of analyzing the intrinsic features of nuclei morphology.

Generalized cylinder decomposition

Decomposing a complex shape into geometrically simple primitives is a fundamental problem in geometry processing. We are interested in a shape decomposition problem where the simple primitives sought are generalized cylinders , which are ubiquitous in both organic forms and man-made artifacts. We introduce a quantitative measure of cylindricity for a shape part and develop a cylindricity-driven optimization algorithm, with a global objective function, for generalized cylinder decomposition. As a measure of geometric simplicity and following the minimum description length principle, cylindricity is defined as the cost of representing a cylinder through skeletal and cross-section profile curves. Our decomposition algorithm progressively builds local to non-local cylinders, which form over-complete covers of the input shape. The over-completeness of the cylinder covers ensures a conservative buildup of the cylindrical parts, leaving the final decision on decomposition to global optimization. We solve the global optimization by finding an exact cover, which optimizes the global objective function. We demonstrate results of our optimal decomposition algorithm on numerous examples and compare with other alternatives.

Generalized Context Modeling With Multi-Directional Structuring and MDL-Based Model Selection for Heterogeneous Data Compression

This paper proposes generalized context modeling (GCM) for heterogeneous data compression. The proposed model extends the suffix of predicted subsequences in classic context modeling to arbitrary combinations of symbols in multiple directions. To address the selection of contexts, GCM constructs a model graph with a combinatorial structuring of finite order combination of predicted symbols as its nodes. The estimated probability for prediction is obtained by weighting over a class of context models that contain all the occurrences of nodes in the model graph. Moreover, separable context modeling in each direction is adopted for efficient prediction. To find optimal class of context models for prediction, the normalized maximum likelihood (NML) function is developed to estimate their structures and parameters, especially for heterogeneous data with large sizes. Furthermore, it is refined by context pruning to exclude the redundant models. Such model selection is optimal in the sense of minimum description length (MDL) principle, whose divergence is proven to be consistent with the actual distribution. It is shown that upper bounds of model redundancy for GCM are irrelevant to the size of data. GCM is validated in an extensive field of applications, e.g., Calgary corpus, executable files, and genomic data. Experimental results show that it outperforms most state-of-the-art context modeling algorithms reported.

Grammar induction using bit masking oriented genetic algorithm and comparative analysis

This paper presents bit masking oriented genetic algorithm (BMOGA) for context free grammar induction. It takes the advantages of crossover and mutation mask-fill operators together with a Boolean based procedure in two phases to guide the search process from ith generation to (i+1)th generation. Crossover and mutation mask-fill operations are performed to generate the proportionate amount of population in each generation. A parser has been implemented checks the validity of the grammar rules based on the acceptance or rejection of training data on the positive and negative strings of the language. Experiments are conducted on collection of context free and regular languages. Minimum description length principle has been used to generate a corpus of positive and negative samples as appropriate for the experiment. It was observed that the BMOGA produces successive generations of individuals, computes their fitness at each step and chooses the best when reached to threshold (termination) condition. As presented approach was found effective in handling premature convergence therefore results are compared with the approaches used to alleviate premature convergence. The analysis showed that the BMOGA performs better as compared to other algorithms such as: random offspring generation approach, dynamic allocation of reproduction operators, elite mating pool approach and the simple genetic algorithm. The term success ratio is used as a quality measure and its value shows the effectiveness of the BMOGA. Statistical tests indicate superiority of the BMOGA over other existing approaches implemented.

An information theoretic approach to pedigree reconstruction

Network structure is a dominant feature of many biological systems, both at the cellular level and within natural populations. Advances in genotype and gene expression screening made over the last few decades have permitted the reconstruction of these networks. However, resolution to a single model estimate will generally not be possible, leaving open the question of the appropriate method of formal statistical inference. The nonstandard structure of the problem precludes most traditional statistical methodologies. Alternatively, a Bayesian approach provides a natural methodology for formal inference. Construction of a posterior density on the space of network structures allows formal inference regarding features of network structure using specific marginal posterior distributions.An information theoretic approach to this problem will be described, based on the Minimum Description Length principle. This leads to a Bayesian inference model based on the information content of data rather than on more commonly used probabilistic models. The approach is applied to the problem of pedigree reconstruction based on genotypic data. Using this application, it is shown how the MDL approach is able to provide a truly objective control for model complexity.A two-cohort model is used for a simulation study. The MDL approach is compared to COLONY-2, a well known pedigree reconstruction application. The study highlights the problem of genotyping error modeling. COLONY-2 requires prior error rate estimates, and its accuracy proves to be highly sensitive to these estimates. In contrast, the MDL approach does not require prior error rate estimates, and is able to accurately adjust for genotyping error across the range of models considered.

Discovering compressing serial episodes from event sequences

Most pattern mining methods yield a large number of frequent patterns, and isolating a small relevant subset of patterns is a challenging problem of current interest. In this paper, we address this problem in the context of discovering frequent episodes from symbolic time-series data. Motivated by the Minimum Description Length principle, we formulate the problem of selecting relevant subset of patterns as one of searching for a subset of patterns that achieves best data compression. We present algorithms for discovering small sets of relevant non-redundant episodes that achieve good data compression. The algorithms employ a novel encoding scheme and use serial episodes with inter-event constraints as the patterns. We present extensive simulation studies with both synthetic and real data, comparing our method with the existing schemes such as GoKrimp and SQS. We also demonstrate the effectiveness of these algorithms on event sequences from a composable conveyor system; this system represents a new application area where use of frequent patterns for compressing the event sequence is likely to be important for decision support and control.

Estimation of Multiple-Regime Threshold Autoregressive Models With Structural Breaks

The threshold autoregressive (TAR) model is a class of nonlinear time series models that have been widely used in many areas. Due to its nonlinear nature, one major difficulty in fitting a TAR model is the estimation of the thresholds. As a first contribution, this article develops an automatic procedure to estimate the number and values of the thresholds, as well as the corresponding AR order and parameter values in each regime. These parameter estimates are defined as the minimizers of an objective function derived from the minimum description length (MDL) principle. A genetic algorithm (GA) is constructed to efficiently solve the associated minimization problem. The second contribution of this article is the extension of this framework to piecewise TAR modeling; that is, the time series is partitioned into different segments for which each segment can be adequately modeled by a TAR model, while models from adjacent segments are different. For such piecewise TAR modeling, a procedure is developed to estimate the number and locations of the breakpoints, together with all other parameters in each segment. Desirable theoretical results are derived to lend support to the proposed methodology. Simulation experiments and an application to an U.S. GNP data are used to illustrate the empirical performances of the methodology. Supplementary materials for this article are available online.

CCSpan: Mining closed contiguous sequential patterns

Existing closed sequential pattern mining generates a more compact yet complete resulting set compared with general sequential pattern mining. However, conventional closed sequential pattern mining algorithms pose a great challenge at spawning a large number of inefficient and redundant patterns, especially when using low support thresholds or pattern-enriched databases. Driven by wide applications of sequential patterns with contiguous constraint, we propose CCSpan (Closed Contiguous Sequential pattern mining), an efficient algorithm for mining closed contiguous sequential patterns, which contributes to a much more compact pattern set but with the same information w.r.t. closed sequential patterns. Moreover, with the shorter feature of patterns, the closed contiguous sequential patterns are preferred for feature selection and sequence classification based on the Minimum Description Length principle. CCSpan adopts a novel snippet-growth paradigm to generate a series of snippets as candidates, each of which is attached with a set of item(s) that precisely record the pattern’s occurrences in the database, and CCSpan leverages three pruning techniques to improve the computational efficiency significantly. Our experiments based on both sparse and dense datasets demonstrated that CCSpan is efficient and scalable in terms of both database size and support threshold.

Mining and clustering mobility evolution patterns from social media for urban informatics

In this paper, given a set of check-in data, we aim at discovering representative daily movement behavior of users in a city. For example, daily movement behavior on a weekday may show users moving from one to another spatial region associated with time information. Since check-in data contain both spatial and temporal information, we propose a mobility evolution pattern to capture the daily movement behavior of users in a city. Furthermore, given a set of daily mobility evolution patterns, we formulate their similarity distances and then discover representative mobility evolution patterns via the clustering process. Representative mobility evolution patterns are able to infer major movement behavior in a city, which could bring some valuable knowledge for urban planning. Specifically, mobility evolution patterns consist of segments with the spatial region distribution and the corresponding time interval. To measure good segmentation from a set of check-in data, we formulate the problem of mining evolution patterns as a compression problem. In particular, we compute the representation length of the patterns based on the Minimum Description Length principle. Since the number of daily mobility evolution patterns is huge, we further cluster the daily mobility evolution patterns into groups and discover representative patterns. Note that we use the concept of locality-sensitive hashing to accelerate the cluster performance. To evaluate our proposed algorithms, we conducted experiments on the Gowalla and Brightkite datasets, and the experimental results show the effectiveness and efficiency of our proposed algorithms.

An improved unsupervised approach to word segmentation

ESA is an unsupervised approach to word segmentation previously proposed by Wang, which is an iterative process consisting of three phases: Evaluation, Selection and Adjustment. In this article, we propose ExESA, the extension of ESA. In ExESA, the original approach is extended to a 2-pass process and the ratio of different word lengths is introduced as the third type of information combined with cohesion and separation. A maximum strategy is adopted to determine the best segmentation of a character sequence in the phrase of Selection. Besides, in Adjustment, ExESA re-evaluates separation information and individual information to overcome the overestimation frequencies. Additionally, a smoothing algorithm is applied to alleviate sparseness. The experiment results show that ExESA can further improve the performance and is time-saving by properly utilizing more information from un-annotated corpora. Moreover, the parameters of ExESA can be predicted by a set of empirical formulae or combined with the minimum description length principle.

The flexibility of models of recognition memory: The case of confidence ratings

The normalized maximum likelihood (NML) index is a model-selection index derived from the minimum-description length principle. In contrast to traditional model-selection indices, it also quantifies differences in flexibility between models related to their functional form. We present a new method for computing the NML index for models of categorical data that parameterize multinomial or product-multinomial distributions and apply it to comparing the flexibility of major models of recognition memory for confidence-rating based receiver-operating-characteristic (ROC) data. NML penalties are tabulated for datasets of typical sizes and interpolation functions are fitted that allow one to interpolate NML penalties for datasets with sizes between the tabulated ones. Recovery studies suggest that the NML index performs better than traditional model-selection indices in model selection from ROC data. In an NML-based meta-analysis of 850 ROC datasets, versions of the dual-process signal detection models received most support followed by the finite mixture signal detection model and constrained versions of two-high threshold models.

Predefined pattern detection in large time series

Predefined pattern detection from time series is an interesting and challenging task. In order to reduce its computational cost and increase effectiveness, a number of time series representation methods and similarity measures have been proposed. Most of the existing methods focus on full sequence matching, that is, sequences with clearly defined beginnings and endings, where all data points contribute to the match. These methods, however, do not account for temporal and magnitude deformations in the data and result to be ineffective on several real-world scenarios where noise and external phenomena introduce diversity in the class of patterns to be matched. In this paper, we present a novel pattern detection method, which is based on the notions of templates, landmarks, constraints and trust regions. We employ the Minimum Description Length (MDL) principle for time series preprocessing step, which helps to preserve all the prominent features and prevents the template from overfitting. Templates are provided by common users or domain experts, and represent interesting patterns we want to detect from time series. Instead of utilising templates to match all the potential subsequences in the time series, we translate the time series and templates into landmark sequences, and detect patterns from landmark sequence of the time series. Through defining constraints within the template landmark sequence, we effectively extract all the landmark subsequences from the time series landmark sequence, and obtain a number of landmark segments (time series subsequences or instances). We model each landmark segment through scaling the template in both temporal and magnitude dimensions. To suppress the influence of noise, we introduce the concept of trust region, which not only helps to achieve an improved instance model, but also helps to catch the accurate boundaries of instances of the given template. Based on the similarities derived from instance models, we introduce the probability density function to calculate a similarity threshold. The threshold can be used to judge if a landmark segment is a true instance of the given template or not. To evaluate the effectiveness and efficiency of the proposed method, we apply it to two real-world datasets. The results show that our method is capable of detecting patterns of temporal and magnitude deformations with competitive performance.