Minimum Description Length Criterion Research Articles

Research on the Speech Enhancement Method based on PCA/KLT Algorithms

In this paper, we propose a speech enhancement technique in terms of subspace methods to reduce the white or colored noise in strong background noise environment. This subspace approach based on Karhunen-Loève transform (KLT) and implemented via Principal Component Analysis (PCA). The subspace selection provided by the minimum description length (MDL) criterion. An offset factor generated from the white noise was used to modify the variance to adapt to the specified colored noise. The objective speech quality measures SegSNR have been introduced to evaluate the performance of the proposed method in time domain. A large amount of data and figures testify that our algorithm provides high performance for a large scale of input signal-to-noise ratio (-5~10dB). The performance of our algorithm is assessed in white and colored noise.

Dynamic Modeling for a Non-Linear System Based on the MOESP Algorithm

In order to measure accurately air fuel ratio in the engine exhaust gas, the Hammerstein model of the exhaust gas oxygen (EGO) sensor was identified using the multivariable output error state space (MOESP) algorithm. Firstly, a static model of the EGO sensor was identified based on engine experiment data as a static nonlinear part of the Hammerstein model. Then, the MOESP algorithm was used to build a state space model (SSM) of dynamic linear part of the Hammerstein model. To estimate model order, the Akaike Information Criterion (AIC) and the Minimum Description Length (MDL) criterion were computed, and validity of structure identification was verified by residual analysis. Then, the auto-regressive model with exogenous input (ARX) was compared with a subspace model. The result shows that a two-order SSM using MOESP algorithm is suitable to dynamic part of the Hammerstein model for EGO sensors.

The Consistency of MDL for Linear Regression Models With Increasing Signal-to-Noise Ratio

Recent work by Ding and Kay has demonstrated that the Bayesian information criterion (BIC) is an inconsistent estimator of model order in nested model selection as the noise variance τ*→ 0. Unfortunately, Ding and Kay have erroneously concluded that the minimum description length (MDL) principle also leads to inconsistent estimates of model order in this setting by equating BIC with MDL. This correspondence shows that only the earlier MDL criterion based on asymptotic assumptions has this problem, and proves that the new MDL linear regression criteria based on normalized maximum likelihood and Bayesian mixture codes satisfy the notion of consistency as τ*→ 0. The main result may be used as a basis to easily establish similar consistency results for other closely related information theoretic regression criteria.

Dimension estimation-based spectrum sensing for cognitive radio

In this article, we will derive closed-form expressions of false alarm probabilities for a given threshold for the dimension estimation-based detector (DED) using Akaike information criterion (AIC) and the minimum description length (MDL) criterion. Specifically, the DED algorithm will be formulated as a binary hypothesis test using AIC and MDL curves. Based on the proposed statistic test, we will express the probability of false alarm of the DED algorithm for a fixed threshold using the cumulative density function for the distribution of Tracy-Widom of order two. The derived analytical decision thresholds are verified with Monte-Carlo simulations and a comparison between simulation and analytical results to confirm the theoretical results are presented. These results confirm the very good match between simulation and theoretic results.

Source enumeration in large arrays using moments of eigenvalues and relatively few samples

This study presents a method based on minimum description length criterion to enumerate the incident waves impinging on a large array using a relatively small number of samples. The proposed scheme exploits the statistical properties of eigenvalues of the sample covariance matrix (SCM) of Gaussian processes. The authors use a number of moments of noise eigenvalues of the SCM in order to separate noise and signal subspaces more accurately. In particular, the authors assume a Marcenko–Pastur probability density function (pdf) for the eigenvalues of SCM associated with the noise subspace. We also use an enhanced noise variance estimator to reduce the bias leakage between the subspaces. Numerical simulations demonstrate that the proposed method estimates the true number of signals for large arrays and a relatively small number of snapshots. In particular, the proposed method requires less number of samples to achieve the same correct enumeration probability compared to the state-of-the-art methods. The authors evaluated the assumed pdf in order to justify the limitation and the behaviour of the proposed method for small number of snapshots and array sizes.

The right tool for the job? Comparing an evidence accumulation and a naive strategy selection model of decision making

AbstractAnalyses of multi‐attribute decision problems are dominated by accounts which assume people select from a repertoire of cognitive strategies to make decisions. This paper explores an alternative account based on sequential sampling and evidence accumulation. Two experiments varied aspects of a decision environment to examine competing models of decision behavior. The results highlighted the intra‐participant consistency but inter‐participant differences in the amount of evidence considered in decisions. This pattern was best captured by a sequential evidence accumulation model (SEQ) which treated pure Take‐The‐Best (TTB) and pure “rational” (RAT) models as special cases of a single model. The SEQ model was also preferred by the minimum description length (MDL) criterion to a naive strategy‐selection model (NSS) which assumed that TTB or RAT could be selected with some probability for each decision. Copyright © 2010 John Wiley & Sons, Ltd.

On Testing the Extent of Noncircularity

In this correspondence, we provide a multiple hypothesis test to detect the number of latent noncircular signals in a complex Gaussian random vector. Our method sequentially tests the results of individual generalized likelihood ratio test (GLRT) statistics with known asymptotic distributions to form the multiple hypothesis detector. Specifically, we are able to set a threshold yielding a precise probability of error. This test can be used to statistically determine if a given complex observation is circular Gaussian, and if not, how many latent signals in the observation are noncircular. Simulations are used to quantify the performance of the detector as compared to a detector based on the minimum description length (MDL) criterion. The utility of the detector is shown by applying it to a beamforming application using independent component analysis (ICA).

On the Second-Order Statistics of the EVD of Sample Covariance Matrices—Application to the Detection of Noncircular Or/and NonGaussian Components

This correspondence presents an asymptotic analysis of the eigenvalue decomposition (EVD) of the sample covariance matrix associated with independent identically distributed (i.i.d.) non necessarily circular and Gaussian data that extends the well known analysis presented in the literature for circular and Gaussian data. Closed-form expressions of the asymptotic bias and variance of the sample eigenvalues and eigenvectors are given. As an application of these extended expressions, the statistical performance analysis of the widely used minimum description length (MDL) criterion applied to the detection of the number of noncircular or/and non-Gaussian sources impinging on an array of sensors is considered with a particular attention paid to uncorrelated rectilinear sources.

An Object-Oriented Semantic Clustering Algorithm for High-Resolution Remote Sensing Images Using the Aspect Model

In this letter, we present a novel object-oriented semantic clustering algorithm for high-spatial-resolution remote sensing images using the probabilistic latent semantic analysis (PLSA) model coupled with neighborhood spatial information. First of all, an image collection is generated by partitioning a large satellite image into densely overlapped subimages. Then, the PLSA model is employed to model the image collection. Specifically, the image collection is partitioned into two subsets. One is used to learn topic models, where the number of topics is determined using a minimum description length criterion. The other is folded in using the learned topic models. Therefore, every pixel in each subimage has been allocated a topic label. At last, the cluster label of every pixel in the large satellite image is derived from the topic labels of multiple subimages which cover the pixel in the image collection. Experimental results over a QUICKBIRD image show that the clusters of the proposed algorithm are better than K-means and Iterative Self-Organizing Data Analysis Technique Algorithm in terms of object-oriented property.

Inconsistency of the MDL: On the Performance of Model Order Selection Criteria With Increasing Signal-to-Noise Ratio

In the problem of model order selection, it is well known that the widely used minimum description length (MDL) criterion is consistent as the sample size N → ∞ . However, the consistency as the noise variance σ2 → 0 has not been studied. In this paper, we find that the MDL is inconsistent as σ2 → 0. The result shows that the MDL has a tendency to overestimate the model order. We also prove that another criterion, the exponentially embedded family (EEF), is consistent as σ2 → 0. Therefore, in a high signal-to-noise (SNR) scenario, the EEF provides a better criterion to use for model order selection.

A Blind Collision Resolution Protocol Based on Cooperative Transmission

Network-assisted diversity multiple access (NDMA) was recently proposed to resolve collision. The blind NDMA method using independent component analysis (ICA) ICA-NDMA, has been developed to overcome the difficulties of synchronization and orthogonal identification codes required by the training-based NDMA protocols. In this paper, we propose a novel blind collision resolution protocol, named as ICA-CoopNDMA, by employing cooperative transmission in ICA-NDMA. When K-node collision happened, a cooperative transmission mechanism is adopted to collect different mixtures of the K original packets in the following slots. After that, the destination can recover the original packets using ICA. This protocol offers spatial diversity by using relay nodes and is robust to the wireless channel. In order to determine the collision multiplicity K, a cooperative blind detection algorithm is also proposed based on minimum description length (MDL) criterion. This algorithm saves one detection slot compared with conventional MDL-based method used by ICA-NDMA. Hence, the delay of ICA-CoopNDMA is reduced. Simulation and theoretical analysis results show that, the proposed protocol can work effectively under both fast and slow fading channels. Compared with ICA-NDMA, the maximum stable throughput and average packet delay are obviously better, and the energy efficiency is higher under heavy traffic load.

An information-theoretic framework for semantic-multimedia retrieval

This article is set in the context of searching text and image repositories by keyword. We develop a unified probabilistic framework for text, image, and combined text and image retrieval that is based on the detection of keywords (concepts) using automated image annotation technology. Our framework is deeply rooted in information theory and lends itself to use with other media types. We estimate a statistical model in a multimodal feature space for each possible query keyword. The key element of our framework is to identify feature space transformations that make them comparable in complexity and density. We select the optimal multimodal feature space with a minimum description length criterion from a set of candidate feature spaces that are computed with the average-mutual-information criterion for the text part and hierarchical expectation maximization for the visual part of the data. We evaluate our approach in three retrieval experiments (only text retrieval, only image retrieval, and text combined with image retrieval), verify the framework's low computational complexity, and compare with existing state-of-the-art ad-hoc models.

On Semi-Supervised Learning Genetic-Based and Deterministic Annealing EM Algorithm for Dirichlet Mixture Models

We propose a genetic-based and deterministic annealing expectation-maximization (GA&DA-EM) algorithm for learning Dirichlet mixture models from multivariate data. This algorithm is capable of selecting the number of components of the model using the minimum description length (MDL) criterion. Our approach benefits from the properties of Genetic algorithms and deterministic annealing algorithm by combination of both into a single procedure. The population-based stochastic search of the GA&DA explores the search space more thoroughly than the EM method. Therefore, our algorithm enables escaping from local optimal solutions since the algorithm becomes less sensitive to its initialization. The GA&DA-EM algorithm is elitist which maintains the monotonic convergence property of the EM algorithm. We conducted experiments on the WebKB and 20NEWSGROUPS. The results show that show that 1) the GA&DA-EM outperforms the EM method since: Our approach identifies the number of components which were used to generate the underlying data more often than the EM algorithm. 2) the algorithm alternatives to EM that overcoming the challenges of local maxima.

A New Distance Measure for a Variable-Sized Acoustic Model Based on MDL Technique

Embedding a large vocabulary speech recognition system in mobile devices requires a reduced acoustic model obtained by eliminating redundant model parameters. In conventional optimization methods based on the minimum description length (MDL) criterion, a binary Gaussian tree is built at each state of a hidden Markov model by iteratively finding and merging similar mixture components. An optimal subset of the tree nodes is then selected to generate a downsized acoustic model. To obtain a better binary Gaussian tree by improving the process of finding the most similar Gaussian components, this paper proposes a new distance measure that exploits the difference in likelihood values for cases before and after two components are combined. The mixture weight of Gaussian components is also introduced in the component merging step. Experimental results show that the proposed method outperforms MDL-based optimization using either a Kullback-Leibler (KL) divergence or weighted KL divergence measure. The proposed method could also reduce the acoustic model size by 50% with less than a 1.5% increase in error rate compared to a baseline system.

Adaptive Sequential Prediction of Multidimensional Signals With Applications to Lossless Image Coding

We investigate the problem of designing adaptive sequential linear predictors for the class of piecewise autoregressive multidimensional signals, and adopt an approach of minimum description length (MDL) to determine the order of the predictor and the support on which the predictor operates. The design objective is to strike a balance between the bias and variance of the prediction errors in the MDL criterion. The predictor design problem is particularly interesting and challenging for multidimensional signals (e.g., images and videos) because of the increased degree of freedom in choosing the predictor support. Our main result is a new technique of sequentializing a multidimensional signal into a sequence of nested contexts of increasing order to facilitate the MDL search for the order and the support shape of the predictor, and the sequentialization is made adaptive on a sample by sample basis. The proposed MDL-based adaptive predictor is applied to lossless image coding, and its performance is empirically established to be the best among all the results that have been published till present.

B-Spline Model Order Selection With Optimal MDL Criterion Applied to Speech Fundamental Frequency Stylization

In the speech processing field, stylization of fundamental frequency <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> has been subjected to numerous works. Models proposed in the literature rely on knowledge stemming from phonology and linguistics. We propose an approach that deals with the issue of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> curve stylization requiring as few linguistic assumptions as possible and in the framework of B-spline models. A B-spline model, characterized by a sequence of knots with which control points are associated, enables the formalization of discontinuities in the derivatives of the observed values sequence. Beyond the implementation of a B-spline model to stylize an open curve sampled using a constant step, we address the problem of the optimal model order choice. We propose to use a parsimony criterion based on a minimum description length (MDL) approach, in order to optimize the number of knots. We derive several criteria relying on bounds estimated from parameter values. We demonstrate the optimality of these choices in the theoretical MDL framework. We introduce a notion of variable precision of parameters which enables a good compromise between the modeling precision and degrees of freedom of the estimated models. Experiments are performed on a French speech corpus and compare three MDL criteria. The use of both B-spline model and MDL methodology enables an efficient modeling of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> curves and provides an RMS error around 1 Hz while allowing a relatively high compression rate about 40%.

Identification of time‐varying Wiener systems with unknown parameters

AbstractWiener systems which consist of a dynamic linear block followed by a static nonlinear element have been used in numerous applications. In many cases, the system parameters are affected by changes in the environmental conditions. This paper describes a new approach to the on‐line identification of time‐varying Wiener systems. The time‐varying linear parameters and the static nonlinear characteristics are estimated by neural networks which can represent various nonlinear characteristics. The initial states of the linear model in each estimation window are not available in the Wiener systems. Thus, the initial states and the other system parameters are estimated simultaneously by nonlinear optimization techniques. Furthermore, the optimal numbers of hidden units in the neural networks are determined by the minimum description length (MDL) criterion. As a result of simulation by this method, more accurate parameters can be obtained than without estimation of the initial states and MDL. © 2010 Wiley Periodicals, Inc. Electron Comm Jpn, 93(4): 1–9, 2010; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ecj.10247

Simplicity and likelihood: An axiomatic approach

We suggest a model in which theories are ranked given various databases. Certain axioms on such rankings imply a numerical representation that is the sum of the log-likelihood of the theory and a fixed number for each theory, which may be interpreted as a measure of its complexity. This additive combination of log-likelihood and a measure of complexity generalizes both the Akaike Information Criterion and the Minimum Description Length criterion, which are well known in statistics and in machine learning, respectively. The axiomatic approach is suggested as a way to analyze such theory-selection criteria and judge their reasonability based on finite databases.

MMSE-Based MDL Method for Accurate Source Number Estimation

In civilian communication systems, the signature sequence of the desired signal in training phase is known to the receiver. In this letter, using the mutual information, we bridge the probability density function and minimum mean-square error (MMSE) between the observed data and training sequence of the desired signal, and then employ the MMSE to construct a minimum description length (MDL) criterion for accurate source enumeration. Numerical results demonstrate that the proposed method is superior to existing MDL methods in terms of detection performance particularly for small number of snapshots and/or source angular separation.

Detection and Localization of Multiple Unresolved Extended Targets via Monopulse Radar Signal Processing

When the size of targets is comparable to the range resolution of monopulse radars, these targets should be considered as extended rather than point targets. If several closely-spaced targets fall within the same radar beam and between adjacent matched filter samples in range, the full monopulse information from all of these samples can and should be used to resolve these targets, i.e., estimate the number of targets and their respective angles-of-arrival and ranges. To detect and localize multiple unresolved extended targets, we establish a model for monopulse radar returns from extended objects, and present a maximum likelihood estimator (MLE) to localize the targets. Rissanen's minimum description length (MDL) criterion will be used to decide the number of existing extended objects. We also derive the upper limit on the number of targets and their scattering centers that can be resolved, and we provide necessary conditions for these targets to be uniquely identified. We compare the new extended target monopulse processing scheme with previously developed point-target monopulse techniques in the simulations.