Infinite Mixture Research Articles

Automated Assessment of Disease Progression in Acute Myeloid Leukemia by Probabilistic Analysis of Flow Cytometry Data.

Flow cytometry (FC) is a widely acknowledged technology in diagnosis of acute myeloid leukemia (AML) and has been indispensable in determining progression of the disease. Although FC plays a key role as a posttherapy prognosticator and evaluator of therapeutic efficacy, the manual analysis of cytometry data is a barrier to optimization of reproducibility and objectivity. This study investigates the utility of our recently introduced nonparametric Bayesian framework in accurately predicting the direction of change in disease progression in AML patients using FC data. The highly flexible nonparametric Bayesian model based on the infinite mixture of infinite Gaussian mixtures is used for jointly modeling data from multiple FC samples to automatically identify functionally distinct cell populations and their local realizations. Phenotype vectors are obtained by characterizing each sample by the proportions of recovered cell populations, which are, in turn, used to predict the direction of change in disease progression for each patient. We used 200 diseased and nondiseased immunophenotypic panels for training and tested the system with 36 additional AML cases collected at multiple time points. The proposed framework identified the change in direction of disease progression with accuracies of 90% (nine out of ten) for relapsing cases and 100% (26 out of 26) for the remaining cases. We believe that these promising results are an important first step toward the development of automated predictive systems for disease monitoring and continuous response evaluation. Automated measurement and monitoring of therapeutic response is critical not only for objective evaluation of disease status prognosis but also for timely assessment of treatment strategies.

Unsupervised Profiling of Microglial Arbor Morphologies and Distribution Using a Nonparametric Bayesian Approach

The morphologies of microglial arbors in brain tissue are important indicators of their state of activation, and their spatial distribution indicates patterns of tissue perturbation. This paper presents a non-parametric Bayesian approach to unsupervised quantitative profiling of microglial activation states, and mapping their 3-D spatial distributions across extended brain tissue regions imaged by mosaiced confocal microscopy. Our approach is to perform large-scale automated reconstruction of all the microglia using a learned label-consistent dictionary model for the microglial processes, and extract 136 arbor features per cell. We then fit an infinite Gaussian mixture model with collapsed Gibbs sampling to the arbor features, to discover arbor-morphological classes, their morphological ‘signatures’, and the associated covariance matrices all in an unsupervised manner. This nonparametric approach can handle clusters with arbitrarily shaped covariance matrices, perform automatic model selection, handle unbalanced clusters, and detect overlapping classes, without the need to specify the number of clusters. The results can be used for diverse investigational purposes, including: population-scale modeling, automated discovery of arbor classes and their distinguishing quantitative signatures and features, quantifying the nature and extent of the cellular heterogeneity, and spatial mapping of cellular arbor morphologies, and mapping tissue alterations.

Precise periodic components estimation for chronobiological signals through Bayesian Inference with sparsity enforcing prior.

The toxicity and efficacy of more than 30 anticancer agents present very high variations, depending on the dosing time. Therefore, the biologists studying the circadian rhythm require a very precise method for estimating the periodic component (PC) vector of chronobiological signals. Moreover, in recent developments, not only the dominant period or the PC vector present a crucial interest but also their stability or variability. In cancer treatment experiments, the recorded signals corresponding to different phases of treatment are short, from 7 days for the synchronization segment to 2 or 3 days for the after-treatment segment. When studying the stability of the dominant period, we have to consider very short length signals relative to the prior knowledge of the dominant period, placed in the circadian domain. The classical approaches, based on Fourier transform (FT) methods are inefficient (i.e., lack of precision) considering the particularities of the data (i.e., the short length). Another particularity of the signals considered in such experiments is the level of noise: such signals are very noisy and establishing the periodic components that are associated with the biological phenomena and distinguishing them from the ones associated with the noise are difficult tasks. In this paper, we propose a new method for the estimation of the PC vector of biomedical signals, using the biological prior informations and considering a model that accounts for the noise. The experiments developed in cancer treatment context are recording signals expressing a limited number of periods. This is a prior information that can be translated as the sparsity of the PC vector. The proposed method considers the PC vector estimation as an Inverse Problem (IP) using the general Bayesian inference in order to infer the unknown of our model, i.e. the PC vector but also the hyperparameters (i.e the variances). The sparsity prior information is modeled using a sparsity enforcing prior law. In this paper, we propose a Student’s t distribution, viewed as the marginal distribution of a bivariate normal-inverse gamma distribution. We build a general infinite Gaussian scale mixture (IGSM) hierarchical model where we assign prior distributions also for the hyperparameters. The expression of the joint posterior law of the unknown PC vector and hyperparameters is obtained via Bayes rule, and then, the unknowns are estimated via joint maximum a posteriori (JMAP) or posterior mean (PM). For the PM estimator, the expression of the posterior distribution is approximated by a separable one, via variational Bayesian approximation (VBA), using the Kullback-Leibler (KL) divergence. For the PM estimation, two possibilities are considered: an approximation with a partially separable distribution and an approximation with a fully separable one. Both resulting algorithms corresponding to the PM estimation and the one corresponding to the JMAP estimation are iterative algorithms. The algorithms are presented in detail and are compared with the ones corresponding to the Gaussian model. We examine the convergency of the algorithms and give simulation results to compare their performances. Finally, we show simulation results on synthetic and real data in cancer treatment applications. The real data considered in this paper examines the rest-activity patterns of KI/KI Per2::luc mouse, aged 10 weeks, singly housed in RealTime Biolumicorder (RT-BIO).Electronic supplementary materialThe online version of this article (doi:10.1186/s13637-015-0033-6) contains supplementary material, which is available to authorized users.

Simple approximate MAP inference for Dirichlet processes mixtures

The Dirichlet process mixture model (DPMM) is a ubiquitous, flexible Bayesian nonparametric statistical model. However, full probabilistic inference in this model is analytically intractable, so that computationally intensive techniques such as Gibbs sampling are required. As a result, DPMM-based methods, which have considerable potential, are restricted to applications in which computational resources and time for inference is plentiful. For example, they would not be practical for digital signal processing on embedded hardware, where computational resources are at a serious premium. Here, we develop a simplified yet statistically rigorous approximate maximum a-posteriori (MAP) inference algorithm for DPMMs. This algorithm is as simple as DP-means clustering, solves the MAP problem as well as Gibbs sampling, while requiring only a fraction of the computational effort. (For freely available code that implements the MAP-DP algorithm for Gaussian mixtures see http://www.maxlittle.net/.) Unlike related small variance asymptotics (SVA), our method is non-degenerate and so inherits the “rich get richer” property of the Dirichlet process. It also retains a non-degenerate closed-form likelihood which enables out-of-sample calculations and the use of standard tools such as cross-validation. We illustrate the benefits of our algorithm on a range of examples and contrast it to variational, SVA and sampling approaches from both a computational complexity perspective as well as in terms of clustering performance. We demonstrate the wide applicabiity of our approach by presenting an approximate MAP inference method for the infinite hidden Markov model whose performance contrasts favorably with a recently proposed hybrid SVA approach. Similarly, we show how our algorithm can applied to a semiparametric mixed-effects regression model where the random effects distribution is modelled using an infinite mixture model, as used in longitudinal progression modelling in population health science. Finally, we propose directions for future research on approximate MAP inference in Bayesian nonparametrics.

Bayesian semiparametric modeling of realized covariance matrices

This paper introduces several new Bayesian nonparametric models suitable for capturing the unknown conditional distribution of realized covariance (RCOV) matrices. Existing dynamic Wishart models are extended to countably infinite mixture models of Wishart and inverse-Wishart distributions. In addition to mixture models with constant weights we propose models with time-varying weights to capture time dependence in the unknown distribution. Each of our models can be combined with returns to provide a coherent joint model of returns and RCOV. The extensive forecast results show the new models provide very significant improvements in density forecasts for RCOV and returns and competitive point forecasts of RCOV.

CLUSTERnGO: a user-defined modelling platform for two-stage clustering of time-series data.

Motivation: Simple bioinformatic tools are frequently used to analyse time-series datasets regardless of their ability to deal with transient phenomena, limiting the meaningful information that may be extracted from them. This situation requires the development and exploitation of tailor-made, easy-to-use and flexible tools designed specifically for the analysis of time-series datasets.Results: We present a novel statistical application called CLUSTERnGO, which uses a model-based clustering algorithm that fulfils this need. This algorithm involves two components of operation. Component 1 constructs a Bayesian non-parametric model (Infinite Mixture of Piecewise Linear Sequences) and Component 2, which applies a novel clustering methodology (Two-Stage Clustering). The software can also assign biological meaning to the identified clusters using an appropriate ontology. It applies multiple hypothesis testing to report the significance of these enrichments. The algorithm has a four-phase pipeline. The application can be executed using either command-line tools or a user-friendly Graphical User Interface. The latter has been developed to address the needs of both specialist and non-specialist users. We use three diverse test cases to demonstrate the flexibility of the proposed strategy. In all cases, CLUSTERnGO not only outperformed existing algorithms in assigning unique GO term enrichments to the identified clusters, but also revealed novel insights regarding the biological systems examined, which were not uncovered in the original publications.Availability and implementation: The C++ and QT source codes, the GUI applications for Windows, OS X and Linux operating systems and user manual are freely available for download under the GNU GPL v3 license at http://www.cmpe.boun.edu.tr/content/CnG.Contact: sgo24@cam.ac.ukSupplementary information: Supplementary data are available at Bioinformatics online.

Latent dirichlet allocation

We describe latent Dirichlet allocation (LDA), a generative probabilistic model for collections of discrete data such as text corpora. LDA is a three-level hierarchical Bayesian model, in which each item of a collection is modeled as a finite mixture over an underlying set of topics. Each topic is, in turn, modeled as an infinite mixture over an underlying set of topic probabilities. In the context of text modeling, the topic probabilities provide an explicit representation of a document. We present efficient approximate inference techniques based on variational methods and an EM algorithm for empirical Bayes parameter estimation. We report results in document modeling, text classification, and collaborative filtering, comparing to a mixture of unigrams model and the probabilistic LSI model.

Calculating infinite-medium α-eigenvalue spectra with Monte Carlo using a transition rate matrix method

The time-dependent behavior of the energy spectrum in neutron transport was investigated with a formulation, based on continuous-time Markov processes, for computing α eigenvalues and eigenvectors in an infinite medium. For this, a research Monte Carlo code called “TORTE” (To Obtain Real Time Eigenvalues) was created and used to estimate elements of a transition rate matrix. TORTE is capable of using both multigroup and continuous-energy nuclear data, and verification was performed. Eigenvalue spectra for infinite homogeneous mixtures were obtained, and an eigenfunction expansion was used to investigate transient behavior of the neutron energy spectrum.

Expectation propagation learning of a Dirichlet process mixture of Beta-Liouville distributions for proportional data clustering

We propose a nonparametric Bayesian model for the clustering of proportional data. Our model is based on an infinite mixture of Beta-Liouville distributions and allows a compact description of complex data. The choice of the Beta-Liouville as the basis of our model is justified by the fact that it has been shown to be a good alternative to the Dirichlet and generalized Dirichlet distributions for the statistical representation of proportional data. Using this infinite mixture, we show how a careful modeling can achieve good results by allowing the elicitation of prior belief about the parameters and the number of clusters through suitable learning. Indeed, we develop an efficient learning algorithm, based on expectation propagation, to estimate the parameters of our infinite Beta-Liouville mixture model. The feasibility and effectiveness of the proposed method are demonstrated by two challenging applications namely action and facial expression recognition.

A variation of the Canadisation algorithm for the pricing of American options driven by Lévy processes

We introduce an algorithm for the pricing of finite expiry American options driven by Lévy processes. The idea is to tweak Carr’s ‘Canadisation’ method, cf. Carr (1998) (see also Bouchard et al. (2005)), in such a way that the adjusted algorithm is viable for any Lévy process whose law at an independent, exponentially distributed time consists of a (possibly infinite) mixture of exponentials. This includes Brownian motion plus (hyper)exponential jumps, but also the recently introduced rich class of so-called meromorphic Lévy processes, cf. Kyprianou et al. (2012). This class contains all Lévy processes whose Lévy measure is an infinite mixture of exponentials which can generate both finite and infinite jump activity. Lévy processes well known in mathematical finance can in a straightforward way be obtained as a limit of meromorphic Lévy processes. We work out the algorithm in detail for the classic example of the American put, and we illustrate the results with some numerics.

Risk, Return, and Volatility Feedback: A Bayesian Nonparametric Analysis

The relationship between risk and return is one of the most studied topics in finance. The majority of the literature is based on a linear, parametric relationship between expected returns and conditional volatility. This paper models the contemporaneous relationship between market excess returns and contemporaneous log-realized variances nonparametrically with an infinite mixture representation of their joint distribution. The conditional distribution of excess returns given log-realized variance will also have an infinite mixture representation but with probabilities and arguments depending on the value of realized variance. Our nonparametric approach allows for deviation from Gaussianity by allowing for higher-order nonzero moments and a smooth nonlinear relationship between the conditional mean of excess returns and contemporaneous log-realized variance. We find strong robust evidence of volatility feedback in monthly data. Once volatility feedback is accounted for, there is an unambiguous positive relationship between expected excess returns and expected log-realized variance. This relationship is nonlinear. Volatility feedback impacts the whole distribution and not just the conditional mean.

Objective Bayesian Survival Analysis Using Shape Mixtures of Log-Normal Distributions

Survival models such as the Weibull or log-normal lead to inference that is not robust to the presence of outliers. They also assume that all heterogeneity between individuals can be modeled through covariates. This article considers the use of infinite mixtures of lifetime distributions as a solution for these two issues. This can be interpreted as the introduction of a random effect in the survival distribution. We introduce the family of shape mixtures of log-normal distributions, which covers a wide range of density and hazard functions. Bayesian inference under nonsubjective priors based on the Jeffreys’ rule is examined and conditions for posterior propriety are established. The existence of the posterior distribution on the basis of a sample of point observations is not always guaranteed and a solution through set observations is implemented. In addition, we propose a method for outlier detection based on the mixture structure. A simulation study illustrates the performance of our methods under different scenarios and an application to a real dataset is provided. Supplementary materials for the article, which include R code, are available online.

Empirical Bayes Conditional Density Estimation

The problem of nonparametric estimation of the conditional density of a response, given a vector of explanatory variables, is classical and of prominent importance in many prediction problems since the conditional density provides a more comprehensive description of the association between the response and the predictor than, for instance, does the regression function. The problem has applications across different fields like economy, actuarial sciences and medicine. We investigate empirical Bayes estimation of conditional densities establishing that an automatic data-driven selection of the prior hyper-parameters in infinite mixtures of Gaussian kernels, with predictor-dependent mixing weights, can lead to estimators whose performance is on par with that of frequentist estimators in being minimax-optimal (up to logarithmic factors) rate adaptive over classes of locally Holder smooth conditional densities and in performing an adaptive dimension reduction if the response is independent of (some of) the explanatory variables which, containing no information about the response, are irrelevant to the purpose of estimating its conditional density.

A flexible mixed-effect negative binomial regression model for detecting unusual increases in MRI lesion counts in individual multiple sclerosis patients.

We develop a new modeling approach to enhance a recently proposed method to detect increases of contrast-enhancing lesions (CELs) on repeated magnetic resonance imaging, which have been used as an indicator for potential adverse events in multiple sclerosis clinical trials. The method signals patients with unusual increases in CEL activity by estimating the probability of observing CEL counts as large as those observed on a patient's recent scans conditional on the patient's CEL counts on previous scans. This conditional probability index (CPI), computed based on a mixed-effect negative binomial regression model, can vary substantially depending on the choice of distribution for the patient-specific random effects. Therefore, we relax this parametric assumption to model the random effects with an infinite mixture of beta distributions, using the Dirichlet process, which effectively allows any form of distribution. To our knowledge, no previous literature considers a mixed-effect regression for longitudinal count variables where the random effect is modeled with a Dirichlet process mixture. As our inference is in the Bayesian framework, we adopt a meta-analytic approach to develop an informative prior based on previous clinical trials. This is particularly helpful at the early stages of trials when less data are available. Our enhanced method is illustrated with CEL data from 10 previous multiple sclerosis clinical trials. Our simulation study shows that our procedure estimates the CPI more accurately than parametric alternatives when the patient-specific random effect distribution is misspecified and that an informative prior improves the accuracy of the CPI estimates.

Bayesian Estimation and Clustering of Latent Attitudinal Parameters Using Cross-Sectional Survey Data: Application to an Online Banking Survey

Firms routinely collect cross-sectional survey data on attitudinal constructs such as satisfaction, brand awareness, behavioral intentions etc. from current and prospective customers. The single observation per individual in such survey data offers limited scope for estimation and inference of heterogeneous response parameters, and consequently, for downstream analyses that presuppose a heterogeneous response. We present a Bayesian treatment to this problem. We estimate these individual-specific latent attitudinal parameters, segment the respondents on the basis of these parameters, and finally, empirically validate our clustering solutions. Our approach considers both finite and infinite mixtures of component densities and enables exact finite sample inference on both individual level parameters and cluster level parameters. We examine the performance of the latent class, the mixture of normals and the semiparametric Bayesian estimators under varying conditions of latent data structure, cluster separation and individual-level idiosyncratic variance.

Bayesian Nonparametric Calibration and Combination of Predictive Distributions

We introduce a Bayesian approach to predictive density calibration and combination that accounts for parameter uncertainty and model set incompleteness through the use of random calibration functionals and random combination weights. Building on the work of Ranjan, R. and Gneiting, T. (2010) and Gneiting, T. and Ranjan, R. (2013), we use infinite beta mixtures for the calibration. The proposed Bayesian nonparametric approach takes advantage of the flexibility of Dirichlet process mixtures to achieve any continuous deformation of linearly combined predictive distributions. The inference procedure is based on Gibbs sampling and allows accounting for uncertainty in the number of mixture components, mixture weights, and calibration parameters. The weak posterior consistency of the Bayesian nonparametric calibration is provided under suitable conditions for unknown true density. We study the methodology in simulation examples with fat tails and multimodal densities and apply it to density forecasts of daily S&P returns and daily maximum wind speed at the Frankfurt airport.

Uncertainty propagation using infinite mixture of Gaussian processes and variational Bayesian inference

Uncertainty propagation in flow through porous media problems is a challenging problem. This is due to the high-dimensionality of the random property fields, e.g. permeability and porosity, as well as the computational complexity of the models that are involved. The usual approach is to construct a surrogate response surface and then use it instead of the expensive model to carry out the uncertainty propagation task. However, the construction of the surrogate surface is hampered by various aspects such as the limited number of model evaluations that one can afford, the curse of dimensionality, multi-variate responses with non-trivial correlations, potential localized features of the response and/or discontinuities. In this work, we extend upon the concept of the Multi-output Gaussian Process (MGP) to effectively deal with all of these difficulties simultaneously. This non-trivial extension involves an infinite mixture of MGP's that is trained using variational Bayesian inference. Prior to observing any data, a Dirichlet process is used to generate the components of the MGP mixture. The Bayesian nature of the model allows for the quantification of the uncertainties due to the limited number of simulations, i.e., we can derive error bars for the statistics of interest. The automatic detection of the mixture components by the variational inference algorithm is able to capture discontinuities and localized features without adhering to ad hoc constructions. Finally, correlations between the components of multi-variate responses are captured by the underlying MGP model in a natural way.

An adaptive truncation method for inference in Bayesian nonparametric models

Many exact Markov chain Monte Carlo algorithms have been developed for posterior inference in Bayesian nonparametric models which involve infinite-dimensional priors. However, these methods are not generic and special methodology must be developed for different classes of prior or different models. Alternatively, the infinite-dimensional prior can be truncated and standard Markov chain Monte Carlo methods used for inference. However, the error in approximating the infinite-dimensional posterior can be hard to control for many models. This paper describes an adaptive truncation method which allows the level of the truncation to be decided by the algorithm and so can avoid large errors in approximating the posterior. A sequence of truncated priors is constructed which are sampled using Markov chain Monte Carlo methods embedded in a sequential Monte Carlo algorithm. Implementational details for infinite mixture models with stick-breaking priors and normalized random measures with independent increments priors are discussed. The methodology is illustrated on infinite mixture models, a semiparametric linear mixed model and a nonparametric time series model.

An infinite Gaussian mixture model with its application in hyperspectral unmixing

Spectral unmixing is a critical issue in multi-spectral data processing, which has the ability to identify the constituent components of a pixel. Most of the hyperspectral unmixing current methods are based on Linear Mixture Model (LMM) and have been widely used in many scenarios. However, both the noise contained in the LMM and the requirement of essential prior knowledge strongly limit their practical applications. In order to address these issues, this paper proposes an iterative approach named CBIGMM. It utilizes infinite Gaussian mixture model to describe the hyperspectral data, which is robust to the noise due to the intrinsic randomness of the Gaussian components; and extracts the endmembers and their corresponding abundance in a fully unsupervised way without prior knowledge. A set of experiment is conducted on both synthetic and real data set from pesticide-contaminated vegetables. The results and analyses show CBIGMM outperforms other methods in addressing hyperspectral unmixing problem.

Second-Order Slepian-Wolf Coding Theorems for Non-Mixed and Mixed Sources

The second-order achievable rate region in Slepian-Wolf source coding systems is investigated. The concept of second-order achievable rates, which enables us to make a finer evaluation of achievable rates, has already been introduced and analyzed for general sources in the single-user source coding problem. Analogously, in this paper, we first define the second-order achievable rate region for the Slepian-Wolf coding system to establish the source coding theorem in the second-order sense. The Slepian-Wolf coding problem for correlated sources is one of typical problems in the multiterminal information theory. In particular, Miyake and Kanaya, and Han have established the first-order source coding theorems for general correlated sources. On the other hand, in general, the second-order achievable rate problem for the Slepian-Wolf coding system with general sources remains still open up to present. In this paper, we present the analysis concerning the second-order achievable rates for general sources, which are based on the information spectrum methods developed by Han and Verdu. Moreover, we establish the explicit second-order achievable rate region for independently and identically distributed (i.i.d.) correlated sources with countably infinite alphabets and mixtures of i.i.d. correlated sources, respectively, using the relevant asymptotic normality.