Mixed Membership Models Research Articles

Joint mixed membership modeling of multivariate longitudinal and survival data for learning the individualized disease progression

Joint mixed membership modeling of multivariate longitudinal and survival data for learning the individualized disease progression

Optimal Network Pairwise Comparison

We are interested in the problem of two-sample network hypothesis testing: given two networks with the same set of nodes, we wish to test whether the underlying Bernoulli probability matrices of the two networks are the same or not. We propose Interlacing Balance Measure (IBM) as a new two-sample testing approach. We consider the Degree-Corrected Mixed-Membership (DCMM) model for undirected networks, where we allow severe degree heterogeneity, mixed-memberships, flexible sparsity levels, and weak signals. In such a broad setting, how to find a test that has a tractable limiting null and optimal testing performances is a challenging problem. We show that IBM is such a test: in a broad DCMM setting with only mild regularity conditions, IBM has N ( 0 , 1 ) as the limiting null and achieves the optimal phase transition. While the above is for undirected networks, IBM is a unified approach and is directly implementable for directed networks. For a broad directed-DCMM (extension of DCMM for directed networks) setting, we show that IBM has N ( 0 , 1 / 2 ) as the limiting null and continues to achieve the optimal phase transition. We have also applied IBM to the Enron email network and a gene co-expression network, with interesting results.

Flexible regularized estimation in high-dimensional mixed membership models

Mixed membership models are an extension of finite mixture models, where each observation can partially belong to more than one mixture component. A probabilistic framework for mixed membership models of high-dimensional continuous data is proposed with a focus on scalability and interpretability. The novel probabilistic representation of mixed membership is based on convex combinations of dependent multivariate Gaussian random vectors. In this setting, scalability is ensured through approximations of a tensor covariance structure through multivariate eigen-approximations with adaptive regularization imposed through shrinkage priors. Conditional weak posterior consistency is established on an unconstrained model, allowing for a simple posterior sampling scheme while keeping many of the desired theoretical properties of our model. The model is motivated by two biomedical case studies: a case study on functional brain imaging of children with autism spectrum disorder (ASD) and a case study on gene expression data from breast cancer tissue. These applications highlight how the typical assumption made in cluster analysis, that each observation comes from one homogeneous subgroup, may often be restrictive in several applications, leading to unnatural interpretations of data features.

Sub-communities of the vaginal microbiota in pregnant and non-pregnant women.

Diverse and non-Lactobacillus-dominated vaginal microbial communities are associated with adverse health outcomes such as preterm birth and the acquisition of sexually transmitted infections. Despite the importance of recognizing and understanding the key risk-associated features of these communities, their heterogeneous structure and properties remain ill-defined. Clustering approaches are commonly used to characterize vaginal communities, but they lack sensitivity and robustness in resolving substructures and revealing transitions between potential sub-communities. Here, we address this need with an approach based on mixed membership topic models. Using longitudinal data from cohorts of pregnant and non-pregnant study participants, we show that topic models more accurately describe sample composition, longitudinal changes, and better predict the loss of Lactobacillus dominance. We identify several non-Lactobacillus-dominated sub-communities common to both cohorts and independent of reproductive status. In non-pregnant individuals, we find that the menstrual cycle modulates transitions between and within sub-communities, as well as the concentrations of half of the cytokines and 18% of metabolites. Overall, our analyses based on mixed membership models reveal substructures of vaginal ecosystems which may have important clinical and biological associations.

Estimating Mixed Memberships in Directed Networks by Spectral Clustering

Community detection is an important and powerful way to understand the latent structure of complex networks in social network analysis. This paper considers the problem of estimating community memberships of nodes in a directed network, where a node may belong to multiple communities. For such a directed network, existing models either assume that each node belongs solely to one community or ignore variation in node degree. Here, a directed degree corrected mixed membership (DiDCMM) model is proposed by considering degree heterogeneity. An efficient spectral clustering algorithm with a theoretical guarantee of consistent estimation is designed to fit DiDCMM. We apply our algorithm to a small scale of computer-generated directed networks and several real-world directed networks.

Mixed membership estimation for social networks

In economics and social science, network data are regularly observed, and a thorough understanding of the network community structure facilitates the comprehension of economic patterns and activities. Consider an undirected network with n nodes and K communities. We model the network using the Degree-Corrected Mixed-Membership (DCMM) model, where for each node i=1,2,…,n, there exists a membership vector πi=(πi(1),πi(2),…,πi(K))′, where πi(k) is the weight that node i puts on community k, 1≤k≤K. In comparison to the well-known stochastic block model (SBM), the DCMM permits both severe degree heterogeneity and mixed memberships, making it more realistic and general. We present an efficient approach, Mixed-SCORE, for estimating the mixed membership vectors of all nodes and the other DCMM parameters. This approach is inspired by the discovery of a delicate simplex structure in the spectral domain. We derive explicit error rates for the Mixed-SCORE algorithm and demonstrate that it is rate-optimal over a broad parameter space. Our findings provide a novel statistical tool for network community analysis, which can be used to understand network formations, extract nodal features, identify unobserved covariates in dyadic regressions, and estimate peer effects. We applied Mixed-SCORE to a political blog network, two trade networks, a co-authorship network, and a citee network, and obtained interpretable results.

Discovering Online Shopping Preference Structures in Large and Frequently Changing Store Assortments

The authors develop an attribute-based mixed-membership model of consumers’ preference for stockkeeping units in store assortments. The model represents the underlying “topics of interest” that drive shopping behaviors as probability distributions over product attributes. It overcomes several limitations of latent Dirichlet allocation topic models and is particularly useful for making preference predictions in large and frequently changing store assortments. The authors apply the proposed model to investigate topics driving browsing and purchase activities in an online deal marketplace of fashion products and explore how preference structures evolve over time. They find commonalities and differences in the topics that drive the browsing and purchase stages of online shopping processes. In general, browsing covers a broader range of product attributes than purchases. Consumers tend to browse products of premium positioning and/or deep discounts in the deal marketplace, but when purchasing, they tend to gravitate toward lower-tiered products at their original prices and modest depths of discounts. The authors illustrate how insights from the proposed model can be utilized to profile consumers based on their price preferences and to improve personalized product recommendations. They show that the model's performance is particularly strong in predicting preferences for new products that are not in the existing assortment.

A Mixed-Membership Model for Social Network Clustering

We propose a simple mixed membership model for social network clustering in this paper. A flexible function is adopted to measure affinities among a set of entities in a social network. The model not only allows each entity in the network to possess more than one membership, but also provides accurate statistical inference about network structure. We estimate the membership parameters using an MCMC algorithm. We evaluate the performance of the proposed algorithm by applying our model to two empirical social network data, the Zachary club data and the bottlenose dolphin network data. We also conduct some numerical studies based on synthetic networks for further assessing the effectiveness of our algorithm. In the end, some concluding remarks and future work are addressed briefly.

Mixed membership Gaussians

In recent years, there has been significant progress in unsupervised representation learning. Classical statistical and machine learning techniques can be used on these representations for downstream tasks and applications. Here, we consider an unsupervised downstream task, namely, topic modeling with mixed membership models. Prototypical mixed membership models like the latent Dirichlet allocation (LDA) topic model use only simple discrete observed features. However, state-of-the-art representations of images, text, and other modalities are given by continuous feature vectors. Therefore, we study mixed membership Gaussians that operate on continuous feature vectors. We prove that the parameters of this model can be learned efficiently and effectively by Pearson’s method of moments and corroborate our theoretical findings on synthetic data. Experiments on standard labeled image data sets show that mixed membership Gaussians on state-of-the-art image representations yield topics that are semantically meaningful, coherent, and cover the labels that have not been used during training well.

Estimating mixed-memberships using the symmetric laplacian inverse matrix

Mixed membership community detection is a challenging problem. In this paper, to detect mixed memberships, we propose a new method Mixed-SLIM which is a spectral clustering method on the symmetrized Laplacian inverse matrix under the degree-corrected mixed membership model. We provide theoretical bounds for the estimation error on the proposed algorithm and its regularized version under mild conditions. Meanwhile, we provide some extensions of the proposed method to deal with large networks in practice. These Mixed-SLIM methods outperform state-of-art methods in simulations and substantial empirical datasets for both community detection and mixed membership community detection problems.

Microbiome subcommunity learning with logistic-tree normal latent Dirichlet allocation.

Mixed-membership (MM) models such as latent Dirichlet allocation (LDA) have been applied to microbiome compositional data to identify latent subcommunities of microbial species. These subcommunities are informative for understanding the biological interplay of microbes and for predicting health outcomes. However, microbiome compositions typically display substantial cross-sample heterogeneities in subcommunity compositions-that is, the variability in the proportions of microbes in shared subcommunities across samples-which is not accounted for in prior analyses. As a result, LDA can produce inference, which is highly sensitive to the specification of the number of subcommunities and often divides a single subcommunity into multiple artificial ones. To address this limitation, we incorporate the logistic-tree normal (LTN) model into LDA to form a new MM model. This model allows cross-sample variation in the composition of each subcommunity around some "centroid" composition that defines the subcommunity. Incorporation of auxiliary Pólya-Gamma variables enables a computationally efficient collapsed blocked Gibbs sampler to carry out Bayesian inference under this model. By accounting for such heterogeneity, our new model restores the robustness of the inference in the specification of the number of subcommunities and allows meaningful subcommunities to beidentified.

A Useful Criterion on Studying Consistent Estimation in Community Detection.

In network analysis, developing a unified theoretical framework that can compare methods under different models is an interesting problem. This paper proposes a partial solution to this problem. We summarize the idea of using a separation condition for a standard network and sharp threshold of the Erdös–Rényi random graph to study consistent estimation, and compare theoretical error rates and requirements on the network sparsity of spectral methods under models that can degenerate to a stochastic block model as a four-step criterion SCSTC. Using SCSTC, we find some inconsistent phenomena on separation condition and sharp threshold in community detection. In particular, we find that the original theoretical results of the SPACL algorithm introduced to estimate network memberships under the mixed membership stochastic blockmodel are sub-optimal. To find the formation mechanism of inconsistencies, we re-establish the theoretical convergence rate of this algorithm by applying recent techniques on row-wise eigenvector deviation. The results are further extended to the degree-corrected mixed membership model. By comparison, our results enjoy smaller error rates, lesser dependence on the number of communities, weaker requirements on network sparsity, and so forth. The separation condition and sharp threshold obtained from our theoretical results match the classical results, so the usefulness of this criterion on studying consistent estimation is guaranteed. Numerical results for computer-generated networks support our finding that spectral methods considered in this paper achieve the threshold of separation condition.

Psychosocial-Behavioral Phenotyping: A Novel Precision Health Approach to Modeling Behavioral, Psychological, and Social Determinants of Health Using Machine Learning.

The context in which a behavioral intervention is delivered is an important source of variability and systematic approaches are needed to identify and quantify contextual factors that may influence intervention efficacy. Machine learning-based phenotyping methods can contribute to a new precision health paradigm by informing personalized behavior interventions. Two primary goals of precision health, identifying population subgroups and highlighting behavioral intervention targets, can be addressed with psychosocial-behavioral phenotypes. We propose a method for psychosocial-behavioral phenotyping that models social determinants of health in addition to individual-level psychological and behavioral factors. To demonstrate a novel application of machine learning for psychosocial-behavioral phenotyping, the identification of subgroups with similar combinations of psychosocial characteristics. In this secondary analysis of psychosocial and behavioral data from a community cohort (n = 5,883), we optimized a multichannel mixed membership model (MC3M) using Bayesian inference to identify psychosocial-behavioral phenotypes and used logistic regression to determine which phenotypes were associated with elevated weight status (BMI ≥ 25kg/m2). We identified 20 psychosocial-behavioral phenotypes. Phenotypes were conceptually consistent as well as discriminative; most participants had only one active phenotype. Two phenotypes were significantly positively associated with elevated weight status; four phenotypes were significantly negatively associated. Each phenotype suggested different contextual considerations for intervention design. By depicting the complexity of psychological and social determinants of health while also providing actionable insight about similarities and differences among members of the same community, psychosocial-behavioral phenotypes can identify potential intervention targets in context.

Simple: Statistical Inference on Membership Profiles in Large Networks

AbstractNetwork data are prevalent in many contemporary big data applications in which a common interest is to unveil important latent links between different pairs of nodes. Yet a simple fundamental question of how to precisely quantify the statistical uncertainty associated with the identification of latent links still remains largely unexplored. In this paper, we propose the method of statistical inference on membership profiles in large networks (SIMPLE) in the setting of degree-corrected mixed membership model, where the null hypothesis assumes that the pair of nodes share the same profile of community memberships. In the simpler case of no degree heterogeneity, the model reduces to the mixed membership model for which an alternative more robust test is also proposed. Both tests are of the Hotelling-type statistics based on the rows of empirical eigenvectors or their ratios, whose asymptotic covariance matrices are very challenging to derive and estimate. Nevertheless, their analytical expressions are unveiled and the unknown covariance matrices are consistently estimated. Under some mild regularity conditions, we establish the exact limiting distributions of the two forms of SIMPLE test statistics under the null hypothesis and contiguous alternative hypothesis. They are the chi-square distributions and the noncentral chi-square distributions, respectively, with degrees of freedom depending on whether the degrees are corrected or not. We also address the important issue of estimating the unknown number of communities and establish the asymptotic properties of the associated test statistics. The advantages and practical utility of our new procedures in terms of both size and power are demonstrated through several simulation examples and real network applications.

MULTIVARIATE MIXED MEMBERSHIP MODELING: INFERRING DOMAIN-SPECIFIC RISK PROFILES.

Characterizing the shared memberships of individuals in a classification scheme poses severe interpretability issues, even when using a moderate number of classes (say 4). Mixed membership models quantify this phenomenon, but they typically focus on goodness-of-fit more than on interpretable inference. To achieve a good numerical fit, these models may in fact require many extreme profiles, making the results difficult to interpret. We introduce a new class of multivariate mixed membership models that, when variables can be partitioned into subject-matter based domains, can provide a good fit to the data using fewer profiles than standard formulations. The proposed model explicitly accounts for the blocks of variables corresponding to the distinct domains along with a cross-domain correlation structure, which provides new information about shared membership of individuals in a complex classification scheme. We specify a multivariate logistic normal distribution for the membership vectors, which allows easy introduction of auxiliary information leveraging a latent multivariate logistic regression. A Bayesian approach to inference, relying on Pólya gamma data augmentation, facilitates efficient posterior computation via Markov Chain Monte Carlo. We apply this methodology to a spatially explicit study of malaria risk over time on the Brazilian Amazon frontier.

Mixed Membership Models for Exploring User Roles in Online Fora

Discussion boards are a form of social media which allow users to discuss topics and exchange information in a complex manner, in a number of different settings. As the popularity of such message boards has increased, communities of users have emerged, and several prominent types of social role have been identified, such as Question Answerer, Celebrity, Discussion Person and Topic Initiator. Recent studies have noted the structural similarity of the egocentric network of users assigned the same role by qualitative criteria. In this paper a methodology is developed with which to cluster together users with similar ego-centric network structures. This is achieved using a mixed membership formulation which allows for the fact that different groups of users may have characteristics in common. The method is then applied to data taken from boards.ie, a medium sized message boards website. Prominent clusters of users are identified and discussed, and illustrative examples of user behaviour provided. The type of interaction, both locally and globally, taking place within forums is examined.

Learning Latent Jet Structure

We summarize our recent work on how to infer on jet formation processes directly from substructure data using generative statistical models. We recount in detail how to cast jet substructure observables’ measurements in terms of Bayesian mixed membership models, in particular Latent Dirichlet Allocation. Using a mixed sample of QCD and boosted tt¯ jet events and focusing on the primary Lund plane observable basis for event measurements, we show how using educated priors on the latent distributions allows to infer on the underlying physical processes in a semi-supervised way.

Spatio-Temporal Mixed Membership Models for Criminal Activity

Abstract We suggest a probabilistic approach to study crime data in London and highlight the benefits of defining a statistical joint crime distribution model which provides insights into urban criminal activity. This is achieved by developing a hierarchical mixture model for observations, crime occurrences over a geographical study area, that are grouped according to multiple time stamps and crime categories. The mixture components correspond to spatial crime distributions over the study area and the goal is to infer, based on the observations, how and to what degree the latent distributions are shared across the groups.

Learning endometriosis phenotypes from patient-generated data

Endometriosis is a systemic and chronic condition in women of childbearing age, yet a highly enigmatic disease with unresolved questions: there are no known biomarkers, nor established clinical stages. We here investigate the use of patient-generated health data and data-driven phenotyping to characterize endometriosis patient subtypes, based on their reported signs and symptoms. We aim at unsupervised learning of endometriosis phenotypes using self-tracking data from personal smartphones. We leverage data from an observational research study of over 4000 women with endometriosis that track their condition over more than 2 years. We extend a classical mixed-membership model to accommodate the idiosyncrasies of the data at hand, i.e., the multimodality and uncertainty of the self-tracked variables. The proposed method, by jointly modeling a wide range of observations (i.e., participant symptoms, quality of life, treatments), identifies clinically relevant endometriosis subtypes. Experiments show that our method is robust to different hyperparameter choices and the biases of self-tracking data (e.g., the wide variations in tracking frequency among participants). With this work, we show the promise of unsupervised learning of endometriosis subtypes from self-tracked data, as learned phenotypes align well with what is already known about the disease, but also suggest new clinically actionable findings. More generally, we argue that a continued research effort on unsupervised phenotyping methods with patient-generated health data via new mobile and digital technologies will have significant impact on the study of enigmatic diseases in particular, and health in general.

A probabilistic assessment of the Indo-Aryan Inner–Outer Hypothesis

Abstract This paper uses a novel data-driven probabilistic approach to address the century-old Inner-Outer hypothesis of Indo-Aryan. I develop a Bayesian hierarchical mixed-membership model to assess the validity of this hypothesis using a large data set of automatically extracted sound changes operating between Old Indo-Aryan and Modern Indo-Aryan speech varieties. I employ different prior distributions in order to model sound change, one of which, the Logistic Normal distribution, has not received much attention in linguistics outside of Natural Language Processing, despite its many attractive features. I find evidence for cohesive dialect groups that have made their imprint on contemporary Indo-Aryan languages, and find that when a Logistic Normal prior is used, the distribution of dialect components across languages is largely compatible with a core-periphery pattern similar to that proposed under the Inner-Outer hypothesis.