Collapsed Gibbs Sampling Research Articles

Decoding Spatial Tissue Architecture: A Scalable Bayesian Topic Model for Multiplexed Imaging Analysis.

Recent progress in multiplexed tissue imaging is advancing the study of tumor microenvironments to enhance our understanding of treatment response and disease progression. Cellular neighborhood analysis is a popular computational approach for these complex image data. Despite its popularity, there are significant challenges, including high computational demands that limit feasibility for largescale applications and the lack of a principled strategy for integrative analysis across images. This absence hampers the precise and consistent identification of spatial features and tracking of their dynamics over disease progression. To overcome these challenges, we introduce SpatialTopic , a spatial topic model designed to decode high-level spatial architecture across multiplexed tissue images. SpatialTopic integrates both cell type and spatial information within a topic modelling framework, originally developed for natural language processing and adapted for computer vision. Spatial information is incorporated into the flexible design of documents, representing densely overlapping regions in images. We employ an efficient collapsed Gibbs sampling algorithm for model inference. We benchmarked the performance against five state-of-the-art algorithms through various case studies using different single-cell spatial transcriptomic and proteomic imaging platforms across different tissue types. We show that SpatialTopic is highly scalable on large-scale image datasets with millions of cells, along with high precision and interpretability. Our findings demonstrate that SpatialTopic consistently identifies biologically and clinically significant spatial "topics" such as tertiary lymphoid structures (TLSs) and tracks dynamic changes in spatial features over disease progression. Its computational efficiency and broad applicability across various molecular imaging platforms will enhance the analysis of large-scale tissue imaging datasets.

Bayesian L 1/2 Regression

It is well known that Bridge regression enjoys superior theoretical properties when compared to traditional LASSO. However, the current latent variable representation of its Bayesian counterpart, based on the exponential power prior, is computationally expensive in higher dimensions. In this article, we show that the exponential power prior has a closed form scale mixture of normal decomposition for α = ( 1 2 ) γ , γ ∈ { 1 , 2 , … } . We call these types of priors L 1 2 prior for short. We develop an efficient partially collapsed Gibbs sampling scheme for computation using the L 1 2 prior and study theoretical properties when p > n . In addition, we introduce a non-separable Bridge penalty function inspired by the fully Bayesian formulation and a novel, efficient coordinate descent algorithm. We prove the algorithm’s convergence and show that the local minimizer from our optimization algorithm has an oracle property. Finally, simulation studies were carried out to illustrate the performance of the new algorithms. Supplementary materials for this article are available online.

Augmentation Samplers for Multinomial Probit Bayesian Additive Regression Trees

The multinomial probit (MNP) (Imai and van Dyk, 2005) framework is based on a multivariate Gaussian latent structure, allowing for natural extensions to multilevel modeling. Unlike multinomial logistic models, MNP does not assume independent alternatives. Kindo et al. (2016) proposed multinomial probit BART (MPBART) to accommodate Bayesian additive regression trees (BART) formulation in MNP. The posterior sampling algorithms for MNP and MPBART are collapsed Gibbs samplers. Because the collapsing augmentation strategy yields a geometric rate of convergence no greater than that of a standard Gibbs sampling step, it is recommended whenever computationally feasible (Liu, 1994a; Imai and van Dyk, 2005). While this strategy necessitates simple sampling steps and a reasonably fast converging Markov chain, the complexity of the stochastic search for posterior trees may undermine its benefit. We address this problem by sampling posterior trees conditional on the constrained parameter space and compare our proposals to that of Kindo et al. (2016), who sample posterior trees based on an augmented parameter space. We also compare to the approach by Sparapani et al. (2021) that specified the multinomial model in terms of conditional probabilities. In terms of MCMC convergence and posterior predictive accuracy, our proposals are comparable to the conditional probability approach and outperform the augmented tree sampling approach. We also show that the theoretical mixing rates of our proposals are guaranteed to be no greater than the augmented tree sampling approach. Appendices and codes for simulations and demonstrations are available online.

Hidden Variable Models in Text Classification and Sentiment Analysis

In this paper, we are proposing extensions to the multinomial principal component analysis (MPCA) framework, which is a Dirichlet (Dir)-based model widely used in text document analysis. The MPCA is a discrete analogue to the standard PCA (it operates on continuous data using Gaussian distributions). With the extensive use of count data in modeling nowadays, the current limitations of the Dir prior (independent assumption within its components and very restricted covariance structure) tend to prevent efficient processing. As a result, we are proposing some alternatives with flexible priors such as generalized Dirichlet (GD) and Beta-Liouville (BL), leading to GDMPCA and BLMPCA models, respectively. Besides using these priors as they generalize the Dir, importantly, we also implement a deterministic method that uses variational Bayesian inference for the fast convergence of the proposed algorithms. Additionally, we use collapsed Gibbs sampling to estimate the model parameters, providing a computationally efficient method for inference. These two variational models offer higher flexibility while assigning each observation to a distinct cluster. We create several multitopic models and evaluate their strengths and weaknesses using real-world applications such as text classification and sentiment analysis.

Bayesian collapsed Gibbs sampling for a stochastic volatility model with a Dirichlet process mixture

SummaryThis paper replicates the results of the stochastic volatility–Dirichlet process mixture (SV‐DPM) models proposed in Jensen and Maheu (2010) in both a narrow and a wide sense. By using a normal‐Wishart prior and the collapsed Gibbs sampling method, our algorithm can be applied for more general settings, and it is more efficient for sampling the Dirichlet process mixture. For the stochastic volatility component, we adopt the method in Chan (2017) to further increase the overall efficiency of our algorithm. Using the same dataset, we obtain mixed results. Some of the results have significant differences. If we use recent time period dataset, which includes the COVID‐19 pandemic period, the log market portfolio volatility seems to increase in terms of the number of clusters and size of magnitude.

Application Research on Semantic Analysis Using Latent Dirichlet Allocation and Collapsed Gibbs Sampling for Topic Discovery

Topic discovery is a process of identifying the main topics present in a collection of documents. It is a crucial step in text mining, digital humanities, and information retrieval, as it allows one to extract meaningful information from large volumes of unstructured text data. The most widely used algorithm for topic discovery is Latent Dirichlet Allocation (LDA). LDA assumes that the words in each document are generated by a small number of underlying topics, and the algorithm learns the topics from the text data automatically. One of the main problems of LDA is that the topics extracted are of poor quality if the document does not coherently belong to a single topic. However, Gibbs sampling operates on a word-by-word basis, which allows it to be used on documents with a variety of topics and modifies the topic assignment of a single word. The paper presents application research on Latent Dirichlet Allocation and Collapsed Gibbs Sampling Semantic Analysis for topic discovery.

Exploring topic modelling for generalising design requirements in complex design

As the redesign process progresses in product lifecycle management, effectively managing engineering changes becomes increasingly challenging, often leading to catastrophic and costly project failures. In response, the study provides a framework for generalising design requirements documents into topics that engineers can use to understand complex designs. Based on previous work, this study employs and compares four different models, including latent Dirichlet allocation (LDA), the collapsed Gibbs sampling algorithm for the Dirichlet multinomial mixtures model (GSDMM), LDA-BERT, and GSDMM-BERT to determine the appropriate representation of requirements documents. Both heatmaps and UMAPs are used to illustrate the correlation between topics and words. The results indicate that the combined vector representation of topic modelling and the sentence-BERT model outperforms single topic modelling. This combined model leverages the additional knowledge from a pre-trained sentence-BERT model, thereby improving model performance and word distribution in all three industrial projects. Through this proposed framework, engineers can potentially generalise high-quality requirements topics for large requirements documents.

A composite Bayesian approach for quantile curve fitting with non-crossing constraints

To fit a set of quantile curves, Bayesian simultaneous quantile curve fitting methods face some challenges in properly specifying a feasible formulation and efficiently accommodating the non-crossing constraints. In this article, we propose a new minimization problem and develop its corresponding Bayesian analysis. The new minimization problem imposes two penalties to control not only the smoothness of fitted quantile curves but also the differences between quantile curves. This enables a direct inference on differences of quantile curves and facilitates improved information sharing among quantiles. After adopting B-spline approximation for the positive smoothing functions in the minimization problem, we specify the pseudo composite asymmetric Laplace likelihood and derive its priors. The computation algorithm, including partially collapsed Gibbs sampling for model parameters and Monte Carlo Expectation-Maximization algorithm for penalty parameters, are provided to carry out the proposed approach. The extensive simulation studies show that, compared with other candidate methods, the proposed approach yields more robust estimation. More advantages of the proposed approach are observed for the extreme quantiles, heavy-tailed random errors, and inference on the differences of quantiles. We also demonstrate the relative performances of the proposed approach and other competing methods through two real data analyses.

Multi-View Scholar Clustering With Dynamic Interest Tracking

Scholar clustering has garnered increasing attention due to the explosive growth of scholar data. Although researchers have proposed many algorithms to cluster scholars, they typically focus on clustering scholars from the intrinsic view (scholars' contents). These algorithms may lead to inaccurate and biased clustering results because they ignore the extrinsic view (scholar's specialty) and the changeability of scholars' interest in each view. In this paper, we propose a multi-view scholar clustering topic model (MSCT), which integrates complementary information from both intrinsic and extrinsic views while considering dynamic scholar interests. Specifically, MSCT involves two novel schemes. The first one is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multi-view integration</i> , where MSCT collaboratively tracks scholars' time-varying topic distribution from two views: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> intrinsic view</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">extrinsic view</i> . The former exploits the details of different academic degrees in the title and information in the abstract; the latter leverages the specialty of different categories in the corresponding research field and research discipline. The second one is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">dynamic interest tracking</i> , which dynamically models each scholar's interest distribution in terms of the current scholar texts and previously estimated distribution through a newly designed collapsed Gibbs sampling algorithm. Experimental results demonstrate that MSCT can significantly outperform state-of-the-art algorithms.

Online short text clustering using infinite extensions of discrete mixture models

AbstractShort text clustering is one of the fundamental tasks in natural language processing. Different from traditional documents, short texts are ambiguous and sparse due to their short form and the lack of recurrence in word usage from one text to another, making it very challenging to apply conventional machine learning algorithms directly. In this article, we propose two novel approaches for short texts clustering: collapsed Gibbs sampling infinite generalized Dirichlet multinomial mixture model infinite GSGDMM) and collapsed Gibbs sampling infinite Beta‐Liouville multinomial mixture model (infinite GSBLMM). We adopt two flexible and practical priors to the multinomial distribution where in the first one the generalized Dirichlet distribution is integrated, while the second one is based on the Beta‐Liouville distribution. We evaluate the proposed approaches on two famous benchmark datasets, namely, Google News and Tweet. The experimental results demonstrate the effectiveness of our models compared to basic approaches that use Dirichlet priors. We further propose to improve the performance of our methods with an online clustering procedure. We also evaluate the performance of our methods for the outlier detection task, in which we achieve accurate results.

Tree-based data filtering for online user-generated reviews

Analysis of online user-generated reviews has attracted extensive attention with broad applications in recent years. However, the high-volume and low-value density of online reviews bring challenges to a timely and effective data utilization. To address that challenge, this work proposes an unsupervised review filtering method based on the inherent tree-structured hierarchies among review data that reflect the general-to-specific characteristics of various quality aspects discussed in reviews. In particular, the reviews with aspects distributed near the leaf nodes of the tree are capable of providing more specific and detailed information about the examined product, which is more likely to be reserved after the tree-based filtering. To enable an effective extraction of aspect hierarchies from a broad variety of review corpora, a Bayesian nonparametric hierarchical topic model has been constructed and incorporated with an enhanced Pólya urn scheme. The approximate inference of model parameters is obtained by an efficient collapsed Gibbs sampling procedure. The proposed method can enhance the layered effect of individual reviews according to their general-to-specific characteristics and reserve an information-rich subset filtered from the raw review corpus. The merits of the proposed method have been elaborated by case studies on two real-world data sets and an extensive simulation study.

WETM: A word embedding-based topic model with modified collapsed Gibbs sampling for short text

Short texts are a common source of knowledge, and the extraction of such valuable information is beneficial for several purposes. Traditional topic models are incapable of analyzing the internal structural information of topics. They are mostly based on the co-occurrence of words at the document level and are often unable to extract semantically relevant topics from short text datasets due to their limited length. Although some traditional topic models are sensitive to word order due to the strong sparsity of data, they do not perform well on short texts. In this paper, we propose a novel word embedding-based topic model (WETM) for short text documents to discover the structural information of topics and words and eliminate the sparsity problem. Moreover, a modified collapsed Gibbs sampling algorithm is proposed to strengthen the semantic coherence of topics in short texts. WETM extracts semantically coherent topics from short texts and finds relationships between words. Extensive experimental results on two real-world datasets show that WETM achieves better topic quality, topic coherence, classification, and clustering results. WETM also requires less execution time compared to traditional topic models.

An Approximated Collapsed Variational Bayes Approach to Variable Selection in Linear Regression

In this work, we propose a novel approximated collapsed variational Bayes approach to model selection in linear regression. The approximated collapsed variational Bayes algorithm offers improvements over mean field variational Bayes by marginalizing over a subset of parameters and using mean field variational Bayes over the remaining parameters in an analogous fashion to collapsed Gibbs sampling. We have shown that the proposed algorithm, under typical regularity assumptions, (a) includes variables in the true underlying model at an exponential rate in the sample size, or (b) excludes the variables at least at the first order rate in the sample size if the variables are not in the true model. Simulation studies show that the performance of the proposed method is close to that of a particular Markov chain Monte Carlo sampler and a path search based variational Bayes algorithm, but requires an order of magnitude less time. The proposed method is also highly competitive with penalized methods, expectation propagation, stepwise AIC/BIC, BMS, and EMVS under various settings. Supplementary materials for the article are available online.

Cross-collection latent Beta-Liouville allocation model training with privacy protection and applications.

Cross-collection topic models extend previous single-collection topic models, such as Latent Dirichlet Allocation (LDA), to multiple collections. The purpose of cross-collection topic modeling is to model document-topic representations and reveal similarities between each topic and differences among groups. However, the restriction of Dirichlet prior and the significant privacy risk have hampered those models' performance and utility. Training those cross-collection topic models may, in particular, leak sensitive information from the training dataset. To address the two issues mentioned above, we propose a novel model, cross-collection latent Beta-Liouville allocation (ccLBLA), which operates a more powerful prior, Beta-Liouville distribution with a more general covariance structure that enhances topic correlation analysis. To provide privacy protection for the ccLBLA model, we leverage the inherent differential privacy guarantee of the Collapsed Gibbs Sampling (CGS) inference scheme and then propose a hybrid privacy protection algorithm for the ccLBLA model (HPP-ccLBLA) that prevents inferring data from intermediate statistics during the CGS training process without sacrificing its utility. More crucially, our technique is the first attempt to use the cross-collection topic model in image classification applications and investigate the cross-collection topic model's capabilities beyond text analysis. The experimental results for comparative text mining and image classification will show the merits of our proposed approach.

Bayesian Graphical Entity Resolution using Exchangeable Random Partition Priors

Abstract Entity resolution (record linkage or deduplication) is the process of identifying and linking duplicate records in databases. In this paper, we propose a Bayesian graphical approach for entity resolution that links records to latent entities, where the prior representation on the linkage structure is exchangeable. First, we adopt a flexible and tractable set of priors for the linkage structure, which corresponds to a special class of random partition models. Second, we propose a more realistic distortion model for categorical/discrete record attributes, which corrects a logical inconsistency with the standard hit-miss model. Third, we incorporate hyperpriors to improve flexibility. Fourth, we employ a partially collapsed Gibbs sampler for inferential speedups. Using a selection of private and nonprivate data sets, we investigate the impact of our modeling contributions and compare our model with two alternative Bayesian models. In addition, we conduct a simulation study for household survey data, where we vary distortion, duplication rates and data set size. We find that our model performs more consistently than the alternatives across a variety of scenarios and typically achieves the highest entity resolution accuracy (F1 score). Open source software is available for our proposed methodology, and we provide a discussion regarding our work and future directions.

TopicBERT: A Topic-Enhanced Neural Language Model Fine-Tuned for Sentiment Classification.

Sentiment classification is a form of data analytics where people's feelings and attitudes toward a topic are mined from data. This tantalizing power to "predict the zeitgeist" means that sentiment classification has long attracted interest, but with mixed results. However, the recent development of the BERT framework and its pretrained neural language models is seeing new-found success for sentiment classification. BERT models are trained to capture word-level information via mask language modeling and sentence-level contexts via next sentence prediction tasks. Out of the box, they are adequate models for some natural language processing tasks. However, most models are further fine-tuned with domain-specific information to increase accuracy and usefulness. Motivated by the idea that a further fine-tuning step would improve the performance for downstream sentiment classification tasks, we developed TopicBERT-a BERT model fine-tuned to recognize topics at the corpus level in addition to the word and sentence levels. TopicBERT comprises two variants: TopicBERT-ATP (aspect topic prediction), which captures topic information via an auxiliary training task, and TopicBERT-TA, where topic representation is directly injected into a topic augmentation layer for sentiment classification. With TopicBERT-ATP, the topics are predetermined by an LDA mechanism and collapsed Gibbs sampling. With TopicBERT-TA, the topics can change dynamically during the training. Experimental results show that both approaches deliver the state-of-the-art performance in two different domains with SemEval 2014 Task 4. However, in a test of methods, direct augmentation outperforms further training. Comprehensive analyses in the form of ablation, parameter, and complexity studies accompany the results.

Fitting sparse Markov models through a collapsed Gibbs sampler

Fitting sparse Markov models through a collapsed Gibbs sampler

EXTENDED STOCHASTIC BLOCK MODELS WITH APPLICATION TO CRIMINAL NETWORKS.

Reliably learning group structures among nodes in network data is challenging in several applications. We are particularly motivated by studying covert networks that encode relationships among criminals. These data are subject to measurement errors, and exhibit a complex combination of an unknown number of core-periphery, assortative and disassortative structures that may unveil key architectures of the criminal organization. The coexistence of these noisy block patterns limits the reliability of routinely-used community detection algorithms, and requires extensions of model-based solutions to realistically characterize the node partition process, incorporate information from node attributes, and provide improved strategies for estimation and uncertainty quantification. To cover these gaps, we develop a new class of extended stochastic block models (esbm) that infer groups of nodes having common connectivity patterns via Gibbs-type priors on the partition process. This choice encompasses many realistic priors for criminal networks, covering solutions with fixed, random and infinite number of possible groups, and facilitates the inclusion of node attributes in a principled manner. Among the new alternatives in our class, we focus on the Gnedin process as a realistic prior that allows the number of groups to be finite, random and subject to a reinforcement process coherent with criminal networks. A collapsed Gibbs sampler is proposed for the whole esbm class, and refined strategies for estimation, prediction, uncertainty quantification and model selection are outlined. The esbm performance is illustrated in realistic simulations and in an application to an Italian mafia network, where we unveil key complex block structures, mostly hidden from state-of-the-art alternatives.

An improved author-topic (AT) model with authorship credit allocation schemes

Authorship credit allocation schemes have attracted considerable research attention. However, no consensus about which one is the best has been attained until now, and limited evidence from practical tasks has been reported. Therefore, this study uses the author interest discovery task as a real-world task case to provide valuable insights into authorship credit allocation schemes and guidelines for further practical applications. For this purpose, a novel model, ATcredit, is proposed to strengthen the Author-Topic (AT) model with an authorship credit allocation scheme, and collapsed Gibbs sampling is used to approximate the posterior and estimate model parameters. Extensive experiments using the SynBio dataset reveal several interesting findings as follows. (a) Any scheme for allocating unequal authorship credits performs better than its equal-credit counterpart with our ATcredit model in terms of perplexity. (b) The fixed versions of four out of the six schemes work better than their flexible counterparts with our ATcredit model, regardless of the hyper-authorship strategy. (c) The variation coefficient of credit awards can serve as a criterion to decide whether the hyper-authorship strategy should be used. (d) When the number of authors in a scholarly article is less than three, the six authorship credit allocation schemes are similar to each other with our ATcredit model in terms of perplexity. (e) The harmonic counting scheme performs the best, followed by the arithmetic counting scheme, and the network-based counting scheme performs the worst with our ATcredit model in terms of perplexity. (f) The arithmetic counting scheme is similar to the harmonic counting scheme in terms of the normalised mutual information (NMI) of discovered interests, but the geometric counting scheme is different from the axiomatic and network-based counting schemes.

Tiered sentence based topic model for multi-document summarization

In this work, a probabilistic two level topic model named Tiered Sentence based Topic Model is proposed which models the document at sentence and word levels and infer hierarchical latent topics for sentences. The proposed model uses two latent variables for the generation of words- a super topic and a subtopic for each sentence of the document, to model word groupings at sentence level. Popular super topics identify general theme of the documents and are used for selecting summary sentences. The model parameters are used for ranking sentences considering sentence importance and topic coverage. Collapsed Gibbs sampling is used for inference and parameter estimation. The proposed model is used to compare with two sentence based topic models- SenLDA and LDCC on query focused multi-document summarization task, over standard DUC2005 dataset using ROUGE-1 and ROUGE-2 precision and recall scores. The proposed model performs better than Latent Dirichlet Allocation and SenLDA but has been outperformed by LDCC.