Probabilistic Topic Model Research Articles

Learning Hierarchical Document Graphs From Multilevel Sentence Relations.

Organizing the implicit topology of a document as a graph, and further performing feature extraction via the graph convolutional network (GCN), has proven effective in document analysis. However, existing document graphs are often restricted to expressing single-level relations, which are predefined and independent of downstream learning. A set of learnable hierarchical graphs are built to explore multilevel sentence relations, assisted by a hierarchical probabilistic topic model. Based on these graphs, multiple parallel GCNs are used to extract multilevel semantic features, which are aggregated by an attention mechanism for different document-comprehension tasks. Equipped with variational inference, the graph construction and GCN are learned jointly, allowing the graphs to evolve dynamically to better match the downstream task. The effectiveness and efficiency of the proposed multilevel sentence relation graph convolutional network (MuserGCN) is demonstrated via experiments on document classification, abstractive summarization, and matching.

Topic modeling methods for short texts: A survey

In the present day, online users are incentivized to engage in short text-based communication. These short texts harbor a significant amount of implicit information, including opinions, topics, and emotions, which are of notable value for both exploration and analysis. By alleviating the sparsity in short texts, topic models can be used to discover topics from large collections of short texts. While there is a large body of surveys focused on topic modeling, but only a few of them have focused on the short texts. This paper presents a comprehensive overview of topic modeling methods for short texts from a novel perspective. Firstly, it discusses short text probabilistic topic models and outlines the directions in which they can be improved. Secondly, it explores short text neural topic models, which can be categorized into three groups based on their underlying structures. In addition, this paper provides a detailed investigation of embedding methods in topic modeling. Moreover, various applications and corresponding works are surveyed, with a focus on short texts. The commonly used public corpora and evaluation indicators for topic modeling are also summarized. Finally, the advantages and disadvantages of short text topic modeling are discussed in detail, and future research directions are proposed.

Combining Knowledge Graph and Word Embeddings for Spherical Topic Modeling.

Probabilistic topic models are considered as an effective framework for text analysis that uncovers the main topics in an unlabeled set of documents. However, the inferred topics by traditional topic models are often unclear and not easy to interpret because they do not account for semantic structures in language. Recently, a number of topic modeling approaches tend to leverage domain knowledge to enhance the quality of the learned topics, but they still assume a multinomial or Gaussian document likelihood in the Euclidean space, which often results in information loss and poor performance. In this article, we propose a Bayesian embedded spherical topic model (ESTM) that combines both knowledge graph and word embeddings in a non-Euclidean curved space, the hypersphere, for better topic interpretability and discriminative text representations. Extensive experimental results show that our proposed model successfully uncovers interpretable topics and learns high-quality text representations useful for common natural language processing (NLP) tasks across multiple benchmark datasets.

Sağlık Bilgi Yönetiminde Kariyer İçin Gerekli Uzmanlık Rollerinin ve Beceri Setlerinin Tespit Edilmesi

This study aims to investigate the competency areas and skill sets demanded on the job market for health information management (HIM), which plays a vital role in sustaining and enhancing the quality and efficacy of health services. In accordance with this objective, a semantic content analysis was performed on online HIM job postings using a quantitative method based on text mining and probabilistic topic modeling to identify the expertise roles and skill sets as semantic topics. Our findings revealed ten expertise roles and twenty-four skills that represent a broad spectrum of HIM professions’ competency requirements. “Specialist” (17.57%), “Director” (17.05%), “Manager” (13.18%), “Coder” (12.40%), and “Technician” (11.11%) are the top five expertise roles for HIM. A competency taxonomy was developed for HIM professions based on the knowledge and skills revealed by 24 topics using topic modeling analysis. The HIM competencies were categorized as “Medical Knowledge” (39.92%), “Management Skills” (29.80%), “IT Skills” (16.09%), and “Soft Skills” (14.18%). Our findings may have significant implications for HIM candidates and professionals, healthcare industries, and academic institutions in their efforts to comprehend, evaluate, and develop the necessary competencies and skills for HIM careers.

Digital Transformation Strategies, Practices, and Trends: A Large-Scale Retrospective Study Based on Machine Learning

The purpose of this research is to identify the areas of interest, research topics, and application areas that reflect the research nature of digital transformation (DT), as well as the strategies, practices, and trends of DT. To accomplish this, the Latent Dirichlet allocation algorithm, a probabilistic topic modeling technique, was applied to 5350 peer-reviewed journal articles on DT published in the last ten years, from 2013 to 2022. The analysis resulted in the discovery of 34 topics. These topics were classified, and a systematic taxonomy for DT was presented, including four sub-categories: implementation, technology, process, and human. As a result of time-based trend analysis, “Sustainable Energy”, “DT in Health”, “E-Government”, “DT in Education”, and “Supply Chain” emerged as top topics with an increasing trend. Our findings indicate that research interests are focused on specific applications of digital transformation in industrial and public settings. Based on our findings, we anticipate that the next phase of DT research and practice will concentrate on specific DT applications in government, health, education, and economics. “Sustainable Energy” and “Supply Chain” have been identified as the most prominent topics in current DT processes and applications. This study can help researchers and practitioners in the field by providing insights and implications about the evolution and applications of DT. Our findings are intended to serve as a guide for DT in understanding current research gaps and potential future research topics.

Keyword‐Assisted Topic Models

AbstractIn recent years, fully automated content analysis based on probabilistic topic models has become popular among social scientists because of their scalability. However, researchers find that these models often fail to measure specific concepts of substantive interest by inadvertently creating multiple topics with similar content and combining distinct themes into a single topic. In this article, we empirically demonstrate that providing a small number of keywords can substantially enhance the measurement performance of topic models. An important advantage of the proposed keyword‐assisted topic model (keyATM) is that the specification of keywords requires researchers to label topics prior to fitting a model to the data. This contrasts with a widespread practice of post hoc topic interpretation and adjustments that compromises the objectivity of empirical findings. In our application, we find that keyATM provides more interpretable results, has better document classification performance, and is less sensitive to the number of topics.

The development of a competence framework for artificial intelligence professionals using probabilistic topic modelling

PurposeThe current gap between the required and available artificial intelligence (AI) professionals poses significant challenges for organisations and academia. Organisations are challenged to identify and secure the appropriate AI competencies. Simultaneously, academia is challenged to design, offer and quickly scale academic programmes in line with industry needs and train new generations of AI professionals. Therefore, identifying and structuring AI competencies is necessary to effectively overcome the AI competence shortage.Design/methodology/approachA probabilistic topic model was applied to explore the AI competence categories empirically. The authors analysed 1159 AI-related online job ads published on LinkedIn.FindingsThe authors identified five predominant competence categories: (1) Data Science, (2) AI Software Development, (3) AI Product Development and Management, (4) AI Client Servicing, and (5) AI Research. These five competence categories were summarised under the developed AI competence framework.Originality/valueThe AI competence framework contributes to clarifying and structuring the diverse AI landscape. These findings have the potential to aid various stakeholders involved in the process of training, recruiting and selecting AI professionals. They may guide organisations in constructing a complementary portfolio of AI competencies by helping users match the right competence requirements with an organisation's needs and business objectives. Similarly, they can support academia in designing academic programmes aligned with industry needs. Furthermore, while focusing on AI, this study contributes to the research stream of information technology (IT) competencies.

Ranking coherence in topic models using statistically validated networks

Probabilistic topic models have become one of the most widespread machine learning techniques in textual analysis. Topic discovering is an unsupervised process that does not guarantee the interpretability of its output. Hence, the automatic evaluation of topic coherence has attracted the interest of many researchers over the last decade, and it is an open research area. This article offers a new quality evaluation method based on statistically validated networks (SVNs). The proposed probabilistic approach consists of representing each topic as a weighted network of its most probable words. The presence of a link between each pair of words is assessed by statistically validating their co-occurrence in sentences against the null hypothesis of random co-occurrence. The proposed method allows one to distinguish between high-quality and low-quality topics, by making use of a battery of statistical tests. The statistically significant pairwise associations of words represented by the links in the SVN might reasonably be expected to be strictly related to the semantic coherence and interpretability of a topic. Therefore, the more connected the network, the more coherent the topic in question. We demonstrate the effectiveness of the method through an analysis of a real text corpus, which shows that the proposed measure is more correlated with human judgement than the state-of-the-art coherence measures.

Matter of Opinion: Assessing the Role of Individual Judicial Opinions at the International Criminal Court

Abstract This article takes a closer look at the individual opinions of judges at the International Criminal Court (icc). The issuance of separate opinions is one of the most effective ways to investigate individual judicial behaviour, because a judge will only issue opinions if in her/his estimation the benefits outweigh the costs. The number of opinions a judge issues is an important measurement as is their timing. Building on an original dataset, the article identifies patterns by uncovering the predominant issuers, the cases and trial phases where individual opinions are most frequently issued, the dominant topics, and developments over time. Using a probabilistic topic modelling approach, this article suggests that opinions are a common way for several icc judges to engage in judicial politics about several topics with limited judicial restraint and finds that a small group of judges is most avid in using opinions as a judicial tool.

Independent topical structures identification: Theoretical approach and proof-of-concept study

This paper introduces a new approach to the identification of topical structures from text data using independent component analysis (ICA). The approach resembles current probabilistic topic modelling approaches in some respects, however introduces an axiomatic definition of topic structures as independent feature functions over a given vocabulary. In addition, the identification of such structures from data is decoupled from estimates of topical compositions of the training documents. Considerations motivating these choices are discussed, and a proof-of-concept study on a small corpus is presented to demonstrate feasibility and interpretative features of the approach. As computational approach to ICA, a method based on distance covariance is used.

Semantic Recognition of Ship Motion Patterns Entering and Leaving Port Based on Topic Model

Recognition and understanding of ship motion patterns have excellent application value for ship navigation and maritime supervision, i.e., route planning and maritime risk assessment. This paper proposes a semantic recognition method for ship motion patterns entering and leavingport based on a probabilistic topic model. The method enables the discovery of ship motion patterns from a large amount of trajectory data in an unsupervised manner and makes the results more interpretable. The method includes three modules: trajectory preprocessing, semantic process, and knowledge discovery. Firstly, based on the activity types and characteristics of ships in the harbor waters, we propose a multi-criteria ship motion state recognition and voyage division algorithm (McSMSRVD), and ship trajectory is divided into three sub-trajectories: hoteling, maneuvering, and normal-speed sailing. Secondly, considering the influence of port traffic rules on ship motion, the semantic transformation and enrichment of port traffic rules and ship location, course, and speed are combined to construct the trajectory text document. Ship motion patterns hidden in the trajectory document set are recognized using the Latent Dirichlet allocation (LDA) topic model. Meanwhile, topic coherence and topic correlation metrics are introduced to optimize the number of topics. Thirdly, a visualization platform based on ArcGIS and Electronic Navigational Charts (ENCs) is designed to analyze the knowledge of ship motion patterns. Finally, the Tianjin port in northern China is used as the experimental object, and the results show that the method is able to identify 17 representative inbound and outbound motion patterns from AIS data and discover the ship motion details in each pattern.

Web Service Network Embedding Based on Link Prediction and Convolutional Learning

Extensive efforts have been applied to develop efficient feature extraction algorithms, which aim to achieve optimal results in many fundamental tasks such as Web-based software service clustering, recommendation and composition. However, one common issue for existing methods is that mined features are problem dependent, causing poor generalization ability across different applications. Recent studies show that we can represent networked data (e.g., citation networks and social networks) as low-dimensional vectors with rich structure and content information preserved, which can then greatly facilitate many downstream tasks such as classification and clustering. In this article, we focus on the problem of Web service network embedding, which aims to learn low-dimensional vectors to represent services by encoding both Mashup-API composition structure and service functional content. We first propose a novel probabilistic topic model to predict potential links between Mashups and APIs in the service network. Then, we develop a Service Graph Convolutional Network (Service-GCN) to learn vector representations of services, where each service (e.g., Mashup or API) forms its representation through message passing between neighborhood services over the network. We evaluate the network embedding quality on two real-world datasets for downstream classification and clustering tasks. Experimental results show that the average performance of our method improves 20.7 percent (Micro-F1) in service classification and 19.0 percent (Accuracy) in Mashup clustering compared to the state-of-the-art, which verified the effectiveness of the proposed approach for learning vector representations of Web services.

51. Discerning cell types and states in spatial transcriptomics data using topic modelling

Glioblastoma multiforme (GBM) is the most common adult brain tumour. GBM cells include a diverse collection of normal and tumour cells with distinct molecular capabilities and with differential levels of sensitivity to treatment that led to treatment resistance. Spatial transcriptomics technologies (e.g. 10X Visium) have enabled analyzing tumour-normal cells interactions in a physical tissue context. Using Visium, a well-annotated cohort of adult GBM patient samples grown as patient-derived xenograft (PDX) models were profiled at different time points of the disease. However, the identification of cell types or states of Visium spots is a challenging problem as each spot contains a mixture of cells. Current tools for cell-type deconvolution are limited in discovering new novel cell types because they require a well-annotated reference dataset. Moreover, the PDX data has a mixture of human and mouse cell types at the same spot. In this work, we have applied topic modelling, a well-known probabilistic approach in natural language processing to discover the distribution of topics in language documents. Using the analogy between documents and spatial transcriptomic data, we were able to identify the main cell types in spatial PDX data without gene expression references dataset of tumour states, normal mouse, and immune cell types. Finally, we compared various probabilistic topic modelling algorithms to fit the PDX data, and to understand the cell-to-cell interactions between tumour and normal cells.

A new method for mining information of gut microbiome with probabilistic topic models

Microbiome is closely related to many major human diseases, but it is generally analyzed by the traditional statistical methods such as principal component analysis, principal coordinate analysis, etc. These methods have shortcomings and do not consider the characteristics of the microbiome data itself (i.e., the “probability distribution” of microbiome). A new method based on probabilistic topic model was proposed to mine the information of gut microbiome in this paper, taking gut microbiome of type 2 diabetes patients and healthy subjects as an example. Firstly, different weights were assigned to different microbiome according to the degree of correlation between different microbiome and subjects. Then a probabilistic topic model was employed to obtain the probabilistic distribution of gut microbiome (i.e., per-topic OTU (operational taxonomic units, OTU) distribution and per-patient topic distribution). Experimental results showed that the output topics can be used as the characteristics of gut microbiome, and can describe the differences of gut microbiome over different groups. Furthermore, in order to verify the ability of this method to characterize gut microbiome, clustering and classification operations on the distributions over topics for gut microbiome in each subject were performed, and the experimental results showed that the clustering and classification performance has been improved, and the recognition rate of three groups reached 100%. The proposed method could mine the information hidden in gut microbiome data, and the output topics could describe the characteristics of gut microbiome, which provides a new perspective for the study of gut microbiome.

Using SVD for Topic Modeling

The probabilistic topic model imposes a low-rank structure on the expectation of the corpus matrix. Therefore, singular value decomposition (SVD) is a natural tool of dimension reduction. We propose an SVD-based method for estimating a topic model. Our method constructs an estimate of the topic matrix from only a few leading singular vectors of the data matrix, and has a great advantage in memory use and computational cost for large-scale corpora. The core ideas behind our method include a pre-SVD normalization to tackle severe word frequency heterogeneity, a post-SVD normalization to create a low-dimensional word embedding that manifests a simplex geometry, and a post-SVD procedure to construct an estimate of the topic matrix directly from the embedded word cloud. We provide the explicit rate of convergence of our method. We show that our method attains the optimal rate in the case of long and moderately long documents, and it improves the rates of existing methods in the case of short documents. The key of our analysis is a sharp row-wise large-deviation bound for empirical singular vectors, which is technically demanding to derive and potentially useful for other problems. We apply our method to a corpus of Associated Press news articles and a corpus of abstracts of statistical papers. Supplementary materials for this article are available online.

Detecting the research structure and topic trends of social media using static and dynamic probabilistic topic models

PurposeA huge volume of published research articles is available on social media which evolves because of the rapid scientific advances and this paper aims to investigate the research structure of social media.Design/methodology/approachThis study employs an integrated topic modeling and text mining-based approach on 30381 Scopus index titles, abstracts, and keywords published between 2006 and 2021. It combines analytical analysis of top-cited reviews with topic modeling as means of semantic validation. The output sequences of the dynamic model are further analyzed using the statistical techniques that facilitate the extraction of topic clusters, communities, and potential inter-topic research directions.FindingsThis paper brings into vision the research structure of social media in terms of topics, temporal topic evolutions, topic trends, emerging, fading, and consistent topics of this domain. It also traces various shifts in topic themes. The hot research topics are the application of the machine or deep learning towards social media in general, alcohol consumption in different regions and its impact, Social engagement and media platforms. Moreover, the consistent topics in both models include food management in disaster, health study of diverse age groups, and emerging topics include drug violence, analysis of social media news for misinformation, and problems of Internet addiction.Originality/valueThis study extends the existing topic modeling-based studies that analyze the social media literature from a specific disciplinary viewpoint. It focuses on semantic validations of topic-modeling output and correlations among the topics and also provides a two-stage cluster analysis of the topics.

Experiences of Women With Interstitial Cystitis/Bladder Pain Syndrome: What Can We Learn From Women's Online Discussions?

Interstitial cystitis/bladder pain syndrome is a debilitating chronic condition that disproportionately affects women at a ratio of 5:1. We sought to capture women's experiences with interstitial cystitis/bladder pain syndrome by conducting a large-scale digital ethnographic analysis of anonymous posts on Internet forums. Online posts were identified using condition-specific keywords and data mining extraction services. Once posts were identified, a random sample of 200 online posts was coded and analyzed by hand using qualitative methods. A Latent Dirichlet Allocation probabilistic topic model was applied to the complete dataset to substantiate the qualitative analysis and allow for further thematic discovery. A total of 6,842 posts written by 3,902 unique users from 224 websites were identified. There was a significant overlap between the hand coding and Latent Dirichlet Allocation themes. Our analysis yielded the following themes: online community engagement, triggers and disease etiologies, medical comorbidities, quality of life impact, patient experience with medical care, and alternative therapies and self-management strategies. Additionally, our population appeared to have a high burden of nonurological associated syndromes. We identified barriers to patient-centered care and found that online peer support was important for women. Our digital ethnographic analysis is a novel application of qualitative methods using online sources. Social media analytics appears to capture a broader patient population than that typically included in clinic-based qualitative studies, such as patient interviews and focus groups. Understanding patient behaviors and concerns are important to guide strategies for improving care and the overall experience with this difficult-to-treat condition.

A Study on Stochastic Variational Inference for Topic Modeling with Word Embeddings

Probabilistic topic models based on Latent Dirichlet Allocation (LDA) is widely used to extract latent topics from document collections. In recent years, a number of extended topic models have been proposed, especially Gaussian LDA (G-LDA) has attracted a lot of attention. G-LDA integrates topic modeling with word embeddings by replacing discrete topic distributions over words with multivariate Gaussian distributions on the word embedding space. This can reflect semantic information into topics. In this paper, we use G-LDA for our base topic model and apply Stochastic Variational Inference (SVI), an efficient inference algorithm, to estimate topics. Through experiments, we could extract the topics with high coherence in practical time.

Epistemological aspect of topic modelling in the social sciences: Latent Dirichlet Allocation

Aware of the challenges faced by the social sciences in publishing a massive volume of research papers, it is worth looking at a novel but no longer so new ways of machine learning for the purposes of literature review. To this end, I explore a probabilistic topic model called Latent Dirichlet Allocation (LDA) in the context of the epistemological challenge of analysing texts on social welfare. This paper aims to describe how the LDA algorithm works for large corpora of data, along with its advantages and disadvantages. This preliminary characterisation of an inductive method for automated text analysis is intended to give a brief overview of how LDA can be used in the social sciences.

Telescoping bimodal latent Dirichlet allocation to identify expression QTLs across tissues.

Expression quantitative trait loci (eQTLs), or single-nucleotide polymorphisms that affect average gene expression levels, provide important insights into context-specific gene regulation. Classic eQTL analyses use one-to-one association tests, which test gene-variant pairs individually and ignore correlations induced by gene regulatory networks and linkage disequilibrium. Probabilistic topic models, such as latent Dirichlet allocation, estimate latent topics for a collection of count observations. Prior multimodal frameworks that bridge genotype and expression data assume matched sample numbers between modalities. However, many data sets have a nested structure where one individual has several associated gene expression samples and a single germline genotype vector. Here, we build a telescoping bimodal latent Dirichlet allocation (TBLDA) framework to learn shared topics across gene expression and genotype data that allows multiple RNA sequencing samples to correspond to a single individual's genotype. By using raw count data, our model avoids possible adulteration via normalization procedures. Ancestral structure is captured in a genotype-specific latent space, effectively removing it from shared components. Using GTEx v8 expression data across 10 tissues and genotype data, we show that the estimated topics capture meaningful and robust biological signal in both modalities and identify associations within and across tissue types. We identify 4,645 cis-eQTLs and 995 trans-eQTLs by conducting eQTL mapping between the most informative features in each topic. Our TBLDA model is able to identify associations using raw sequencing count data when the samples in two separate data modalities are matched one-to-many, as is often the case in biological data. Our code is freely available at https://github.com/gewirtz/TBLDA.