Traditional Topic Models Research Articles

A Hybrid Semantic Representation Method Based on Fusion Conceptual Knowledge and Weighted Word Embeddings for English Texts

The accuracy of traditional topic models may be compromised due to the sparsity of co-occurring vocabulary in the corpus, whereas conventional word embedding models tend to excessively prioritize contextual semantic information and inadequately capture domain-specific features in the text. This paper proposes a hybrid semantic representation method that combines a topic model that integrates conceptual knowledge with a weighted word embedding model. Specifically, we construct a topic model incorporating the Probase concept knowledge base to perform topic clustering and obtain topic semantic representation. Additionally, we design a weighted word embedding model to enhance the contextual semantic information representation of the text. The feature-based information fusion model is employed to integrate the two textual representations and generate a hybrid semantic representation. The hybrid semantic representation model proposed in this study was evaluated based on various English composition test sets. The findings demonstrate that the model presented in this paper exhibits superior accuracy and practical value compared to existing text representation methods.

Efficient topic identification for urgent MOOC Forum posts using BERTopic and traditional topic modeling techniques

AbstractMOOC platforms provide a means of communication through forums, allowing learners to express their difficulties and challenges while studying various courses. Within these forums, some posts require urgent attention from instructors. Failing to respond promptly to these posts can contribute to higher dropout rates and lower course completion rates. While existing research primarily focuses on identifying urgent posts through various classification techniques, it has not adequately addressed the underlying reasons behind them. This research aims to delve into these reasons and assess the extent to which they vary. By understanding the root causes of urgency, instructors can effectively address these issues and provide appropriate support and solutions. BERTopic utilizes the advanced language capabilities of transformer models and represents an advanced approach in topic modeling. In this study, a comparison was conducted to evaluate the performance of BERTopic in topic modeling on MOOCs discussion forums, alongside traditional topic models such as LDA, LSI, and NMF. The experimental results revealed that the NMF and BERTopic models outperformed the other models. Specifically, the NMF model demonstrated superior performance when a lower number of topics was required, whereas the BERTopic model excelled in generating topics with higher coherence when a larger number of topics was needed.The results considering all urgent posts from the dataset were as follows: Optimal number of topics is 6 for NMF and 50 for BERTopic; coherence scores is 0.66 for NMF and 0.616 for BERTopic; and IRBO scores is 1 for both models. This highlights the BERTopic model capability to distinguish and extract diverse topics comprehensively and coherently, aiding in the identification of various reasons behind MOOC Forum posts.

A relational background knowledge boosting based topic model for Chinese poems

Classical Chinese poetry has been increasingly popular in recent years, and modeling its topic is quite a promising area of research. Chinese poems have the characteristic of short in length, but traditional topic models perform poorly when faced with short texts due to the text sparsity. Therefore, topic model should be improved to satisfy the scenario of classical Chinese poems. In this paper, a relational background knowledge boosting based topic model (RBKBTM) was proposed to overcome the text sparsity of Chinese poems. We incorporated background information into the model, which expanded the text content from the semantic perspective. The background knowledge was combined using word embedding and TextRank and was then fed into the core computing process. Subsequently, a new sampling formula was derived. Our proposed model was tested on three different tasks using three different datasets. The results demonstrate that the incorporated background knowledge can effectively overcomes text sparsity, improving the performance and effectiveness of the topic model.

Sparseness-constrained nonnegative tensor factorization for detecting topics at different time scales

Temporal text data, such as news articles or Twitter feeds, often comprises a mixture of long-lasting trends and transient topics. Effective topic modeling strategies should detect both types and clearly locate them in time. We first demonstrate that nonnegative CANDECOMP/PARAFAC decomposition (NCPD) can automatically identify topics of variable persistence. We then introduce sparseness-constrained NCPD (S-NCPD) and its online variant to control the duration of the detected topics more effectively and efficiently, along with theoretical analysis of the proposed algorithms. Through an extensive study on both semi-synthetic and real-world datasets, we find that our S-NCPD and its online variant can identify both short- and long-lasting temporal topics in a quantifiable and controlled manner, which traditional topic modeling methods are unable to achieve. Additionally, the online variant of S-NCPD shows a faster reduction in reconstruction error and results in more coherent topics compared to S-NCPD, thus achieving both computational efficiency and quality of the resulting topics. Our findings indicate that S-NCPD and its online variant are effective tools for detecting and controlling the duration of topics in temporal text data, providing valuable insights into both persistent and transient trends.

Joint modeling of causal phrases-sentiments-aspects using Hierarchical Pitman Yor Process

Traditional sentiment-aware topic models assume that topic or sentiment transition occurs from either a sentence to the next sentence or from a word to the next word. Such models cannot capture a topic or sentiment transition at phrase boundaries. Further, most of the models adopt a sentiment lexicon to initialize sentiment priors and this approach induces coverage problems. To overcome the above-cited limitations, we have proposed a topic model that extracts aspects, sentiments, and causal phrases simultaneously by leveraging Hierarchical Pitman Yor Process (HPYP) that is modified using a sentiment component, a word-tagger to guide the causal phrase generation and a sentiment prior initialized through a sequential model to address coverage problems. We have evaluated our model on six datasets and found that the proposed model outperforms the baselines in terms of perplexity by 14%, topical coherence by 20%, topic diversity by 5%, sentiment classification task’s accuracy by 4% and, precision, recall and F1 score by 2%. Ablation studies assert that sequence model based sentiment prior initialization results in increasing the accuracy of sentiment classification by 2%.

Temporal topic model for clinical pathway mining from electronic medical records

BackgroundIn recent years, the discovery of clinical pathways (CPs) from electronic medical records (EMRs) data has received increasing attention because it can directly support clinical doctors with explicit treatment knowledge, which is one of the key challenges in the development of intelligent healthcare services. However, the existing work has focused on topic probabilistic models, which usually produce treatment patterns with similar treatment activities, and such discovered treatment patterns do not take into account the temporal process of patient treatment which does not meet the needs of practical medical applications.MethodsBased on the assumption that CPs can be derived from the data of EMRs which usually record the treatment process of patients, this paper proposes a new CPs mining method from EMRs, an extended form of the traditional topic model - the temporal topic model (TTM). The method can capture the treatment topics and the corresponding treatment timestamps for each treatment day.ResultsExperimental research conducted on a real-world dataset of patients’ hospitalization processes, and the achieved results demonstrate the applicability and usefulness of the proposed methodology for CPs mining. Compared to existing benchmarks, our model shows significant improvement and robustness.ConclusionOur TTM provides a more competitive way to mine potential CPs considering the temporal features of the EMR data, providing a very prospective tool to support clinical diagnostic decisions.

Semantic-Driven Topic Modeling Using Transformer-Based Embeddings and Clustering Algorithms

Topic modeling is a powerful technique to discover hidden topics and patterns within a collection of documents without prior knowledge. Traditional topic modeling and clustering-based techniques encounter challenges in capturing contextual semantic information. This study introduces an innovative end-to-end semantic-driven topic modeling technique for the topic extraction process, utilizing advanced word and document embeddings combined with a powerful clustering algorithm. This semantic-driven approach represents a significant advancement in topic modeling methodologies. It leverages contextual semantic information to extract coherent and meaningful topics. Specifically, our model generates document embeddings using pre-trained transformer-based language models, reduces the dimensions of the embeddings, clusters the embeddings based on semantic similarity, and generates coherent topics for each cluster. Compared to ChatGPT and traditional topic modeling algorithms, our model provides more coherent and meaningful topics.

Top2Label: Explainable zero shot topic labelling using knowledge graphs

Automatic topic labelling aims to generate coherent, interpretable, and meaningful labels to facilitate the interpretation of various topics within a corpus of documents. Typically, we represent a topic as a list of terms and documents ranked according to probability. In our research, we employ Top2Vec for topic modelling. We propose a novel three-phase, zero-shot topic labelling framework leveraging the ConceptNet knowledge graph (a comprehensive semantic network of words and phrases) and pre-trained language models as external sources of information. The first phase enriches the top n words of a given topic (based on probability) by expanding their neighbourhood in ConceptNet, bridging missing connections and information gaps, and subsequently yielding a semantically enhanced set of potential labels through the generation of a sub-graph. The second phase constructs a neighbourhood graph (organized as a subgraph of ConceptNet) for each candidate label, evaluating each node’s semantic similarity to the topic and retaining the best sub-graph according to semantic similarity. In this phase, a one-word label is extracted from the final graph, concisely representing the topic. In the third phase, we use the language model to derive the final labels by taking the optimal graph as input. In the third phase, we utilize the language model to derive the sentence and summary labels using the optimal graph as input. These additional labels offer more comprehensive and contextually rich topic representations, facilitating more profound understanding and interpretation. By harnessing the power of knowledge graphs and language models, our framework expands the knowledge beyond the topic documents, optimizing the discovered topics with more representative terms while preserving the topic information. The proposed zero-shot approach (employing pre-trained language models and the ConceptNet knowledge graph without additional training) alleviates computational burdens. It reduces the cognitive and interpretative load on end-users by generating three types of labels for each topic: a one-word, sentence, and summary. Experimental results demonstrate that our model significantly surpasses unsupervised baselines and traditional topic labelling models while remaining competitive with supervised baselines in topic labelling performance.

Benchmarking topic models on scientific articles using BERTeley

The introduction of BERTopic marked a crucial advancement in topic modeling and presented a topic model that outperformed both traditional and modern topic models in terms of topic modeling metrics on a variety of corpora. However, unique issues arise when topic modeling is performed on scientific articles. This paper introduces BERTeley, an innovative tool built upon BERTopic, designed to alleviate these shortcomings and improve the usability of BERTopic when conducting topic modeling on a corpus consisting of scientific articles. This is accomplished through BERTeley’s three main features: scientific article preprocessing, topic modeling using pre-trained scientific language models, and topic model metric calculation. Furthermore, an experiment was conducted comparing topic models using four different language models in three corpora consisting of scientific articles.

WES-BTM: A Short Text-Based Topic Clustering Model

User comments often contain their most practical requirements. Using topic modeling of user comments, it is possible to classify and downscale text data, mine the information in user comments, and understand users’ requirements and preferences. However, user comment texts are usually short and lack rich word frequency and contextual information with sparsity. The traditional topic model cannot model and analyze these short texts well. The biterm topic model (BTM), while solving the sparsity problem, suffers from accuracy and noise problems. In order to eliminate information barriers and further ensure information symmetry, a new topic clustering model, termed the word-embedding similarity-based BTM (WES-BTM), is proposed in this paper. The WES-BTM builds on the BTM by converting word pairs into word vectors and calculating their similarity to perform word pair filtering, which in turn improves clustering accuracy. Based on the experimental results using actual data, the WES-BTM outperforms the BTM, LDA, and NMF models in terms of topic coherence, perplexity, and Jensen–Shannon divergence. It is verified that the WES-BTM can effectively reduce noise and improve the quality of topic clustering. In this way, the information in user comments can be better mined.

Combining Knowledge Graph and Word Embeddings for Spherical Topic Modeling.

Probabilistic topic models are considered as an effective framework for text analysis that uncovers the main topics in an unlabeled set of documents. However, the inferred topics by traditional topic models are often unclear and not easy to interpret because they do not account for semantic structures in language. Recently, a number of topic modeling approaches tend to leverage domain knowledge to enhance the quality of the learned topics, but they still assume a multinomial or Gaussian document likelihood in the Euclidean space, which often results in information loss and poor performance. In this article, we propose a Bayesian embedded spherical topic model (ESTM) that combines both knowledge graph and word embeddings in a non-Euclidean curved space, the hypersphere, for better topic interpretability and discriminative text representations. Extensive experimental results show that our proposed model successfully uncovers interpretable topics and learns high-quality text representations useful for common natural language processing (NLP) tasks across multiple benchmark datasets.

WETM: A word embedding-based topic model with modified collapsed Gibbs sampling for short text

Short texts are a common source of knowledge, and the extraction of such valuable information is beneficial for several purposes. Traditional topic models are incapable of analyzing the internal structural information of topics. They are mostly based on the co-occurrence of words at the document level and are often unable to extract semantically relevant topics from short text datasets due to their limited length. Although some traditional topic models are sensitive to word order due to the strong sparsity of data, they do not perform well on short texts. In this paper, we propose a novel word embedding-based topic model (WETM) for short text documents to discover the structural information of topics and words and eliminate the sparsity problem. Moreover, a modified collapsed Gibbs sampling algorithm is proposed to strengthen the semantic coherence of topics in short texts. WETM extracts semantically coherent topics from short texts and finds relationships between words. Extensive experimental results on two real-world datasets show that WETM achieves better topic quality, topic coherence, classification, and clustering results. WETM also requires less execution time compared to traditional topic models.

Multi-channel hypergraph topic neural network for clinical treatment pattern mining

Recently, increasing attention has been paid to mining clinical treatment patterns from electronic medical records (EMRs), which provide physicians with explicit knowledge to guide the patient’s treatment. However, the current work mainly focuses on the topic probability models. These shallow models fail to consider the complex high-order correlations. Hypergraphs provide a natural way to establish complex high-order correlations, but their potential to discover the semantics of latent topic has not been fully explored. In this paper, we integrate the topic model in hypergraph learning and propose a multi-channel hypergraph topic neural network (C3-HGTNN) to discover latent topic treatment patterns with learning high-order correlations. Specifically, the hypergraph network is constructed based on the interactions in the treatment traces, which describe the latent high-order correlations. Then, based on the heterogeneity of the nodes in the hypergraph, we generate multiple channels and perform a convolution operation on each channel to fully encode the hypergraph. Finally, we develop an extensive representation of the treatment trace and activity by combining the embeddings from various channels and generating relevant topic distributions to identify latent treatment patterns. Also, we demonstrate that our model is a form of deep inference for traditional relational topic models, filling the gap in learning high-order correlation in traditional topic models. Extensive experiments on our dataset and public dataset show the effectiveness and superiority of our model compared to the competitive baselines.

A tale of heroes and villains: Russia’s strategic narratives on Twitter during the COVID-19 pandemic

ABSTRACT This research studies how Russian authorities disseminated strategic narratives on Twitter from January 1st, 2020, to March 11th, 2021, this last date coinciding with the first anniversary of the COVID-19 pandemic. 111,077 tweets in English from 203 Russian authorities’ accounts, including Governmental profiles, embassies, and diplomats, are considered. To recognize the narratives, a sequential design is implemented. First, Bertopic is used to perform both traditional topic modeling and dynamic topic modeling. The frequency and thematic significance of the 15 most salient topics are analyzed. Second, the strategic narratives’ approach is adopted to identify thematic and structural patterns among the most representative tweets of each topic. The findings disclose that Russia’s efforts to shape perceptions on Twitter are grounded in a gradual and coordinated approach informed by identity and system narratives. They portray Russia as a long-established beneficial force to the world unlike the West, depicted as a hostile antagonist. Issue narratives during the pandemic, such as the supply of health resources, are presented as a logical outcome of this rationale. Thus, these seemingly scattered tweets can be better understood when they are interpreted within the overarching narratives they form over time.

Ad-Hoc Monitoring of COVID-19 Global Research Trends for Well-Informed Policy Making

The COVID-19 pandemic has affected millions of people worldwide with severe health, economic, social, and political implications. Healthcare Policy Makers (HPMs) and medical experts are at the core of responding to this continuously evolving pandemic situation and are working hard to contain the spread and severity of this relatively unknown virus. Biomedical researchers are continually discovering new information about this virus and communicating the findings through scientific articles. As such, it is crucial for HPMs and funding agencies to monitor the COVID-19 research trend globally on a regular basis. However, given the influx of biomedical research articles, monitoring COVID-19 research trends has become more challenging than ever, especially when HPMs want on-demand guided search techniques with a set of topics of interest in mind. Unfortunately, existing topic trend modeling techniques are unable to serve this purpose as (1) traditional topic models are unsupervised, and (2) HPMs in different regions may have different topics of interest that they want to track. To address this problem, we introduce a novel computational task in this article called Ad-Hoc Topic Tracking , which is essentially a combination of zero-shot topic categorization and the spatio-temporal analysis task. We then propose multiple zero-shot classification methods to solve this task by building on state-of-the-art language understanding techniques. Next, we picked the best-performing method based on its accuracy on a separate validation dataset and then applied it to a corpus of recent biomedical research articles to track COVID-19 research endeavors across the globe using a spatio-temporal analysis. A demo website has also been developed for HPMs to create custom spatio-temporal visualizations of COVID-19 research trends. The research outcomes demonstrate that the proposed zero-shot classification methods can potentially facilitate further research on this important subject matter. At the same time, the spatio-temporal visualization tool will greatly assist HPMs and funding agencies in making well-informed policy decisions for advancing scientific research efforts.

Graph neural topic model with commonsense knowledge

Traditional topic models are based on the bag-of-words assumption, which states that the topic assignment of each word is independent of the others. However, this assumption ignores the relationship between words, which may hinder the quality of extracted topics. To address this issue, some recent works formulate documents as graphs based on word co-occurrence patterns. It assumes that if two words co-occur frequently, they should have the same topic. Nevertheless, it introduces noise edges into the model and thus hinders topic quality since two words co-occur frequently do not mean that they are on the same topic. In this paper, we use the commonsense relationship between words as a bridge to connect the words in each document. Compared to word co-occurrence, the commonsense relationship can explicitly imply the semantic relevance between words, which can be utilized to filter out noise edges. We use a relational graph neural network to capture the relation information in the graph. Moreover, manifold regularization is utilized to constrain the documents’ topic distributions. Experimental results on a public dataset show that our method is effective at extracting topics compared to baseline methods.

Consumer Demand Behavior Mining and Product Recommendation Based on Online Product Review Mining and Fuzzy Sets

Consumer demand is the need for product characteristics expressed in their own words, which is the basis for producers to develop product recommendations. The extraction and analysis of consumer demands is the most critical input information in quality function deployment (QFD), which has a significant impact on the final prioritization of product technical features, product optimization, and subsequent configuration decisions in QFD, and is directly related to the success of product development. However, the traditional QFD approach to demand analysis lacks reliability and feasibility, and its application often requires time and labor costs that exceed the company’s actual capabilities. Therefore, this paper uses online reviews as the data source and constructs a latent Dirichlet allocation (LDA) topic model based on fuzzy sets to explore the consumer demand information reflected in user reviews. We also introduce the concept of word vector to improve the LDA topic model and compare it with the traditional topic model to verify the performance of the model, so as to explore the consumer demand behavior more accurately and efficiently.

Use of Neural Topic Models in conjunction with Word Embeddings to extract meaningful topics from short texts

Unsupervised machine learning is utilized as a part of the process of topic modeling to discover dormant topics hidden within a large number of documents. The topic model can help with the comprehension, organization, and summarization of large amounts of text. Additionally, it can assist with the discovery of hidden topics that vary across different texts in a corpus. Traditional topic models like pLSA (probabilistic latent semantic analysis) and LDA suffer performance loss when applied to short-text analysis caused by the lack of word co-occurrence information in each short text. One technique being developed to solve this problem is pre-trained word embedding (PWE) with an external corpus used with topic models. These techniques are being developed to perform interpretable topic modeling on short texts. Deep neural networks (DNN) and deep generative models have recently advanced, allowing neural topic models (NTM) to achieve flexibility and efficiency in topic modeling. There have been few studies on neural-topic models with pre-trained word embedding for producing significant topics from short texts. An extensive study with five NTMs was accomplished to test the efficacy of additional PWE in generating comprehensible topics through experiments with different datasets in Arabic and French concerning Moroccan news published on Facebook pages. Several metrics, including topic coherence and topic diversity, are utilized in the process of evaluating the extracted topics. Our research shows that the topic coherence of short texts can be significantly improved using a word embedding with an external corpus.

A hierarchical topic analysis tool to facilitate digital humanities research

PurposeThis study aims to develop a hierarchical topic analysis tool (HTAT) based on hierarchical Latent Dirichelet allocation (hLDA) to support digital humanities research that is associated with the need of topic exploration on the Digital Humanities Platform for Mr. Lo Chia-Lun’s Writings (DHP-LCLW). HTAT can assist humanities scholars on distant reading with analysis of hierarchical text topics, through classifying time-stamped texts into multiple historical eras, conducting hierarchical topic modeling (HTM) according to the texts from different eras and presenting through visualization. The comparative network diagram is another function provided to assist humanities scholars in comparing the difference in the topics they wish to explore and to track how the concept of a topic changes over time from a particular perspective. In addition, HTAT can also provide humanities scholars with the feature to view source texts, thus having high potential to be applied in promoting the effectiveness of topic exploration due to simultaneously integrating both the topic exploration functions of distant reading and close reading.Design/methodology/approachThis study adopts a counterbalanced experimental design to examine whether there is significant differences in the effectiveness of topic inquiry, the number of relevant topics inquired and the time spent on them when research participants were alternately conducting text exploration using DHP-LCLW with HTAT or DHP-LCLW with Single-layer Topic Analysis Tool (SLTAT). A technology acceptance questionnaire and semi-structured interviews were also conducted to understand the research participants' perception and feelings toward using the two different tools to assist topic inquiry.FindingsThe experimental results show that DHP-LCLW with HTAT could better assist the research participants, in comparison with DHP-LCLW with SLTAT, to grasp the topic context of the texts from two particular perspectives assigned by this study within a short period. In addition, the results of the interviews revealed that DHP-LCLW with HTAT, in comparison with SLTAT, was able to provide a topic terms that better met research participnats' expectations and needs, and effectively guided them to the corresponding texts for close reading. In the analysis of technology acceptance and interview data, it can be found that the research participants have a high and positive tendency toward using DHP-LCLW with HTAT to assist topic inquiry.Research limitations/implicationsThe Jieba Chinese word segmentation system was used in the Mr. Lo Chia-Lun’s Writings Database in this study, to perform word segmentation on Mr. Lo Chia-Lun’s writing texts for topic modeling based on hLDA. Since Jieba word segmentation system is a lexicon based word segmentation system, it cannot identify new words that have still not been collected in the lexicon well. In this case, the correctness of word segmentation on the target texts will affect the results of hLDA topic modeling, and the effectiveness of HTAT in assisting humanities scholars for topic inquiry.Practical implicationsAn HTAT was developed to support digital humanities research in this study. With HTAT, DHP-LCLW provides hmanities scholars with topic clues from different hierarchical perspectives for textual exploration, and with temporal and comparative network diagrams to assist humanities scholars in tracking the evolution of the topics of specific perspectives over time, to gain a more comprehensive understanding of the overall context of the texts.Originality/valueIn recent years, topic analysis technology that can automatically extract key topic information from a large amount of texts has been developed rapidly, but the topics generated from traditional topic analysis models like LDA (Latent Dirichelet allocation) make it difficult for users to understand the differences in the topics of texts with different hierarchical levels. Thus, this study proposes HTAT which uses hLDA to build a hierarchical topic tree with a tree-like structure without the need to define the number of topics in advance, enabling humanities scholars to quickly grasp the concept of textual topics and use different hierarchical perspectives for further textual exploration. At the same time, it also provides a combination function of temporal division and comparative network diagram to assist humanities scholars in exploring topics and their changes in different eras, which helps them discover more useful research clues or findings.

A clustering-based topic model using word networks and word embeddings

Online social networking services like Twitter are frequently used for discussions on numerous topics of interest, which range from mainstream and popular topics (e.g., music and movies) to niche and specialized topics (e.g., politics). Due to the popularity of such services, it is a challenging task to automatically model and determine the numerous discussion topics given the large amount of tweets. Adding on this complexity is the need to identify these topics with the absence of prior knowledge about both the types and number of topics, while having the requirement of the relevant technical expertise to tune the numerous parameters for the various models. To address this challenge, we develop the Clustering-based Topic Modelling (ClusTop) algorithm that first constructs different types of word networks based on different types of n-grams co-occurrence and word embedding distances. Using these word networks, ClusTop is then able to automatically determine the discussion topics using community detection approaches. In contrast to traditional topic models, ClusTop does not require the tuning or setting of numerous parameters and instead uses community detection approaches to automatically determine the appropriate number of topics. The ClusTop algorithm is also able to capture the syntactic meaning in tweets via the use of bigrams, trigrams, other word combinations and word embedding techniques in constructing the word network graph, and utilizes edge weights based on word embedding. Using three Twitter datasets with labelled crises and events as topics, we show that ClusTop outperforms various traditional baselines in terms of topic coherence, pointwise mutual information, precision, recall and F-score.