Topic Modeling Algorithms Research Articles

Expert Opinion and Coherence Based Topic Modeling

In this paper, we propose a novel algorithm that rearrange the topic assignment results obtained from topic modeling algorithms, including NMF and LDA. The effectiveness of the algorithm is measured by how much the results conform to expert opinion, which is a data structure called TDAG that we defined to represent the probability that a pair of highly correlated words appear together. In order to make sure that the internal structure does not get changed too much from the rearrangement, coherence, which is a well known metric for measuring the effectiveness of topic modeling, is used to control the balance of the internal structure. We developed two ways to systematically obtain the expert opinion from data, depending on whether the data has relevant expert writing or not. The final algorithm which takes into account both coherence and expert opinion is presented. Finally we compare amount of adjustments needed to be done for each topic modeling method, NMF and LDA.

Analysis of publication activity of computational science society in 2001–2017 using topic modelling and graph theory

This paper presents the results of topic modelling and analysis of topic networks using the corpus of the International Conference on Computational Science (ICCS), which contains 5982 domain-specific papers over seventeen years 2001–2017. We discuss the topical structures of ICCS, and show how these topics have evolved over time in response to the topicality of various domains, technologies and methods, and how all these topics relate to one another. This analysis illustrates the multidisciplinary research and collaborations among scientific communities, by constructing static and dynamic networks from the topic modelling results and from the authors’ keywords. The results of this study provide insights regarding the past and future trends of core discussion topics in computational science and show how “computational thinking” has propagated across different fields of study. We used the Non-negative Matrix Factorization (NMF) topic modelling algorithm to discover topics. The resulting topics were then manually labelled and grouped hierarchically on three levels. Next, we applied trend analysis and Change Point Analysis (CPA) to study the evolution of topics over seventeen years and to identify the growing and disappearing topics. We used Gephi to examine the static networks of topics, and an R library called DyA to analyse the dynamic networks of topics. We also analysed the conference as a platform for potential collaboration development through the perspective of collaboration networks. The results show that authors of ICCS papers continue to actively collaborate after the conference − on average authors collaborate with three other ICCS authors, − which suggests that ICCS is a valuable platform for collaboration development.

Supervised topic models with weighted words: multi-label document classification

Supervised topic modeling algorithms have been successfully applied to multi-label document classification tasks. Representative models include labeled latent Dirichlet allocation (L-LDA) and dependency-LDA. However, these models neglect the class frequency information of words (i.e., the number of classes where a word has occurred in the training data), which is significant for classification. To address this, we propose a method, namely the class frequency weight (CF-weight), to weight words by considering the class frequency knowledge. This CF-weight is based on the intuition that a word with higher (lower) class frequency will be less (more) discriminative. In this study, the CF-weight is used to improve L-LDA and dependency-LDA. A number of experiments have been conducted on real-world multi-label datasets. Experimental results demonstrate that CF-weight based algorithms are competitive with the existing supervised topic models.

Weakly supervised topic sentiment joint model with word embeddings

Topic sentiment joint model aims to deal with the problem about the mixture of topics and sentiment simultaneously from online reviews. Most of existing topic sentiment modeling algorithms are mainly based on the state-of-art latent Dirichlet allocation (LDA) and probabilistic latent semantic analysis (PLSA), which infer sentiment and topic distributions from the co-occurrence of words. These methods have been proposed and successfully used for topic and sentiment analysis. However, when the training corpus is small or when the documents are short, the textual features become sparse, so that the results of the sentiment and topic distributions might be not very satisfied. In this paper, we propose a novel topic sentiment joint model called weakly supervised topic sentiment joint model with word embeddings (WS-TSWE), which incorporates word embeddings and HowNet lexicon simultaneously to improve the topic identification and sentiment recognition. The main contributions of WS-TSWE include the following two aspects. (1) Existing models generate the words only from the sentiment-topic-to-word Dirichlet multinomial component, but the WS-TSWE model replaces it with a mixture of two components, a Dirichlet multinomial component and a word embeddings component. Since the word embeddings are trained on a very large corpora and can be used to extend the semantic information of the words, they can provide a certain solution for the problem of the textual sparse. (2) Most of previous models incorporate sentiment knowledge in the β priors. And the priors are usually set from a dictionary and completely rely on previous domain knowledge to identify positive and negative words. In contrast, the WS-TSWE model calculates the sentiment orientation of each word with the HowNet lexicon and automatically infers sentiment-based β priors for sentiment analysis and opinion mining. Furthermore, we implement WS-TSWE with Gibbs sampling algorithms. The experimental results on Chinese and English data sets show that WS-TSWE achieved significant performance in the task of detecting sentiment and topics simultaneously.

Тематическое моделирование русскоязычных текстов с опорой на леммы и лексические конструкции

Исследование посвящено усовершенствованию методов вероятностного тематического моделирования, направленных на выявление скрытых взаимосвязей между словами, документами и темами в текстовых коллекциях. В большинстве тематических моделей темы представлены исключительно униграммами, что в некоторых случаях влечет за собой ухудшение точности и повышает сложность содержательной интерпретации выделяемых тем. Нами предложен новый алгоритм на основе метода LDA, позволяющий автоматически выделять в корпусе словосочетания, состоящие из двух слов, и добавлять их в тематические модели. В статье изложена работа алгоритма и приведены результаты его применения в автоматической обработке корпусов специальных текстов.

Expert Opinion and Coherence Based Topic Modeling

In this paper, we propose a novel algorithm that rearrange the topic assignment results obtained from topic modeling algorithms, including NMF and LDA. The effectiveness of the algorithm is measured by how much the results conform to expert opinion, which is a data structure called TDAG that we defined to represent the probability that a pair of highly correlated words appear together. In order to make sure that the internal structure does not get changed too much from the rearrangement, coherence, which is a well known metric for measuring the effectiveness of topic modeling, is used to control the balance of the internal structure. We developed two ways to systematically obtain the expert opinion from data, depending on whether the data has relevant expert writing or not. The final algorithm which takes into account both coherence and expert opinion is presented. Finally we compare amount of adjustments needed to be done for each topic modeling method, NMF and LDA.

A framework for recommending health-related topics based on topic modeling in conversational data (Twitter)

Social media is an environment where it enables the user’s to create and share content or participate in all social networking activities; it has given rise to information overloading problem and knowledge starvation. Generally, in large conversational dataset like Twitter it is very difficult to identifying public health-related topics. Machine learning is the area which helps us to overcome this situation. By applying probabilistic unsupervised machine learning we have identified latent topics from the text corpora. Topic modeling algorithms are probabilistic generative models which is used to analyze the words of the original texts to identify themes that run through them, how connected to each other by themes, and how the topics change over (Blei in Commun ACM 55(4):77–84, 2012). In this paper we have proposed a framework for recommending health-related topics by applying an enhanced topic model hierarchical latent Dirichlet allocation for identifying public health-related topics and themes in tweets. We have tested our model by exploring tobacco usage across conversational data, as well as tested with the subset of data generated by tobacco related queries. Our test results indicate most of the public health related issues are uncovered in our model. However, this model provides relevant topic across the tobacco subset.

Modeling Structure and Content: Socio-Semantic Network Analysis of the Mahābhārata

There is a demand to incorporate content information into social networks. The authors constructed and visualized a network of the most important gods and heroes in the Sanskrit epic Mahābhārata. The network includes semantic information about the actors and their relationships. These two types of information were collected automatically with the help of the Nubbi topic modeling algorithm, which assigns separate sets of topics to both persons and their relations. The visualization of such a network provides intuitive access to a high density of information, like the topic distribution for each actor and the predominant topic for each relation.

Stability of topic modeling via matrix factorization

The problem of the instability of standard topic modeling algorithms is investigated.Three new stability measures for topic models are proposed.Two new ensemble approaches for topic modeling with matrix factorization are proposed.A detailed evaluation of these approaches is performed on 10 text corpora. Topic models can provide us with an insight into the underlying latent structure of a large corpus of documents. A range of methods have been proposed in the literature, including probabilistic topic models and techniques based on matrix factorization. However, in both cases, standard implementations rely on stochastic elements in their initialization phase, which can potentially lead to different results being generated on the same corpus when using the same parameter values. This corresponds to the concept of instability which has previously been studied in the context of k-means clustering. In many applications of topic modeling, this problem of instability is not considered and topic models are treated as being definitive, even though the results may change considerably if the initialization process is altered. In this paper we demonstrate the inherent instability of popular topic modeling approaches, using a number of new measures to assess stability. To address this issue in the context of matrix factorization for topic modeling, we propose the use of ensemble learning strategies. Based on experiments performed on annotated text corpora, we show that a K-Fold ensemble strategy, combining both ensembles and structured initialization, can significantly reduce instability, while simultaneously yielding more accurate topic models.

Progressive Learning of Topic Modeling Parameters: A Visual Analytics Framework.

Topic modeling algorithms are widely used to analyze the thematic composition of text corpora but remain difficult to interpret and adjust. Addressing these limitations, we present a modular visual analytics framework, tackling the understandability and adaptability of topic models through a user-driven reinforcement learning process which does not require a deep understanding of the underlying topic modeling algorithms. Given a document corpus, our approach initializes two algorithm configurations based on a parameter space analysis that enhances document separability. We abstract the model complexity in an interactive visual workspace for exploring the automatic matching results of two models, investigating topic summaries, analyzing parameter distributions, and reviewing documents. The main contribution of our work is an iterative decision-making technique in which users provide a document-based relevance feedback that allows the framework to converge to a user-endorsed topic distribution. We also report feedback from a two-stage study which shows that our technique results in topic model quality improvements on two independent measures.

Automated competitor analysis using big data analytics

PurposeCompetitor analysis is a key component in operations management. Most business decisions are rooted in the analysis of rival products inferred from market structure. Relative to more traditional competitor analysis methods, the purpose of this paper is to provide operations managers with an innovative tool to monitor a firm’s market position and competitors in real time at higher resolution and lower cost than more traditional competitor analysis methods.Design/methodology/approachThe authors combine the techniques of Web Crawler, Natural Language Processing and Machine Learning algorithms with data visualization to develop a big data competitor-analysis system that informs operations managers about competitors and meaningful relationships among them. The authors illustrate the approach using the fitness mobile app business.FindingsThe study shows that the system supports operational decision making both descriptively and prescriptively. In particular, the innovative probabilistic topic modeling algorithm combined with conventional multidimensional scaling, product feature comparison and market structure analyses reveal an app’s position in relation to its peers. The authors also develop a user segment overlapping index based on user’s social media data. The authors combine this new index with the product functionality similarity index to map indirect and direct competitors with and without user lock-in.Originality/valueThe approach improves on previous approaches by fully automating information extraction from multiple online sources. The authors believe this is the first system of its kind. With limited human intervention, the methodology can easily be adapted to different settings, giving quicker, more reliable real-time results. The approach is also cost effective for market analysis projects covering different data sources.

Machine Learning-Based Classification of 38 Years of Spine-Related Literature Into 100 Research Topics.

Retrospective review. To identify the top 100 spine research topics. Recent advances in "machine learning," or computers learning without explicit instructions, have yielded broad technological advances. Topic modeling algorithms can be applied to large volumes of text to discover quantifiable themes and trends. Abstracts were extracted from the National Library of Medicine PubMed database from five prominent peer-reviewed spine journals (European Spine Journal [ESJ], The Spine Journal [SpineJ], Spine, Journal of Spinal Disorders and Techniques [JSDT], Journal of Neurosurgery: Spine [JNS]). Each abstract was entered into a latent Dirichlet allocation model specified to discover 100 topics, resulting in each abstract being assigned a probability of belonging in a topic. Topics were named using the five most frequently appearing terms within that topic. Significance of increasing ("hot") or decreasing ("cold") topic popularity over time was evaluated with simple linear regression. From 1978 to 2015, 25,805 spine-related research articles were extracted and classified into 100 topics. Top two most published topics included "clinical, surgeons, guidelines, information, care" (n = 496 articles) and "pain, back, low, treatment, chronic" (424). Top two hot trends included "disc, cervical, replacement, level, arthroplasty" (+0.05%/yr, P < 0.001), and "minimally, invasive, approach, technique" (+0.05%/yr, P < 0.001). By journal, the most published topics were ESJ-"operative, surgery, postoperative, underwent, preoperative"; SpineJ-"clinical, surgeons, guidelines, information, care"; Spine-"pain, back, low, treatment, chronic"; JNS- "tumor, lesions, rare, present, diagnosis"; JSDT-"cervical, anterior, plate, fusion, ACDF." Topics discovered through latent Dirichlet allocation modeling represent unbiased meaningful themes relevant to spine care. Topic dynamics can provide historical context and direction for future research for aspiring investigators and trainees interested in spine careers. Please explore https://singdc.shinyapps.io/spinetopics. N A.

Semi-automatic terminology ontology learning based on topic modeling

Ontologies provide features like a common vocabulary, reusability, machine-readable content, and also allows for semantic search, facilitate agent interaction and ordering & structuring of knowledge for the Semantic Web (Web 3.0) application. However, the challenge in ontology engineering is automatic learning, i.e., the there is still a lack of fully automatic approach from a text corpus or dataset of various topics to form ontology using machine learning techniques. In this paper, two topic modeling algorithms are explored, namely LSI & SVD and Mr.LDA for learning topic ontology. The objective is to determine the statistical relationship between document and terms to build a topic ontology and ontology graph with minimum human intervention. Experimental analysis on building a topic ontology and semantic retrieving corresponding topic ontology for the user's query demonstrating the effectiveness of the proposed approach.

Improving Automated Bug Triaging with Specialized Topic Model

Bug triaging refers to the process of assigning a bug to the most appropriate developer to fix. It becomes more and more difficult and complicated as the size of software and the number of developers increase. In this paper, we propose a new framework for bug triaging, which maps the words in the bug reports (i.e., the term space) to their corresponding topics (i.e., the topic space). We propose a specialized topic modeling algorithm named multi-feature topic model (MTM) which extends Latent Dirichlet Allocation (LDA) for bug triaging. MTM considers product and component information of bug reports to map the term space to the topic space. Finally, we propose an incremental learning method named TopicMiner which considers the topic distribution of a new bug report to assign an appropriate fixer based on the affinity of the fixer to the topics. We pair TopicMiner with MTM ( TopicMiner $^{MTM}$ ). We have evaluated our solution on 5 large bug report datasets including GCC, OpenOffice, Mozilla, Netbeans, and Eclipse containing a total of 227,278 bug reports. We show that TopicMiner $^{MTM}$ can achieve top-1 and top-5 prediction accuracies of 0.4831-0.6868, and 0.7686-0.9084, respectively. We also compare TopicMiner $^{MTM}$ with Bugzie, LDA-KL, SVM-LDA, LDA-Activity, and Yang et al.'s approach. The results show that TopicMiner $^{MTM}$ on average improves top-1 and top-5 prediction accuracies of Bugzie by 128.48 and 53.22 percent, LDA-KL by 262.91 and 105.97 percent, SVM-LDA by 205.89 and 110.48 percent, LDA-Activity by 377.60 and 176.32 percent, and Yang et al.'s approach by 59.88 and 13.70 percent, respectively.

PIVE: Per-Iteration Visualization Environment for Real-Time Interactions with Dimension Reduction and Clustering

One of the key advantages of visual analytics is its capability to leverage both humans's visual perception and the power of computing. A big obstacle in integrating machine learning with visual analytics is its high computing cost. To tackle this problem, this paper presents PIVE (Per-Iteration Visualization Environment) that supports real-time interactive visualization with machine learning. By immediately visualizing the intermediate results from algorithm iterations, PIVE enables users to quickly grasp insights and interact with the intermediate output, which then affects subsequent algorithm iterations. In addition, we propose a widely-applicable interaction methodology that allows efficient incorporation of user feedback into virtually any iterative computational method without introducing additional computational cost. We demonstrate the application of PIVE for various dimension reduction algorithms such as multidimensional scaling and t-SNE and clustering and topic modeling algorithms such as k-means and latent Dirichlet allocation.

A general framework to expand short text for topic modeling

Short texts are everywhere in the Web, including messages posted in social media, status messages and blog comments, and uncovering the topics of this type of messages is crucial to a wide range of applications, e.g., context analysis and user characterization. Extracting topics from short text is challenging because of the dependence of conventional methods, such as Latent Dirichlet Allocation, in words co-occurrence, which in short text is rare and make these methods suffer from severe data sparsity. This paper proposes a general framework for topic modeling of short text by creating larger pseudo-document representations from the original documents. In the framework, document components (e.g., words or bigrams) are defined over a metric space, which provides information about the similarity between them. We present two simple, effective and efficient methods that specialize our general framework to create larger pseudo-documents. While the first method considers word co-occurrence to define the metric space, the second relies on distributed word vector representations. The pseudo-documents generated can be given as input to any topic modeling algorithm. Experiments run in seven datasets and compared against state-of-the-art methods for extracting topics by generating pseudo-documents or modifying current topic modeling methods for short text show the methods significantly improve results in terms of normalized pointwise mutual information. A classification task was also used to evaluate the quality of the topics in terms of document representation, where improvements in F1 varied from 1.5 to 15%.

A Topic Modeling Guided Approach for Semantic Knowledge Discovery in e-Commerce

The task of mining large unstructured text archives, extracting useful patterns and then organizing them into a knowledgebase has attained a great attention due to its vast array of immediate applications in business. Businesses thus demand new and efficient algorithms for leveraging potentially useful patterns from heterogeneous data sources that produce huge volumes of unstructured data. Due to the ability to bring out hidden themes from large text repositories, topic modeling algorithms attained significant attention in the recent past. This paper proposes an efficient and scalable method which is guided by topic modeling for extracting concepts and relationships from e-commerce product descriptions and organizing them into knowledgebase. Semantic graphs can be generated from such a knowledgebase on which meaning aware product discovery experience can be built for potential buyers. Extensive experiments using proposed unsupervised algorithms with e-commerce product descriptions collected from open web shows that our proposed method outperforms some of the existing methods of leveraging concepts and relationships so that efficient knowledgebase construction is possible.

Studying Patterns of Communication in Virtual Urban Groups with Different Modes of Privacy

This paper is concerned with online communication of apartment buildings' residents on general purpose social networking site (SNS) VKontakte (VK), focusing on how groups' participants use instruments of SNS to separate place-based discussions and participation in wider community initiatives. With the help of topic modeling algorithm LDA, we analyzed posts collected from online groups related to apartment buildings in St. Petersburg to reveal differences in communication in open groups and restricted access groups. We also looked at overlaps between local groups of apartment buildings and city-wide movements. Our study shows that inside SNS there is a functional differentiation between restricted access groups and open groups, which have different audiences and communicative strategies. Restricted access (private) groups play an important role in the formation of neighbors' communities of trust and, supposedly, can be useful substitutes of face-to-face interaction for people moving into new buildings. Open (public) groups function as public forums for fostering neighbors' cooperation and attracting the attention of the broader public to local issues and conflicts.

Analysis of Computational Science Papers from ICCS 2001-2016 using Topic Modeling and Graph Theory

This paper presents results of topic modeling and network models of topics using the ICCS corpus, which contains domain-specific (computational science) papers over sixteen years (a total of 5695 papers). We discuss topical structures of ICCS, how these topics evolve over time in response to the topicality of various problems, technologies and methods, and how all these topics relate to one another. This analysis illustrates multidisciplinary research and collaborations among scientific communities, by constructing static and dynamic networks from the topic modeling results and the authors’ keywords. The results of this study give insights about the past and future trends of core discussion topics in computational science. We used the Non-negative Matrix Factorization(NMF) topic modeling algorithm to discover topics and labeled and grouped results hierarchically. We used Gephi to study static networks of topics, and an R library called DyA to analyze the dynamic networks of topics.

Detecting Polarizing Language in Twitter using Topic Models and ML Algorithms

The upsurge in the use of social media in public discourses has made it possible for social scientists to engage in emerging and interesting areas of research. Normally, public debates tend to assume polar positions along political, social or ideological lines. Generally, polarity in the language used is more of blaming the opposing group in such debates. In this paper, we investigated the detection of polarizing language in tweets in the event of a disaster. Our approach entails combining topic modeling and Machine Learning (ML) algorithms to generate topics that we consider to be polarized thereby classifying a given tweet as polar or not. Our latent Dirichlet allocation (LDA)-based model incorporates external resources in the form of a lexicon of blame-oriented words to induce the generation of polar topics. The Collapsed Gibbs sampling is used to infer new documents and to estimate the values of parameters employed in our model. We computed the log likelihood (LL) ratios using our model and two other state-of-the-art LDA-based models for evaluation. Furthermore, we compared polarized detection classification accuracy using the features extracted from polarized topics, bag of words (BOW) and part of speech (POS)-based features. Preliminary experiments returned higher overall accuracy results of 87.67% using topic-based features compared to BOW and POS-based features.