Hierarchical Latent Dirichlet Allocation Research Articles

Gaussian hierarchical latent Dirichlet allocation: Bringing polysemy back.

Topic models are widely used to discover the latent representation of a set of documents. The two canonical models are latent Dirichlet allocation, and Gaussian latent Dirichlet allocation, where the former uses multinomial distributions over words, and the latter uses multivariate Gaussian distributions over pre-trained word embedding vectors as the latent topic representations, respectively. Compared with latent Dirichlet allocation, Gaussian latent Dirichlet allocation is limited in the sense that it does not capture the polysemy of a word such as "bank." In this paper, we show that Gaussian latent Dirichlet allocation could recover the ability to capture polysemy by introducing a hierarchical structure in the set of topics that the model can use to represent a given document. Our Gaussian hierarchical latent Dirichlet allocation significantly improves polysemy detection compared with Gaussian-based models and provides more parsimonious topic representations compared with hierarchical latent Dirichlet allocation. Our extensive quantitative experiments show that our model also achieves better topic coherence and held-out document predictive accuracy over a wide range of corpus and word embedding vectors which significantly improves the capture of polysemy compared with GLDA and CGTM. Our model learns the underlying topic distribution and hierarchical structure among topics simultaneously, which can be further used to understand the correlation among topics. Moreover, the added flexibility of our model does not necessarily increase the time complexity compared with GLDA and CGTM, which makes our model a good competitor to GLDA.

Web content topic modeling using LDA and HTML tags.

An immense volume of digital documents exists online and offline with content that can offer useful information and insights. Utilizing topic modeling enhances the analysis and understanding of digital documents. Topic modeling discovers latent semantic structures or topics within a set of digital textual documents. The Internet of Things, Blockchain, recommender system, and search engine optimization applications use topic modeling to handle data mining tasks, such as classification and clustering. The usefulness of topic models depends on the quality of resulting term patterns and topics with high quality. Topic coherence is the standard metric to measure the quality of topic models. Previous studies build topic models to generally work on conventional documents, and they are insufficient and underperform when applied to web content data due to differences in the structure of the conventional and HTML documents. Neglecting the unique structure of web content leads to missing otherwise coherent topics and, therefore, low topic quality. This study aims to propose an innovative topic model to learn coherence topics in web content data. We present the HTML Topic Model (HTM), a web content topic model that takes into consideration the HTML tags to understand the structure of web pages. We conducted two series of experiments to demonstrate the limitations of the existing topic models and examine the topic coherence of the HTM against the widely used Latent Dirichlet Allocation (LDA) model and its variants, namely the Correlated Topic Model, the Dirichlet Multinomial Regression, the Hierarchical Dirichlet Process, the Hierarchical Latent Dirichlet Allocation, the pseudo-document based Topic Model, and the Supervised Latent Dirichlet Allocation models. The first experiment demonstrates the limitations of the existing topic models when applied to web content data and, therefore, the essential need for a web content topic model. When applied to web data, the overall performance dropped an average of five times and, in some cases, up to approximately 20 times lower than when applied to conventional data. The second experiment then evaluates the effectiveness of the HTM model in discovering topics and term patterns of web content data. The HTM model achieved an overall 35% improvement in topic coherence compared to the LDA.

A Comprehensive Analysis of COVID-19 Vaccine Discourse by Vaccine Brand on Twitter in Korea: Topic and Sentiment Analysis.

The unprecedented speed of COVID-19 vaccine development and approval has raised public concern about its safety. However, studies on public discourses and opinions on social media focusing on adverse events (AEs) related to COVID-19 vaccine are rare. This study aimed to analyze Korean tweets about COVID-19 vaccines (Pfizer, Moderna, AstraZeneca, Janssen, and Novavax) after the vaccine rollout, explore the topics and sentiments of tweets regarding COVID-19 vaccines, and examine their changes over time. We also analyzed topics and sentiments focused on AEs related to vaccination using only tweets with terms about AEs. We devised a sophisticated methodology consisting of 5 steps: keyword search on Twitter, data collection, data preprocessing, data analysis, and result visualization. We used the Twitter Representational State Transfer application programming interface for data collection. A total of 1,659,158 tweets were collected from February 1, 2021, to March 31, 2022. Finally, 165,984 data points were analyzed after excluding retweets, news, official announcements, advertisements, duplicates, and tweets with <2 words. We applied a variety of preprocessing techniques that are suitable for the Korean language. We ran a suite of analyses using various Python packages, such as latent Dirichlet allocation, hierarchical latent Dirichlet allocation, and sentiment analysis. The topics related to COVID-19 vaccines have a very large spectrum, including vaccine-related AEs, emotional reactions to vaccination, vaccine development and supply, and government vaccination policies. Among them, the top major topic was AEs related to COVID-19 vaccination. The AEs ranged from the adverse reactions listed in the safety profile (eg, myalgia, fever, fatigue, injection site pain, myocarditis or pericarditis, and thrombosis) to unlisted reactions (eg, irregular menstruation, changes in appetite and sleep, leukemia, and deaths). Our results showed a notable difference in the topics for each vaccine brand. The topics pertaining to the Pfizer vaccine mainly mentioned AEs. Negative public opinion has prevailed since the early stages of vaccination. In the sentiment analysis based on vaccine brand, the topics related to the Pfizer vaccine expressed the strongest negative sentiment. Considering the discrepancy between academic evidence and public opinions related to COVID-19 vaccination, the government should provide accurate information and education. Furthermore, our study suggests the need for management to correct the misinformation related to vaccine-related AEs, especially those affecting negative sentiments. This study provides valuable insights into the public discourses and opinions regarding COVID-19 vaccination.

Discovering a Failure Taxonomy for Early Design of Complex Engineered Systems Using Natural Language Processing

Abstract Methodologies for failure assessment frequently rely on historical failure modes, causes, and recommendations for prevention. Meanwhile, there are growing databases of narrative-based lessons that are under-utilized due to their size. Advances in natural language processing (NLP) enable unsupervised extraction of this knowledge. We present a methodology for (1) identifying relevant information using a term frequency inverse document frequency (TF-IDF) classifier and (2) extracting knowledge for failure assessment using a hierarchical topic modeling approach, hierarchical latent Dirichlet allocation (LDA). To interpret the extracted topics, we apply an automatic topic labeling technique using pointwise mutual information (PMI) extraction. The methodology is applied to NASA’s Lessons Learned Information System (LLIS), which is publicly available. Partitioned topics enable the extraction of three aspects: cause, failure, and recommendation, while a hierarchy enables organization into a taxonomy. The methodology is generalizable to databases containing narrative-style documents, while the results from the LLIS represent a summary of themes in the dataset, expressed in a format that can be linked to early design failure analyses.

Measuring Technology Complementarity Between Enterprises With an hLDA Topic Model

When considering a joint venture or merger, it is essential for firms to explore innovators with complementary technology to compensate for any internal limitations in R&D resources. In this article, we provide a framework for exploring the technology complementarity between enterprises in a quantitative manner based on text-mining patent data. A hierarchical latent Dirichlet allocation topic model identifies the technology topics hidden in patent documents along with the hierarchical structure of those topics. The technology complementarity between broad classes of technology and their subclassifications across enterprises is then measured with an improved formulation. An empirical study on three-dimensional printing illustrates the validity, reliability, and practicality of this method and the measurement formula used, endorsed by technical experts. This method can be used to identify R&D opportunities, to find appropriate acquisition targets and potential collaborators, and to support managerial decision-making with quantified information on technology complementarity.

Analysis and tuning of hierarchical topic models based on Renyi entropy approach.

Hierarchical topic modeling is a potentially powerful instrument for determining topical structures of text collections that additionally allows constructing a hierarchy representing the levels of topic abstractness. However, parameter optimization in hierarchical models, which includes finding an appropriate number of topics at each level of hierarchy, remains a challenging task. In this paper, we propose an approach based on Renyi entropy as a partial solution to the above problem. First, we introduce a Renyi entropy-based metric of quality for hierarchical models. Second, we propose a practical approach to obtaining the “correct” number of topics in hierarchical topic models and show how model hyperparameters should be tuned for that purpose. We test this approach on the datasets with the known number of topics, as determined by the human mark-up, three of these datasets being in the English language and one in Russian. In the numerical experiments, we consider three different hierarchical models: hierarchical latent Dirichlet allocation model (hLDA), hierarchical Pachinko allocation model (hPAM), and hierarchical additive regularization of topic models (hARTM). We demonstrate that the hLDA model possesses a significant level of instability and, moreover, the derived numbers of topics are far from the true numbers for the labeled datasets. For the hPAM model, the Renyi entropy approach allows determining only one level of the data structure. For hARTM model, the proposed approach allows us to estimate the number of topics for two levels of hierarchy.

A Hybrid Model for Documents Representation

Text representation is a critical issue for exploring the insights behind the text. Many models have been developed to represent the text in defined forms such as numeric vectors where it would be easy to calculate the similarity between the documents using the well-known distance measures. In this paper, we aim to build a model to represent text semantically either in one document or multiple documents using a combination of hierarchical Latent Dirichlet Allocation (hLDA), Word2vec, and Isolation Forest models. The proposed model aims to learn a vector for each document using the relationship between its words’ vectors and the hierarchy of topics generated using the hierarchical Latent Dirichlet Allocation model. Then, the isolation forest model is used to represent multiple documents in one representation as one profile to facilitate finding similar documents to the profile. The proposed text representation model outperforms the traditional text representation models when applied to represent scientific papers before performing content-based scientific papers recommendation for researchers.

Intelligent Extraction of a Knowledge Ontology From Global Patents

The growth of global patents increased over the last decade as enterprises and inventors sought greater protection of their intellectual property (IP) rights. Global patents represent state-of-the-art knowledge for given domains. This research develops a hierarchical Latent Dirichlet Allocation (LDA)-based approach as a computational intelligent method to discover topics and form a top-down ontology, a semantic schema, representing the collective patent knowledge. To validate the knowledge extraction, 1,546 smart retailing patents collected from the Derwent Innovation platform from 2011 and 2016 are used to build the domain ontology schema. The patent set focuses on in-use, globally established, and non-disputed IP covering payment, user experience, and information integration for smart retailing. The clustering and LDA-based ontology system automatically build the knowledge map, which identifies the technology trends and the technology gaps enabling the development of competitive R&amp;D and management strategies.

A Top-Down Binary Hierarchical Topic Model for Biomedical Literature

Over the past two decades, a number of advances in topic modeling have produced sophisticated models that are capable of generating topic hierarchies. In particular, hierarchical Latent Dirichlet Allocation (hLDA) builds a topic tree based on the nested Chinese Restaurant Process (nCRP) or other sampling processes to generate a topic hierarchy that allows arbitrarily large branch structures and adaptive dataset growth. In addition, hierarchical topic models based on the latent tree model, such as Hierarchical Latent Tree Analysis (HLTA), have been developed over the last five years. However, these models do not work well in cases with millions of documents and hundreds of thousands of terms. In addition, the topic trees generated by these models are always poorly interpretable, and the relationships among topics in different levels are relatively simple. The biomedical literature, including Medline abstracts, has large-scale documents in two major categories: biological laboratory research and medical clinical research. We propose a top-down binary hierarchical topic model (biHTM) for biomedical literature by iteratively applying a flat topic model and adaptively processing subtrees of the hierarchy. The biHTM topic hierarchy of complete Medline abstracts with more than 14 topic node levels shows good bimodality and interpretability. Compared to hLDA and HLTA, biHTM shows promising results in experiments assessed in terms of runtime and quality.

HiLDA: a statistical approach to investigate differences in mutational signatures.

We propose a hierarchical latent Dirichlet allocation model (HiLDA) for characterizing somatic mutation data in cancer. The method allows us to infer mutational patterns and their relative frequencies in a set of tumor mutational catalogs and to compare the estimated frequencies between tumor sets. We apply our method to two datasets, one containing somatic mutations in colon cancer by the time of occurrence, before or after tumor initiation, and the second containing somatic mutations in esophageal cancer by sex, age, smoking status, and tumor site. In colon cancer, the relative frequencies of mutational patterns were found significantly associated with the time of occurrence of mutations. In esophageal cancer, the relative frequencies were significantly associated with the tumor site. Our novel method provides higher statistical power for detecting differences in mutational signatures.

Hierarchical Topic Modeling of Twitter Data for Online Analytical Processing

Social platforms, such as Twitter, reveal much about the tastes of the public. Many studies focus on the content analysis of social platforms, which assists in product promotion and sentiment investigation. On the other hand, online analytical processing (OLAP) has been proven to be very effective for analyzing multidimensional structured data. The key purpose of applying OLAP to text messages, (e.g., tweets), called text OLAP, is to mine and construct the hierarchical dimension based on the unstructured text content. In contrast to the plain texts which text OLAP usually handles, the social media content includes a wealth of social relationship information which can be employed to extract a more effective dimensional hierarchy. In this paper, we propose a topic model called twitter hierarchical latent Dirichlet allocation (thLDA). Based on hierarchical latent Dirichlet allocation, thLDA aims to automatically mine the hierarchical dimension of tweets’ topics, which can be further employed for text OLAP on the tweets. Furthermore, thLDA uses word2vec to analyze the semantic relationships of words in tweets to obtain a more effective dimension. We conduct extensive experiments on huge quantities of Twitter data and evaluate the effectiveness of thLDA. The experimental results demonstrate that it outperforms other current topic models in mining and constructing the hierarchical dimension of tweeters’ topics.

A Novel Hierarchical Topic Model for Horizontal Topic Expansion With Observed Label Information

Hierarchical topic models, such as hierarchical Latent Dirichlet Allocation (hLDA)and its variations, can organize topics into a hierarchy automatically. On the other hand, there are lots of documents associated with hierarchical label information. Incorporating these information into the topic modeling process can help users to obtain a more reasonable hierarchical structure. However, after analyzing various real-world datasets, we find that these hierarchical labels are ambiguous and conflicting in some levels, which introduces error and restriction to the latent topic and the hierarchical structure exploration process. We call it the horizontal topic expansion problem. To address this problem, in this paper, we propose a novel hierarchical topic model named horizontal and vertical hierarchical topic model (HV-HTM), which aims to incorporate the observed hierarchical label information into the topic generation process, while keeping the flexibility of horizontal and vertical expansion of the hierarchical structure in the modeling process. We conduct experiments on BBC news and Yahoo! Answers datasets and evaluate the effectiveness of HV-HTM on three evaluation metrics. The experimental results show that HV-HTM has a significant improvement on topic modeling, compared to the state-of-the-art models, and it can also obtain a more interpretable hierarchical structure.

Scalable training of hierarchical topic models

Large-scale topic models serve as basic tools for feature extraction and dimensionality reduction in many practical applications. As a natural extension of flat topic models, hierarchical topic models (HTMs) are able to learn topics of different levels of abstraction, which lead to deeper understanding and better generalization than their flat counterparts. However, existing scalable systems for flat topic models cannot handle HTMs, due to their complicated data structures such as trees and concurrent dynamically growing matrices, as well as their susceptibility to local optima. In this paper, we study the hierarchical latent Dirichlet allocation (hLDA) model which is a powerful nonparametric Bayesian HTM. We propose an efficient partially collapsed Gibbs sampling algorithm for hLDA, as well as an initialization strategy to deal with local optima introduced by tree-structured models. We also identify new system challenges in building scalable systems for HTMs, and propose efficient data layout for vectorizing HTM as well as distributed data structures including dynamic matrices and trees. Empirical studies show that our system is 87 times more efficient than the previous open-source implementation for hLDA, and can scale to thousands of CPU cores. We demonstrate our scalability on a 131-million-document corpus with 28 billion tokens, which is 4--5 orders of magnitude larger than previously used corpus. Our distributed implementation can extract 1,722 topics from the corpus with 50 machines in just 7 hours.

A framework for recommending health-related topics based on topic modeling in conversational data (Twitter)

Social media is an environment where it enables the user’s to create and share content or participate in all social networking activities; it has given rise to information overloading problem and knowledge starvation. Generally, in large conversational dataset like Twitter it is very difficult to identifying public health-related topics. Machine learning is the area which helps us to overcome this situation. By applying probabilistic unsupervised machine learning we have identified latent topics from the text corpora. Topic modeling algorithms are probabilistic generative models which is used to analyze the words of the original texts to identify themes that run through them, how connected to each other by themes, and how the topics change over (Blei in Commun ACM 55(4):77–84, 2012). In this paper we have proposed a framework for recommending health-related topics by applying an enhanced topic model hierarchical latent Dirichlet allocation for identifying public health-related topics and themes in tweets. We have tested our model by exploring tobacco usage across conversational data, as well as tested with the subset of data generated by tobacco related queries. Our test results indicate most of the public health related issues are uncovered in our model. However, this model provides relevant topic across the tobacco subset.

Computational linguistics literature and citations oriented citation linkage, classification and summarization

Scientific literature is currently the most important resource for scholars, and their citations have provided researchers with a powerful latent way to analyze scientific trends, influences and relationships of works and authors. This paper is focused on automatic citation analysis and summarization for the scientific literature of computational linguistics, which are also the shared tasks in the 2016 workshop of the 2nd Computational Linguistics Scientific Document Summarization at BIRNDL 2016 (The Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries). Each citation linkage between a citation and the spans of text in the reference paper is recognized according to their content similarities via various computational methods. Then the cited text span is classified to five pre-defined facets, i.e., Hypothesis, Implication, Aim, Results and Method, based on various features of lexicons and rules via Support Vector Machine and Voting Method. Finally, a summary of the reference paper from the cited text spans is generated within 250 words. hLDA (hierarchical Latent Dirichlet Allocation) topic model is adopted for content modeling, which provides knowledge about sentence clustering (subtopic) and word distributions (abstractiveness) for summarization. We combine hLDA knowledge with several other classical features using different weights and proportions to evaluate the sentences in the reference paper. Our systems have been ranked top one and top two according to the evaluation results published by BIRNDL 2016, which has verified the effectiveness of our methods.

Object-oriented and multi-scale target classification and recognition based on hierarchical ensemble learning

Target classification and recognition (TCR) of high resolution remote-sensing image is the important ability for earth observation system and unmanned autonomous system. It is difficult to improve the precision of TCR because of different imaging mechanism. In this paper, we propose a brain-inspired computing model for TCR using cognitive computing and deep learning. Accordingly, we have built an ensemble learning algorithm based on deep spiking convolutional neural network and hierarchical latent Dirichlet allocation. The hierarchical features were extracted from remote-sensing image. Then a TCR algorithm for small sample sizes and complex target was designed, which uses the incremental and reinforcement learning based on object-oriented and multi-scale data argumentation. Experimental results demonstrate that our algorithm has state-of-the-art performance on public data sets of optical remote-sensing image and synthetic aperture image. The model proposed can provide reference to explore an essential significance in brain-inspired intelligence, and has significant value in military and civil affairs.

Using Hierarchical Latent Dirichlet Allocation to Construct Feature Tree for Program Comprehension

Program comprehension is an important task faced by developers during software maintenance. With the increasing complexity of evolving systems, program comprehension becomes more and more difficult. In practice, programmers are accustomed to getting a general view of the features in a software system and then finding the interesting or necessary files to start the understanding process. Given a system, developers may need a general view of the system. The traditional view of a system is shown in a package-class structure which is difficult to understand, especially for large systems. In this article, we focus on understanding the system in both feature view and file structure view. This article proposes an approach to generate a feature tree based on hierarchical Latent Dirichlet Allocation (hLDA), which includes two hierarchies, the feature hierarchy and file structure hierarchy. The feature hierarchy shows the features from abstract level to detailed level, while the file structure hierarchy shows the classes from whole to part. Empirical results show that the feature tree can produce a view for the features and files, and the clustering of classes in the package in our approach is better (in terms of recall) than the other clustering approach, that is, hierarchical clustering.

THC-DAT helps in reading a multi-topic document

Purpose With a mass of electronic multi-topic documents available, there is an increasing need for evaluating emerging analysis tools to help users and digital libraries analyze these documents better. The purpose of this paper is to evaluate the effectiveness, efficiency and user satisfaction of THC-DAT, a within-document analysis tool, in reading a multi-topic document. Design/methodology/approach The authors reviewed related literature first, then performed a user-centered, comparative evaluation of two within-document analysis tools, THC-DAT and BOOKMARK. THC-DAT extracts a topic hierarchy tree using hierarchical latent Dirichlet allocation (hLDA) method and takes the context information into account. BOOKMARK provides similar functionality to the Table of Contents bookmarks in Adobe Reader. Three novel kinds of tasks were devised for participants to finish on two tools, with objective results to assess reading effectiveness and efficiency. And post-system questionnaires were employed to obtain participants’ subjective judgments about the tools. Findings The results confirm that THC-DAT is significantly more effective than BOOKMARK, while not inferior in efficiency. There is some evidence that suggests THC-DAT can slow down the process of approaching cognitive overload and improve users’ willingness to undertake difficult task. Based on qualitative data from questionnaires, the results indicate that users were more satisfied when using THC-DAT than BOOKMARK. Practical implications Adopting THC-DAT in digital libraries or electrical document reading systems contributes to promoting users’ reading performance, willingness to undertake difficult task and general satisfaction. Moreover, THC-DAT is of great value to addressing cognitive overload problem in the information retrieval field. Originality/value This paper evaluates a novel within-document analysis tool in analyzing a multi-topic document, and proved that this tool is superior to the benchmark in effectiveness and user satisfaction, and not inferior in efficiency.

THC-DAT: a document analysis tool based on topic hierarchy and context information

Purpose – The purpose of this paper is to propose a novel within-document analysis tool (DAT) topic hierarchy and context-based document analysis tool (THC-DAT) which enables users to interactively analyze any multi-topic document based on fine-grained and hierarchical topics automatically extracted from it. THC-DAT used hierarchical latent Dirichlet allocation method and took the context information into account so that it can reveal the relationships between latent topics and related texts in a document. Design/methodology/approach – The methodology is a case study. The authors reviewed the related literature first, then utilized a general “build and test” research model. After explaining the model, interface and functions of THC-DAT, a case study was presented using a scholarly paper that was analyzed with the tool. Findings – THC-DAT can organize and serve document topics and texts hierarchically and context based, which overcomes the drawbacks of traditional DATs. The navigation, browse, search and comparison functions of THC-DAT enable users to read, search and analyze multi-topic document efficiently and effectively. Practical implications – It can improve the document organization and services in digital libraries or e-readers, by helping users to interactively read, search and analyze documents efficiently and effectively, exploringly learn about unfamiliar topics with little cognitive burden, or deepen their understanding of a document. Originality/value – This paper designs a tool THC-DAT to analyze document in a THC way. It contributes to overcoming the coarse-analysis drawbacks of existing within-DATs.

Multimedia News Summarization in Search

It is a necessary but challenging task to relieve users from the proliferative news information and allow them to quickly and comprehensively master the information of the whats and hows that are happening in the world every day. In this article, we develop a novel approach of multimedia news summarization for searching results on the Internet, which uncovers the underlying topics among query-related news information and threads the news events within each topic to generate a query-related brief overview. First, the hierarchical latent Dirichlet allocation (hLDA) model is introduced to discover the hierarchical topic structure from query-related news documents, and a new approach based on the weighted aggregation and max pooling is proposed to identify one representative news article for each topic. One representative image is also selected to visualize each topic as a complement to the text information. Given the representative documents selected for each topic, a time-bias maximum spanning tree (MST) algorithm is proposed to thread them into a coherent and compact summary of their parent topic. Finally, we design a friendly interface to present users with the hierarchical summarization of their required news information. Extensive experiments conducted on a large-scale news dataset collected from multiple news Web sites demonstrate the encouraging performance of the proposed solution for news summarization in news retrieval.