Hierarchical Topic Model Research Articles

Unveiling the knowledge structure of technological forecasting and social change (1969–2020) through an NMF-based hierarchical topic model

• The paper presents a longitudinal analysis of the content and topics of the journal technological forecasting and social change. • There are clear bureaus or tranches of literature in the journal corresponding to its long-term mission of technological forecasting and of technology assessment. • The paper demonstrates the use of the non-negative matrix factorization technique on text data. • The paper demonstrates a hierarchical structure of topics is present in the technological forecasting and social change literature, and that this hierarchical structure can be identified through introspection of topic analysis matrices. This article examines the knowledge structure of the journal Technology Forecasting and Social Change (TFSC) from its inception in 1969 until 2020. In this paper we argue that the structure of knowledge in the field of technological forecasting is more complex than a topic model – a bag of words – can in fact effectively reveal. Therefore we propose and demonstrate a hierarchical model that selectively combines topics at varying levels of generality. The resultant analysis, which is based on non-negative matrix factorization, reveals four distinct branches of technology forecasting work, composed of seven distinct topics. Each topic and branch are examined individually through a detailed examination of terms and keywords. Representative works and authors in each of the branches are also identified. The method enables the examination of the complex structure of knowledge in a scientific journal in a succinct representation. The resultant analysis can assist future researchers, enabling them to better position their work, and to better identify the key references across the various subject silos.

An exploration in remote blood pressure management: Application of daily routine pattern based on mobile data in health management

Blood pressure (BP) is an important indicator of an individual's health status and is closely related to daily behaviors. Thus, a continuous daily measurement of BP is critical for hypertension control. To assist continuous measurement, BP prediction based on non-physiological data (ubiquitous mobile phone data) was studied in the research. An algorithm was proposed that predicts BP based on patients’ daily routine, which includes activities such as sleep, work, and commuting. The aim of the research is to provide insight into the application of mobile data in telemonitoring and the continuous unobtrusive daily measurement of BP. A half-year data set from October 2017 of 320 individuals, including telecom data and BP measurement data, was analyzed. Two hierarchical Bayesian topic models were used to extract individuals’ location-driven daily routine patterns (topics) and calculate probabilities among these topics from their day-level mobile trajectories. Based on the topic probability distribution and patients’ contextual data, their BP were predicted using different models. The prediction model comparison shows that the long short-term memory (LSTM) method exceeds others when the data has a high dependency. Otherwise, the Random Forest regression model outperforms the LSTM method. Also, the experimental results validate the effectiveness of the topics in BP prediction.

Hierarchical neural topic modeling with manifold regularization

Topic models have been widely used for learning the latent explainable representation of documents, but most of the existing approaches discover topics in a flat structure. In this study, we propose an effective hierarchical neural topic model with strong interpretability. Unlike the previous neural topic models, we explicitly model the dependency between layers of a network, and then combine latent variables of different layers to reconstruct documents. Utilizing this network structure, our model can extract a tree-shaped topic hierarchy with low redundancy and good explainability by exploiting dependency matrices. Furthermore, we introduce manifold regularization into the proposed method to improve the robustness of topic modeling. Experiments on real-world datasets validate that our model outperforms other topic models in several widely used metrics with much fewer computation costs.

Measuring Technology Complementarity Between Enterprises With an hLDA Topic Model

When considering a joint venture or merger, it is essential for firms to explore innovators with complementary technology to compensate for any internal limitations in R&D resources. In this article, we provide a framework for exploring the technology complementarity between enterprises in a quantitative manner based on text-mining patent data. A hierarchical latent Dirichlet allocation topic model identifies the technology topics hidden in patent documents along with the hierarchical structure of those topics. The technology complementarity between broad classes of technology and their subclassifications across enterprises is then measured with an improved formulation. An empirical study on three-dimensional printing illustrates the validity, reliability, and practicality of this method and the measurement formula used, endorsed by technical experts. This method can be used to identify R&D opportunities, to find appropriate acquisition targets and potential collaborators, and to support managerial decision-making with quantified information on technology complementarity.

Topic Modeling Using Latent Dirichlet allocation

We are not able to deal with a mammoth text corpus without summarizing them into a relatively small subset. A computational tool is extremely needed to understand such a gigantic pool of text. Probabilistic Topic Modeling discovers and explains the enormous collection of documents by reducing them in a topical subspace. In this work, we study the background and advancement of topic modeling techniques. We first introduce the preliminaries of the topic modeling techniques and review its extensions and variations, such as topic modeling over various domains, hierarchical topic modeling, word embedded topic models, and topic models in multilingual perspectives. Besides, the research work for topic modeling in a distributed environment, topic visualization approaches also have been explored. We also covered the implementation and evaluation techniques for topic models in brief. Comparison matrices have been shown over the experimental results of the various categories of topic modeling. Diverse technical challenges and future directions have been discussed.

TopicBank: Collection of coherent topics using multiple model training with their further use for topic model validation

Probabilistic topic modeling of a text collection is a tool for unsupervised learning of the inherent thematic structure of the collection. Given only the text of documents as input, the topic model aims to reveal latent topics as probability distributions over words. The shortcomings of topic models are that they are unstable in the sense that topics may depend on the random initialization, and incomplete in the sense that each new run of the model on the same collection may discover some new topics. This means that data exploration using topic modeling usually requires too many experiments for looking over many topic models and tuning their parameters in search of a model that describes the data best. To deal with the instability and incompleteness of topic models, we propose to gradually accumulate interpretable topics in a “topic bank” using multiple model training. To add topics into the bank, we learn a child level in a hierarchical topic model, then we analyze the coherence of child subtopics and their relationships with parent bank topics in order to exclude irrelevant and duplicate subtopics instead of adding them to the bank. Then we introduce a new way to topic model evaluation by comparing the topics found by the model with the ones that were collected beforehand in a bank. Our experiments with several datasets and topic models show that the proposed method does help in finding a model with more interpretable topics.

Analysis and tuning of hierarchical topic models based on Renyi entropy approach.

Hierarchical topic modeling is a potentially powerful instrument for determining topical structures of text collections that additionally allows constructing a hierarchy representing the levels of topic abstractness. However, parameter optimization in hierarchical models, which includes finding an appropriate number of topics at each level of hierarchy, remains a challenging task. In this paper, we propose an approach based on Renyi entropy as a partial solution to the above problem. First, we introduce a Renyi entropy-based metric of quality for hierarchical models. Second, we propose a practical approach to obtaining the “correct” number of topics in hierarchical topic models and show how model hyperparameters should be tuned for that purpose. We test this approach on the datasets with the known number of topics, as determined by the human mark-up, three of these datasets being in the English language and one in Russian. In the numerical experiments, we consider three different hierarchical models: hierarchical latent Dirichlet allocation model (hLDA), hierarchical Pachinko allocation model (hPAM), and hierarchical additive regularization of topic models (hARTM). We demonstrate that the hLDA model possesses a significant level of instability and, moreover, the derived numbers of topics are far from the true numbers for the labeled datasets. For the hPAM model, the Renyi entropy approach allows determining only one level of the data structure. For hARTM model, the proposed approach allows us to estimate the number of topics for two levels of hierarchy.

A Statistical Perspective on the Challenges in Molecular Microbial Biology.

High throughput sequencing (HTS)-based technology enables identifying and quantifying non-culturable microbial organisms in all environments. Microbial sequences have enhanced our understanding of the human microbiome, the soil and plant environment, and the marine environment. All molecular microbial data pose statistical challenges due to contamination sequences from reagents, batch effects, unequal sampling, and undetected taxa. Technical biases and heteroscedasticity have the strongest effects, but different strains across subjects and environments also make direct differential abundance testing unwieldy. We provide an introduction to a few statistical tools that can overcome some of these difficulties and demonstrate those tools on an example. We show how standard statistical methods, such as simple hierarchical mixture and topic models, can facilitate inferences on latent microbial communities. We also review some nonparametric Bayesian approaches that combine visualization and uncertainty quantification. The intersection of molecular microbial biology and statistics is an exciting new venue. Finally, we list some of the important open problems that would benefit from more careful statistical method development.

Hierarchical features-based targeted aspect extraction from online reviews

With the prevalence of online review websites, large-scale data promote the necessity of focused analysis. This task aims to capture the information that is highly relevant to a specific aspect. However, the broad scope of the aspects of the various products makes this task overarching but challenging. A commonly used solution is to modify the topic models with additional information to capture the features for a specific aspect (referred to as a targeted aspect). However, the existing topic models, either perform the full analysis to capture features as many as possible or estimate the similarity to capture features as coherent as possible, overlook the fine-grained semantic relations between the features, resulting in the captured features coarse and confusing. In this paper, we propose a novel Hierarchical Features-based Topic Model (HFTM) to extract targeted aspects from online reviews, then to capture the aspect-specific features. Specifically, our model can not only capture the direct features posing target-to-feature semantics but also capture the latent features posing feature-to-feature semantics. The experiments conducted on real-world datasets demonstrate that HFTMl outperforms the state-of-the-art baselines in terms of both aspect extraction and document classification.

Hierarchical Interpretable Topical Embeddings for Exploratory Search and Real-Time Document Tracking

Real-time monitoring of scientific papers and technological news requires fast processing of complicated search demands motivated by thematically relevant information acquisition. For this case, the authors develop an exploratory search engine based on probabilistic hierarchical topic modeling. Topic model gives a low dimensional sparse interpretable vector representation (topical embedding) of a text, which is used for ranking documents by their similarity to the query. They explore several ways of comparing topical vectors including searching with thematically homogeneous text segments. Topical hierarchies are built using the regularized EM-algorithm from BigARTM project. The topic-based search achieves better precision and recall than other approaches (TF-IDF, fastText, LSTM, BERT) and even human assessors who spend up to an hour to complete the same search task. They also discover that blending hierarchical topic vectors with neural pretrained embeddings is a promising way of enriching both models that helps to get precision and recall higher than 90%.

The Narrated Global Orientation and the Liabilities of Newness and Foreignness

This research builds upon the liabilities that firms undergo during foreign initial public offerings (IPOs). While many studies have explored global orientation in internationalization, not much research has focused on the IPO context and the effect upon the liabilities of newness and foreignness. Using a sample from 2008 to 2014 of foreign IPO firms in the United States (US), our study finds evidence about the impact of the narrated global orientation. To detect how a firm narratively focuses on which dimensions of global orientation, we use hierarchical topic modeling and propose a global orientation narrative strategy of a firm using the modeling. The results show that the narrated market-driven and technology-driven global orientation in an IPO process have moderating effects on the relationship between the liabilities of newness and foreignness and IPO performance.

Understanding hierarchical structural evolution in a scientific discipline: A case study of artificial intelligence

Detecting what type of knowledge constitutes a discipline, tracking how the knowledge changes, and understanding why the changes are triggered are the key issues in analyzing scientific development from a macro perspective, which is usually analyzed by the topic of evolution. However, traditional methods assume that the disciplinary structure is flat with only one-layer topics, rather than a tree-like structure with hierarchical topics, which leads to the inability of existing methods to effectively examine the details of the evolution, such as the interactions between different research directions. In this paper, we take artificial intelligence (AI) as a case in which we study its hierarchical structural evolution, more precisely inspecting disciplinary development, by analyzing 65,887 AI-related research papers published during a 10-year period from 2009 to 2018. From a hierarchical topic model that can construct a topic-tree with multi-layer organizations, we design a visual analysis model for the topic-tree to systematically and visually investigate how knowledge transfers along the topic-tree and how the topic-tree changes over time. Moreover, some assistant indicators are employed to help in the exploration of the complicated structural evolution. Then, we discover the latent relationship between the sub-structures within a topic as well as the triggering reason for the knowledge migration. Based on these results, we conclude that different topics have different development patterns and that the recent artificial intelligence revolution stems from the interactions among the different topics.

ArchiText: Interactive Hierarchical Topic Modeling.

Human-in-the-loop topic modeling allows users to explore and steer the process to produce better quality topics that align with their needs. When integrated into visual analytic systems, many existing automated topic modeling algorithms are given interactive parameters to allow users to tune or adjust them. However, this has limitations when the algorithms cannot be easily adapted to changes, and it is difficult to realize interactivity closely supported by underlying algorithms. Instead, we emphasize the concept of tight integration, which advocates for the need to co-develop interactive algorithms and interactive visual analytic systems in parallel to allow flexibility and scalability. In this article, we describe design goals for efficiently and effectively executing the concept of tight integration among computation, visualization, and interaction for hierarchical topic modeling of text data. We propose computational base operations for interactive tasks to achieve the design goals. To instantiate our concept, we present ArchiText, a prototype system for interactive hierarchical topic modeling, which offers fast, flexible, and algorithmically valid analysis via tight integration. Utilizing interactive hierarchical topic modeling, our technique lets users generate, explore, and flexibly steer hierarchical topics to discover more informed topics and their document memberships.

A latent topic model with Markov transition for process data.

We propose a latent topic model with a Markov transition for process data, which consists of time-stamped events recorded in a log file. Such data are becoming more widely available in computer-based educational assessment with complex problem-solving items. The proposed model can be viewed as an extension of the hierarchical Bayesian topic model with a hidden Markov structure to accommodate the underlying evolution of an examinee's latent state. Using topic transition probabilities along with response times enables us to capture examinees' learning trajectories, making clustering/classification more efficient. A forward-backward variational expectation-maximization (FB-VEM) algorithm is developed to tackle the challenging computational problem. Useful theoretical properties are established under certain asymptotic regimes. The proposed method is applied to a complex problem-solving item in the 2012 version of the Programme for International Student Assessment (PISA).

A Top-Down Binary Hierarchical Topic Model for Biomedical Literature

Over the past two decades, a number of advances in topic modeling have produced sophisticated models that are capable of generating topic hierarchies. In particular, hierarchical Latent Dirichlet Allocation (hLDA) builds a topic tree based on the nested Chinese Restaurant Process (nCRP) or other sampling processes to generate a topic hierarchy that allows arbitrarily large branch structures and adaptive dataset growth. In addition, hierarchical topic models based on the latent tree model, such as Hierarchical Latent Tree Analysis (HLTA), have been developed over the last five years. However, these models do not work well in cases with millions of documents and hundreds of thousands of terms. In addition, the topic trees generated by these models are always poorly interpretable, and the relationships among topics in different levels are relatively simple. The biomedical literature, including Medline abstracts, has large-scale documents in two major categories: biological laboratory research and medical clinical research. We propose a top-down binary hierarchical topic model (biHTM) for biomedical literature by iteratively applying a flat topic model and adaptively processing subtrees of the hierarchy. The biHTM topic hierarchy of complete Medline abstracts with more than 14 topic node levels shows good bimodality and interpretability. Compared to hLDA and HLTA, biHTM shows promising results in experiments assessed in terms of runtime and quality.

Personal research idea recommendation using research trends and a hierarchical topic model

In the era of rapid technological advance, it is an important task for all researchers to keep up with trends when performing research. How to efficiently find suitable research topics while the number of papers is increasing rapidly is worthwhile to explore. To solve such problems, some researchers attempted to find research ideas by topic detection and tracking methods. However, these methods do not consider the users’ background knowledge and preferences, and they express a topic with general keywords, which does not effectively help researchers to develop new research ideas. Existing studies support that the title expresses the research idea the best. This study adapts this concept to propose an automatic title generation method that combines personalized recommendation methods and topic trend analysis methods to achieve this task. First, it uses hierarchical latent tree analysis to find the users’ interests for a topic structure and its representative keywords hidden in the existing research. Second, the interesting topic trends, popularity and user preferences in a hybrid recommendation method are considered. Finally, a natural language generation algorithm that is suitable for the titles of academic papers converts the original recommended-keywords into fluent title sentences that are designed for the users. Experiments have found that adding Google Trend indicators and personal factors can improve the performance of topic recommendations. The automatic title generation method using template-based and statistical information methods leads to excellent performances in both grammatical correctness and semantic expression. Moreover, for the users, the title is indeed more inspirational than the simple keywords for users to develop new research ideas.

Modeling hierarchical usage context for software exceptions based on interaction data

Traces of user interactions with a software system, captured in production, are commonly used as an input source for user experience testing. In this paper, we present an alternative use, introducing a novel approach of modeling user interaction traces enriched with another type of data gathered in production—software fault reports consisting of software exceptions and stack traces. The model described in this paper aims to improve developers’ comprehension of the circumstances surrounding a specific software exception and can highlight specific user behaviors that lead to a high frequency of software faults. Modeling the combination of interaction traces and software crash reports to form an interpretable and useful model is challenging due to the complexity and variance in the combined data source. Therefore, we propose a probabilistic unsupervised learning approach, adapting the nested hierarchical Dirichlet process, which is a Bayesian non-parametric hierarchical topic model originally applied to natural language data. This model infers a tree of topics, each of whom describes a set of commonly co-occurring commands and exceptions. The topic tree can be interpreted hierarchically to aid in categorizing the numerous types of exceptions and interactions. We apply the proposed approach to large scale datasets collected from the ABB RobotStudio software application, and evaluate it both numerically and with a small survey of the RobotStudio developers.

Modeling stack overflow tags and topics as a hierarchy of concepts

Developers rely on online Q&A forums to look up technical solutions, to pose questions on implementation problems, and to enhance their community profile by contributing answers. Many popular developer communication platforms, such as the Stack Overflow Q&A forum, require threads of discussion to be tagged by their contributors for easier lookup in both asking and answering questions. In this paper, we propose to leverage Stack Overflow’s tags to create a hierarchical organization of concepts discussed on this platform. The resulting concept hierarchy couples tags with a model of their relevancy to prospective questions and answers. For this purpose, we configure and apply a supervised multi-label hierarchical topic model to Stack Overflow questions and demonstrate the quality of the model in several ways: by identifying tag synonyms, by tagging previously unseen Stack Overflow posts, and by exploring how the hierarchy could aid exploratory searches of the corpus. The results suggest that when traversing the inferred hierarchical concept model of Stack Overflow the questions become more specific as one explores down the hierarchy and more diverse as one jumps to different branches. The results also indicate that the model is an improvement over the baseline for the detection of tag synonyms and that the model could enhance existing ensemble methods for suggesting tags for new questions. The paper indicates that the concept hierarchy as a modeling imperative can create a useful representation of the Stack Overflow corpus. This hierarchy can be in turn integrated into development tools which rely on information retrieval and natural language processing, and thereby help developers more efficiently navigate crowd-sourced online documentation.

Bayesian Sparse Topical Coding

Sparse topic models (STMs) are widely used for learning a semantically rich latent sparse representation of short texts in large scale, mainly by imposing sparse priors or appropriate regularizers on topic models. However, it is difficult for these STMs to model the sparse structure and pattern of the corpora accurately, since their sparse priors always fail to achieve real sparseness, and their regularizers bypass the prior information of the relevance between sparse coefficients. In this paper, we propose a novel Bayesian hierarchical topic models called Bayesian Sparse Topical Coding with Poisson Distribution (BSTC-P) on the basis of Sparse Topical Coding with Sparse Groups (STCSG). Different from traditional STMs, it focuses on imposing hierarchical sparse prior to leverage the prior information of relevance between sparse coefficients. Furthermore, we propose a sparsity-enhanced BSTC, Bayesian Sparse Topical Coding with Normal Distribution (BSTC-N), via mathematic approximation. We adopt superior hierarchical sparse inducing prior, with the purpose of achieving the sparsest optimal solution. Experimental results on datasets of Newsgroups and Twitter show that both BSTC-P and BSTC-N have better performance on finding clear latent semantic representations. Therefore, they yield better performance than existing works on document classification tasks.

Sentiment Lexicon Construction With Hierarchical Supervision Topic Model

In this paper, we propose a novel hierarchical supervision topic model to construct a topic-adaptive sentiment lexicon (TaSL) for higher-level classification tasks. It is widely recognized that sentiment lexicon as a useful prior knowledge is crucial in sentiment analysis or opinion mining. However, many existing sentiment lexicons are constructed ignoring the variability of the sentiment polarities of words in different topics or domains. For example, the word “amazing” can refer to causing great surprise or wonder but can also refer to very impressive and excellent. In TaSL, we solve this issue by jointly considering the topics and sentiments of words. Documents are represented by multiple pairs of topics and sentiments, where each pair is characterized by a multinomial distribution over words. Meanwhile, this generating process is supervised under hierarchical supervision information of documents and words. The main advantage of TaSL is that the sentiment polarity of each word in different topics can be sufficiently captured. This model is beneficial to construct a domain-specific sentiment lexicon and then effectively improve the performance of sentiment classification. Extensive experimental results on four publicly available datasets, MR , OMD , semEval13A , and semEval16B were presented to demonstrate the usefulness of the proposed approach. The results have shown that TaSL performs better than the existing manual sentiment lexicon (MPQA), the topic model based domain-specific lexicon (ssLDA), the expanded lexicons(Weka-ED, Weka-STS, NRC, Liu's), and deep neural network based lexicons (nnLexicon, HIT, HSSWE).