Topic Extraction of Crawled Documents Collection using Correlated Topic Model in Mapreduce Framework

Mi Khine Oo,May Aye Khine

doi:10.5121/ijnlc.2019.8602

Abstract

The tremendous increase in the amount of available research documents impels researchers to propose topic models to extract the latent semantic themes of a documents collection. However, how to extract the hidden topics of the documents collection has become a crucial task for many topic model applications. Moreover, conventional topic modeling approaches suffer from the scalability problem when the size of documents collection increases. In this paper, the Correlated Topic Model with variational Expectation-Maximization algorithm is implemented in MapReduce framework to solve the scalability problem. The proposed approach utilizes the dataset crawled from the public digital library. In addition, the full-texts of the crawled documents are analysed to enhance the accuracy of MapReduce CTM. The experiments are conducted to demonstrate the performance of the proposed algorithm. From the evaluation, the proposed approach has a comparable performance in terms of topic coherences with LDA implemented in MapReduce framework.

Highlights

With increased online digital documents, researchers have started to focus on large documents collection for the extraction of hidden semantic themes and the summarization of these large collection
Correlated Topic Model (CTM) [3] proposed a solution to solve the incapability of Latent Dirichlet Allocation (LDA) by substituting the Dirichlet distribution with the logistic normal distribution to exhibit the correlations of the latent topics
The MapReduce CTM with variational EM algorithm is implemented for the crawled documents collection in a Hadoop cluster to extract the latent topics in order to understand the whole documents collection

Summary

Introduction

With increased online digital documents, researchers have started to focus on large documents collection for the extraction of hidden semantic themes and the summarization of these large collection. Probabilistic topic models discover the underlying thematic structures in a collection of documents by extracting the topics. With these extracted topics, the whole documents collection can be summarized and categorized without human annotation effort. Latent Dirichlet Allocation (LDA) [2], one of the most widely known topic models, uses statistical methods to infer the latent topics contained in the document collection. A main shortcoming of LDA is the lack of ability to model the correlations between topics because of using a Dirichlet distribution in order to model the topic proportions. Correlated Topic Model (CTM) [3] proposed a solution to solve the incapability of LDA by substituting the Dirichlet distribution with the logistic normal distribution to exhibit the correlations of the latent topics

Methods

Results

Conclusion