Incremental technique with set of frequent word item sets for mining large Indonesian text data

Abstract

Indonesian text data from social media is one of large text data that interesting to be mined. Mining the insight knowledge from large text data need more effort and time to processed. Moreover, Indonesian text data from social media contains natural language, including slang that require special treatment. We propose incremental technique for more efficient mining process of large text data with Set of Frequent Word Itemset (SFWI) representation that had been proven capable to keep the meaning of Indonesian text well. We compared Frequent Pattern Growth (FP-Growth) algorithm for not incremental mining and Compact Pattern Growth (CP-Tree) algorithm for incremental mining. The result of experiment with 3,200, 5,000, 110,000, and 239,496 text data form Twitter showed that the incremental technique capable to reduce time process and memory usage for mining Indonesian large text data. Incremental technique with CP-Tree could decrease time process and memory usage so that time process was about 1.66 times faster and 1.84 times more efficient for memory usage than with FP-Growth which was not incremental.