Effective Garbage Data Filtering Algorithm for SNS Big Data Processing by Machine Learning

Abstract

Recently, as the use of social network services (SNS) increases in daily modern life, the amount of SNS data generated has become very large. In addition, increasing efforts are being directed to extracting various pieces of information by collecting, processing and analyzing large amounts of SNS data. While various pieces of information can be extracted from SNS data through big data processing, this is a highly resource-intensive task. Therefore, in order to obtain information from SNS data, a lot of time and material resources are required. In this paper, we propose a data filtering algorithm that filters out garbage data that has no meaning as data among SNS data. The proposed algorithm improves the filtering accuracy by recursive learning based on the initial learning data. Experimental results show that the proposed algorithm has a filtering effect of over 70% on the experimental keywords.