Feasibility analysis of big log data real time search based on Hbase and ElasticSearch

Abstract

There are a lot of log events generated very day from modern enterprises. Form the pattern of these log data, enterprises can mine business value; it's even better if they can do it in real time. But managing this big log data is a big challenge because traditional technology is not powerful enough to process huge data. Luckily, Hadoop echo system provides a new way to process big data; ElasticSearch, which is based on Lucene, is the modern search engineen for cloud environment. This paper presents a real-time big data search method: First, Flume agent from the end user's machine collect log events, then ElasticSearch according to the search conditions are needed rowkey list; finally Hbase using these rowkey directly from the database to get the data, the paper-based hardware to create a virtual machine environment, the experiment proved, with the search for more log events, the search response time does not increase linearly. This article will explain Hbase based on modern distributed systems and cloud real-time search search engine ElasticSearch feasibility of large data logs.