Abstract
In this paper, we address a novel method to efficiently manage and analyze a large amount of log data. First, we present a new Apache Hive-based data storage and analysis architecture to process a large volume of Hadoop log data, which rapidly occur in multiple nodes. Second, we design and implement three simple but efficient anomaly detection methods. These methods use moving average and 3-sigma techniques to detect anomalies in log data. Finally, we show that all the three methods detect abnormal intervals properly, and the weighted anomaly detection methods are more precise than the basic one. These results indicate that our research is an excellent and simple approach in detecting anomalies of log data on a Hadoop ecosystem.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
