ELK Stack Research Articles

Innovative Approaches to Infrastructure Monitoring and Alerting Using the ELK Stack

Innovative Approaches to Infrastructure Monitoring and Alerting Using the ELK Stack

Automation in Cloud Infrastructure Management: Enhancing Efficiency and Reliability

This paper explores the role of automation in cloud infrastructure management, focusing on its impact on efficiency and reliability. Key use cases, including Infrastructure as Code (IaC) with Terraform, Continuous Integration/ Continuous De- ployment (CI/CD), and real-time monitoring with the ELK stack are examined. The paper discusses how automation reduces manual effort, promotes consistency, and enhances scalability in cloud environments. It highlights the benefits of automated alerting and its role in improving security and system stability. By embracing automation, organizations can streamline cloud infrastructure management, ensuring a robust and adaptable framework that supports business continuity and agility. Index Terms—Cloud Infrastructure Management, Automation, Infrastructure as Code (IaC), Terraform, ELK Stack (Elastic- search, Logstash, Kibana), Automated Alerting

Using big data to assess an affective domain for distance education

Distance education is popular worldwide, especially after the COVID-19 pandemic. Assessing the affective domain of distance education becomes challenging because the instructor cannot observe the student’s behavior. This article proposes an assessment of the affective domain for distance education systems implemented with big data technologies. An assessment model called AMADEUS has been developed. Exploratory factor analysis (EFA) was used to explore assessment factors and formulate the model. Then, the constructed model was verified by confirmatory factor analysis (CFA). AMADEUS consists of 12 factors in 4 components: responsibility, interest in course material, class participation, and honesty. The experimental results indicated that AMADEUS was consistent with the empirical data and passed all criteria for consideration. After that, an assessment and reporting tool called MOZART has been developed. MOZART employed the AMADEUS model to assess students’ affective domain from log files using Hadoop and ELK stack technologies. The MOZART usability assessment results are at a good level in all aspects.

MAAS - Monitoring as a Service through Payment Gateway

Abstract: This paper describes the creation of a cloud-based Monitoring as a Service (MaaS) platform designed specifically for thorough and safe payment gateway monitoring. Scalability, strong security, and real-time transaction insights are provided by the MaaS project through the use of the ELK Stack (Elasticsearch, Logstash, Kibana). Flexibility in monitoring modules allows for the smooth integration of the ELK Stack for effective analysis and display of vital payment gateway data. An interface that is easy to use enables operators to keep an eye on transactions and spot possible problems. Further integration with internal security and fraud detection tools is made easier by thoroughly described APIs. The goal of this MaaS platform is to improve oversight capabilities for crucial payment gateway operations by providing a consolidated, secure monitoring solution. Future development will involve adding machine learning for anomaly detection and extending data source support to cover other payment channels.

MAAS- Monitoring as a Service through Payment Gateway

Abstract: This paper describes the creation of a cloud-based Monitoring as a Service (MaaS) platform designed specifically for thorough and safe payment gateway monitoring. Scalability, strong security, and real-time transaction insights are provided by the MaaS project through the use of the ELK Stack (Elasticsearch, Logstash, Kibana). Flexibility in monitoring modules allows for the smooth integration of the ELK Stack for effective analysis and display of vital payment gateway data. An interface that is easy to use enables operators to keep an eye on transactions and spot possible problems. Further integration with internal security and fraud detection tools is made easier by thoroughly described APIs. The goal of this MaaS platform is to improve oversight capabilities for crucial payment gateway operations by providing a consolidated, secure monitoring solution. Future development will involve adding machine learning for anomaly detection and extending data source support to cover other payment channels.

Internet of Things Analystics Menggunakan ELK Stacks Dalam Pabrik Perangkat Server

The internet is an information and communication technology that makes it easier for human activities to access data and send information into networks. For industry, the internet is also used to store data both internally and externally. There are several application facilities that are open source to help the industry save data via the internet to the cloud, one of which is the ELK stack. In this research, a simulation of a temperature and humidity monitoring system is designed for a server device factory and simultaneously calculates the amount of production. The data obtained is sent to Elasticsearch as a cloud so that the measurement data can be monitored and visualized in the form of graphs. The system is realized using a microcontroller, ESP8266, DHT11 sensor and proximity sensor. This design is equipped with an ELK system that can collect data and filter the required data so that it can be monitored and visualized in real time. The results of testing the temperature and humidity obtained are in the optimal value range and also proves that this research can be used on a large scale. It is recommended that this design be implemented in a factory or smart home to make it easier to obtain and retrieve the required data.

A general and modular framework for dark web analysis

The dark web, often linked with illegal activities, can be monitored with different solutions. However, these tools are typically purpose-specific and designed for unique use cases. In this study, we propose a flexible and scalable framework that facilitates the easy integration of new workflows for dark web analysis. The design is based on the control, logic and operations layers, supplemented by a tools module, logs management, asynchronous message-based communication and a database. The implementation maps the features into a microservice approach, utilizing the open-source technologies Docker Swarm, Kafka, ELK Stack (Elastic Search, Logstash and Kibana), and PostgreSQL. A workflow to scrape web elements of Tor onion services is deployed and validated, demonstrating considerable framework performance despite the time-consuming task of navigating the dark web. Over 16 h, the framework collected over half million onion domains (84,371 unique ones) and made 78,555 accesses to them.

Real-time bot infection detection system using DNS fingerprinting and machine-learning

In today’s cyberattacks, botnets are used as an advanced technique to generate sophisticated and coordinated attacks. Infected systems connect to a command and control (C&C) server to receive commands and attack. Thus, detecting infected hosts makes it possible to protect the network’s resources and prevent them from illicit activities toward third parties. This research elaborates on the design, implementation, and results of a bot infection detection system based on Domain Name System (DNS) traffic events for a network corporation. An infection detection feasibility analysis is performed by creating fingerprints. The traces are generated from a numerical analysis of 13 attributes. These attributes are obtained from the DNS logs of a DNS server. It looks for fingerprint anomalies using Isolation Forest to label a host as infected or not. In addition, on the traces cataloged as anomalous, a search will be carried out for queries to domains generated by Domain Generation Algorithms (DGA). Then, Random Forest generates a model that detects future bot infections on hosts. The devised system integrates the ELK stack and Python. This integration facilitates the management, transformation, and storage of events, generation of fingerprints, machine learning application, and analysis of fingerprint classification results with a precision greater than 99%.

A New Approach Based on ELK Stack for the Analysis and Visualisation of Geo-referenced Sensor Data

A New Approach Based on ELK Stack for the Analysis and Visualisation of Geo-referenced Sensor Data

Building Learning Analysis System with GQM Methodology and ELK Stack

<p>With the development of Internet technology, Massive Open Online Courses (MOOCs) are becoming popular around the world, and more and more people are learning new knowledge through online learning platforms. Learners generate a lot of information on learning platforms every day, and most of the information is recorded in logs. However, it is difficult to start to extract valuable information from the huge and messy log data of learning platforms, which requires a methodology to guide and a lot of time and cost to process the data. Therefore, this research proposes a data analysis process, using the GQM (Goal Question Metric) method as a guide for the analysis process combined with the Banerjee analysis model to build a series of question lists and metrics to evaluate students’ learning behavior performance, and using the ELK Stack (Elasticsearch Logstash Kibana) as an analysis environment to solve the problem of data processing. Finally, we conduct a case study of a programming course in the OpenEdu e-learning platform to help educators transform the log data into analyzable information to understand students’ learning behaviors in the online course and to propose effective decisions for improving learning outcomes.</p> <p> </p>

Flexible visualization of a 3rd party Intrusion Prevention (Security) tool: A use case with the ELK stack

A difficult aspect of cyber security is the ability to achieve automated real time intrusion prevention across various sets of systems. To this extent, several companies are offering comprehensive solutions that leverage an “accuracy of scale” and moving much of the intelligence and detection on the Cloud, relying on an ever-growing set of data and analytics to increase decision accuracy. Often, they provide tools to visualize the decision workflows in attack prevention (as well as tune the algorithm) but those solutions are not always practical as companies see the problem as “global” that is, from a unified Cyber-security standpoint. However, a key to a successful Cyber-security program is transparency and trust: from an experimental team viewpoint, this specifically means having the ability to immediately see what and from where, who has been blocked and being able to inform the community in case of a revoked access without the need for filing a “ticket” (that may eventually be answered) – in other words, rapid response to their user-base is essential but solutions targeting “sub-groups” in an organization are not often available.We have come up with a versatile solution leveraging the ELK stack (Elasticsearch, Logstash, & Kibana) and an IPS (Intrusion Prevention System) based WAF (Web Application Firewall) from Signal Sciences. Signal Science allows the streaming of detailed logs in a Logstash format suitable for custom solutions for visualization. By combining these two tools, we have strengthened our security posture and enabled individual experiments to monitor their own traffic. Specifically, the IPS WAF provides unique data such as country of origin, protocol, response code, source IP, and paths accessed.In this contribution, we will show how we engineered a visualization solution so experiment groups could access a dashboard with predefined graphs but also, where they can create individual customizable dashboards used to display blocked traffic and troubleshoot latency issues. We will discuss the details and procedures for developing and configuring these tools and how it benefits cyber security postures across our scientific based environment.

Optimization of DevOps Transformation for Cloud-Based Applications

Rapid software development is critical for meeting company objectives and competing more effectively in the competitive IoT infrastructure. DevOps is a growing technique that enables enterprises to provide high-quality software capabilities through automation, to improve team communication, and to increase efficiency across the software product lifecycle. Research problem: Due to the increased demand for new products and technologies, a huge overwork shifted on the organizations for introducing software with pace and to become stable to compete with others. Due to this, the majority of organizations prefer an automated system for product development and require cloud-based applications. The git version control system is used for version management and Docker is used to package code and provide libraries. AWS services are leveraged to deploy an application as a cloud. Jenkins is used as a CI/CD pipeline to manage various phases of development and to make the development process continuous. The ELK stack is used to monitor and visualize the execution of code. In light of the findings, DevOps is an efficient method for cloud application deployment and resource selection based on the relative importance of each optimized objective in terms of value parameters such as cost, memory, and CPU capacity, and that the method can be tailored to specific application requirements. The findings of this analysis indicate that an application can be deployed to the cloud using DevOps techniques. The proposed approach cost 60% less at full weight 1.0 and 11.3% less with no weight compared to the benchmark solution’s 15.078%

Improving service quality management based on call center monitoring solution.

The presented paper investigates common call center challenges ranging from management to software and agents. Service quality is a complex and multifaceted construction that has a major impact on customer acquisition, retention and loyalty. Because there are a few different factors that influence service quality, call center managers must know what they are before they can work to optimize them. As an important part of any business is communication, both within the organization and with customers and clients, a feature-rich, advanced Private Branch Exchange (PBX) phone system is of great benefit. Call Centers are one of the most commonly used implementations of PBX systems in businesses of any type, shape and size. We suggest an efficient and secure solution for collecting, processing, storing, analyzing and visualizing data from call centers. The proposed approach is driven by applying the type of IP PBX engine based on Asterisk. AWS cloud hosted clustering approaches and load balancing system algorithms have been studied in order to mitigate progressively growing load indicators as we add more customers and thus real data sources to the system. The ELK Stack has been chosen as the optimal approach: it is open source and covers a need in the log management and analytics space. The proposed solution will help to automate the call flow process, where it is necessary, based on the analysis of the collected call centers data. The monitoring results obtained along with the particular call center structure knowledges gives us an opportunity for improving call strategies such as ACDs and IVRs, filling agent's knowledge gaps, eliminating the absence of well-defined practices of call handling, etc. It also eliminates the need to hire additional staff or outsource call centers to share the load.

Network Forensic Serangan DoS pada Jaringan Cloud berdasarkan Generic Framework for Network Forensics (GFNF)

The attacks on cloud-based networks have increased and could lead to various disadvantages such as the inaccessibility of services until the loss of user’s trust. Owncloud is one cloud implementation that runs on a network with more than 200 million users. The aims of these researches are to find digital evidence from DoS attacks. Some DoS attacks are SSH brute force, SYN flood, ping of death, and port scanning on the Owncloud network and then finding the digital evidence such as the attacker's IP, time occurred of the attack, types of the attack, also the resource usage of CPU and RAM. This research uses Wireshark and Snort tools to analyze the network and the method of Generic Framework for Network Forensic (GFNF) as a framework during the simulation process until performing the evidence. The simulation was carried out for 1 minute with 30 trials for each attack. The results of this study found the attacker’s IP, time of the attack occurred, types of attack, and also the increase of the resource usage on CPU and RAM when an attack occurred. The found of results digital evidence such as the attacker's IP, the time occurred of attack, and the types of attack were visualized as a table and presented on the ELK Stack dashboard.

Cyberattacks detection and analysis in a network log system using XGBoost with ELK stack

The usage of artificial intelligence and machine learning methods on cyberattacks increasing significantly recently. For the defense method of cyberattacks, it is possible to detect and identify the attack event by observing the log data and analyzing whether it has abnormal behavior or not. This paper implemented the ELK Stack network log system (NetFlow Log) to visually analyze log data and present several network attack behavior characteristics for further analysis. Additionally, this system evaluated the extreme gradient enhancement (XGBoost), Recurrent Neural Network (RNN), and Deep Neural Network (DNN) model for machine learning methods. Keras was used as a deep learning framework for building a model to detect the attack event. From the experiments, it can be confirmed that the XGBoost model has an accuracy rate of 96.01% for potential threats. The full attack dataset can achieve 96.26% accuracy, which is better than RNN and DNN models.

Detection of Network Attacks in a Heterogeneous Industrial Network Based on Machine Learning

The paper discusses the issues of improving algorithms for detecting network attacks in a heterogeneous industrial Internet of Things network based on machine learning technologies for subsequent integration with the subsystems of the center for monitoring and responding to information security incidents. A structural diagram of a network attack detection system and an algorithm for intelligent analysis of network traffic parameters in the task of detecting malicious network activity have been developed. Variants of constructing ensembles of classifiers based on machine learning models and heterogeneous neural network models are analyzed. The F1-measure score when working with test samples reaches 96 %. The possibility of embedding the obtained models as modules of network equipment to increase the efficiency of the analysis of network traffic of industrial systems or use as part of a network intrusion detection system is considered. The efficiency of the obtained solutions in assessing the quality of network attack detection on the original datasets is comparable for the tested models. The most promising for use in specialized signal processors of network equipment is a classifier based on a committee of random trees, since it provides good quality detection of network attacks and does not require significant computing resources when launching a model with coefficients selected during training. Monitoring the state of information and network infrastructure is implemented on the basis of a deployed solution based on the ELK stack. A virtual testing ground has been developed to assess the effectiveness of ML-models for detecting network attacks. Further research is aimed at developing a methodology for testing models in various scenarios for the implementation of targeted multi-step network attacks.

Integration of Low Interaction Honeypot and ELK Stack as Attack Detection Systems on Servers

The high need for information technology that can be accessed anywhere and anytime indirectly opens a big opportunity for irresponsible parties to attack and destroy the system. The server farm is one of the targets most hunted by attackers, intending to damage, and even retrieving victim data. One of the efforts to deal with this problem is to add server security by using honeypot. The existence of a honeypot is one of the efforts to prevent system hacking by creating a fake server to divert attackers access. In its application, the logs generated from the honeypot are only letters and numbers, making it difficult to analyze the logs. It became a problem it will being a lot of log data being processed. To make it easier for administrators in analyzing logs, a visualization system using the ELK Stack is proposed. Honeypot and ELK Stack integration can be a security system solution in detecting attacks while providing visualization to administrators. Five testing schemes were carried out to provide a comparative study between the low interaction honeypot Cowrie and Dionaea. Cowrie delivers a better performance detection system (real-time) compared to the detection system offered by Dionaea, and the average delay time is 3.75 seconds, while ELK managed to provide better monitoring results to administrators through its visualization.

Monitoring system based on ELK Stack to improve IT governance

Currently, monitoring tools in the market help to maintain in optimal conditions the IT infrastructure and thus improve the IT management of an organization providing a strategic alignment. Objective: The objective of the study is to analyze how to respond quickly to incidents that originate in IT resources, through a monitoring system based on ELK Stack, leading to an improvement in IT governance within the organization. A review of different sources of bibliographic documents no older than 5 years and with articles described as research was made. Within IT governance, a monitoring tool based on ELK Stack generates flexibility by generating reports with clean data and facilitates decision making through dynamic dashboards. ELK Stack being an Open Source framework, divided into 3 tools with different purposes, prioritizes the adaptation of adequate monitoring to an organization's plans with IT. The implementation of a monitoring system not only improves the response time to incidents, but also improves the performance of personnel at the operational level and at the managerial level.

Choice of parallelism: multi-GPU driven pipeline for huge academic backbone network

Science Information Network (SINET) is a Japanese academic backbone network for more than 800 research institutions and universities. In this paper, we present a multi-GPU-driven pipeline for handling huge session data of SINET. Our pipeline consists of ELK stack, multi-GPU server, and Splunk. A multi-GPU server is responsible for two procedures: discrimination and histogramming. Discrimination is dividing session data into ingoing/outgoing with subnet mask calculation and network address matching. Histogramming is grouping ingoing/outgoing session data into bins with map-reduce. In our architecture, we use GPU for the acceleration of ingress/egress discrimination of session data. Also, we use a tiling design pattern for building a two-stage map-reduce of CPU and GPU. Our multi-GPU-driven pipeline has succeeded in processing huge workloads of about 1.2–1.6 billion session streams (500–650 GB) within 24 hours.

Cyberattack detection model using deep learning in a network log system with data visualization

Network log data is significant for network administrators, since it contains information on every event that occurs in a network, including system errors, alerts, and packets sending statuses. Effectively analyzing large volumes of diverse log data brings opportunities to identify issues before they become problems and to prevent future cyberattacks; however, processing of the diverse NetFlow data poses challenges such as volume, velocity, and veracity of log data. In this study, by means of Elasticsearch, Logstash, and Kibana, i.e., the ELK Stack, we construct an analysis and management system for network log data, which provides functions to filter, analyze, and display network log data for further applications and creates data visualization on a Web browser. In addition, an advanced cyberattack detection model is facilitated using deep neural network (DNN), recurrent neural networks (RNN), and long short-term memory (LSTM) approaches. By knowing cyberattack behaviors and cross-validating with the log analysis system, one can learn from this model the characteristics of a variety of cyberattacks. Finally, we also implement Grafana to perform metrics monitoring.