Network Logs Research Articles

AN ASSESSMENT OF FUZZY TEMPORAL EVENT CORRELATION TOWARDS CYBER CRIME INVESTIGATION

Event logging and event logs play an important role in modern IT systems criminal investigation which is generated when end user with each other in web environment and stored in various logs like firewall log file at side, network log file at gateway and web log file at server side. But log file is not to be over emphasized as a source of information in systems and network management. Whereas conduct efficient investigation and gathering of use full information need to correlate different log file. Task of analysing event log files with the ever-increasing size and complexity of today’s event logs has become cumbrous to carry out manually. Nowadays latest spotlighted is automatic analysis of these logs files. . This paper presents a bird eye on two basic concepts one is temporal data mining and another is fuzzy association rules. Using log files it is possible to classify the attacker from the normal user.

Machine Learning for NetFlow Anomaly Detection With Human-Readable Annotations

We propose a framework for anomaly detection in communication network logs along with automated extraction of human-readable annotations that explain the decision logic underlying each anomaly detection. For this purpose, we develop a machine learning methodology formulated in terms of a model comprised of an OR-combination of multiple Boolean logic based sentences. Each sentence is an empirically learned set of inequality conditions involving subsets of features. The feature set, which comprises the “alphabet” for human-readable annotations, is constructed using dynamic graph based spatio-temporal aggregation to extract human-understandable aggregates of network activity. These aggregates are constructed both in terms of computers (nodes in dynamic graph) and communications between computers (edges in dynamic graph). From the alphabet, the learned model identifies subsets of features that relate to each anomaly type and the combinations of conditions in terms of the feature subsets for detection of the specific anomaly type. Given a data point that the learned model detects as anomalous, the model identifies the specific features and their combinations related to the anomaly detection. These human-readable annotations provide a cyber-security analyst a transparent view into the decision logic underlying an anomaly detection.

Network Traffic Anomaly Detection via Deep Learning

Network intrusion detection is a key pillar towards the sustainability and normal operation of information systems. Complex threat patterns and malicious actors are able to cause severe damages to cyber-systems. In this work, we propose novel Deep Learning formulations for detecting threats and alerts on network logs that were acquired by pfSense, an open-source software that acts as firewall on FreeBSD operating system. pfSense integrates several powerful security services such as firewall, URL filtering, and virtual private networking among others. The main goal of this study is to analyse the logs that were acquired by a local installation of pfSense software, in order to provide a powerful and efficient solution that controls traffic flow based on patterns that are automatically learnt via the proposed, challenging DL architectures. For this purpose, we exploit the Convolutional Neural Networks (CNNs), and the Long Short Term Memory Networks (LSTMs) in order to construct robust multi-class classifiers, able to assign each new network log instance that reaches our system into its corresponding category. The performance of our scheme is evaluated by conducting several quantitative experiments, and by comparing to state-of-the-art formulations.

A semantic-aware log generation method for network activities

Context-aware network logging is becoming more prevalent for enterprise networks, data centers, and forensics. Monitoring agents are strategically placed to generate log files from the activity of interests from various network points. In a distributed architecture, these agents are scattered across multiple nodes, and they have limited network visibility. Consequently, the resulting logs become fragmented and less perceptible without a unified network context. Besides, aggregating useful information from a diverse management protocol with various languages, syntax styles, and notations requires complex semantic understanding to synthesize these log files. Currently, general-purpose logs like SNMP's logs only provide parametric values at connection levels but lacks incident-specific information. Meanwhile, proprietary services like AWS CloudTrail identify more contexts at the incident-level, but they only work on selected products and infrastructure. This paper proposed a platform-agnostic log decoding and generation algorithm (SAG) for network logging that is semantic aware using context aggregation. Firstly, a protocol-agnostic controller acts as a master to collect logs from agents running in routers, firewall, IDS/IPS, load balancers, managed switches, and servers. From these logs, three traffic models, namely (1) service-activity model (SaM), (2) general-activity model (GaM), and (3) device-activity model (DaM), are trained using artificial neural network (ANN). The log generator then uses the context-filling technique to resolve and construct log entries using a generic sentence template while inferring from these machine-learning models. A sentence smoothing technique is designed to restructure entities in the logs based on traffic directionality for semantic correctness. The experimental result shows that SAG's logs have 1.8 times more contexts resolved for improved log's perceptibility.

WiFiTrace

Contact tracing is a well-established and effective approach for the containment of the spread of infectious diseases. While Bluetooth-based contact tracing method using phones has become popular recently, these approaches suffer from the need for a critical mass adoption to be effective. In this paper, we present WiFiTrace, a network-centric approach for contact tracing that relies on passive WiFi sensing with no client-side involvement. Our approach exploits WiFi network logs gathered by enterprise networks for performance and security monitoring, and utilizes them for reconstructing device trajectories for contact tracing. Our approach is specifically designed to enhance the efficacy of traditional methods, rather than to supplant them with new technology. We designed an efficient graph algorithm to scale our approach to large networks with tens of thousands of users. The graph-based approach outperforms an indexed PostgresSQL in memory by at least 4.5X without any index update overheads or blocking. We have implemented a full prototype of our system and deployed it on two large university campuses. We validated our approach and demonstrate its efficacy using case studies and detailed experiments using real-world WiFi datasets.

Cyberattack detection model using deep learning in a network log system with data visualization

Network log data is significant for network administrators, since it contains information on every event that occurs in a network, including system errors, alerts, and packets sending statuses. Effectively analyzing large volumes of diverse log data brings opportunities to identify issues before they become problems and to prevent future cyberattacks; however, processing of the diverse NetFlow data poses challenges such as volume, velocity, and veracity of log data. In this study, by means of Elasticsearch, Logstash, and Kibana, i.e., the ELK Stack, we construct an analysis and management system for network log data, which provides functions to filter, analyze, and display network log data for further applications and creates data visualization on a Web browser. In addition, an advanced cyberattack detection model is facilitated using deep neural network (DNN), recurrent neural networks (RNN), and long short-term memory (LSTM) approaches. By knowing cyberattack behaviors and cross-validating with the log analysis system, one can learn from this model the characteristics of a variety of cyberattacks. Finally, we also implement Grafana to perform metrics monitoring.

Detection of network anomalies in log files using machine learning methods

Detection of network anomalies plays an important role in ensuring information security and countering unauthorized access to information infrastructure, including critical facilities. Detecting of abnormal events in log files is complicated by the fact that individual events without any context may be uninformative. The growing importance of log file analysis in large computer systems requires the development of automated methods for processing unstructured data that retrieves information from large log files without human intervention. This article discusses K-means data clustering method and Isolation forest and OSVM machine learning algorithms in terms of searching network anomalies in network log files in order to detect malicious domains.

Real-Time Detection of Dictionary DGA Network Traffic Using Deep Learning

Botnets and malware continue to avoid detection by static rule engines when using domain generation algorithms (DGAs) for callouts to unique, dynamically generated web addresses. Common DGA detection techniques fail to reliably detect DGA variants that combine random dictionary words to create domain names that closely mirror legitimate domains. To combat this, we created a novel hybrid neural network, Bilbo the “bagging” model, that analyses domains and scores the likelihood they are generated by such algorithms and therefore are potentially malicious. Bilbo is the first parallel usage of a convolutional neural network (CNN) and a long short-term memory (LSTM) network for DGA detection. Our unique architecture is found to be the most consistent in performance in terms of AUC, F_1 score, and accuracy when generalising across different dictionary DGA classification tasks compared to current state-of-the-art deep learning architectures. We validate using reverse-engineered dictionary DGA domains and detail our real-time implementation strategy for scoring real-world network logs within a large enterprise. In 4 h of actual network traffic, the model discovered at least five potential command-and-control networks that commercial vendor tools did not flag.

Towards website domain name classification using graph based semi-supervised learning

In this work, we tackle the problem of classifying websites domain names to a category, e.g., mapping bbc.com to the ”News and Media” class. Domain name classification is challenging due to the high number of class labels and the highly skewed class distributions. Differently from prior efforts that need to crawl and use the web pages’ actual content, we rely only on traffic logs passively collected, observing traffic regularly flowing in the network, without the burden to crawl and parse web pages. We exploit the information carried by network logs, using just the name of the websites and the sequence of visited websites by users. For this, we propose and evaluate different classification methods based on machine learning. Using a large dataset with hundreds of thousands of domain names and 25 different categories, we show that semi-supervised learning methods are more suitable for this task than traditional supervised approaches. Using graphs, we incorporate in the classifier aspects not strictly related to the labeled data, and we can classify most of the unlabeled domains. However, in this framework, classification scores are lower than those usually found when exploiting the page-specific content. Our work is the first to perform an extensive evaluation of domain name classification using only passive flow-level logs to the best of our knowledge.

Economic analysis of data protection in systems with complex architecture using neural network methods

Introduction. In the United States, Europe, and Asia, there have been spikes in cyber attacks on protected and confidential information (including bank data, personal data, and confidential business information) over the period 2000-2020. Data protection in systems with complex architectures is a complex and non-trivial solution, which is suitable for flexible self-tuning and self-learning tools, such as neural networks as state modern studies. The described above state of things stipulates the importance and topicality of the direction of our research. Our research raises the question about the economic study of the data but attempts to create a mathematical apparatus to cause a loss of data in digital form, not made before The purpose of the study is to propose and test the calculation of the economic value of the data loss and economic benefit from data protection with multi-level neural networks. Results. Nowadays, almost every business that has access to the Internet has a threat of losing protected data, including banking data. To prevent this, the company must comply with the data protection management regulations. As an example, we studied the process of neural network reaction to an attack and its detection, when two types of attacks are threatened: botnet and UDP flood. Due to the result of a quick response to detect suspicious activity, neural network methods are excellent for use in economic analysis, because the neural network logs every event for every millisecond. Conclusions. As a result of using neural network tools for economic analysis of data protection in modern systems with complex architecture, we were able to obtain stable results of responding to an emerging attack in order to obtain protected data. Such protection is a preventive measure for the possibility of reducing business losses in the event of data loss. As we can see, the management of protection of systems with complex architecture is necessary for each company with the specified data level.

RDP-based Lateral Movement detection using Machine Learning

Detecting cyber threats has been an on-going research endeavor. In this era, Advanced Persistent Threats (APTs) can incur significant costs for organizations and businesses. The ultimate goal of cybersecurity is to thwart attackers from achieving their malicious intent, whether it is credential stealing, infrastructure takeover, or program sabotage. Every cyber attack goes through several stages before its termination. Lateral Movement (LM) is one of those stages that is of particular importance. Remote Desktop Protocol (RDP) is a method used in LM to successfully authenticate to an unauthorized host that leaves footprints on both host and network logs. In this paper, we propose to detect evidence of LM using Machine Learning (ML) and Windows RDP event logs. We explore different feature sets extracted from these logs and evaluate various supervised ML techniques for classifying RDP sessions with high precision and recall. We also compare the performance of our proposed approach to a state-of-the-art approach and demonstrate that our ML model outperforms in classifying RDP sessions in Windows event logs. In addition, we show that our model is robust against certain types of adversarial attacks.

An Emotional Recommender System for Music

Nowadays, recommender systems have become essential to users for finding “what they need” within large collections of items. Meanwhile, recent studies have demonstrated as user personality can effectively provide a more valuable information to significantly improve recommenders’ performance, especially considering behavioral data captured from social network logs. In this work, we describe a novel music recommendation technique based on the identification of personality traits, moods, and emotions of a single user, starting from solid psychological observations recognized by the analysis of user behavior within a social environment. In particular, users’ personality and mood have been embedded within a content-based filtering approach to obtain more accurate and dynamic results. Several experiments are then reported to show effectiveness of user personality and mood recognition recommendation, thus, encouraging research in this direction.

Identifying Intrusion Behaviour using Enhanced Hidden Markov Model

Data Mining is a method for detecting network intrusion detection in networks. It brings ideas from variety of areas including statistics, machine learning and database processes. Decreasing price of digital networking is now economically viable for network intrusion detection. This analysis chiefly examines the system intrusion detection with machine learning and DM methods. To improve the accuracy and efficiency of SHMM, we are collecting multiple observation in SHMM that will be called as Multiple Hidden Markov Model (MHMM). It is used to improve better Detection accuracy compare with SHMM. In the standard Hidden Markov Model, we have observed three fundamental problems are Evaluation and decoding another one is learning problem. The Evaluation problem can be used for word recognition. And the Decoding problem is related to constant attention and also the segmentation. In this Proposed Research, the primary purpose is to model the sequence of observation in Network log and credit card log transactions process using Enhanced Hidden Markov Model (EHMM). And show how it can be used for intrusion detection in Network. In this procedure, an EHMM is primarily trained with the conventional manners of a intruders. If the trained EHMM does not recognize an incoming Intruder transaction with adequately high probability, it is thought to be fraudulent.

Detection of application layer DDoS attacks using big data technologies

In today’s age, where data is of prime importance and rate at which data generated and sources of data generation is quite huge. The attacks on the data have increased manifold due to which its security is of prime importance. The traditional defense mechanisms are proving to be inadequate. In this paper, a new method to analyses network logs using R and Big data technologies is introduced. The approach detects fraudulent ip address in the network logs based on a set of conditions using pig scripting language. The results of using this method to detect the attack on the dataset is also presented.

Analysis in big data of satellite communication network based on machine learning algorithms

AbstractWith the development of satellite communication network technology, the amount of data generated every day and the rate of data growth are amazing. These different types of data (not necessarily structured) contain rich information. Based on this, a satellite for machine learning is proposed. First, a correlation analysis model is established, and a machine learning method is applied to the data prediction analysis of the satellite communication network. The basic idea and application process of the AdaBoost‐BP algorithm are introduced in detail. Second, combined with the data features after dimensionality reduction processing, an efficient big data analysis model is trained. Third, in order to quickly locate the abnormal behavior of users under large‐scale data, an algorithm for quantifying abnormal behaviors of users based on satellite communication network log information and quickly detecting and analyzing big data is proposed. Finally, experimental verification is performed. The experimental results verify that the method can improve the big data analysis capability of satellite communication networks.

DLME: Distributed Log Mining Using Ensemble Learning for Fault Prediction

Fault prediction problems in network systems are often manifested as very onerous for better network management. One of the effective measures is to constantly monitor and analyze the unceasing generation of network logs that capture the activities of a network. The learning algorithms are quite useful for this purpose. However, due to the dynamic nature of network systems, a frequent drift in the logged data may occur which in turn affects the efficiency of the learning algorithms. In this paper, we present a general purpose algorithmic framework for developing easily parallelizable distributed log mining approach, which uses machine learning and distributed processing to achieve a better quality of network services. Our proposed approach monotonously handles the dynamic nature of network logs by tracking the changes in the distribution of logs and takes adequate actions according to that. The entire problem is illustrated as a distributed learning environment, where the complete set of logs is partitioned into assorted data chunks and a distributed weighted ensemble of the information is generated from these chunks. Furthermore, our method is tested on real dataset and experimental analysis shows that a fair amount of scalability and accuracy can be obtained.

A Hybrid Deep Learning-Based Model for Anomaly Detection in Cloud Datacenter Networks

With the emergence of the Internet-of-Things (IoT) and seamless Internet connectivity, the need to process streaming data on real-time basis has become essential. However, the existing data stream management systems are not efficient in analyzing the network log big data for real-time anomaly detection. Further, the existing anomaly detection approaches are not proficient because they cannot be applied to networks, are computationally complex, and suffer from high false positives. Thus, in this paper a hybrid data processing model for network anomaly detection is proposed that leverages grey wolf optimization (GWO) and convolutional neural network (CNN). To enhance the capabilities of the proposed model, GWO and CNN learning approaches were enhanced with: 1) improved exploration, exploitation, and initial population generation abilities and 2) revamped dropout functionality, respectively. These extended variants are referred to as Improved-GWO (ImGWO) and Improved-CNN (ImCNN). The proposed model works in two phases for efficient network anomaly detection. In the first phase, ImGWO is used for feature selection in order to obtain an optimal trade-off between two objectives, i.e., reduced error rate and feature-set minimization. In the second phase, ImCNN is used for network anomaly classification. The efficacy of the proposed model is validated on benchmark (DARPA’98 and KDD’99) and synthetic datasets. The results obtained demonstrate that the proposed cloud-based anomaly detection model is superior in comparison to the other state-of-the-art models (used for network anomaly detection), in terms of accuracy, detection rate, false positive rate, and F-score. In average, the proposed model exhibits an overall improvement of 8.25%, 4.08%, and 3.62% in terms of detection rate, false positives, and accuracy, respectively; relative to standard GWO with CNN.

On construction of a network log management system using ELK Stack with Ceph

A log management system is essential for the networks administrator. With a log management tool, we can collect, store, analyze, archive, and finally dispose of the log information. In this paper, we propose the architecture model of a log management system using ELK Stack with Ceph to provide a safe network, good Wi-Fi signal strength, and adequate backup data mechanism. In this case, we use our campus data of Wi-Fi log and NetFlow log. First, we collect and store data of our Wi-Fi log using Filebeats tool, and then, we use Elasticsearch, Logstash, and Kibana Stack to visualize the Wi-Fi log data. Second, we collect and store our NetFlow log using NFDUMP, and then, we also use ELK Stack to visualize the NetFlow log data. Third, we integrate the Wi-Fi log and NetFlow log data in one architecture using a distributed storage Ceph file system (CephFS). Moreover, we also compare the performance of RADOS Gateway and CephFS for better storage mechanism.

Frequent Attack Sequences-based Network log Mining

As an important part of modern IT infrastructure, network provides great convenience for people to exchange information and share resources. However, network still faces with many threats such as network virus, hacker attacks, data theft and tampering and so on. Network logs includes a lot of valuable information about all behaviours happened in the network. How to analyse these network log to enhance the security of network becomes consequently the focus of many researchers. In this paper, we first design three similarity functions between two network attack records to create the network attack sequences, and then present a PrefixSpan-based frequent attack sequence mining algorithm to identify all frequent attack sequences in network log. The experimental results show that the PrefixSpan-based frequent attack sequence mining algorithm has shorter executing time and less running space than the Apriori algorithm. The PrefixSpan-based frequent attack sequence mining algorithm provides a network log analysing method for intrusion detection.

Proactive Failure Detection Learning Generation Patterns of Large-Scale Network Logs

With the growth of services in IP networks, network operators are required to perform proactive operation that quickly detects the signs of critical failures and prevents future problems. Network log data, including router syslog, are rich sources for such operations. However, it has become impossible to find genuinely important logs that lead to serious problems due to the large volume and complexity of log data. We propose a log analysis system for proactive detection of failures. Our key observation is that the abnormality of logs depends on not just the keywords in the messages (e.g. ERROR, FAIL), but generation patterns such as burstiness. Our system consists of three functions: (i) extracting log templates automatically and quickly from a massive amount of unstructured log data; (ii) constructing log feature vectors to characterize the generation patterns of logs; and (iii) using a supervised machine learning approach to associate failures with the log data that appeared before them. We validated our system using real log data collected from a large network and determined its effectiveness.