Alert Aggregation Research Articles

Leveraging Large Language Models for Efficient Alert Aggregation in AIOPs

Alerts are an essential tool for the detection of anomalies and ensuring the smooth operation of online service systems by promptly notifying engineers of potential issues. However, the increasing scale and complexity of IT infrastructure often result in “alert storms” during system failures, overwhelming engineers with a deluge of often correlated alerts. Therefore, effective alert aggregation is crucial in isolating root causes and accelerating failure resolution. Existing approaches typically rely on either semantic similarity or statistical methods, both of which have significant limitations, such as ignoring causal relationships or struggling to handle infrequent alerts. To overcome these drawbacks, we propose a novel two-phase alert aggregation approach. We employ temporal–spatial clustering to group alerts based on their temporal proximity and spatial attributes. In the second phase, we utilize large language models to trace the cascading effects of service failures and aggregate alerts that share the same root cause. Experimental evaluations on datasets from real-world cloud platforms demonstrate the effectiveness of our method, achieving superior performance compared to traditional aggregation techniques.

Dealing with Security Alert Flooding: Using Machine Learning for Domain-independent Alert Aggregation

Intrusion Detection Systems (IDS) secure all kinds of IT infrastructures through automatic detection of malicious activities. Unfortunately, they are known to produce large numbers of alerts that often become overwhelming for manual analysis. Therefore, aggregation methods have been developed for filtering, grouping, and correlating alerts. However, existing techniques either rely on manually defined attack scenarios or require specific alert formats, such as IDMEF that include IP addresses. This makes the application of existing aggregation methods infeasible for alerts from host-based or anomaly-based IDSs that frequently lack such network-related data. In this paper, we therefore present a domain-independent alert aggregation technique. We introduce similarity measures and merging strategies for arbitrary semi-structured alerts and alert groups. Based on these metrics and techniques we propose an incremental procedure for the generation of abstract alert patterns that enable continuous classification of incoming alerts. Evaluations show that our approach is capable of reducing the number of alert groups for human review by around \( 80\% \) and assigning attack classifiers to the groups with true positive rates of \( 80\% \) and false positive rates lower than \( 5\% \) .

An Efficient Alert Aggregation Method Based on Conditional Rough Entropy and Knowledge Granularity.

With the emergence of network security issues, various security devices that generate a large number of logs and alerts are widely used. This paper proposes an alert aggregation scheme that is based on conditional rough entropy and knowledge granularity to solve the problem of repetitive and redundant alert information in network security devices. Firstly, we use conditional rough entropy and knowledge granularity to determine the attribute weights. This method can determine the different important attributes and their weights for different types of attacks. We can calculate the similarity value of two alerts by weighting based on the results of attribute weighting. Subsequently, the sliding time window method is used to aggregate the alerts whose similarity value is larger than a threshold, which is set to reduce the redundant alerts. Finally, the proposed scheme is applied to the CIC-IDS 2018 dataset and the DARPA 98 dataset. The experimental results show that this method can effectively reduce the redundant alerts and improve the efficiency of data processing, thus providing accurate and concise data for the next stage of alert fusion and analysis.

An Improved Frequent Pattern Growth Based Approach to Intrusion Detection System Alert Aggregation

This paper introduces different approaches to intrusion detection system (IDS) alert aggregation and proposes an improved frequent pattern growth (FP-growth) algorithm for it. This approach can be divided into three parts, which are removal of noisy data, mining association rules and text similarity check. According to the experiment on Snort alarm dataset provided by an enterprise, all the association rules found by the proposed approach are valid. Therefore, compared with FP-growth algorithm, the proposed approach can increase the precision of the result and is useful for alert aggregation.

An Alert Aggregation Algorithm Based on K-means and Genetic Algorithm

The Intrusion Detection System often produces a large number of alerts, in which 90% are useless. This makes it difficult for security administrator to identify real attack alerts. Using clustering algorithm, such as K-means, DBSCAN, can efficiently cluster the similar alerts, thus greatly reducing the number of alerts that need to be processed. However, the original clustering algorithm has some shortcomings, for example, the K-means has great dependence on the initial value selection, and it is easy to fall into local optimum. Therefore, this paper proposed a new alert aggregation method based on the genetic algorithm and K-means algorithm. We use the Darpa99 dataset to test the performance of our algorithm. And the experimental results show that the algorithm can obtain good aggregation results.

An Alert Fusion Method Based on Grey Relation and Attribute Similarity Correlation

Various security devices which produce a large volume of logs and alerts have been used widely. It is such a troublesome and time-consuming task for network managers to analyze and deal with the information. This paper presented an improved alerts aggregation method based on grey correlation and attribute similarity method. We used grey correlation to ascertain the importance of alert attributes in network security, and considered it as the weight of attributes. Then we combined with the attribute similarity method and calculated the overall feature similarity in order to complete alert aggregation. Experiments results showed that this method had a strict mathematical theory basis and a higher practical value, which can effectively reduce raw alerts and reduce redundancy for alert data fusion.

Reducing the Cognitive Load on Analysts through Hamming Distance Based Alert Aggregation

Previous work introduced the idea of grouping alerts at a Hamming distance of 1 to achieve lossless alert aggregation; such aggregated meta-alerts were shown to increase alert interpretability. However, a mean of 84023 daily Snort alerts were reduced to a still formidable 14099 meta-alerts. In this work, we address this limitation by investigating several approaches that all contribute towards reducing the burden on the analyst and providing timely analysis. We explore minimizing the number of both alerts and data elements by aggregating at Hamming distances greater than 1. We show how increasing bin sizes can improve aggregation rates. And we provide a new aggregation algorithm that operates up to an order of magnitude faster at Hamming distance 1. Lastly, we demonstrate the broad applicability of this approach through empirical analysis of Windows security alerts, Snort alerts, netflow records, and DNS logs. The result is a reduction in the cognitive load on analysts by minimizing the overall number of alerts and the number of data elements that need to be reviewed in order for an analyst to evaluate the set of original alerts.

Blackout Avoidance and Energy Saving Services with Alert Aggregation

Peak electricity demands from huge number of households in a mega-city would cause contention, leading to potential blackout. This paper proposes bi-directional collaboration via a Smart Energy Monitor System (SEMS) between consumers and energy suppliers, exchanging real-time energy usage data with smart meters over the Internet and mobile devices for well-informed decisions and even predictions. The authors further propose the use of an Alert Management System (AMS) to monitor and aggregate critical energy consumption events for this purpose.

An Alert Aggregation Algorithm Based on Iterative Self-Organization

Considering the problem that intrusion detection systems always produced duplicated alarm information, in this paper we propose an iterative self-organization clustering algorithm. It begins with calculating average value of classes as the new clustering center on the basis of random selection, merging and dividing dynamically, then finish the clustering procedure through the iteration finally. Experimental results with DARPA1999 testing data set show that the clustering method is more excellent than traditional clustering methods in both aggregation rate and error aggregation rate. Besides, it reduces duplicated alarm effectively and provides assistance to further related work.

Online Intrusion Alert Aggregation with Generative Data Stream Modeling

Alert aggregation is an important subtask of intrusion detection. The goal is to identify and to cluster different alerts-produced by low-level intrusion detection systems, firewalls, etc.-belonging to a specific attack instance which has been initiated by an attacker at a certain point in time. Thus, meta-alerts can be generated for the clusters that contain all the relevant information whereas the amount of data (i.e., alerts) can be reduced substantially. Meta-alerts may then be the basis for reporting to security experts or for communication within a distributed intrusion detection system. We propose a novel technique for online alert aggregation which is based on a dynamic, probabilistic model of the current attack situation. Basically, it can be regarded as a data stream version of a maximum likelihood approach for the estimation of the model parameters. With three benchmark data sets, we demonstrate that it is possible to achieve reduction rates of up to 99.96 percent while the number of missing meta-alerts is extremely low. In addition, meta-alerts are generated with a delay of typically only a few seconds after observing the first alert belonging to a new attack instance.

Reducing false positives in anomaly detectors through fuzzy alert aggregation

In this paper we focus on the aggregation of IDS alerts, an important component of the alert fusion process. We exploit fuzzy measures and fuzzy sets to design simple and robust alert aggregation algorithms. Exploiting fuzzy sets, we are able to robustly state whether or not two alerts are “close in time”, dealing with noisy and delayed detections. A performance metric for the evaluation of fusion systems is also proposed. Finally, we evaluate the fusion method with alert streams from anomaly-based IDS.

Alert aggregation based on IDMEF and category

How to aggregate and reduce duplicated alerts from different Intrusion Detection System (IDS) is one of the most important problems in distributed IDS research area. The article proposed a type of Intrusion Detection Message Exchange Format (IDMEF) alert specification extending the alert class structure of IDMEF data model to universally define the format of the alerts originating from different IDS. In order to efficiently aggregate this new type of IDMEF alerts, the article proposed an improvement to the original algorithm that was based on category and similarity. The prototype of alert aggregation module and IDMEF alert generation module has been implemented on the Snort platform. Experiments on DARPA99 data sets demonstrate the effectiveness of our method.

Alert aggregation algorithm based on genetic clustering algorithm

An alert aggregation algorithm was proposed,in which alerts of Intrusion Detection System(IDS)based on genetic clustering algorithm were aggregated.To convert the dissimilarity between two alert attributes into the value range of [0.0,1.0],the dissimilarity of two alerts was represented by using a dissimilarity matrix;the more excellent clustering centers were chosen by the genetic algorithm,and the similar alerts would be clustered according to the dissimilarity matrix.Hierarchical aggregation method was adopted for every kind of alerts to aggregate according to the results of clustering.The experimental results show that the repeat alerts can be decreased effectively by this method.

The Implementation of Alert Aggregation and Dataset Testing

The Implementation of Alert Aggregation and Dataset Testing

A Survey of Intrusion-Detection Alert Aggregation and Correlation Techniques

A Survey of Intrusion-Detection Alert Aggregation and Correlation Techniques

TRINETR: An architecture for collaborative intrusion detection and knowledge-based alert evaluation

Current reactive and standalone network security products are not capable of withstanding the onslaught of diversified network threats. As a result, a new security paradigm, where integrated security devices or systems collaborate closely to achieve enhanced protection and provide multi-layer defenses is emerging. In this paper, we present the design of a collaborative architecture for multiple intrusion detection systems to work together to detect real-time network intrusions. The detection is made more efficient and effective by using collaborative intelligent agents, relevant knowledge base and combination of multiple detection sensors. The architecture is composed of three parts: Collaborative Alert Aggregation, Knowledge-based Alert Evaluation and Alert Correlation. The architecture is aimed at reducing the alert overload by correlating results from multiple sensors to generate condensed views, reducing false positives by integrating network and host system information into the evaluation process and correlating events based on logical relations to generate global and synthesized alert report. The architecture is designed as a layer above intrusion detection for post-detection alert analysis and security actions. The first two parts of the architecture have been implemented and the implementation results are presented in this paper.

혼합형 침입 탐지 시스템에서 데이터 및 정책 전달 통신 모델과 성능 평가

침입 탐지 시스템에 대하여 많은 연구 노력들이 진행되고 있다. 그러나 침입 탐지 시스템의 모델과 성능 평가에 대한 작업은 거의 찾아 볼 수 없다. 본 논문에서는 지역적인 침입 탐지를 위한 에이전트들과 전역적인 침입 탐지를 위한 집중 데이터 분석 컴포넌트를 가지고 있는 다중 도메인 환경에서 혼합 침입 탐지를 위한 통신 프레임워크를 제안한다. 또한 전체적인 프레임워크에서 호스트 기반과 네트워크 기반 침입 탐지 시스템의 결합을 가정한다. 지역 도메인에서 경보와 로그 데이터 같은 정보 집합은 상위 레벨로 보고 된다. 계위의 루트에는 데이터 합동을 수행하는 전역 매니저가 있다. 전역 매니저는 침입 탐지 경보의 집합과 상호관련의 결과로 보안 정책을 하위 레벨로 전달하게 된다. 본 논문에서는 혼합 침입 탐지 시스템을 위한 통심 메커니즘을 모델링하고 데이터 및 정책 전달을 위한 전송 능력의 성능 평가를 위하여 OPNET 모델러를 이용한 시뮬레이터를 개발한다. 여러 가지 시나리오에 기반하여 통신 지연에 초점을 두고 모의실험 결과를 제시하고 비교한다. Much research efforts are being exerted for the study of intrusion detection system(IDS). However little work has been for the communication medels and performance eveluation of the IDS. Here we present a communication framework for doing hybrid intrusion detection in which agents are used for local intrusion detections with a centralized data anaysis componenta for a global intrusion detection at multiple domains environment. We also assume the combination of host-based and network-based intrusion detection systems in the oberall framework. From the local domain, a set of information such as alert, and / or log data are reported to the upper level. At the root of the hierarchy, there is a global manager where data coalescing is performed. The global manager delivers a security policy to its lower levels as the result of aggregation and correlation of intrusion detection alerts. In this paper, we model the communication mechanisms for the hybrid IDS and develop a simular using OPNET modeller for the performance evaluation of transmission capabillities for the delivery of data and policy. We present and compare simulation results based on several scenarios focuding on communication delay.