Provenance Graphs Research Articles

A dynamic provenance graph-based detector for advanced persistent threats

Advanced Persistent Threats (APTs) pose a major cyber threat due to their stealthy, long-term nature and intricate complexity, making them particularly challenging to detect. Provenance graphs map interactions between system entities as directed, heterogeneous networks, offering rich semantic information valuable for threat identification. However, most existing approaches rely on static graph analysis, overlooking the critical dynamics of evolving threats. Dynamic graph analysis offers the potential to capture both temporal and structural insights, but current methods focus on single-perspective learning, failing to fully exploit the inherent relationships and evolving patterns within the data. To address this gap, we propose CGL-AD, a Contextualized Graph Learning APT Detector. CGL-AD leverages temporal graph learning to capture subtle temporal changes and overall structural transformations over time. It then integrates hash-based data stream frequency estimation techniques to identify local topological alterations, subsequently feeding these rich embeddings into a powerful sequence learning model. Experimental results on three widely used datasets: Streamspot, Camflow-apt and Shellshock datasets demonstrate that CGL-AD significantly outperforms existing methods. Specifically, CGL-AD outperforms the best baseline FLASH on these datasets by 1.5%, 3.7%, and 5.6% respectively in terms of Receiver Operating Characteristic -Area Under the Curve (ROC-AUC), effectively revealing hidden APT attack patterns in dynamic provenance graphs.

Unveiling scientific articles from paper mills with provenance analysis.

The increasing prevalence of fake publications created by paper mills poses a significant challenge to maintaining scientific integrity. While integrity analysts typically rely on textual and visual clues to identify fake articles, determining which papers merit further investigation can be akin to searching for a needle in a haystack, as these fake publications have non-related authors and are published on non-related venues. To address this challenge, we developed a new methodology for provenance analysis, which automatically tracks and groups suspicious figures and documents. Our approach groups manuscripts from the same paper mill by analyzing their figures and identifying duplicated and manipulated regions. These regions are linked and organized in a provenance graph, providing evidence of systematic production. We tested our solution on a paper mill dataset of hundreds of documents and also on a larger version of the dataset that deliberately included thousands of documents intentionally selected to distract our method. Our approach successfully identified and linked systematically produced articles on both datasets by pinpointing the figures they reused and manipulated from one another. The technique herein proposed offers a promising solution to identify fraudulent manuscripts, and it could be a valuable tool for supporting scientific integrity.

Provenance-based APT campaigns detection via masked graph representation learning

Advanced Persistent Threats (APTs) are well-planned, persistent, and highly stealthy cyberattacks designed to steal confidential information or disrupt specific target systems. Recent studies have used system audit logs to construct provenance graphs that describe system interactions to detect potentially malicious activities. Although they are effective, they still suffer from problems such as the need for a priori knowledge, lack of attack data, and high computational overhead that limit their application. In this paper, we propose a self-supervised learning-based APT detection model, APT-MGL, which learns the embedded representations of nodes through a graph mask self-encoder and transforms the detection problem into an outlier detection problem for malicious nodes. APT-MGL characterizes the behavior of nodes based on node type, action, and interaction frequency, and fuses the features through a multi-head self-attention mechanism. Then the node embedding is obtained by combining graph features and structural information using masked graph representation learning. Finally, the unsupervised outlier detection method is used to analyze the computed embeddings and obtain the final detection results. The experimental results show that APT-MGL outperforms existing monitoring models and achieves a small overhead.

RT-APT: A real-time APT anomaly detection method for large-scale provenance graph

Advanced Persistent Threats (APTs) are prevalent in the field of cyber attacks, where attackers employ advanced techniques to control targets and exfiltrate data without being detected by the system. Existing APT detection methods heavily rely on expert rules or specific training scenarios, resulting in the lack of both generality and reliability. Therefore, this paper proposes a novel real-time APT attack anomaly detection system for large-scale provenance graphs, named RT-APT. Firstly, a provenance graph is constructed with kernel logs, and the WL subtree kernel algorithm is utilized to aggregate contextual information of nodes in the provenance graph. In this way we obtain vector representations. Secondly, the FlexSketch algorithm transforms the streaming provenance graph into a sequence of feature vectors. Finally, the K-means clustering algorithm is performed on benign feature vector sequences, where each cluster represents a different system state. Thus, we can identify abnormal behaviors during system execution. Therefore RT-APT enables to detect unknown attacks and extract long-term system behaviors. Experiments have been carried out to explore the optimal parameter settings under which RT-APT can perform best. In addition, we compare RT-APT and the state-of-the-art approaches on three datasets, Laboratory, StreamSpot and Unicorn. Results demonstrate that our proposed method outperforms the state-of-the-art approaches from the perspective of runtime performance, memory overhead and CPU usage.

ProcSAGE: an efficient host threat detection method based on graph representation learning

Advanced Persistent Threats (APTs) achieves internal networks penetration through multiple methods, making it difficult to detect attack clues solely through boundary defense measures. To address this challenge, some research has proposed threat detection methods based on provenance graphs, which leverage entity relationships such as processes, files, and sockets found in host audit logs. However, these methods are generally inefficient, especially when faced with massive audit logs and the computational resource-intensive nature of graph algorithms. Effectively and economically extracting APT attack clues from massive system audit logs remains a significant challenge. To tackle this problem, this paper introduces the ProcSAGE method, which detects threats based on abnormal behavior patterns, offering high accuracy, low cost, and independence from expert knowledge. ProcSAGE focuses on processes or threads in host audit logs during the graph construction phase to effectively control the scale of provenance graphs and reduce performance overhead. Additionally, in the feature extraction phase, ProcSAGE considers information about the processes or threads themselves and their neighboring nodes to accurately characterize them and enhance model accuracy. In order to verify the effectiveness of the ProcSAGE method, this study conducted a comprehensive evaluation on the StreamSpot dataset. The experimental results show that the ProcSAGE method can significantly reduce the time and memory consumption in the threat detection process while improving the accuracy, and the optimization effect becomes more significant as the data size expands.

Using provenance and replay for qualitative analysis of gameplay sessions

There is an increasing interest to use game telemetry for analyzing gameplay sessions, with numerous techniques created to help game developers analyze different game aspects like game balancing and behavioral analysis. Among different gameplay session analysis techniques, the collection of provenance data has stood out due to a crucial advantage of this approach: the possibility to identify cause–effect relationships between game events. In previous work, we presented our conceptual framework called Prov-Replay, which provides a replay synchronized with an interactive provenance graph visualization. We validate Prov-Replay by creating PinGU Replay, a tool that implements our conceptual framework and applied it in a commercial game. Due to the promising results, this paper extends our previous work by presenting a detailed overview about Prov-Replay implementation, introducing a new feature that provides an analysis dashboard, and applying our experiment methodology to a new commercial game. We also enhance the concept of analytics related to provenance through the replay pipeline. Finally, we made PinGU Replay available as open-source software. Our new experiment results reinforce that, when using our conceptual framework fundamentals, it is possible to improve the efficiency and effectiveness of qualitative analysis process.

Game provenance graph-based representation learning vs metrics-based machine learning: An empirical comparison on predictive game analytics tasks

Predicting events and behavior in games is a crucial task of game analytics, both during gameplay, such as predicting wins, and after, like forecasting player churn. Game metrics have been widely adopted due to their human-readable and task-oriented nature. However, these metrics vary between games and require expert knowledge. Game analytics has evolved with techniques that primarily handle non-tabular and sequential data, like graphs and time series. However, existing methods still depend on traditional game metrics from extensive feature engineering. In parallel, graph representation learning has been applied to game provenance graphs — structures that log game events as nodes and link them through causal relationships. This approach may allow analysts to bypass traditional game metrics and learn relevant downstream features during training. This paper compares this metric-less graph-based representation learning with traditional metric-based machine learning in two predictive game analytics tasks: win prediction and hit prediction. Our results show that the same graph-based representation learning architecture achieves competitive results in win prediction and outperforms metric-based approaches in hit prediction, suggesting its suitability for predictive game analytics tasks. This method could significantly benefit as it does not require game-specific metric expertise, and its learned features are versatile across multiple tasks.

T-Trace: Constructing the APTs Provenance Graphs Through Multiple Syslogs Correlation

Advanced Persistent Threats (APTs) employ sophisticated and covert tactics to infiltrate target systems, leading to increased vulnerability and an elevated risk of exposure. Consequently, it is essential for us to proactively create an extensive and clearly outlined attack chain for APTs in order to effectively combat these threats. Unlike traditional malware or application threats, APTs can sidestep cyber security efforts and cause severe damage to organizations or even state security. Nonetheless, earlier methods struggle to accurately track APTs and may face a dependency explosion issue, as identifying the intricate and complex unknown malicious activities within APTs proves to be challenging. In this paper, we propose and build an approach, T-trace, which constructs the events provenance graphs by analyzing the correlations among logs. The approach precisely finds the log communities with tensor decomposition and calculates significance scores to extract the events. The APTs can be inferred by discovering the event communities and constructing the provenance graph with log correlation. In the experiment, we used DARPA data sets and launched four current practical APTs. Compared with current approaches, the results show that T-trace can efficiently reduce time cost by 90% and achieve a 92% accuracy rate in constructing the provenance graph, which can be practically applied in APTs provenance.

ConGraph: Advanced Persistent Threat Detection Method Based on Provenance Graph Combined with Process Context in Cyber-Physical System Environment

With the wide use of Cyber-Physical Systems (CPS) in many applications, targets of advanced persistent threats (APTs) have been extended to the IoT and industrial control systems. Provenance graph analysis based on system audit logs has become a promising way for APT detection and investigation. However, we cannot afford to ignore that existing provenance-based APT detection systems lack the process–context information at system runtime, which seriously limits detection performance. In this paper, we proposed ConGraph, an approach for detecting APT attacks using provenance graphs combined with process context; we presented a module for collecting process context to detect APT attacks. This module collects file access behavior, network access behavior, and interactive relationship features of processes to enrich semantic information of the provenance graph. It was the first time that the provenance graph was combined with multiple process–context information to improve the detection performance of APT attacks. ConGraph extracts process activity features from the provenance graphs and submits the features to a CNN-BiLSTM model to detect underlying APT activities. Compared to some state-of-the-art models, our model raised the average precision rate, recall rate, and F-1 score by 13.12%, 25.61%, and 24.28%, respectively.

Prov-Dominoes: An approach for knowledge discovery from provenance data

Provenance has become increasingly relevant to understanding, auditing, and reproducing computational tasks. The provenance analysis processes can often be overwhelming to the user due to the large volume of data, the multiple relationships among data, and the implicit information buried in the data. Existing provenance analysis tools use either visual exploration (which is overwhelming for large provenance graphs) or do not support the exploration of implicit provenance data, such as the inferences of the PROV Data Model Constraints. To fill in this gap, we introduce Prov-Dominoes, a tool designed to interactively enable knowledge discovery on provenance data. Prov-Dominoes promotes the provenance relationships among entities, activities, and agents into first-class elements represented by domino tiles. It allows users to combine and compose such domino tiles visually and interactively, using GPU. The benefits of Prov-Dominoes are three-fold: first, it uses matrices to display provenance data, which is more compact than graphs; second, it allows users to easily explore implicit information; third, it is capable of efficiently processing large datasets using GPUs. We evaluated Prov-Dominoes over distinct case studies, allowing the observation of Prov-Dominoes in action. We also evaluated the performance of sequential combinations executed in Prov-Dominoes when dealing with provenance data with thousands of relations, contrasting their executions in GPU and CPU. The results showed that, for a large dataset, GPU was more than a hundred times faster than CPU.

A PT-based approach to construct efficient provenance graph for threat alert investigation

The Provenance Graph, constructed based on audit logs, encapsulates a wealth of causal relationships. When existing threat detection software issues an Event of Alert (EOA), security analysts conduct threat alert investigations based on the provenance graph. However, a provenance graph comprising tens of thousands of nodes and edges can lead to substantial time overhead. Moreover, coarse-grained audit logs can cause a dependency explosion in the provenance graph, thereby generating false causal relationships and interfering with the accuracy of the results. Hardware-assisted Processor Tracing (PT) can address the problem of false causal relationships caused by dependency explosion, by extracting the execution trace of a process and restoring instruction-level data flow. Leveraging PT features, we propose an efficient scheme, termed as GETSUS, which can reduce the size of the provenance graph while fully preserving the event sequences leading to the EOA. Empirical evidence shows that GETSUS can shrink a large provenance graph (approximately 200,000 edges) to a small one (around 50 edges). Compared to other schemes, the partial provenance graph output by GETSUS can fully retain the semantics related to the EOA, while improving the reduction efficiency by about tenfold.

Detecting Malicious Websites from the Perspective of System Provenance Analysis

Malicious websites are considered one of the top threats to the modern Internet. Thus, it is critical to effectively detect malicious websites for the security of the Internet. Conventional technologies typically rely on URL blacklists, or static and dynamic code analysis, which are known to have limitations. In order to effectively detect malicious websites, in this paper, we study malicious websites from the perspective of system provenance analysis for the first time. We first conduct a systematic feature engineering study on thousands of benign and malicious websites from the perspective of system provenance data. In our study, we discover eight useful features for malicious website detection. Based on these eight features, we propose ProvWeb, a novel non-intrusive system provenance-based tool, for malicious website detection. In our evaluation, ProvWeb can achieve an F1 score of 93.7% ∼ 99.7% for the four combinations of browsers and OSes (Windows Chrome, Windows Firefox, Linux Chrome, Linux Firefox). This result confirms that the features discovered in provenance graphs are effective in detecting malicious websites.

FPGAA: A Multi-Feature Provenance Graph for the Accurate Alert System

FPGAA: A Multi-Feature Provenance Graph for the Accurate Alert System

APT-KGL: An Intelligent APT Detection System Based on Threat Knowledge and Heterogeneous Provenance Graph Learning

APTs (Advanced Persistent Threats) have caused serious security threats worldwide. Most existing APT detection systems are implemented based on sophisticated forensic analysis rules. However, the design of these rules requires in-depth domain knowledge and the rules lack generalization ability. On the other hand, deep learning technique could automatically create detection model from training samples with little domain knowledge. However, due to the persistence, stealth, and diversity of APT attacks, deep learning technique suffers from a series of problems including difficulties of capturing contextual information, low scalability, dynamic evolving of training samples, and scarcity of training samples. Aiming at these problems, this paper proposes APT-KGL, an intelligent APT detection system based on provenance data and graph neural networks. First, APT-KGL models the system entities and their contextual information in the provenance data by a HPG (Heterogeneous Provenance Graph), and learns a semantic vector representation for each system entity in the HPG in an offline way. Then, APT-KGL performs online APT detection by sampling a small local graph from the HPG and classifying the key system entities as malicious or benign. In addition, to conquer the difficulty of collecting training samples of APT attacks, APT-KGL creates virtual APT training samples from open threat knowledge in a semi-automatic way. We conducted a series of experiments on two provenance datasets with simulated APT attacks. The experiment results show that APT-KGL outperforms other current deep learning based models, and has competitive performance against state-of-the-art rule-based APT detection systems.

ProvGRP: A Context-Aware Provenance Graph Reduction and Partition Approach for Facilitating Attack Investigation

Attack investigation is a crucial technique in proactively defending against sophisticated attacks. Its purpose is to identify attack entry points and previously unknown attack traces through comprehensive analysis of audit data. However, a major challenge arises from the vast and redundant nature of audit logs, making attack investigation difficult and prohibitively expensive. To address this challenge, various technologies have been proposed to reduce audit data, facilitating efficient analysis. However, most of these techniques rely on defined templates without considering the rich context information of events. Moreover, these methods fail to remove false dependencies caused by the coarse-grained nature of logs. To address these limitations, this paper proposes a context-aware provenance graph reduction and partition approach for facilitating attack investigation named ProvGRP. Specifically, three features are proposed to determine whether system events are the same behavior from multiple dimensions. Based on the insight that information paths belonging to the same high-level behavior share similar information flow patterns, ProvGRP generates information paths containing context, and identifies and merges paths that share similar flow patterns. Experimental results show that ProvGRP can efficiently reduce provenance graphs with minimal loss of crucial information, thereby facilitating attack investigation in terms of runtime and results.

System-level data management for endpoint advanced persistent threat detection: Issues, challenges and trends

Advanced persistent threat (APT) attacks pose significant security threats to governments and large enterprises. Endpoint detection and response (EDR) methods, which are standard solutions to combat APT attacks, can efficaciously respond to associated security threats by leveraging the semantic richness of provenance graphs and clear causality relations to resist illegal tampering. However, the large number of audit logs produced over time, which provide key supporting information for EDR methods, lead to substantial computational overhead and increased storage costs. Therefore, a robust data management framework must be developed. However, most existing reviews discuss data collection, compression, and storage methods independently. Due to the lack of a comprehensive, structured survey of data management strategies, current data management analyses tend to be separated into individual modules, making it difficult to obtain prompt and precise guidance for higher-level security analysis tasks from these analyses. In this paper, a comprehensive and structured survey of data management strategies based on provenance graphs is conducted, the core ideas of the mainstream approaches to each aspect of data management are summarized, and existing approaches are systematically classified and compared. Then, the problems with individual data management modules are investigated, and potential complementary and collaborative strategies are examined based on the insights and challenges of existing work as a basis for recommending best practices for practical deployment. Finally, an ideal data management framework is described to guide future research.

E-Audit: Distinguishing and investigating suspicious events for APTs attack detection

To detect Advanced Persistent Threats (APTs), recent research efforts focus on modeling the common attack kill chain. The provenance graph is one of the proven techniques which maintains a long-term historic correlation of the attack stages to model the kill chain. However, it is overwhelming to model the kill chain by inspecting the high volume of system events. The paper proposes E-Audit, a hybrid (event-correlation and classification) approach to model the APT kill chain based on threat-likely events that are suspicious events on the system. E-Audit first distinguishes the threat-likely events and builds a provenance graph for threat-likely events only. Second, it assesses the APT campaigns, identifies the attack-irrelevant campaigns, and removes the corresponding paths from the graph. The novel sequence-based model assesses the APT campaigns as a sequence to identify the attack-relevant and attack-irrelevant campaigns. We evaluated the effectiveness of E-Audit over the audit logs dataset. We validated it in real-time using audit logs generated over the enterprise network set-up. The experimental results show that E-Audit can distinguish 668 threat-likely events from the 183223 system events in 174 ms on average and effectively detect the APT attacks with 99.71% accuracy and generate the kill chain.

OneProvenance: Efficient Extraction of Dynamic Coarse-Grained Provenance from Database Query Event Logs

Provenance encodes information that connects datasets, their generation workflows, and associated metadata (e.g., who or when executed a query). As such, it is instrumental for a wide range of critical governance applications (e.g., observability and auditing). Unfortunately, in the context of database systems, extracting coarse-grained provenance is a long-standing problem due to the complexity and sheer volume of database workflows. Provenance extraction from query event logs has been recently proposed as favorable because, in principle, can result in meaningful provenance graphs for provenance applications. Current approaches, however, (a) add substantial overhead to the database and provenance extraction workflows and (b) extract provenance that is noisy, omits query execution dependencies, and is not rich enough for upstream applications. To address these problems, we introduce OneProvenance: an efficient provenance extraction system from query event logs. OneProvenance addresses the unique challenges of log-based extraction by (a) identifying query execution dependencies through efficient log analysis, (b) extracting provenance through novel event transformations that account for query dependencies, and (c) introducing effective filtering optimizations. Our thorough experimental analysis shows that OneProvenance can improve extraction by up to ~18X compared to state-of-the-art baselines; our optimizations reduce the extraction noise and optimize performance even further. OneProvenance is deployed at scale by Microsoft Purview and actively supports customer provenance extraction needs (https://bit.ly/3N2JVGF).

ProvLink-IoT: A novel provenance model for Link-Layer Forensics in IoT networks

Provenance has been instrumental in networked systems to solve issues related to data trustworthiness, network diagnostics, security, and forensic analysis. Existing provenance based solutions for the Internet of Things (IoT) are either device-centric or platform-centric, which address only application layer attacks. Whereas any stealthy attacks performed at MAC, network, and other sub-layers which alter the behavior of IoT devices often go undetected. Provenance based forensic solutions offer effective mechanisms for investigating link and network layer attacks in IoT networks. Realizing this potential, we propose ProvLink-IoT, a novel provenance model for link-layer forensic analysis in IoT networks. ProvLink-IoT employs PROV-DM and PROV-TEMPLATE standards to model the provenance of the network. Provenance graphs are generated under both benign and attack scenarios based on the provenance logs and network traffic collected from the network. ProvLink-IoT is implemented in a simulated environment with a case study on the 6TiSCH protocol stack. We implemented three link-layer attacks on TSCH and the 6top layers of the 6TiSCH network to study their impact and perform forensic analysis. Link-IoT, a comprehensive link-layer dataset, is generated from the network provenance, which can be used in the further incident and forensic analysis. The performance impact of ProvLink-IoT on IoT network is analyzed in terms of provenance growth rate and storage overhead. Experimental results showed the efficacy of the proposed solution in correlating evidence during incident analysis and its relevance to real-time scenarios.

Visualizing the Scripts of Data Wrangling With Somnus.

Data workers use various scripting languages for data transformation, such as SAS, R, and Python. However, understanding intricate code pieces requires advanced programming skills, which hinders data workers from grasping the idea of data transformation at ease. Program visualization is beneficial for debugging and education and has the potential to illustrate transformations intuitively and interactively. In this article, we explore visualization design for demonstrating the semantics of code pieces in the context of data transformation. First, to depict individual data transformations, we structure a design space by two primary dimensions, i.e., key parameters to encode and possible visual channels to be mapped. Then, we derive a collection of 23 glyphs that visualize the semantics of transformations. Next, we design a pipeline, named Somnus, that provides an overview of the creation and evolution of data tables using a provenance graph. At the same time, it allows detailed investigation of individual transformations. User feedback on Somnus is positive. Our study participants achieved better accuracy with less time using Somnus, and preferred it over carefully-crafted textual description. Further, we provide two example applications to demonstrate the utility and versatility of Somnus.