Cloud security in the age of adaptive adversaries: A game theoretic approach to hypervisor-based intrusion detection

Abstract

Recent advancements in cloud computing have underscored the critical need for robust security mechanisms to counter evolving cyber-threats. Traditional security solutions such as Intrusion Detection Systems (IDSs) often fall short due to their inability to anticipate the strategies of adaptive cyber adversaries. Game theory is considered a popular analytical tool for understanding the strategic interactions between defenders and adversaries, providing a more informed decision-making process. However, existing game-theoretic IDSs often employ non-comprehensive utility functions with limited parameters that fail to capture the complexity of real-world dynamics. This paper introduces a novel Game-Theoretic Hypervisor-based IDS (GHyIDS), which employs comprehensive utility functions and an innovative belief update model to enhance detection accuracy and adaptability in dynamic cloud environments. To overcome the limitations of existing models, we design comprehensive utility functions by incorporating a wider range of real-world parameters, such as trust score, risk, vulnerability, damage severity, worth of the VM, means, opportunities, and access available to the attacker, as well as success rates of attack detection and execution. We propose a Resource-Aware Static Intrusion Detection Bayesian Game (S-IDBG) and extend it into a Dynamic Multi-Stage IDBG (D-IDBG), enabling the system to dynamically adapt to changes in attack patterns and system vulnerabilities. The belief update model is pivotal in continuously refining the system’s strategies based on observed behaviors and outcomes, allowing for precise adjustments to the evolving threats. Our experimental results show a significant improvement over existing models, with our approach achieving approximately 10% increase in detection rate, 20% reduction in false positive rate and 10% reduction in false negative rate in comparative analysis against state-of-the-art models namely, the trust-based Maxmin game and the repeated Bayesian Stackelberg game.