Sneaking Through Security: Mutating Live Network Traffic to Evade Learning-Based NIDS

Abstract

Machine learning based network intrusion system (NIDS) is known to be vulnerable to evasions. Attackers conceal intrusion activities to make them undetected. Researching evasion techniques contributes to evaluating and increasing the robustness of NIDS. Previous evasion approaches modify feature values or packets of an offline network trace as a whole. However, in real scenarios, attackers are constrained to manipulate only outbound packets on the fly. To bridge this assumption gap, we present the first evasion solution for live network traffic against learning based NIDSs. The solution consists of three components: a devised Kalman filter based algorithm to predicate the feature values of live flows, a set of formally constructed atomic packet mutation operators, and a proposed Strength Enhanced Deep Q-learning (SE-DQN) to determine effective mutation operators on outbound packets according to the predicted features. A defense scheme based on adaptive decision threshold adjustment is also provided. Experimental evaluation is presented on various NIDS classifiers and cyber attacks. Results show that SE-DQN achieves an evasion rate of at least 64.2% on most classifiers and even more than 90% on certain ones, and it is three times faster than DQN on learning mutation policy. The defense scheme shows an improvement of at least 76.4% on recall measurement.