Remote access trojan traffic early detection method based on Markov matrices and deep learning

Abstract

Remote Access Trojan (RAT) allows the attacker to gain remote control of an infected system and steal data from it. Due to over-reliance on expert experience and statistical features, most feature-based RAT detection methods perform inefficiently and can only achieve high accuracy with network traffic collected over a long period of time. Byte-based RAT detection methods often use sequence truncation to process network traffic to meet the uniform-sized input requirements of convolutional neural networks (CNNs). However, the sequence truncation brings a negative impact on detection accuracy for some information loss. Towards effective and efficient RAT detection, we define the early stage from the damage degree caused on victim hosts and propose a new RAT traffic early detection method based on Markov matrices and deep learning (RATMD). In RATMD, the byte sequence of the TCP payloads of the TCP service packets collected in the early stage is represented by a byte transition probability matrix with a fixed size of 256 × 256, and the generated matrices are used to construct a detection model using a CNN architecture. Experiments are conducted on the network traffic of 58 benign applications and 61 RATs. Employing only the byte sequences derived from the TCP payloads of the TCP service packets in the traffic between the first TCP connection established by the application client and the third TCP service packet sent by the application server, RATMD achieves a detection accuracy of 95.5%.