An Approach to Detect Remote Access Trojan in the Early Stage of Communication

Abstract

As data leakage accidents occur every year, the security of confidential information is becoming increasingly important. Remote Access Trojans (RAT), a kind of spyware, are used to invade the PC of a victim through targeted attacks. After the intrusion, the attacker can monitor and control the victim's PC remotely, to wait for an opportunity to steal the confidential information. Since it is hard to prevent the intrusion of RATs completely, preventing confidential information being leaked back to the attacker is the main issue. Various existing approaches introduce different network behaviors of RAT to construct detection systems. Unfortunately, two challenges remain: one is to detect RAT sessions as early as possible, the other is to remain a high accuracy to detect RAT sessions, while there exist normal applications whose traffic behave similarly to RATs. In this paper, we propose a novel approach to detect RAT sessions in the early stage of communication. To differentiate network behaviors between normal applications and RAT, we extract the features from the traffic of a short period of time at the beginning. Afterward, we use machine learning techniques to train the detection model, then evaluate it by K-Fold cross-validation. The results show that our approach is able to detect RAT sessions with a high accuracy. In particular, our approach achieves over 96% accuracy together with the FNR of 10% by Random Forest algorithm, which means that our approach is valid to detect RAT sessions in the early stage of communication.