DTRAB: Combating Against Attacks on Encrypted Protocols Through Traffic-Feature Analysis

Abstract

The unbridled growth of the Internet and the network-based applications has contributed to enormous security leaks. Even the cryptographic protocols, which are used to provide secure communication, are often targeted by diverse attacks. Intrusion detection systems (IDSs) are often employed to monitor network traffic and host activities that may lead to unauthorized accesses and attacks against vulnerable services. Most of the conventional misuse-based and anomaly-based IDSs are ineffective against attacks targeted at encrypted protocols since they heavily rely on inspecting the payload contents. To combat against attacks on encrypted protocols, we propose an anomaly-based detection system by using strategically distributed monitoring stubs (MSs). We have categorized various attacks against cryptographic protocols. The MSs, by sniffing the encrypted traffic, extract features for detecting these attacks and construct normal usage behavior profiles. Upon detecting suspicious activities due to the deviations from these normal profiles, the MSs notify the victim servers, which may then take necessary actions. In addition to detecting attacks, the MSs can also trace back the originating network of the attack. We call our unique approach DTRAB since it focuses on both Detection and TRAceBack in the MS level. The effectiveness of the proposed detection and traceback methods are verified through extensive simulations and Internet datasets.