Malicious DNS Tunnel Tool Recognition Using Persistent DoH Traffic Analysis

Abstract

DNS over HTTPS (DoH) protocol can mitigate the risk of privacy breaches but makes it difficult to control network security services due to the DNS traffic encryption. However, since malicious DNS tunnel tools for the DoH protocol pose network security threats, network administrators need to recognize malicious communications even after the DNS traffic encryption has become widespread. In this paper, we propose a malicious DNS tunnel tool recognition system using persistent DoH traffic analysis based on machine learning. The proposed system can accomplish continuous knowledge updates for emerging malicious DNS tunnel tools on the machine learning model. The system is based on hierarchical machine learning classification and focuses on DoH traffic analysis. The evaluation results confirm that the proposed system is able to recognize the six malicious DNS tunnel tools in total, not only well-known ones, including dns2tcp, dnscat2, and iodine, but also the emerging ones such as dnstt, tcp-over-dns, and tuns with 98.02% classification accuracy.