VINCENT: Cyber-threat detection through vision transformers and knowledge distillation

Abstract

Vision Transformers (ViTs) denote a family of attention-based deep learning techniques that have recently achieved amazing results in various problems related to the field of computer vision. In this paper, we explore the use of ViTs in problems of cyber-threat detection related to malware and network intrusion detection. In particular, we propose VINCENT, that is a novel deep neural method, which resorts to a color imagery representation of cyber-data by encoding related cyber-data features into neighboring color pixels. ViTs are trained from cyber-data images as teacher models, to extract explainable imagery signatures of cyber-data classes. This knowledge is extracted by leveraging the self-attention mechanism to give paired attention values between pairs of imagery patches. The signature knowledge, extracted through the ViT teacher, is, finally, used to train a smaller neural student model according to the knowledge distillation theory. Experiments with various benchmark cybersecurity datasets assess the accuracy of the student model VINCENT also compared to that of several state-of-the-art methods. In addition, it shows that VINCENT can obtain insights from explanations recovered through the self-attention mechanism of the ViT teacher.