Analysis of Encrypted Network Traffic for Enhancing Cyber-security in Dynamic Environments

Abstract

ABSTRACT At present, encrypted data is the cornerstone of Internet communication, providing the maximum degree of privacy and security protection for all transmitted data while shielding users against potential cyber threats and attacks. However, since the Deep Packet Inspection (DPI) system is the primary layer of defense against numerous cyberattacks, applying encrypted network data poses severe issues for detection and prevention systems. In dynamic contexts such as the Internet of Things (IoT), detecting intrusion inside encrypted network traffic is vital. Yet, it is equally important to predict and prevent any cyber-attacks that may compromise the integrity and security of the network infrastructure. As a result, there is a fundamental need for methodologies based on intelligent analysis of patterns and attributes of encrypted network traffic. To satisfy security requirements in such a context, we propose an application of deep learning models for enhanced intrusion detection systems (IDS). The Tree-based Spider-Net Multipath (TBSNM) methodology is utilized, while an Advanced Encryption Standard (AES) technique is used to authenticate users. User selection is accomplished through robust Deep Reinforcement Learning with the Tabu Search (DRL-TS) algorithm, while channel selection is optimized through rigorous training employing Proximal Policy Optimization (PPO). Path selection is then determined by analyzing traffic statistics extracted from the Routing Information Protocol (RIP). Finally, an optimized IDS is established based on a Lightweight Deep Neural Network with Hunger Games Search and Remora Optimization Algorithm (LDNN-HGS-ROA). Evaluation results have shown that the proposed system architecture is effective in detecting attacks, achieving an enhanced IDS architecture with higher performance rates.