DROPSYS: Detection of ROP attacks using system information

Abstract

As modern return-oriented programming (ROP) attacks have become more sophisticated, preventing or detecting these attacks is essential for real-world exploit defense. As an alternative to many defense techniques of ROP attacks that require software modification and hardware assistance, researchers have proposed ROP defense techniques using hardware performance counters (HPCs) to mitigate concerns about additional cost and compatibility issues. However the existing HPC data-based ROP detection techniques typically suffer from low detection performance mainly because of the non-deterministic nature of HPCs.To address these issues, we propose DROPSYS, an enhanced detection of ROP attacks using system information. DROPSYS is based on the detection of the abnormal change of system information that takes place during ROP attacks. Differing from the existing techniques, DROPSYS harnesses not only HPC data, but also system utilization data to mitigate the non-deterministic nature of HPCs. Using both HPCs of processors and system utilization of operating systems makes transparent operation without requiring any modifications to the protected programs. DROPSYS uses a long short-term memory-based variational autoencoder to effectively analyze the multivariate time-series HPC data and system utilization data for better detection performance. DROPSYS also performs feature selection for low computational overhead while maintaining the attack detection performance.In our experiments with real-world ROP exploits, DROPSYS successfully detected ROP code execution in all tested programs. Evaluation results show that DROPSYS effectively captures the behaviors and effects of ROP attacks and can detect the attacks with a 0.028% false positive rate. The accuracy of DROPSYS is 95.3%, and its F1 score is 94.9%—a figure much higher than those of existing techniques that utilize only HPC data.