Making vulnerability prediction more practical: Prediction, categorization, and localization

Abstract

Context:Due to the prevalence of software vulnerabilities, vulnerability detection becomes a fundamental problem in system security. Objective:To solve this problem, academics and industries have made great efforts to propose deep-learning-based (DL-based) approaches but these attempts have three main limitations: (1) perform poorly on real-world projects (e.g., Accuracy below 74.33% and F1 below 73.55%); (2) perform poorly in catching vulnerable patterns due to incomplete code representations; (3) mostly perform coarse-grained function-level prediction and lack interpretability analysis. Methods:In this paper, we propose VulPCL, a BLSTM and CodeBERT based approach, which makes the first attempt to perform vulnerability prediction, categorization, and localization automatically within a framework. To alleviate the above-mentioned limitations, our VulPCL considers multi-dimension (i.e., text-based, sequence-based, and graph-based) representations to catch latent vulnerable patterns and multi-model training to learn high-level semantics. Results:Through experiments on four real-world datasets containing 114+ CWE (Common Weakness Enumeration) types spanning from 2005 to 2022, we find that our VulPCL outperforms the baselines by (1) 13.51%∼60.64% and 14.34%∼180.23% on Accuracy, and F1 respectively on vulnerability prediction; (2) 10.32%∼46.79%, and 10.71%∼127.80% on Accuracy, and macro-F1 respectively on vulnerability categorization; (3) 9.23%∼36.54% on Top-10 Accuracy on vulnerability localization. Conclusion:These results indicate that our VulPCL is considerably more accurate, effective, fine-grained, and practical than previous studies. Besides, our further analyses show that VulPCL is indeed capable of capturing all vulnerability lines, and the result of line-level vulnerability localization is consistent with the function-level vulnerability prediction as the increase of predicted lines. Thus making VulPCL more interpretable than previous studies. Our additional investigation also shows that VulPCL effectively detects the Most Dangerous 25 CWEs in 2022, which is instructive for security researchers.