Vulnerability datasets for software security: A survey of existing resources, challenges, and future directions

Software security is a very important aspect for software development organizations who wish to provide high-quality and dependable software to their consumers. A crucial part of software security is the early detection of software vulnerabilities. Vulnerability prediction is a mechanism that facilitates the identification (and, in turn, the mitigation) of vulnerabilities early enough during the software development cycle. The scientific community has recently focused a lot of attention on developing Deep Learning models using text mining techniques for predicting the existence of vulnerabilities in software components. However, there are also studies that examine whether the utilization of statically extracted software metrics can lead to adequate Vulnerability Prediction Models. In this paper, both software metrics- and text mining-based Vulnerability Prediction Models are constructed and compared. A combination of software metrics and text tokens using deep-learning models is examined as well in order to investigate if a combined model can lead to more accurate vulnerability prediction. For the purposes of the present study, a vulnerability dataset containing vulnerabilities from real-world software products is utilized and extended. The results of our analysis indicate that text mining-based models outperform software metrics-based models with respect to their F2-score, whereas enriching the text mining-based models with software metrics was not found to provide any added value to their predictive performance.

Examining the characteristics of software vulnerabilities and the code that contains them can lead to the development of more secure software. We present a dataset (∼1.4 GB) containing vulnerable source code files together with the corresponding, patched versions. Contrary to other existing vulnerability datasets, ours includes vulnerable files written in more than 40 programming languages. Each file is associated to (1) a Common Vulnerability Exposures identifier (CVE ID) and (2) the repository it came from. Further, our dataset can be the basis for machine learning applications that identify defects, as we show in specific examples. We also present a supporting dataset that contains commit messages derived from Git commits that serve as security patches. This dataset can be used to train ML models that in turn, can be used to detect security patch commits as we highlight in a specific use case.

The Bugs Framework (BF) is a classification of security bugs and related faults that features a formal language for the unambiguous specification of software and hardware security weaknesses and vulnerabilities. BF bugs models, multidimensional weakness and failure taxonomies, and vulnerability models define the lexis, syntax, and semantics of the BF formal language and form the basis for the definition of secure coding principles. The BF formalism supports a deeper understanding of vulnerabilities as chains of weaknesses that adhere to strict causation, propagation, and composition rules. It enables the generation of comprehensively labeled weakness and vulnerability datasets and multidimensional vulnerability classifications. It also enables the development of new algorithms for code analysis and the use of AI models and formal methods to identify bugs and detect, analyze, prioritize, and resolve or mitigate vulnerabilities.

PurposeEffective and efficient software security inspection is crucial as the existence of vulnerabilities represents severe risks to software users. The purpose of this paper is to empirically evaluate the potential application of Stochastic Gradient Boosting Trees (SGBT) as a novel model for enhanced prediction of vulnerable Web components compared to common, popular and recent machine learning models.Design/methodology/approachAn empirical study was conducted where the SGBT and 16 other prediction models have been trained, optimized and cross validated using vulnerability data sets from multiple versions of two open-source Web applications written in PHP. The prediction performance of these models have been evaluated and compared based on accuracy, precision, recall and F-measure.FindingsThe results indicate that the SGBT models offer improved prediction over the other 16 models and thus are more effective and reliable in predicting vulnerable Web components.Originality/valueThis paper proposed a novel application of SGBT for enhanced prediction of vulnerable Web components and showed its effectiveness.

Effective and efficient software security inspection is crucial since the existence of vulnerabilities represents severe risks to software users. This paper aims to empirically evaluate and compare different types of Artificial Neural Networks (ANNs) in predicting components that are prone to vulnerabilities in web applications. We considered five different types of ANNs, which were trained, optimized, and cross validated using vulnerability datasets from multiple versions of two open source web applications written in PHP. The prediction performance of these models have been evaluated and compared based on accuracy, precision, recall and F-measure. The results indicate that the Multilayer Perceptron (MLP) neural network outperforms the other models, in both datasets, in accuracy and precision. The Cascade Correlation Neural Network (CCNN), however, outperforms the other models in recall and F-measure in one dataset, whereas the Radial Basis Function (RBF) neural network was the best in the other dataset. It has been also observed that variables selection improved some performance measures and degraded the other measures of several models.

Data-driven research on the automated discovery and repair of security vulnerabilities in source code requires comprehensive datasets of real-life vulnerable code and their fixes. To assist in such research, we propose a method to automatically collect and curate a comprehensive vulnerability dataset from Common Vulnerabilities and Exposures (CVE) records in the National Vulnerability Database (NVD). We implement our approach in a fully automated dataset collection tool and share an initial release of the resulting vulnerability dataset named CVEfixes. The CVEfixes collection tool automatically fetches all available CVE records from the NVD, gathers the vulnerable code and corresponding fixes from associated open-source repositories, and organizes the collected information in a relational database. Moreover, the dataset is enriched with meta-data such as programming language, and detailed code and security metrics at five levels of abstraction. The collection can easily be repeated to keep up-to-date with newly discovered or patched vulnerabilities. The initial release of CVEfixes spans all published CVEs up to 9 June 2021, covering 5365 CVE records for 1754 open-source projects that were addressed in a total of 5495 vulnerability fixing commits. CVEfixes supports various types of data-driven software security research, such as vulnerability prediction, vulnerability classification, vulnerability severity prediction, analysis of vulnerability-related code changes, and automated vulnerability repair.

The increasing number of security vulnerabilities has become an important problem that needs to be solved urgently in the field of software security, which means that the current vulnerability mining technology still has great potential for development. However, most of the existing AI-based vulnerability detection methods focus on designing different AI models to improve the accuracy of vulnerability detection, ignoring the fundamental problems of data-driven AI-based algorithms: first, there is a lack of sufficient high-quality vulnerability data; second, there is no unified standardized construction method to meet the standardized evaluation of different vulnerability detection models. This all greatly limits security personnel's in-depth research on vulnerabilities. In this survey, we review the current literature on building high-quality vulnerability datasets, aiming to investigate how state-of-the-art research has leveraged data mining and data processing techniques to generate vulnerability datasets to facilitate vulnerability discovery. We also identify the challenges of this new field and share our views on potential research directions.

Detecting vulnerabilities is vital for software security, yet deep learning-based vulnerability detectors (DLVD) face a data shortage, which limits their effectiveness. Data augmentation can potentially alleviate the data shortage, but augmenting vulnerable code is challenging and requires a generative solution that maintains vulnerability. Previous works have only focused on generating samples that contain single statements or specific types of vulnerabilities. Recently, large language models (LLMs) have been used to solve various code generation and comprehension tasks with inspiring results, especially when fused with retrieval augmented generation (RAG). Therefore, we propose VulScribeR , a novel LLM-based solution that leverages carefully curated prompt templates to augment vulnerable datasets. More specifically, we explore three strategies to augment both single and multi-statement vulnerabilities, with LLMs, namely Mutation, Injection, and Extension. Our extensive evaluation across four vulnerability datasets and DLVD models, using three LLMs, show that our approach beats two SOTA methods VulGen and VGX, and Random Oversampling (ROS) by 27.48%, 27.93%, and 15.41% in F1-score with 5K generated vulnerable samples on average, and 53.84%, 54.10%, 69.90%, and 40.93% with 15K generated vulnerable samples. Our approach demonstrates its feasibility for large-scale data augmentation by generating 1K samples at as cheap as US$1.88.

INTRODUCTION: Vulnerability detection is crucial for preventing severe security incidents like hacker attacks, data breaches, and network paralysis. Traditional methods, however, face challenges such as low efficiency and insufficient detail in identifying code vulnerabilities. OBJECTIVES: This paper introduces E-GVD, an advanced method for source code vulnerability detection, aiming to address the limitations of existing methods. The objective is to enhance the accuracy of function-level vulnerability detection and provide detailed, understandable insights into the vulnerabilities. METHODS: E-GVD combines Graph Neural Networks (GNNs), which are adept at handling graph-structured data, with residual connections and advanced Programming Language (PL) pre-trained models. RESULTS: Experiments conducted on the real-world vulnerability dataset CodeXGLUE show that E-GVD significantly outperforms existing baseline methods in detecting vulnerabilities. It achieves a maximum accuracy gain of 4.98%, indicating its effectiveness over traditional methods. CONCLUSION: E-GVD not only improves the accuracy of vulnerability detection but also contributes by providing fine-grained explanations. These explanations are made possible through an interpretable Machine Learning (ML) model, which aids developers in quickly and efficiently repairing vulnerabilities, thereby enhancing overall software security.

The detection of program vulnerabilities remains a challenging task in software security. The existing vulnerability detection methods rarely consider the multidimensional feature space complementarity of program graph structures, which easily overlooks contextual environment features and syntax structure features. This disadvantage leads to insufficient performance in capturing complex structural features, which hinders the improvement in detection accuracy. To address this issue, this paper introduces a novel vulnerability detection method, EnGS2F, which adopts the representation learning of an enhanced graph structure to improve the efficiency of capturing vulnerability information. On the dimension of the graph structure, a context relationship graph (CRG) is integrated on the basis of a program dependency graph (PDG) to enrich the global structural context representation. On the dimension of graph nodes, abstract syntax tree (AST) embedding and paragraph embedding are integrated to solve the problem of insufficient feature space complementarity. Moreover, the combination of a gated graph neural network (GGNN) with a graph attention mechanism further improves the learning performance of the enhanced graph structure. EnGS2F has been rigorously evaluated on program slices from open-source vulnerability datasets, demonstrating significant improvements over current competitive methods in detecting program vulnerabilities. Specifically, EnGS2F achieved a significant increase in the F1 score, outperforming existing technologies by 6%.

\nQuantitative methods for evaluating and managing software security are becoming reliable with the ever\nincreasing vulnerability datasets. The Common Vulnerability Scoring System (CVSS) provides a way to\nquantitatively evaluate individual vulnerability. However it cannot be applied to evaluate software risk directly and some metrics of CVSS are hard to assess. To overcome these shortcomings, this paper presents a novel method, which combines the CVSS base score with market share and software patches, to quantitatively evaluate the software risk. It is based on CVSS and includes three indicators: Absolute Severity Value (ASV), Relative Severity Value (RSV) and Severity Value Variation Rate (SVVR). Experimental results indicate that by using these indicators, the method can quantitatively describe the risk level of software systems, and thus strengthen software security.\n

As more users suffer serious security threats from software vulnerabilities, software security becomes increasingly important. Vulnerability prediction and risk evaluation are two of the most concerning issues in software security management. In this paper, we propose a prediction model for software vulnerability in which the probability and severity of vulnerability occurrence are determined by the logistic function and binomial distribution, respectively. Using the parameters obtained by prediction, we developed a new risk metric model. We provided some metrics, including mean time to vulnerability, local risk rate, mean risk rate, and overall risk value, from the viewpoint of time and probability. Experiments were conducted on real software vulnerability datasets. The results show that the prediction is effective and the evaluation is easy to operate. Our work has several features: (1) users can predict the vulnerability state in the future, in particular, vulnerability severity; (2) unlike traditional evaluation methods with expert scoring, our evaluation model is based on prediction and uses historical vulnerability data; and (3) the risk metric value can be used in risk assessment, security rating, and patch management.

MSGVUL: Multi-semantic integration vulnerability detection based on relational graph convolutional neural networks

Software security is a critical consideration for software development companies that want to provide their customers with high-quality and dependable software. The automated detection of software vulnerabilities is a critical aspect in software security. Vulnerability prediction is a mechanism that enables the detection and mitigation of software vulnerabilities early enough in the development cycle. Recently the scientific community has dedicated a lot of effort on the design of Deep learning models based on text mining techniques. Initially, Bag-of-Words was the most promising method but recently more complex models have been proposed focusing on the sequences of instructions in the source code. Recent research endeavors have started utilizing word embedding vectors, which are widely used in text classification tasks like semantic analysis, for representing the words (i.e., code instructions) in vector format. These vectors could be trained either jointly with the other layers of the neural network, or they can be pre-trained using popular algorithms like word2vec and fast-text. In this paper, we empirically examine whether the utilization of word embedding vectors that are pre-trained separately from the vulnerability predictor could lead to more accurate vulnerability prediction models. For the purposes of the present study, a popular vulnerability dataset maintained by NIST was utilized. The results of the analysis suggest that pre-training the embedding vectors separately from the neural network leads to better vulnerability predictors with respect to their effectiveness and performance.

The explosive growth of vulnerabilities poses a significant threat to the security of software systems. While various deep-learning-based vulnerability detection methods have emerged, they primarily rely on semantic features extracted from a single code representation structure, which limits their ability to detect vulnerabilities hidden deep within the code. To address this limitation, we propose S2FVD, short for Sequence and Structure Fusion-based Vulnerability Detector, which fuses vulnerability-indicative features learned from the multiple views of the code for more accurate vulnerability detection. Specifically, S2FVD employs either well-matched or carefully extended neural network models to extract vulnerability-indicative semantic features from the token sequence, attributed control flow graph (ACFG) and abstract syntax tree (AST) representations of a function, respectively. These features capture different perspectives of the code, which are then fused to enable S2FVD to accurately detect vulnerabilities that are well-hidden within a function. The experiments conducted on two large vulnerability datasets demonstrated the superior performance of S2FVD against state-of-the-art approaches, with its accuracy and F1 scores reaching 98.07% and 98.14% respectively in detecting the presence of vulnerabilities, and 97.93% and 97.94%, respectively, in pinpointing specific vulnerability types. Furthermore, with regard to the real-world dataset D2A, S2FVD achieved average performance gains of 6.86% and 14.84% in terms of accuracy and F1 metrics, respectively, over the state-of-the-art baselines. This ablation study also confirms the superiority of fusing the semantics implied in multiple distinct code views to further enhance vulnerability detection performance.