A Database of Existing Vulnerabilities to Enable Controlled Testing Studies

Abstract

From holding worldwide companies' information hostage to keeping several distributed systems down for hours, the last years were marked by several security attacks which are the result of complex software and its fast production. There are already tools which can be used to help companies detect vulnerabilities responsible for such attacks. However, their reliability is still not the best and well discriminated. In software testing, researchers tend to use hand-seeded test cases or mutations due to the challenges involved in the extraction or reproduction of real test cases which might not be suitable for testing techniques, since both approaches can create samples that inadvertently differ from the real vulnerabilities and thus might lead to misleading assessments of the tools' capabilities. The lack of databases of real security vulnerabilities is an issue since it hampers the tools' evaluation and categorization. To study these tools, the researchers created a database of 682 real test cases which is the outcome of mining 248 repositories for 16 different vulnerability patterns.