An Empirical Study in Software Verification Tools

Abstract

Competitions related to software verification, which systematically compare state-of-art software verification systems, have further contributed to the development of software verification. However, so far there is no systematic study on the relationship between code structures and tools' verification answers, which can be another boost to the development of software verification. In this paper, we do a study to understand the different performance of tools based on different code structures. First, we divide programs into eight categories in accordance with their code structures and analyze the data under each category using evaluation schema. Second, we study how the different properties of the same program affect the answers of tools under each category. Third, we investigate the stability of tools when they treat slightly different programs from each other. Fourth, we probe into the existing software verification methods. We find that programs in the memory category are still challenges for software verification tools, although there are some special methods only designed for it. Through our research, we point out problems base on code structures, which should be taken attention by developments of tools.