Challenges in the QBF Arena: the SAT’03 Evaluation of QBF Solvers

Abstract

The implementation of effective reasoning tools for deciding the satisfiability of Quantified Boolean Formulas (QBFs) is an important issue in several research fields such as Formal Verification, Planning, and Reasoning about Knowledge. Several QBF solvers have been implemented in the last few years, most of them extending the well-known Davis, Putnam, Logemann, Loveland procedure (DPLL) for propositional satisfiability (SAT). At the same time, a substantial breed of QBF benchmarks emerged, both in the form of statistical models for the generation of random formulas, and in the form of real-world instances. In this paper we report about the – first ever – evaluation of QBF solvers that was run as a joint event to SAT’03 Conference on Theory and Applications of Satisfiability Testing. Owing to the relative youngness of QBF tools and applications, we decided to run the comparison on a non-competitive basis, using the same technology that powered SAT’02 and SAT’03 competitions of SAT solvers. Running the evaluation enabled us to collect all sorts of data regarding the relative strength of different solvers and methods, the quality of the benchmarks, and to understand some of the current challenges for researchers involved in the QBF arena.