An analysis of empirical testing for modal decision procedures

Abstract

Recent years have seen the emergence of a new generation of heavily-optimised modal decision procedures. Several systems based on such procedures are now available and have proved to be much more effective than the previous generation of modal decision procedures. As both computational complexity and algorithm complexity are generally unchanged, neither is useful in analysing and comparing these new systems and their various optimisations. Instead, empirical testing has been widely used, both for comparison and as a tool for tuning systems and identifying their strengths and weaknesses. However, the very effectiveness of the new systems has revealed serious weaknesses in existing empirical test suites and methodologies. This paper provides a detailed survey of empirical testing methodologies, analyses the current state of the art and presents new results obtained with a recently developed test method.