Seventy-five problems for testing automatic theorem provers

Abstract

The purpose of this note is to provide a graduated selection of problems for use in testing an automatic theorem proving (ATP) system. Included in these problems were the ones I have used in testing my own ATP system (Pelletier 1982, and more recent updates to the system). Some of the problems, especially some of the more difficult ones described at the end of this note, are due to discussions with Len Schubert (Computing, Univ. Alberta), Alasdair Urquhart (Philosophy, Univ. Toronto), and Charles Morgan (Philosophy, Univ. Victoria). People who have tried to compile lists of problems for ATPs in the past (for example, the Association for Automated Reasoning) have discovered that the pro- duction of such a list is difficult because (a) what is ‘easy’ for one system might not be for another, (b) researchers are understandably shy about saying what problems their ATP might have, unless they know that it is so difficult that any ATP will have trouble with it, (c) especially with problems at the ‘easy end’ of the scale, researchers are prone to think that any ATP system can prove it and so there is no call to write them up, (d) the goals of ‘natural’ system ATPs and resolution-based ATPs are different: the former tries to produce a ‘natural’ proof usually without prior conver- sion to clause form. This means that some problems, especially at the ‘easy end’ of the scale, will be trivial for resolution systems but difficult for ‘natural’ systems. On the other hand, proponents of ‘natural’ systems think that at the ‘difficult end’ of the scale, there will be problems within the grasp of the ‘natural’ systems which are beyond the reach of resolution systems. It is therefore difficult to even give a graduated scale of problems. All this has led to publication of dificuft problems, but not to publication of lists of problems suitable for developing an ATP. The locus clussicus of problems is McCharen et al. (1976) which contains a wide range of problems all in clause form. More recently, the Journal of Automated Reasoning has instituted its ‘Problem Corner’. A notable item herein is Lusk and Overbeek (1985). All of us involved in ATP wish to encourage others - graduate students, for example - to look into the field. But where is such a person to start? It is with such neophyte