What can we learn by testing a program?

Abstract

It is conventional to start research papers on engineering, particularly on testing and quality, with a statement of how important has become in the world, and the potential dangers of using it when those who construct it really don't know what they are doing. The author then goes on to show that his or her theory/method/insights/tools will make the world safe (or safer, if the author is modest) by providing the understanding that is lacking. ISSTA authors (I among them) have started many of their papers in just this way, but speaking myself, these statements are window dressing --- they disguise my real concern. Long before was any part of the workaday world, before there was any software or even much software, I was interested in program analysis because programs are arguably the most intriguing mathematical objects people create. I have happily pursued the study of programs over 30 years.I wrote my first program (in ALGOL 58) in 1962 --- it failed to properly calculate a table of values of the error integral, being somewhat off in the third significant figure. (I remember the failure, and the debugging, poring over a decimal memory dump. But I can't recall the fault.) It took me until the mid-1960s to recognize that programs per se were much more interesting than their applications: I stumbled on Maurice Halstead's book [4] on self-compiling NELIAC compilers. Here was magical stuff: the master program defining a language, and itself written in that language! In my application to the University of Washington, I explained that I wanted to study computers for themselves, not as they are used. Paul Klee, the topologist who was saddled with the task of replying to mathematics grad students that year, wrote back to assure me that we've got a computer around here somewhere, and by the time you arrive I'm sure I can locate it. It was an IBM 7090, complete with IBSYS and FORTRAN, and who could have asked more in need of analysis?There were standards of respectability a PhD dissertation even in those days, so I took up recursive function theory and the program-equivalence problem. Its application to testing is that we would like to know if the program under test differs from its functional specification --- that is, can it fail? Any understanding of programs through testing (a mechanical process) must come up against the program-equivalence problem: we cannot hope to gain full understanding, because to do so would be to solve the unsolvable problem. My dissertation was an exploration of the additional information (beyond the purely functional) needed to make the program-equivalence problem solvable [5]. What brought me to the first ISSTA in 1978 was Bill Howden's 1976 paper Reliability of the path analysis strategy [6].