The limits of correctness

Abstract

On October 5, 19601, the American Ballistic Missile Early–Warning System station at Thule, Greenland, indicated a large contingent of Soviet missiles headed towards the United States2. Fortunately, common sense prevailed at the informal threat–assessment conference that was immediately convened: international tensions weren’t particularly high at the time. The system had only recently been installed. Kruschev was in New York, and all in all a massive Soviet attack seemed very unlikely. As a result no devastating counter–attack was launched. What was the problem? The moon had risen, and was reflecting radar signals back to earth. Needless to say, this lunar reflection hadn’t been predicted by the system’s designers. Over the last ten years, the Defense Department has spent many millions of dollars on a new computer technology called “program verification” a branch of computer science whose business, in its own terms, is to “prove programs correct”. Program verification has been studied in theoretical computer science departments since a few seminal papers in the 1960s3, but it has only recently started to gain in public visibility, and to be applied to real world problems. General Electric, to consider just one example, has initiated verification projects in their own laboratories: they would like to prove that the programs used in their latest computer–controlled washing machines won’t have any “bugs” (even one serious one can destroy their profit margin)4. Although it used to be that only the simplest programs could be “proven correct” — programs to put simple lists into order, to compute simple arithmetic functions –slow but steady progress has been made in extending the range of verification techniques. Recent papers have reported correctness proofs for somewhat more complex programs, including small operating systems, compilers, and other material of modern system design. 5 What, we do well to ask, does this new technology mean? How good are we at it? For example, if the 1960 warning system had been proven correct (which it was not), could we have avoided the problem with the moon? If it were possible to prove that the programs being written to control automatic launch–on–warning systems were correct, would that mean there could not be a catastrophic accident? In systems now being proposed computers will make launching decisions in a matter of seconds, with no time for any human intervention (let alone for musings about