The Effect of Problem Complexity, Computer Power, and Feedback Aids on the Accuracy and Completness of Computer Programs

Abstract

An essential step in the development of a computer program is the specification of the solution logic to be implemented by the program. The importance of accurate problem solution specification is further stressed by researchers in programming language and artificial intelligence who suggest that in a relatively short time computer programs will be synthesized automatically and that the job of the user will be only to specify the problem solution. The research reported in this paper involved an investigation of the ability of computer users to specify problem solutions in the form of example solutions. This ability was evaluated as a function of the user's background and experience, the complexity of the available processor (i.e., degree of generalization of the inputs), and the available feedback aids. Further, the ability of experienced programmers to implement the problem solution in FORTRAN IV logic code was investigated. The research reported here is part of study in multi-level communications, both abstract and concrete (i.e., general statements and examples), that may provide a method for effective two-way communication between users and computers. Two participant groups (programmers and bookkeepers/accountants) working with three levels of problem complexity and three levels of processor complexity were used. The experimental task employed in this study required specification of a logic for solution of a Navy task force problem. This task involved choosing ships from a ship list which identified the ship type, the transiting time (the time required for the ship to get from its present position to the desired site), and stationing time (the number of days the ship can remain on station with available provisions). In addition to this specification of ship combinations, the participants had to specify by the example solution the range of transiting and stationing times required. It was found that both programmers' and bookkeepers/accountants' scores decreased with increasing levels of problem and processor complexity, but the scores for the bookkeepers/accountants were significantly less than those of the programmers. In a regression analysis it was found that the degree of computer generalization of the user input (processor complexity) explained more variance than did problem complexity. Further for those experiment conditions where little computer generalization of user input was provided, performance was significantly lower than for other experiment conditions. This result suggests that computer generalization of user inputs is an important factor to accurate specification of problem solutions. Finally, the results show that participant strategy in generating solutions was the most significant factor affecting performance. The strategy measures, which indicated the degree of systematization of the participants, explained 58 percent of the score variance.