Empirical evaluation of constraint diagrams notation

Abstract

An empirical evaluation of constraint diagrams (CD) as a program specification language is conducted by comparing it to natural language (NL) with computer science students included two experiments, one on the interpretation of CD to evaluate the comprehension of notational system, and the other on the construction of program specifications. The first experiment took the form of a web-based competition in which 33 participants were given instructions and training either on CD or on equivalent NL specification expressions. After each example, they responded to three multiple-choice questions requiring the interpretation of expressions in their particular notation. Although the CD group spent more time on the training and had less confidence, they obtained comparable interpretation scores as the NL group and took less time to answer the questions, although they had no prior experience of CD notation. In the second experiment, which focused on the construction of CD, 20 participants were given instructions and training either on CD or on equivalent NL specification expressions. After each example, they responded to three questions requiring the construction of expressions in their particular notation. We built a mini-editor to allow the construction of the two notations, which automatically logged their interactions. Although the CD group supplied more accurate answers, they spent more time in answering the questions. The NL group supplied answers that were partially correct but with some missing information. Moreover, the CD group had spent more time in training, but their returns to the training examples were fewer than the NL group.