An experimental study of the impact of visual semantic feedback on novice programming

Abstract

Prior empirical studies of programming have shown that novice programmers tend to program by exploration, relying on frequent compilation and execution of their code in order to make progress. One way visual and end-user programming environments have attempted to facilitate this exploratory programming process is through their support of “live” editing models, in which immediate visual feedback on a program's execution is provided automatically at edit time. Notice that the notion of “liveness” actually encompasses two distinct dimensions: (a) the amount of time a programmer must wait between editing a program and receiving visual feedback ( feedback delay); and (b) whether such feedback is provided automatically, or whether the programmer must explicitly request it ( feedback self-selection). While a few prior empirical studies of “live” editing do exist, none has specifically evaluated the impact of these dimensions of “live” editing within the context of the imperative programming paradigm commonly taught in first-semester computer science courses. As a preliminary step toward that end, we conducted an experimental study that investigated the impact of feedback self-selection on novice imperative programming. Our within-subjects design compared the impact of three different levels of feedback self-selection on syntactic and semantic correctness: (a) no visual feedback at all (the No Feedback treatment); (b) visual feedback, in the form of a visualization of the program's execution state, provided on request when a “run” button is hit (the Self-Select treatment); and (c) visual feedback, in the form of a visualization of the program's execution state, updated on every keystroke (the Automatic treatment). Participants in the Automatic and Self-Select treatments produced programs that had significantly fewer syntactic and semantic errors than those of the No Feedback treatment; however, no significant differences were found between the Automatic and Self-Select treatments. These results suggest that, at least in the case of novice imperative programming environments, the benefits of delivering a continuously updated visual representation of a program's execution may fail to justify the substantial costs of implementing such feedback. We recommend that programming environment designers instead direct their efforts toward carefully considering when programmers will be ready to take advantage of the feedback that is coming toward them, along with what content will be of most benefit to them.