Live programming in practice: A controlled experiment on state machines for robotic behaviors

Abstract

Abstract Context Live programming environments are gaining momentum across multiple programming languages. A tenet of live programming is a development feedback cycle, resulting in faster development practices. Although practitioners of live programming consider it a positive inclusion in their workflow, no in-depth investigations have yet been conducted on its benefits in a realistic scenario, nor using complex API. Objective This paper carefully studies the advantage of using live programming in defining nested state machines for robot behaviors. We analyzed two important aspects of developing robotic behaviors using these machines: program comprehension and program writing. We analyzed both development practices in terms of speed and accuracy. Method We conducted two controlled experiments, one for program comprehension and another for program writing. We measured the speed and accuracy of randomized assigned participants on completing programming tasks, against a baseline. Results In a robotic behavior context, we found that a live programming system for nested state machine programs does not significantly outperform a non-live language in program comprehension nor in program writing in terms of speed and accuracy. However, the feedback of test subjects indicates their preference for the live programming system. Conclusions The results of this work seem to contradict the studies of live programming in other areas, even while participants still favor using live programming techniques. We learned that the complex API chosen in this work has a strong negative influence on the results. To the best of our knowledge, this is the first in-depth live programming experiment in a complex domain.