CRITIQUE AND RESPONSE: Critique of “A Longitudinal Study of the Effects of Ecological Interface Design on Skill Acquisition” by Christoffersen, Hunter, and Vicente

Abstract

As one of the reviewers of this article, I've been asked by the editor to write a critique describing my comments and reservations about the study. This is a new feature for Human Factors, so I undertake this task with the knowledge that many readers will not have been exposed to such explicit professional criticism. However, this type of (I hope) constructive criticism is common in our review process for journal manuscripts. The authors are to be commended for allowing the issues I will raise in this critique to be aired publicly. Before I begin my critique, let me say that the work described here contains many excellent features. First and foremost, this research was conducted over a long period with nearly continuous empirical data gathering. This, by itself, represents a unique and valuable contribution to our body of knowledge. Anyone who has ever conducted a research study using human subjects can appreciate the logistical difficulties inherent in long-term experiments. I commend both the authors and subjects for their perseverance. Also, the topic of the research is relevant to many practical situations involving process control. This publication and the human factors research community are often taken to task for the lack of practical relevance in our work. This article and the work itself directly rebuts such criticism. The review process for the present article, which has involved two other reviewers and me, along with the editor, has taken about one year. In the two revisions since the original manuscript, I believe the authors have conscientiously tried to address reviewers' concerns. What we are left with are only a few issues about which we disagree. Other reviewers have questioned the number and selection procedure for subjects. I believe these issues are related directly to the difficult nature of long-term studies, and I'm content to let readers make their own determination regarding their relevance to the results. Only one of the issues I raised during the review process has any fundamental bearing on the interpretation of the work presented here. In my opinion, this issue has not been adequately addressed in either of the revisions. My critique of this work is quite simple. I do not believe the study addresses the primary factor it was intended to address. In the study, the performance of subjects using one of two different types of process control interfaces was compared. Statistical analyses show that although overall performance levels did not vary between the two interfaces, subjects' performance was more variable with one than with the other. The authors conclude that the type of interface - more important, the principles used to develop one of the interfaces - is the determining factor in these performance differences. I believe this factor was confounded by certain elements in the design of the two interfaces used in the study. The basic hypothesis underlying this study is that presenting simple physical information about a process, such as pressure, temperature, valve status, and so on, is not necessarily the best portrayal for supporting operator tasks, such as controlling the process. To best support various types of behavior, the authors, and others, contend that information must be presented in forms that support an abstraction hierarchy, as described by Rasmussen (1985). The authors are correct in asserting that ecological interfaces have not been widely evaluated, although at least one recent study examined operator performance with a number of different types of interfaces - including ecological - in the process control domain (Edlund & Lewis, 1994). However, the idea of presenting process information according to abstraction hierarchies is certainly not new. After the Three Mile Island accident in 1979, various studies examined the information available to nuclear power plant control room operators. With the advent of symptom-based emergency procedures, there was general agreement that presenting simple physical information was not sufficient to allow operators to make certain task-related decisions. …