The Cost of Production in Elicitation Studies and the Legacy Bias-Consensus Trade off

Adam S Williams,Julia Sharp,Jason Garcia,Francisco R Ortega,Fidel Hernandez,Fernando De Zayas

doi:10.3390/mti4040088

Abstract

Gesture elicitation studies are a popular means of gaining valuable insights into how users interact with novel input devices. One of the problems elicitation faces is that of legacy bias, when elicited interactions are biased by prior technologies use. In response, methodologies have been introduced to reduce legacy bias. This is the first study that formally examines the production method of reducing legacy bias (i.e., repeated proposals for a single referent). This is done through a between-subject study that had 27 participants per group (control and production) with 17 referents placed in a virtual environment using a head-mounted display. This study found that over a range of referents, legacy bias was not significantly reduced over production trials. Instead, production reduced participant consensus on proposals. However, in the set of referents that elicited the most legacy biased proposals, production was an effective means of reducing legacy bias, with an overall reduction of 11.93% for the chance of eliciting a legacy bias proposal.

Highlights

Gesture elicitation has become a popular study design that can be used to gain an understanding of interactions for emerging technologies and user behavior [1]
An expert rater who was knowledgeable on gesture production classified the gesture proposals into bins of equivalent gestures
Over the 17 referents used in the study, the results indicate that the occurrence of legacy proposals did alter significantly when participants were asked to propose multiple gestures

Summary

Introduction

Gesture elicitation has become a popular study design that can be used to gain an understanding of interactions for emerging technologies and user behavior [1]. Through the use of elicitation methodologies paired with Wizard-of-Oz (WoZ) enabled systems, researchers can observe users’. Interactions with systems before accurate input recognition exists. Elicitation study design was popularized by Wobbrock et al [2] in 2005. This study methodology is under continual change and often improvement. Work has improved the metrics of consensus for interaction proposals [4,5]. Metrics to assess the dissimilarity of gestures [6], and ways to identify the level of chance agreement in the study were added [7]

Methods

Results

Discussion

Conclusion