Quantity over quality: Do single-item scales reflect what users truly experienced?

Abstract

Single-item scales are widely used in the field of user experience to report the emotions. However, they are strongly criticized and discouraged by the scientific community. While they have several practical advantages, single-item scales are mainly criticized for their psychometric weakness. Our research explores to what extent single-item scales reflect what a user has experienced during his or her interaction with technology, overall, but also at specific moments such as first and last impressions. This research also explores the sensitivity of these measures across different contexts of use brought by the presence or absence of certain interface features. We conducted a correlational study with 40 users while interacting with financial institution websites. We used two methods to evaluate the experience: lived and measured implicitly using psychophysiological instruments on one hand and self-perceived and measured using single-item scales, on the other hand. Overall, our results suggest limited correlations and in most cases contradictory between lived and perceived experience. We conclude by highlighting the limits to be taken into account when professionals are led to use single-item scales.