Touchable pixels: Examining the affordance effect between an on-screen object and a user-elicited gesture on the touchscreen

Abstract

Touching a smartphone screen is one of the most frequent activities in our everyday life. These touch gestures have recently been investigated under the user-centered approach, which collects gestures proposed by end-users. Despite its advantages in cognitive and behavioral performance, these gestural studies neglect the effect of visual properties of interface objects, i.e., digital affordances. This study aims to explore the effect of digital affordances on the screen through two sequential studies. In the preliminary study, an online questionnaire and interview were conducted to investigate user-defined gestures. Twenty participants responded to the survey to select the most appropriate gesture in the combination between four functions and five gestures for eight stimuli manipulated in several visual properties. The collected gestures for each function were statistically compared in terms of visual properties. In the main study, a behavioral experiment was conducted to examine the object-based correspondence effect. Twenty-six participants executed real touch gestures on the eight stimuli, and the gestural response time was collected to compare corresponding and non-corresponding conditions between gestures and stimuli. The results of both the preliminary and main studies indicated significant effects of visual properties in gesture execution. We concluded that digital affordance exists on the touchscreen; furthermore, it has unique characteristics grounded on the hybrid materiality of a digital environment consisting of both hardware and software. The new findings on digital affordance can be summarized as 1) a digital entity is perceived as freely manipulable regardless of physical laws, 2) a visuo-perceptible entity can be only perceptually constrained by other visual obstacles, and 3) the gestural priority depends on the embodied direction inherent in particular content. This study contributes theoretically to a better understanding of digital affordance and practically designing touch gestures based on the characteristics of affordances on the screen.