Smart-Pockets: Body-deictic gestures for fast access to personal data during ambient interactions

Abstract

This work introduces Smart-Pockets, a new set of whole-body gesture recognition techniques that enables users to access their personal digital content efficiently for visualization on ambient displays. Smart-Pockets works by recognizing users' body-deictic gestures entailing access to their pockets, for which associations between specific pockets and personal digital content anchored to those pockets has been managed a priori. The pocket metaphor that we explore in this work enables links to digital content using physical personal containers (i.e., pockets) placed at convenient locations on the user's body, containers that have been specifically devised over decades of fashion design to store and carry people's personal belongings comfortably and conveniently. Consequently, Smart-Pockets gestures are fast, require absolutely no precision to perform effectively, and are robustly recognized in user-independent scenarios with absolutely no training required from the user of the ambient display. Also, the Smart-Pockets technique is flexible and easily extensible to other physical containers, such as bags and hand-held objects, which we demonstrate in the form of Smart-Containers. We evaluate the accuracy of several techniques for recognizing Smart-Pockets access gestures, for which we report +99% accuracy for user-independent classification and explicit segmentation. We discuss users' kinematic performance with Smart-Pockets and Smart-Containers and show that the average pocket access time of 2.2s is comparable to the average production time of touch gestures on smart mobile devices and is much smaller than the time required to produce other whole-body gestures. Beyond their practical implications for advancing knowledge in gesture-based interface design for ambient interactions, we believe that the contributions introduced by the Smart-Pockets concept will also foster new developments by pointing the community attention toward (i) more examination of the potential of a new class of whole-body gestures, i.e., body-deictics, (ii) more attention toward how users access their personal digital content on public displays, an important preliminary step before actual interaction, and (iii) inspiring work in the community to examine new and creative associations between users' physical personal objects and their digital content visualized on ambient displays. HighlightsSmart-Pockets is a new technique to access personal digital content on ambient displays.The pocket metaphor creates the link between physical objects and personal digital content.Smart-Pockets employ a new type of hybrid deictic and body-deictic gestures.Smart-Pockets actions are fast to perform (2.2s) and robustly recognized (+99%).The Smart-Pockets concept is flexible and extendable to Smart-Containers.