Gesture Elicitation Research Articles

Different Ways of Making a Point: A Study of Gestural Communication in Typical and Atypical Early Development.

Deictic pointing is among the most impaired gestures in children with autism. Research on typical development demonstrates that contact with the referent and handshape when pointing, are associated with different communicative intentions and developmental stages. Despite their importance, the morphological features of pointing remain largely unexplored in autism. The aim of the present study was to map out pointing production in autism with a focus on handshape and contact with the referent. Participants (age range = 1–6 years old) with ASD (n = 16), at high risk for autism (n = 13) and typically developing children (n = 18) interacted with their caregivers in a gesture elicitation task. Results showed that children with ASD produced fewer pointing gestures overall and fewer index finger pointing without contact with the referent compared to the typically developing children.Lay SummaryChildren with autism produce less gestures than typical children, and pointing gestures appear to be more affected than other gesture types. Whether children point using their index finger or the palm, and whether they touch or not the referent is crucial for understanding communicative intentions. This is the first study to document experimentally exactly how pointing gestures differ in autism in comparison to typical development. We found important qualitative differences in the communicative patterns of children with autism and at risk for autism, that may serve to identify potential new markers for early diagnosis. Autism Res 2021, 14: 984–996. © 2020 The Authors. Autism Research published by International Society for Autism Research and Wiley Periodicals LLC.

A grammar for specifying full-body gestures elicited for abstract tasks

A gesture elicitation study consists of a popular method for eliciting a sample of end end users to propose gestures for executing functions in a certain context of use, specified by its users and their functions, the device or the platform used, and the physical environment in which they are working. Gestures proposed in such a study needs to be classified and, perhaps, extended in order to feed a gesture recognizer. To support this process, we conducted a full-body gesture elicitation study for executing functions in a smart home environment by domestic end users in front of a camera. Instead of defining functions opportunistically, we define them based on a taxonomy of abstract tasks. From these elicited gestures, a XML-compliant grammar for specifying resulting gestures is defined, created, and implemented to graphically represent, label, characterize, and formally present such full-body gestures. The formal notation for specifying such gestures is also useful to generate variations of elicited gestures to be applied on-the-fly on gestures in order to allow one-shot learning.

Exploring frame-based gesture design for immersive VR shopping environments

ABSTRACT In the design of gesture-based user interfaces, traditional gesture elicitation studies suffer from the legacy bias problem. In this paper, we conducted an exploratory study about the practical effects of frame-based design for gestural interaction with immersive VR shopping applications. In this study, we derived gestures via the traditional guessability and the framed guessability approaches. Experimental results indicated that priming participants with a frame, or a scenario, could significantly reduce the impact of legacy bias, and result in superior gesture vocabulary. However, no evidence was found that the priming technique would generate more gesture types, which may lead to lower agreement scores due to the reduction of legacy bias. Based on our findings, we propose some concrete design guidelines for gesture-based interaction. We highlight the implications of this work for the design of all gesture-based applications.

An investigation into the relationship between Quality of pantomime gestures and visuospatial skills

While children with developmental language disorder or Williams syndrome appear to use hand gestures to compensate for specific cognitive and communicative difficulties, they have different cognitive strength-weakness profiles. Their semantic and visuospatial skills potentially affect gesture quality such as iconicity. The present study focuses on untangling the unique contribution of these skills in the quality of gestures. An explicit gesture elicitation task was presented to 25 participants with developmental language disorder between 7 and 10 years of age, 25 age-matched peers with typical development, and 14 participants with Williams Syndrome (8−23 years). They gestured pictures of objects without using speech (pantomime). The iconicity, semantic richness, and representation technique of the pantomimes were coded. Participants’ semantic association and visuospatial skills were formally assessed. Iconicity was slightly lower in individuals with Williams syndrome, which seems related to their visuospatial deficit. While semantic saliency was similar across participant groups, small differences in representation technique were found. Partial correlations showed that visuospatial skills and semantic skills were instrumental in producing clear pantomimes. These findings indicate that clinicians aiming to enhance individuals’ natural iconic gestures should consider achieved iconicity, particularly in individuals with low visuospatial skills.

Influence of cultural factors on freehand gesture design

Abstract In the design of gesture-based user interfaces, gesture elicitation methods are often used to obtain gesture preferences of end users. User choices of gestures can be affected by various factors, such as their cultural backgrounds. Considering cultural factors in designing gesture-based interfaces is important to systems that target for users from different cultures. However, so little empirical research has been conducted on the impact of cultures on gesture commands. This paper reports a study with a series of three experiments on the gesture preferences for tasks in three different application domains with participants from two cultures. We find that some gesture choices are strongly influenced by the cultural background of participants. We discuss the characteristics of those gestures that exhibit cultural dependence and those tasks that do not. We also provide some design guidelines for freehand gesture-based interfaces.

A fuzzy interaction scheme of mid-air gesture elicitation

In a virtual assembly scenario, a semantic model of mid-air gesture interaction is established through a user-defined elicitation experiment. Considering the spatial-temporal continuity of the gesture movement, a fuzzy interaction scheme of gesture segments is proposed based on Trie Tree and Levenshtein distance. The experiment result proves that this design can effectively alleviate the effect of missing input sequences and recognition errors. Moreover, it helps relieve the memory load for users by establishing a close correlation of input actions.

User-defined gesture interaction for in-vehicle information systems

Gesture elicitation study, a technique emerging from the field of participatory design, has been extensively applied in emerging interaction and sensing technologies in recent years. However, traditional gesture elicitation study often suffers from the gesture disagreement and legacy bias problem and may not generate optimal gestures for a target system. This paper reports a research project on user-defined gestures for interacting with in-vehicle information systems. The main contribution of our research lies in a 3-stage, participatory design method we propose for deriving more reliable gestures than traditional gesture elicitation methods. Using this method, we generated a set of user-defined gestures for secondary tasks in an in-vehicle information system. Drawing on our research, we develop a set of design guidelines for freehand gesture design. We highlight the implications of this work for the gesture elicitation for all gestural interfaces.

Preference elicitation: Obtaining gestural guidelines for PACS in neurosurgery

ObjectiveAccessing medical records is an integral part of neurosurgical procedures in the Operating Room (OR). Gestural interfaces can help reduce the risks for infections by allowing the surgical staff to browse Picture Archiving and Communication Systems (PACS) without touch. The main objectives of this work are to: a) Elicit gestures from neurosurgeons to analyze their preferences, b) Develop heuristics for gestural interfaces, and c) Produce a lexicon that maximizes surgeons’ preferences. Materials and methodsA gesture elicitation study was conducted with nine neurosurgeons. Initially, subjects were asked to outline the gestures on a drawing board for each of the PACS commands. Next, the subjects performed one of three imaging tasks using gestures instead of the keyboard and mouse. Each gesture was annotated with respect to the presence/absence of gesture descriptors. Next, K-nearest neighbor approach was used to obtain the final lexicon that complies with the preferred/popular descriptors. ResultsThe elicitation study resulted in nine gesture lexicons, each comprised of 28 gestures. A paired t-test between the popularity of the overall gesture and the top three descriptors showed that the latter is significantly higher than the former (89.5%-59.7% vs 19.4%, p < 0.001), meaning more than half of the subjects agreed on these descriptors. Next, the gesture heuristics were generated for each command using the popular descriptors. Lastly, we developed a lexicon that complies with surgeons’ preferences. ConclusionsNeurosurgeons do agree on fundamental characteristics of gestures to perform image manipulation tasks. The proposed heuristics could potentially guide the development of future gesture-based interaction of PACS for the OR.

On the Use of Mobile Devices as Controllers for First-Person Navigation in Public Installations

User navigation in public installations displaying 3D content is mostly supported by mid-air interactions using motion sensors, such as Microsoft Kinect. On the other hand, smartphones have been used as external controllers of large-screen installations or game environments, and they may also be effective in supporting 3D navigations. This paper aims to examine whether a smartphone-based control is a reliable alternative to mid-air interaction for four degrees of freedom (4-DOF) fist-person navigation, and to discover suitable interaction techniques for a smartphone controller. For this purpose, we setup two studies: A comparative study between smartphone-based and Kinect-based navigation, and a gesture elicitation study to collect user preferences and intentions regarding 3D navigation methods using a smartphone. The results of the first study were encouraging, as users with smartphone input performed at least as good as with Kinect and most of them preferred it as a means of control, whilst the second study produced a number of noteworthy results regarding proposed user gestures and their stance towards using a mobile phone for 3D navigation.

Gelicit

A gesture elicitation study, as originally defined, consists of gathering a sample of participants in a room, instructing them to produce gestures they would use for a particular set of tasks, materialized through a representation called referent, and asking them to fill in a series of tests, questionnaires, and feedback forms. Until now, this procedure is conducted manually in a single, physical, and synchronous setup. To relax the constraints imposed by this manual procedure and to support stakeholders in defining and conducting such studies in multiple contexts of use, this paper presents Gelicit, a cloud computing platform that supports gesture elicitation studies distributed in time and space structured into six stages: (1) define a study: a designer defines a set of tasks with their referents for eliciting gestures and specifies an experimental protocol by parameterizing its settings; (2) conduct a study: any participant receiving the invitation to join the study conducts the experiment anywhere, anytime, anyhow, by eliciting gestures and filling forms; (3) classify gestures: an experimenter classifies elicited gestures according to selected criteria and a vocabulary; (4) measure gestures: an experimenter computes gesture measures, like agreement, frequency, to understand their configuration; (5) discuss gestures: a designer discusses resulting gestures with the participants to reach a consensus; (6) export gestures: the consensus set of gestures resulting from the discussion is exported to be used with a gesture recognizer. The paper discusses Gelicit advantages and limitations with respect to three main contributions: as a conceptual model for gesture management, as a method for distributed gesture elicitation based on this model, and as a cloud computing platform supporting this distributed elicitation. We illustrate Gelicit through a study for eliciting 2D gestures executing Internet of Things tasks on a smartphone.

Utilizing gestures to enable visually impaired for computer interaction

Despite active research in input device development, the visually impaired community still find difficulty in interacting with computers. Braille and other conventional methods have limitations while inputting data to the computer due to their vision loss. Gesture-based interaction can offer them a new vista of computer interaction. However, one needs to consider their performance and preference towards hand gesture. So, a gesture elicitation study is done with 25 visually impaired users. A quantitative rating analysis is performed with them, and an optimal set of gestures is obtained. Further, a dactylology is proposed using which visually impaired users can interact with computers. A recognition module is also presented to recognise the dactylology symbols.

Interactive spaces for children: gesture elicitation for controlling ground mini-robots

Interactive spaces for education are emerging as a mechanism for fostering children’s natural ways of learning by means of play and exploration in physical spaces. The advanced interactive modalities and devices for such environments need to be both motivating and intuitive for children. Among the wide variety of interactive mechanisms, robots have been a popular research topic in the context of educational tools due to their attractiveness for children. However, few studies have focused on how children would naturally interact and explore interactive environments with robots. While there is abundant research on full-body interaction and intuitive manipulation of robots by adults, no similar research has been done with children. This paper therefore describes a gesture elicitation study that identified the preferred gestures and body language communication used by children to control ground robots. The results of the elicitation study were used to define a gestural language that covers the different preferences of the gestures by age group and gender, with a good acceptance rate in the 6–12 age range. The study also revealed interactive spaces with robots using body gestures as motivating and promising scenarios for collaborative or remote learning activities.

Systematic literature review of hand gestures used in human computer interaction interfaces

Abstract Gestures, widely accepted as a humans’ natural mode of interaction with their surroundings, have been considered for use in human-computer based interfaces since the early 1980s. They have been explored and implemented, with a range of success and maturity levels, in a variety of fields, facilitated by a multitude of technologies. Underpinning gesture theory however focuses on gestures performed simultaneously with speech, and majority of gesture based interfaces are supported by other modes of interaction. This article reports the results of a systematic review undertaken to identify characteristics of touchless/in-air hand gestures used in interaction interfaces. 148 articles were reviewed reporting on gesture-based interaction interfaces, identified through searching engineering and science databases (Engineering Village, Pro Quest, Science Direct, Scopus and Web of Science). The goal of the review was to map the field of gesture-based interfaces, investigate the patterns in gesture use, and identify common combinations of gestures for different combinations of applications and technologies. From the review, the community seems disparate with little evidence of building upon prior work and a fundamental framework of gesture-based interaction is not evident. However, the findings can help inform future developments and provide valuable information about the benefits and drawbacks of different approaches. It was further found that the nature and appropriateness of gestures used was not a primary factor in gesture elicitation when designing gesture based systems, and that ease of technology implementation often took precedence.

Empirical Research in Mid-Air Interaction: A Systematic Review

ABSTRACTMid-air interaction is a distinct style of natural HCI (Human-Computer Interaction). In mid-air interaction, users make use of their whole body—with a strong focus on hands—and apply gestures, postures, and movements to interact with digital content on distant displays or remote devices. Although the idea of exploiting body movements and gestures in HCI is not new, mid-air interaction is now possible due to the availability of reasonably priced depth cameras and sensors. This article presents a systematic review of empirical research in mid-air interaction, based on a corpus of 104 publications from 2011 to 2018. The review includes: (a) a retrospective on mid-air interaction, which outlines its historical development and clarifications on important concepts; (b) current application domains of mid-air interaction; (c) user requirements methods, focusing on gesture elicitation studies; (d) dimensions of prototyping and design; (e) empirical evaluation methods and issues; (f) a discussion on several trends and challenges for further research and development.

Eliciting Contact-Based and Contactless Gestures With Radar-Based Sensors

Radar sensing technologies now offer new opportunities for gesturally interacting with a smart environment by capturing microgestures via a chip that is embedded in a wearable device, such as a smartwatch, a finger or a ring. Such microgestures are issued at a very small distance from the device, regardless of whether they are contact-based, such as on the skin, or contactless. As this category of microgestures remains largely unexplored, this paper reports the results of a gesture elicitation study that was conducted with twenty-five participants who expressed their preferred user-defined gestures for interacting with a radar-based sensor on nineteen referents that represented frequent Internet-of-things tasks. This study clustered the $25 \times 19=475$ initially elicited gestures into four categories of microgestures, namely, micro, motion, combined, and hybrid, and thirty-one classes of distinct gesture types and produced a consensus set of the nineteen most preferred microgestures. In a confirmatory study, twenty new participants selected gestures from this classification for thirty referents that represented tasks of various orders; they reached a high rate of agreement and did not identify any new gestures. This classification of radar-based gestures provides researchers and practitioners with a larger basis for exploring gestural interactions with radar-based sensors, such as for hand gesture recognition.

HeadGesture

We propose HeadGesture, a hands-free input approach to interact with Head Mounted Display (HMD) devices. Using HeadGesture, users do not need to raise their arms to perform gestures or operate remote controllers in the air. Instead, they perform simple gestures with head movement to interact with the devices. In this way, users' hands are free to perform other tasks, e.g., taking notes or manipulating tools. This approach also reduces the hand occlusion of the field of view [11] and alleviates arm fatigue [7]. However, one main challenge for HeadGesture is to distinguish the defined gestures from unintentional movements. To generate intuitive gestures and address the issue of gesture recognition, we proceed through a process of Exploration - Design - Implementation - Evaluation. We first design the gesture set through experiments on gesture space exploration and gesture elicitation with users. Then, we implement algorithms to recognize the gestures, including gesture segmentation, data reformation and unification, feature extraction, and machine learning based classification. Finally, we evaluate user performance of HeadGesture in the target selection experiment and application tests. The results demonstrate that the performance of HeadGesture is comparable to mid-air hand gestures, measured by completion time. Additionally, users feel significantly less fatigue than when using hand gestures and can learn and remember the gestures easily. Based on these findings, we expect HeadGesture to be an efficient supplementary input approach for HMD devices.

User-defined gesture interaction for immersive VR shopping applications

ABSTRACTGesture elicitation studies, which are a popular technology for collecting requirements and expectations by involving real users in gesture design processes, often suffer from gesture disagreement and legacy bias and may not generate optimal gestures for a target system in practice. This paper reports a research project on user-defined gestures for interacting with immersive VR shopping applications. The main contribution of this work is the proposal of a more practical method for deriving more reliable gestures than traditional gesture elicitation studies. We applied this method to a VR shopping application and obtained empirical evidence for the benefits of deriving two gestures in the a priori stage and selecting the top-two gestures in the a posteriori stage of traditional elicitation studies for each referent. We hope that this research can help lay a theoretical foundation for freehand-gesture-based user interface design and be generalised to all freehand-gesture-based applications.

Seeking common ground while reserving differences in gesture elicitation studies

Gesture elicitation studies have been frequently conducted in recent years for gesture design. However, most elicitation studies adopted the frequency ratio approach to assign top gestures derived from end-users to the corresponding target tasks, which may cause the results get caught in local minima, i.e., the gestures discovered in an elicitation study are not the best ones. In this paper, we propose a novel approach of seeking common ground while reserving differences in gesture elicitation research. To verify this point, we conducted a four-stage case study on the derivation of a user-defined mouse gesture vocabulary for web navigation and then provide new empirical evidences on our proposed method, including 1) gesture disagreement is a serious problem in elicitation studies, e.g., the chance for participants to produce the same mouse gesture for a given target task without any restriction is very low, below 0.26 on average; 2) offering a set of gesture candidates can improve consistency; and 3) benefited from the hindsight effect, some unique but highly teachable gestures produced in the elicitation study may also have a chance to be chosen as the top gestures. Finally, we discuss how these findings can be applied to inform all gesture-based interaction design.

Gesture Elicitation Studies for Mid-Air Interaction: A Review

Mid-air interaction involves touchless manipulations of digital content or remote devices, based on sensor tracking of body movements and gestures. There are no established, universal gesture vocabularies for mid-air interactions with digital content or remote devices based on sensor tracking of body movements and gestures. On the contrary, it is widely acknowledged that the identification of appropriate gestures depends on the context of use, thus the identification of mid-air gestures is an important design decision. The method of gesture elicitation is increasingly applied by designers to help them identify appropriate gesture sets for mid-air applications. This paper presents a review of elicitation studies in mid-air interaction based on a selected set of 47 papers published within 2011–2018. It reports on: (1) the application domains of mid-air interactions examined; (2) the level of technological maturity of systems at hand; (3) the gesture elicitation procedure and its variations; (4) the appropriateness criteria for a gesture; (5) participants number and profile; (6) user evaluation methods (of the gesture vocabulary); (7) data analysis and related metrics. This paper confirms that the elicitation method has been applied extensively but with variability and some ambiguity and discusses under-explored research questions and potential improvements of related research.

Looking Beyond the Gesture: Vocabulary Acceptability Criteria for Gesture Elicitation Studies

The choice of what gestures should be part of a gesture language is a critical step in the design of gesturebased interfaces. This step is especially important when time and accuracy are key factors of the user experience, such as gestural interfaces in vehicle control and sterile control of a picture archiving and communication system (PACS) in the operating room (OR). Agreement studies are commonly used to find the gesture preference of the end users. These studies hypothesize that the best available gesture lexicon is the one preferred by a majority. However, these agreement approaches cannot offer a metric to assess the qualitative aspects of gestures. In this work, we propose an experimental framework to quantify, compare and evaluate gestures. This framework is grounded in the expert knowledge of speech and language professionals (SLPs). The development consisted of three studies: 1) Creation, 2) Evaluation and 3) Validation. In the creation study, we followed an adapted version of the Delphi’s interview/discussion procedure with SLPs. The purpose was to obtain the Vocabulary Acceptability Criteria (VAC) to evaluate gestures. Next, in the evaluation study, a modified method of pairwise comparisons was used to rank and quantify the gestures based on each criteria (VAC). Lastly, in the validation study, we formulated an odd one out procedure, to prove that the VAC values of a gesture are representative and sufficiently distinctive, to select that particular gesture from a pool of gestures. We applied this framework to the gestures obtained from a gesture elicitation study conducted with nine neurosurgeons, to control an imaging software. In addition, 29 SLPs comprising of 17 experts and 12 graduate students participated in the VAC study. The best lexicons from the available pool were obtained through both agreement and VAC metrics. We used binomial tests to show that the results obtained from the validation procedure are significantly better than the baseline. These results verify our hypothesis that the VAC are representative of the gestures and the subjects should be able to select the right gesture given its VAC values.