Mid-air Gestures Research Articles

RingGesture: A Ring-Based Mid-Air Gesture Typing System Powered by a Deep-Learning Word Prediction Framework.

Text entry is a critical capability for any modern computing experience, with lightweight augmented reality (AR) glasses being no exception. Designed for all-day wearability, a limitation of lightweight AR glass is the restriction to the inclusion of multiple cameras for extensive field of view in hand tracking. This constraint underscores the need for an additional input device. We propose a system to address this gap: a ring-based mid-air gesture typing technique, RingGesture, utilizing electrodes to mark the start and end of gesture trajectories and inertial measurement units (IMU) sensors for hand tracking. This method offers an intuitive experience similar to raycast-based mid-air gesture typing found in VR headsets, allowing for a seamless translation of hand movements into cursor navigation. To enhance both accuracy and input speed, we propose a novel deep-learning word prediction framework, Score Fusion, comprised of three key components: a) a word-gesture decoding model, b) a spatial spelling correction model, and c) a lightweight contextual language model. In contrast, this framework fuses the scores from the three models to predict the most likely words with higher precision. We conduct comparative and longitudinal studies to demonstrate two key findings: firstly, the overall effectiveness of RingGesture, which achieves an average text entry speed of 27.3 words per minute (WPM) and a peak performance of 47.9 WPM. Secondly, we highlight the superior performance of the Score Fusion framework, which offers a 28.2% improvement in uncorrected Character Error Rate over a conventional word prediction framework, Naive Correction, leading to a 55.2% improvement in text entry speed for RingGesture. Additionally, RingGesture received a System Usability Score of 83 signifying its excellent usability.

Working Memory Capacity for Mid-Air Gestures in Human–Computer Interaction

Mid-air gesture interactions have emerged as one of the predominant modalities used in human-computer interaction (HCI). However, extant research on gestures predominantly concentrates on the design of gestures for singular functionalities, which requires a comprehensive investigation into gesture interaction design from an integrative HCI system perspective. Working memory capacity plays a pivotal role in constraining the number of gestures accommodated within an HCI system. This study adopts the change detection paradigm to conduct an empirical investigation into the working memory capacities associated with mid-air gestures, aiming to elucidate the working memory capacity for such gestures. The experimental design included two independent variables: five distinct categories of mid-air gestures (physical, symbolic, metaphorical, abstract, and mixed) and memory set sizes ranging from two to nine items. The findings revealed that working memory capacities for different types of gestures fluctuate between three and five items, with symbolic gestures presenting the most negligible cognitive load for recall and abstract gestures proving to be the most challenging. The outcomes of this research have significant implications for both the design of individual gestures and the overarching design of gesture interaction systems, serving as a valuable reference for future endeavors in HCI design optimization.

Head-up Displays Improve Drivers’ Performance and Subjective Perceptions with the In-Vehicle Gesture Interaction System

In-vehicle infotainment systems can cause various distractions, increasing the risk of car accidents. To address this problem, mid-air gesture systems have been introduced. This study investigated the potential of a novel interface that integrates a Head-Up Display (HUD) with auditory displays (spearcons: compressed speech) in a gesture-based menu navigation system to minimize visual distraction and improve driving and secondary task performance. The experiment involved 24 participants who navigated through 12 menu items using mid-air gestures while driving on a simulated road under four conditions: HUD (with, without spearcons) and Head-Down Display (HDD) (with, without spearcons). Results showed that the HUD condition significantly outperformed the HDD condition in participants’ level 1 situation awareness, perceived workload, menu navigation performance, and system usability. However, there were trade-offs on visual fixation duration on the menu, and lane deviation. These findings will guide future research in developing safer and more effective HUD-supported in-vehicle gesture interaction systems.

DigCode—A generic mid-air gesture coding method on human-computer interaction

With high flexibility and rich semantic expressiveness, mid-air gesture interaction is an important part of natural human-computer interaction (HCI) and has broad application prospects. However, there is no unified representation frame for designing, recording, investigating and comparing HCI mid-air gestures. Therefore, this paper proposes an interpretable coding method, DigCode, for HCI mid-air gestures. DigCode converts the unstructured continuous actions into structured discrete string encoding. From the perspective of human cognition and expression, the research employed psychophysical methods to divide gesture actions into discrete intervals, defined the coding rules of representation in letters and numbers, and developed automated programs to enable encoding and decoding by using gesture sensors. The coding method can cover the existing representations of HCI mid-air gestures by considering human understanding and computer recognition and can be applied to HCI mid-air gesture design and gesture library construction.

How Do Users Regulate Interaction Behaviors While Performing a Drag-and-Drop Task in a Virtual Reality Environment?

In virtual reality (VR) environments, users interact with objects using mid-air gestures; drag-and-drop (DND) is one of the most frequently performed tasks. However, few studies have considered the characteristics of VR and the interaction behavior of DND. This study aimed to investigate the interaction behavior of DND in a VR environment using the Oculus Quest 2 system by controlling the target width, movement amplitude, and movement direction. A DND task in VR has three phases: acceleration, deceleration, and correction. We observed that the target width, movement amplitude, and movement direction had a significant effect on the three phases of DND behavior. The effects were different for each behavioral phase, and an in-depth interaction analysis was conducted through the segmentation of behavior and consideration of vertical movement. These findings can contribute to the evaluation of work performance and interaction correction techniques in VR environments.

Exploiting Physical Referent Features as Input for Multidimensional Data Selection in Augmented Reality

Embedding data into the physical environment using augmented reality (AR) is a practical approach for data visualization as it offers a large and flexible display space on or around the physical referent, i.e. the physical object to which the data is related. Yet current interaction in such context is often performed using cumbersome dedicated devices, tiring mid-air gestures or awkward on-body input. In this paper, we investigate the use of the physical referent itself as a support for input interaction with an embedded space-time cube (STC) representation. Hence, we first identify the most promising mappings between the physical features of the referent (edges, faces, corners) and the STC dimensions. Then, we design three data selection techniques using the physical referent and compare them to mid-air gestures when performing selection tasks on the STC. Overall, our work demonstrates that using the physical referent to support input interaction with embedded data representations is an efficient and comfortable approach for data selection in standing and sitting situations.

SpatialTouch: Exploring Spatial Data Visualizations in Cross-reality.

We propose and study a novel cross-reality environment that seamlessly integrates a monoscopic 2D surface (an interactive screen with touch and pen input) with a stereoscopic 3D space (an augmented reality HMD) to jointly host spatial data visualizations. This innovative approach combines the best of two conventional methods of displaying and manipulating spatial 3D data, enabling users to fluidly explore diverse visual forms using tailored interaction techniques. Providing such effective 3D data exploration techniques is pivotal for conveying its intricate spatial structures-often at multiple spatial or semantic scales-across various application domains and requiring diverse visual representations for effective visualization. To understand user reactions to our new environment, we began with an elicitation user study, in which we captured their responses and interactions. We observed that users adapted their interaction approaches based on perceived visual representations, with natural transitions in spatial awareness and actions while navigating across the physical surface. Our findings then informed the development of a design space for spatial data exploration in cross-reality. We thus developed cross-reality environments tailored to three distinct domains: for 3D molecular structure data, for 3D point cloud data, and for 3D anatomical data. In particular, we designed interaction techniques that account for the inherent features of interactions in both spaces, facilitating various forms of interaction, including mid-air gestures, touch interactions, pen interactions, and combinations thereof, to enhance the users' sense of presence and engagement. We assessed the usability of our environment with biologists, focusing on its use for domain research. In addition, we evaluated our interaction transition designs with virtual and mixed-reality experts to gather further insights. As a result, we provide our design suggestions for the cross-reality environment, emphasizing the interaction with diverse visual representations and seamless interaction transitions between 2D and 3D spaces.

Fast and Robust Mid-Air Gesture Typing for AR Headsets using 3D Trajectory Decoding.

We present a fast mid-air gesture keyboard for head-mounted optical see-through augmented reality (OST AR) that supports users in articulating word patterns by merely moving their own physical index finger in relation to a virtual keyboard plane without a need to indirectly control a visual 2D cursor on a keyboard plane. To realize this, we introduce a novel decoding method that directly translates users' three-dimensional fingertip gestural trajectories into their intended text. We evaluate the efficacy of the system in three studies that investigate various design aspects, such as immediate efficacy, accelerated learning, and whether it is possible to maintain performance without providing visual feedback. We find that the new 3D trajectory decoding design results in significant improvements in entry rates while maintaining low error rates. In addition, we demonstrate that users can maintain their performance even without fingertip and gesture trace visualization.

A multimodal smartwatch-based interaction concept for immersive environments

Augmented and Virtual Reality (AR/VR) environments require user interaction concepts beyond the traditional mouse-and-keyboard setup for seated desktop computer usage. Although advanced input modalities such as hand or gaze tracking have been developed, they have yet to be widely adopted in available hardware. Modern smartwatches have been shown to provide a powerful and intuitive means of input, thereby overcoming the limitation of the current AR/VR headsets. They typically offer a set of interesting input modalities, such as a touchscreen, rotary buttons, and an Inertial Measurement Unit (IMU), which can be used for mid-air gesture recognition. Compared to other input devices, they have the benefit that they are hands-free as soon as the user stops interacting since they are attached to the wrist. As many concepts have been proposed, comparative evaluations of their effectiveness and user-friendliness are still rare. In this paper, we evaluate the usability of two commonly found approaches for using a smartwatch as an interaction device, specifically in immersive environments provided by AR/VR HMDs: using the physical inputs of the watch (touchscreen, rotary buttons) or mid-air gestures. We conducted a user study with 20 participants, where they tested both of the interaction methods, and we compared them in their usability and performance. Based on a prototypical AR application, we evaluated the performance and user experience of these two smartwatch-based interaction concepts. We have found that the input using a touchscreen and buttons was generally favored by the participants and led to shorter task completion times.

“I Can See Your Password”: A Case Study About Cybersecurity Risks in Mid-Air Interactions of Mixed Reality-Based Smart Manufacturing Applications

Abstract This paper aims to present a potential cybersecurity risk existing in mixed reality (MR)-based smart manufacturing applications that decipher digital passwords through a single RGB camera to capture the user’s mid-air gestures. We first created a test bed, which is an MR-based smart factory management system consisting of mid-air gesture-based user interfaces (UIs) on a video see-through MR head-mounted display. To interact with UIs and input information, the user’s hand movements and gestures are tracked by the MR system. We setup the experiment to be the estimation of the password input by users through mid-air hand gestures on a virtual numeric keypad. To achieve this goal, we developed a lightweight machine learning-based hand position tracking and gesture recognition method. This method takes either video streaming or recorded video clips (taken by a single RGB camera in front of the user) as input, where the videos record the users’ hand movements and gestures but not the virtual UIs. With the assumption of the known size, position, and layout of the keypad, the machine learning method estimates the password through hand gesture recognition and finger position detection. The evaluation result indicates the effectiveness of the proposed method, with a high accuracy of 97.03%, 94.06%, and 83.83% for 2-digit, 4-digit, and 6-digit passwords, respectively, using real-time video streaming as input with known length condition. Under the unknown length condition, the proposed method reaches 85.50%, 76.15%, and 77.89% accuracy for 2-digit, 4-digit, and 6-digit passwords, respectively.

Abacus Gestures

Designing an extensive set of mid-air gestures that are both easy to learn and perform quickly presents a significant challenge. Further complicating this challenge is achieving high-accuracy detection of such gestures using commonly available hardware, like a 2D commodity camera. Previous work often proposed smaller, application-specific gesture sets, requiring specialized hardware and struggling with adaptability across diverse environments. Addressing these limitations, this paper introduces Abacus Gestures, a comprehensive collection of 100 mid-air gestures. Drawing on the metaphor of Finger Abacus counting, gestures are formed from various combinations of open and closed fingers, each assigned different values. We developed an algorithm using an off-the-shelf computer vision library capable of detecting these gestures from a 2D commodity camera feed with an accuracy exceeding 98% for palms facing the camera and 95% for palms facing the body. We assessed the detection accuracy, ease of learning, and usability of these gestures in a user study involving 20 participants. The study found that participants could learn Abacus Gestures within five minutes after executing just 15 gestures and could recall them after a four-month interval. Additionally, most participants developed motor memory for these gestures after performing 100 gestures. Most of the gestures were easy to execute with the designated finger combinations, and the flexibility in executing the gestures using multiple finger combinations further enhanced the usability. Based on these findings, we created a taxonomy that categorizes Abacus Gestures into five groups based on motor memory development and three difficulty levels according to their ease of execution. Finally, we provided design guidelines and proposed potential use cases for Abacus Gestures in the realm of mid-air interaction.

Realtime Recognition of Dynamic Hand Gestures in Practical Applications

Dynamic hand gesture acting as a semaphoric gesture is a practical and intuitive mid-air gesture interface. Nowadays benefiting from the development of deep convolutional networks, the gesture recognition has already achieved a high accuracy, however, when performing a dynamic hand gesture such as gestures of direction commands, some unintentional actions are easily misrecognized due to the similarity of the hand poses. This hinders the application of dynamic hand gestures and cannot be solved by just improving the accuracy of the applied algorithm on public datasets, thus it is necessary to study such problems from the perspective of human-computer interaction. In this article, two methods are proposed to avoid misrecognition by introducing activation delay and using asymmetric gesture design. First the temporal process of a dynamic hand gesture is decomposed and redefined, then a realtime dynamic hand gesture recognition system is built through a two-dimensional convolutional neural network. In order to investigate the influence of activation delay and asymmetric gesture design on system performance, a user study is conducted and experimental results show that the two proposed methods can effectively avoid misrecognition. The two methods proposed in this article can provide valuable guidance for researchers when designing realtime recognition system in practical applications.

RadarSense: Accurate Recognition of Mid-air Hand Gestures with Radar Sensing and Few Training Examples

Microwave radars bring many benefits to mid-air gesture sensing due to their large field of view and independence from environmental conditions, such as ambient light and occlusion. However, radar signals are highly dimensional and usually require complex deep learning approaches. To understand this landscape, we report results from a systematic literature review of ( N =118) scientific papers on radar sensing, unveiling a large variety of radar technology of different operating frequencies and bandwidths and antenna configurations but also various gesture recognition techniques. Although highly accurate, these techniques require a large amount of training data that depend on the type of radar. Therefore, the training results cannot be easily transferred to other radars. To address this aspect, we introduce a new gesture recognition pipeline that implements advanced full-wave electromagnetic modeling and inversion to retrieve physical characteristics of gestures that are radar independent, i.e., independent of the source, antennas, and radar-hand interactions. Inversion of radar signals further reduces the size of the dataset by several orders of magnitude, while preserving the essential information. This approach is compatible with conventional gesture recognizers, such as those based on template matching, which only need a few training examples to deliver high recognition accuracy rates. To evaluate our gesture recognition pipeline, we conducted user-dependent and user-independent evaluations on a dataset of 16 gesture types collected with the Walabot, a low-cost off-the-shelf array radar. We contrast these results with those obtained for the same gesture types collected with an ultra-wideband radar made of a vector network analyzer with a single horn antenna and with a computer vision sensor, respectively. Based on our findings, we suggest some design implications to support future development in radar-based gesture recognition.

Exploring Gaze-assisted and Hand-based Region Selection in Augmented Reality

Region selection is a fundamental task in interactive systems. In 2D user interfaces, users typically use a rectangle selection tool to formulate a region using a mouse or touchpad. Region selection in 3D spaces, especially in Augmented Reality (AR) Head-Mounted Displays (HMDs) is different and challenging because users need to select an intended region via freehand mid-air gestures or eye-based actions that are touchless interactions. In this work, we aim to fill in the gap in the design of region selection techniques in AR HMDs. We first analyzed and discretized the interaction procedure of region selection and explored design possibilities for each step. We then developed four techniques for region selection in AR HMDs, which leveraged users' hand and gaze for unimodal or multimodal interaction. The techniques were evaluated via a user study with a controlled region selection task. The findings led to three design recommendations and two proof-of-concept application examples.

Gesture Vocabularies for Hand Gestures for Controlling Air Conditioners in Home and Vehicle Environments

With the growing prevalence of modern technologies as part of everyday life, mid-air gestures have become a promising input method in the field of human–computer interaction. This paper analyses the gestures of actual users to define a preliminary gesture vocabulary for home air conditioning (AC) systems and suggests a gesture vocabulary for controlling the AC that applies to both home and vehicle environments. In this study, a user elicitation experiment was conducted. A total of 36 participants were filmed while employing their preferred hand gestures to manipulate a home air conditioning system. Comparisons were drawn between our proposed gesture vocabulary (HomeG) and a previously proposed gesture vocabulary which was designed to identify the preferred hand gestures for in-vehicle air conditioners. The findings indicate that HomeG successfully identifies and describes the employed gestures in detail. To gain a gesture taxonomy that is suitable for manipulating the AC at home and in a vehicle, some modifications were applied to HomeG based on suggestions from other studies. The modified gesture vocabulary (CrossG) can identify the gestures of our study, although CrossG has a less detailed gesture pattern. Our results will help designers to understand user preferences and behaviour prior to designing and implementing a gesture-based user interface.

Evaluating the usability of a prototype gesture-controlled illuminative textile

In the field of human-computer interaction (HCI), recent years have witnessed an increasing interest in smart textiles with gesture recognition functions. This paper reports usability evaluations of a prototype gesture-controlled illuminative textile developed by the authors. This is an interactive woven POF textile system with computer vision-enabled mid-air gesture recognition. For the usability test, 25 participants (11 males and 14 females) aged between 19 and 56 (M = 30.85, SD = 10.8) were randomly recruited. The study was conducted by the means of the user experience questionnaire (UEQ) and the system usability scale (SUS). Results show that participants responded positively to our gesture-controlled illuminative textile in terms of its attractiveness, perspicuity and stimulation but were less positive about its efficiency. The results will be used to make targeted improvements to the existing product design.

User-defined mid-air gestures for multiscale GIS interface interaction

ABSTRACT Multisize geographic information system (GIS) has widely used in social life, and the introduction of natural interactive methods such as gesture interaction further broadens the applications of GIS. Gestures, as a natural means of expression, bring a convenient experience to the use of GIS. The user-elicitation method can determine the user’s original operational intention and take advantage of the natural and convenient gesture interaction. However, current research on elicitation methods mainly focuses on defining the function factors of operation commands and rarely considers the influence of environmental factors such as spatial sizes. This paper takes GIS as the object, which is closely related to multisize space. This paper reports a research project on user-defined gestures for interacting with two-dimensional maps in a virtual reality environment, including three types of GIS operation commands and six sizes of maps. The results showed that the user-defined gestures for the same GIS operation commands varied across map sizes. The increase in map size increased the amount of gestural movement. Drawing on our research, we developed a set of gestures for multisize GIS interface interaction. We hope that this research can provide a guiding foundation for mid-air gesture design for all multisize interfaces.

Interpretative Structural Modeling Analyzes the Hierarchical Relationship between Mid-Air Gestures and Interaction Satisfaction

Mid-air gestures as a new form of human–computer interaction has a wide range of satisfaction factors, for which the primary and secondary relationships and hierarchical relationships between factors are unclear. By examining usability definitions, collecting satisfaction questionnaires and user interviews, 30 observed variables were obtained and a scale was developed. A total of 310 valid questionnaires were collected, and six latent variables were summarized through factor analysis. The matrix quantitative analysis of latent variables based on interpretative structural model theory was used to construct a hierarchical model of influencing factors of satisfaction with mid-air gestures. The study shows that the influencing factors of mid-air gesture satisfaction can be divided into three levels. The first layer of attractiveness is the direct influencing factor on the surface and the goal of mid-air gesture design. In the second layer, Simplicity and Efficiency, Simplicity and Tiredness, and Tiredness and Friendliness interact with each other. Simplicity positively affects Friendliness, and Efficiency positively affects Tiredness. The third layer, Intuitiveness is the root layer influencing factor, which affects Simplicity. This study provides a theoretical basis for the design of mid-air gesture so that it can be designed and selected more objectively.

Gesture-ProxylessNAS: A Lightweight Network for Mid-Air Gesture Recognition Based on UWB Radar

Gesture-ProxylessNAS: A Lightweight Network for Mid-Air Gesture Recognition Based on UWB Radar

Usability of Cross-Device Interaction Interfaces for Augmented Reality in Physical Tasks

The shortcomings of established input methods for Augmented Reality (AR) head-mounted displays (HMDs) motivate us to investigate the use of AR HMDs with smartphones and smartwatches to improve AR interaction in physical tasks. However, it is unclear whether the cross-device interaction interfaces are efficient for AR systems on physical tasks because the physical tasks can break the interaction flow. In this work, we conducted a user study to explore this. The user study consists of three subtasks, respectively requiring one of the three representative user interface (UI) controls (buttons, sliders, or text input). We implemented them with mid-air gestures, a smartphone, and a smartwatch. We compared the three interfaces and found that the smartphone and the smartwatch interaction interfaces have greater usability, provide a better user experience, and have lower workloads than the mid-air gesture interface. However, there is no significant difference in terms of efficiency for simple interactions. We discuss the limitations of this research and directions for future work.