Gesture Design Research Articles

Working Memory Capacity for Mid-Air Gestures in Human–Computer Interaction

Mid-air gesture interactions have emerged as one of the predominant modalities used in human-computer interaction (HCI). However, extant research on gestures predominantly concentrates on the design of gestures for singular functionalities, which requires a comprehensive investigation into gesture interaction design from an integrative HCI system perspective. Working memory capacity plays a pivotal role in constraining the number of gestures accommodated within an HCI system. This study adopts the change detection paradigm to conduct an empirical investigation into the working memory capacities associated with mid-air gestures, aiming to elucidate the working memory capacity for such gestures. The experimental design included two independent variables: five distinct categories of mid-air gestures (physical, symbolic, metaphorical, abstract, and mixed) and memory set sizes ranging from two to nine items. The findings revealed that working memory capacities for different types of gestures fluctuate between three and five items, with symbolic gestures presenting the most negligible cognitive load for recall and abstract gestures proving to be the most challenging. The outcomes of this research have significant implications for both the design of individual gestures and the overarching design of gesture interaction systems, serving as a valuable reference for future endeavors in HCI design optimization.

Designing Gestures for Data Exploration with Public Displays via Identification Studies

In-lab elicitation studies inform the design of gestures by having the participants suggest actions to activate the system functions. Conversely, crowd-sourced identification studies follow the opposite path, asking the users to associate the control actions with functions. Identification studies have been used to validate the gestures produced by elicitation studies, but not to design interactive systems. In this paper, we show that identification studies can be combined with in situ observations to design the gestures for data exploration with public displays. To illustrate this method, we developed two versions of a gesture-controlled system for data exploration with 368 users: one designed through an elicitation study, and one designed through in situ observations followed by an identification study. Our results show that the users discovered the majority of the gestures with similar accuracy across the two prototypes. Additionally, the in situ approach enabled the direct recruitment of target users, and the crowd-sourced approach typical of identification studies expedited the design process.

DigCode—A generic mid-air gesture coding method on human-computer interaction

With high flexibility and rich semantic expressiveness, mid-air gesture interaction is an important part of natural human-computer interaction (HCI) and has broad application prospects. However, there is no unified representation frame for designing, recording, investigating and comparing HCI mid-air gestures. Therefore, this paper proposes an interpretable coding method, DigCode, for HCI mid-air gestures. DigCode converts the unstructured continuous actions into structured discrete string encoding. From the perspective of human cognition and expression, the research employed psychophysical methods to divide gesture actions into discrete intervals, defined the coding rules of representation in letters and numbers, and developed automated programs to enable encoding and decoding by using gesture sensors. The coding method can cover the existing representations of HCI mid-air gestures by considering human understanding and computer recognition and can be applied to HCI mid-air gesture design and gesture library construction.

Design and Development of Gesture Controlled Robotic Arm

Design and Development of Gesture Controlled Robotic Arm

Quality Assessment of Interactive Mobile Application - Maternal and Infant Care for Tribal Birth Attendants.

Digital health interventions can overcome geographical barriers and prepare health-care providers for better health outcomes in rural and remote tribal areas, however, it has not been explored among traditional birth attendants (TBAs). A mobile application, "maternal and infant care" (MAI) for capacity building of tribal birth attendants was developed and its quality was evaluated using the Mobile Application Rating Scale for user's interest in and satisfaction with the esthetics, information, and functionality. Thirteen Android user TBAs with the MAI application were piloted with the MARS checklist. Engagement, functionality, esthetics, and information quality; and one subjective quality scale having 29 items were used. The application was found to be entertaining excellent rating (mean score ± standard deviation) (4.00 ± 0.58), and scored high on performance (3.77 ± 0.93); layout design (3.85 ± 0.90); subjective quality (4.23 ± 0.93), however, scored minimum on interest; gestural design; visual appeal, etc. MAI is a user-friendly, culturally acceptable Android app that can be used for the capacity building of frontline workers.

Design and Development of Gesture Controlled Light Weight Robotic Arm

The Gesture-Controlled Robotic Arm revolutionizes cleaning and manufacturing industries by automating tasks and enhancing efficiency. Traditional methods often rely on manual labor, leading to inefficiencies and safety concerns. Our device utilizes advanced robotics and gesture recognition technology for precise control and maneuverability. It minimizes human intervention, reducing risks and enhancing workplace safety. With its adaptability to various environments, it performs tasks with unparalleled accuracy and versatility. Seamlessly integrating into existing workflows.

Feasibility of NepaDengue mobile application for dengue prevention and control: user and stakeholder perspectives in Nepal

BackgroundDengue has emerged as a significant public health issue in Nepal since 2006, necessitating innovative approaches to disseminating timely information and promoting preventive behaviour. Our study focuses on developing and...

Commonsense Knowledge-Driven Joint Reasoning Approach for Object Retrieval in Virtual Reality

National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), China Retrieving out-of-reach objects is a crucial task in virtual reality (VR). One of the most commonly used approaches for this task is the gesture-based approach, which allows for bare-hand, eyes-free, and direct retrieval. However, previous work has primarily focused on assigned gesture design, neglecting the context. This can make it challenging to accurately retrieve an object from a large number of objects due to the one-to-one mapping metaphor, limitations of finger poses, and memory burdens. There is a general consensus that objects and contexts are related, which suggests that the object expected to be retrieved is related to the context, including the scene and the objects with which users interact. As such, we propose a commonsense knowledge-driven joint reasoning approach for object retrieval, where human grasping gestures and context are modeled using an And-Or graph (AOG). This approach enables users to accurately retrieve objects from a large number of candidate objects by using natural grasping gestures based on their experience of grasping physical objects. Experimental results demonstrate that our proposed approach improves retrieval accuracy. We also propose an object retrieval system based on the proposed approach. Two user studies show that our system enables efficient object retrieval in virtual environments (VEs).

Elicitation and Evaluation of Hand-based Interaction Language for 3D Conceptual Design in Mixed Reality

Conceptual design is a fundamental stage in a design process. Traditional conceptual design tools impose limitations on the intuitive creation process of 3D objects, hindering their full potential. Applying hand-based interaction in mixed reality (MR) provides an immersive and intuitive creation mode without dimensionality transformations, facilitating conceiving ideas in 3D conceptual design. Although studies on hand-based languages have explored different gesture design methodologies and techniques from many individual perspectives, they lack holistic understanding and comprehensive suggestions for gesture application in conceptual design scenarios. This study aims to establish a set of hand-based languages that are available in cognitive, physical, and system-based aspects for 3D conceptual designs in the MR environment. A two-stage study was conducted in an MR environment, combining an elicitation design and a comprehensive evaluation experiment. Through the elicitation design, approximately 930 gesture actions, focusing on 31 targeted functionalities, were collected. By performing an evaluation experiment, a set of theoretically optimal hand-based interaction languages for 3D conceptual design was proposed, and the corresponding guidelines for hand-based interactions were clarified. Our results can further expand human-computer interactions in MR environments and inspire software builders to create novel hand-driven interaction modes or tools.

An empirical practice of design and evaluation of freehand interaction gestures in virtual reality

An empirical practice of design and evaluation of freehand interaction gestures in virtual reality

The Comfort and Measurement Precision-Based Multi-Objective Optimization Method for Gesture Interaction.

As an advanced interaction mode, gestures have been widely used for human-computer interaction (HCI). This paper proposes a multi-objective optimization method based on the objective function JCP to solve the inconsistency between the gesture comfort JCS and measurement precision JPH in the gesture interaction. The proposed comfort model CS takes seventeen muscles and six degrees of freedom into consideration based on the data from muscles and joints, and is capable of simulating the energy expenditure of the gesture motion. The CS can provide an intuitive indicator to predict which act has the higher risk of fatigue or injury for joints and muscles. The measurement precision model ∆PH is calculated from the measurement error (∆XH,∆YH,∆ZH) caused by calibration, that provides a means to evaluate the efficiency of the gesture interaction. The modeling and simulation are implemented to analyze the effectiveness of the multi-objective optimization method proposed in this paper. According to the result of the comparison between the objective function JCS, based on the comfort model CS, and the objective function JPH, based on the measurement precision models ∆PH, the consistency and the difference can be found due to the variation of the radius rB_RHO and the center coordinates PB_RHOxB_RHO,yB_RHO,zB_RHO. The proposed objective function JCP compromises the inconsistency between the objective function JCS and JPH. Therefore, the multi-objective optimization method proposed in this paper is applied to the gesture design to improve the ergonomics and operation efficiency of the gesture, and the effectiveness is verified through usability testing.

Realtime Recognition of Dynamic Hand Gestures in Practical Applications

Dynamic hand gesture acting as a semaphoric gesture is a practical and intuitive mid-air gesture interface. Nowadays benefiting from the development of deep convolutional networks, the gesture recognition has already achieved a high accuracy, however, when performing a dynamic hand gesture such as gestures of direction commands, some unintentional actions are easily misrecognized due to the similarity of the hand poses. This hinders the application of dynamic hand gestures and cannot be solved by just improving the accuracy of the applied algorithm on public datasets, thus it is necessary to study such problems from the perspective of human-computer interaction. In this article, two methods are proposed to avoid misrecognition by introducing activation delay and using asymmetric gesture design. First the temporal process of a dynamic hand gesture is decomposed and redefined, then a realtime dynamic hand gesture recognition system is built through a two-dimensional convolutional neural network. In order to investigate the influence of activation delay and asymmetric gesture design on system performance, a user study is conducted and experimental results show that the two proposed methods can effectively avoid misrecognition. The two methods proposed in this article can provide valuable guidance for researchers when designing realtime recognition system in practical applications.

In-vehicle air gesture design: impacts of display modality and control orientation

In-vehicle air gesture design: impacts of display modality and control orientation

Design of Gesture Controlled Arm – An Overview

The Design of Gesture Controlled Arm project aims to explore the potential of gesture control technology in robotics by designing a functional and reliable robotic arm that can be controlled using hand gestures. The system will use machine learning algorithms and computer vision techniques to recognize and interpret hand gestures, allowing users to control the arm's movement and perform a variety of tasks, such as picking and placing objects. The project will involve designing and fabricating the robotic arm, developing the gesture recognition system, and integrating the two systems to create a cohesive and efficient solution. The project's outcome will demonstrate the practical application of gesture control technology in robotics, potentially opening new avenues for automation in various industries. The project's success will also provide valuable insights into the design and development of advanced robotic systems, paving the way for future advancements in the field.

Importance-performance analysis in fitness apps. A study from the viewpoint of gender and age.

We are currently undergoing a profound process of digital transformation that has favoured the development and use of apps in multiple facets of people's daily lives. In the fitness industry, this situation has facilitated the control of exercise and the maintenance of healthier lifestyles. However, it is not known how the perceived quality and importance of fitness apps vary for users based on gender and age, which is the objective of this study conducted among users of fitness centres. By means of a convenience sample, 321 users from different centres of the boutique fitness chain Sano Centre (238 females and 83 males) took part in the study. They answered the 16 items of the MARS scale, distributed in four dimensions, in terms of importance and perceived quality. The existence of significant differences was analysed using non-parametrical statistics such as the U-Mann-Whitney (gender) and the H-Kruskal-Wallis (age). In addition, a cluster analysis, combining hierarchical and non-hierarchical methods, was analysed considering as a dependent variable the level of recommendation of fitness apps. Considering gender, in importance-performance analysis (IPA), credibility was the most important attribute for females and quality of information for males. In the case of age, credibility was the most important attribute in all the ranges. The cluster analysis established two groups (high and low recommendations of the fitness app). In importance, the first group scored better on all factors except entertainment and interest. In valuation, the scores were lower than on importance, especially in the low recommendation group. Regarding usage behaviour, credibility is the factor to which users attach the highest importance and rating, regardless of gender and age. The main demand focuses on improving the gestural design and visual appeal, which will facilitate a better user experience.

Review for Optimal Human-Gesture Design Methodology and Motion representation of Medical Images using segmentation from depth data and gesture recognition.

Human gesture recognition and motion representation has become a vital base of current intelligent human-machine interfaces because of ubiquitous and more comfortable interaction. Human-Gesture recognition chiefly deals with recognizing meaningful, expressive body movements involving physical motions of face, head, arms, fingers, hands or body. This review article presents a concise overview of optimal human-gesture and motion representation of medical images. This paper surveys various works undertaken on human gesture design and discusses various design methodologies used for image segmentation and gesture recognition. It further provides a general idea of modeling techniques for analysing hand gesture images and even discusses the diverse techniques involved in motion recognition. This survey provides an insight into various efforts and developments made in the gesture/motion recognition domain through analyzing and reviewing the procedures, datasets, recognition rates and approaches employed for identifying diverse human motions and gestures for supporting better and devising improved applications in near future.

SequenceSense: A Tool for Designing Usable Foot-Based Gestures Using a Sequence-Based Gesture Recognizer

Foot-based gestures enable people to interact with mobile and wearable devices when their hands are unavailable for interaction. For the foot gestures to be truly usable, the gestures should be recognizable by the system without being confused by daily activities and still be easy to perform. However, designing such gestures often requires multiple iterations of gesture design, model training, and evaluation. In this paper, we present SequenceSense, a tool developed to help designers efficiently design a usable gesture set using inertial sensors, which eliminates the need for multiple data collection studies to evaluate the gestures’ usability through gesture modification by sequencing atomic actions and instant false positive analysis, and instead requires only the initial gesture sample collection. Unlike gesture recognizers using complete gestures to train a model, SequenceSense segments gesture into a sequence of atomic actions. For example, a foot tap to the right may have (1) lift the foot, (2) move the foot to the right, and (3) land the foot. SequenceSense also compares the gesture sequence with the sequence database created from the daily activities to identify possible conflicts. This allows gesture designers to build easily usable foot-based gestures without the need for recollecting and evaluating gestures. We validated SequenceSense’s efficacy in designing usable gestures with low false positives through a user study with nine gesture designers.

User-defined mid-air gestures for multiscale GIS interface interaction

ABSTRACT Multisize geographic information system (GIS) has widely used in social life, and the introduction of natural interactive methods such as gesture interaction further broadens the applications of GIS. Gestures, as a natural means of expression, bring a convenient experience to the use of GIS. The user-elicitation method can determine the user’s original operational intention and take advantage of the natural and convenient gesture interaction. However, current research on elicitation methods mainly focuses on defining the function factors of operation commands and rarely considers the influence of environmental factors such as spatial sizes. This paper takes GIS as the object, which is closely related to multisize space. This paper reports a research project on user-defined gestures for interacting with two-dimensional maps in a virtual reality environment, including three types of GIS operation commands and six sizes of maps. The results showed that the user-defined gestures for the same GIS operation commands varied across map sizes. The increase in map size increased the amount of gestural movement. Drawing on our research, we developed a set of gestures for multisize GIS interface interaction. We hope that this research can provide a guiding foundation for mid-air gesture design for all multisize interfaces.

Interpretative Structural Modeling Analyzes the Hierarchical Relationship between Mid-Air Gestures and Interaction Satisfaction

Mid-air gestures as a new form of human–computer interaction has a wide range of satisfaction factors, for which the primary and secondary relationships and hierarchical relationships between factors are unclear. By examining usability definitions, collecting satisfaction questionnaires and user interviews, 30 observed variables were obtained and a scale was developed. A total of 310 valid questionnaires were collected, and six latent variables were summarized through factor analysis. The matrix quantitative analysis of latent variables based on interpretative structural model theory was used to construct a hierarchical model of influencing factors of satisfaction with mid-air gestures. The study shows that the influencing factors of mid-air gesture satisfaction can be divided into three levels. The first layer of attractiveness is the direct influencing factor on the surface and the goal of mid-air gesture design. In the second layer, Simplicity and Efficiency, Simplicity and Tiredness, and Tiredness and Friendliness interact with each other. Simplicity positively affects Friendliness, and Efficiency positively affects Tiredness. The third layer, Intuitiveness is the root layer influencing factor, which affects Simplicity. This study provides a theoretical basis for the design of mid-air gesture so that it can be designed and selected more objectively.

The Design and Observed Effects of Robot-performed Manual Gestures: A Systematic Review

Communication using manual (hand) gestures is considered a defining property of social robots, and their physical embodiment and presence, therefore, we see a need for a comprehensive overview of the state-of-the-art in social robots that use gestures. This systematic literature review aims to address this need by (1) describing the gesture production process of a social robot, including the design and planning steps, and (2) providing a survey of the effects of robot-performed gestures on human-robot interactions in a multitude of domains. We identify patterns and themes from the existing body of literature, resulting in nine outstanding questions for research on robot-performed gestures regarding: developments in sensor technology and AI, structuring the gesture design and evaluation process, the relationship between physical appearance and gestures, the effects of planning on the overall interaction, standardizing measurements of gesture “quality,” individual differences, gesture mirroring, whether human-likeness is desirable, and universal accessibility of robots. We also reflect on current methodological practices in studies of robot-performed gestures and suggest improvements regarding replicability, external validity, measurement instruments used, and connections with other disciplines. These outstanding questions and methodological suggestions can guide future work in this field of research.