Gesture Interface Research Articles

GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents

Existing gesture interfaces only work with a fixed set of gestures defined either by interface designers or by users themselves, which introduces learning or demonstration efforts that diminish their naturalness. Humans, on the other hand, understand free-form gestures by synthesizing the gesture, context, experience, and common sense. In this way, the user does not need to learn, demonstrate, or associate gestures. We introduce GestureGPT, a free-form hand gesture understanding framework that mimics human gesture understanding procedures to enable a natural free-form gestural interface. Our framework leverages multiple Large Language Model agents to manage and synthesize gesture and context information, then infers the interaction intent by associating the gesture with an interface function. More specifically, our triple-agent framework includes a Gesture Description Agent that automatically segments and formulates natural language descriptions of hand poses and movements based on hand landmark coordinates. The description is deciphered by a Gesture Inference Agent through self-reasoning and querying about the interaction context (e.g., interaction history, gaze data), which is managed by a Context Management Agent. Following iterative exchanges, the Gesture Inference Agent discerns the user’s intent by grounding it to an interactive function. We validated our framework offline under two real-world scenarios: smart home control and online video streaming. The average zero-shot Top-1/Top-5 grounding accuracies are 44.79%/83.59% for smart home tasks and 37.50%/73.44% for video streaming tasks. We also provide an extensive discussion that includes rationale for model selection, generalizability, and future research directions for a practical system etc.

Investigating User Experience of an Immersive Virtual Reality Simulation Based on a Gesture-Based User Interface

Investigating User Experience of an Immersive Virtual Reality Simulation Based on a Gesture-Based User Interface

OmniCharger: CNN-based Hand Gesture Interface to Operate an Electric Car Charging Robot through Teleconference

The automation of the car charging process is motivated by the rapid development of technologies for self-driving cars and the increasing importance of ecological transportation units. Automation of this process requires the implementation of Computer Vision (CV) techniques. However, it remains challenging to precisely position the charger plug autonomously due to the sensitivity of CV algorithms to lighting and weather conditions. We introduce a novel robotic operation system based on hand gesture recognition through teleconferencing software. The users, connected by teleconference, use their hand gestures to teleoperate the electric plug located on the collaborative robot end-effector. We conducted a user study to evaluate the system performance and suitability using OmniCharger and two baseline interfaces (a UR10 Teach Pendant and a Logitech F710 Wireless Gamepad). Except for two trials, all the users were able to locate the plug inside of a 5 cm target using the interfaces. The distance to the target and the orientation error did not present statistically significant differences ( \(p=0.1099\,{\gt}\,0.05\) and \(p=0.0903\,{\gt}\,0.05\) , respectively) in the use of the three interfaces. The NASA TLX questionnaire results showed low values in all the sub-classes, the SUS results rated the usability of the proposed interface above average (68%), and the UEQ showed excellent performance of the OmniCharger interface in the attractiveness, stimulation, and novelty attributes.

Entering the Next Dimension: A Review of 3D User Interfaces for Virtual Reality

In the last decade, the use of virtual reality (VR) technologies has been increasing, as head-mounted display devices, such as the Meta Quest series, have become commercially accessible to everyday consumers. There are plenty of software applications developed for VR, ranging from games to serious training. Despite the fast-growing emergence of novel and natural gesture-based interface components, traditional windows, icons, menus, and pointer user interfaces still remain popular in these implementations. Therefore, navigating and interacting within 3-dimensional environments can still be challenging in virtual reality. This body of work will serve as a reference to the literature survey on 3-dimensional user interfaces and their implementations, and categorize them to reiterate a set of design recommendations for implementing 3-dimensional user interfaces in virtual reality. While questions revolve around the need for a 3-dimensional user interface if a 2-dimensional user interface suffices, we cannot ignore the capabilities and benefits offered when utilising 3-dimensional space. Unlike other work, this work proposes a novel metaphor-guided quadrant model to delineate the 3-dimensional user interface problem in an end-to-end manner. To achieve this, over 30 unique articles selected using a meta-analyses methodology were reviewed. The result highlights the dimensional semantics of the inputs and dimensional outcomes of the virtual targets. Notable reiterated design recommendations would match devices to interaction and then to the tasks in 3-dimensional environments, reducing the degrees of freedom, imposing constraints when possible, and not implementing one single best interaction technique. The results of this survey will be beneficial for virtual reality simulation designers or researchers attempting to determine the appropriate model when developing user interfaces for virtual reality content.

Designing a 3D gestural interface to support user interaction with time-oriented data as immersive 3D radar charts

The design of intuitive three-dimensional user interfaces is vital for interaction in virtual reality, allowing to effectively close the loop between a human user and the virtual environment. The utilization of 3D gestural input allows for useful hand interaction with virtual content by directly grasping visible objects, or through invisible gestural commands that are associated with corresponding features in the immersive 3D space. The design of such interfaces remains complex and challenging. In this article, we present a design approach for a three-dimensional user interface using 3D gestural input with the aim to facilitate user interaction within the context of Immersive Analytics. Based on a scenario of exploring time-oriented data in immersive virtual reality using 3D Radar Charts, we implemented a rich set of features that is closely aligned with relevant 3D interaction techniques, data analysis tasks, and aspects of hand posture comfort. We conducted an empirical evaluation (n=12)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$(n=12)$$\\end{document}, featuring a series of representative tasks to evaluate the developed user interface design prototype. The results, based on questionnaires, observations, and interviews, indicate good usability and an engaging user experience. We are able to reflect on the implemented hand-based grasping and gestural command techniques, identifying aspects for improvement in regard to hand detection and precision as well as emphasizing a prototype’s ability to infer user intent for better prevention of unintentional gestures.

How Do People with Limited Movement Personalize Upper-Body Gestures? Considerations for the Design of Personalized and Accessible Gesture Interfaces.

Always-on, upper-body input from sensors like accelerometers, infrared cameras, and electromyography hold promise to enable accessible gesture input for people with upper-body motor impairments. When these sensors are distributed across the person's body, they can enable the use of varied body parts and gestures for device interaction. Personalized upper-body gestures that enable input from diverse body parts including the head, neck, shoulders, arms, hands and fingers and match the abilities of each user, could be useful for ensuring that gesture systems are accessible. In this work, we characterize the personalized gesture sets designed by 25 participants with upper-body motor impairments and develop design recommendations for upper-body personalized gesture interfaces. We found that the personalized gesture sets that participants designed were highly ability-specific. Even within a specific type of disability, there were significant differences in what muscles participants used to perform upper-body gestures, with some pre-dominantly using shoulder and upper-arm muscles, and others solely using their finger muscles. Eight percent of gestures that participants designed were with their head, neck, and shoulders, rather than their hands and fingers, demonstrating the importance of tracking the whole upper-body. To combat fatigue, participants performed 51% of gestures with their hands resting on or barely coming off of their armrest, highlighting the importance of using sensing mechanisms that are agnostic to the location and orientation of the body. Lastly, participants activated their muscles but did not visibly move during 10% of the gestures, demonstrating the need for using sensors that can sense muscle activations without movement. Both inertial measurement unit (IMU) and electromyography (EMG) wearable sensors proved to be promising sensors to differentiate between personalized gestures. Personalized upper-body gesture interfaces that take advantage of each person's abilities are critical for enabling accessible upper-body gestures for people with upper-body motor impairments.

Realtime Recognition of Dynamic Hand Gestures in Practical Applications

Dynamic hand gesture acting as a semaphoric gesture is a practical and intuitive mid-air gesture interface. Nowadays benefiting from the development of deep convolutional networks, the gesture recognition has already achieved a high accuracy, however, when performing a dynamic hand gesture such as gestures of direction commands, some unintentional actions are easily misrecognized due to the similarity of the hand poses. This hinders the application of dynamic hand gestures and cannot be solved by just improving the accuracy of the applied algorithm on public datasets, thus it is necessary to study such problems from the perspective of human-computer interaction. In this article, two methods are proposed to avoid misrecognition by introducing activation delay and using asymmetric gesture design. First the temporal process of a dynamic hand gesture is decomposed and redefined, then a realtime dynamic hand gesture recognition system is built through a two-dimensional convolutional neural network. In order to investigate the influence of activation delay and asymmetric gesture design on system performance, a user study is conducted and experimental results show that the two proposed methods can effectively avoid misrecognition. The two methods proposed in this article can provide valuable guidance for researchers when designing realtime recognition system in practical applications.

Study on the brightness and graphical display object directions of the Single-Gaze-Gesture user interface

Single gaze gestures (SGGs), as the basic unit of gaze gestures, interact with graphical display objects (GDOs) on a graphical user interface (GUI) through user saccades to control the system. The brightness design of the interface and saccade direction can significantly affect interaction performance and user experience. Therefore, this study adopts the method of ergonomic experiments combined with subjective evaluation, proposing the design suggestions for the background brightness (BB), brightness contrast (BC), GDO (completion field) directions, and target brightness (TB) of the gaze gesture interface: (1) Background brightness: To ensure the efficiency and experience in actual use, the background of the interface should choose a brightness value slightly lower than 50%; (2) Brightness contrast: At 25% BB, it is advisable to choose a brightness value between 21.5% and 43.5%, and the best choice is 33.25%. Under the condition of BB above 50%, the brightness contrast value of 50% can meet the requirements of the operation task; (3) Target brightness: In the gaze gesture interface design, light color should be used to mark the target; (4) GDO Directions: The GDO should be placed on the upper side of the initiation field (IF), and it is best to be directly above the IF. The contribution of this research is the proposal of an interface design strategy that is beneficial for improving the performance and experience of gaze gesture interaction and can provide future directions and recommendations for interface design.

Head Gesture Interface for Mouse Stick Users by AAGI.

We developed a gesture interface (AAGI) for individuals with motor dysfunction who cannot use standard interface switches. These users have cerebral palsy, quadriplegia, or traumatic brain injury and experience involuntary movement, spasticity, and so on. In this paper, we describe a disabled user who utilizes a mouth stick for laptop PC input in daily life. Our objective is to lower the burden on his body by using gestures. To this end, we developed a "home position" for the head that enables gestures to coexist with the mouse stick usage. The results of basic experiments with five healthy participants indicate that our system has reached the level where it can be applied to actual disabled persons. Finally, we applied the system to a user with cerebral palsy asked him to perform web browsing.

Hand Gesture Interface for Robot Path Definition in Collaborative Applications: Implementation and Comparative Study.

The article explores the possibilities of using hand gestures as a control interface for robotic systems in a collaborative workspace. The development of hand gesture control interfaces has become increasingly important in everyday life as well as professional contexts such as manufacturing processes. We present a system designed to facilitate collaboration between humans and robots in manufacturing processes that require frequent revisions of the robot path and that allows direct definition of the waypoints, which differentiates our system from the existing ones. We introduce a novel and intuitive approach to human-robot cooperation through the use of simple gestures. As part of a robotic workspace, a proposed interface was developed and implemented utilising three RGB-D sensors for monitoring the operator's hand movements within the workspace. The system employs distributed data processing through multiple Jetson Nano units, with each unit processing data from a single camera. MediaPipe solution is utilised to localise the hand landmarks in the RGB image, enabling gesture recognition. We compare the conventional methods of defining robot trajectories with their developed gesture-based system through an experiment with 20 volunteers. The experiment involved verification of the system under realistic conditions in a real workspace closely resembling the intended industrial application. Data collected during the experiment included both objective and subjective parameters. The results indicate that the gesture-based interface enables users to define a given path objectively faster than conventional methods. We critically analyse the features and limitations of the developed system and suggest directions for future research. Overall, the experimental results indicate the usefulness of the developed system as it can speed up the definition of the robot's path.

Survey on Vision Based Hand Gesture Interface for Controlling Multimedia Player

As computers become additional pervasive in society, facilitating natural human–computer interaction (HCI) can have a positive impact on their use. Hence, there has been growing interest within the development of recent approaches and technologies for bridging the human-computer barrier. the last word aim is to bring HCI to a regime wherever interactions with computers are going to be as natural as associate degree interaction between humans, and to the current finish, incorporating gestures in HCI is a crucial analysis space. Gestures have long been thought-about as associate degree interaction technique that may doubtless deliver additional natural, creative, and intuitive strategies for human activity with our computers. Hand gesture recognition is one amongll|one amongst|one in every of} the systemsthat may notice the gesture of the hand in a period of time video. The gesture of hand is classed inside a definite space of interest. during this study, planning hand gesture recognition is one among the difficult jobs that involves 2 major issues. foremost is that the detection of the hand. Another drawback is to form an indication that's appropriate to be used one hand at a time. This project concentrates on however a system might notice, acknowledge and interpret hand gesture recognition through computer vision with the difficult factors that variability within the create, orientation, location, and scale. To perform well for developing this project, differing kinds of gestures like numbers and sign languages got to be created during this system. The image taken from the period of time video is analyzed via Haar-cascade Classifier to notice the gesture of hand before the image process is finished or in different words to notice the looks of hand in a very frame. during this project, the detection of hand are going to be done mistreatment the theories of Region of Interest (ROI) via Python programming. the reason of the results are going to be targeted on the simulation half since the distinction for the hardware implementation is that the ASCII text file to scan the period of time input video. the event of hand gesture recognition mistreatment Python, OpenCV, and YOLO V3 will be enforced by applying the theories of hand segmentation and also the hand detection system that uses the Haar-cascade classifier

Gesture Interface and Transfer Method for AMR by Using Recognition of Pointing Direction and Object Recognition

This paper describes a gesture interface for a factory transfer robot. Our proposed interface used gesture recognition to recognize the pointing direction, instead of estimating the point as in conventional pointing gesture estimation. When the autonomous mobile robot (AMR) recognized the pointing direction, it performed position control based on the object recognition. The AMR traveled along our unique path to ensure that its camera detected the object to be referenced for position control. The experimental results confirmed that the position and angular errors of the AMR controlled with our interface were 0.058 m and 4.7° averaged over five subjects and two conditions, which were sufficiently accurate for transportation. A questionnaire showed that our interface was user-friendly compared with manual operation with a commercially available controller.

Using a Gesture Recognition Modeling approach to improve graphical user interface

The gesture recognition technology based on visual detection employs image processing for detection, segmentation, tracking, and recognition of hand gestures. Static and dynamic gesture recognition are the two types of gesture recognition. In this paper, we present our hand gesture interface consisting of three main layers: Hand Gesture Interface, Mapping Gesture, Action, and Graphical User Interface with our approach named IGMA (Improving Graphical User Interface Based on Gesture Recognition Modeling Approach). IGMA organized in two stages: the modeling stage and the code generation stage. The model is Domain Specific Model; it allows action and gesture modeling. At the code generation stage, a specific code generator transforms all models to final source code using a framework specifically designed for a given target platform. Finally, to simplify the implementation of this method, we propose defining a process for improving user interface control based on gesture recognition. We validate our approach by performing experiments on a gastronomy application. The result is an adaptation framework that can guide software engineers in developing Graphical User Interface Based on Gesture Recognition.

GESTURAL DESIGN - HAND TRACKING FOR DIGITAL DRAWING

Computational design is increasingly interested in the active feedback between the user/designer and the digital space. Often, our initial instinct as designers comes from a gesture, a movement of the hands that gets translated into sketches and 3D models via the tools available to us. While the physical realm allows for muscle memory, tactile feedback, and creative output via movement, digital design often negates the body of the designer as it sequesters us into a screen-mouse-hand relationship. Moreover, current CAD software tools often reinforce this standardization, further limiting the potential of physical bodily gestures as a vehicle for architectural form-making. Seeking new opportunities for a gestural interface, this research explores how Machine Learning and parametric design tools can be used to translate active movements and gestural actions into rich and complex digital models without the need of specialized equipment. In this paper, we present an open-source and economically accessible methodology for designers to translate hand movements into the digital world, implementing the MediaPipe Hands tracking library. In developing this workflow, this research explores opportunities to create more direct, vital links between expressive gesture and architectural form, with an emphasis on creating platforms that are accessible not only to design experts, but also the broader public.

Gesture Vocabularies for Hand Gestures for Controlling Air Conditioners in Home and Vehicle Environments

With the growing prevalence of modern technologies as part of everyday life, mid-air gestures have become a promising input method in the field of human–computer interaction. This paper analyses the gestures of actual users to define a preliminary gesture vocabulary for home air conditioning (AC) systems and suggests a gesture vocabulary for controlling the AC that applies to both home and vehicle environments. In this study, a user elicitation experiment was conducted. A total of 36 participants were filmed while employing their preferred hand gestures to manipulate a home air conditioning system. Comparisons were drawn between our proposed gesture vocabulary (HomeG) and a previously proposed gesture vocabulary which was designed to identify the preferred hand gestures for in-vehicle air conditioners. The findings indicate that HomeG successfully identifies and describes the employed gestures in detail. To gain a gesture taxonomy that is suitable for manipulating the AC at home and in a vehicle, some modifications were applied to HomeG based on suggestions from other studies. The modified gesture vocabulary (CrossG) can identify the gestures of our study, although CrossG has a less detailed gesture pattern. Our results will help designers to understand user preferences and behaviour prior to designing and implementing a gesture-based user interface.

From Warfighting Needs to Robot Actuation: A Complete Rapid Integration Swarming Solution

Swarm robotics systems have the potential to transform warfighting in urban environments but until now have not seen large-scale field testing. We present the Rapid Integration Swarming Ecosystem (rise), a platform for future multi-agent research and deployment. rise enables rapid integration of third-party swarm tactics and behaviors, which was demonstrated using both physical and simulated swarms. Our physical testbed is composed of more than 250 networked heterogeneous agents and has been extensively tested in mock warfare scenarios at five urban combat training ranges. rise implements live, virtual, constructive simulation capabilities to allow the use of both virtual and physical agents simultaneously, while our “fluid fidelity” simulation enables adaptive scaling between low and high fidelity simulation levels based on dynamic runtime requirements. Both virtual and physical agents are controlled with a unified gesture-based interface that enables a greater than 150:1 agent-to-operator ratio. Through this interface, we enable efficient swarm-based mission execution. rise translates mission needs to robot actuation with rapid tactic integration, a reliable testbed, and efficient operation.

Usability of Cross-Device Interaction Interfaces for Augmented Reality in Physical Tasks

The shortcomings of established input methods for Augmented Reality (AR) head-mounted displays (HMDs) motivate us to investigate the use of AR HMDs with smartphones and smartwatches to improve AR interaction in physical tasks. However, it is unclear whether the cross-device interaction interfaces are efficient for AR systems on physical tasks because the physical tasks can break the interaction flow. In this work, we conducted a user study to explore this. The user study consists of three subtasks, respectively requiring one of the three representative user interface (UI) controls (buttons, sliders, or text input). We implemented them with mid-air gestures, a smartphone, and a smartwatch. We compared the three interfaces and found that the smartphone and the smartwatch interaction interfaces have greater usability, provide a better user experience, and have lower workloads than the mid-air gesture interface. However, there is no significant difference in terms of efficiency for simple interactions. We discuss the limitations of this research and directions for future work.

Interactive 3D touch and gesture capable holographic light field display with automatic registration between user and content

AbstractThis paper presents an interactive three‐dimensional (3D) touch and gesture user interface that utilizes a light‐field display and a 3D sensor. In contrast to conventional gesture interfaces, in which usually the image space and gesture space are separated, a user can directly interact with the 3D image in this interface system. To achieve this trait, the 3D registration between the sensor and display coordinate systems is necessary. In this study, we propose a method to automatically match the position of the reconstructed 3D image with the position of the hand of the user based on scattered light detection. The proposed calibration process is automatic and enables self‐calibration during practical operation without laborious measurement work. The method allows us to use the gesture detection capabilities of the leap motion controller to develop an interface where content and gesture positions match. We show some examples of 3D touch and gesture interactions in which the gesture position and the generated content match on a light field display.

DAVE: Deep Learning-Based Asymmetric Virtual Environment for Immersive Experiential Metaverse Content

In this study, we design an interface optimized for the platform by adopting deep learning in an asymmetric virtual environment where virtual reality (VR) and augmented reality (AR) users participate together. We also propose a novel experience environment called deep learning-based asymmetric virtual environment (DAVE) for immersive experiential metaverse content. First, VR users use their real hands to intuitively interact with the virtual environment and objects. A gesture interface is designed based on deep learning to directly link gestures to actions. AR users interact with virtual scenes, objects, and VR users via a touch-based input method in a mobile platform environment. A text interface is designed using deep learning to directly link handwritten text to actions. This study aims to propose a novel asymmetric virtual environment via an intuitive, easy, and fast interactive interface design as well as to create metaverse content for an experience environment and a survey experiment. This survey experiment is conducted with users to statistically analyze and investigate user interface satisfaction, user experience, and user presence in the experience environment.

QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion Controller

Despite the tremendous progress made for recognizing gestures acquired by various devices, such as the Leap Motion Controller, developing a gestural user interface based on such devices still induces a significant programming and software engineering effort before obtaining a running interactive application. To facilitate this development, we present QuantumLeap, a framework for engineering gestural user interfaces based on the Leap Motion Controller. Its pipeline software architecture can be parameterized to define a workflow among modules for acquiring gestures from the Leap Motion Controller, for segmenting them, recognizing them, and managing their mapping to functions of the application. To demonstrate its practical usage, we implement two gesture-based applications: an image viewer that allows healthcare workers to browse DICOM medical images of their patients without any hygiene issues commonly associated with touch user interfaces and a large-scale application for managing multimedia contents on wall screens. To evaluate the usability of QuantumLeap, seven participants took part in an experiment in which they used QuantumLeap to add a gestural interface to an existing application.