3D Hand Gesture Research Articles

OBT-Trace: An approach to trace and recognition object motion through prosthetic hand gestures

The development of prosthetic hands has advanced significantly in recent years, aiming to provide more intuitive and functional solutions for individuals with upper limb amputations. This research presents a novel approach to prosthetic hand control by integrating 3D hand gesture recognition with object manipulation and recognition capabilities. Our proposed system utilizes a pre-trained object recognition model, based on transfer learning, to enable the prosthetic hand to perceive and identify objects in its vicinity. The model leverages a vast dataset of objects, enabling the prosthetic hand to recognize a wide array of everyday items, thus enhancing its versatility.In addition, the prosthetic hand incorporates a sophisticated 3D hand gesture recognition system, allowing users to control the hand's movements and actions seamlessly. By recognizing specific gestures, such as grasping, lifting, and releasing, users can intuitively interact with their environment and perform various tasks with ease. This research leverages the synergy between gesture recognition and object recognition, creating a powerful framework for prosthetic hand control. The system's adaptability and versatility make it suitable for a broad range of applications, from assisting with daily tasks to enhancing the quality of life for individuals with upper limb amputations.The results of this study demonstrate the feasibility and effectiveness of combining 3D hand gesture recognition with pre-trained object recognition through transfer learning. This approach opens up new possibilities for enhancing prosthetic hand functionality and usability, ultimately improving the lives of those who rely on these devices for daily living. The proposed model combines the features of YOLO V7 object detection with pre-trained models. The proposed model achieves 99.8% of accuracy compared to the existing models.

Hand Cricket Game Application Using Computer Vision

Hand cricket is an attempt to experience the childhood entrainment of playing cricket using our hands. This paper presents an application built to play Hand cricket using a computer vision-based real-time 3D hand gesture recognition system. To play the game the user will give inputs using hand gestures. These hand gestures will be identified using a real-time computer vision system that runs based on a CNN model. Since in this game both players must play simultaneously the computer also comes up with a number from 1 to 6 when a hand gesture is being read and recognized. The core engine of building this app is an image classifier that classifies a picture of a hand into 1,2,3,4,5,6 and none. This paper shows an interactive simple game application developed, that tries to make the game feel more natural and as if playing with a real person.

JengASL: A Gamified Approach to Sign Language Learning in VR

Learning sign language has many advantages ranging from being able to communicate with millions of hearing impaired people, to improving cognitive function and communication skills. Sign language is recognised as an official language in 74 countries, including Germany, Japan, and the UK. Despite that only a small percentage of people attempt to learn sign language. In this research we investigate how virtual reality and gamification can be used to make learning sign language more enjoyable and motivating. We present JengASL, a gamified approach using 3D hand models, gesture recognition, and interactive gameplay in Virtual Reality to teach American Sign Language. We evaluate this system with a pilot study using eight participants and found that while it is less effective for sign memorisation than traditional 2D image-based learning methods, learning is more, but not significantly more, enjoyable and motivating.

EMG-based 3D hand gesture prediction using transformer–encoder classification

One of the most common task of electromyography (EMG)-based human–machine interface (HMI) is the hand gesture recognition. Robust and accurate predictions leveraging the surface EMG (sEMG) is a key challenge. In this paper, we present a 3D hand gesture prediction application, leveraging the sEMG signal and the optical hand tracking information. A transformer–encoder classifier (TEC) module is introduced in an IPC-system to predict the 3D-hand gestures using eight monopolar channels from sEMG as input. An experimental testbed is setup to acquire, train, and predict the 3D hand gestures within a feasible range of performance. The performance has been evaluated in terms of percentage of correctly classified keypoints (PCK). PCK is measured by first estimating the euclidean distance between the actual and the predicted keypoints. The percentage of keypoints within a threshold distance value are then calculated. Results from the ablation study indicate that the proposed scheme shows a percentage of correctly classified keypoints of up to 72.8%, 92.7%, 97.2%, and 98.6% with a PCK threshold of 5 mm, 10 mm, 20 mm, and 30 mm, respectively.

Ultrasonic positioning and IMU data fusion for pen-based 3D hand gesture recognition

Ultrasonic positioning and IMU data fusion for pen-based 3D hand gesture recognition

Deep Reinforcement Learning Based Unmanned Aerial Vehicle (UAV) Control Using 3D Hand Gestures

The evident change in the design of the autopilot system produced massive help for the aviation industry and it required frequent upgrades. Reinforcement learning delivers appropriate outcomes when considering a continuous environment where the controlling Unmanned Aerial Vehicle (UAV) required maximum accuracy. In this paper, we designed a hybrid framework, which is based on Reinforcement Learning and Deep Learning where the traditional electronic flight controller is replaced by using 3D hand gestures. The algorithm is designed to take the input from 3D hand gestures and integrate with the Deep Deterministic Policy Gradient (DDPG) to receive the best reward and take actions according to 3D hand gestures input. The UAV consist of a Jetson Nano embedded testbed, Global Positioning System (GPS) sensor module, and Intel depth camera. The collision avoidance system based on the polar mask segmentation technique detects the obstacles and decides the best path according to the designed reward function. The analysis of the results has been observed providing best accuracy and computational time using novel design framework when compared with traditional Proportional Integral Derivatives (PID) flight controller. There are six reward functions estimated for 2500, 5000, 7500, and 10000 episodes of training, which have been normalized between 0 to −4000. The best observation has been captured on 2500 episodes where the rewards are calculated for maximum value. The achieved training accuracy of polar mask segmentation for collision avoidance is 86.36%.

Vision Based Hand Gesture Recognition Using 3D Shape Context

Hand gesture recognition is a popular topic in computer vision and makes human-computer interaction more flexible and convenient. The representation of hand gestures is critical for recognition. In this paper, we propose a new method to measure the similarity between hand gestures and exploit it for hand gesture recognition. The depth maps of hand gestures captured via the Kinect sensors are used in our method, where the 3D hand shapes can be segmented from the cluttered backgrounds. To extract the pattern of salient 3D shape features, we propose a new descriptor-3D Shape Context, for 3D hand gesture representation. The 3D Shape Context information of each 3D point is obtained in multiple scales because both local shape context and global shape distribution are necessary for recognition. The description of all the 3D points constructs the hand gesture representation, and hand gesture recognition is explored via dynamic time warping algorithm. Extensive experiments are conducted on multiple benchmark datasets. The experimental results verify that the proposed method is robust to noise, articulated variations, and rigid transformations. Our method outperforms state-of-the-art methods in the comparisons of accuracy and efficiency.

3D Hand Gestures Segmentation and Optimized Classification Using Deep Learning

Hand gestures recognition system has massive applications which are mainly utilized in robotics and computer vision specially to control Unmanned Aerial Vehicles (UAV). These methods bypass the presence of electronic control to UAVs and provide an ease to the operators. In this paper, we present a method for 3D hand gestures segmentation and classification by combining MASK-RCNN with Grass Hopper Optimization. We created a private 3D and RGB hand gestures dataset using Intel Kinetic and Intel Real sense d435i camera, then proposed a model for RGB hand gestures to estimate the key points using human kinematics, the key points later then utilize to get the best degree of freedom (DoF). The grass hopper optimization besides minimum distance function was applied to achieve the finest deep features from the 3D hand gestures dataset. The ResNet50 network is used as the backbone to calculate the Overlap Coefficient (OC) for segmentation and the ResNet50, ResNet101 networks to calculate the classification for 3D hand gestures. The classification accuracy achieved on the private dataset is 99.05% and 99.29% on public Microsoft Kinect and Leap Motion dataset where the OC are 88.16%. and 88.19% respectively.

DeepHandsVR: Hand Interface Using Deep Learning in Immersive Virtual Reality

This paper proposes a hand interface through a novel deep learning that provides easy and realistic interactions with hands in immersive virtual reality. The proposed interface is designed to provide a real-to-virtual direct hand interface using a controller to map a real hand gesture to a virtual hand in an easy and simple structure. In addition, a gesture-to-action interface that expresses the process of gesture to action in real-time without the necessity of a graphical user interface (GUI) used in existing interactive applications is proposed. This interface uses the method of applying image classification training process of capturing a 3D virtual hand gesture model as a 2D image using a deep learning model, convolutional neural network (CNN). The key objective of this process is to provide users with intuitive and realistic interactions that feature convenient operation in immersive virtual reality. To achieve this, an application that can compare and analyze the proposed interface and the existing GUI was developed. Next, a survey experiment was conducted to statistically analyze and evaluate the positive effects on the sense of presence through user satisfaction with the interface experience.

A hand gesture action-based emotion recognition system by 3D image sensor information derived from Leap Motion sensors for the specific group with restlessness emotion problems

The popular 3D image sensor on market urges fast developments of hand gesture-based applications. With utilizations of 3D space hand gesture data, hand gesture command classifications for control of the specific target device, such as smart speaker and smart TV equipments. However, context aware cognition such as human emotion recognition using 3D hand gesture action characteristics has been extremely rare to be explored. In this work, focused on the specific group with the restlessness emotion problem, a hand gesture action-based emotion recognition system with gesture-making user identification using the well-known Leap Motion sensor is developed. With acquiring 3D-space action variation information extracted from the gesture action made by the person, recognition of the specific human emotion categorization can then be done. In this study, there are ten different degrees of restlessness emotion behaviors are defined according to the designed restlessness emotion hand gesture categorization table that denotes restlessness characteristics of a restlessness hand gesture action, speed, repeat, attack. Three types of feature parameters derived from the Leap Motion sensor are presented, 78-dimension data with 3D hand gesture action variations, 7-dimension data with physical characteristics of the hand and 85-dimension data with combinations of 78-dimension and 7-dimension data. The presented 7-dimension feature parameter set is employed on hand gesture-making user identification, and the other two designed feature parameter sets are carried out for classifying the defined hand gesture actions with different restlessness emotion degrees. The popular K-nearest neighbor (KNN) approach is adopted to be as a gesture data classifier for performance evaluations of all these feature parameters in this study. Experiments on recognition calculations of 4 subjects in a laboratory office show that KNN with 85-dimension feature data has the averaged recognition accuracy of 80.6%, slightly superior to 80% of 78-dimension feature data. KNN with the 7-dimension feature parameter set will have the averaged performance of 66.1% on gesture-making user identification.

3D Hand Gesture Representation and Recognition through Deep Joint Distance Measurements

Hand gestures with finger relationships are among the toughest features to extract for machine recognition. In this paper, this particular research challenge is addressed with 3D hand joint features extracted from distance measurements which are then colour mapped as spatio temporal features. Further patterns are learned using an 8-layer convolutional neural network (CNN) to estimate the hand gesture. The results showed a higher degree of recognition accuracy when compared to similar 3D hand gesture methods. The recognition accuracy for our dataset KL 3DHG with 220 classes was around 94.32%. Robustness of the proposed method was validated with only available benchmark 3D skeletal hand gesture dataset DGH 14/28.

The implementation of a haptic feedback virtual reality simulation clinic with dynamic patient interaction and communication for medical imaging students.

IntroductionAn immersive virtual reality (VR) simulation clinic with dynamic patient interaction and communication was developed to facilitate the training of medical radiation science students. The software “CETSOL VR Clinic” was integrated into the Medical Imaging programme at Monash University in 2016 in order to benchmark student experiences against existing simulation techniques (Shaderware™).MethodsAn iterative approach to development, based on two cycles of user feedback, was used to develop and refine the simulated clinical environment. This environment uses realistic 3D models, embedded clinical scenarios, dynamic communication, 3D hand gesture interaction, gaze and positional stereoscopic tracking and online user capabilities using the Unity™ game and physics engines. Students’ perceptions of educational enhancement of their positioning skills following the use of the simulation tools were analysed via a 5‐point Likert scale questionnaire.ResultsStudent perception scores indicated a significant difference between simulation modalities in favour of the immersive CETSOL VR Clinic, χ2 (4, N = 92) = 9.5, P‐value <0.001.ConclusionStudent perception scores on improvement of their clinical and technical skills were higher for the hand‐positioning tasks performed with the CETSOL VR Clinic™ than with the comparative benchmark simulation that did not provide dynamic patient interaction and communication.

“Floating heart” application of holographic 3D imaging in nursing education

This study combined 3D holographic imaging and hand gesture recognition technologies to develop an intuitive and control-based interactive holographic imaging teaching aid system for practical application in nursing education to enhance teaching effectiveness. Cardiovascular structure was developed through 3D scanning, which was then subjected to postproduction to be converted into a 3D vector layer object. After segmenting the 3D object into different images with different viewing angles, a self-made holographic projection display equipment was used to project a 3D floating image to enhance the on-site experience of instructors and learners. By using a gesture sensing module, instructors could freely operate teaching objects by using intuitive gesture controls during on-site teaching. This system was applied to the 2-year adult nursing curriculum offered by the nursing department. After the coursereflections of 90 students were summarized into the following categories: rescue imagination, conquer th. abstract, transform 2D into 3D, facilitate self-controlled learning, and dynamically understand memory. The system effectively enhanced the convenience of teaching as well as the learning motivation and effectiveness among students. In the future, we hope that this system is applied to other professional courses and becomes a powerful on-site teaching aid.

Survey on 3D Hand Gesture Recognition

Three-dimensional hand gesture recognition has attracted increasing research interests in computer vision, pattern recognition, and human-computer interaction. The emerging depth sensors greatly inspired various hand gesture recognition approaches and applications, which were severely limited in the 2D domain with conventional cameras. This paper presents a survey of some recent works on hand gesture recognition using 3D depth sensors. We first review the commercial depth sensors and public data sets that are widely used in this field. Then, we review the state-of-the-art research for 3D hand gesture recognition in four aspects: 1) 3D hand modeling; 2) static hand gesture recognition; 3) hand trajectory gesture recognition; and 4) continuous hand gesture recognition. While the emphasis is on 3D hand gesture recognition approaches, the related applications and typical systems are also briefly summarized for practitioners.

An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition

In this paper, we present an Image-to-Class Dynamic Time Warping (I2C-DTW) approach for the recognition of both 3D static hand gestures and 3D hand trajectory gestures. Our contribution is twofold. First, we propose a technique to compute the image-to-class dynamic time warping distance instead of the Image-to-Image distance. By doing so, we obtain better generalization capability using the Image-to-Class distance than the Image-to-Image distance. Second, we propose a compositional model called fingerlets for static gesture representation, and a compositional model called strokelets for trajectory gesture representation. The compositional models make it possible to compute the DTW distance between a data sample and a gesture category. We have evaluated the static gesture recognition performance on several public 3D hand gesture datasets. For better evaluating the performance on trajectory gesture recognition, we collected a 3D hand trajectory gesture dataset, called UESTC-HTG, using a Kinect device. The experiment results show that the proposed I2C-DTW approach significantly improves the recognition accuracy on both static gestures and trajectory gestures.

Control and Guidance of Low-Cost Robots via Gesture Perception for Monitoring Activities in the Home.

This paper describes the development of a low-cost mini-robot that is controlled by visual gestures. The prototype allows a person with disabilities to perform visual inspections indoors and in domestic spaces. Such a device could be used as the operator's eyes obviating the need for him to move about. The robot is equipped with a motorised webcam that is also controlled by visual gestures. This camera is used to monitor tasks in the home using the mini-robot while the operator remains quiet and motionless. The prototype was evaluated through several experiments testing the ability to use the mini-robot’s kinematics and communication systems to make it follow certain paths. The mini-robot can be programmed with specific orders and can be tele-operated by means of 3D hand gestures to enable the operator to perform movements and monitor tasks from a distance.

주거 공간에서의 3차원 핸드 제스처 인터페이스에 대한 사용자 요구사항

The aim of this study is to find out the user’s needs of a 3D hand gesture interface in the smart home environment. To find out the users’ needs, we investigated which object the users want to use with a 3D hand gesture interface and why they want to use a 3D hand gesture interface. 3D hand gesture interfaces are studied to be applied to various devices in the smart environment. 3D hand gesture interfaces enable the users to control the smart environment with natural and intuitive hand gestures. With these advantages, finding out the user’s needs of a 3D hand gesture interface would improve the user experience of a product. This study was conducted using a diary method to find out the user’s needs with 20 participants. They wrote the needs of a 3D hand gesture interface during one week filling in the forms of a diary. The form of the diary is comprised of who, when, where, what and how to use a 3D hand gesture interface with each consisting of a usefulness score. A total of 322 data (209 normal data and 113 error data) were collected from users. There were some common objects which the users wanted to control with a 3D hand gesture interface and reasons why they want to use a 3D hand gesture interface. Among them, the users wanted to use a 3D hand gesture interface mostly to control the light, and to use a 3D hand gesture interface mostly to overcome hand restrictions. The results of this study would help develop effective and efficient studies of a 3D hand gesture interface giving valuable insights for the researchers and designers. In addition, this could be used for creating guidelines for 3D hand gesture interfaces. †

3D Hand Gesture Based Computer Interaction for Non Specific User

3D Hand Gesture Based Computer Interaction for Non Specific User

Three-Dimensional Hand Tracking Based on Microstructure of Hand Motion

At present, hand tracking based on particle filter obtains current 3D hand gesture mainly by two elements, that is, by current frame of gesture images and by status information of the previous 3D hand gesture. Compared with current algorithms, the proposed algorithm focuses on behavior analysis and process modeling. Focusing on mathematical linkages establishment between microstructure model and the current frame gesture, the status of current 3D hand gesture is obtained not only on the basis of local frame information, but gesture model of overall movement process. After that, a unified and efficient data structure is presented, upon which the particle number can be reduced. First, hand motion law in specific circumstance is obtained by experiments based on data glove in the three-dimensional virtual platform. Second, the law is summarized as the microstructure of hand motion. Hand motion model is put forward and applied to forecast period of the particle filter algorithm, upon which hand tracking can be guided by the summarized microstructure. At last, hand tracking algorithm is carried out by a camera-oriented virtual three-dimensional interactive platform. The experimental result shows that, compared with Gaussian Process Annealed Particle Filter (GPAPF) gesture tracking algorithm, the time cost of the proposed algorithm was nearly 29% lower, while the accuracy 26% higher.

A taxonomy and notation method for three-dimensional hand gestures

Recently, studies on gesture-based interfaces have made an effort to improve the intuitiveness of gesture commands by asking users to define a gesture for a command. However, there are few methods to organize and notate user-defined gestures in a systematic approach. To resolve this, we propose a three-dimensional (3D) Hand Gesture Taxonomy and Notation Method. We first derived elements of a hand gesture by analyzing related studies and subsequently developed the 3D Hand Gesture Taxonomy based on the elements. Moreover, we devised a Notation Method based on a combination of the elements and also matched a code to each element for easy notation. Finally, we have verified the usefulness of the Notation Method by training participants to notate hand gestures and by asking another set of participants to recreate the notated gestures. In short, this research proposes a novel and systematic approach to notate hand gesture commands. Relevance to industryThis study develops a 3D Hand Gesture Taxonomy and Notation Method. The results of this study can be used as a guideline to organize hand gestures for enhancing the usability of gesture-based interfaces.