3D Gesture Research Articles

ALHK: Integrating 3D Holograms and Gesture Interaction for Elementary Education

The integration of technology in elementary education offers innovative ways to enhance learning. One such advancement is the use of three-dimensional holograms (3DH), which provide immersive displays that merge seamlessly with the learner’s environment, creating a dynamic and engaging atmosphere. Educators have found that 3D visual tools significantly improve student comprehension, with 94.4% agreeing in a preliminary study. However, using interactive 3D holography alone has limitations, such as the inability for students to physically touch or manipulate holographic objects. To address this, Active Learning with Holo-Kid (ALHK) is introduced as a desktop application for elementary school students (grades 1 to 6). ALHK combines Leap Motion technology’s precision with interactive 3D holography to overcome these limitations. The combination allows students to interact with virtual objects in a more immersive and realistic manner. Holograms provide visual representation, while Leap Motion enables precise gesture recognition and hand tracking, resulting in a seamless and intuitive user experience. Initial evaluations demonstrate improved student engagement and comprehension. Future iterations aim to enhance scalability by incorporating features like custom object upload, multi-user interaction, and broader age applicability. ALHK shows promise as a tool for creating an immersive and intuitive learning environment using 3D holograms and interactive technology in elementary education.

AL-MobileNet: a novel model for 2D gesture recognition in intelligent cockpit based on multi-modal data

As the degree of automotive intelligence increases, gesture recognition is gaining more attention in human-vehicle interaction. However, existing gesture recognition methods are computationally intensive and perform poorly in multi-modal sensor scenarios. This paper proposes a novel network structure, AL-MobileNet (MobileNet with Attention and Lightweight Modules), which can quickly and accurately estimate 2D gestures in RGB and infrared (IR) images. The innovations of this paper are as follows: Firstly, to enhance multi-modal data, we created a synthetic IR dataset based on real 2D gestures and employed a coarse-to-fine training approach. Secondly, to speed up the model's computation on edge devices, we introduced a new lightweight computational module called the Split Channel Attention Block (SCAB). Thirdly, to ensure the model maintains accuracy in large datasets, we incorporated auxiliary networks and Angle-Weighted Loss (AWL) into the backbone network. Experiments show that AL-MobileNet requires only 0.4 GFLOPs of computational power and 1.2 million parameters. This makes it 1.5 times faster than MobileNet and allows for quick execution on edge devices. AL-MobileNet achieved a running speed of up to 28 FPS on the Ambarella CV28. On both general datasets and our dataset, our algorithm achieved an average PCK0.2 score of 0.95. This indicates that the algorithm can quickly generate accurate 2D gestures. The demonstration of the algorithm can be reviewed in gesturebaolong.

Early gesture detection in untrimmed streams: A controlled CTC approach for reliable decision-making

This paper focuses on the problem of online action detection for interactive systems, with a special emphasis on earliness. Online Action Detection (OAD) refers to the challenging task of recognizing gestures in untrimmed, streaming videos where the actions occur in unpredictable orders and durations. To address these challenges, we present a skeleton-based system for OAD incorporating a decision mechanism to accurately detect ongoing gestures. This allows us to provide instance-level output, achieving a high level of stream understanding. This mechanism relies on a novel Connectionist Temporal Classification (CTC) loss design that restricts the path possibilities according to the action boundaries. We also present a mechanism to tune the trade-off between accuracy and earliness according to the needs of the interactive system using a weighted label prior. This system includes a 3D CNN network, referred to as DOLT-C3D, exploiting the spatial–temporal information provided by the euclidean skeleton representation. We extensively evaluate our approach on eight publicly available datasets, demonstrating its superior performance compared to state-of-the-art methods in terms of both accuracy and earliness. We also successfully applied our approach to early 2D gestures detection. Furthermore, our system shows real-time performance, making it a suitable choice for interactive systems.

SwivelTouch: Boosting Touchscreen Input with 3D Finger Rotation Gesture

Today, touchscreens stand as the most prevalent input devices of mobile computing devices (smartphones, tablets, smartwatches). Yet, compared with desktop or laptop computers, the limited shortcut keys and physical buttons on touchscreen devices, coupled with the fat finger problem, often lead to slower and more error-prone input and navigation, especially when dealing with text editing and other complex interaction tasks. We introduce an innovative gesture set based on finger rotations in the yaw, pitch, and roll directions on a touchscreen, diverging significantly from traditional two-dimensional interactions and promising to expand the gesture library. Despite active research in estimation of finger angles, however, the previous work faces substantial challenges, including significant estimation errors and unstable sequential outputs. Variability in user behavior further complicates the isolation of movements to a single rotational axis, leading to accidental disturbances and screen coordinate shifts that interfere with the existing sliding gestures. Consequently, the direct application of finger angle estimation algorithms for recognizing three-dimensional rotational gestures is impractical. SwivelTouch leverages the analysis of finger movement characteristics on the touchscreen captured through original capacitive image sequences, which aims to rapidly and accurately identify these advanced 3D gestures, clearly differentiating them from conventional touch interactions like tapping and sliding, thus enhancing user interaction with touch devices and meanwhile compatible with existing 2D gestures. User study further confirms that the implementation of SwivelTouch significantly enhances the efficiency of text editing on smartphones.

OBT-Trace: An approach to trace and recognition object motion through prosthetic hand gestures

The development of prosthetic hands has advanced significantly in recent years, aiming to provide more intuitive and functional solutions for individuals with upper limb amputations. This research presents a novel approach to prosthetic hand control by integrating 3D hand gesture recognition with object manipulation and recognition capabilities. Our proposed system utilizes a pre-trained object recognition model, based on transfer learning, to enable the prosthetic hand to perceive and identify objects in its vicinity. The model leverages a vast dataset of objects, enabling the prosthetic hand to recognize a wide array of everyday items, thus enhancing its versatility.In addition, the prosthetic hand incorporates a sophisticated 3D hand gesture recognition system, allowing users to control the hand's movements and actions seamlessly. By recognizing specific gestures, such as grasping, lifting, and releasing, users can intuitively interact with their environment and perform various tasks with ease. This research leverages the synergy between gesture recognition and object recognition, creating a powerful framework for prosthetic hand control. The system's adaptability and versatility make it suitable for a broad range of applications, from assisting with daily tasks to enhancing the quality of life for individuals with upper limb amputations.The results of this study demonstrate the feasibility and effectiveness of combining 3D hand gesture recognition with pre-trained object recognition through transfer learning. This approach opens up new possibilities for enhancing prosthetic hand functionality and usability, ultimately improving the lives of those who rely on these devices for daily living. The proposed model combines the features of YOLO V7 object detection with pre-trained models. The proposed model achieves 99.8% of accuracy compared to the existing models.

Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control

This study aims to improve the generation of 3D gestures by utilizing multimodal information from human speech. Previous studies have focused on incorporating additional modalities to enhance the quality of generated gestures. However, these methods perform poorly when certain modalities are missing during inference. To address this problem, we suggest using speech-derived multimodal priors to improve gesture generation. We introduce a novel method that separates priors from speech and employs multimodal priors as constraints for generating gestures. Our approach utilizes a chain-like modeling method to generate facial blendshapes, body movements, and hand gestures sequentially. Specifically, we incorporate rhythm cues derived from facial deformation and stylization prior based on speech emotions, into the process of generating gestures. By incorporating multimodal priors, our method improves the quality of generated gestures and eliminate the need for expensive setup preparation during inference. Extensive experiments and user studies confirm that our proposed approach achieves state-of-the-art performance.

3D Motion Gesture Control : Gesture Recognition and Adaptation for Human Computer Interaction

Advancements in human-computer interaction (HCI) have paved the way for more intuitive and immersive interfaces. The first part of the paper delves into the fundamental principles of 3D gesture recognition, including sensor technologies, machine learning algorithms, and computer vision techniques. It discusses the challenges associated with accurate recognition in various environmental conditions and the ways in which these challenges are being addressed by researchers. The second part focuses on the adaptation aspect of the technology. It highlights how 3D gesture recognition can be integrated into adaptive HCI systems, enabling personalized and context-aware interactions. These adaptations can range from adjusting the interface layout to suit the user's preferences to dynamically changing the system's behavior based on the user's gestures. Additionally, the paper discusses the potential applications of 3D gesture recognition in fields such as gaming, virtual reality, healthcare, and beyond. It emphasizes the need for continued research to improve accuracy, robustness, and user-friendliness, ultimately driving the widespread adoption of 3D gesture recognition in HCI.

A TinyML Model for Gesture-Based Air Handwriting Arabic Numbers Recognition

In an era where the demand for efficient and practical machine learning (ML) solutions on resource-constrained devices is evergrowing, the realm of tiny machine learning (TinyML) emerges as a promising frontier. Motivated by the need for lightweight, low-power models that can be deployed on edge devices, this research paper presents an innovative TinyML model tailored to recognize Arabic hand gestures executed in mid-air. With a primary emphasis on the precise classification of Arabic numbers through these expressive hand movements, the paper unveils a comprehensive dataflow architecture. This intricate architecture processes accelerometer and gyroscope data to derive exact 2D gesture coordinates, a fundamental component of the recognition process. The cornerstone of the proposed model is the integration of Convolutional Neural Networks (CNNs), elucidating their exceptional role in achieving an impressive 93.8% accuracy rate in the classification of diverse Arabic Numbers gestures. This remarkable level of precision underscores the model's efficacy and resilience, rendering it an ideal candidate for real-time deployment in various gesture recognition scenarios.

Traffic Police 3D Gesture Recognition Based on Spatial–Temporal Fully Adaptive Graph Convolutional Network

It is critical for autonomous vehicles to recognize traffic police gestures timely and accurately. During the movement of the vehicle, the collected traffic police scales change all the time, in addition, the frequency and amplitude of actions of different traffic police are different. First, we use gesture normalization to fix the traffic police actions at a unified scale and remove the influence of scale changes on traffic police gesture recognition. Meanwhile, a fully adaptive spatial-temporal graph convolution network (FA-STGCN) is proposed to recognize the actions with different amplitude and frequencies. The adaptive spatial graph network can dig the latent joints connection relation of the traffic police under different gestures, which weakens the amplitude impact on the action recognition. The adaptive temporal graph network is composed of the global temporal module and the local temporal module. The global temporal module can obtain the coarse-grained features of the traffic police gestures’ speed and then naturally use the coarse-grained features to guide the local temporal module to adaptively learn the fine-grained temporal features of the traffic police action. The adaptive spatial graph network and the temporal graph network are alternately stacked to finally output accurate traffic police gestures. We thoroughly evaluated our method through intensive experiments, the result shows that our method achieved the best results on public datasets. What’s more, we proofed the effectiveness of each module and verified our methods for moving vehicles for the first time, the performance present meets the vehicle’s practical requirements.

Bodyformer: Semantics-guided 3D Body Gesture Synthesis with Transformer

Automatic gesture synthesis from speech is a topic that has attracted researchers for applications in remote communication, video games and Metaverse. Learning the mapping between speech and 3D full-body gestures is difficult due to the stochastic nature of the problem and the lack of a rich cross-modal dataset that is needed for training. In this paper, we propose a novel transformer-based framework for automatic 3D body gesture synthesis from speech. To learn the stochastic nature of the body gesture during speech, we propose a variational transformer to effectively model a probabilistic distribution over gestures, which can produce diverse gestures during inference. Furthermore, we introduce a mode positional embedding layer to capture the different motion speeds in different speaking modes. To cope with the scarcity of data, we design an intra-modal pre-training scheme that can learn the complex mapping between the speech and the 3D gesture from a limited amount of data. Our system is trained with either the Trinity speech-gesture dataset or the Talking With Hands 16.2M dataset. The results show that our system can produce more realistic, appropriate, and diverse body gestures compared to existing state-of-the-art approaches.

JengASL: A Gamified Approach to Sign Language Learning in VR

Learning sign language has many advantages ranging from being able to communicate with millions of hearing impaired people, to improving cognitive function and communication skills. Sign language is recognised as an official language in 74 countries, including Germany, Japan, and the UK. Despite that only a small percentage of people attempt to learn sign language. In this research we investigate how virtual reality and gamification can be used to make learning sign language more enjoyable and motivating. We present JengASL, a gamified approach using 3D hand models, gesture recognition, and interactive gameplay in Virtual Reality to teach American Sign Language. We evaluate this system with a pilot study using eight participants and found that while it is less effective for sign memorisation than traditional 2D image-based learning methods, learning is more, but not significantly more, enjoyable and motivating.

EMG-based 3D hand gesture prediction using transformer–encoder classification

One of the most common task of electromyography (EMG)-based human–machine interface (HMI) is the hand gesture recognition. Robust and accurate predictions leveraging the surface EMG (sEMG) is a key challenge. In this paper, we present a 3D hand gesture prediction application, leveraging the sEMG signal and the optical hand tracking information. A transformer–encoder classifier (TEC) module is introduced in an IPC-system to predict the 3D-hand gestures using eight monopolar channels from sEMG as input. An experimental testbed is setup to acquire, train, and predict the 3D hand gestures within a feasible range of performance. The performance has been evaluated in terms of percentage of correctly classified keypoints (PCK). PCK is measured by first estimating the euclidean distance between the actual and the predicted keypoints. The percentage of keypoints within a threshold distance value are then calculated. Results from the ablation study indicate that the proposed scheme shows a percentage of correctly classified keypoints of up to 72.8%, 92.7%, 97.2%, and 98.6% with a PCK threshold of 5 mm, 10 mm, 20 mm, and 30 mm, respectively.

Intelligent hybrid hand gesture recognition system using deep recurrent neural network with chaos game optimisation

ABSTRACT Hand gesture recognition is considered an essential task in various human-computer interaction (HCI) applications. Therefore, developing a robust system for hand gesture recognition is a challenge. This work proposed a new hybrid approach named hybrid deep recurrent neural network (RNN) incorporated with a chaos game optimisation (CGO) algorithm for efficiently recognising hand gestures. The main objective of a hybrid recurrent neural network with chaos game optimisation (RNN-CGO) is to achieve the recognition of alphabet signs from 2D gesture images. The different stages included in this technique are pre-processing, feature extraction, feature selection, and classification. The proposed approach is executed using American Sign Language (ASL) dataset with the help of the PYTHON platform. The precision, accuracy, f1-score, and recall obtained for the proposed approach are 99.28%, 99.96%, 99.25%, and 99.28%, respectively. The inference time required for the proposed approach is 0.121 s. The proposed technique has less computational complexity. The results showed that the proposed approach is better than the existing hand gesture recognition model approaches.

Virtual steering with gesture control using 3D hand modeling

Hand gesture consciousness has surprisingly improved research in computer vision. Hand pose estimation identifies wrist and hand key points to precisely apprehend the postures of folks given via way of video currently. So, the above step is integral perquisite to perform a couple of tasks of computer vision which embody human laptop computer interaction, sign languages, video surveillance, human motion recognition and hand gesture analysis. The advancing depth sensors are going to inspire exclusive hand gesture techniques and applications, in which these are constrained in 2D with traditional model cameras. If we check the lookup for 3D hand gesture focus in distinct factors then, it would be static hand gesture recognition, hand trajectory recognition, nonstop hand gesture recognition, 3D hand modeling. During the exploration of hand gesture cognizance approaches, meanwhile the related purposes and ordinary systems additionally summarized for practioners. From introduction by way of classifying it as a single hand or more than one hands estimation based totally on range of people needed to be tracked, then slowly the processes used in hand gesture estimation are defined earlier than listing functions and flaws going through pose estimation.

Kinesiological Approach to the Interpretation of Gestures in a Written Literary Text

The interpretation of the universal hand gestures of spreading arms and clapping hands in a written literary text is considered using the kinesiological approach to non-verbal communication of participants while speaking, which is currently widely developed. This approach makes it possible to identify the structuring role of the human body in the implementation of gestures based on the biomechanics of individual body parts involved in the process of gesticulation. For movement analysis researchers of French Sign Language designed a real-time 3D gesture visualization tool named ThirdEye to help study the importance of movement in the production of meaning within the context of sign language. Some studies support the idea that research on gestures accompanying speech can provide ways of studying speakers’ conceptualization of grammatical notions as they are speaking. So, the research on co-verbal gestures in French shows that gestures marking a boundary are predominantly associated with the Passé composé (perfect tense) whereas the gestures without boundaries are more often associated with the Imparfait (indefinite tense).As to literary text, it has been proved that the material embodiment of a gesture will obviously differ in each individual case, taking into account the author’s commentary, which gives an ambiguous expressive and evaluative characteristic of the action being performed. The importance of reliance on the extralinguistic context is emphasized, which increases significantly in the situation of translating a text that interprets the body language of the “third” culture representatives who are not native speakers of the source or the target languages. Given the lack of a visual representation illustrating a specific gesture, the interpretation of the relevant fragments of the text requires an extensive cognitive baggage and a developed creative imagination of the translator.

Linked motion image‐based dynamic hand gesture recognition

AbstractThe researchers have paid significant attention to dynamic images for hand gesture recognition. Dynamic images are gesture representation patterns that simultaneously capture spatial, temporal, and structural information from the video. Existing techniques to generate dynamic images provide low discriminability for the gestures, which follow the same trajectory, but in opposite directions, such as “swiping hand right” versus “swiping hand left.” Also, limited to similar gestures such as “Snap fingers” versus “Dual fingers heart.” To address these issues, we have proposed an algorithm to convert a depth video into a single dynamic image known as a linked motion image (LMI). We give the LMI to a classifier consisting of an ensemble of three modified pretrained convolutional neural networks. We conduct the experiments using a multimodal large‐scale EgoGesture dataset and The MSR Gesture 3D dataset. For the EgoGesture dataset, the proposed method achieved an accuracy of 92.91%, which is better than the state‐of‐the‐art methods. For the MSR Gesture 3D dataset, the proposed method accuracy is 100%, which outperforms the state‐of‐the‐art methods. This work also highlights the recognition accuracy and precision of each gesture. The experiments demonstrate the work's economic efficiency using a web‐based data science environment called Kaggle rather than high‐end systems like GPU.

Anomaly detection by using a combination of generative adversarial networks and convolutional autoencoders

With the development of full digitalization, the amount of time series data generated by sensors is ever-increasing; thus, time series outlier detection has become crucial. Moreover, in practice, discovering and flagging anomalies is very time-consuming and expensive. To solve this problem, unsupervised anomaly detection methods have often been used in the past, in which the model is trained with normal data to learn its behavioral patterns. Generative adversarial networks (GANs) can simulate complex and high-dimensional distributions of data and can be used to learn the behavioral patterns of normal data for unsupervised anomaly detection. However, because of the problem of convergence, GANs are difficult to train. Thus, USADs (an unsupervised anomaly detection model) utilize an autoencoder (AE) to undertake the task of the generator and discriminator and enhance the stability during adversarial training by using the AE to alleviate the problem of non-convergence encountered in GANs. Therefore, in this study, we used the USAD’s generative adversarial training architecture combined with convolutional AEs to improve the model’s feature extraction capabilities. In addition, to reduce false-positive outcomes caused by the prominent sharp points in the reconstructed data, we used the exponential weighted moving average method to smooth the reconstruction error, thereby improving the anomaly detection accuracy of the model. Finally, we experimented with real-world time-series data (ECG and 2D gesture) and verified that our approach could improve accuracy. Compared to the best in the comparison method, our model improved by 0.028% in AUROC, 0.233% in AUPRC, and 0.187% in F1 on average.

Printed 3D Gesture Recognition Thermoformed Half Sphere Compatible with In‐Mold Electronic Applications

The development of conformable electronic systems’ focuses on the multifunctional smart materials transform areas from the automotive industry to biomedicine, based on improved integration and advanced functionalities. In‐mold electronics (IME) combines thermoforming, injection molding, and printed electronics, resulting in sensing structural parts integrated into smart surfaces, communicating with computer systems, and enabling advanced experiences in terms of human‐machine interaction. Contrary to flexible electronics, designed as bend conductors around a single axis, thermoformed electronics allow deformations in the three‐axial directions for different geometries. Herein, a 2D screen‐printed pattern using conductive silver ink thermoformed into a spherical surface suitable for 3D gesture recognition is reported. The mechanical deformation of the high‐impact polystyrene sheet (HIPS) and polyethylene terephthalate‐glycol (PETG) substrates during the thermoforming process of the electrical circuit is characterized experimentally. The electrical resistance changes through the circuit in the stages of the fabrication process due the thickness decrease and the propagation of the small cracks. Both evaluations contribute to a better understanding of the performance of the materials when subject to the IME process. Validation of the 3D gesture recognition operation is carried out, demonstrating their full operability, as well as the possibilities that the combination of technologies allows.

Interactive 3D touch and gesture capable holographic light field display with automatic registration between user and content

AbstractThis paper presents an interactive three‐dimensional (3D) touch and gesture user interface that utilizes a light‐field display and a 3D sensor. In contrast to conventional gesture interfaces, in which usually the image space and gesture space are separated, a user can directly interact with the 3D image in this interface system. To achieve this trait, the 3D registration between the sensor and display coordinate systems is necessary. In this study, we propose a method to automatically match the position of the reconstructed 3D image with the position of the hand of the user based on scattered light detection. The proposed calibration process is automatic and enables self‐calibration during practical operation without laborious measurement work. The method allows us to use the gesture detection capabilities of the leap motion controller to develop an interface where content and gesture positions match. We show some examples of 3D touch and gesture interactions in which the gesture position and the generated content match on a light field display.

Learning Unseen Emotions from Gestures via Semantically-Conditioned Zero-Shot Perception with Adversarial Autoencoders

We present a novel generalized zero-shot algorithm to recognize perceived emotions from gestures. Our task is to map gestures to novel emotion categories not encountered in training. We introduce an adversarial autoencoder-based representation learning that correlates 3D motion-captured gesture sequences with the vectorized representation of the natural-language perceived emotion terms using word2vec embeddings. The language-semantic embedding provides a representation of the emotion label space, and we leverage this underlying distribution to map the gesture sequences to the appropriate categorical emotion labels. We train our method using a combination of gestures annotated with known emotion terms and gestures not annotated with any emotions. We evaluate our method on the MPI Emotional Body Expressions Database (EBEDB) and obtain an accuracy of 58.43%. We see an improvement in performance compared to current state-of-the-art algorithms for generalized zero-shot learning by an absolute 25-27%. We also demonstrate our approach on publicly available online videos and movie scenes, where the actors' pose has been extracted and map to their respective emotive states.