Gestural Activity Research Articles

Multi-scale control and action recognition based human-robot collaboration framework facing new generation intelligent manufacturing

Facing the new generation intelligent manufacturing, traditional manufacturing models are transitioning towards large-scale customized productions, improving the efficiency and flexibility of complex manufacturing processes. This is crucial for enhancing the stability and core competitiveness of the manufacturing industry, and human-robot collaboration systems are an important means to achieve this goal. At present, mainstream manufacturing human-robot collaboration systems are modeled for specific scenarios and actions, with poor scalability and flexibility, making it difficult to flexibly handle actions beyond the set. Therefore, this article proposes a new human-robot collaboration framework based on action recognition and multi-scale control, designs 27 basic gesture actions for motion control, and constructs a robot control instruction library containing 70 different semantics based on these actions. By integrating static gesture recognition, dynamic action process recognition, and You-Only-Look-Once V5 object recognition and positioning technology, accurate recognition of various control actions has been achieved. The recognition accuracy of 27 types of static control actions has reached 100%, and the dynamic action recognition accuracy of the gearbox assembly process based on lightweight MF-AE-NNOBJ has reached 90%. This provides new ideas for simplifying the complexity of human-robot collaboration problems, improving system accuracy, efficiency, and stability.

Multi-View Fusion Network-Based Gesture Recognition Using sEMG Data.

sEMG(surface electromyography) signals have been widely used in rehabilitation medicine in the past decades because of their non-invasive, convenient and informative features, especially in human action recognition, which has developed rapidly. However, the research on sparse EMG in multi-view fusion has made less progress compared to high-density EMG signals, and for the problem of how to enrich sparse EMG feature information, a method that can effectively reduce the information loss of feature signals in the channel dimension is needed. In this article, a novel IMSE (Inception-MaxPooling-Squeeze- Excitation) network module is proposed to reduce the loss of feature information during deep learning. Then, multiple feature encoders are constructed to enrich the information of sparse sEMG feature maps based on the multi-core parallel processing method in multi-view fusion networks, while SwT (Swin Transformer) is used as the classification backbone network. By comparing the feature fusion effects of different decision layers of the multi-view fusion network, it is experimentally obtained that the fusion of decision layers can better improve the classification performance of the network. In NinaPro DB1, the proposed network achieves 93.96% average accuracy in gesture action classification with the feature maps obtained in 300ms time window, and the maximum variation range of action recognition rate of individuals is less than 11.2%. The results show that the proposed framework of multi-view learning plays a good role in reducing individuality differences and augmenting channel feature information, which provides a certain reference for non-dense biosignal pattern recognition.

Hand Gesture Recognition Based on High-Density Myoelectricity in Forearm Flexors in Humans.

Electromyography-based gesture recognition has become a challenging problem in the decoding of fine hand movements. Recent research has focused on improving the accuracy of gesture recognition by increasing the complexity of network models. However, training a complex model necessitates a significant amount of data, thereby escalating both user burden and computational costs. Moreover, owing to the considerable variability of surface electromyography (sEMG) signals across different users, conventional machine learning approaches reliant on a single feature fail to meet the demand for precise gesture recognition tailored to individual users. Therefore, to solve the problems of large computational cost and poor cross-user pattern recognition performance, we propose a feature selection method that combines mutual information, principal component analysis and the Pearson correlation coefficient (MPP). This method can filter out the optimal subset of features that match a specific user while combining with an SVM classifier to accurately and efficiently recognize the user's gesture movements. To validate the effectiveness of the above method, we designed an experiment including five gesture actions. The experimental results show that compared to the classification accuracy obtained using a single feature, we achieved an improvement of about 5% with the optimally selected feature as the input to any of the classifiers. This study provides an effective guarantee for user-specific fine hand movement decoding based on sEMG signals.

DigCode—A generic mid-air gesture coding method on human-computer interaction

With high flexibility and rich semantic expressiveness, mid-air gesture interaction is an important part of natural human-computer interaction (HCI) and has broad application prospects. However, there is no unified representation frame for designing, recording, investigating and comparing HCI mid-air gestures. Therefore, this paper proposes an interpretable coding method, DigCode, for HCI mid-air gestures. DigCode converts the unstructured continuous actions into structured discrete string encoding. From the perspective of human cognition and expression, the research employed psychophysical methods to divide gesture actions into discrete intervals, defined the coding rules of representation in letters and numbers, and developed automated programs to enable encoding and decoding by using gesture sensors. The coding method can cover the existing representations of HCI mid-air gestures by considering human understanding and computer recognition and can be applied to HCI mid-air gesture design and gesture library construction.

A channel-fused gated temporal convolutional network for EMG-based gesture recognition

Background:EMG gesture recognition can be widely applied in many fields, such as prosthetic control and human–computer interaction, to enhance the standard of human life. Purpose:This study aims to develop a deep learning-based model to identify multiple complex gestures from raw EMG signals. Methods:We propose a new channel-fused gated temporal convolutional network. First, a channel fusion and gating mechanism is designed to improve temporal convolutional networks, allowing the model to obtain higher-level features. Second, we improve the channel fusion module by the short-term average energy to fuse the EMG signals of multi-channels more accurately. Results:The model is evaluated on a public dataset of EMG gestures, NinaPro DB5. Results demonstrated that the unbalanced accuracy rate of 53 gesture actions reaches 92.71%, and the balanced accuracy rate 74.79%. Further, ablation experiments validates the effectiveness of each model module. Conclusions:This study demonstrates our proposed approach can improve gesture recognition accuracy for complex gestures and has great potential for practical applications.

ST-TGR: Spatio-Temporal Representation Learning for Skeleton-Based Teaching Gesture Recognition.

Teaching gesture recognition is a technique used to recognize the hand movements of teachers in classroom teaching scenarios. This technology is widely used in education, including for classroom teaching evaluation, enhancing online teaching, and assisting special education. However, current research on gesture recognition in teaching mainly focuses on detecting the static gestures of individual students and analyzing their classroom behavior. To analyze the teacher's gestures and mitigate the difficulty of single-target dynamic gesture recognition in multi-person teaching scenarios, this paper proposes skeleton-based teaching gesture recognition (ST-TGR), which learns through spatio-temporal representation. This method mainly uses the human pose estimation technique RTMPose to extract the coordinates of the keypoints of the teacher's skeleton and then inputs the recognized sequence of the teacher's skeleton into the MoGRU action recognition network for classifying gesture actions. The MoGRU action recognition module mainly learns the spatio-temporal representation of target actions by stacking a multi-scale bidirectional gated recurrent unit (BiGRU) and using improved attention mechanism modules. To validate the generalization of the action recognition network model, we conducted comparative experiments on datasets including NTU RGB+D 60, UT-Kinect Action3D, SBU Kinect Interaction, and Florence 3D. The results indicate that, compared with most existing baseline models, the model proposed in this article exhibits better performance in recognition accuracy and speed.

AI-based real-time hand gesture-controlled virtual mouse

ABSTRACT Hand gesture control technology is an emerging alternative to conventional input devices like keyboards and mice, enabling users to interact with computers through natural hand movements. Although, many researchers worked on gesture recognition and its challenges, yet wide range of gestures has not been addressed. In this direction, the author proposed a novel gesture-controlled virtual mouse system that uses hand gestures and hand tip detection with a webcam or internal camera for computer navigation that accurately recognizes various gestures, including cursor movement, left and right clicks, double clicks, scrolling, drag and drop, multiple item selection, volume control. To deal with this real-world problem, the authors proposed a solution for limited space situations and assist individuals with hand impairments who cannot use a physical mouse. Furthermore, it surpasses the limitations of traditional mice, such as mechanical wear and tear, adaptability to different environments, and restricted functionalities. By leveraging hand gestures and tip detection through a camera proposed system offers an efficient and adaptable approach to operating a PC mouse. The proposed methodology uses computer vision and AI for the accurate detection of gestures, even in challenging environments. It also offers a comprehensive set of gestures for various actions and functions.

Application of machine learning algorithms to character interaction in single-player role-playing games

Character interaction in single-player role-playing games refers to the role interactions between the player character and the non-player characters (NPCs) in the game including dialogue, social interaction, cooperation, and competition, which are designed and implemented to achieve certain game goals and effects. This paper summarizes and compares machine learning algorithms applied to game development using a paper review and theoretical analysis, aiming to provide new solutions for optimizing and improving algorithms commonly used in game design. The ADEM algorithm can automate the evaluation of semantic relevance and logical coherence of character dialogues, but it is more costly in the pre-training phase and more difficult to perform domain adaptation for specific game content. The BERT algorithm can improve the speed of character dialogue generation through the training of unlabeled text, and then improve the flexibility of the dialogue through a small amount of labeled text for domain adaptation and fine-tuning, but it lacks in the generation of smooth and complex text and other aspects. The GAN algorithm can learn from a single or a small number of action sequences to achieve the transformation and mixing of character actions, thus generating multiple action gestures. However, its training process may suffer from problems such as pattern collapse leading to degradation in the quality of the generation, and it is also difficult to set a uniform objective criterion to determine the loss function.

The Path of Integrating Traditional Music Culture into Piano Playing Teaching in the Background of Internet

Abstract This paper proposes two primary pathways for online teaching and virtual music classrooms constructed using virtual reality technology and establishes a corresponding technical framework. In the online teaching mode, an online teaching platform is built, and an improved genetic algorithm is proposed for the intelligent grouping problem, which introduces the individual similarity elimination operation and incorporates simulated annealing to solve the problem of insufficient search capability. Meanwhile, in order to build a virtual music classroom, a dynamic light projection algorithm is proposed to reconstruct a three-dimensional model of human movement, and a multi-feature fusion algorithm is used to match the weights assigned to the human body's action gestures. In the evaluation of the teaching effect of the application of the technology framework, the piano art students are taken as the research object, and the experimental class and the control class are set up to carry out the teaching practice. In the evaluation of the learning effect, the completion degree of music and artistic performance of students in the experimental class was 83.65 and 82.66, which was significantly better than that of the control class (P<0.05). The overall degree of cognition, emotional attitude, and value recognition of traditional music culture was also significantly better than that of the control class (P<0.05).

A Comparative Study of Aesthetic Differences between Guangxi Folk Dance and ASEAN Dance

Abstract Clarifying the aesthetic differences between Guangxi folk dance and ASEAN dance can promote the in-depth exchange of dance art and culture between the two. In this paper, the aesthetic characteristics of Guangxi folk dance and ASEAN dance are discussed, and the HRNet network is used as the basis, combined with the sequence multi-scale feature fusion and the PCA action gesture segmentation model for the action gesture estimation. The dataset was constructed according to different kinds of Guangxi folk dance and ASEAN dance, and the aesthetic differences between Guangxi folk dance and ASEAN dance in terms of intrinsic culture and extrinsic performance were analyzed using the dataset. The results show that on the inner culture, Guangxi folk dance is mainly based on traditional culture and folk customs. ASEAN dance is more influenced by religious culture, although it has differences, but the correlation coefficients of emotional expression and national custom are 0.963 and 0.954, respectively. On the outer manifestation, Guangxi folk dance has as many as 11 kinds of action dance forms, while ASEAN dance emphasizes more on The ASEAN dance emphasizes more on the “flat, taut, straight and standing” forms. The aesthetic differences between Guangxi folk dance and ASEAN dance can promote the exchange and integration of dance art and culture between the two and provide a good artistic foundation for the in-depth promotion of the Belt and Road.

Analysis of tennis technical movement training strategy in colleges and universities under the background of big data technology

Abstract This paper first explains the tennis technical action representation and matching technology, using the Euler angle method to represent the coordinates of the tennis action gesture, realizing the deconstruction of the action gesture through the gesture-solving algorithm, and then using the weighted Euclidean distance to match the tennis gesture to the action. Secondly, we constructed the tennis technical movement training strategy for colleges and universities based on the task-driven teaching method, and gave the teaching experiment method to analyze the effectiveness of the strategy. Finally, the data was analyzed by examining the muscle characteristics of tennis technical movements and the teaching effect of task-driven teaching. The results showed that the maximal EMG integral value of the muscles was 32.91uV.s in the level 1 group and 87.91uV.s in the level 2 group, and the maximal angular velocity of the ankle-knee-hip joints was 645.23±189.42°/s and the minimum angular velocity of the hip joints was 461.87±115.08°/s in all the landing modes. 115.08°/s. The task-driven teaching method can effectively improve the students’ tennis learning and technical level and enhance the quality of tennis technical teaching in colleges and universities.

Elicitation and Evaluation of Hand-based Interaction Language for 3D Conceptual Design in Mixed Reality

Conceptual design is a fundamental stage in a design process. Traditional conceptual design tools impose limitations on the intuitive creation process of 3D objects, hindering their full potential. Applying hand-based interaction in mixed reality (MR) provides an immersive and intuitive creation mode without dimensionality transformations, facilitating conceiving ideas in 3D conceptual design. Although studies on hand-based languages have explored different gesture design methodologies and techniques from many individual perspectives, they lack holistic understanding and comprehensive suggestions for gesture application in conceptual design scenarios. This study aims to establish a set of hand-based languages that are available in cognitive, physical, and system-based aspects for 3D conceptual designs in the MR environment. A two-stage study was conducted in an MR environment, combining an elicitation design and a comprehensive evaluation experiment. Through the elicitation design, approximately 930 gesture actions, focusing on 31 targeted functionalities, were collected. By performing an evaluation experiment, a set of theoretically optimal hand-based interaction languages for 3D conceptual design was proposed, and the corresponding guidelines for hand-based interactions were clarified. Our results can further expand human-computer interactions in MR environments and inspire software builders to create novel hand-driven interaction modes or tools.

Overcoming the effect of muscle fatigue on gesture recognition based on sEMG via generative adversarial networks

Surface electromyography (sEMG)-controlled bionic prostheses have been extensively investigated recently. However, the majority of investigations pertaining to gesture recognition grounded in sEMG predominantly center their focus on the human body in a static and non-fatigued condition, thus neglecting the ramifications of muscle fatigue on the precision of gesture recognition. This study explores the effect of muscle fatigue on recognition accuracy in a gesture recognition task based on sEMG. The muscle fatigue induction experiment was designed, and eight subjects were recruited to participate in the experiment to collect the sEMG signal data sets of seven gesture actions under non-fatigue and fatigue conditions. Seven gesture actions under non-fatigue and fatigue conditions were identified by four classifiers, namely K-nearest neighbor (K-NN), support vector machine (SVM), decision tree (DT), and deep convolution neural network (CNN). The experimental results show that muscle fatigue has a great impact on the accuracy of gesture recognition. Specifically, using the four classifiers K-NN, SVM, DT and CNN trained in the non-fatigue state, the test accuracy of sEMG signals in the non-fatigue state is 96.7 %, 89.0 %, 87.3 % and 97.5 % respectively, while the test accuracy in the fatigue state is reduced to 53.3 %, 55.4 %, 45.8 % and 64.8 % respectively. In this regard, we propose a new method to overcome the effects of muscle fatigue, namely, the data enhancement method based on Wasserstein General Adversary Networks-Gradient Penalty (WGAN-GP), which is used to enhance the sEMG signal under fatigue. Through the data enhancement of the fatigue sEMG signal, the experimental results show that the final test accuracy in the fatigue state is improved by more than 20 %, which can reach 72.3 %, 80.9 %, 69.9 % and 92.1 % respectively. This shows that the method proposed by us can effectively overcome the influence of muscle fatigue on the accuracy of gesture recognition, and has made a great contribution to the improvement of the robustness of gesture recognition model based on sEMG signal.

EMG-FRNet: A feature reconstruction network for EMG irrelevant gesture recognition.

With the development of deep learning technology, gesture recognition based on surface electromyography (EMG) signals has shown broad application prospects in various human-computer interaction fields. Most current gesture recognition technologies can achieve high recognition accuracy on a wide range of gesture actions. However, in practical applications, gesture recognition based on surface EMG signals is susceptible to interference from irrelevant gesture movements, which affects the accuracy and security of the system. Therefore, it is crucial to design an irrelevant gesture recognition method. This paper introduces the GANomaly network from the field of image anomaly detection into surface EMG-based irrelevant gesture recognition. The network has a small feature reconstruction error for target samples and a large feature reconstruction error for irrelevant samples. By comparing the relationship between the feature reconstruction error and the predefined threshold, we can determine whether the input samples are from the target category or the irrelevant category. In order to improve the performance of EMG irrelevant gesture recognition, this paper proposes a feature reconstruction network named EMG-FRNet for EMG irrelevant gesture recognition. This network is based on GANomaly and incorporates structures such as channel cropping (CC), cross-layer encoding-decoding feature fusion (CLEDFF), and SE channel attention (SE). In this paper, Ninapro DB1, Ninapro DB5 and self-collected datasets were used to verify the performance of the proposed model. The Area Under the receiver operating characteristic Curve (AUC) values of EMG-FRNet on the above three datasets were 0.940, 0.926 and 0.962, respectively. Experimental results demonstrate that the proposed model achieves the highest accuracy among related research.

Novel Time-Distance Parameters Based Hand Gesture Recognition System Using Multi-UWB Radars

Hand gesture recognition (HGR) has a vital role to develop an intelligent interface for human–computer interaction. This letter proposed an automated system for HGR using three ultrawideband radars. In this system, the received radar reflection is measured for each hand gesture action which is recorded with respect to time duration and distance of hand from radar during hand motion. A time-distance matrix is formed for each gesture action, and reflection density is obtained for time-distance analysis. Furthermore, mean density function-based parameters are suggested, namely, reflection density per unit time, reflection density per unit distance, and mixed moments of time and distance with higher order functions. Features are computed in entire time-distance region and further limited to local time-distance ranges to extract the gesture distinguishable information. Computed features are ranked and given to the classification model. The achieved high accuracy justifies the primacy of features and the developed system.

GESTURAL DESIGN - HAND TRACKING FOR DIGITAL DRAWING

Computational design is increasingly interested in the active feedback between the user/designer and the digital space. Often, our initial instinct as designers comes from a gesture, a movement of the hands that gets translated into sketches and 3D models via the tools available to us. While the physical realm allows for muscle memory, tactile feedback, and creative output via movement, digital design often negates the body of the designer as it sequesters us into a screen-mouse-hand relationship. Moreover, current CAD software tools often reinforce this standardization, further limiting the potential of physical bodily gestures as a vehicle for architectural form-making. Seeking new opportunities for a gestural interface, this research explores how Machine Learning and parametric design tools can be used to translate active movements and gestural actions into rich and complex digital models without the need of specialized equipment. In this paper, we present an open-source and economically accessible methodology for designers to translate hand movements into the digital world, implementing the MediaPipe Hands tracking library. In developing this workflow, this research explores opportunities to create more direct, vital links between expressive gesture and architectural form, with an emphasis on creating platforms that are accessible not only to design experts, but also the broader public.

A Fine-grained Channel State Information-based Deep Learning System for Dynamic Gesture Recognition

Indoor gesture recognition technology is concerned with making the machine accurately recognize dynamic gestures within a certain range. Remarkably, most of this technology is based on passive recognition methods. This is quite striking because the high cost is a crucial factor in active recognition methods and ignoring this aspect can increase the reality gap. In this paper, we tend to use the fine-grained channel state information (CSI) in Wi-Fi to build a dynamic CNN-GRU-Attention (CGA) model to implement a gesture recognition system and thus alleviate this problem. Firstly, we study the influence of gestures on the amplitude and phase difference in CSI, and prove the feasibility of proposed method by analyzing the fluctuation of amplitude and phase difference under different conditions. Then, we use data processing methods such as phase correction and unwrapping with a new proposed adaptive gesture action truncation algorithm to extract the phase difference and remove redundant information, thus ensuring the validity of data. Finally, we propose to segment gesture fragment into 3-channel CSI images as input information of model. Extensive comparison experiments are conducted under the influence of different people, different indoor environments, and different sampling rates. The results show that the system has high accuracy.

Wi-NN: Human Gesture Recognition System Based on Weighted KNN

Gesture recognition, the basis of human–computer interaction (HCI), is a significant component for the development of smart home, VR, and senior care management. Most gesture recognition methods still depend on sensors worn by the user or video-based gestures for recognition, can be used for fine-grained gesture recognition. our paper implements a gesture recognition method that is independent of environment and gesture drawing direction, and it achieves gesture recognition classification by using small sample data. Wi-NN, proposed in this study, does not require the user to wear additional device. In this case, channel state information (CSI) extracted from Wi-Fi signal is used to capture the action information of the human body via CSI. After pre-processing to reduce the interference of environmental noise as much as possible, clear action information is extracted using the feature extraction method based on time domain to obtain the gesture action feature data. The gathered data are integrated with the weighted k-nearest neighbor (KNN) classification recognizer for classification task. The experiment outcomes revealed that the accuracy scores of the same gesture for different users and different gestures for the same user under the same environment were 93.1% and 89.6%, respectively. The experiments in different environments also achieved good recognition results, and by comparing with other experimental methods, the experiments in this paper have better recognition results. Evidently, good classification results were generated after the original data were processed and incorporated into the weighted KNN.

GesturalOrigins: A bottom-up framework for establishing systematic gesture data across ape species

Current methodologies present significant hurdles to understanding patterns in the gestural communication of individuals, populations, and species. To address this issue, we present a bottom-up data collection framework for the study of gesture: GesturalOrigins. By “bottom-up”, we mean that we minimise a priori structural choices, allowing researchers to define larger concepts (such as ‘gesture types’, ‘response latencies’, or ‘gesture sequences’) flexibly once coding is complete. Data can easily be re-organised to provide replication of, and comparison with, a wide range of datasets in published and planned analyses. We present packages, templates, and instructions for the complete data collection and coding process. We illustrate the flexibility that our methodological tool offers with worked examples of (great ape) gestural communication, demonstrating differences in the duration of action phases across distinct gesture action types and showing how species variation in the latency to respond to gestural requests may be revealed or masked by methodological choices. While GesturalOrigins is built from an ape-centred perspective, the basic framework can be adapted across a range of species and potentially to other communication systems. By making our gesture coding methods transparent and open access, we hope to enable a more direct comparison of findings across research groups, improve collaborations, and advance the field to tackle some of the long-standing questions in comparative gesture research.

Cheer for me: effect of non-player character audience feedback on older adult users of virtual reality exergames.

The presence of an audience and its feedback could affect people's performance and experience during an event, especially related to sports such as tennis or boxing. Similarly, in videogames, players' gameplay could be affected if there is an audience and its feedback in response to players' performance in the environment. The inclusion of an audience with non-player characters (NPC) is common in videogames in general. However, there is a limited exploration of the use of an NPC audience in virtual reality (VR) exergames, especially focusing on elderly players. To fill this gap, this work examines the effect of an NPC audience and its associated feedback (with/without) on elderly users of VR exergames. In a user study, we used 120 NPC in a virtual audience. Results showed that the presence of the NPC audience with responsive feedback led to higher performance (with a higher success rate of performing gesture actions, more successful combinations of actions (or combos for short) performed, and more opponent's combos prevented) and better gameplay experience (with higher levels of competence, autonomy, relatedness, immersion, and intuitive controls) of elderly players. Our results can help frame the design and engineering of VR exergames that are targeted at elderly users to help them have an enhanced gameplay experience and improve their health.