Gesture Features Research Articles

Educational Data Mining by Optimally Fusing Shallow and Deep Features.

In order to meet the requirement of designing the educational curriculum and extract the relevant data toward students, we in this paper utilize the deep learning architecture to study the personalized understanding and educational feature extraction algorithm of the high school and college students. Aiming at the insufficiency of deep collaborative filtering recommendation algorithm that can only discover the paths that have attracted students' attention, the requirements of intelligent course are proposed by us. First, the word vector permutation discourse poke model is extracted to produce audible vectors of message data, a well-designed learning algorithm is proposed subsequently to extract and fuse multiple heterogeneous educational data. Then, the SVD algorithm program is used to reduce the range of feature vectors. And thus, the model is completely functional and we identify unified channels and users. The method of combining hand-extracted low-level features with high-level human gesture features (image visual gate features and image deep features) automatically extracted by our adopted deep network is used to identify students' various emotions. The conclusive product is a flat semantic description that effectively detects copying and objection. Experiments are performed on a generic dataset, and we use the old-fashioned manual birth method and two well-known data sets VGG16 and fine-tune AlexNet.

Redefining the human factors approach to 3D gestural HCI by exploring the usability-accuracy tradeoff in gestural computer systems

3D gestural technology for HCI could transform the way people interact with computing systems. There are traditionally two approaches to developing gestural technology systems: a human-based approach where usability is maximized and a technology-based approach where system accuracy is maximized. The tradeoff between usability and accuracy may negatively affect the overall trust and reliability in the system. Therefore, this study seeks to redefine the human-based approach to gestural system development by introducing a bottom-up approach to identifying the lower-level features that produce a gesture, thus allowing the technology to accurately recognize features. A user elicitation study was performed, and gestures were classified according to a novel feature extraction gesture taxonomy and a traditional taxonomy of classifying gestures as a unit. The feature-extraction approach revealed several advantages because it fosters a bottom-up approach to identifying gesture features. Using this approach may mitigate the effects of the usability-accuracy tradeoff in gestural system development.

The LET Procedure for Gesture Recognition With Multiple Forearm Angles

A-mode ultrasound, like other biological signals, has a certain deviation in the signals obtained by performing the same gesture at different arm positions. This problem hinders the clinical application of gesture recognition based on A-mode ultrasound. To tackle this problem, we propose the linearly enhanced training (LET) procedure to compensate for the deviation of gesture signals after forearm position changes. The training set does not contain the gesture data of the new position, so no additional training is required. Instead, we determine the scale parameters to construct enhanced features for the new positions by the original position gesture features. We tested the method on 10 gestures after the forearm angle is changed. Results show that the classification accuracy can be improved by 7.8% and 9.4% after the forearm bent and stretched 40° respectively. Since the LET procedure is a step between feature extraction and model construction, it is suitable for various features and algorithms, offering a multi-scene solution based on wearable A-mode ultrasound.

A Feasibility Study on Hand Gesture Intention Interpretation Based on Gesture Detection and Speech Recognition

In this paper, considering the characteristics of hand gestures that occur during a natural, unscripted human conversation, we report the results of experiments involving the data analysis of the video recordings of experiments. The recorded conversations between two subjects and the assistant in the experiments (the first author) are analyzed for findings that might be useful to future human-robot interaction (HRI), findings regarding the recognition of hand gestures in natural conversations. In the experiment, hand gestures that appeared naturally during the conversations were manually selected, and their elapsed times and number of appearances were analyzed. In addition, these unscripted conversations were analyzed with a voice recognition system, and the meanings of the hand gestures that appear in the recorded video were interpreted. It was observed that some gestures were made consciously and others unconsciously. Some gestures appeared multiple times with similar meanings and the same intention, especially gestures involving pointing at something to emphasize a sentence or word. Other observations included that there were times when (i) each subject made different hand gestures to convey the same meaning, (ii) both subjects made the same hand gesture but intended different meanings, and (iii) a specific gesture was made by the same subject but with different intended meanings. After considering the results of these experiments, we conclude that it is better to combine gestures with the utterances at the time in order to interpret their intended meanings correctly.

Attention Mechanism Based on Improved Spatial-Temporal Convolutional Neural Networks for Traffic Police Gesture Recognition

Human action recognition has attracted extensive research efforts in recent years, in which traffic police gesture recognition is important for self-driving vehicles. One of the crucial challenges in this task is how to find a representation method based on spatial-temporal features. However, existing methods performed poorly in spatial and temporal information fusion, and how to extract features of traffic police gestures has not been well researched. This paper proposes an attention mechanism based on the improved spatial-temporal convolutional neural network (AMSTCNN) for traffic police gesture recognition. This method focuses on the action part of traffic police and uses the correlation between spatial and temporal features to recognize traffic police gestures, so as to ensure that traffic police gesture information is not lost. Specifically, AMSTCNN integrates spatial and temporal information, uses weight matching to pay more attention to the region where human action occurs, and extracts region proposals of the image. Finally, we use Softmax to classify actions after spatial-temporal feature fusion. AMSTCNN can strongly make use of the spatial-temporal information of videos and select effective features to reduce computation. Experiments on AVA and the Chinese traffic police gesture datasets show that our method is superior to several state-of-the-art methods.

Comparing mnemonic effects of iconic gestures and pictures on word memory.

Previous studies using intermodal semantic priming have found that gestures improve language memory. In the present study, we ask whether the inherent characteristics of representative gestures (iconic gestures) facilitate word memory, or it is simply the semantic content shared with the words. Two analogous experiments were carried out presenting iconic gestures, pictures, or null primes to target words (nouns and action verbs). In Experiment 1, participants performed a free word recall task. In Experiment 2, the task was one of recognition. The results showed that participants recalled (Experiment 1) an equivalent number of words preceded by gestures or pictures compared with words alone, with no prime. However, a significantly higher number of words were recognised (Experiment 2) when they were primed by iconic gestures compared with the other two conditions, an advantage also found in reaction times (RTs) and both effects being larger in verbs than in nouns. These findings are discussed regarding the differences between recall and recognition processes as well as the particular characteristics of representative gestures.

Characteristics of articulatory gestures in stuttered speech: A case study using real-time magnetic resonance imaging

IntroductionMost of the previous articulatory studies of stuttering have focussed on the fluent speech of people who stutter. However, to better understand what causes the actual moments of stuttering, it is necessary to probe articulatory behaviors during stuttered speech. We examined the supralaryngeal articulatory characteristics of stuttered speech using real-time structural magnetic resonance imaging (RT-MRI). We investigated how articulatory gestures differ across stuttered and fluent speech of the same speaker. MethodsVocal tract movements of an adult man who stutters during a pseudoword reading task were recorded using RT-MRI. Four regions of interest (ROIs) were defined on RT-MRI image sequences around the lips, tongue tip, tongue body, and velum. The variation of pixel intensity in each ROI over time provided an estimate of the movement of these four articulators. ResultsAll disfluencies occurred on syllable-initial consonants. Three articulatory patterns were identified. Pattern 1 showed smooth gestural formation and release like fluent speech. Patterns 2 and 3 showed delayed release of gestures due to articulator fixation or oscillation respectively. Block and prolongation corresponded to either pattern 1 or 2. Repetition corresponded to pattern 3 or a mix of patterns. Gestures for disfluent consonants typically exhibited a greater constriction than fluent gestures, which was rarely corrected during disfluencies. Gestures for the upcoming vowel were initiated and executed during these consonant disfluencies, achieving a tongue body position similar to the fluent counterpart. ConclusionDifferent perceptual types of disfluencies did not necessarily result from distinct articulatory patterns, highlighting the importance of collecting articulatory data of stuttering. Disfluencies on syllable-initial consonants were related to the delayed release and the overshoot of consonant gestures, rather than the delayed initiation of vowel gestures. This suggests that stuttering does not arise from problems with planning the vowel gestures, but rather with releasing the overly constricted consonant gestures.

Multimode Gesture Recognition Algorithm Based on Convolutional Long Short-Term Memory Network.

Gesture recognition utilizes deep learning network model to automatically extract deep features of data; however, traditional machine learning algorithms rely on manual feature extraction and poor model generalization ability. In this paper, a multimodal gesture recognition algorithm based on convolutional long-term memory network is proposed. First, a convolutional neural network (CNN) is employed to automatically extract the deeply hidden features of multimodal gesture data. Then, a time series model is constructed using a long short-term memory (LSTM) network to learn the long-term dependence of multimodal gesture features on the time series. On this basis, the classification of multimodal gestures is realized by the SoftMax classifier. Finally, the method is experimented and evaluated on two dynamic gesture datasets, VIVA and NVGesture. Experimental results indicate that the accuracy rates of the proposed method on the VIVA and NVGesture datasets are 92.55% and 87.38%, respectively, and its recognition accuracy and convergence performance are better than those of other comparison algorithms.

Gesture Recognition Algorithm Introducing Ghost Feature Mapping and Channel Attention Mechanism

To solve the problem that the lightweight object detection network has insufficient ability to extract static gesture features, high false detection rate and missed detection rate, a lightweight gesture recognition algorithm is proposed based on YOLOv4-tiny network structure. First, a more powerful and low-cost ghost feature mapping is introduced to enhance the ability of network to obtain multi-scale gesture features. Then, the embedded channel attention mechanism realizes feature recalibration and achieves the purpose of reducing background interference. Finally, Swish is used as the main activation function to further improve the accuracy of gesture recognition. The experimental results on the gesture dataset show that the proposed algorithm has better recognition performance than YOLOv4-tiny. For multi-scale gestures under different environmental conditions, the algorithm achieves accurate classification as well as real-time detection, and has better recognition performance for small-scale gestures.

Content-Adaptive and Attention-Based Network for Hand Gesture Recognition

For hand gesture recognition, recurrent neural networks and 3D convolutional neural networks are the most commonly used methods for learning the spatial–temporal features of gestures. The calculation of the hidden state of the recurrent neural network at a specific time is determined by both input at the current time and the output of the hidden state at the previous time, therefore limiting its parallel computation. The large number of weight parameters that need to be optimized leads to high computational costs associated with 3D convolution-based methods. We introduced a transformer-based network for hand gesture recognition, which is a completely self-attentional architecture without any convolution or recurrent layers. The framework classifies hand gestures by focusing on the sequence information of the whole gesture video. In addition, we introduced an adaptive sampling strategy based on the video content to reduce the input of gesture-free frames to the model, thus reducing computational consumption. The proposed network achieved 83.2% and 93.8% recognition accuracy on two publicly available benchmark datasets, NVGesture and EgoGesture datasets, respectively. The results of extensive comparison experiments show that our proposed approach outperforms the existing state-of-the-art gesture recognition systems.

Automatic identification of smoking behaviour in public places based on improved YOLO algorithm

This paper proposes an automatic identification method of smoking behaviour in public places based on the improved You Only Look Once (YOLO) algorithm. The histogram of oriented gradient (HOG) feature method is used to detect the face part of smokers, taking the detected face image as the input, the edge of the image is extracted by using the structured edge detection operator of the edge box. The gesture dataset is re-clustered by K-means++ clustering algorithm to obtain a priori frame that is more suitable for gesture recognition; and in the traditional YOLO network, the SPP spatial pyramid pooling module is added to realise the fusion of local gesture features and global gesture features, to complete the automatic identification of smoking behaviour in public places. The results show that the highest recognition rate of this method is 94.3%, the accuracy and recall rate are higher than those of the traditional methods, and the recognition results are more comprehensive.

Evaluation of Deep Learning Models for Person Authentication Based on Touch Gesture

Touch gesture biometrics authentication system is the study of user's touching behavior on his touch device to identify him. The features traditionally used in touch gesture authentication systems are extracted using hand-crafted feature extraction approach. In this work, we investigate the ability of Deep Learning (DL) to automatically discover useful features of touch gesture and use them to authenticate the user. Four different models are investigated Long-Short Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Neural Network (CNN) combined with LSTM (CNN-LSTM), and CNN combined with GRU(CNN-GRU). In addition, different regularization techniques are investigated such as Activity Regularizer, Batch Normalization (BN), Dropout, and LeakyReLU. These deep networks were trained from scratch and tested using TouchAlytics and BioIdent datasets for dynamic touch authentication. The result reported in terms of authentication accuracy, False Acceptance Rate (FAR), False Rejection Rate (FRR). The best result we have been obtained was 96.73%, 96.07% and 96.08% for training, validation and testing accuracy respectively with dynamic touch authentication system on TouchAlytics dataset with CNN-GRU DL model, while the best result of FAR and FRR obtained on TouchAlytics dataset was with CNN-LSTM were FAR was 0.0009 and FRR was 0.0530. For BioIdent dataset the best results have been obtained was 84.87%, 78.28% and 78.35% for Training, validation and testing accuracy respectively with CNN-LSTM model. The use of a learning based approach in touch authentication system has shown good results comparing with other state-of-the-art using TouchAlytics dataset.

Design of Human-Computer Interaction Product Interface of Intelligent Service System based on User Experience

The current intelligent service platform for human-computer interaction products and services on the user experience is not comprehensive enough, resulting in user satisfaction cannot reach the ideal level. Therefore, a new human-computer interface of intelligent service system based on user experience is designed. Build a user experience based intelligent service system hardware platform, the introduction of the full name of hypertext markup language, so that more personalized design to be met. Design user experience PC terminal and Bluetooth/RS-485 gateway module to achieve two-way signal conversion between Bluetooth and RS-485. Based on ARM processor, the speech recognition of human-computer interaction is completed, the features of collected data are extracted, and the hand gesture recognition is completed. In order to optimize the human-computer interaction effect, Kinect is used to track and identify moving objects, and the 3D interactive image is simulated by fused texture. Experimental results show that the proposed method has a higher probability of receiving data, and the recognition rate of gesture features and recognition accuracy can reach more than 90%.

Hidden Markov Model-Based Video Recognition for Sports

In this paper, we conduct an in-depth study and analysis of sports video recognition by improved hidden Markov model. The feature module is a complex gesture recognition module based on hidden Markov model gesture features, which applies the hidden Markov model features to gesture recognition and performs the recognition of complex gestures made by combining simple gestures based on simple gesture recognition. The combination of the two modules forms the overall technology of this paper, which can be applied to many scenarios, including some special scenarios with high-security levels that require real-time feedback and some public indoor scenarios, which can achieve different prevention and services for different age groups. With the increase of the depth of the feature extraction network, the experimental effect is enhanced; however, the two-dimensional convolutional neural network loses temporal information when extracting features, so the three-dimensional convolutional network is used in this paper to extract features from the video in time and space. Multiple binary classifications of the extracted features are performed to achieve the goal of multilabel classification. A multistream residual neural network is used to extract features from video data of three modalities, and the extracted feature vectors are fed into the attention mechanism network, then, the more critical information for video recognition is selected from a large amount of spatiotemporal information, further learning the temporal dependencies existing between consecutive video frames, and finally fusing the multistream network outputs to obtain the final prediction category. By training and optimizing the model in an end-to-end manner, recognition accuracies of 92.7% and 64.4% are achieved on the dataset, respectively.

Gesture estimation for 3D martial arts based on neural network

A 3D martial arts gestures estimation method is proposed, for it is difficult to accurately describe the gestures, which is caused by the high degree of freedom and high similarity of 3D martial arts gestures. This method is combined with finger kinematics analysis model. And beyond that, it is based on the morphological topological structure of hand and combined with the CNN neural network. Firstly, the CNN is used to extract and classify 3D gesture features of martial arts. Then, using the morphological topological structure of hands to simulate the dependence of hand joints, and the 3D coordinates of hand joints were obtained. Finally, the attitude regression module is used to realize the attitude estimation of martial arts gesture action. Simulation results show that the accuracy of gesture estimation can be improved through cascade splicing in this research. Compared with existing 3D gesture estimation methods such as V2V, Pose-REN and CrossInfoNet, the proposed method performs better in MSE and FS indicators. It has lower estimation errors and an inference speed of 220.7 frames per second. In addition, the three-dimensional visualization results show that the predicted joint points obtained by the proposed estimation method coincide with the labeled joint points, and there is no occlusion. So it is proved that the proposed method is feasible, and it can be used to estimate martial arts gestures in the future.

Chinese Traffic Police Gesture Recognition Based on Graph Convolutional Network in Natural Scene

For an automated driving system to be robust, it needs to recognize not only fixed signals such as traffic signs and traffic lights, but also gestures used by traffic police. With the aim to achieve this requirement, this paper proposes a new gesture recognition technology based on a graph convolutional network (GCN) according to an analysis of the characteristics of gestures used by Chinese traffic police. To begin, we used a spatial–temporal graph convolutional network (ST-GCN) as a base network while introducing the attention mechanism, which enhanced the effective features of gestures used by traffic police and balanced the information distribution of skeleton joints in the spatial dimension. Next, to solve the problem of the former graph structure only representing the physical structure of the human body, which cannot capture the potential effective features, this paper proposes an adaptive graph structure (AGS) model to explore the hidden feature between traffic police gesture nodes and a temporal attention mechanism (TAS) to extract features in the temporal dimension. In this paper, we established a traffic police gesture dataset, which contained 20,480 videos in total, and an ablation study was carried out to verify the effectiveness of the method we proposed. The experiment results show that the proposed method improves the accuracy of traffic police gesture recognition to a certain degree; the top-1 is 87.72%, and the top-3 is 95.26%. In addition, to validate the method’s generalization ability, we also carried out an experiment on the Kinetics–Skeleton dataset in this paper; the results show that the proposed method is better than some of the existing action-recognition algorithms.

Die rol van handgebare in oudiovisuele aanbiedings van teologiestudente: Belyning van teorie en praktyk

Kommunikasie in die 21ste eeu het grootliks multimodaal geword. 'n Verskeidenheid primêre modusse, byvoorbeeld geskrewe teks, mondelinge vertelling, beweging, kleur, klank en musiek, word ook elektronies "vertaal" en sekondêr deur rekenaars, selfone en tablette afgelewer. Een van die genres wat gekenmerk word deur multimodaliteit is oudiovisuele aanbieding. In studente se aanbiedings is spraak gewoonlik die primêre modus, en word aangevul deur teks, animasie (beweging), foto's en kleur. Die mondelinge aanbieding tree as deel van 'n multimodale ensemble op, waar die unieke eienskappe van elke modus benut word om betekenis oor te dra. Eers gee ek 'n definisie van die konsep "handgebaar", gevolg deur 'n uiteensetting van algemeen erkende gebareteorieë en tipologieë wat gebare indeel volgens semiotiese gebaretipes, handigheid (links, regs of beide), gekonvensionaliseerde handvorme en palm-oriëntasies, en beweging en posisie in die gebareveld. Daarna volg 'n beskrywing van 'n navorsingsprojek wat berus op die analise van ko-taalgebare in 'n korpus van 17 gevideo-graveerde mondelinge aanbiedings deur teologiestudente. Die artikel word afgesluit deur puntsgewyse advies vir die gebruik van gebare in studente se oudiovisuele aanbiedings. Die advies is sowel gegrond op teorieë en tipologieë van gebare as op die analise van die korpus. Waar dit relevant is, word verwys na handleidings oor kommunikasie en openbare redevoering.

Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction

Augmented reality assisted assembly training (ARAAT) is an effective and affordable technique for labor training in the automobile and electronic industry. In general, most tasks of ARAAT are conducted by real-time hand operations. In this paper, we propose an algorithm of dynamic gesture recognition and prediction that aims to evaluate the standard and achievement of the hand operations for a given task in ARAAT. We consider that the given task can be decomposed into a series of hand operations and furthermore each hand operation into several continuous actions. Then, each action is related with a standard gesture based on the practical assembly task such that the standard and achievement of the actions included in the operations can be identified and predicted by the sequences of gestures instead of the performance throughout the whole task. Based on the practical industrial assembly, we specified five typical tasks, three typical operations, and six standard actions. We used Zernike moments combined histogram of oriented gradient and linear interpolation motion trajectories to represent 2D static and 3D dynamic features of standard gestures, respectively, and chose the directional pulse-coupled neural network as the classifier to recognize the gestures. In addition, we defined an action unit to reduce the dimensions of features and computational cost. During gesture recognition, we optimized the gesture boundaries iteratively by calculating the score probability density distribution to reduce interferences of invalid gestures and improve precision. The proposed algorithm was evaluated on four datasets and proved to increase recognition accuracy and reduce the computational cost from the experimental results.

Differences in the production and perception of communicative kinematics in autism.

In human communication, social intentions and meaning are often revealed in the way we move. In this study, we investigate the flexibility of human communication in terms of kinematic modulation in a clinical population, namely, autistic individuals. The aim of this study was twofold: to assess (a) whether communicatively relevant kinematic features of gestures differ between autistic and neurotypical individuals, and (b) if autistic individuals use communicative kinematic modulation to support gesture recognition. We tested autistic and neurotypical individuals on a silent gesture production task and a gesture comprehension task. We measured movement during the gesture production task using a Kinect motion tracking device in order to determine if autistic individuals differed from neurotypical individuals in their gesture kinematics. For the gesture comprehension task, we assessed whether autistic individuals used communicatively relevant kinematic cues to support recognition. This was done by using stick‐light figures as stimuli and testing for a correlation between the kinematics of these videos and recognition performance. We found that (a) silent gestures produced by autistic and neurotypical individuals differ in communicatively relevant kinematic features, such as the number of meaningful holds between movements, and (b) while autistic individuals are overall unimpaired at recognizing gestures, they processed repetition and complexity, measured as the amount of submovements perceived, differently than neurotypicals do. These findings highlight how subtle aspects of neurotypical behavior can be experienced differently by autistic individuals. They further demonstrate the relationship between movement kinematics and social interaction in high‐functioning autistic individuals.Lay SummaryHand gestures are an important part of how we communicate, and the way that we move when gesturing can influence how easy a gesture is to understand. We studied how autistic and typical individuals produce and recognize hand gestures, and how this relates to movement characteristics. We found that autistic individuals moved differently when gesturing compared to typical individuals. In addition, while autistic individuals were not worse at recognizing gestures, they differed from typical individuals in how they interpreted certain movement characteristics.

Dynamic Gesture Recognition Method Based on Millimeter-wave Radar by One-Dimensional Series Neural Network

For the most of the existing gesture recognition methods based on the radar sensor, the parameters such as the distance, Doppler, and angle are estimated using the radar echo at first. And then the obtained data spectra are inputted into the convolutional neural networks to classify the gestures. The implementation process is complicated. A dynamic gesture recognition method is proposed based on the millimeter-wave radar using the One-Dimensional Series connection Neural Networks (1D-ScNN) in this paper. Firstly, the original echo of dynamic gesture is obtained by the millimeter-wave radar. The gesture features are extracted by the one-dimensional convolution and pooling operations, and then are inputted into the one-dimensional inception v3 structure. In order to aggregate the one-dimensional features, the Long Short-Term Memory (LSTM) modular is connected to the end of the network. The inter-frame correlation of dynamic gestures echo is fully utilized to improve the recognition accuracy and the convergence speed of training. The experimental results show that the proposed method is simple in implementation and has a fast convergence speed. The classification accuracy can reach more than 96.0%, which is higher than the traditional gesture classification methods.