Complex Hand Movements Research Articles

Hand-aware graph convolution network for skeleton-based sign language recognition

Skeleton-based sign language recognition (SLR) is a challenging research area mainly due to the fast and complex hand movement. Currently, graph convolution networks (GCNs) have been employed in skeleton-based SLR and achieved remarkable performance. However, existing GCN-based SLR methods suffer from a lack of explicit attention to hand topology which plays an important role in the sign language representation. To address this issue, we propose a novel hand-aware graph convolution network (HA-GCN) to focus on hand topological relationships of skeleton graph. Specifically, a hand-aware graph convolution layer is designed to capture both global body and local hand information, in which two sub-graphs are defined and incorporated to represent hand topology information. In addition, in order to eliminate the over-fitting problem, an adaptive DropGraph is designed in construction of hand-aware graph convolution block to remove the spatial and temporal redundancy in the sign language representation. With the aim to further improve the performance, the joints information, bones, together with their motion information are simultaneously modeled in a multi-stream framework. Extensive experiments on the two open-source datasets, AUTSL and INCLUDE, demonstrate that our proposed algorithm outperforms the state-of-the-art with a significant margin. Our code is available at https://github.com/snorlaxse/HA-SLR-GCN.

Verification of Criterion-Related Validity for Developing a Markerless Hand Tracking Device.

Physicians, physical therapists, and occupational therapists have traditionally assessed hand motor function in hemiplegic patients but often struggle to evaluate complex hand movements. To address this issue, in 2019, we developed Fahrenheit, a device and algorithm that uses infrared camera image processing to estimate hand paralysis. However, due to Fahrenheit's dependency on specialized equipment, we conceived a simpler solution: developing a smartphone app that integrates MediaPipe. The objective of this study was to measure hand movements in stroke patients using both MediaPipe and Fahrenheit and to assess their criterion-related validity. The analysis revealed moderate-to-high correlations between the two methods. Consistent results were also observed in the peak angle and velocity comparisons across the severity stages. Because Fahrenheit determines finger recovery status based on these measures, it has the potential to transfer this function to MediaPipe. This study highlighted the potential use of MediaPipe in paralysis estimation applications.

Age-related differences in finger interdependence during complex hand movements.

The well-known decrease in finger dexterity during healthy aging leads to a significant reduction in quality of life. Still, the exact patterns of altered finger kinematics of older adults in daily life are fairly unexplored. Finger interdependence is the unintentional comovement of fingers that are not intended to move, and it is known to vary across the lifespan. Nevertheless, the magnitude and direction of age-related differences in finger interdependence are ambiguous across studies and tasks and have not been explored in the context of daily life finger movements. We investigated five different free and daily-life-inspired finger movements of the right, dominant hand as well as a sequential finger tapping task of the thumb against the other fingers, in 17 younger (22-37 yr) and 17 older (62-80 yr) adults using an exoskeleton data glove for data recording. Using inferential statistics, we found that the unintentional comovement of fingers generally decreases with age in all performed daily-life-inspired movements. Finger tapping, however, showed a trend towards higher finger interdependence for older compared with younger adults. Using machine learning, we predicted the age group of a person from finger interdependence features of single movement trials significantly better than chance level for the daily-life-inspired movements, but not for finger tapping. Taken together, we show that for specific tasks, decreased finger interdependence (i.e., less comovement) could potentially act as a marker of human aging that specifically characterizes older adults' complex finger movements in daily life.NEW & NOTEWORTHY Kinematic finger movement data were analyzed with regard to age-related differences. Extensive analyses of complex and daily-life-inspired movements reveal that the direction of age effects is not uniform but task-dependent: Although older adults generally show more finger interdependence than younger adults in a simple finger tapping task, this effect is reversed for daily-life-inspired movement tasks. For these tasks, finger interdependence indices offer potential new markers to predict the age group of an individual using machine learning approaches.

Capturing complex hand movements and object interactions using machine learning-powered stretchable smart textile gloves

Capturing complex hand movements and object interactions using machine learning-powered stretchable smart textile gloves

Simulation of FES on the forearm with muscle-specific activation resolution.

Functional electrical stimulation (FES) is an established method of supporting neurological rehabilitation. However, particularly on the forearm, it still cannot elicit selective muscle activations that form the basis of complex hand movements. Current research approaches in the context of selective muscle activation often attempt to enable targeted stimulation by increasing the number of electrodes and combining them in electrode arrays. In order to determine the best stimulation positions and settings, manual or semi-automated algorithms are used. This approach is limited due to experimental limitations. The supportive use of simulation studies is well-established, but existing simulation models are not suitable for analyses of selective muscle activation due to missing or arbitrarily arranged innervation zones. This study introduces a new modeling method to design a person-specific digital twin that enables the prediction of muscle activations during FES on the forearm. The designed individual model consists of three parts: an anatomically based 3D volume conductor, a muscle-specific nerve fiber arrangement in various regions of interest (ROIs), and a standard nerve model. All processes were embedded in scripts or macros to enable automated changes to the model and the simulation setup. The experimental evaluation of simulated strength-duration diagrams showed good coincidence. The relative differences of the simulated amplitudes to the mean amplitude of the four experiments were in the same range as the inter-experimental differences, with mean values between 0.005 and 0.045. Based on these results, muscle-specific activation thresholds were determined and integrated into the simulation process. With this modification, simulated force-intensity curves showed good agreement with additionally measured curves. The results show that the model is suitable for simulating realistic muscle-specific activations. Since complex hand movements are physiologically composed of individual, selective muscle activations, it can be assumed that the model is also suitable for simulating these movements. Therefore, this study presents a new and very promising approach for developing new applications and products in the context of the rehabilitation of sensorimotor disorders.

Accuracy of Video-Based Hand Tracking for People With Upper-Body Disabilities.

Utilization of hand-tracking cameras, such as Leap, for hand rehabilitation and functional assessments is an innovative approach to providing affordable alternatives for people with disabilities. However, prior to deploying these commercially-available tools, a thorough evaluation of their performance for disabled populations is necessary. In this study, we provide an in-depth analysis of the accuracy of Leap's hand-tracking feature for both individuals with and without upper-body disabilities for common dynamic tasks used in rehabilitation. Leap is compared against motion capture with conventional techniques such as signal correlations, mean absolute errors, and digit segment length estimation. We also propose the use of dimensionality reduction techniques, such as Principal Component Analysis (PCA), to capture the complex, high-dimensional signal spaces of the hand. We found that Leap's hand-tracking performance did not differ between individuals with and without disabilities, yielding average signal correlations between 0.7-0.9. Both low and high mean absolute errors (between 10-80mm) were observed across participants. Overall, Leap did well with general hand posture tracking, with the largest errors associated with the tracking of the index finger. Leap's hand model was found to be most inaccurate in the proximal digit segment, underestimating digit lengths with errors as high as 18mm. Using PCA to quantify differences between the high-dimensional spaces of Leap and motion capture showed that high correlations between latent space projections were associated with high accuracy in the original signal space. These results point to the potential of low-dimensional representations of complex hand movements to support hand rehabilitation and assessment.

The weaving art of mechanically and electronically robust hydrogel realizes multi-dimensional response of stretchable sensors

In recent years, hydrogel-based soft electronics have been developed for the application of human motion, flexible robots and foldable displays. However, most existing conductive hydrogels have low toughness, single electron conduction or ionic conduction and poor adaptability towards complex applications. In this work, we obtained the toughness fiber-like hydrogel in three steps by doping nanofillers, directional extrusion freezing, and salting out. The proposed tough fiber-like hydrogel has a rich porous structure, which is achieved by doping the hydrophilic graphene nanosheets (HGNs) into a physically cross-linking polyvinyl alcohol (PVA) gel matrix. Directional extrusion freezing and salting out effect endow fiber-like hydrogel with better orientation ultimate nominal stress (∼9.5 MPa) and nominal strain (∼2200%). Meanwhile, the abundant porous structure facilitates ion migration and greatly enhances the ionic conductivity (up to 2.8 S m −1, at f = 1 MHz) of the hydrogel. The presence of HGNs endows the fiber-like hydrogel with excellent electronic conduction properties. Moreover, the weaving strategy is proposed to fabricate fiber-like hydrogel into hydrogel fabric film, which is effective for increased safety of stress-induced deformation and crack propagation. The woven hydrogel mesh bag can lift 6 kg of watermelon and the woven hydrogel net can bear a top-down weight impact of 1 kg. We also develop a novel type of woven hydrogel glove that can monitor complex hand movement. This work will provide new insight into the design of multifunctional materials with applications on electronic skin, wearable devices, and health monitoring.

The Effect of Mirror Visual Feedback Therapy on the Hand Mental Rotation in Stroke Patients: An ERP study.

Mirror visual feedback (MVF) intervention is an adjunctive approach for motor recovery after stroke. It has been hypothesized that MVF can increase visual perception, motor imagery, and attention of/to the hands. However, neuroimaging evidence for this hypothesis is still lacking. In this study, we used a hand mental rotation task and event-related potential (ERP) analysis to explore the effect of MVF intervention on visual perception, motor preparation, and motor imagery of hands. We recruited 46 patients and randomly divided them into a mirror visual feedback group (MG) and a conventional intervention group (CG). By comparing ERP amplitude between the two groups and between before and after the intervention, we found that the N200 component, which was considered to be related to motor preparation, was significantly less negative in the affected hemisphere than that in the unaffected counterpart. After intervention, the N200 amplitude became more negative, reflecting a recovery of motor preparation. Specifically, MG showed a significant effect on the N200 for the hand pictures at large orientations, while the CG showed an effect mainly for the upright hand stimuli. The results suggested an improvement of preparation for motor imagery of complex and precise hand movements after MVF intervention.Clinical Relevance- This study might be helpful for understanding the neural mechanisms of MVF which can help stroke patients regain upper extremity function.

A Novel Sensor Fusion Approach for Precise Hand Tracking in Virtual Reality-Based Human-Computer Interaction.

The rapidly evolving field of Virtual Reality (VR)-based Human-Computer Interaction (HCI) presents a significant demand for robust and accurate hand tracking solutions. Current technologies, predominantly based on single-sensing modalities, fall short in providing comprehensive information capture due to susceptibility to occlusions and environmental factors. In this paper, we introduce a novel sensor fusion approach combined with a Long Short-Term Memory (LSTM)-based algorithm for enhanced hand tracking in VR-based HCI. Our system employs six Leap Motion controllers, two RealSense depth cameras, and two Myo armbands to yield a multi-modal data capture. This rich data set is then processed using LSTM, ensuring the accurate real-time tracking of complex hand movements. The proposed system provides a powerful tool for intuitive and immersive interactions in VR environments.

Улучшение речи у детей с ДЦП на фоне реабилитации с применением нейроинтерфейса «мозг–компьютер–экзоскелет кисти»

As explained earlier, neurorehabilitation sessions involving the use of the non-invasive “brain – computer – hand exoskeleton” interface reduce hand muscle spasticity and improve motor skills in children with cerebral palsy (CP). However, the changes in the patients’ speech functions and their relationship with the upper limb mobility have not been analyzed. The study was aimed to assess the correlation between the motor and speech functions of children with CP, as well as to detect the changes in motor realization of speech production following complex treatment of patients including sessions of neurorehabilitation. The study involved children with CP aged 6–15. The index group (n = 40, 16 girls, 24 boys) received complex resort treatment with the course of neurorehabilitation, while the comparison group (n = 20, 10 girls, 10 boys) received standard resort treatment. A significant (р < 0.001) correlation between the total ABILHAND-Kids score and the indicators of speech production motor realization was revealed. In patients of the index group, complex treatment with the course of neurorehabilitation resulted in the significant (р < 0.001) decrease in hand spasticity and the increase in the total ABILHAND-Kids score and speech scores. No significant changes of these indicators were revealed in children of the comparison group. Beneficial effects of neurorehabilitation may be based on the enhanced plasticity of the neural circuits responsible for planning and execution of complex hand movements, as well as speech processes. The findings can be used to develop new methods for correction of motor and cognitive spheres in children with CP.

A Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps

In this work, we present a reconfigurable data glove design to capture different modes of human hand–object interactions, which are critical in training embodied artificial intelligence (AI) agents for fine manipulation tasks. To achieve various downstream tasks with distinct features, our reconfigurable data glove operates in three modes sharing a unified backbone design that reconstructs hand gestures in real time. In the tactile-sensing mode, the glove system aggregates manipulation force via customized force sensors made from a soft and thin piezoresistive material; this design minimizes interference during complex hand movements. The virtual reality (VR) mode enables real-time interaction in a physically plausible fashion: A caging-based approach is devised to determine stable grasps by detecting collision events. Leveraging a state-of-the-art finite element method, the simulation mode collects data on fine-grained four-dimensional manipulation events comprising hand and object motions in three-dimensional space and how the object’s physical properties (e.g., stress and energy) change in accordance with manipulation over time. Notably, the glove system presented here is the first to use high-fidelity simulation to investigate the unobservable physical and causal factors behind manipulation actions. In a series of experiments, we characterize our data glove in terms of individual sensors and the overall system. More specifically, we evaluate the system’s three modes by ① recording hand gestures and associated forces, ② improving manipulation fluency in VR, and ③ producing realistic simulation effects of various tool uses, respectively. Based on these three modes, our reconfigurable data glove collects and reconstructs fine-grained human grasp data in both physical and virtual environments, thereby opening up new avenues for the learning of manipulation skills for embodied AI agents.

Random forest-based simultaneous and proportional myoelectric control system for finger movements

A classification scheme for myoelectric control systems (MCS) cannot mimic complex hand movements. This paper presents simultaneous and proportional MCS by estimating the angles of fourteen finger joints using time-domain feature extraction and random forest. The experimental results show that the best feature was the root mean square (RMS). Furthermore, the random forest attained an average coefficient of determination (R2) of 0.85 compared to other regressors which perform below 0.75. The ANOVA tests indicated that the performance of the proposed system was significantly different. Therefore, the proposed system will be the best option for real-time MCS applications in the future.

Advancing Communication for the Deaf: A Convolutional Model for Arabic Sign Language Recognition

For the deaf population that speaks Arabic, Arabic Sign Language (ArSL) is an essential means of communication. This research presents a convolutional model for recognizing Arabic sign language because of the importance of clear communication. We hope to improve the deaf community's access to communication and broaden its sense of belonging by harnessing deep learning's power and fine-tuning the model to ArSL's particularities. To represent the complex hand movements and visual patterns that are characteristic of ArSL, the proposed model makes use of a variety of carefully made architectural decisions, such as the number of layers, the size of the kernels, the activation functions, and the pooling approaches. Our model outperforms state-of-the-art machine learning techniques, as shown by experimental findings on a large dataset. These results not only lay the groundwork for future developments in sign language recognition, but also demonstrate the promise of our technique in improving communication for the Arabic-speaking deaf community.

Variability in Hemispheric Functional Segregation Phenotypes: A Review and General Mechanistic Model.

Many functions of the human brain are organized asymmetrically and are subject to strong population biases. Some tasks, like speaking and making complex hand movements, exhibit left hemispheric dominance, whereas others, such as spatial processing and recognizing faces, favor the right hemisphere. While pattern of preference implies the existence of a stereotypical way of distributing functions between the hemispheres, an ever-increasing body of evidence indicates that not everyone follows this pattern of hemispheric functional segregation. On the contrary, the review conducted in this article shows that departures from the standard hemispheric division of labor are routinely observed and assume many distinct forms, each having a different prevalence rate. One of the key challenges in human neuroscience is to model this variability. By integrating well-established and recently emerged ideas about the mechanisms that underlie functional lateralization, the current article proposes a general mechanistic model that explains the observed distribution of segregation phenotypes and generates new testable hypotheses.

Decomposition into dynamic features reveals a conserved temporal structure in hand kinematics

The human hand is a unique and highly complex effector. The ability to describe hand kinematics with a small number of features suggests that complex hand movements are composed of combinations of simpler movements. This would greatly simplify the neural control of hand movements. If such movement primitives exist, a dimensionality reduction approach designed to exploit these features should outperform existing methods. We developed a deep neural network to capture the temporal dynamics of movements and demonstrate that the features learned allow accurate representation of functional hand movements using lower-dimensional representations than previously reported. We show that these temporal features are highly conserved across individuals and can interpolate previously unseen movements, indicating that they capture the intrinsic structure of hand movements. These results indicate that functional hand movements are defined by a low-dimensional basis set of movement primitives with important temporal dynamics and that these features are common across individuals.

Intrinsically piezoelectric elastomer based on crosslinked polyacrylonitrile for soft electronics

Soft and elastic piezoelectric materials have critical applications in emerging fields such as wearable electronics and soft robotics. Although there is much progress towards piezoelectric elastomers, existing methods to achieve such materials rely mainly on blending piezoelectric ceramic or polymer fillers into elastomers. Here, we report a soft tissue-like polyacrylonitrile (PAN)-based piezoelectric elastomer, which shows a high piezoelectric coefficient d33 of ∼ 40 pC/N and an intrinsic super-elasticity (100 % self-recovery). Moreover, ultrahigh compressibility (> 99 % strain) and stretchability (> 250 % strain) are achieved simultaneously for the elastomer. The elastomer can produce long-term stable electrical outputs and exhibit a fast piezoelectric response and high sensitivity due to the stress-induced poling effect. We proceed to demonstrate that the piezoelectric elastomers can be used for simultaneous impact protection and self-sensing, as well as continuous human pulse and complex hand movements monitoring. These results suggest that the material has potential applications for soft electronics.

Discriminating Free Hand Movements Using Support Vector Machine and Recurrent Neural Network Algorithms.

Decoding natural hand movements is of interest for human–computer interaction and may constitute a helpful tool in the diagnosis of motor diseases and rehabilitation monitoring. However, the accurate measurement of complex hand movements and the decoding of dynamic movement data remains challenging. Here, we introduce two algorithms, one based on support vector machine (SVM) classification combined with dynamic time warping, and the other based on a long short-term memory (LSTM) neural network, which were designed to discriminate small differences in defined sequences of hand movements. We recorded hand movement data from 17 younger and 17 older adults using an exoskeletal data glove while they were performing six different movement tasks. Accuracy rates in decoding the different movement types were similarly high for SVM and LSTM in across-subject classification, but, for within-subject classification, SVM outperformed LSTM. The SVM-based approach, therefore, appears particularly promising for the development of movement decoding tools, in particular if the goal is to generalize across age groups, for example for detecting specific motor disorders or tracking their progress over time.

Musculoskeletal Synergies in the Grasping Hand.

Investigations on how the central nervous system (CNS) effortlessly conducts complex hand movements have led to an extensive study of synergies or movement primitives. Of the different types of hand synergies, kinematic and muscle synergies have been widely studied in literature, but only a few studies have fused both. In this paper kinematic and muscle activities recorded from the activities of daily living were first fused and then dimensionally reduced through principal component analysis (PCA). By using these principal components or musculoskeletal synergies in a weighted linear combination, the recorded kinematics and muscle activities were reconstructed. The performance of these musculoskeletal synergies in reconstructing the movements was compared to the kinematic and muscle synergies reported previously in the literature by us and others. The results from these findings indicate that musculoskeletal synergies perform better than the synergies extracted without fusion. These newly demonstrated musculoskeletal synergies might improve neural control of robotics, prosthetics and exoskeletons. Clinical Relevance- In this paper, musculoskeletal synergies were extracted from the fusion of kinematic and muscle activities recorded from the activities of daily living. These newly demonstrated musculoskeletal synergies might enhance our understanding of neural control of robotics, prosthetics and exoskeletons.

Recurrent Convolutional Neural Networks as an Approach to Position-Aware Myoelectric Prosthesis Control.

Persons with normal arm function can perform complex wrist and hand movements over a wide range of limb positions. However, for those with transradial amputation who use myoelectric prostheses, control across multiple limb positions can be challenging, frustrating, and can increase the likelihood of device abandonment. In response, the goal of this research was to investigate convolutional neural network (RCNN)-based position-aware myoelectric prosthesis control strategies. Surface electromyographic (EMG) and inertial measurement unit (IMU) signals, obtained from 16 non-disabled participants wearing two Myo armbands, served as inputs to RCNN classification and regression models. Such models predicted movements (wrist flexion/extension and forearm pronation/supination), based on a multi-limb-position training routine. RCNN classifiers and RCNN regressors were compared to linear discriminant analysis (LDA) classifiers and support vector regression (SVR) regressors, respectively. Outcomes were examined to determine whether RCNN-based control strategies could yield accurate movement predictions, while using the fewest number of available Myo armband data streams. An RCNN classifier (trained with forearm EMG data, and forearm and upper arm IMU data) predicted movements with 99.00% accuracy (versus the LDA's 97.67%). An RCNN regressor (trained with forearm EMG and IMU data) predicted movements with R2 values of 84.93% for wrist flexion/extension and 84.97% for forearm pronation/supination (versus the SVR's 77.26% and 60.73%, respectively). The control strategies that employed these models required fewer than all available data streams. RCNN-based control strategies offer novel means of mitigating limb position challenges. This research furthers the development of improved position-aware myoelectric prosthesis control.

Deep Residual Network for Smartwatch-Based User Identification through Complex Hand Movements

Wearable technology has advanced significantly and is now used in various entertainment and business contexts. Authentication methods could be trustworthy, transparent, and non-intrusive to guarantee that users can engage in online communications without consequences. An authentication system on a security framework starts with a process for identifying the user to ensure that the user is permitted. Establishing and verifying an individual’s appearance usually requires a lot of effort. Recent years have seen an increase in the usage of activity-based user identification systems to identify individuals. Despite this, there has not been much research into how complex hand movements can be used to determine the identity of an individual. This research used a one-dimensional residual network with squeeze-and-excitation (SE) configurations called the 1D-ResNet-SE model to investigate hand movements and user identification. According to the findings, the SE modules have enhanced the one-dimensional residual network’s identification ability. As a deep learning model, the proposed methodology is capable of effectively identifying features from the input smartwatch sensor and could be utilized as an end-to-end model to clarify the modeling process. The 1D-ResNet-SE identification model is superior to the other models. Hand movement assessment based on deep learning is an effective technique to identify smartwatch users.