Hand Trajectory Research Articles

Subject-independent trajectory prediction using pre-movement EEG during grasp and lift task

Electroencephalogram (EEG) based motor trajectory decoding for efficient control of brain–computer interface (BCI) systems has been an active area of research. The systems include prosthesis, rehabilitation and human-power augmenting devices. In this work, three-dimensional (3D) hand kinematics is estimated using pre-movement EEG signals during grasp and lift motion. Twelve subjects’ data from the publicly available database WAY-EEG-GAL is utilized for this purpose. Multi-layer perceptron (MLP) and convolutional neural network-long short-term memory (CNN-LSTM) based deep learning frameworks are proposed that utilize the motor-neural information encoded in the EEG data preceding the actual movement execution. Frequency band features are analyzed for hand kinematics decoding using EEG data filtered in seven distinct ranges. The best performing frequency band features is taken for further analysis with different EEG window sizes and lag windows. Additionally, inter-subject hand trajectory decoding analysis is performed using leave-one-subject-out (LOSO) approach. The Pearson correlation coefficient along with hand trajectory are taken to evaluate decoding performance for the proposed neural decoders. This study explores the feasibility of inter-subject 3D hand trajectory decoding using EEG signals during reach and grasp task. The proposed CNN-LSTM decoder is able to achieve the grand correlation in three axes upto 0.730 and 0.627 in intra-subject and inter-subject settings, respectively, thus providing viable information regarding decoding hand position from pre-movement EEG signals for practical BCI applications.

Control of interjoint coordination in the performance of manual circular movements can explain lateral specialization

Between-arm performance asymmetry can be seen in different arm movements requiring specific interjoint coordination to generate the desired hand trajectory. In the current investigation, we assessed between-arm asymmetry of shoulder-elbow coordination and its stability in the performance of circular movements. Participants were 16 healthy right-handed university students. The task consisted of performing cyclic circular movements with either the dominant right arm or the nondominant left arm at movement frequencies ranging from 40% of maximum to maximum frequency in steps of 15%. Kinematic analysis of shoulder and elbow motions was performed through an optoelectronic system in the three-dimensional space. Results showed that as movement frequency increased circularity of left arm movements diminished, taking an elliptical shape, becoming significantly different from the right arm at higher movement frequencies. Shoulder-elbow coordination was found to be asymmetric between the two arms across movement frequencies, with lower shoulder-elbow angle coefficients and higher relative phase for the left compared to the right arm. Results also revealed greater variability of left arm movements in all variables assessed, an outcome observed from low to high movement frequencies. From these findings, we propose that specialization of the left cerebral hemisphere for motor control resides in its higher capacity to generate appropriate and stable interjoint coordination leading to the planned hand trajectory.

Class specific nullspace marginal discriminant analysis with overfitting-prevention kernel estimation for hand trajectory recognitions

Class specific nullspace marginal discriminant analysis with overfitting-prevention kernel estimation for hand trajectory recognitions

Mindreading by body: incorporating mediolateral balance and mouse-tracking measures to examine the motor basis of adults' false-belief tracking.

The role played by motor representations in tracking others' belief-based actions remains unclear. In experiment 1, the dynamics of adults' anticipatory mediolateral motor activity (leftwards-rightwards leaning on a balance board) as well as hand trajectories were measured as they attempted to help an agent who had a true or false belief about an object's location. Participants' leaning was influenced by the agent's belief about the target's location when the agent was free to act but not when she was motorically constrained. However, the hand trajectories participants produced to provide a response were not modulated by the other person's beliefs. Therefore, we designed a simplified second experiment in which participants were instructed to click as fast as possible on the location of a target object. In experiment 2, mouse-movements deviated from an ideal direct path to the object location, with trajectories that were influenced by the location in which the agent falsely believed the object to be located. These experiments highlight that information about an agent's false-belief can be mapped onto the motor system of a passive observer, and that there are situations in which the motor system plays an important role in accurate belief-tracking.

A Human–Robot Collaboration Method Using a Pose Estimation Network for Robot Learning of Assembly Manipulation Trajectories From Demonstration Videos

The wide application of industrial robots has greatly improved assembly efficiency and reliability. However, determining how to efficiently teach a robot to perform assembly manipulation trajectories from demonstration videos is a challenging issue. This paper proposes a method integrating deep learning, image processing, and an iteration model to predict the real assembly manipulation trajectory of a human hand from a video without specific depth information. First, a pose estimation network, Keypoint-RCNN, is used to accurately estimate hand pose in the two-dimensional (2-D) image of each frame in a video. Second, image processing is applied to map the 2-D hand pose estimated by the neural network with the real 3-D assembly space. An iteration model based on the trust region algorithm is proposed to solve for the quaternions and translation vectors of two frames. All the quaternions and translation vectors form the predicted assembly manipulation trajectories. Finally, a UR3 robot is used to imitate the assembly operation based on the predicted manipulation trajectories. The results show that the robot could successfully imitate various operations based on the predicted manipulation trajectories.

New Training Method for Non-Dominant Hand Pitching Motion Based on Reversal Trajectory of Dominant Hand Pitching Motion Using AR and Vibration

New Training Method for Non-Dominant Hand Pitching Motion Based on Reversal Trajectory of Dominant Hand Pitching Motion Using AR and Vibration

Smoothness and Efficiency Metrics Behavior after an Upper Extremity Training with Robic Humanoid Robot in Paediatric Spinal Cord Injured Patients

The upper extremity behavior in smoothness and efficiency metrics should be different between paraplegic and tetraplegic patients. The aim of this article was to analyze the behavior of these metrics after receiving upper extremity training with the humanoid robot Robic as a treatment. Ten pediatric patients participated in the study and completed ten experimental sessions with Robic. Patients were assessed at baseline and at ending the training using the Box and Block test and a non-immersive virtual application based on the Leap Motion Controller available in the RehabHand software. From this application, the smoothness metric was calculated as the number of peaks or units of movement detected in the velocity profile of the hand during the execution of the task, and the efficiency metric was assessed by calculating the length of the hand trajectory. Patients with tetraplegia had a significantly longer trajectory (286.01 ± 59.87 mm) than paraplegics (123.61 ± 17.14 mm) in the baseline situation. However, at the end of the training, there were no differences between them. In the Box and Block test, the paraplegic group passed more cubes than tetraplegics. In conclusion, the first experience with a Robic robot in SCI was very positive, with observed improvements in upper extremity dexterity in trained patients.

Transfer Learning in Trajectory Decoding: Sensor or Source Space?

In this study, across-participant and across-session transfer learning was investigated to minimize the calibration time of the brain-computer interface (BCI) system in the context of continuous hand trajectory decoding. We reanalyzed data from a study with 10 able-bodied participants across three sessions. A leave-one-participant-out (LOPO) model was utilized as a starting model. Recursive exponentially weighted partial least squares regression (REW-PLS) was employed to overcome the memory limitation due to the large pool of training data. We considered four scenarios: generalized with no update (Gen), generalized with cumulative update (GenC), and individual models with cumulative (IndC) and non-cumulative (Ind) updates, with each one trained with sensor-space features or source-space features. The decoding performance in generalized models (Gen and GenC) was lower than the chance level. In individual models, the cumulative update (IndC) showed no significant improvement over the non-cumulative model (Ind). The performance showed the decoder's incapability to generalize across participants and sessions in this task. The results suggested that the best correlation could be achieved with the sensor-space individual model, despite additional anatomical information in the source-space features. The decoding pattern showed a more localized pattern around the precuneus over three sessions in Ind models.

Improvements (or not!) in the hand trajectory of stroke patients due to practice of a virtual game

Movement trajectories contain important spatio-temporal information to characterise human activities that require displacements (eg grasp an object). A trajectory (dis)similarity measure is highly relevant in trajectory data analysis. The main purpose of this study was to develop a running version of the Procrustes Method to quantify dissimilarity between trajectories along time, as a method that can be used in further research. Empirical data was used to quantify changes in stroke patients’ movements in a daily life task (drinking water) after participating in a combined rehabilitation program (virtual reality plus conventional therapy). Results of the simulation study reflected the reliability of the Running Procrustes Method to quantify the dissimilarity between trajectories continuously over time. For the empirical data, this method identified critical parts of the drinking water task, providing information that might suggest beneficial effects of the combined program in stroke patients’ daily life tasks.

A novel mathematical model of the badminton smash: simulation and modeling in biomechanics

Applied physics and computer methods in biomechanics have been extensively used in sports science research, including performance and biomechanics analysis. The Brachistochrone problem, which expresses the curve that an object draws quickly under gravitational forces in a vertical position, is one of the most widely used studies in classical mechanics. A similar problem arises when a badminton player intends to hit a smash with the shortest shot time. This paper aims to determine the optimal stroke trajectory for a shuttlecock smash in the shortest time. We simulate the badminton smash movement using a computer program after analyzing the shuttlecock smash analytically and numerically for several conditions. The modeling results show that a cycloid trajectory allows badminton players to smash the shuttlecock in the shortest time. Based on the experimental findings of Tsai, Huang, and Jih’s study and our models, the ratio of clear speed to smash speed is 0.75, which is still in the range of 0.71 to 0.76, and we find that a cycloid trajectory gives the shortest shuttlecock smash time. We concluded that the experimental data from this study’s literature supported our model. The novelty of this study is that we found the first powerful model and simulation of conventional Brachistochrone in the case of a badminton smash of badminton players. For badminton coaches and players, this model formulation is intended as a reference for optimizing shuttlecock shots. Furthermore, another novelty of this research is that it may lead to software that can be used to analyze the muscle strength of badminton players based on their cycloid hand trajectory and shuttlecock speed.

State-Based Decoding of Continuous Hand Movements Using EEG Signals

Recently, the advent of the non-invasive brain-computer interface (BCI) for continuous decoding of upper limb motions opens a new horizon for motor-disabled people. However, the performance of discrete-decoding BCIs based on discriminating different brain states are still more robust. In this study, we aimed to cascade a discrete state decoder with a continuous decoder to enhance the prediction of hand trajectories. EEG data were recorded from nine healthy subjects performing a center-out task with four orthogonal targets on the horizontal plane. The pre-movement data of each trial has been used for training a binary discrete decoder which identifies the axis of the movement based on common spatial pattern (CSP) features. Two non-parametric continuous decoders based on Gaussian process regression (GPR) have been designed for continuous decoding of hand movements along each axis using the envelope features of EEG signals in six frequency bands. In addition to those four principal orthogonal targets, some targets at random directions on the horizontal plane were recorded to evaluate the generalizability of the proposed model. The discrete decoder attained the average binary classification of 97.1% for discriminating movement along the x-axis and y-axis. The proposed state-based method achieved the mean correlation coefficient of 0.54 between actual and predicted trajectories for principal targets over all subjects. The trajectories of random targets were also decoded with a mean correlation of 0.37. The generalizability of the proposed paradigm proved by the findings of this study could open new possibilities in developing novel types of neuroprostheses for rehabilitation purposes.

Smoothness Evaluation by Hand Trajectory of Box and Block Test During Recovery Process of Stroke Patients

Smoothness Evaluation by Hand Trajectory of Box and Block Test During Recovery Process of Stroke Patients

Low Complexity Radar Gesture Recognition Using Synthetic Training Data

Developments in radio detection and ranging (radar) technology have made hand gesture recognition feasible. In heat map-based gesture recognition, feature images have a large size and require complex neural networks to extract information. Machine learning methods typically require large amounts of data and collecting hand gestures with radar is time- and energy-consuming. Therefore, a low computational complexity algorithm for hand gesture recognition based on a frequency-modulated continuous-wave (FMCW) radar and a synthetic hand gesture feature generator are proposed. In the low computational complexity algorithm, two-dimensional Fast Fourier Transform is implemented on the radar raw data to generate a range-Doppler matrix. After that, background modelling is applied to separate the dynamic object and the static background. Then a bin with the highest magnitude in the range-Doppler matrix is selected to locate the target and obtain its range and velocity. The bins at this location along the dimension of the antenna can be utilised to calculate the angle of the target using Fourier beam steering. In the synthetic generator, the Blender software is used to generate different hand gestures and trajectories and then the range, velocity and angle of targets are extracted directly from the trajectory. The experimental results demonstrate that the average recognition accuracy of the model on the test set can reach 89.13% when the synthetic data are used as the training set and the real data are used as the test set. This indicates that the generation of synthetic data can make a meaningful contribution in the pre-training phase.

Continuous Bimanual Trajectory Decoding of Coordinated Movement From EEG Signals.

While many voluntary movements involve bimanual coordination, few attempts have been made to simultaneously decode the trajectory of bimanual movements from electroencephalogram (EEG) signals. In this study, we proposed a novel bimanual brain-computer interface (BCI) paradigm to reconstruct the continuous trajectory of both hands during coordinated movements from EEG. The protocol required human subjects to complete a bimanual reaching task to the left, middle, or right target while EEG data were collected. A multi-task deep learning model combining the EEGNet and long short-term memory network (LSTM) was proposed to decode bimanual trajectories, including position and velocity. Decoding performance was evaluated in terms of the correlation coefficient (CC) and normalized root mean square error (NRMSE) between decoded and real trajectories. Experimental results from 13 human subjects showed that the grand-averaged combined CC values achieved 0.54 and 0.42 for position and velocity decoding, respectively. The corresponding combined NRMSE values were 0.22 and 0.23. Both CC and NRMSE were significantly superior to the chance level (p<0.05). Comparative experiments also indicated that the proposed model significantly outperformed some other commonly-used methods in terms of CC and NRMSE for continuous trajectory decoding. These findings demonstrated the feasibility of simultaneously decoding bimanual trajectory from EEG, indicating the potential of bimanual control for coordinated tasks.

Radar-Based Air-Writing Gesture Recognition Using a Novel Multistream CNN Approach

Hand gestures, being a convenient and natural way of communication, is getting huge attention for human–computer interface designs. Among these gestures, detecting mid-air writing is one of the most promising applications. Existing radar-based solutions often perform the mid-air writing recognition by tracking the hand trajectory using multiple monostatic or bistatic radars. This article presents a multistream convolutional neural network (MS-CNN)-based in-air digits recognition method using a frequency-modulated continuous-wave (FMCW) radar. With one FMCW radar comprising of two receiving channels, a novel three-stream CNN network with range-time, Doppler-time, and angle-time spectrograms as inputs is constructed and the features are fused together in the later stage before making a final recognition. Unlike the traditional CNN, MS-CNN with multiple independent input layers enables the creation of a multidimensional deep-learning model for FMCW radars. Twelve human volunteers were invited to writing the digits from zero to nine in the air in both home and lab environments. The three-stream CNN architecture-based air writing for digits has shown a promising accuracy of 95%. A comparison of the proposed MS-CNN system was made with 45 different variants of CNN and preliminary results shows that MS-CNN outperforms the other traditional CNN architectures for air-writing application. The gestures radar data have also been made available to the research community.

From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation From Single-Camera Teleoperation

We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the simulator, which is a manipulator resembling the same structure of the operator's hand. It provides an intuitive interface and avoid unstable human-robot hand retargeting for data collection, leading to large-scale and high quality data. Once the data is collected, the customized robot hand trajectories can be converted to different specified robot hands (models that are manufactured) to generate training demonstrations. With imitation learning using our data, we show large improvement over baselines with multiple complex manipulation tasks. Importantly, we show our learned policy is significantly more robust when transferring to the real robot.

Dangerous Driving Behavior Recognition Based on Hand Trajectory

Dangerous driving behaviors in the process of driving will produce road traffic safety hazards, and even cause traffic accidents. Common dangerous driving behavior includes: eating, smoking, fetching items, using a handheld phone, and touching a control monitor. In order to accurately identify the dangerous driving behaviors, this study first uses the hand trajectory data to construct the dangerous driving behavior recognition model based on the dynamic time warping algorithm (DTW) and the longest common sub-sequence algorithm (LCS). Secondly, 45 subjects’ hand trajectory data were obtained by driving simulation test, and 30 subjects’ hand trajectory data were used to determine the dangerous driving behavior label. The matching degree of hand trajectory data of 15 subjects was calculated based on the dangerous driving behavior recognition model, and the threshold of dangerous driving behavior recognition was determined according to the calculation results. Finally, the dangerous driving behavior recognition algorithm and neural network algorithm are compared and analyzed. The dangerous driving behavior recognition algorithm has a fast calculation speed, small memory consumption, and simple program structure. The research results can be applied to dangerous driving behavior recognition and driving distraction warning based on wrist wearable devices.

Biological motion elicits between-person Inhibition of Return in temporal and spatial movement parameters

The present study was designed to gain a deeper understanding of the processing of biological motion stimuli. To this end, we investigated if the inhibition of return (IoR) effect emerges in initiation times and trajectories of pointing movements to targets in left and right space where the preceding cues were pointing movement of a human model or a dot with the same biological motion. Targets were randomly presented in the same or opposite side from the direction of the motion cue. It was hypothesised that the visuomotor system should resonate with the biological motion of a dot, but that the human model should exaggerate the effect. Thus, the human model should trigger stronger attention shifts compared with the dot model and lead to more robust IoR effects in both spatial (movement) and temporal parameters of the observer's pointing responses. Initiation times and the spatial parameters (angle of the hand trajectory) of the pointing movements were analysed. Results indicate that facilitation and IoR effects triggered by human and dot stimuli did not differ. Based on these findings, it seems that the crucial feature of motion cues that generate shifts in attention is biological motion, rather than human appearance per se.

Human Arm Motion Prediction for Collision Avoidance in a Shared Workspace

Industry 4.0 transforms classical industrial systems into more human-centric and digitized systems. Close human–robot collaboration is becoming more frequent, which means security and efficiency issues need to be carefully considered. In this paper, we propose to equip robots with exteroceptive sensors and online motion generation so that the robot is able to perceive and predict human trajectories and react to the motion of the human in order to reduce the occurrence of the collisions. The dataset for training is generated in a real environment in which a human and a robot are sharing their workspace. An Encoder–Decoder based network is proposed to predict the human hand trajectories. A Model Predictive Control (MPC) framework is also proposed, which is able to plan a collision-free trajectory in the shared workspace based on this human motion prediction. The proposed framework is validated in a real environment that ensures collision free collaboration between humans and robots in a shared workspace.

PreMovNet: Premovement EEG-Based Hand Kinematics Estimation for Grasp-and-Lift Task

Kinematics decoding from brain activity helps in developing rehabilitation or power-augmenting brain-computer interface devices. Low-frequency signals recorded from non-invasive electroencephalography (EEG) are associated with the neural motor correlation utilised for motor trajectory decoding (MTD). In this communication, the ability to decode motor kinematics trajectory from pre-movement delta-band (0.5-3&#x00A0;Hz) EEG is investigated for the healthy participants. In particular, two deep learning-based neural decoders called PreMovNet-I and PreMovNet-II, are proposed that make use of motor-related neural information existing in the pre-movement EEG data. EEG data segments with various time lags of 150&#x00A0;ms, 200&#x00A0;ms, 250&#x00A0;ms, 300&#x00A0;ms, and 350&#x00A0;ms before the movement onset are utilised for the same. The MTD is presented for grasp-and-lift task (WAY-EEG-GAL dataset) using EEG with the various lags taken as input to the neural decoders. The performance of the proposed decoders are compared with the state-of-the-art multi-variable linear regression (mLR) model. Pearson correlation coefficient and hand trajectory are utilized as performance metric. The results demonstrate the viability of decoding 3D hand kinematics using pre-movement EEG data, enabling better control of BCI-based external devices such as exoskeleton/exosuit.