Human Hand Motion Research Articles

Sign Language Recognition System

In this paper an attempt has been made to design a sign language recognition system. An intelligent glove has been designed to automate the communication between a deaf-mute with others by converting sign language into speech or understandable language. The sensory gloves provide data of the human hand shape or movement and translate it to text and speech. It comprises hardware and software for translating sensor data. It is wearable devices that can be put on human hands and convert hand's gestures into signs letter by letter and send the data into the firebase for further processing. The glove is equipped with flex sensors and an inertial measurement unit to recognize the movement by monitoring the finger orientation and hand motion in three-dimensional spaces that senses a person's gestures in the form of finger bend and hand fist tilt. The Hall sensor has been used to process and collect data for training and model development. The three different machine learning algorithms, i.e., support vector machine, Naive Bayes, decision tree, have been used for analysis. It has been observed that the support vector machine has the highest accuracy, i.e., 90%. After Analyzing, the data has been sent to the speech converting function, and then audible results have been produced.

The Effect of the Waveform of the Functional Electrical Stimulation on the Movement of MP Joint in Index Finger

Recently, many types of Functional Electrical Stimulation (FES) devices have been developed for rehabilitation and neural prosthesis. The effect on human motion by changing the waveform pattern of electrical stimulation was evaluated for suitable parameters for the precise control of MP joint of index finger, In this study, we use a combination of square wave waveforms and then we focused on the stimulation period, stimulation amplitude, and pulse width. The results showed that the stimulus amplitude and pulse width had a strong linear correlation with MP joint angle increase. However, the stimulation period may have nonlinearity. These results suggest that the voltage and pulse width are possible parameters for the precious human hand motion control with FES.

Research on Hand Action Pattern Recognition of Bionic Limb Based on Surface Electromyography

Hands are important parts of a human body. It is not only the main tool for people to engage in productive labor, but also an important communication tool. When the hand moves, the human body produces a kind of signal named surface electromyography (sEMG), which is a kind of electrophysiological signal that accompanies muscle activity. It contains a lot of information about human movement consciousness. The bionic limb is driven by multi-degree-freedom control, which is got by converting the recognition result and this can meet the urgent need of people with disabilities for autonomous operation. A profound study of hand action pattern technology based on sEMG signals can achieve the ability of the bionic limb to distinguish the hand action fast and accurately. From the perspective of the pattern recognition of the bionic limb, this paper discussed the human hand action pattern recognition technology of sEMG. By analyzing and summarizing the current development of human hand movement recognition, the author proposed a bionic limb schema based on artificial neural network and the improved DT-SVM hand action recognition system. According to the research results, it is necessary to expand the type and total amount of hand movements and gesture recognition, in order to adapt to the objective requirements of the diversity of hand action patterns in the application of the bionic limb.

Dimensionality Reduction and Motion Clustering During Activities of Daily Living: Decoupling Hand Location and Orientation.

This article is the second in a two-part series analyzing human arm and hand motion during a wide range of unstructured tasks. In this work, we track the hand of healthy individuals as they perform a variety of activities of daily living (ADLs) in three ways decoupled from hand orientation: end-point locations of the hand trajectory, whole path trajectories of the hand, and straight-line paths generated using start and end points of the hand. These data are examined by a clustering procedure to reduce the wide range of hand use to a smaller representative set. Hand orientations are subsequently analyzed for the end-point location clustering results and subsets of orientations are identified in three reference frames: global, torso, and forearm. Data driven methods that are used include dynamic time warping (DTW), DTW barycenter averaging (DBA), and agglomerative hierarchical clustering with Ward's linkage. Analysis of the end-point locations, path trajectory, and straight-line path trajectory identified 5, 5, and 7 ADL task categories, respectively, while hand orientation analysis identified up to 4 subsets of orientations for each task location, discretized and classified to the facets of a rhombicuboctahedron. Together these provide insight into our hand usage in daily life and inform an implementation in prosthetic or robotic devices using sequential control.

Dimensionality Reduction and Motion Clustering During Activities of Daily Living: Three-, Four-, and Seven-Degree-of-Freedom Arm Movements.

This paper is the first in a two-part series analyzing human arm and hand motion during a wide range of unstructured tasks. The wide variety of motions performed by the human arm during daily tasks makes it desirable to find representative subsets to reduce the dimensionality of these movements for a variety of applications, including the design and control of robotic and prosthetic devices. This paper presents a novel method and the results of an extensive human subjects study to obtain representative arm joint angle trajectories that span naturalistic motions during Activities of Daily Living (ADLs). In particular, we seek to identify sets of useful motion trajectories of the upper limb that are functions of a single variable, allowing, for instance, an entire prosthetic or robotic arm to be controlled with a single input from a user, along with a means to select between motions for different tasks. Data driven approaches are used to discover clusters and representative motion averages for the wrist 3 degree of freedom (DOF), elbow-wrist 4 DOF, and full-arm 7 DOF motions. The proposed method makes use of well-known techniques such as dynamic time warping (DTW) to obtain a divergence measure between motion segments, Ward's distance criterion to build hierarchical trees, and functional principal component analysis (fPCA) to evaluate cluster variability. The emerging clusters associate various recorded motions into primarily hand start and end location for the full-arm system, motion direction for the wrist-only system, and an intermediate between the two qualities for the elbow-wrist system.

REAL-TIME HAND GESTURE CLASSIFICATION USING CRNN WITH SCALE AVERAGE WAVELET TRANSFORM

It is very useful in the human computer interface to quickly and accurately recognize human hand movements in real time. In this paper, we aimed to robustly recognize hand gestures in real time using Convolutional Recurrent Neural Network (CRNN) with pre-processing and overlapping window. The CRNN is a deep learning model that combines Long Short-Term Memory (LSTM) for time-series information classification and Convolutional Neural Network (CNN) for feature extraction. The sensor for hand gesture detection uses Myo-armband, and six hand gestures are recognized and classified, including two grips, three hand signs, and one rest. As the essential pre-processing due to the characteristics of EMG data, the existing Short Time Fourier Transform (STFT), Continuous-time Wavelet Transform (CWT), and newly proposed Scale Average Wavelet Transform (SAWT) are used, and thus, the SAWT showed relatively high accuracy in the stationary environmental test. The CRNN with overlapping window has been proposed that can improve the degradation of real-time prediction accuracy, which is caused by inconsistent start time and hand motion speed when acquiring the EMG signal. In the stationary environmental test, the CRNN model with SAWT and overlapping window showed the highest accuracy of 92.5%. In the real-time environmental test, for all subjects learning, 80% accuracy and 0.99 s time delay were obtained on average, and for individual learning, 91.5% accuracy and 0.32 s time delay were obtained on average. As a result, in both stationary and real-time tests, the CRNN with SAWT and overlapping window showed better performance than the other methods.

A Robust Data-Driven Soft Sensory Glove for Human Hand Motions Identification and Replication

Biomechanical sensors are essential components for wearable robots because of their broad spectrum of industrial and medical applications. A wide range of these sensors uses soft materials with deformation-related electrical properties, namely variable resistors and capacitors. Previously, conductive materials were injected into silicone structures that require several molding steps to build the microchannels. This paper proposes a versatile soft sensing glove, using a simple process for preparing sensors of different sizes without molding. We used commercially available silicone tubes to host the conductive liquid. Ten sensors were attached to the back side of the glove to measure flexion-extension and four sensors were placed on the glove at the interdigital folds between the fingers to measure abduction-adduction. The sensory glove successfully replicates hand motion. We used machine learning algorithms to estimate the angles of the joints in the hand and also to identify 15 gestures. The system's robustness was evaluated in two experiments. The gesture prediction is robust to shocks from contact with a punching ball and also submersion of the glove in water. The proposed sensory glove overcomes challenges of comfort, rigidity, and robustness. It can be used, therefore, to replicate human hand motion in industrial and medical applications.

An Ultra-Stretchable and Highly Sensitive Photoelectric Effect-Based Strain Sensor: Implementation and Applications

High performance strain sensors have recently attracted considerable attention because of their applications in human motion detection, soft robotics, and human-machine interface. However, it remains some challenges in achieving satisfactory features (e.g., high sensitivity, high stretchable, linearity, and easy fabrication) for soft strain sensors. Inspired by the working principle of the eye's adaptation to different light luminance, this paper presents an ultra-stretchable, highly sensitive and high linear strain sensor based on the photoelectric effect. The sensor has a simple structure which is made of light-emitting diode (LED), photosensitive element, silicone rod, and sensor shell. When the silicone rod is stretched, the light's shielding area becomes larger, causing the increase of light luminous received by the photosensitive element. Hence, the sensitivity of the sensor can be tuned by changing luminance of LED and the amplification factor. Experiments demonstrate that the sensor has a high sensitivity (the maximum gauge factor was 1050), wide strain-sensing range (the max sensing range was up to 800%), high linearity range (from 100% to 300%), good repeatability (decreased by 5.48% over 5000 cycles at the wide tensile strain of 0 - 300%), low drift, and high signal-to-noise ratio (over 60dB). The proposed strain sensor has been applied in the bending detection of the wrist and finger, the recording of the human moving patterns, and the human's pulse counting. The sensor can be further fabricated into a data glove for detecting the human hand movement.

A Task-Learning Strategy for Robotic Assembly Tasks from Human Demonstrations.

In manufacturing, traditional task pre-programming methods limit the efficiency of human–robot skill transfer. This paper proposes a novel task-learning strategy, enabling robots to learn skills from human demonstrations flexibly and generalize skills under new task situations. Specifically, we establish a markerless vision capture system to acquire continuous human hand movements and develop a threshold-based heuristic segmentation algorithm to segment the complete movements into different movement primitives (MPs) which encode human hand movements with task-oriented models. For movement primitive learning, we adopt a Gaussian mixture model and Gaussian mixture regression (GMM-GMR) to extract the optimal trajectory encapsulating sufficient human features and utilize dynamical movement primitives (DMPs) to learn for trajectory generalization. In addition, we propose an improved visuo-spatial skill learning (VSL) algorithm to learn goal configurations concerning spatial relationships between task-relevant objects. Only one multioperation demonstration is required for learning, and robots can generalize goal configurations under new task situations following the task execution order from demonstration. A series of peg-in-hole experiments demonstrate that the proposed task-learning strategy can obtain exact pick-and-place points and generate smooth human-like trajectories, verifying the effectiveness of the proposed strategy.

A brief analysis of gesture recognition in VR

With the rapid development of artificial intelligence technology, human‐computer interaction, as the core of virtual reality system, has become an increasingly important part of modern science and technology products. At present, human‐computer interaction is mainly based on contact, which must be completed through direct contact between hand and receiving equipment, rather than through indirect contact between hand and equipment. Direct interaction can not give full play to the advantages of flexible human hand, so gesture recognition becomes extremely important in interaction.The development of gesture recognition technology based on vision is a process from 2D to 3D. Early gesture recognition is based on the recognition technology of two‐dimensional color image, which means to get two‐dimensional static image after taking the scene with ordinary camera, and then recognize the content of the image through the computer graphic algorithm. With the development of camera and sensor technology, the depth information of gesture can be captured, and 3D gesture recognition technology can recognize various hand shapes, gestures and actions.Virtual reality is a kind of simulated virtual environment generated by computer according to reality and imagination space, which makes users have immersive feeling in the real world. Through various interactive behaviors, it stimulates users’ perception systems such as vision, touch and hearing, so as to establish a comprehensive sensory experience. Nowadays, virtual reality interaction is more based on the handle of virtual reality device or operating gloves, but using these devices is too complex, so gesture interaction based on bare hands will have a broader prospect.The virtual reality interaction technology based on gesture recognition is to recognize human hand movements with cameras and other devices and interact with the virtual scene. Users can control the virtual scene and interact through simple gestures. Gesture recognition is now applied in many industries, but the interaction of gesture in virtual reality is not so popular. This paper will analyze the development status of gesture recognition in virtual reality technology, and prospect the development trend of virtual reality technology based on gesture recognition.

A Method for Measuring the Height of Hand Movements Based on a Planar Array of Electrostatic Induction Electrodes.

This paper proposes a method based on a planar array of electrostatic induction electrodes, which uses human body electrostatics to measure the height of hand movements. The human body is electrostatically charged for a variety of reasons. In the process of a hand movement, the change of a human body’s electric field is captured through the electrostatic sensors connected to the electrode array. A measurement algorithm for the height of hand movements is used to measure the height of hand movements after the direction of it has been obtained. Compared with the tridimensional array, the planar array has the advantages of less space and easy deployment; therefore, it is more widely used. In this paper, a human hand movement sensing system based on human body electrostatics was established to perform verification experiments. The results show that this method can measure the height of hand movements with good accuracy to meet the requirements of non-contact human-computer interactions.

Comparison Analysis of Hand Gesture Recognition using Different Prediction Based Algorithm

In previous years, the issue of sign motion acknowledgment is happen on the grounds that it is hard to perceive the human hand motion by utilizing diverse calculation and by using device .So in this paper we explained different algorithm to explain gesture recognition algorithm because it has been gaining lots of attention. We can interact with computer without touch the computer screen using our hand gesture we can give instruction to computer so in this paper we explain about the detection of hands signed gesture using Kinect. We are using dynamic time warping method of gesture recognition. We explain an efficient method for gesture recognition. We also used easy K-NN classifier. In this method we used DTW (Dynamic Time wrapping) alignment. Using different algorithms and methods we are explain about the sign hand gesture recognition results. We used MPLCS algorithm to recognize hand gesture in free air and give good results after that we additionally utilized MCC calculation which have identify the beginning and end purpose of significant motion and overlook unused signals. So by using this algorithm we are giving our hand gesture reorganization results which are better than previous all results.

Plucking Motions for Tea Harvesting Robots Using Probabilistic Movement Primitives

This letter proposes a harvesting robot capable of plucking tea leaves. In order to harvest high-quality tea, the robot is required to pluck the petiole of the leaf without cutting it using blades. To pluck the leaves, it is necessary to reproduce a complicated human hand motion of pulling while rotating. Furthermore, the rotation and pulling of the hand, and the time taken, vary greatly depending on conditions that include the maturity of the leaves, thickness of the petioles, and thickness and length of the branches. Therefore, it is necessary to determine the amount of translational and rotational movements, and the length of time of the motion, according to each situation. In this study, the complicated motion is reproduced by learning from demonstration. The condition is judged in terms of the stiffness of the branches, which is defined as the force received from the branches per unit length when the gripped leaf is slightly pulled up. Combining the learned motions probabilistically at a ratio determined by the branch stiffness, the appropriate motion is generated, even for situations where no motion is taught. We compared the motions generated by the proposed method with the motions taught by humans, and verified the effectiveness of the proposed method. It was confirmed by experiment that the proposed method can harvest high-quality tea.

A Survey on Hand Pose Estimation with Wearable Sensors and Computer-Vision-Based Methods.

Real-time sensing and modeling of the human body, especially the hands, is an important research endeavor for various applicative purposes such as in natural human computer interactions. Hand pose estimation is a big academic and technical challenge due to the complex structure and dexterous movement of human hands. Boosted by advancements from both hardware and artificial intelligence, various prototypes of data gloves and computer-vision-based methods have been proposed for accurate and rapid hand pose estimation in recent years. However, existing reviews either focused on data gloves or on vision methods or were even based on a particular type of camera, such as the depth camera. The purpose of this survey is to conduct a comprehensive and timely review of recent research advances in sensor-based hand pose estimation, including wearable and vision-based solutions. Hand kinematic models are firstly discussed. An in-depth review is conducted on data gloves and vision-based sensor systems with corresponding modeling methods. Particularly, this review also discusses deep-learning-based methods, which are very promising in hand pose estimation. Moreover, the advantages and drawbacks of the current hand gesture estimation methods, the applicative scope, and related challenges are also discussed.

Effects of Human Synchronous Hand Movements in Eliciting a Sense of Agency and Ownership

The self is built as an entity independent from the external world using the human ability to experience the senses of agency and ownership. Humans usually experience these senses during movement. Nevertheless, researchers recently reported that another person’s synchronous mirror-symmetrical movements elicited both agency and ownership in research participants. However, it is unclear whether this elicitation was caused by the synchronicity or the mirror symmetry of the movements. To address this question, we investigated the effect of interpersonal synchronization on the self-reported sense of agency and ownership in two conditions, using movements with and without mirror symmetry. Participants performed rhythmic hand movements while viewing the experimenter’s synchronous or random hand movements, and then reported their perceptions of agency and ownership in a questionnaire. We observed that agency and ownership were significantly elicited by the experimenter’s synchronous hand movements in both conditions. The results suggested that the synchronous movements of another person—rather than mirror- or non-mirror-symmetrical movements—appear to elicit the experience of a sense of agency and ownership. The results also suggested that people could experience these senses not only from their own movements but also from another person’s synchronous movements.

Classification of human hand movements based on EMG signals using nonlinear dimensionality reduction and data fusion techniques

Surface electromyography (EMG) is non-invasive signal acquisition technique that plays a central role in many application, including clinical diagnostics, control for prosthetic devices and for human-machine interactions. The processing typically begins with a feature extraction step, which may be followed by the application of a dimensionality reduction technique. The obtained reduced features are input for a machine learning classifier. The constructed machine learning model may then classify new recorded movements.The features extracted for EMG signals usually capture information both from the time and from the frequency domain. Short time Fourier transform (STFT) is commonly used for signal processing and in particular for EMG processing since it captures the temporal and the frequency characteristics of the data. Since the number of calculated STFT features is large, a common approach in signal processing and machine learning applications is to apply a linear or a nonlinear dimensionality reduction technique for simplifying the feature space. Another aspect that arises in medical applications in general and in EMG based hand classification in particular, is the large variability between subjects. Due to this variability, many studies focus on single subject classification. This requires acquiring a large training set for each tested participant which is not practical in real life application.The objectives of this study were first to compare between the performances of a nonlinear dimensionality technique to a standard linear dimensionality method when applied for single subject EMG based hand movement classification, and to examined their performances in case of limited amount of training data samples. The second objective was to propose an algorithm for multi-subjects classification that utilized a data alignment step for overcoming the large variability between subjects.The data set included EMG signals from 5 subjects who perform 6 different hand movements. STFT was calculated for feature extraction, principal component analysis (PCA) and diffusion maps (DM) were compared for dimension reductions. An affine transformation for aligning between the reduced feature spaces of two subjects, was investigated. K-nearest neighbors (KNN) was used for single and multi-subject classification.The results of this study clearly show that the DM outperformed the PCA in case of limited training data. In addition, the multi-subject classification approach, which utilizes dimension reduction methods along with an alignment algorithm enable robust classification of a new subject based on another subjects’ data sets. The proposed framework is general and can be adopted for many EMG classification task.

Design and Implementation of Artificial Human Hand Model Based on Eye Blink Detection

This paper proposes two paradigms for controlling a robotic arm by integrating eye blink and IOT based techniques. The purpose of our study is to develop a feasible paradigm for helping disabled persons with their every-day needs.“Artificial Human Hand Model (AHHM)” which can precisely follow the real movements of human hand by means of real human eye blinks and IOT based methods. Simple algorithms were implemented for detecting electrophysiological signals like eye saccades, blinking and eye closure events. Interactive resources to integrate myRIO into our project increases efficiency of this concept.

Vision-Based Assistance for Myoelectric Hand Control

Conventional control systems for prosthetic hands use myoelectric signals as an interface, but it is impossible to realize complex and flexible human hand movements with only myoelectric signals. A promising control scheme for prosthetic hands uses computer vision to assist in grasping objects. It features an imaging sensor, and the control system is capable of recognizing an object placed in the environment. Then, a gripping pattern can be selected from some predefined candidates according the recognized object. However, previous studies assumed that only one object exists in the environment. If there are multiple target objects in the environment, the hand could become confused in attempting to find the target object. This study addresses this problem and proposes a method to determine the target object from multiple objects. The proposed method is able to determine the target object by estimating the positional relationship between the artificial hand and the objects, as well as the motion of the hand. To verify the validity and effectiveness, we implemented the proposed method in a vision-based prosthetic hands control system and conducted pick-and-place experiments. The experiments confirm that the proposed method can accurately estimate the target object in accordance with the user's intention.

Classification of hand movements using variational mode decomposition and composite permutation entropy index with surface electromyogram signals

Research of human hand movements recognition can be applied to artificial limb control, motion recognition of wearable exoskeleton, human–computer interaction in virtual reality and so on. Surface Electromyogram (sEMG) signal is the preferred source. There are many researches on how to extract information from sEMG signal and apply it to human motion recognition. However, how to extract the feature signal from sEMG signal is a difficult problem in the research of human hand movement recognition based on sEMG signal. In this paper, a method based on Variational Mode Decomposition (VMD) and composite permutation entropy index (CPEI) method is proposed for hand motion classification. Previously, the VMD method had not been used in human hand motion recognition studies. The method proposed in this work applies the VMD method to decompose the original sEMG signal into multiple Variational Mode Functions (VMFs) and calculate the corresponding CPEI of each signal component. Three feature selection methods (Infinite Latent Feature Selection (ILFS), ReliefF, and Laplacian Score) were applied to rank the features and remove the unimportant features. Three classifiers (Naive Bayes, K-NN, and Bagging) were used to recognize the hand actions. Ten volunteers participated in the experiment, and the experimental data were used to verify the proposed method. The average accuracy was 94.28 ± 1.26% for the proposed method with Laplacian Score for feature sorting and selection, and Bagging as classifier. Besides, 600 randomly selected hand movements are predicted (CPU is i5-8250U, ram is 8g, processing software is Spyder, python 3.7), and the corresponding execution time of proposed method is 0.56 s.

Virtual Reality Robot-Assisted Welding Based on Human Intention Recognition

We propose an innovative approach to enhance welding operations by using a cyber-physical system (CPS) with layered architecture and enabling a robot to be effectively operated by its commanding human. This article focuses on the recognition of the commanding human’s intention that should be executed by the robot. To this end, a virtual reality (VR) system based on the HTC Vive is used to create a remote virtual welding environment. Human hand movement speed data is collected and used to train a hidden Markov model (HMM) using the Baum-Welch algorithm. The Bayesian information criterion (BIC) is applied to determine the number of hidden states. The state occupancy probability distribution (the probability of each state at a given time) is estimated based on the human hand movement speed sequence using the forward algorithm. The human intention, defined as the intended movement in this article, is then estimated as the statistical expectation of the observable variables. Using the proposed human intention estimation algorithm, the intended movement recognized from the raw movement data is smoother, which is preferable in welding tasks. A 6-DoF industrial robot, UR-5 with a custom gas tungsten arc welding (GTAW) torch installed, works as the final performer of the welding jobs. The robot receives the intended movement data from the HMM and uses this to assist the human welding operators. Welding experiments have been conducted both with and without the proposed human intention recognition (IR) algorithm. The results show that the robot can help the operators complete welding tasks with better performance using the proposed IR system, supporting the effectiveness of the proposed VR robot-assisted welding system. Note to Practitioners —Welding is not only labor-intensive but also requires real-time adaption to the process, which is challenging for robots/machines but relatively straightforward for humans. Using a human commanded robot to perform welding can liberate humans from laborious operations and hazardous environments. To this end, a virtual reality (VR) system is used to create a virtual welding environment for a human to view the process remotely and for a human to pass his/her resultant adaptation to the robot through hand movements. However, hand movements do not always fully represent the intended adaptation of the commanding human, and the recognition of such human intention is fundamental in such a proposed method. This article established the mathematical framework for the recognition of the human intention and, thus, the foundation for effective assistance of robots to humans.