Hand Motion Detection Research Articles

Soft Modular Glove with Multimodal Sensing and Augmented Haptic Feedback Enabled by Materials' Multifunctionalities.

Immersive communications rely on smart perception based on diversified and augmented sensing and feedback technologies. However, the increasing of functional components also raises the issue of increased system complexity. Here, we propose a modular soft glove with multimodal sensing and feedback functions by exploring and utilizing the multiple properties of glove materials. With a single design of basic structure, the main functional unit possesses triboelectric-based sensing of static and dynamic contact, vibration, strain, and pneumatic actuation. Additionally, the same unit is also capable of offering pneumatic tactile haptic feedback and electroresistive thermal haptic feedback. Together with a machine learning algorithm, the proposed glove not only performs real-time detection of dexterous hand motion and direct feedback but also realizes intelligent object recognition and augmented feedback, which significantly enhance the communication and perception of more comprehensive information. In general, this glove utilizes a facile designed sensing and feedback device to achieve dual-way and multimodal communication among humans, machines, and the virtual world via smart perceptions.

Low-Cost Wearable Band Sensors of Surface Electromyography for Detecting Hand Movements.

Surface electromyography (sEMG) is a non-invasive measure of electrical activity generated due to muscle contraction. In recent years, sEMG signals have been increasingly used in diverse applications such as rehabilitation, pattern recognition, and control of orthotic and prosthetic systems. This study presents the development of a versatile multi-channel sEMG low-cost wearable band system to acquire 4 signals. In this case, the signals acquired with the proposed device have been used to detect hand movements. However, the WyoFlex band could be used in some sections of the arm or the leg if the section’s diameter matches the diameter of the WyoFlex band. The designed WyoFlex band was fabricated using three-dimensional (3D) printing techniques employing thermoplastic polyurethane and polylactic acid as manufacturing materials. Then, the proposed wearable electromyographic system (WES) consists of 2 WyoFlex bands, which simultaneously allow the wireless acquisition of 4 sEMG channels of each forearm. The collected sEMG can be visualized and stored for future post-processing stages using a graphical user interface designed in Node-RED. Several experimental tests were conducted to verify the performance of the WES. A dataset with sEMG collected from 15 healthy humans has been obtained as part of the presented results. In addition, a classification algorithm based on artificial neural networks has been implemented to validate the usability of the collected sEMG signals.

A non-contact system for intraoperative quantitative assessment of bradykinesia in deep brain stimulation surgery

Background and Objective: Deep brain stimulation (DBS) is an effective treatment for a number of neurological diseases, especially for the advanced stage of Parkinson’s disease (PD). Objective assessment of patients’ motor symptoms is crucial for accurate electrode targeting and treatment. Existing approaches suffer from subjective variability or interference with voluntary motion. This work is aimed to establish an objective assessment system to quantify bradykinesia in DBS surgery. Methods: Based on the analysis of the requirements for intraoperative assessment, we developed a system with non-contact measurement, online movement feature extraction, and interactive data analysis and visualization. An optical sensor, Leap Motion Controller (LMC), was taken to detect hand movement in three clinical tasks. A graphic user interface was designed to process, compare and visualize the collected data and assessment results online. Quantified movement features include amplitude, frequency, velocity, their decrement and variability, etc. Technical validation of the system was performed with a motion capture system (Mocap), with respect to data-level and feature-level accuracy and reliability. Clinical validation was conducted with 20 PD patients for intraoperative assessments in DBS surgery. Treatment responses with respect to the bradykinesia movement features were analyzed. Single case analysis and group statistical analysis were performed to examine the differences between preoperative and intraoperative performance, and the correlation between the clinical ratings and the quantified assessment was analyzed. Results:For the movements measured by LMC and Mocap, the average Pearson’s correlation coefficient was 0.986, and the mean amplitude difference was 2.11 mm. No significant difference was found for all movement features quantified by LMC and Mocap. For the clinical tests, key movement features showed significant differences between the preoperative baseline and intraoperative performance when the brain stimulation was ON. The assessment results were significantly correlated with the MDS-UPDRS clinical ratings. Conclusions:The proposed non-contact system has established itself as an objective intraoperative assessment, analysis, and visualization tool for DBS treatment of Parkinson’s disease.

SEMG Signals Characterization and Identification of Hand Movements by Machine Learning Considering Sex Differences

Developing a robust machine-learning algorithm to detect hand motion is one of the most challenging aspects of prosthetic hands and exoskeleton design. Machine-learning methods that considered sex differences were used to identify and describe hand movement patterns in healthy individuals. To this purpose, surface Electromyographic (sEMG) signals have been acquired from muscles in the forearm and hand. The results of statistical analysis indicated that most of the same muscle pairs in the right hand (females and males) showed significant differences during the six hand movements. Time features were used an as input to machine-learning algorithms for the recognition of six gestures. Specifically, two types of hand-gesture recognition methods that considered sex differences(differentiating sex datasets and adding a sex label)were proposed and applied to the k-nearest neighbor (k-NN), support vector machine (SVM) and artificial neural network (ANN) algorithms for comparison. In addition, a t-test statistical analysis approach and 5-fold cross validation were used as complements to verify whether considering sex differences could significantly improve classification performance. It was demonstrated that considering sex differences can significantly improve classification performance. The ANN algorithm with the addition of a sex label performed best in movement classification (98.4% accuracy). In the future, hand movement recognition algorithms considering sex differences could be applied to control systems for prosthetic hands or exoskeletons.

Enhanced Recognition of Amputated Wrist and Hand Movements by Deep Learning Method Using Multimodal Fusion of Electromyography and Electroencephalography.

Motion classification can be performed using biometric signals recorded by electroencephalography (EEG) or electromyography (EMG) with noninvasive surface electrodes for the control of prosthetic arms. However, current single-modal EEG and EMG based motion classification techniques are limited owing to the complexity and noise of EEG signals, and the electrode placement bias, and low-resolution of EMG signals. We herein propose a novel system of two-dimensional (2D) input image feature multimodal fusion based on an EEG/EMG-signal transfer learning (TL) paradigm for detection of hand movements in transforearm amputees. A feature extraction method in the frequency domain of the EEG and EMG signals was adopted to establish a 2D image. The input images were used for training on a model based on the convolutional neural network algorithm and TL, which requires 2D images as input data. For the purpose of data acquisition, five transforearm amputees and nine healthy controls were recruited. Compared with the conventional single-modal EEG signal trained models, the proposed multimodal fusion method significantly improved classification accuracy in both the control and patient groups. When the two signals were combined and used in the pretrained model for EEG TL, the classification accuracy increased by 4.18–4.35% in the control group, and by 2.51–3.00% in the patient group.

A BPR-CNN Based Hand Motion Classifier Using Electric Field Sensors

In this paper, we propose a BPR-CNN (Biometric Pattern Recognition-Convolution Neural Network) classifier for hand motion classification as well as a dynamic threshold algorithm for motion signal detection and extraction by EF (Electric Field) sensors. Currently, an EF sensor or EPS (Electric Potential Sensor) system is attracting attention as a next-generation motion sensing technology due to low computation and price, high sensitivity and recognition speed compared to other sensor systems. However, it remains as a challenging problem to accurately detect and locate the authentic motion signal frame automatically in real-time when sensing body-motions such as hand motion, due to the variance of the electric-charge state by heterogeneous surroundings and operational conditions. This hinders the further utilization of the EF sensing; thus, it is critical to design the robust and credible methodology for detecting and extracting signals derived from the motion movement in order to make use and apply the EF sensor technology to electric consumer products such as mobile devices. In this study, we propose a motion detection algorithm using a dynamic offset-threshold method to overcome uncertainty in the initial electrostatic charge state of the sensor affected by a user and the surrounding environment of the subject. This method is designed to detect hand motions and extract its genuine motion signal frame successfully with high accuracy. After setting motion frames, we normalize the signals and then apply them to our proposed BPR-CNN motion classifier to recognize their motion types. Conducted experiment and analysis show that our proposed dynamic threshold method combined with a BPR-CNN classifier can detect the hand motions and extract the actual frames effectively with 97.1% accuracy, 99.25% detection rate, 98.4% motion frame matching rate and 97.7% detection & extraction success rate.

Participant-specific classifier tuning increases the performance of hand movement detection from EEG in patients with amyotrophic lateral sclerosis

Objective. Brain–computer interface (BCI) systems can be employed to provide motor and communication assistance to patients suffering from neuromuscular diseases, such as amyotrophic lateral sclerosis (ALS). Movement related cortical potentials (MRCPs), which are naturally generated during movement execution, can be used to implement a BCI triggered by motor attempts. Such BCI could assist impaired motor functions of ALS patients during disease progression, and facilitate the training for the generation of reliable MRCPs. The training aspect is relevant to establish a communication channel in the late stage of the disease. Therefore, the aim of this study was to investigate the possibility of detecting MRCPs associated to movement intention in ALS patients with different levels of disease progression from slight to complete paralysis. Approach. Electroencephalography signals were recorded from nine channels in 30 ALS patients at various stages of the disease while they performed or attempted to perform hand movements timed to a visual cue. The movement detection was implemented using offline classification between movement and rest phase. Temporal and spectral features were extracted using 500 ms sliding windows with 50% overlap. The detection was tested for each individual channel and two surrogate channels by performing feature selection followed by classification using linear and non-linear support vector machine and linear discriminant analysis. Main results. The results demonstrated that the detection performance was high in all patients (accuracy 80.5 ± 5.6%) but that the classification parameters (channel, features and classifier) leading to the best performance varied greatly across patients. When the same channel and classifier were used for all patients (participant-generic analysis), the performance significantly decreased (accuracy 74 ± 8.3%). Significance. The present study demonstrates that to maximize the detection of brain waves across ALS patients at different stages of the disease, the classification pipeline should be tuned to each patient individually.

Case report: restored vision after ocular Closantel intoxication and blindness

BackgroundClosantel is the best-known anti-parasitic medicine for veterinarians, which is contraindicated in humans. After reviewing the literature on ocular toxicity following mistaken usage of Closantel in humans, this report was found as the first complete restoration of visual function after Closantel intoxication. This report could be useful in anticipating the possibility of a further improvement based on a dose-response relationship. An important point of this report is the apparent reversibility of the vision and Electrophysiological parameters after Closantel intoxication and blindness. To conclude, the present case report demonstrates the importance of immediate referral and management in Closantel intoxication to avoid the long-term adverse effects of drug on visual function.Case presentationA 47-year-old man mistakenly took about 20 cc of Closantel 5% (15.87 mg/kg). Four hours after mistaken usage of Closantel, he was transferred to the district hospital due to dizziness and nausea. His stomach was washed out immediately after hospital arrival. He was being hospitalized in that hospital for 3 days. Then, he was referred to our clinic due to progressive vision loss. Methylprednisolone acetate 250 mg was injected once on 5th day after taking Closantel. His vision was reducing gradually so low that he could only detect hand motion (HM) on the 14th day after taking Closantel. ERG test was requested. It showed an exclusive reduction in b-wave amplitude under photopic and scotopic conditions. Later, his vision surprisingly improved gradually and his visual acuity was fully restored on the 28th day after the incident. After 3 years, we checked him again. His visual acuity was 20/20 in both eyes and the patient did not have any problem and his ERG report was completely normal.ConclusionsIn low dose of Closantel and immediate referral, ocular toxicity could be resolved.

Design and Build of Automatic Hand Sanitizer Using Arduino

An automatic handwashing deviceis a handwashing devicethat works automatically by utilizing an infrared sensor as a hand detector and using the Arduino Uno as the main controller. In this study, the authors discussed the design of an Automatic Handwashing Devicethat is placed on the hand-sanitizer faucet using the InfraRed sensor (detecting hand movements) based on Arduino Uno. This system includes the design of hardware (Hardware) and software (Software). Researchers use descriptive analysis techniques that are presented in table form. In this study, the authors conducted an analysis and design of the devices used to build the input and output processes including System Algorithms, Research Design, and Research Flowcharts. The results showed that the Hands Sanitizer deviceautomatically runs well and can be assembled using Arduino Uno microcontroller components and IR (Infrared) sensors. In making this program, the Arduino Uno application software is needed.

Design and Build of Automatic Hand Sanitizer Using Arduino

An automatic handwashing deviceis a handwashing devicethat works automatically by utilizing an infrared sensor as a hand detector and using the Arduino Uno as the main controller. In this study, the authors discussed the design of an Automatic Handwashing Devicethat is placed on the hand-sanitizer faucet using the InfraRed sensor (detecting hand movements) based on Arduino Uno. This system includes the design of hardware (Hardware) and software (Software). Researchers use descriptive analysis techniques that are presented in table form. In this study, the authors conducted an analysis and design of the devices used to build the input and output processes including System Algorithms, Research Design, and Research Flowcharts. The results showed that the Hands Sanitizer deviceautomatically runs well and can be assembled using Arduino Uno microcontroller components and IR (Infrared) sensors. In making this program, the Arduino Uno application software is needed.

P‐15.4: Virtual reality gesture recognition based on depth information

With the rapid development of computer technology, the information revolution is constantly affecting and changing our lives. Today's virtual reality technology is no longer just a virtual model for viewing. Experiencers are more willing to personally interact with the virtual environment in a virtual scene to increase their sense of gaming experience.Today's virtual reality interaction methods mainly interact through the handles that come with VR helmets. Previously, some scholars have researched gesture interaction in virtual environments. They use data gloves to obtain information about hand movements. This method makes users have to wear heavy data gloves, which limits the natural movement of their hands. To overcome this shortcoming, the need for low‐cost vision‐based virtual reality gesture interaction is becoming more and more urgent.Traditional gesture recognition technologies based on artificial modeling include template‐based matching, data‐based gloves, and machine learning. As machine learning gesture classifiers, Hidden Markov Model (HMM) and Support Vector Machine (SVM) [1] have been widely used. However, in the hand motion detection of ordinary cameras, the image is prone to speckles, lighting changes, and other factors, which seriously affects the performance of the classifier, making it difficult to accurately and efficiently detect the first image when the background is complex and the lighting is affected. In recent years, deep learning‐based object detection methods have achieved great breakthroughs [2]. In the past decade, many deep learning‐based dynamic gesture recognition algorithms have been introduced. In the case of changing lighting conditions, in order to improve the classification accuracy of gesture recognition [3]. Therefore, this article chooses a depth camera combined with HTC VIVE device for gesture recognition in virtual reality. First obtain the depth image of the hand through the depth camera, then label the depth image, and then learn through the deep learning network hand movement training [4], preprocess the detected human hand image, and recognize it by hand detection The position of the hand in the image is obtained, and the movement of the hand is recognized as a result of recognition.The algorithm in this paper uses the advanced deep learning network YOLOv3, and changes the network structure and multiscale structure based on the original network. This algorithm is compared with other deep learning algorithms for gesture recognition, and the recognition effect is better than other algorithms in the same category.

Effective Multi-Mode Grasping Assistance Control of a Soft Hand Exoskeleton Using Force Myography.

Human intention detection is fundamental to the control of robotic devices in order to assist humans according to their needs. This paper presents a novel approach for detecting hand motion intention, i.e., rest, open, close, and grasp, and grasping force estimation using force myography (FMG). The output is further used to control a soft hand exoskeleton called an SEM Glove. In this method, two sensor bands constructed using force sensing resistor (FSR) sensors are utilized to detect hand motion states and muscle activities. Upon placing both bands on an arm, the sensors can measure normal forces caused by muscle contraction/relaxation. Afterwards, the sensor data is processed, and hand motions are identified through a threshold-based classification method. The developed method has been tested on human subjects for object-grasping tasks. The results show that the developed method can detect hand motions accurately and to provide assistance w.r.t to the task requirement.

Classification of Hand Movements from EMG Signals for People with Motor Disabilities

People with disabilities correspond to about 25% of the Brazilian population. A great part of these people have physical impairments that difficult the use computer peripherals. This article presents the development of a system for detection of hand movements through the acquisition and classification of electromyographic (EMG) signals using machine learning techniques. The purpose of the proposed system is to be used by people with disabilities to control an adapted text editor. The signals are capture by surface EMG electrodes and used to the detect 4 different hand movements. In addition, a database with 3200 EMG signals generated by the hand movements was created, made by one user diagnosed with cerebral palsy and another user without diagnosed motor disabilities. Several tests were carried out, showing the good accuracy of the proposed system, with a success classification rate of 96% to 98%.

Virtual reality gesture recognition based on depth information

With the rapid development of computer technology, the information revolution is constantly affecting and changing our lives. Today’s virtual reality technology is no longer just a virtual model for viewing. Experiencers are more willing to personally interact with the virtual environment in a virtual scene to increase their sense of gaming experience.Today’s virtual reality interaction methods mainly interact through the handles that come with VR helmets. Previously, some scholars have researched gesture interaction in virtual environments. They use data gloves to obtain information about hand movements. This method makes users have to wear heavy data gloves, which limits the natural movement of their hands. To overcome this shortcoming, the need for low‐cost vision‐based virtual reality gesture interaction is becoming more and more urgent.Traditional gesture recognition technologies based on artificial modeling include template‐based matching, data‐based gloves, and machine learning. As machine learning gesture classifiers, Hidden Markov Model (HMM) and Support Vector Machine (SVM) [1] have been widely used. However, in the hand motion detection of ordinary cameras, the image is prone to speckles, lighting changes, and other factors, which seriously affects the performance of the classifier, making it difficult to accurately and efficiently detect the first image when the background is complex and the lighting is affected. In recent years, deep learning‐based object detection methods have achieved great breakthroughs [2]. In the past decade, many deep learning‐based dynamic gesture recognition algorithms have been introduced. In the case of changing lighting conditions, in order to improve the classification accuracy of gesture recognition [3]. Therefore, this article chooses a depth camera combined with HTC VIVE device for gesture recognition in virtual reality. First obtain the depth image of the hand through the depth camera, then label the depth image, and then learn through the deep learning network hand movement training [4], preprocess the detected human hand image, and recognize it by hand detection The position of the hand in the image is obtained, and the movement of the hand is recognized as a result of recognition.The algorithm in this paper uses the advanced deep learning network YOLOv3, and changes the network structure and multi‐scale structure based on the original network. This algorithm is compared with other deep learning algorithms for gesture recognition, and the recognition effect is better than other algorithms in the same category.

Efficient Hand Movement Detection Using k-Means Clustering and k-Nearest Neighbor Algorithms

Electromyography (EMG) signals are commonly used in prosthetic limb studies. We have proposed a system to detect six basic hand movements using unsupervised and supervised classification algorithms. In this study, two-channel EMG recordings belonging to six different hand movements are analyzed and the performance of the wavelet-based features for hand movement clustering and classification are examined for six subjects (three females and three males). The approximation and detail components are obtained by four-level symmetric wavelet transform. The energy, mean, standard deviation, and entropy values of the wavelet components are calculated and the feature sets are generated. After feature extraction, feature set dimensionality is reduced using principal component analysis, and then the k-nearest neighbor method and k-means clustering are applied for classification and clustering, respectively. The analyses are performed subject-specifically and gender-specifically. Thus, it is possible to evaluate the gender effect on classification performances. Subject-specific hand movements were detected with accuracy in the range of 86.33–100%. Gender-specific hand movements were detected with an accuracy of 96.67% for males and 92.78% for females. The classification and clustering results support each other. It was observed that the samples of hand movements that were classified incorrectly were concentrated in the same clusters. Similarly, it was found that the hand movements that were easily detected were homogeneously clustered.

Hand Motion Analysis using CNN

-Hand motion detection and gesture recognition research has attracted large interest due to its wide range of applications in the field of Human computer interaction such as sign language recognition, 3D printing, virtual reality. There have been several approaches to create a robust algorithm to ease human computer interaction and perform in unfavourable environments. The real time recognition and learning of the model are big challenges. In this work, we use Convolutional Neural Network architecture to detect and classify hand motions, the region of interest of the image is passed through the neural network for the hand motion analysis and detection. Our system has achieved testing accuracy of 98%.

Textile-Based Stretchable and Flexible Glove Sensor for Monitoring Upper Extremity Prosthesis Functions

To detect various signals and motions from the human body, textile substrates having the compatibility with human skin and high electrical/mechanical properties are strongly required in wearable healthcare sensors. However, there is still limitation in forming electrical materials directly on fabric substrates due to inherent fabric properties such as porousness and roughness of surface. In this study, by using stretchable and flexible polydimethylsiloxane (PDMS) films as both interface/ planarization and passivation layers, we demonstrate a textile-based AgNW sensor with high conductivity (434.7 S/cm) on polyester-based spandex fabric substrate. In addition, in order to detect hand motions, we designed a full-sized glove-type sensor by integrating five strain sensor units, showing stable responsibility to each finger motions individually. Furthermore, to expand the possibilities of the textile-based glove sensor in prosthesis monitoring applications, the glove sensor was applied to a 3D-printed prosthetic hand to detect hand motion or gestures and showed stable corresponding responses to different grasps (against normal and small-sized balls). From successful results in practical 3D-printed prosthetic application, we believe that the textile-based sensor developed in this study is appropriate for human motion monitoring and potential candidates for integrated prosthetic hands and medical robotic hand.

Designing and Developing Four Games for Rehabilitation of the Wrist Complex and Forearm Complex: An Action Research

Introduction: Virtual reality (VR) games are among the rehabilitation strategies that, while effective, are attractive to the individual and facilitate and encourage proper movement. In this study, four VR games were designed and assessed to encourage the movements of the forearm and wrist complexs. Materials and Methods: The games were designed using the leap motion device, a small non-tactile device with good spatial resolution. Cameras were responsible for detecting the depth and distance of the hand from the device, and the infrared sensors were responsible for detecting hand movements. In order to determine the level of attractiveness of the game for people with upper motor neuron disorders, three individuals aged 18 to 41 were randomly selected from among the people referred to the physiotherapy department of Al-Zahra Educational and Medical Center, Tehran, Iran. Results: The effectiveness of the games was confirmed by board members of department of physical therapy. Three games were applicable in the early stages of rehabilitation for all three participants. One game could not be used in the early stages of rehabilitation due to the complexity of the required movement. Conclusion: The four designed games required the basic movements required for daily and self-care activities and seemed to appeal the younger users.

Speeding up the detection of non-iconic and iconic gestures (SPUDNIG): A toolkit for the automatic detection of hand movements and gestures in video data

In human face-to-face communication, speech is frequently accompanied by visual signals, especially communicative hand gestures. Analyzing these visual signals requires detailed manual annotation of video data, which is often a labor-intensive and time-consuming process. To facilitate this process, we here present SPUDNIG (SPeeding Up the Detection of Non-iconic and Iconic Gestures), a tool to automatize the detection and annotation of hand movements in video data. We provide a detailed description of how SPUDNIG detects hand movement initiation and termination, as well as open-source code and a short tutorial on an easy-to-use graphical user interface (GUI) of our tool. We then provide a proof-of-principle and validation of our method by comparing SPUDNIG’s output to manual annotations of gestures by a human coder. While the tool does not entirely eliminate the need of a human coder (e.g., for false positives detection), our results demonstrate that SPUDNIG can detect both iconic and non-iconic gestures with very high accuracy, and could successfully detect all iconic gestures in our validation dataset. Importantly, SPUDNIG’s output can directly be imported into commonly used annotation tools such as ELAN and ANVIL. We therefore believe that SPUDNIG will be highly relevant for researchers studying multimodal communication due to its annotations significantly accelerating the analysis of large video corpora.

A Model-Based RF Hand Motion Detection System for Shadowing Scenarios

With the explosive growth of mobile computing, new modes of human-computer interaction (HCI) are emerging and becoming feasible. Compared to vision-based systems that require lighting, radio frequency (RF)-based hand motion detection systems are becoming more popular in HCI applications. In real RF hand motion detection scenarios, the line-of-sight between the transmitter (Tx) and receiver (Rx) is usually blocked. Hence, shadowing significantly affects the detection accuracy. To design better RF hand motion detection systems, we propose a simple diffraction and interference model (DIM) to interpret the received signal strength (RSS) variation caused by hand motions in the shadowing scenario. Based on theories of knife-edge diffraction and mutual radio interference, DIM provides a simple theoretical foundation for analyzing the RSS variation with hand size, signal frequency, and Tx-Rx distance. Furthermore, a model-based RF hand motion detection system benefiting from DIM is presented. Unlike existing systems that require a large number of motion features to train a motion classifier, the model-based system achieves training-free motion classification, which has potential for hand motion detection on a real-time basis. Empirical data collected from a vector network analyzer validate our system as well as demonstrate a simple diffraction model can help hand motion detection processing for commonly growing HCI applications.