Hand Finger Movements Research Articles

Applying learning-from-observation to household service robots: three task common-sense formulations

Utilizing a robot in a new application requires the robot to be programmed at each time. To reduce such programmings efforts, we have been developing “Learning-from-observation (LfO)” that automatically generates robot programs by observing human demonstrations. So far, our previous research has been in the industrial domain. From now on, we want to expand the application field to the household-service domain. One of the main issues with introducing this LfO system into the domain is the cluttered environments, which makes it difficult to discern which movements of the human body parts and their relationships with environment objects are crucial for task execution when observing demonstrations. To overcome this issue, it is necessary for the system to have task common-sense shared with the human demonstrator to focus on the demonstrator's specific movements. Here, task common-sense is defined as the movements humans take almost unconsciously to streamline or optimize the execution of a series of tasks. In this paper, we extract and define three types of task common-sense (semi-conscious movements) that should be focused on when observing demonstrations of household tasks and propose representations to describe them. Specifically, the paper proposes to use labanotation to describe the whole-body movements with respect to the environment, contact-webs to describe the hand-finger movements with respect to the tool for grasping, and physical and semantic constraints to describe the movements of the hand with the tool with respect to the environment. Based on these representations, the paper formulates task models, machine-independent robot programs, that indicate what-to-do and where-to-do. In this design process, the necessary and sufficient set of task models to be prepared in the task-model library are determined on the following criteria: for grasping tasks, according to the classification of contact-webs along the purpose of the grasping, and for manipulation tasks, corresponding to possible transitions between states defined by either physical constraints and semantic constraints. The skill-agent library is also prepared to collect skill-agents corresponding to tasks. The skill-agents in the library are pre-trained using reinforcement learning with the reward functions designed based on the physical and semantic constraints to execute the task when specific parameters are provided. Third, the paper explains the task encoder to obtain task models and task decoder to execute the task models on the robot hardware. The task encoder understands what-to-do from the verbal input and retrieves the corresponding task model in the library. Next, based on the knowledge of each task, the system focuses on specific parts of the demonstration to collect where-to-do parameters for executing the task. The decoder constructs a sequence of skill-agents retrieving from the skill-agnet library corresponding and inserts those parameters obtained from the demonstration into these skill-agents, allowing the robot to perform task sequences with following the Labanotation postures. Finally, this paper presents how the system actually works through several example scenes.

Intelligent Image Processing System Based on Virtual Painting

Image processing and artificial intelligence have come so far in the age of technology that they are now used in every field. AI is meant to help people do their jobs better and with less help. Virtual painting is a clever tool that lets us paint by just moving our hands. This app uses hand tracing and finger movement recording to show marks. With this program, we can open the camera on our computer or laptop and draw with our hands, colored pens, or pens we made ourselves. Image processing is used to figure out how the hand is moving. This turns the person in front of the camera into a great illustrator who can pick out colors and draw in similar directions on the screen. OpenCV, which is the brain of the system, and Media Pipe have been added to this application.

Extra-neural signals from severed nerves enable intrinsic hand movements in transhumeral amputations

Robotic prostheses controlled by myoelectric signals can restore limited but important hand function in individuals with upper limb amputation. The lack of individual finger control highlights the yet insurmountable gap to fully replacing a biological hand. Implanted electrodes around severed nerves have been used to elicit sensations perceived as arising from the missing limb, but using such extra-neural electrodes to record motor signals that allow for the decoding of phantom movements has remained elusive. Here, we showed the feasibility of using signals from non-penetrating neural electrodes to decode intrinsic hand and finger movements in individuals with above-elbow amputations. We found that information recorded with extra-neural electrodes alone was enough to decode phantom hand and individual finger movements, and as expected, the addition of myoelectric signals reduced classification errors both in offline and in real-time decoding.

Modeling-Based EMG Signal (MBES) Classifier for Robotic Remote-Control Purposes

The fast-growing human–robot collaboration predicts that a human operator could command a robot without mechanical interface if effective communication channels are established. In noisy, vibrating and light sensitive environments, some sensors for detecting the human intention could find critical issues to be adopted. On the contrary, biological signals, as electromyographic (EMG) signals, seem to be more effective. In order to command a laboratory collaborative robot powered by McKibben pneumatic muscles, promising actuators for human–robot collaboration due to their inherent compliance and safety features have been researched, a novel modeling-based electromyographic signal (MBES) classifier has been developed. It is based on one EMG sensor, a Myotrac one, an Arduino Uno and a proper code, developed in the Matlab environment, that performs the EMG signal recognition. The classifier can recognize the EMG signals generated by three hand-finger movements, regardless of the amplitude and time duration of the signal and the muscular effort, relying on three mathematical models: exponential, fractional and Gaussian. These mathematical models have been selected so that they are the best fitting with the EMG signal curves. Each of them can be assigned a consent signal for performing the wanted pick-and-place task by the robot. An experimental activity was carried out to test and achieve the best performance of the classifier. The validated classifier was applied for controlling three pressure levels of a McKibben-type pneumatic muscle. Encouraging results suggest that the developed classifier can be a valid command interface for robotic purposes.

Voluntary and FES-Induced Finger Movement Estimation Using Muscle Deformation Features

It is practically crucial to estimate dexterous hand movement for applications such as prosthetics and rehabilitation, especially scenarios actively assisted with functional electrical stimulation (FES). This paper attempts to estimate hand finger movement through forearm muscle deformation for both clinical scenarios with voluntary and FES-induced muscle contractions based on A-mode ultrasound sensing approaches. Six healthy subjects participated in the experiment of middle finger movement. In order to evaluate the sensing approach, a least-squares support-vector-machine (LS-SVM) model and a linear-ridge (LR) model were developed to estimate the hand finger movement for datasets captured from the flexor digitorum superficialis (FDS) muscle belly, extensor digitorum muscle belly with or without FES. The average results of using FDS datasets as inputs revealed excellent evaluation performance ( $R^2>0.9$ and $\text{RMSE} ) both for LR and LS-SVM regression technologies. Furthermore, finger-movements-related muscle deformation features corresponding to voluntary and FES-induced contraction were proved to be different. These results demonstrated the potential application of the forearm muscle deformation in wearable A-mode ultrasound-based prosthesis finger movements control and optimizing the closed-loop FES system for motor function rehabilitation.

Changes of γ Band Oscillation during Non-dominant Hand Finger Movements

Changes of γ Band Oscillation during Non-dominant Hand Finger Movements

1-DOF Finger Mechanism Design for Anthropomorphic Prosthetic Hand with Similarity Criteria for Human Hand Finger Movement

Development of prosthetic hands continues to be made to get prosthetic which has special characteristics, namely anthropomorphism. The anthropomorphic prosthetic design refers to the improvement and development of the design to the stage of the similarity of the prosthetic movement to human hand movements. This study carries the design of the anthropomorphic 1-DOF prosthetic finger mechanism to get prosthetic at an affordable price. The optimization criteria for the similarity of movement with the human hand are formulated with two objective functions, namely the similarity of the range of motion and the total length of the finger that is completed simultaneously. The human hand movement that is a reference is the movement of conical object grasping on a standard size conical according to the maximum hand-held diameter anthropometry of an Indonesian people.

Highly Stretchable, Elastic, and Ionic Conductive Hydrogel for Artificial Soft Electronics

AbstractHigh conductivity, large mechanical strength, and elongation are important parameters for soft electronic applications. However, it is difficult to find a material with balanced electronic and mechanical performance. Here, a simple method is developed to introduce ion‐rich pores into strong hydrogel matrix and fabricate a novel ionic conductive hydrogel with a high level of electronic and mechanical properties. The proposed ionic conductive hydrogel is achieved by physically cross‐linking the tough biocompatible polyvinyl alcohol (PVA) gel as the matrix and embedding hydroxypropyl cellulose (HPC) biopolymer fibers inside matrix followed by salt solution soaking. The wrinkle and dense structure induced by salting in PVA matrix provides large stress (1.3 MPa) and strain (975%). The well‐distributed porous structure as well as ion migration–facilitated ion‐rich environment generated by embedded HPC fibers dramatically enhances ionic conductivity (up to 3.4 S m−1, at f = 1 MHz). The conductive hybrid hydrogel can work as an artificial nerve in a 3D printed robotic hand, allowing passing of stable and tunable electrical signals and full recovery under robotic hand finger movements. This natural rubber‐like ionic conductive hydrogel has a promising application in artificial flexible electronics.

Computational Modeling of Touchscreen Drag Gestures Using a Cognitive Architecture and Motion Tracking

ABSTRACTThis article presents a computational model that predicts finger-drag gesture performance on touchscreen devices, by integrating the queueing network (QN) cognitive architecture and motion tracking. Specifically, the QN-based model was developed to predict two execution times: the finger movement time of drag-gesture (i.e., only the motion time of the finger touched and dragged on the surface of touchscreen) and the comprehensive process time of drag-gesture (i.e., the entire process time to complete the finger-drag task, including visual attention shift, memory storage and retrieval, and hand-finger movements). To develop predictive models for the finger movement time of drag-gesture, 11 participants’ motion data were collected and a regression analysis with parameters of hand-finger anthropometric data and eight angular directions was conducted. Human subject data from our previous study (Jeong & Liu, 2017a) were used to evaluate the QN-based model, generating similar outputs (R2 was more than 80% and root-mean square was less than 300 msec) for both execution times.

B-19. What does eye tracking tell us about neurological patients?

Performances of experimental oculomotor (saccade) paradigms have provided insights into the pathophysiology of neurological patients. However, to investigate the behavioral correlates and the physiological significance of these saccade abnormalities in daily life, we need to look into the difficulty patients actually experience in a more ecological setting. We review studies in which the eye tracking methodology was applied to normal subjects, e.g., when they viewed line drawings presented on a monitor screen to memorize them for later recall. In addition, simultaneous recording of hand (finger) movements with gaze movements allows us to evaluate how the eye and hand coordinates in performing daily actions. For example, eye-hand coordination was studied whey subjects wrote letters on a touch panel, and during a trail making test or in a task simulating finger-nose test. Even eye-voice coordination can be studied by recording the voice of subjects as they read aloud a text. Investigations in neurological patients such as Parkinson’s disease and cerebellar ataxia holds promise for giving insights into how saccade abnormalities observed in oculomotor tasks contribute to their clinical symptoms. They can also be used to quantitatively evaluate motor symptoms of neurological patients from a new perspective.

Behavioral and neural effects of congruency of visual feedback during short-term motor learning

Visual feedback can facilitate or interfere with movement execution. Here, we describe behavioral and neural mechanisms by which the congruency of visual feedback during physical practice of a motor skill modulates subsequent performance gains. 18 healthy subjects learned to execute rapid sequences of right hand finger movements during fMRI scans either with or without visual feedback. Feedback consisted of a real-time, movement-based display of virtual hands that was either congruent (right virtual hand movement), or incongruent (left virtual hand movement yoked to the executing right hand). At the group level, right hand performance gains following training with congruent visual feedback were significantly higher relative to training without visual feedback. Conversely, performance gains following training with incongruent visual feedback were significantly lower. Interestingly, across individual subjects these opposite effects correlated. Activation in the Supplementary Motor Area (SMA) during training corresponded to individual differences in subsequent performance gains. Furthermore, functional coupling of SMA with visual cortices predicted individual differences in behavior. Our results demonstrate that some individuals are more sensitive than others to congruency of visual feedback during short-term motor learning and that neural activation in SMA correlates with such inter-individual differences.

A new hand finger movements’ classification system based on bicoherence analysis of two-channel surface EMG signals

In this study, two-channel surface electromyography (sEMG) signals were used to classify hand finger movements. Bicoherence analysis of the sEMG signal recorded with surface electrodes for flexor and extensor muscle bundles on the front and back of the forearm, respectively, was classified by extreme learning machines (ELM) based on phase matches in the electromyography (EMG) signal. EMG signals belonging to 42 human, 22 males and 20 females, with an average age of 21.4 were used in the study. The finger movements were also classified by using different learning algorithms. The best classification was performed by using ELM algorithm with 98.95 and 97.83% accuracies in average for subjects individually and all together, respectively. On the other hand, a maximum of 95.81 and 94.30% accuracies were reached for subjects individually and all together, respectively, with other learning methods used in the present study. From the information obtained through this study, it is possible to control finger movements by using flexor and extensor muscle activities of the forearm. Furthermore, by this method, it may be possible controlling of the intelligent prosthesis hands with high degree of freedom.

Primary motor cortex contributes to the implementation of implicit value-based rules during motor decisions

In the present study, we investigated the functional contribution of the human primary motor cortex (M1) to motor decisions. Continuous theta burst stimulation (cTBS) was used to alter M1 activity while participants performed a decision-making task in which the reward associated with the subjects’ responses (right hand finger movements) depended on explicit and implicit value-based rules. Subjects performed the task over two consecutive days and cTBS occurred in the middle of Day 2, once the subjects were just about to implement implicit rules, in addition to the explicit instructions, to choose their responses, as evident in the control group (cTBS over the right somatosensory cortex). Interestingly, cTBS over the left M1 prevented subjects from implementing the implicit value-based rule while its implementation was enhanced in the group receiving cTBS over the right M1. Hence, cTBS had opposite effects depending on whether it was applied on the contralateral or ipsilateral M1. The use of the explicit value-based rule was unaffected by cTBS in the three groups of subject. Overall, the present study provides evidence for a functional contribution of M1 to the implementation of freshly acquired implicit rules, possibly through its involvement in a cortico-subcortical network controlling value-based motor decisions.

Effective Connectivity of Cortical Sensorimotor Networks During Finger Movement Tasks: A Simultaneous fNIRS, fMRI, EEG Study.

Recently, interest has been growing to understand the underlying dynamic directional relationship between simultaneously activated regions of the brain during motor task performance. Such directionality analysis (or effective connectivity analysis), based on non-invasive electrophysiological (electroencephalography-EEG) and hemodynamic (functional near infrared spectroscopy-fNIRS; and functional magnetic resonance imaging-fMRI) neuroimaging modalities can provide an estimate of the motor task-related information flow from one brain region to another. Since EEG, fNIRS and fMRI modalities achieve different spatial and temporal resolutions of motor-task related activation in the brain, the aim of this study was to determine the effective connectivity of cortico-cortical sensorimotor networks during finger movement tasks measured by each neuroimaging modality. Nine healthy subjects performed right hand finger movement tasks of different complexity (simple finger tapping-FT, simple finger sequence-SFS, and complex finger sequence-CFS). We focused our observations on three cortical regions of interest (ROIs), namely the contralateral sensorimotor cortex (SMC), the contralateral premotor cortex (PMC) and the contralateral dorsolateral prefrontal cortex (DLPFC). We estimated the effective connectivity between these ROIs using conditional Granger causality (GC) analysis determined from the time series signals measured by fMRI (blood oxygenation level-dependent-BOLD), fNIRS (oxygenated-O2Hb and deoxygenated-HHb hemoglobin), and EEG (scalp and source level analysis) neuroimaging modalities. The effective connectivity analysis showed significant bi-directional information flow between the SMC, PMC, and DLPFC as determined by the EEG (scalp and source), fMRI (BOLD) and fNIRS (O2Hb and HHb) modalities for all three motor tasks. However the source level EEG GC values were significantly greater than the other modalities. In addition, only the source level EEG showed a significantly greater forward than backward information flow between the ROIs. This simultaneous fMRI, fNIRS and EEG study has shown through independent GC analysis of the respective time series that a bi-directional effective connectivity occurs within a cortico-cortical sensorimotor network (SMC, PMC and DLPFC) during finger movement tasks.

EEG pattern decoding of rhythmic individual finger imaginary movements of one hand

The results of four-class classification of the motor imagery EEG patterns corresponding to the right hand finger movements (little finger, thumb, index and middle fingers) of eight healthy subjects are presented in this study. The motor imagery of individual right-hand finger movements was executed by the subjects in a prescribed rhythm and the trials contained no external stimuli. Classification was performed by means of a specially developed two-level committee of classifiers on the basis of support vector machine and artificial neural networks at the first level and by generalizing an artificial neural network at the second level. The area under the EEG signal curve and the curve length calculated in a sliding time window for sites F3, C3, and Cz of the International 10?20 System were selected as the key features of signals from the sensorimotor and adjoining frontal cortical areas contralateral to the movements. The average accuracy of four-class singletrial classification for all subjects was 50 ± 7 [SD] (maximum, 58%) for the pair of sites F3–C3 and 46 ± 11% [SD] (maximum 62%) for the pair of sites C3–Cz with a theoretical guessing level 25%.

Subthalamic stimulation modulates cortical motor network activity and synchronization in Parkinson’s disease

Dynamic modulations of large-scale network activity and synchronization are inherent to a broad spectrum of cognitive processes and are disturbed in neuropsychiatric conditions including Parkinson's disease. Here, we set out to address the motor network activity and synchronization in Parkinson's disease and its modulation with subthalamic stimulation. To this end, 20 patients with idiopathic Parkinson's disease with subthalamic nucleus stimulation were analysed on externally cued right hand finger movements with 1.5-s interstimulus interval. Simultaneous recordings were obtained from electromyography on antagonistic muscles (right flexor digitorum and extensor digitorum) together with 64-channel electroencephalography. Time-frequency event-related spectral perturbations were assessed to determine cortical and muscular activity. Next, cross-spectra in the time-frequency domain were analysed to explore the cortico-cortical synchronization. The time-frequency modulations enabled us to select a time-frequency range relevant for motor processing. On these time-frequency windows, we developed an extension of the phase synchronization index to quantify the global cortico-cortical synchronization and to obtain topographic differentiations of distinct electrode sites with respect to their contributions to the global phase synchronization index. The spectral measures were used to predict clinical and reaction time outcome using regression analysis. We found that movement-related desynchronization of cortical activity in the upper alpha and beta range was significantly facilitated with 'stimulation on' compared to 'stimulation off' on electrodes over the bilateral parietal, sensorimotor, premotor, supplementary-motor, and prefrontal areas, including the bilateral inferior prefrontal areas. These spectral modulations enabled us to predict both clinical and reaction time improvement from subthalamic stimulation. With 'stimulation on', interhemispheric cortico-cortical coherence in the beta band was significantly attenuated over the bilateral sensorimotor areas. Similarly, the global cortico-cortical phase synchronization was attenuated, and the topographic differentiation revealed stronger desynchronization over the (ipsilateral) right-hemispheric prefrontal, premotor and sensorimotor areas compared to 'stimulation off'. We further demonstrated that the cortico-cortical phase synchronization was largely dominated by genuine neuronal coupling. The clinical improvement with 'stimulation on' compared to 'stimulation off' could be predicted from this cortical decoupling with multiple regressions, and the reduction of synchronization over the right prefrontal area showed a linear univariate correlation with clinical improvement. Our study demonstrates wide-spread activity and synchronization modulations of the cortical motor network, and highlights subthalamic stimulation as a network-modulating therapy. Accordingly, subthalamic stimulation may release bilateral cortical computational resources by facilitating movement-related desynchronization. Moreover, the subthalamic nucleus is critical to balance inhibitory and facilitatory cortical players within the motor program.

Oscillatory motor network activity during rest and movement: an fNIRS study.

Coherent network oscillations (<0.1 Hz) linking distributed brain regions are commonly observed in the brain during both rest and task conditions. What oscillatory network exists and how network oscillations change in connectivity strength, frequency and direction when going from rest to explicit task are topics of recent inquiry. Here, we study network oscillations within the sensorimotor regions of able-bodied individuals using hemodynamic activity as measured by functional near-infrared spectroscopy (fNIRS). Using spectral interdependency methods, we examined how the supplementary motor area (SMA), the left premotor cortex (LPMC) and the left primary motor cortex (LM1) are bound as a network during extended resting state (RS) and between-tasks resting state (btRS), and how the activity of the network changes as participants execute left, right, and bilateral hand (LH, RH, and BH) finger movements. We found: (i) power, coherence and Granger causality (GC) spectra had significant peaks within the frequency band (0.01–0.04 Hz) during RS whereas the peaks shifted to a bit higher frequency range (0.04–0.08 Hz) during btRS and finger movement tasks, (ii) there was significant bidirectional connectivity between all the nodes during RS and unidirectional connectivity from the LM1 to SMA and LM1 to LPMC during btRS, and (iii) the connections from SMA to LM1 and from LPMC to LM1 were significantly modulated in LH, RH, and BH finger movements relative to btRS. The unidirectional connectivity from SMA to LM1 just before the actual task changed to the bidirectional connectivity during LH and BH finger movement. The uni-directionality could be associated with movement suppression and the bi-directionality with preparation, sensorimotor update and controlled execution. These results underscore that fNIRS is an effective tool for monitoring spectral signatures of brain activity, which may serve as an important precursor before monitoring the recovery progress following brain injury.

Detecting delay in visual feedback of an action as a monitor of self recognition

How do we distinguish "self" from "other"? The correlation between willing an action and seeing it occur is an important cue. We exploited the fact that this correlation needs to occur within a restricted temporal window in order to obtain a quantitative assessment of when a body part is identified as "self". We measured the threshold and sensitivity (d') for detecting a delay between movements of the finger (of both the dominant and non-dominant hands) and visual feedback as seen from four visual perspectives (the natural view, and mirror-reversed and/or inverted views). Each trial consisted of one presentation with minimum delay and another with a delay of between 33 and 150ms. Participants indicated which presentation contained the delayed view. We varied the amount of efference copy available for this task by comparing performances for discrete movements and continuous movements. Discrete movements are associated with a stronger efference copy. Sensitivity to detect asynchrony between visual and proprioceptive information was significantly higher when movements were viewed from a "plausible" self perspective compared with when the view was reversed or inverted. Further, we found differences in performance between dominant and non-dominant hand finger movements across the continuous and single movements. Performance varied with the viewpoint from which the visual feedback was presented and on the efferent component such that optimal performance was obtained when the presentation was in the normal natural orientation and clear efferent information was available. Variations in sensitivity to visual/non-visual temporal incongruence with the viewpoint in which a movement is seen may help determine the arrangement of the underlying visual representation of the body.

Arthritic Hand‐Finger Movement Similarity Measurements: Tolerance Near Set Approach

The problem considered in this paper is how to measure the degree of resemblance between nonarthritic and arthritic hand movements during rehabilitation exercise. The solution to this problem stems from recent work on a tolerance space view of digital images and the introduction of image resemblance measures. The motivation for this work is both to quantify and to visualize differences between hand-finger movements in an effort to provide clinicians and physicians with indications of the efficacy of the prescribed rehabilitation exercise. The more recent introduction of tolerance near sets has led to a useful approach for measuring the similarity of sets of objects and their application to the problem of classifying image sequences extracted from videos showing finger-hand movement during rehabilitation exercise. The approach to measuring the resemblance between hand movement images introduced in this paper is based on an application of the well-known Hausdorff distance measure and a tolerance nearness measure. The contribution of this paper is an approach to measuring as well as visualizing the degree of separation between images in arthritic and nonarthritic hand-finger motion videos captured during rehabilitation exercise.

Design of an electronic instrumentation for measuring repetitive hand movements during computer use to help prevent work related upper extremity disorder

Repetitive movements for computer users can result in complaints caused by extreme hand posture, finger movements, and force when using the computer, which is known as Work Related Upper Extremity Disorder (WRUED). This study is about the construction of electronic instrumentation for monitoring and quantifying these movements and forces, using sensors to register wrist posture and fingertip force with software developed to collect and process the data. Tests evaluated the performance of the instrumentation with seventeen subjects participating in this study. The maximum extension observed for the first test was 41°, however after training the subject decreased this value to 33°. Six subjects had a wrist extension of between 15° and 41° for the first test; five reduced their wrist extension (between 3° and 33°) during the second test ( p = 0.08) while one subject increased instead of decreased it. No subject performed fingertip force greater than 0.77N during the first test; this was reduced to 0.57N during the second test ( p = 0.04). The average typing frequency in the group decreased from 3.2Hz to 2.5Hz during the second test ( p = 0.01). Results confirm that this solution may potentially contribute to hand movement reeducation, thereby reducing the risk of WRUED for computer users. Relevance to industry Knowledge of repetitive movements during computer use and associated WRUED is essential for prevention. This electronic instrumentation aids the correction of hand movements, which reduces the risk of injury due to inappropriate posture, extreme range of movement, or force during computer use.