Control Of Upper Limb Prostheses Research Articles

A GAN based PID controller for highly adaptive control of a pneumatic-artificial-muscle driven antagonistic joint

AbstractUpper limb prostheses are commonly propelled by pneumatic artificial muscles organized in an antagonistic arrangement. Nonetheless, the control of upper limb prostheses under changing/unknown situations is difficult and necessary for a variety of real-world applications. Adaptive control, learning-based control, and robust control have been studied to deal with such challenges. However, their adaptability is insufficient for prostheses used in daily life, which are exposed to variable task levels, user motor characteristics, and prosthetic features. This paper introduces a highly adaptive controller for the first time based on Generative Adversarial Nets and proportional–integral–derivative controller (G-PID controller). G-PID controller comprises a generator for generating compensation actions to enhance PID responsiveness when controlling the unknown/changing system. Moreover, it incorporates a discriminator that receives responses from both a user-preselected reference system and the compensated changing/unknown system, and simultaneously determines the source of these responses. Through continuous updates, the compensator modifies the response of unknown/changing system to align with the reference system, thereby facilitating adaptive control. The G-PID controller’s effectiveness is evaluated through 1-degree of freedom (DoF) joint and 2-DoF shoulder prostheses in simulation experiments, and further validated in prototype experiments focusing on online learning for unknown and time-varying payload. The results demonstrate its ability to deal with diverse types of unknowns/changes, marking a significant advancement towards incorporating prostheses seamlessly into daily life.

A geometric algebra-based approach for myoelectric pattern recognition control and faster prosthesis recalibration

ObjectiveAlthough many advancements have been made on myoelectric pattern-recognition, the control of poly-articulated upper-limb prostheses remains insufficiently robust. Electrode-shift, sweat or fatigue degrade the performance of classifiers over time, resulting in unfruitful device usage and frequent re-calibration. To tackle this issue, here we introduce two models − µP6 and µP8 − that combine Geometric Algebra with nearest-neighbor classification. We aim at reducing both the necessary training data and training time and, unlike most current state-of-the-art algorithms, we exploit an alternative geometric representation (and visualization) of the EMG signal as different polygons for different types of gestures, facilitating the explanation of the decision-making process to a layman. Moreover, we explore four abstention strategies to reduce the number of misclassifications. MethodsWe perform an offline analysis on two datasets, alongside two other standard models: nonlinear logistic regression (NLR) and linear discriminant analysis (LDA). ResultsEven with few training data, the proposed algorithms achieve high F1-scores (>0.95), significantly higher or non-significantly different from the values obtained with NLR and LDA, while maintaining relatively low abstentions rates and training times (<2 ms). ConclusionThe proposed algorithms allow to reduce the amount of training data and training times without compromising recognition rates. SignificanceThe proposed algorithms may contribute for a faster prosthesis re-calibration procedure while allowing to re-gain high recognition rates. Furthermore, the decision-making process is explainable and interpretable, potentially improving user trust and acceptance.

Toward Impedance Control in Human-Machine Interfaces for Upper-Limb Prostheses.

Adaptation of upper-limb impedance (stiffness, damping, inertia) is crucial for humans to physically interact with the external environment during grasping and manipulation tasks. Here, we present a novel framework for Adaptive Impedance Control of Upper-limb Prosthesis (AIC-UP) based on surface electromyography (sEMG) signals. AIC-UP uses muscle-tendon models driven by sEMG signals from agonist-antagonist muscle groups to estimate the human motor intent as joint kinematics, stiffness and damping. These estimates are used to implement a variable impedance controller on a simulated robot. Designed for use by amputees, joint torque or stiffness measurements are not used for model calibration. AIC-UP was evaluated with eight able-bodied subjects and a transradial amputee performing target-reaching tasks in simulation through wrist flexion-extension. The control performance was tested in free space and in the presence of unexpected perturbations. We show that AIC-UP outperformed a neural network that regresses the desired kinematics from sEMG signals, in terms of robustness to muscle coactivations needed to complete the task. These results were in agreement with the qualitative feedback from the participants. Additionally, we observed that AIC-UP enables the user to adapt the stiffness and damping to the tasks at hand.

Reducing Motor Variability Enhances Myoelectric Control Robustness Across Untrained Limb Positions

Reducing Motor Variability Enhances Myoelectric Control Robustness Across Untrained Limb Positions

Sonomyography for Control of Upper-Limb Prostheses: Current State and Future Directions.

ABSTRACT Problem Statement Despite the recent advancements in technology, many individuals with upper-limb loss struggle to achieve stable control over multiple degrees of freedom in a prosthesis. There is an ongoing need to develop noninvasive prosthesis control modalities that could improve functional patient outcomes. Proposed Solution Ultrasound-based sensing of muscle deformation, known as sonomyography, is an emerging sensing modality for upper-limb prosthesis control with the potential to significantly improve functionality. Sonomyography enables spatiotemporal characterization of both superficial and deep muscle activity, making it possible to distinguish the contributions of individual muscles during functional movements and derive a large set of independent prosthesis control signals. Using sonomyography to control a prosthesis has shown great promise in the research literature but has not yet been fully adapted for clinical use. This article describes the implementation of sonomyography for upper-limb prosthesis control, ongoing technological development, considerations for deploying this technology in clinical settings, and recommendations for future study. Clinical Relevance Sonomyography may soon become a clinically viable modality for upper-limb prosthesis control that could offer prosthetists an additional solution when selecting optimal treatment plans for their patients.

Recognition of Upper Limb Movements Based on Hybrid EEG and EMG Signals for Human-Robot Interaction

Upper limb amputation is a condition that severely limits the amputee’s movement. Patients who have lost the use of one or more of their upper extremities have difficulty performing activities of daily living. To help improve the control of upper limb prosthesis with pattern recognition, non-invasive approaches (EEG and EMG signals) is proposed in this paper and are integrated with machine learning techniques to recognize the upper-limb motions of subjects. EMG and EEG signals are combined, and five features are utilized to classify seven hand movements such as (wrist flexion (WF), outward part of the wrist (WE), hand open (HO), hand close (HC), pronation (PRO), supination (SUP), and rest (RST)). Experiments demonstrate that using mean absolute value (MAV), waveform length (WL), Wilson Amplitude (WAMP), Sine Slope Changes (SSC), and Cardinality features of the proposed algorithm achieves a classification accuracy of 89.6% when classifying seven distinct types of hand and wrist movement. Index Terms— Human Robot Interaction, Bio-signals Analysis, LDA classifier.

Predicting Wrist Joint Angles from the Kinematics of the Arm: Application to the Control of Upper Limb Prostheses.

Automation of wrist rotations in upper limb prostheses allows simplification of the human-machine interface, reducing the user's mental load and avoiding compensatory movements. This study explored the possibility of predicting wrist rotations in pick-and-place tasks based on kinematic information from the other arm joints. To do this, the position and orientation of the hand, forearm, arm, and back were recorded from five subjects during transport of a cylindrical and a spherical object between four different locations on a vertical shelf. The rotation angles in the arm joints were obtained from the records and used to train feed-forward neural networks (FFNNs) and time-delay neural networks (TDNNs) in order to predict wrist rotations (flexion/extension, abduction/adduction, and pronation/supination) based on the angles at the elbow and shoulder. Correlation coefficients between actual and predicted angles of 0.88 for the FFNN and 0.94 for the TDNN were obtained. These correlations improved when object information was added to the network or when it was trained separately for each object (0.94 for the FFNN, 0.96 for the TDNN). Similarly, it improved when the network was trained specifically for each subject. These results suggest that it would be feasible to reduce compensatory movements in prosthetic hands for specific tasks by using motorized wrists and automating their rotation based on kinematic information obtained with sensors appropriately positioned in the prosthesis and the subject's body.

Forearm sEMG data from young healthy humans during the execution of hand movements

This work provides a complete dataset containing surface electromyography (sEMG) signals acquired from the forearm with a sampling frequency of 1000 Hz. The dataset is named WyoFlex sEMG Hand Gesture and recorded the data of 28 participants between 18 and 37 years old without neuromuscular diseases or cardiovascular problems. The test protocol consisted of sEMG signals acquisition corresponding to ten wrist and grasping movements (extension, flexion, ulnar deviation, radial deviation, hook grip, power grip, spherical grip, precision grip, lateral grip, and pinch grip), considering three repetitions for each gesture. Also, the dataset contains general information such as anthropometric measures of the upper limb, gender, age, laterally of the person, and physical condition. Likewise, the implemented acquisition system consists of a portable armband with four sEMG channels distributed equidistantly for each forearm. The database could be used for the recognition of hand gestures, evaluation of the evolution of patients in rehabilitation processes, control of upper limb orthoses or prostheses, and biomechanical analysis of the forearm.

Delaying feedback during pre-device training facilitates the retention of novel myoelectric skills: a laboratory and home-based study

Objective. The objective of this study was to assess the impact of delayed feedback training on the retention of novel myoelectric skills, and to demonstrate the use of this training approach in the home environment. Approach. We trained limb-intact participants to use a motor learning-based upper-limb prosthesis control scheme called abstract decoding. A delayed feedback paradigm intended to prevent within-trial adaptation and to facilitate motor learning was used. We conducted two multi-day experiments. Experiment 1 was a laboratory-based study consisting of two groups trained over a 4 day period with concurrent or delayed feedback. An additional follow-up session took place after 18 days to assess the retention of motor skills. Experiment 2 was a home-based pilot study that took place over five consecutive days to investigate delayed feedback performance when using bespoke training structures. Main Results. Approximately 35 000 trials were collected across both experiments. Experiment 1 found that the retention of motor skills for the delayed feedback group was significantly better than that of their concurrent feedback counterparts. In addition, the delayed feedback group improved their retention of motor skills across days, whereas the concurrent feedback group did not. Experiment 2 demonstrated that by using a bespoke training protocol in an environment that is more conducive to learning, it is possible for participants to become highly accurate in the absence of feedback. Significance. These results show that with delayed feedback training, it is possible to retain novel myoelectric skills. Using abstract decoding participants can activate four distinct muscle patterns without using complex algorithms. The accuracy achieved in the pilot study supports the feasibility of motor learning-based upper-limb prosthesis control after home-based myoelectric training.

Bio-robotics research for non-invasive myoelectric neural interfaces for upper-limb prosthetic control: a 10-year perspective review.

A decade ago, a group of researchers from academia and industry identified a dichotomy between the industrial and academic state-of-the-art in upper-limb prosthesis control, a widely used bio-robotics application. They proposed that four key technical challenges, if addressed, could bridge this gap and translate academic research into clinically and commercially viable products. These challenges are unintuitive control schemes, lack of sensory feedback, poor robustness and single sensor modality. Here, we provide a perspective review on the research effort that occurred in the last decade, aiming at addressing these challenges. In addition, we discuss three research areas essential to the recent development in upper-limb prosthetic control research but were not envisioned in the review 10 years ago: deep learning methods, surface electromyogram decomposition and open-source databases. To conclude the review, we provide an outlook into the near future of the research and development in upper-limb prosthetic control and beyond.

Real-Time Wrist Motion Decoding with High Framerate Electrical Impedance Tomography (EIT).

Human wrist motion decoding with a biological-signal-based interface is a key technique in the upper-limb exoskeleton and prosthesis control. One critical issue in this field is achieving high recognition precision and fast time response while against external disturbances of sensor re-wearing. In this study, we proposed a high-framerate Electrical Impedance Tomography (EIT) system combined with an adaptive recognition algorithm for real-time wrist kinematics decoding. The high-framerate EIT system was developed by a parallel stimulation-measurement sequence, and the sampling rate was as high as 104 Hz. Compared to the most widely used myoelectric techniques, the EIT-based interface can provide extra deep muscular spatial information with similar surface electrodes. It greatly benefited the subsequent recognition algorithms, in which the key EIT regions indicating muscle morphology kept consistent after an arbitrary sensor re-donning. The designed adaptive algorithm achieved equally high performance with an automatic update of the classifier mean values with a fast self-operated calibration process. We validated the approach on 12 subjects with a 2-dimensional Fitts' law test. The wrist gestures and joint angles were mapped to the direction and speed of the cursor movement, respectively. The average throughputs (TPs) of Fitts' law tests were 1.0269 ± 0.0971 bits/s and 1.0095 ± 0.0931 bits/s without and with sensor re-donning, respectively, which were comparable to the TPs of sEMG-based studies. The results showed the promise of the EIT-based interface on real-time human motion intent recognition. Future endeavors are worth being paid in this direction for more complicated robotic tasks.

Composite Recurrent Convolutional Neural Networks Offer a Position-Aware Prosthesis Control Alternative While Balancing Predictive Accuracy with Training Burden.

To mitigate the "limb position effect" that hinders myoelectric upper limb prosthesis control, pattern recognition-based models must accurately predict user-intended movements across a multitude of limb positions. Such models can use electromyography (EMG) and inertial measurement units to capture necessary multi-position data. However, this data capture solution requires lengthy user-performed model training routines, with movements in many limb positions, plus retraining thereafter due to inherent signal variations over time. While a general-purpose control model (trained with a dataset that represents numerous device users) eliminates the user-training requirement altogether, it yields low movement predictive accuracy. Conversely, a user-specific control model (trained with a smaller dataset from an individual) yields high predictive accuracy, but requires retraining over time. This study capitalizes on the benefits offered by both such control options, and contributes an alternative control solution-a novel recurrent convolutional neural network (RCNN)-based Composite Model that combines the representation portion of a general-purpose model, with the decision portion of a user-specific model. The resulting Composite Model offers moderate movement predictive accuracy across various limb positions and a reduction in user training routine requirements, suggesting a new research direction to help mitigate the limb position effect along with model training burden.

An Exploration of the Optimal Feature-Classifier Combinations for Transradial Prosthesis Control.

Within state-of-the-art gesture-based upper-limb myoelectric prosthesis control, gesture recognition commonly relies on the classification of features extracted from electromyographic (EMG) data gathered from the amputee's residual forearm musculature. Despite best efforts in broadly maximizing gesture recognition accuracy, there does not yet exist a feature-classifier combination accepted as best-practice. In turn, this work hypothesizes that no single feature-classifier combination can consistently maximize accuracy across subjects, positing instead that control schemes should be personalized to the individual. To investigate this hypothesis, the study employed the 40-subject, 49-gesture Ninapro Database 2 (DB2) to compare the performance of 7 different historic, more recent, and state-of-the-art feature sets when classified by 5 machine learning algorithms commonly seen within EMG-based pattern recognition literature. The results demonstrate the ability of Linear Discriminant Analysis (LDA) to marginally exceed other more computationally intensive classifiers in terms of mean accuracy, while the feature set which maximized the highest proportion of individuals' accuracies was shown to vary with both classifier choice and gesture count.

Biosignal-based transferable attention Bi-ConvGRU deep network for hand-gesture recognition towards online upper-limb prosthesis control

Background and Objective: Upper-limb amputation can significantly affect a person’s capabilities with a dramatic impact on their quality of life. As a biological signal, surface electromyogram (sEMG) provides a non-invasive means to measure underlying muscle activation patterns, corresponding to specific hand gestures. This project aims to develop a real-time deep learning based recognition model to automatically and reliably recognise these complex signals of a wide range of daily hand gestures from amputees and non-amputees.Methods: This paper proposes an attention bidirectional Convolutional Gated Recurrent Unit (Bi-ConvGRU) deep neural network for hand-gesture recognition. By training on sEMG data from both amputees and non-amputees, the model can learn to recognise a group of fine-grained hand movements. This is a significantly more challenging and underexplored area, compared to existing studies on coarse-control in lower limbs. One dimensional CNNs are initially used to extract intra-channel features. The novel use of a bidirectional sequential GRU (Bi-GRU) deep neural network allows the exploration of correlation of muscle activation among multi-channel sEMG signals from both prior and posterior time sequences. Importantly, the attention mechanism is employed following Bi-GRU layers. This enables the model to learn vital parts and feature weights, increasing robustness to bio-data noise and irregularity. Finally, we introduce the first of its kind transfer learning, demonstrating that a baseline model pre-trained with non-amputee data can be effectively refined with amputee data to build a personalised model for amputees.Results: The attention Bi-ConvGRU was evaluated on the benchmark database Ninapro, and achieved an average accuracy of 88.7%, outperforming the state-of-the-art on 18 gesture recognition by 6.7%.Conclusions: To our knowledge, the developed end-to-end deep learning model is the first of its kind that enables reliable predictive decision making in short time windows (160ms). This reduced latency limits physiological awareness, enabling the potential for real-time, online and thus more intuitive bio-control of prosthetic devices for amputees.

Sensory substitution of elbow proprioception to improve myoelectric control of upper limb prosthesis: experiment on healthy subjects and amputees

BackgroundCurrent myoelectric prostheses lack proprioceptive information and rely on vision for their control. Sensory substitution is increasingly developed with non-invasive vibrotactile or electrotactile feedback, but most systems are designed for grasping or object discriminations, and few were tested for online control in amputees. The objective of this work was evaluate the effect of a novel vibrotactile feedback on the accuracy of myoelectric control of a virtual elbow by healthy subjects and participants with an upper-limb amputation at humeral level.MethodsSixteen, healthy participants and 7 transhumeral amputees performed myoelectric control of a virtual arm under different feedback conditions: vision alone (VIS), vibration alone (VIB), vision plus vibration (VIS + VIB), or no feedback at all (NO). Reach accuracy was evaluated by angular errors during discrete as well as back and forth movements. Healthy participants’ workloads were assessed with the NASA-TLX questionnaire, and feedback conditions were ranked according to preference at the end of the experiment.ResultsReach errors were higher in NO than in VIB, indicating that our vibrotactile feedback improved performance as compared to no feedback. Conditions VIS and VIS+VIB display similar levels of performance and produced lower errors than in VIB. Vision remains therefore critical to maintain good performance, which is not ameliorated nor deteriorated by the addition of vibrotactile feedback. The workload associated with VIB was higher than for VIS and VIS+VIB, which did not differ from each other. 62.5% of healthy subjects preferred the VIS+VIB condition, and ranked VIS and VIB second and third, respectively.ConclusionOur novel vibrotactile feedback improved myoelectric control of a virtual elbow as compared to no feedback. Although vision remained critical, the addition of vibrotactile feedback did not improve nor deteriorate the control and was preferred by participants. Longer training should improve performances with VIB alone and reduce the need of vision for close-loop prosthesis control.

Classification Performance and Feature Space Characteristics in Individuals With Upper Limb Loss Using Sonomyography

Objective: Sonomyography, or ultrasound-based sensing of muscle deformation, is an emerging modality for upper limb prosthesis control. Although prior studies have shown that individuals with upper limb loss can achieve successful motion classification with sonomyography, it is important to better understand the time-course over which proficiency develops. In this study, we characterized user performance during their initial and subsequent exposures to sonomyography. Method: Ultrasound images corresponding to a series of hand gestures were collected from individuals with transradial limb loss under three scenarios: during their initial exposure to sonomyography (Experiment 1), during a subsequent exposure to sonomyography where they were provided biofeedback as part of a training protocol (Experiment 2), and during testing sessions held on different days (Experiment 3). User performance was characterized by offline classification accuracy, as well as metrics describing the consistency and separability of the sonomyography signal patterns in feature space. Results: Classification accuracy was high during initial exposure to sonomyography (96.2 ± 5.9%) and did not systematically change with the provision of biofeedback or on different days. Despite this stable classification performance, some of the feature space metrics changed. Conclusions: User performance was strong upon their initial exposure to sonomyography and did not improve with subsequent exposure. Clinical Impact: Prosthetists may be able to quickly assess if a patient will be successful with sonomyography without submitting them to an extensive training protocol, leading to earlier socket fabrication and delivery.

Improving Automatic Control of Upper-Limb Prosthesis Wrists Using Gaze-Centered Eye Tracking and Deep Learning.

Many upper-limb prostheses lack proper wrist rotation functionality, leading to users performing poor compensatory strategies, leading to overuse or abandonment. In this study, we investigate the validity of creating and implementing a data-driven predictive control strategy in object grasping tasks performed in virtual reality. We propose the idea of using gaze-centered vision to predict the wrist rotations of a user and implement a user study to investigate the impact of using this predictive control. We demonstrate that using this vision-based predictive system leads to a decrease in compensatory movement in the shoulder, as well as task completion time. We discuss the cases in which the virtual prosthesis with the predictive model implemented did and did not make a physical improvement in various arm movements. We also discuss the cognitive value in implementing such predictive control strategies into prosthetic controllers. We find that gaze-centered vision provides information about the intent of the user when performing object reaching and that the performance of prosthetic hands improves greatly when wrist prediction is implemented. Lastly, we address the limitations of this study in the context of both the study itself as well as any future physical implementations.

A biomechatronics-based EPP topology for upper-limb prosthesis control: Modeling & benchtop prototype

This article reintroduces the concept of Extended Physiological Proprioception (EPP) for below the elbow amputations, eliminating the use of a harness, cineplasty, or Bowden cables, and proposing instead a contemporary scheme inspired by teleoperation theory and its applications. A biomechatronics-based master/slave topology is developed to provide EPP-equivalent control using implanted micro servo actuators directly attached to amputee’s residual muscles and wirelessly communicating with the prosthesis. A Position-Force (P-F) control architecture guarantees the dynamic coupling of the actuating muscles with the prosthetic limb, enabling amputee kinesthetic sense. Simulations and Hardware in the Loop (HIL) preliminary experiments with an able-bodied individual are conducted and provide an evaluation of the proposed teleoperation scheme in terms of control transparency and time delay with promising results. An initial feasibility analysis on implants size and power demands based on specific performance criteria, is encouraging, while the materialization potential of the topology is assessed.

Impact of Shared Control Modalities on Performance and Usability of Semi-autonomous Prostheses.

Semi-autonomous (SA) control of upper-limb prostheses can improve the performance and decrease the cognitive burden of a user. In this approach, a prosthesis is equipped with additional sensors (e.g., computer vision) that provide contextual information and enable the system to accomplish some tasks automatically. Autonomous control is fused with a volitional input of a user to compute the commands that are sent to the prosthesis. Although several promising prototypes demonstrating the potential of this approach have been presented, methods to integrate the two control streams (i.e., autonomous and volitional) have not been systematically investigated. In the present study, we implemented three shared control modalities (i.e., sequential, simultaneous, and continuous) and compared their performance, as well as the cognitive and physical burdens imposed on the user. In the sequential approach, the volitional input disabled the autonomous control. In the simultaneous approach, the volitional input to a specific degree of freedom (DoF) activated autonomous control of other DoFs, whereas in the continuous approach, autonomous control was always active except for the DoFs controlled by the user. The experiment was conducted in ten able-bodied subjects, and these subjects used an SA prosthesis to perform reach-and-grasp tasks while reacting to audio cues (dual tasking). The results demonstrated that, compared to the manual baseline (volitional control only), all three SA modalities accomplished the task in a shorter time and resulted in less volitional control input. The simultaneous SA modality performed worse than the sequential and continuous SA approaches. When systematic errors were introduced in the autonomous controller to generate a mismatch between the goals of the user and controller, the performance of SA modalities substantially decreased, even below the manual baseline. The sequential SA scheme was the least impacted one in terms of errors. The present study demonstrates that a specific approach for integrating volitional and autonomous control is indeed an important factor that significantly affects the performance and physical and cognitive load, and therefore these should be considered when designing SA prostheses.

The effects of forearm movements on human gait during walking with various self-selected speeds

The forearms significantly contribute to the upper extremity movements and, consequently, whole-body responses during locomotion. The purpose of this study is to provide a more in-depth understanding of the mechanism controlling forearm movements during walking by comprehensively investigating the effects of the forearms on the lower and upper limb movements. Such an understanding can provide critical information for the design and control of robotic upper-limb prostheses. Twelve healthy young participants were recruited to compare their gait during (1) natural walking, (2) walking while wearing a pair of artificial passive forearms and having their actual forearms restrained by orthopedic braces, and (3) walking with only having their forearms restrained by the braces (i.e., no artificial forearms). While the passive forearms in condition 2 were to determine if the forearm movements were passively or actively controlled, condition 3 was to account for the effects of restraining the forearms in condition 2. The participants' lower-limb joint angles and spatiotemporal parameters remained unchanged across the three conditions while walking at their normal and fast self-selected gait speeds. However, significant decreases were observed in the shoulder and trunk angles, the interlimb coordination, and the shoulder-trunk correlations when walking with the artificial forearms. These observations were in tandem with the increased muscle activity of the biceps, trapeziuses, and posterior deltoids, which controlled the shoulder motion and trunk rotation during walking with the artificial forearms across both normal and fast self-selected speeds. Although not significant, the metabolic energy analysis of five participants revealed an increase during walking with artificial forearms. The results support the idea that the body actively controls the forearm movements through the shoulder and trunk rotations to mitigate the undesired disturbances induced by the passive forearm movements during locomotion.