Dynamic Manipulation Skill Learning for Tactile Myoelectric Prosthetic Hands in Tool Handling

An approach for measuring local hand motions is explored by using induction coils as sensors for a data glove. The primary goal is to deduce finger joint angles in real time and recognize the corresponding hand gestures. The task is accomplished by relating the finger bending with the induced voltage measurements. The induced voltages are subsequently processed to extract the information of the finger angles. Induction coil properties are explored to realize their significance in finger bending measurement. The principle of electromagnetic induction is used to accomplish near field wireless transfer of energy between the coils. Different human hand motion constraints are also exploited to increase the effectiveness of the proposed approach. The technique is realized and tested by prototyping an embedded system that ensures easy comprehension of its working. Experimental results show the efficacy of the proposed design.

Surface electromyography (EMG) provides a non-invasive human-machine interaction interface that can promote the coherence of human-machine interaction operations. Decomposing surface electromyographic signals into hand joint angles in real time can be applied to prosthetic control, rehabilitation engineering and other fields. However, existing methods of using surface electromyography signals suffer from high end-to-end latency, high memory consumption, and high power consumption, which hinder their dissemination in clinical edge devices and public wearable devices. After a thorough analysis of the state-of-the-art surface EMG based architecture, we observed that the time complexity of the attention mechanism in using Transformer for continuous motion estimation results in longer inference time. The attention mechanism requires a large number of parameters to achieve good results, leading to higher model power consumption. This will reduce its performance in continuous motion statistics. To tackle the existing Surface EMGs challenges, PET, a lightweight parallel efficient transformer model, is proposed. We elaborately develop a thorough bottom-up architecture of PET, from model structure and power mechanism. The PET’s parallel and lightweight architecture can decompose the surface electromyography in real time and output the hand joint angles while compacting memory consumption and affordable power expenditure without sacrificing the accuracy of extracting motion statistics. Compared to the state-of-the-art surface EMG architectures, the experimental results demonstrate that PET outperforms SVR, TCN, LSTM, GRU, LE-LSTM, LE-ConvMN, Transformer, Bert, MAFN and Conformer by Correlation Coefficient, RMSE, NRMSE, AME, End-to-end latency in variety of challenging Surface EMG programs, including Ninapro DB2, Ninapro DB7, FMHD, and SEEDS. The PET correlation coefficient for all 60 subjects in the Ninapro dataset was 0.85 ± 0.01, the root mean square error was 7.26 ± 0.32, the normalized RMSE was 0.11 ± 0.01, and the AME was 6.183. The PET correlation coefficient in the test of the Finger Movement HD was 0.81 ± 0.01, and the root mean square error was 10.15 ± 0.52 with a normalized RMSE of 0.11 ± 0.01. The PET correlation coefficient in the test of the SEEDS was 0.82 ± 0.01, and the root mean square error was 10.09 ± 0.01 with a normalized RMSE of 0.10 ± 0.01. Our method achieved state-of-the-art performance in the above tests. The results of the above tests were based on the same subjects.

Here we describe and validate a fabric sensing glove for hand finger movement monitoring. After a quick calibration procedure, and by suitably processing of the outputs of the glove, it is possible to estimate hand joint angles in real time. Moreover, we tested the fMRI compatibility of the glove and ran a pilot fMRI experiment on the neural correlates of handshaking during human-to-human and human-to-robot interactions. Here we describe how the glove can be used to monitor correct task execution and to improve modeling of the expected hemodynamic responses during fMRI experimental paradigms.

We present a system that enables a humanoid robot to imitate complex whole-body motions of humans in real time. For recording the human motions, any sensor system capable of inferring the joint angle trajectories can be used. In our work, we capture the human data with an Xsens MVN motion capture system consisting of inertial sensors attached to the body. Our framework converts the human joint angles to the robot's joint angles in real time. Here, we use a mapping between the human's and the robot's joints to ensure feasibility of the motion. The focus of our system lies in ensuring static stability when the motions are executed which is a challenging task, depending on the complexity of the movements. To avoid falls of the robot that might occur when using direct imitation of the joint angle trajectories due to the different weight distribution, we developed an approach that actively balances the center of mass over the support polygon of the robot's feet. At every point in time, our approach ensures that the robot is in a statically stable configuration, i.e., that the ground projection of the center of mass lies within the convex hull of the foot contact points. To achieve this, we apply inverse kinematics given valid foot positions that satisfy the stability criterion and generate the corresponding leg joint angles. In more detail, our system first finds valid positions for the robot's feet by determining a target plane and its orientation, so that the feet can be placed planar and the robot's center of mass is over the support polygon. The new positions of the feet are chosen as the projection on the target plane. Afterwards, the corresponding leg joint angles are calculated via inverse kinematics. To determine whether the configuration is in the double support modus, and if not, which foot is the stance foot, we evaluate the position of the center of mass relative to the feet. As can be seen in the experiments with a Nao humanoid, our approach leads to a highly stable imitation of challenging human movements (see also Fig. 1). In contrast to recent approaches that capture human data using a Kinect-like sensor and only imitate arm movements while keeping the body static, our system can deal with complex, whole-body motions. Note that our approach does not require a prior learning phase but computes stable configurations online and almost in real time as can be seen in the accompanying video. We are currently working on imitating motions to learn complex navigation actions such as climbing up staircases or walking down ramps. Our system can also be used for tele-operated tasks that include whole-body movements where stability needs to be guaranteed in order to successfully fulfill the mission.

In this study, an online predictive control strategy based on the backpropagation neural network (BPNN) is proposed for planetary surface exploration robots with a high manipulator-to-platform mass ratio, with the objective of achieving dynamic stability control. First, a dynamic model of the planetary surface exploration robot is established, addressing the strong coupling effects between the manipulator and the mobile platform. Second, an online adaptive stability strategy based on BPNN is proposed. An adaptive sliding mode control method is applied to adjust the joint angles in real time, dynamically regulating the zero moment point (ZMP) to remain within the support polygon. Finally, simulation and experiment are conducted to validate the proposed method. The experimental results show that the system maintains minimal deviations at low speeds, with the X direction deviation ranging from 7 to 11 mm and the Y direction deviation within ±2 mm. At medium and high speeds, the system observes slight increases in ZMP deviations but effectively maintains dynamic balance. The experimental results closely align with the simulations, confirming the accuracy and reliability of the proposed method. Coordination control between the manipulator and the mobile platform significantly enhances stability and adaptability in rugged terrain, providing crucial technological support for future planetary exploration missions.

When patients wear an exoskeleton for rehabilitation to walk and encounter obstacles or stairs, they often need manual operation by themselves or other people to switch the gait of the exoskeleton robot, which leads to a poor human-machine synergy of common exoskeleton robots. In this paper, the surface electromyography (sEMG) signals of rectus femoris (RF), semimembranosus (SM) and sartorius (SR) muscles of the subjects were collected and the time domain features - root mean square (RMS) of the sEMG were extracted. A back propagation (BP) neural network was used to predict knee joint angles in real time, when the exoskeleton wearer’s knee angle was greater than 70°, the exoskeleton performed gait switching across the obstacle in real time. Experiments show that when encountering obstacles, the method can realize the real-time gait switching of the exoskeleton across obstacles by the wearer using sEMG signals, and the method has good real-time characteristic and accuracy.

Control of planar underactuated manipulators (PUM) with unknown parameter perturbations and external disturbances is still a challenging task due to their complex and peculiar characteristics. The research on it is significant in the view of wide applications in practice. In this paper, taking an uncertain 3-degree of freedom PUM with a free first joint as a benchmark example, we discuss its position control issue. Specifically, an integrated control method is developed, including the nonlinear model prediction control (NMPC) based on an improved particle swarm optimisation (PSO) algorithm and the discrete-time fast terminal sliding mode (FTSM) control. The PSO-based NMPC is proposed for planning discrete trajectories of the active joint angles in real time, along which the manipulator end-point can reach the desired position. Then the discrete-time FTSM controllers are designed to keep the active joints tracking the discrete trajectories, where the uncertainties related to the active links/joints are estimated by time delay estimation method. Besides, the influence of the uncertainties related to the free link/joint on the system can be made up by the NMPC in real time. It is confirmed via simulations that the above method can achieve the accurate positioning of such an uncertain manipulator.

Joint angular sensors are widely used in industry, from highly effective robots in product lines and heavy construction machines to different home appliances. Likewise in the medical field, parameters of human motion, especially the orientations of lower limb segments, are crucial in clinical evaluations and therapeutic treatments in the orthopedic and rehabilitation fields. This paper presents the design and implementation of an embedded system-on-chip for measuring human-body joint angles using MEMS accelerometers and PSoC mixed-signal circuits. It focuses on two algorithms, one called CMR and another DCMR, and utilizes the property of rigid body kinematics to explain their advantages and weaknesses. Unlike CMR algorithm, DCMR algorithm has no requirement on placing the sensors close to the joint center. This provides greater flexibility for the sensor installation. An auto-adjustment procedure is described and both algorithms are characterized on a rigid body robot arm model and compared with a reference system. Experimental results showed that the algorithms were able to measure joint angles in real time, and their accuracy was high enough to be used in ambulatory human-body joint angle measurements and feedback control systems for gait assistance.

Estimation and control of zero-moment point (ZMP) is a widely used concept for planning the locomotion of bipedal robots and is commonly measured using integrated joint angle encoders and foot force sensors. Contemporary methods for ZMP measurement involve built-in contact sensors such as joint encoders or instrumented foot force sensors. This paper presents a novel approach for computing ZMP for a humanoid robot using inertial sensor-based wireless foot sensor modules (WFSMs). The developed WFSMs, strapped at different limb segments of a bipedal robot, measure lower limb joint angles in real time. The joint angle trajectories, further transformed into Cartesian position coordinates, are used for estimating the ZMP positions of humanoid robots using the planar biped model. The whole framework is presented through experimental studies for different real-life walking scenarios. Since the modules work based on the limb motion and inclination, any ground unevenness would be automatically reflected in the module output. Hence, this measurement process can be a convenient method for applications requiring humanoid control on uneven surfaces/outdoor terrains. To compare the performance of the proposed model, ZMP is simultaneously measured from inbuilt foot force sensors and joint encoders of the robot. Statistical tests exhibit a high linear correlation between the proposed method with integrated encoders and foot force sensors (Pearson’s coefficient, [Formula: see text]). Results indicate that ZMP estimated by WFSM is a viable method to monitor the dynamic gait balance of a humanoid robot and has potential application in outdoor and uneven terrains.

Aquatic therapy exercises utilize buoyancy and hydrodynamic resistance forces, providing a safe and effective exercise environment for participants, particularly elderly individuals, those with obesity, and patients undergoing rehabilitation. However, conventional aquatic exercise tools, such as buoyant aids and rigid fins, offer limited capacity for providing active assistance or resistance. To address these challenges, this study proposes a wearable robotic device for aquatic lower-limb exercises, equipped with a bidirectional thruster. The robot measures knee joint angles in real time and controls the thrust moment to enable various exercise modes, including passive exercises such as stretching and continuous-passive-motion (CPM), as well as active exercises with assistance and resistance modes. To evaluate the performance of the proposed system, a dynamic model was derived, and its parameters and relationships were experimentally identified. A prototype was subsequently fabricated and attached to a motor-driven lower-leg mock-up. Control experiments were then conducted underwater across the proposed exercise modes to assess the feasibility of the proposed exercise mode. These results verify the feasibility of closed-loop, thruster-based assistance and resistance for lower-limb aquatic exercise, laying the groundwork for forthcoming human-subject trials aimed at elderly and post-operative populations.

Continuous joint motion estimation plays an important part in accomplishing more compliant and safer human-machine interaction (HMI). Surface electromyogram (sEMG) signals, which contain abundant motion information, can be used as a source for continuous joint motion estimation. In this paper, a knee joint angle prediction system based on muscle synergy theory and generalized regression neural network (GRNN) was proposed. The wavelet transform threshold method was used for sEMG signals and angle trajectories denoising. The time-domain features wave-length extracted from four-channel sEMG signals were decomposed into a synergy matrix and an activation coefficient matrix by using nonnegative matrix factorization based on muscle synergy theory. A GRNN based on golden-section search was employed to build the activation model mapping from the activation coefficients to the knee joint angles, so as to realize the continuous knee joint angle estimation. The experimental results show that the average coefficient of determination is 0.933. In addition, a user graphic interface based on the Java platform was designed to display the dynamic sEMG data and predicted knee joint angles in real time.

Industrial workers often perform awkward gaits such as squatting, kneeling, etc. for a prolonged time when conducting skilled tasks. We present a real-time wearable sensing and exoskeleton control design to provide assistance for the industrial workers under awkward gaits. A wearable sensor-based gait activity detection and pose estimation scheme is designed to predict the human motion and lower-limb joint angles in real time during a sequence of walking, standing, squatting, and kneeling gaits. Wearable bilateral exoskeletons provide assistive torques at various gaits under a multi-level controller. Human-subject experiments are presented to demonstrate the gait detection and exoskeleton control performance. The results show that the overall accuracy of human gait recognition is up to 95% and the average detection latency is around 50 ms. The exoskeleton control strategy reduces muscle activation in knee extension/flexion up to 25% during various stationary gaits and posture transitions.

Joint angles are commonly measured in physical rehabilitation to evaluate joint function. Evidences showed that wearable inertial sensors can accurately quantify human motion information, however, the most advanced and accurate methodologies require the execution of complex calibration movements which are unsuitable to inexpert users and inadequate for a home context. This way, four different joint angles estimation methods requiring no calibration movement were developed in order to track the main human body joint angles in real time. IMUs mounted in bracelets were used to restrict sensor positioning on the limbs. For six different exercises, the estimated absolute and relative joint angles were evaluated against the marker-based video tracking software Kinovea ground-truth. Correlation analysis between estimated and ground-truth joint angles indicated a very strong and statistically significant correlation. The average error in estimated joint angles is below 5 degrees for all four methods employed, which may be an acceptable result for the rehabilitation at home scenario.

The inverse kinematics of the manipulator is the key and difficult point in the real-time control of the manipulator. Based on Jacobi matrix generalized inverse method, this article obtains the the inverse kinematics of a 6-DOF manipulator which is used for underwater propeller cleaning. Then, based on linear and circular target trajectory, through MATLAB simulation, we obtain all joint angles in real time, the trajectory for inverse solution and the error between simulation results and the target trajectory. The error between the simulation results and the target trajectory is less than $1.5\times 10^{-3}m$, which verifies that the method can obtain the real-time inverse kinematics solution of the manipulator.

With the development of robotics, lower limb exoskeleton robots have broad application prospects in helping patients with rehabilitation. However, the current operating modes of the commercial lower limb exoskeleton robots are mostly driving the lower limb dysfunction patients for rehabilitation walking training. In this case, the patients are only passively involved, lacking voluntary participation. In this paper, by collecting the sEMG signal of the lower extremity of the exoskeleton wearer and applying deep backpropagation (BP) neural network to predict the knee and hip joint angles in real time based on the root mean square (RMS) feature of the sEMG signal, the motion intention of the wearer can be detected to control the exoskeleton. The experimental results show that the method has satisfactory real-time performance and accuracy, and can effectively improve the voluntary participation of the wearer and enhance the synergy performance of the humanexoskeleton system.