Human-machine Interactive Control Research Articles

An Image-Based Interactive Training Method of an Upper Limb Rehabilitation Robot

Aimed at the problem of human–machine interaction between patients and robots in the process of using rehabilitation robots for rehabilitation training, this paper proposes a human–machine interactive control method based on an independently developed upper limb rehabilitation robot. In this method, the camera is used as a sensor, the human skeleton model is used to analyse the moving image, and the key points of the human body are extracted. Then, the three-dimensional coordinates of the key points of the human arm are extracted by depth estimation and spatial geometry, and then the real-time motion data are obtained, and the control instructions of the robot are generated from it to realise the real-time interactive control of the robot. This method can not only improve the adaptability of the system to individual patient differences, but also improve the robustness of the system, which is less affected by environmental changes. The experimental results show that this method can realise real-time control of the rehabilitation robot, and that the robot assists the patient to complete the action with high accuracy. The results show that this control method is effective and can be applied to the fields of robot control and robot-assisted rehabilitation training.

SEMG data driven-based anti-disturbance control enables adaptive interaction of lower limb rehabilitation exoskeleton

The efficient human–machine interaction control is the core of human–machine integration, which can improve the tracking accuracy of the lower limb rehabilitation exoskeleton system and the compliance of human–machine cooperative movement. However, the non-ideal factors such as mechanical friction, model uncertainties, iteration errors and external interference in the training process undoubtedly bring potential dangers to the safety of rehabilitation training. Consequently, to escape unnecessary damage caused by external disturbances (especially the unbounded disturbances) during rehabilitation training, a noise-suppressing zeroing neural network human–machine interaction controller is developed in this work, which is based on the constructed human–machine coupling dynamic model and the active motion intention (active torque) of the subject identified by exploiting the deep convolutional neural network. The simulation experiments and statistical analyses verify that the present noise-suppressing zeroing neural network controller can be applied to monitor the lower limb rehabilitation exoskeleton for assisting the subjects in various task with the external disturbances,and the average root mean square error of the hip, knee and ankle joints’ angles are 0.0015rad, 0.0051rad and 0.0056rad, respectively. Furthermore, a novel model predictive control is developed and analyzed based on noise-suppressing zeroing neural network controller, which can effectively constrain the angle and angular velocity of the lower limb rehabilitation exoskeleton. The root mean square errors of hip, knee and ankle joint angle are 0.0032rad, 0.0078rad, and 0.0085rad, respectively. Finally, the platform experiment verifies that the proposed controllers can be utilized to control the lower limb rehabilitation exoskeleton to assist the subjects in rehabilitation training.

Human–machine interaction controller of upper limb based on iterative learning method with zeroing neural algorithm and disturbance observer

In this paper, an iterative learning controller with zeroing neural algorithm and disturbance observer is proposed to solve the conflicting of human–machine interaction and disturbances in system. This paper presents a theoretical framework, which is able to process rigorous stability analysis of human–machine interaction control. The iterative learning controller is suitable for repetitive upper limb rehabilitation training. Combining tracking error dependent weight vector with zeroing neural algorithm is aimed to reduce the conflicts in various operations of motions between upper limb and machine. The disturbance observer is used to deal with eliminating the disturbance. In addition, simulations of the controller can effectively and safety assist upper limb movement in different training stages.

Research on optimization of human-machine interaction control strategy for exoskeleton based on impedance control

Research on optimization of human-machine interaction control strategy for exoskeleton based on impedance control

Fuzzy model reference learning control in surgical robots

This paper proposes to apply the fuzzy model reference learning control method based on the guide control to the collaborative human-machine dragging process of the surgical robot. Variable damping coefficient adjustment parameter rules for fuzzy conductance controllers are trained by an offline learning mechanism in order to achieve the best human-machine interaction control effect. The surgical risk caused by the doctor's misoperation is lowered. Through the simulation experiment results, the speed curve tracking error is reduced by 70%. Trajectory curve tracking error is reduced by 57%. The both results prove that the strategy can reduce the error of the surgical robot and ensure the safety of robotic surgical operation.

A noise-suppressing neural network approach for upper limb human-machine interactive control based on sEMG signals.

The use of upper limb rehabilitation robots to assist the affected limbs for active rehabilitation training is an inevitable trend in the field of rehabilitation medicine. In particular, the active motion intention-based control of the upper limb rehabilitation robots to assist subjects in rehabilitation training is a hot research topic in human-computer interaction control. Therefore, improving the accuracy of active motion intention recognition is the premise of the human-machine interaction controller design. Furthermore, there are external disturbances (bounded/unbounded disturbances) during rehabilitation training, which seriously threaten the safety of subjects. Thereby, eliminating external disturbances (especially unbounded disturbances) is the difficulty and key to the human-machine interaction control of the upper limb rehabilitation robots. In response to these problems, based on the surface electromyogram signal of the human upper limb, this paper proposes a fuzzy neural network active motion intention recognition method to explore the internal connection between the surface electromyogram signal of the human upper limb and active motion intention, and improve the real-time and accuracy of recognition. Based on this, two types of human-machine interaction controllers, which can be called as zeroing neural network controller and noise-suppressing zeroing neural network controller are designed to establish a safe and comfortable training environment to avoid secondary damage to the affected limb. Numerical experiments verify the feasibility and effectiveness of the proposed theories and methods.

Porous-Structure-Promoted Tribo-Induced High-Performance Self-Powered Tactile Sensor toward Remote Human-Machine Interaction.

Self-powered tactile sensor with versatile functions plays a significant role in the development of an intelligent human-machine interaction (HMI) system. Herein, a hybrid self-powered porous-structured tactile sensor (SPTS) is proposed by monolithically integrating a porous triboelectrification-induced electroluminescence (TIEL) component and a single-electrode triboelectric nanogenerator with the high charge generation in the bulk volume. At a low pressure of 10kPa, TIEL intensity can be significantly improved by three times, which is superior to that in previous reports, with enhanced triboelectricity. Based on the enhancement brought by the porous structure and optimized parameters, the SPTS achieves significant sensing performance in both optical and electrical modes. To demonstrate the potential of practical applications, a programmable optical and electrical dual-mode HMI system is established based on SPTS to remotely control an intelligent vehicle and operate a computer game through identifying finger touch trajectories. This work not only contributes a new economical-effective methodology toward a high-performance tribo-induced self-powered tactile sensor but also facilitates the remote control of HMI with dual-mode functionality, which has broad potential applications in the fields of intelligent robots, augmented reality, flexible wearable electronics, and smart home.

Improving the Robustness of Human-Machine Interactive Control for Myoelectric Prosthetic Hand During Arm Position Changing.

Robust classification of natural hand grasp type based on electromyography (EMG) still has some shortcomings in the practical prosthetic hand control, owing to the influence of dynamic arm position changing during hand actions. This study provided a framework for robust hand grasp type classification during dynamic arm position changes, improving both the “hardware” and “algorithm” components. In the hardware aspect, co-located synchronous EMG and force myography (FMG) signals are adopted as the multi-modal strategy. In the algorithm aspect, a sequential decision algorithm is proposed by combining the RNN-based deep learning model with a knowledge-based post-processing model. Experimental results showed that the classification accuracy of multi-modal EMG-FMG signals was increased by more than 10% compared with the EMG-only signal. Moreover, the classification accuracy of the proposed sequential decision algorithm improved the accuracy by more than 4% compared with other baseline models when using both EMG and FMG signals.

Decoding finger movement patterns from microscopic neural drive information based on deep learning

Recent development of surface electromyogram (sEMG) decomposition technique provides a good basis of decoding movements from individual motor unit (MU) activities that directly representing microscopic neural drives. How to interpret the function and contribution of each decomposed MU to macroscopic movements remains unclear. The objective of this study is to decode finger movement patterns by establishing a relationship between individual MU activities and movements. In this study, high-density sEMG (HD-sEMG) data were recorded by a 16 × 8 electrode array from finger extensor muscles of 10 subjects performing 10 finger movement patterns. The progressive FastICA peel-off algorithm was first applied to decompose the HD-sEMG data to obtain microscopic neural drives in terms of MU firing sequences and their corresponding action potential waveforms. Then, convolutional neural network was used for classification of the decomposed MUs by characterizing their spatial waveforms spanned over all channels of the array. On this basis, a fuzzy weighted decision strategy was designed to give a final decision of movement pattern recognition, where function of an individual MU was measured in the form of contributing into all movement patterns with different weights to solve the issue of MUs shared among multiple patterns due to muscle co-activation. The proposed method yielded an average accuracy approximating to 100%, and it outperformed other common MU-based methods or conventional myoelectric classification methods using macroscopic sEMG features (p < 0.05). The proposed method has a wide application prospect in the field of human-machine interaction and precise motor control.

Musculoskeletal Joint Angle Estimation Based on Isokinetic Motor Coordination

Motion estimation plays an important role in motor coordination exploration and optimal control of human-machine interaction. Muscular strength training, such as isokinetic exercise, aims to improve one’s muscular strength and endurance with adjustable resistance at a constant speed. To explore motion estimation and muscle coordination, this paper proposes a sEMG based musculoskeletal model to estimate joint angle trajectory of elbow isokinetic exercise to build the relationship between surface-Electromyography (sEMG) and its related joint movement. A set of motion protocol is designed to evaluate the model, including isokinetic movements under 100% and 50% joint torque levels (TL), and isotonic exercise. sEMG sensors and isokinetic dynamometer are employed in which muscle activations are calculated from sEMG signals as the input of musculoskeletal model. The experiment of three motion protocols is conducted by seven healthy subjects. The synergy patterns indicate the muscle coordination consistency with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{R}^{2} {&gt;}$ </tex-math></inline-formula> 0.92. The estimated joint angle trajectory confirms that the musculoskeletal model predicts the isokinetic movement and isotonic movement accurately with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text{R}^{2} {&gt;}$ </tex-math></inline-formula> 0.90 and the normalized root mean squared deviation NRMSD < 0.18. The research outcome in different levels of muscle forces paves the way for tailoring isokinetic exercise in clinical therapy such as stroke rehabilitation.

The Role of Surface Electromyography in Data Fusion with Inertial Sensors to Enhance Locomotion Recognition and Prediction.

Locomotion recognition and prediction is essential for real-time human–machine interactive control. The integration of electromyography (EMG) with mechanical sensors could improve the performance of locomotion recognition. However, the potential of EMG in motion prediction is rarely discussed. This paper firstly investigated the effect of surface EMG on the prediction of locomotion while integrated with inertial data. We collected EMG signals of lower limb muscle groups and linear acceleration data of lower limb segments from ten healthy participants in seven locomotion activities. Classification models were built based on four machine learning methods—support vector machine (SVM), k-nearest neighbor (KNN), artificial neural network (ANN), and linear discriminant analysis (LDA)—where a major vote strategy and a content constraint rule were utilized for improving the online performance of the classification decision. We compared four classifiers and further investigated the effect of data fusion on the online locomotion classification. The results showed that the SVM model with a sliding window size of 80 ms achieved the best recognition performance. The fusion of EMG signals does not only improve the recognition accuracy of steady-state locomotion activity from 90% (using acceleration data only) to 98% (using data fusion) but also enables the prediction of the next steady locomotion (∼370 ms). The study demonstrates that the employment of EMG in locomotion recognition could enhance online prediction performance.

Human–machine interaction control for stochastic cell manipulation systems

While many physical robot systems are able to work cooperatively with humans to perform numerous tasks, existing optical cell manipulation systems can only perform either manual or automated tasks at a time. As optical cell manipulation is essentially different from robotic manipulation in the physical world due to its stochastic nature, existing human–robot interaction control techniques cannot thus be directly reproduced into optical micro-manipulation of cells. This paper proposes a human–machine interaction control approach for stochastic optical cell manipulation systems. A theoretical foundation is then developed, and thus enabling human to interpose or interact, when applicable or needed, with an automated cell manipulation system without interrupting the manipulation process. Therefore, it offers an opportunity for human to get involved with an automated cell manipulation system so as to deal with unexpected events or complex tasks, rather than to stop or restart the whole manipulation process. Both the capacity of an automated control system – which provides precise and productive cell manipulation, and human ability – in terms of decision making, if needed, are then consolidated into a single optical cell manipulation system. Rigorous mathematical formulation is developed with the consideration of the stochastic perturbations, and the stability of the human–machine interaction control system is analyzed from a stochastic perspective. Experiments are then performed to illustrate the effectiveness of the proposed approach.

A New Projected Active Set Conjugate Gradient Approach for Taylor-Type Model Predictive Control: Application to Lower Limb Rehabilitation Robots With Passive and Active Rehabilitation.

In this paper, a three-order Taylor-type numerical differentiation formula is firstly utilized to linearize and discretize constrained conditions of model predictive control (MPC), which can be generalized from lower limb rehabilitation robots. Meanwhile, a new numerical approach that projected an active set conjugate gradient approach is proposed, analyzed, and investigated to solve MPC. This numerical approach not only incorporates both the active set and conjugate gradient approach but also utilizes a projective operator, which can guarantee that the equality constraints are always satisfied. Furthermore, rigorous proof of feasibility and global convergence also shows that the proposed approach can effectively solve MPC with equality and bound constraints. Finally, an echo state network (ESN) is established in simulations to realize intention recognition for human–machine interactive control and active rehabilitation training of lower-limb rehabilitation robots; simulation results are also reported and analyzed to substantiate that ESN can accurately identify motion intention, and the projected active set conjugate gradient approach is feasible and effective for lower-limb rehabilitation robot of MPC with passive and active rehabilitation training. This approach also ensures computational when disturbed by uncertainties in system.

Design of AGV human-machine interactive control system

The outdoor intelligent transportation of a pharmaceutical factory requires the workshop to put forward task requirements according to the situation and provide multiple AGVs that can meet both man-machine control and networked scheduling. On the basis of existing AGV, an AGV control method based on human-computer interaction is proposed. First, design the overall architecture of the control system, use the touch screen to design and develop the AGV mobile client and task request client to form an industrial wireless network. The establishment of a network enables the AGV mobile client, task request client and vehicle management system to achieve cross-platform communication. Based on this, the client system sets and controls the AGV, which effectively solves the AGV’s man-machine control and networking scheduling problems. After functional testing and performance verification, the system works stably, with good flexibility and scalability.

Conductive cold-resistant and elastic hydrogel: A potential bionic skin for human-machine interaction control over artificial limbs

This work enables conductive cold-resistant and elastic hydrogel for a new application in electric steering engine control, demonstrating a great potential as bionic skin for goal-directed control over artificial limbs. The photocrosslinking of acrylamide provides an elastic matrix that is functionally improved by physical crosslinking of K2CO3, carrageenan and locust bean gum. The hydrogel is highly elastic with the stress of 0.04 MPa at 650% strain, and can recover immediately after being stretched to 500% or pressed to 10% of its original length. It can keep its softness and conductivity at a low temperature of −10 °C. When an intermittent bending force is applied, it synchronously outputs the resistance variation signals between 1.1 and 0.9 times the original value. We take advantages of all these physical features to create a soft controller for the convenient manipulation of electric steering engine. We translate the bending-induced resistance variation signals into digital data of between 0 and 1023, which are then encoded to direct the electric steering engine for various controlled motility. A simple bending of hydrogel can activate the electric steering engine in 0.18 s, showing a highly-synchronous control over the dynamic response of electric steering engine in many models.

Human-machine Interaction: Adapted Safety Assistance in Mentality Using Hidden Markov Chain and Petri Net

This study proposes a cognition-adaptive approach for the administrative control of human-machine safety interaction through Internet of Things (IoT) data. As part of Industry 4.0, a human operator possesses various characteristics, but cannot be consistently understood as well as a machine. Thus, human-machine interaction plays an important role. This study focuses on incumbent challenges on the basis of estimated mental states. Given the operation logs from data recording hardware, a Hidden Markov model on top of a human cognitive model was trained to capture a production line worker’s sequential faults. Our study found that retaining workers’ attention is insufficient and tracking the state of perception is key to accomplishing production tasks. A safe workflow policy requires attention and perception. Accordingly, our proposed Petri Net enhances operation safety and improves production efficiency.

A Pilot Study on Using Forcemyography to Record Upper-limb Movements for Human-machine Interactive Control.

Forcemyography (FMG) is a useful method to record real-time body motions, which has application potentials for human-machine interactive control. The FMG registers the change of force distribution in the normal direction on muscle surface during limb movements, and the body motions can be recognized by decoding the FMG patterns. In this study, we used FMG to record upper-limb movements and evaluated the influence of different configurations of signal channel and feature on motion classification performances. A four-channel wearable FMG acquisition system was developed to record seven upper-limb movements on each of six able-bodied subjects. The preliminary results showed that the signal channel number has significant influence on motion classification performance; however, the influence of signal feature number on motion classification was insignificant. In addition, the influence of channel combination and feature combination were also discussed in this paper. This work would support the application potential of FMG for body motion recording and may provide useful instructions for the application of FMG in human-machine interactive control.

Orthogonal transformation operation theorem of a spatial universal uniform rotating magnetic field and its application in capsule endoscopy

According to the anti-phase sine current superposition theorem, the orientation, the magnetic flux density, the angular speed and the rotational direction of the spatial universal rotating magnetic field (SURMF) can be controlled within the tri-axial orthogonal square Helmholtz coils (TOSHC). Nevertheless, three coupling direction angles of the normal vector of the SURMF in the Descartes coordinate system cannot be separately controlled, thus the adjustment of the orientation of the SURMF is difficult and the flexibility of the robotic posture control is restricted. For the dimension reduction and the decoupling of control variables, the orthogonal transformation operation theorem of the SURMF is proposed based on two independent rotation angular variables, which employs azimuth and altitude angles as two variables of the three-phase sine current superposition formula derived by the orthogonal rotation inverse transformation. Then the unique control rules of the orientation and the rotational direction of the SURMF are generalized in each spatial quadrant, thus the scanning of the normal vector of the SURMF along the horizontal or vertical direction can be achieved through changing only one variable, which simplifies the control process of the orientation of the SURMF greatly. To validate its feasibility and maneuverability, experiments were conducted in the animal intestine utilizing the innovative dual hemisphere capsule robot (DHCR) with active and passive modes. It was demonstrated that the posture adjustment and the steering rolling locomotion of the DHCR can be realized through single variable control, thus the orthogonal transformation operation theorem makes the control of the orientation of the SURMF convenient and flexible significantly. This breakthrough will lay a foundation for the human-machine interaction control of the SURMF.

The Application of Aerospace Robot Based on Human-Machine Interactive Control Technology

As a new hot spot of robot, the research of aerospace robot has been developed rapidly. This paper presents a novel sensing system of aerospace robot based on human-machine interactive control technology. In order to establish the human-machine interactive interface, ten subjects’ biological signal features of electroencephalogram (EEG), electromyography (EMG) and voice were extracted quickly and accurately to discriminate the task and motion control for the aerospace robot. In the system, the multi-mode biological signals mentioned above were acquired by the biologic dry electrode sensor, then they were enlarged and analyzed by a microprocessor module based on AR model method. The simulation space experiments demonstrated the feasibility and accuracy of the aerospace robot control system which realized the multi-mode human-machine interactive interface. Obviously, this research explores a new way for astronaut working with aerospace robot in the outer space and it has great significance under some special environments.

Tool Position Control of an Upper Limb Exoskeleton for Robot-Assisted Surgery

This paper presents a medical robotic system for tool positioning tasks in orthopaedic surgery. The system consists of an 7 DOF upper limb exoskeleton, which is attached to the arm and back of an orthopaedic surgeon, and a real-time control system with an optical tracking camera. The objective is to guide the surgeon during an operation task to follow a predefined trajectory with a surgical tool. Therefore, a control algorithm is introduced to minimize the positioning error between the desired and actual pose (position and orientation) of the tool tip. The required joint velocities for positioning tasks are obtained by solving the inverse kinematic problem of the serial kinematic chain using an extended Jacobian method. For the computation of the Jacobian matrix, a geometric approach is chosen to solve the forward kinematics problem of the manipulator links. The over-determined kinematic system of the redundant robotic manipulator is determined using basic kinematic constraints of the human joint angles. Results of the inverse kinematics solution are evaluated with simulations of positioning trajectories for drilling tasks of the robotic manipulator and show a deviation of the desired path of less than 0.8 mm. The implemented end-effector positioning controller produces satisfactory results and enables future extensions for intuitive human-machine interaction control.