Master Side Research Articles

Online Bilateral Predictive Control for Time-Delayed Teleoperation of Snake-like Robots

Controlling the teleoperation of snake-like robots is challenging due to complex nonlinear dynamics and communication delays. This research proposes an online bilateral predictive control architecture to address these issues. This control structure is established by predicting environment force and the user’s future motion. The former uses a model-mediated approach by creating a virtual environment on the master side and the latter adopts an artificial neural network (ANN) for online operator’s motion prediction. The slave controller utilizes transmitted data from ANN to generate required backbone lengths, which are then transformed into the slave's local bending and torsional degrees of freedom through the inverse kinematics of the robot. Motion prediction is examined in two scenarios: when the ANN predicts the trained motions, and when it predicts a different motion. Simulation studies demonstrate that the proposed online bilateral predictive teleoperation structure successfully achieves real-time position synchronization and force feedback, by effectively bypassing communication delays.

Fundamental Study on Force-Projecting Bilateral Control for Pneumatically Driven Follower Device

This paper proposes the application of force-projecting bilateral control to a master-follower teleoperation system with pneumatic drive on the follower side and evaluates its effectiveness. The proposed method directly projects the operating force on the master side to the driving force on the follower side, eliminating the need for both position control and external force detection on the follower side, thereby solving the problem of low rigidity and response delay of a pneumatic servo system and providing highly stable sensor-less force presentation against variable environments. In this study, dynamic response analyses of a 1-DOF master-follower system were performed by numerical simulation using a linear system model, followed by experimental verification by implementing an actual system with an external force estimator. The results showed that the proposed force-projecting bilateral control has significantly higher positioning rigidity and better force control stability than the conventional force-reflecting bilateral control. A theoretical consideration was also given using the equivalent transformation of force transfer functions to provide evidence of high stability.

A Bilateral Teleoperation System With Learning-Based Cognitive Guiding Force

This article investigates the problem involving a fuzzy logic system-based bilateral teleoperation system with a unified impedance/admittance structure and learning-based assisted cognitive guiding force, enhancing a smooth transition between augmentation and autonomous motion on the master side. The impedance structure regulates the impedance and admittance via the parameters determined by the fuzzy logic system and, besides the variable impedance during interaction with the environment, the fuzzy Q-learning algorithm generates an assisted cognitive guiding force to guide the trainee operator in free space for reaching the interaction surface smoothly and steadily. The degeneration of tracking performance in the bilateral teleoperation system caused by time delay is analyzed and reduced via a mixed-type error. Experimental results verify the stability analyses, and under the proposed control scheme, the human-robot-environment interaction is smooth and stable.

Hierarchical finite-time cooperative control for teleoperation of networked disturbed mobile manipulators

This paper investigates the teleoperation problem of networked disturbed mobile manipulators (NDMMs), in which the human operator remotely controls multiple slave mobile manipulators through a master manipulator. Each individual of the slave ones consisted of a nonholonomic mobile platform and a holonomic constrained manipulator that is mounted on the nonholonomic mobile platform. The cooperative control objective of the considered teleoperation problem includes: (1) synchronizing the states of the slave manipulators to the human-controlled master one; (2) forcing the mobile platforms of the slave ones to form a user-defined formation; (3) controlling the geometric center of all the platforms to track a reference trajectory. We present a hierarchical finite-time cooperative control (HFTCC) framework to achieve the cooperative control goal in a finite time. The presented framework includes the distributed estimator, the weight regulator and the adaptive local controller, where the estimator generates the estimated states of the desired formation and trajectory, the regulator selects the slave robot that the master one needs to track, as well as the presented adaptive local controller guarantees the finite-time convergence of the controlled states with model uncertainties and disturbances. Additionally, for improving the telepresence, a novel super twisting observer is presented to reconstruct the interaction force between the salve mobile manipulators and the remote operating environment on the master (i.e., the human) side. Finally, the effectiveness of the proposed control framework is demonstrated by several simulation results.

Adaptive Control of a Master-Slave Based Robotic Surgical System With Haptic Feedback

A master-slave (M-S) based robotic surgical system with haptic feedback is presented in this work for natural orifice transluminal endoscopic surgery (NOTES) using an in-vivo robot. A novel dual-loop control, including closed-loop inverse kinematics (CLIK) and adaptive control with optimal control parameters, is developed for the slave robot to track a random trajectory upon updating the dynamic parameters of the slave robot. The interaction between the slave robot tip position and environment is controlled by an impedance controller that prevents damage to healthy tissues. Upon tracking a given desired 3D trajectory as input for the slave robot by the human operator handling the master robot, the slave robot-tissue interaction force is sensed as haptic feedback at the master side. Lyapunov direct method based stability analysis and performance evaluations involving both simulation and experimental studies prove the efficacy of our novelly developed M-S robotic system aiming NOTES regulated by the proposed adaptive control strategy with optimized control parameters. Note to Practitioners: The M-S based robotic surgical system is developed to perform NOTES using a cable-driven in-vivo robot attached to the tip of a flexible endoscope. It performs tissue manipulation as per the direction given by the operator. The inclusion of haptic imitates the NOTES performed using the in-vivo robot as conventional surgery. The developed system includes the following aspects: a) The adaptive control scheme with optimal control parameters; b) Interaction force between the slave robot and tissue environment is controlled using impedance control and given as haptic feedback to the surgeon; c) The proposed system can be potentially utilized for actual teleoperated surgery and surgical training for NOTES.

Design of a Novel Haptic Joystick for the Teleoperation of Continuum-Mechanism-Based Medical Robots

Continuum robots are increasingly used in medical applications and the master–slave-based architectures are still the most important mode of operation in human–machine interaction. However, the existing master control devices are not fully suitable for either the mechanical mechanism or the control method. This study proposes a brand-new, four-degree-of-freedom haptic joystick whose main control stick could rotate around a fixed point. The rotational inertia is reduced by mounting all powertrain components on the base plane. Based on the design, kinematic and static models are proposed for position perception and force output analysis, while at the same time gravity compensation is also performed to calibrate the system. Using a continuum-mechanism-based trans-esophageal ultrasound robot as the test platform, a master–slave teleoperation scheme with position–velocity mapping and variable impedance control is proposed to integrate the speed regulation on the master side and the force perception on the slave side. The experimental results show that the main accuracy of the design is within 1.6°. The workspace of the control sticks is −60° to 110° in pitch angle, −40° to 40° in yaw angle, −180° to 180° in roll angle, and −90° to 90° in translation angle. The standard deviation of force output is within 8% of the full range, and the mean absolute error is 1.36°/s for speed control and 0.055 N for force feedback. Based on this evidence, it is believed that the proposed haptic joystick is a good addition to the existing work in the field with well-developed and effective features to enable the teleoperation of continuum robots for medical applications.

A Novel Master-Slave Interventional Surgery Robot with Force Feedback and Collaborative Operation.

In recent years, master-slave vascular robots have been developed to address the problem of radiation exposure during vascular interventions for surgeons. However, the single visual feedback reduces surgeon immersion and transparency of the system. In this work, we have developed a haptic interface based on the magnetorheological fluid (MRF) on the master side. The haptic interface can provide passive feedback force with high force fidelity and low inertia. Additionally, the manipulation of the master device does not change the operating posture of traditional surgery, which allows the surgeon to better adapt to the robotic system. For the slave robot, the catheter and guidewire can be navigated simultaneously which allows the two degrees of action on the catheter and axial action of a guidewire. The resistance force of the catheter navigation is measured and reflected to the user through the master haptic interface. To verify the proposed master-slave robotic system, the evaluation experiments are carried out in vitro, and the effectiveness of the system was demonstrated experimentally.

Design of robust adaptive fuzzy control for uncertain bilateral teleoperation systems based on backstepping approach

AbstractIn this study, a novel method based on a robust adaptive fuzzy control approach is developed for nonlinear teleoperation systems. Its main objectives are to ensure system stability and properly mitigating parametric uncertainties stemming from external disturbances and un‐modelled dynamics. For the communication channel, instead of the direct transmission of environmental torque signals, the approximated environmental parameters by the fuzzy system are transmitted to the master side for the prediction of environmental torque, thus successfully avoiding the transmission of the power signals in the delayed communication channel and solving the passivity problem in the teleoperation system. Besides, a trajectory generator is employed in the master side, whereas a trajectory smoothing is provided in the slave side. Theoretically, it was proven that both position tracking and force feedback problems are attained. Using Lyapunov stability analysis, this work illustrates that the robust adaptive fuzzy controller based on the backstepping approach guarantees the system's asymptotic stability. Simulation results confirm the efficiency of the suggested control technique in achieving the stability and tracking objectives of the uncertain nonlinear teleoperation system.

Design and research of CFETR peg hole assembly virtual platform

To fulfil in-vessel maintenance in the China Fusion Engineering Test Reactor (CFETR), a Master-Slave remote handling system with high reliability, stability, and flexibility is required. Due to the complexity of CFETR's internal environment and the robot system, operators need to be well trained before any physical maintenance is conducted. It is of high risk, high cost, and low efficiency that operators are trained in physical environment. The above problems can be avoided when operators are trained in a virtual environment. However, it is difficult to achieve the realization in the master side and contact force in the slave side. In this paper, we develop a new Virtual Assembly Simulation Platform with high quality realization of the contact force and the operator's telepresence. The platform consists of a master-slave robot system and a simulation environment build upon CoppeliaSim. To remove the influence of the tool's gravity on the feedback force obtained by the force sensor installed at the end of robot. We propose a least square method-based gravity compensation algorithm. The replaceability of the virtual platform to the physical platform and the effectiveness of the algorithm are demonstrated by the peg-in-hole assembly experiment.

Analytical Evaluation of Force-Projecting Bilateral Control for Pneumatic Manipulators

This paper presents performance and stability analyses of the Force-Projecting Bilateral Control (the Force-Projecting Type) for master-follower operation of pneumatic manipulators. In this control method, the operational force on the master side is directly projected to the driving force on the follower. Unlike other bilateral control methods, the Force-Projecting Type does not have a position controller on the follower side. Therefore, it is considered highly suitable for manipulators with low-rigidity and slow response, such as pneumatic manipulators. In this study, the Force-Projecting Type was applied to a one-degree-of-freedom master-follower model with characteristics of low-rigidity and response delay as a pneumatic manipulator on the follower side, and the time and frequency responses were numerically investigated. Then, the responses were compared with those of the conventional Force-Reflecting Bilateral Control. As a result of the analyses, the Force-Projecting Type showed both higher stability and performance. In addition, the equivalent block diagram of the system suggests that the Force-Projecting Type is more robust to changes in environmental characteristics.

A novel catheter interaction simulating method for virtual reality interventional training systems

Endovascular robotic systems have been applied in robot-assisted interventional surgery to improve surgical safety and reduce radiation to surgeons. However, this surgery requires surgeons to be highly skilled at operating vascular interventional surgical robot. Virtual reality (VR) interventional training systems for robot-assisted interventional surgical training have many advantages over traditional training methods. For virtual interventional radiology, simulation of the behaviors of surgical tools (here mainly refers to catheter and guidewire) is a challenging work. In this paper, we developed a novel virtual reality interventional training system. This system is an extension of the endovascular robotic system. Because the master side of this system can be used for both the endovascular robotic system and the VR interventional training system, the proposed system improves training and reduces the cost of education. Moreover, we proposed a novel method to solve catheterization modeling during the interventional simulation. Our method discretizes the catheter by the collision points. The catheter between two adjacent collision points is treated as thin torsion-free elastic rods. The deformation of the rod is mainly affected by the force applied at the collision points. Meanwhile, the virtual contact force is determined by the collision points. This simplification makes the model more stable and reduces the computational complexity, and the behavior of the surgical tools can be approximated. Therefore, we realized the catheter interaction simulation and virtual force feedback for the proposed VR interventional training system. The performance of our method is experimentally validated.

Gesture Recognition and Master–Slave Control of a Manipulator Based on sEMG and Convolutional Neural Network–Gated Recurrent Unit

AbstractSurface electromyography signal (sEMG) is the bio-electric signal accompanied by muscle contraction. For master–slave manipulation scenario such as patients with prosthetic hands, their upper limb sEMG signals can be collected and corresponded to the patient's gesture intention. Therefore, using a slave manipulator that integrated with the sEMG signal recognition module, the master side could control it to make gestures and meet their needs of daily life. In this paper, gesture recognition is carried out based on sEMG and deep learning, and the master–slave control of manipulator is realized. According to the results of training, the network model with the highest accuracy of gesture classification and recognition can be obtained. Then, combined with the integrated manipulator, the control signal of the manipulator corresponding to the gesture is sent to the control module of the manipulator. In the end, a prototype system is built and the master–slave control of the manipulator using the sEMG signal is realized.

Sliding Mode Impedance Control for Dual Hand Master Single Slave Teleoperation Systems

For the purpose of avoiding injury and realizing precise operations, the multilateral teleoperation system is the most efficient way to transport trace toxic or radioactive substances, to perform minimally invasive surgery, etc. It is essential to enhance the transparency of a multilateral teleoperation system including multiple masters and the single slave manipulator. However, there are few researchers focus on the allocation of the contact force of the single slave manipulator to different master manipulators. In this paper, we firstly introduce the concept of force translation for teleoperation systems consisting of dual hand master (left and right hands) manipulators and a single slave manipulator. Force translation reflects how the impedance on the single slave side is translated or allocated to contact forces on different master sides. To maintain the stability of the system and to improve transparency, we elucidate the mechanism of the force translation and analyze the relation among masters and the slave. Furthermore, the force translation mechanism is analyzed through numerical simulations and CHAI 3D virtual physical simulations. It is used to propose a multilateral impedance control, and the Lyapunov function is used to analyze the system stability. The results of numerical simulations and real robot experiments verify the effectiveness of the proposed control methods based on the proposed force translation mechanism.

Adaptive control for position and force tracking of uncertain teleoperation with actuators saturation and asymmetric varying time delays

Abstract This paper presents a new bounded force feedback control law to improve transparency in nonlinear bilateral teleoperation systems in the presence of three problems in practical applications of teleoperation systems such as input saturation, asymmetric time varying communication delays with no restriction on their rates of variation and parametric uncertainties, simultaneously. The proposed controller is a nonlinear-proportional plus nonlinear damping (nP + nD) controller with the addition of a nonlinear adaptive term and nonlinear function of the environment force on the slave side and nonlinear function of the human force and force error on the master side. Using a novel Lyapunov–Krasovskii functional, the asymptotic stability and position and force tracking performance of the teleoperation system are established under specific conditions on the controller parameters, actuator saturation characteristics and maximum allowable time delay. The validity of the theoretical results is corroborated by the simulation results.

Design, development and evaluation of an ergonomically designed dual-use mechanism for robot-assisted cardiovascular intervention.

Robot-assisted cardiovascular intervention has been recently developed, which enables interventionists to avoid x-ray radiation and improve their comfort. However, there are still some challenges in the robotic design, such as the inability of the interventionist to freely perform natural clinical techniques and the limited motion travel of the interventional tool. To overcome these challenges, this paper proposes an ergonomically designed dual-use mechanism for cardiovascular intervention (DMCI). DMCI can work as an ergonomic interface or a compact slave robot with unlimited motion travel. Our kinematic analysis of DMCI includes motion decoupling and coupling. Motion decoupling decomposes the translation and rotation from the interventionist's natural clinical actions at the master side. Motion coupling can calculate the input pulses of motors according to the desired rotation and translation, thus composing the motion of the intervention tool at the slave side. Our kinematic analysis of DMCI has been experimentally verified, where the overall mean rotational errors are all less than 1° and translational errors are all less than 1mm. We also evaluated the performance of the DMCI-based master-slave system, where the overall rotational and translational errors are 0.821 ± 0.753° and 0.608 ± 0.512mm. Moreover, operators were found to be generally more efficient when using the DMCI-based interface compared to the conventional joystick. We have validated our kinematic analysis of DMCI. The master-slave teleoperation experiment demonstrated that operators can freely perform natural clinical techniques through the DMCI-based interface, and the slave robot can replicate the operators' manipulation at the master side well.

Design and Performance Evaluation of a Novel Slave System for Endovascular Tele-Surgery

Vascular interventional robots have attracted growing attention in recent years. However, current vascular interventional robot systems generally lack force feedback and cannot quickly clamp the catheter/guidewire. The structure of slave systems is unstable and the power transmission is imprecise, increasing the system’s safety hazards. Vascular intervention robots generally do not follow traditional surgeons’ operation habits and, thus, it is not easy for them to understand and learn how to operate. Therefore, a novel vascular intervention system is proposed. The slave system can quickly clamp the catheter/guidewire, is compatible with various standard catheter/guidewire sizes, has precise power transmission, and has a stable structure. The surface of the catheter/guidewire is clamped without damage. Whether it is on the master side or the slave side, it follows the habits of traditional operators to a great extent. The results show that the measurement accuracy of the axial force meets the requirements of robot-assisted surgery and the system can track the designed position of the catheter/guidewire in real time. This study makes a certain contribution to the development of master–slave systems for endovascular tele-surgery.

Machine Learning-Based Surgical State Perception and Collaborative Control for a Vascular Interventional Robot

In robot-assisted vascular interventional surg- ery (VIS), surgeons often need to operate outside the operating room to avoid exposure to X-ray. However, it greatly changes the operating ways of surgeons, which affects judgment and operation safety. In this paper, a novel VIS robot system was developed to predict guidewire insertion states and operate collaboratively. To assist the surgeons in perceiving the insertion state, an insertion multi-states prediction model based on softmax logistic regression was proposed. Combined with the prediction model, a human-machine collaborative control strategy was designed, which allows surgeons to perceive the insertion states based on not only the force feedback constructed by the master side but also the prediction results from the slave side. Moreover, a human-machine trust evaluation model and a master-slave collaborative mapping model were proposed for improving safety and efficiency of surgery. To verify the effectiveness of these models, the evaluation experiments in the blood vessel model were carried out. It was indicated by the experiment results that the guidewire insertion states can be predicted by the prediction model in different environments, and the overall accuracy is 93%. The master-slave mapping ratio can be adjusted by the collaborative control strategy automatically to adapt to different surgical conditions. The experimental results showed the usability of the robot-assisted VIS system with the novel force-based perception method.

An Endovascular Catheterization Robotic System Using Collaborative Operation with Magnetically Controlled Haptic Force Feedback.

Robot-assisted technology is often used to perform endovascular catheterization surgeries, which generally depend on the flexible operability and the accurate force feedback of a robotic system. In this paper, an endovascular catheterization robotic system (ECRS) was developed to improve collaborative operation and haptic force feedback. A couple of operating handles were designed to maximize the use of the natural operations of surgeons on the master side, which is a flexible and ergonomic device. A magnetically controlled haptic force feedback structure is proposed based on hydrogel and solid magnetorheological (MR) fluid to offer a sense of haptic feedback to operators; this has potential influence on the field of force feedback. In addition, a unique tremor-reduction structure is introduced to enhance operating safety. Tracking performance experiments and in vitro experiments were conducted to evaluate the performance of the developed ECRS. According to these experimental results, the average translation-tracking error is 0.94 mm, and the average error of rotation is 0.89 degrees. Moreover, in vitro experiments demonstrated that haptic feedback has the advantage of reducing workload and shortening surgery completion time. The developed ECRS also has the benefits of inspiring other researchers to study collaborative robots and magnetically controlled feedback.

Safe operation of scaling bilateral control system under application of excessive force by operator

Physical interaction between humans and robots in a remote environment can be realized safely with bilateral teleoperation. The acceleration-based scaling bilateral controller and the reaction force estimator implemented with the disturbance observer facilitate robust control and vivid touch sensation between different structures. This paper introduces a replica-side security enhancement method when the operator applies excessive force on the master during scaling bilateral motion. Initially, the system follows four-channel acceleration-based scaling bilateral control. It switches to the master side virtual impedance controller and replica side force controller as the bearable force limit is reached on the replica. As a result, the operator-applied excessive force is not transmitted to the remote end, and the master maneuvers freely following the operator's intention. This method supports the operator's freedom while protecting the replica object. This study derives the value of virtual impedance aiming at high reproducibility. As a result, the operator receives replica side object impedance throughout the operation even when controllers are switched. This study analyzes the stability considering pole placement with system parameter variations. Experiments verified the proposed method. The results demonstrate that the proposed method ensures the safety of remote objects while the operator freely applies excessive force. The operator was able to successfully sense object impedance irrespective of whether the system followed scaling bilateral control or constrained scaling motion control.

Experimental Evaluation on Haptic Feedback Accuracy by Using Two Self-Made Haptic Devices and One Additional Interface in Robotic Teleoperation

The goal of haptic feedback in robotic teleoperation is to enable users to accurately feel the interaction force measured at the slave side and precisely understand what is happening in the slave environment. The accuracy of the feedback force describing the error between the actual feedback force felt by a user at the master side and the measured interaction force at the slave side is the key performance indicator for haptic display in robotic teleoperation. In this paper, we evaluate the haptic feedback accuracy in robotic teleoperation via experimental method. A special interface iHandle and two haptic devices, iGrasp-T and iGrasp-R, designed for robotic teleoperation are developed for experimental evaluation. The device iHandle integrates a high-performance force sensor and a micro attitude and heading reference system which can be used to identify human upper limb motor abilities, such as posture maintenance and force application. When a user is asked to grasp the iHandle and maintain a fixed position and posture, the fluctuation value of hand posture is measured to be between 2 and 8 degrees. Based on the experimental results, human hand tremble as input noise sensed by the haptic device is found to be a major reason that results in the noise of output force from haptic device if the spring-damping model is used to render feedback force. Therefore, haptic rendering algorithms should be independent of hand motion information to avoid input noise from human hand to the haptic control loop in teleoperation. Moreover, the iHandle can be fixed at the end effector of haptic devices; iGrasp-T or iGrasp-R, to measure the output force/torque from iGrasp-T or iGrasp-Rand to the user. Experimental results show that the accuracy of the output force from haptic device iGrasp-T is approximately 0.92 N, and using the force sensor in the iHandle can compensate for the output force inaccuracy of device iGrasp-T to 0.1 N. Using a force sensor as the feedback link to form a closed-loop feedback force control system is an effective way to improve the accuracy of feedback force and guarantee high-fidelity of feedback forces at the master side in robotic teleoperation.