Control Framework For Teleoperation Research Articles

Haptic Shared Control Framework with Interaction Force Constraint Based on Control Barrier Function for Teleoperation.

Current teleoperated robotic systems for retinal surgery cannot effectively control subtle tool-to-tissue interaction forces. This limitation may lead to patient injury caused by the surgeon's mistakes. To improve the safety of retinal surgery, this paper proposes a haptic shared control framework for teleoperation based on a force-constrained supervisory controller. The supervisory controller leverages Control Barrier Functions (CBFs) and the interaction model to modify teleoperated inputs when they are deemed unsafe. This method ensures that the interaction forces at the slave robot's end-effector remain within the safe range without the robot's dynamic model and the safety margin. Additionally, the master robot provides haptic feedback to enhance the surgeon's situational awareness during surgery, reducing the risk of misjudgment. Finally, simulated membrane peeling experiments are conducted in a controlled intraocular surgical environment using a teleoperated robotic system controlled by a non-expert. The experimental results demonstrate that the proposed control framework significantly reduces the rate of force constraint violation.

Cooperative vs. Teleoperation Control of the Steady Hand Eye Robot with Adaptive Sclera Force Control: A Comparative Study.

A surgeon's physiological hand tremor can significantly impact the outcome of delicate and precise retinal surgery, such as retinal vein cannulation (RVC) and epiretinal membrane peeling. Robot-assisted eye surgery technology provides ophthalmologists with advanced capabilities such as hand tremor cancellation, hand motion scaling, and safety constraints that enable them to perform these otherwise challenging and high-risk surgeries with high precision and safety. Steady-Hand Eye Robot (SHER) with cooperative control mode can filter out surgeon's hand tremor, yet another important safety feature, that is, minimizing the contact force between the surgical instrument and sclera surface for avoiding tissue damage cannot be met in this control mode. Also, other capabilities, such as hand motion scaling and haptic feedback, require a teleoperation control framework. In this work, for the first time, we implemented a teleoperation control mode incorporated with an adaptive sclera force control algorithm using a PHANTOM Omni haptic device and a force-sensing surgical instrument equipped with Fiber Bragg Grating (FBG) sensors attached to the SHER 2.1 end-effector. This adaptive sclera force control algorithm allows the robot to dynamically minimize the tool-sclera contact force. Moreover, for the first time, we compared the performance of the proposed adaptive teleoperation mode with the cooperative mode by conducting a vessel-following experiment inside an eye phantom under a microscope.

Teleoperated Surgical Robot with Adaptive Interactive Control Architecture for Tissue Identification.

The remote perception of teleoperated surgical robotics has been a critical issue for surgeons in fulfilling their remote manipulation tasks. In this article, an adaptive teleoperation control framework is proposed. It provides a physical human-robot interaction interface to enhance the ability of the operator to intuitively perceive the material properties of remote objects. The recursive least square (RLS) is adopted to estimate the required human hand stiffness that the operator can achieve to compensate for the contact force. Based on the estimated stiffness, a force feedback controller is designed to avoid the induced motion and to convey the haptic information of the slave side. The passivity of the proposed teleoperation system is ensured by the virtual energy tank. A stable contact test validated that the proposed method achieved stable contact between the slave robot and the hard environment while ensuring the transparency of the force feedback. A series of human subject experiments was conducted to empirically verify that the proposed teleoperation framework can provide a more smooth, dexterous, and intuitive user experience with a more accurate perception of the mechanical property of the interacted material on the slave side, compared to the baseline method. After the experiment, the design idea about the force feedback controller of the bilateral teleoperation is discussed.

Synchronized Human-Humanoid Motion Imitation

We present a tele-operation control framework that (i) enhances the upper motion synchrony between a user and a robot using the minimum-jerk model coupled with a recursive least-square filter, and (ii) synchronizes the walking pace by predicting user's stepping frequency using motion capture data and a deep learning model. By integrating (i) and (ii) in a task-space whole-body controller, we achieve full-body synchronization. We assess our humanoid-to-human whole-body synchronized motion model on the HRP-4 humanoid robot in forward, lateral and backward walks with concurrent upper limbs motions experiments.

Unified Human Intention Recognition and Heuristic-Based Trajectory Generation for Haptic Teleoperation of Non-Holonomic Vehicles

In this paper, a novel bilateral shared control approach is proposed to address the issue of strong dependence on the human, and the resulting burden of manipulation, in classical haptic teleoperation systems for vehicle navigation. A Hidden Markov Model (HMM) is utilized to handle the Human Intention Recognition (HIR), according to the force input by the human—including the HMM solution, i.e., Baum–Welch algorithm, and HMM decoding, i.e., Viterbi algorithm—and the communication delay in teleoperation systems is added to generate a temporary goal. Afterwards, a heuristic and sampling method for online generation of splicing trajectory based on the goal is proposed innovatively, ensuring the vehicle can move feasibly after the change in human intention is detected. Once the trajectory is available, the vehicle velocity is then converted to joystick position information as the haptic cue of the human, which enhances the telepresence. The shared teleoperation control framework is verified in the simulation environment, where its excellent performance in the complex environment is evaluated, and its feasibility is confirmed. The experimental results show that the proposed method can achieve simple and efficient navigation in a complex environment, and can also give a certain situational awareness to the human.

A humanoid robot teleoperation approach based on waist–arm coordination

PurposeIn the current teleoperation system of humanoid robots, the control between arms and the control between the waist and arms are individual and lack coordinated motion. This paper aims to solve the above problem and proposes a teleoperation control approach for a humanoid robot based on waist–arm coordination (WAC).Design/methodology/approachThe teleoperation approach based on WAC comprises dual-arm coordination (DAC) and WAC. The DAC method realizes the coordinated motion of both arms through one hand by establishing a mapping relationship between a single hand controller and the manipulated object; the WAC method realizes the coordinated motion of both arms and waist by calculating the inverse kinematic input of robotic arms based on the desired velocity of the waist and the end of both arms. An integrated teleoperation control framework provides interfaces for the above methods, and users can switch control modes online to adapt to different tasks.FindingsAfter conducting experiments on the dual-arm humanoid robot through the teleoperation control framework, it was found that the DAC method can save 27.2% of the operation time and reduce 99.9% of the posture change of the manipulated object compared with the commonly used individual control. The WAC method can accomplish a task that cannot be done by individual control. The experiments proved the improvement of both methods in terms of operation efficiency, operation stability and operation capability compared with individual control.Originality/valueThe DAC method better maintains the constraints of both arms and the manipulated object. The WAC method better maintains the constraints of the manipulated object itself. Meanwhile, the teleoperation framework integrates the proposed methods and enriches the teleoperation modes and control means.

A Novel SMMS Teleoperation Control Framework for Multiple Mobile Agents With Obstacles Avoidance by Leader Selection

Teleoperation of multiple agents has the unique advantage to complete tasks with wide range and is an effective solution to help agents avoid obstacles with human intelligence, especially when encounters the local-minima problem. In this article, a novel nonlinear single-master–multislave (SMMS) teleoperation control framework is proposed for multiple mobile agents to achieve obstacles avoidance under delays, nonlinearities, various uncertainties, and nonholonomic constraints. Namely, the slave trajectory planner is designed to cope with the nonholonomic constraints caused by underactuated characteristics of slave agents, while the slave obstacle avoidance planner is designed to cope with obstacles in the environment, which can avoid the obstacles by artificial potential function (APF)-based obstacle avoidance algorithm. Particularly, considering that the APF usually encounters the local-minima problem, a leader selection algorithm is designed for the slave obstacle avoidance planner and a virtual force feedback is designed for the master subsystem, where the slave agents can get rid of local-minima points while teleoperated by human operator with confident force feedback. The global stability of the overall system can be guaranteed under the proposed radial basis function neural network (RBFNN)-based adaptive sliding mode master controller and slave formation controller under delays, nonlinearities and various uncertainties. The comparative experiment is implemented, and the results show the effectiveness of proposed control framework in the achievement of good performance including position tracking, force feedback, and formation and obstacles avoidance while the stability is guaranteed.

Brain Teleoperation Control of a Nonholonomic Mobile Robot Using Quadrupole Potential Function

This paper presents the development of a brain-machine interfacing teleoperation control framework of a mobile robot using quadrupole potential function (QPF). The online brain-computer interface is based on steady-state visually evoked potentials, which employs the multivariate synchronization index classification algorithm to decode the human electroencephalograph (EEG) signals. In this way, human intentions can be recognized and EEG control commands can be produced for the mobile robot. Besides, a novel artificial potential function named QPF is designed to produce the potential fields to parameterize the EEG control commands. The classification results of the EEG signal are related to the distribution of the potential fields, and the collision-free path to a target position is automatically planned by the potential field. Such semi-automatic design builds the direct mapping from human intentions to the mobile robot's behavior and reduces mental effort or exhaustion on subject's part. Besides, based on the QPF with the capability of handling nonholonomic constraints, the integrated system can not only achieve obstacle avoidance but is also able to automatically stabilize the robot to a target configuration. Extensive experiment studies were conducted on several subjects to demonstrate the effectiveness of the proposed approaches.

A modular bilateral haptic control framework for teleoperation of robots

SUMMARYThis paper presents a novel approach to implement bilateral control loops between local haptic devices and remote industrial manipulators using a layer of simulation and virtual reality. The remote scene of manipulation has been visualized in an open-source software environment, where forward and inverse kinematics of the manipulators can be computed. Therefore, the explicit knowledge of mathematical models of the robots is not required for the implementation of the proposed bilateral control schemes. A haptic coupling has been designed between the human operator and the task in the remote environment. Virtually introduced force feedback has contributed to the performance of the proposed bilateral loop by facilitating the adaptation of unexperienced human operators. Teleoperation of one remote manipulator has been experimentally demonstrated with the proposed controllers. Structural modularity of the bilateral haptic control schemes makes them directly extendable for the teleoperation of multiple collaborative robots. Stability and transparency of the proposed bilateral haptic controllers have been theoretically and experimentally investigated.

Bilateral Adaptive Control of Nonlinear Teleoperation Systems With Uncertain Dynamics and Dead-Zone

Dead-zone is one of the most common hard nonlinearities ubiquitous in master–slave teleoperation systems, particularly in the slave robot joints. However, adaptive control techniques applied in teleoperation systems usually deal with dynamic uncertainty but ignore the presence of dead-zone. Dead-zone has the potential to remarkably deteriorate the transparency of a teleoperation system in the sense of position and force tracking performance or even destabilizing the system if not compensated for in the control scheme. In this paper, an adaptive bilateral control scheme is proposed for nonlinear teleoperation systems in the presence of both uncertain dynamics and dead-zone. An adaptive controller is designed for the master robot with dynamic uncertainties and the other is developed for the slave robot with both dynamic uncertainties and unknown dead-zone. The two controllers are incorporated into the four-channel bilateral teleoperation control framework to achieve transparency. The transparency and stability of the closed-loop teleoperation system is studied via a Lyapunov function analysis. Comparisons with the conventional adaptive control which merely deal with dynamic uncertainties in the simulations demonstrate the validity of the proposed approach.

Brain–Machine Interfacing-Based Teleoperation of Multiple Coordinated Mobile Robots

This paper describes the development of a teleoperation control framework of multiple coordinated mobile robots through a brain–machine interface (BMI). Utilizing the remote images of an environment, transferred to the human operator, visual compressive feedback loop produces imagine errors in nonvector space, where images are considered as a set without image processing of feature extraction. Given an initial set and a goal set, visual evoked potentials are used to generate EEG motion commands to make the image set converge to the goal set. The online BMI, utilizing steady-state visually evoked potentials, analyzes the human EEG data in such a format that human intentions can be recognized by AdaBoostSVM classifier and motion commands produced for the teleoperated robot. Bezier curve is utilized to parameterize the motion commands and leader–follower formation control is proposed to guarantee a good reference trajectory tracking performance. Extensive experimental studies have been carried out to assess the effectiveness of the proposed approaches.

Adaptive Control of Teleoperation Systems With Linearly and Nonlinearly Parameterized Dynamic Uncertainties

Existing work concerning adaptive control of uncertain teleoperation systems only deals with linearly parameterized (LP) dynamic uncertainties. Typical teleoperation system dynamics, however, also posses terms with nonlinearly parameterized (NLP) structures. An example of such terms is friction, which is ubiquitous in the joints of the master and slave robots of practical teleoperation systems. Uncertainties in the NLP dynamic terms may lead to significant position and force tracking errors if not compensated for in the control scheme. In this paper, adaptive controllers are designed for the master and slave robots with both LP and NLP dynamic uncertainties. Next, these controllers are incorporated into the 4-channel bilateral teleoperation control framework to achieve transparency. Then, transparency of the overall teleoperation is studied via a Lyapunov function analysis. Simulation studies demonstrate the effectiveness of the proposed adaptive scheme when exact knowledge of the LP and NLP dynamics is unavailable.

Adaptive Inverse Dynamics Four-Channel Control of Uncertain Nonlinear Teleoperation Systems

Most of the methods to date on bilateral control of nonlinear teleoperation systems lead to nonlinear and coupled closed-loop dynamics, even in the ideal case of perfect knowledge of the master, the slave, the human operator and the environment. Consequently, the transparency of these closed-loop systems is difficult to study. In comparison, inverse dynamics controllers can deal with the nonlinear terms in the dynamics in a way that, in the ideal case, the closed-loop systems become linear and decoupled. In this paper, for multi-d.o.f. nonlinear teleoperation systems with uncertainties, adaptive inverse dynamics controllers are incorporated into the four-channel bilateral teleoperation control framework. The resulting controllers do not need exact knowledge of the dynamics of the master, the slave, the human operator or the environment. A Lyapunov analysis is presented to prove the transparency of the teleoperation system. Simulations are also presented to show the effectiveness of the proposed approach.

Hybrid PD-Based Control Framework for Passive Bilateral Teleoperation over the Internet

AbstractLet us consider the problem of bilateral teleoperation over the Internet. One of its aspects, which has not been much utilized so far, is its hybrid nature, that is, the (nonlinear) master and slave robots are continuous-time systems, while the Internet communication between them defines a discrete-time channel. Exploiting this hybrid nature and combining our prior results [Lee [2009], Huang and Lee [2010]], in this paper, we propose a novel hybrid proportional-derivative (PD) type passivity-enforcing bilateral teleoperation control framework over the Internet with such communication unreliability as varying-delay, packet-loss, data-duplication/swapping, etc. The closed-loop passivity of the teleoperator, position coordination and the force reflection between master and slave robots are mathematically proved.