Teleoperation Control Research Articles

Intuitive Robot Teleoperation Through Multi-Sensor Informed Mixed Reality Visual Aids

Mobile robotic systems have evolved to include sensors capable of truthfully describing robot status and operating environment as accurately and reliably as never before. This possibility is challenged by effective sensor data exploitation, because of the cognitive load an operator is exposed to, due to the large amount of data and time-dependency constraints. This paper addresses this challenge in remote-vehicle teleoperation by proposing an intuitive way to present sensor data to users by means of using mixed reality and visual aids within the user interface. We propose a method for organizing information presentation and a set of visual aids to facilitate visual communication of data in teleoperation control panels. The resulting sensor-information presentation appears coherent and intuitive, making it easier for an operator to catch and comprehend information meaning. This increases situational awareness and speeds up decision-making. Our method is implemented on a real mobile robotic system operating outdoor equipped with on-board internal and external sensors, GPS, and a reconstructed 3D graphical model provided by an assistant drone. Experimentation verified feasibility while intuitive and comprehensive visual communication was confirmed through an assessment, which encourages further developments.

Quantitative Estimation of Dynamic Modulation in Remote Environment Sensing

Teleoperated robots are an effective means to sense a remote environment. Impedance control is often preferred in teleoperated robots due to its capability to modulate the dynamic relationship between the position and force of the robot. One of the challenges in the practical implementation of impedance controllers is the determination of the extent to which the dynamic properties of the robot are to be modified. A methodology for quantitatively estimating the desired dynamic parameters of the robot in order to have stable interaction with the environment is proposed in this paper. The best set of scaling factors by which the environment parameters need to be scaled to obtain a corresponding desired parameter for the robot are formulated. In addition, novel impedance controllers for teleoperated robots, which achieve the desired dynamic performance even under time-varying delays, are formulated in this paper. The uniqueness of the proposed controllers lies in enabling the system to interact with a wide range of environments, unlike the traditional teleoperation controllers which allow interaction with only those environments whose dynamic properties are within a specific range. Passivity analysis is performed to prove the global asymptotic stability of the system. Experimental implementation demonstrates the successful sensing of a wide range of environments.

Admittance parameterization in linear networked bilateral teleoperation control

In this paper we study linear control of delayed and/or sampled bilateral teleoperation with LTI master and slave devices. A complete parameterization of admittance operators achievable with nominally stabilizing control laws is derived. The potential of this parameterization for controller synthesis is illustrated with a numerical example. In addition, a generic structure of nominally stabilizing controllers is revealed and its properties are discussed. It is shown that delay-independent nominal stabilization in linear networked bilateral teleoperation control does not imply limitations on performance and coupled stability.

Disturbance-Observer-Based Fuzzy Control for a Robot Manipulator Using an EMG-Driven Neuromusculoskeletal Model

Robot manipulators have been extensively used in complex environments to complete diverse tasks. The teleoperation control based on human-like adaptivity in the robot manipulator is a growing and challenging field. This paper developed a disturbance-observer-based fuzzy control framework for a robot manipulator using an electromyography- (EMG-) driven neuromusculoskeletal (NMS) model. The motion intention (desired torque) was estimated by the EMG-driven NMS model with EMG signals and joint angles from the user. The desired torque was transmitted into the desired velocity for the robot manipulator system through an admittance filter. In the robot manipulator system, a fuzzy logic system, utilizing an integral Lyapunov function, was applied for robot manipulator systems subject to model uncertainties and external disturbances. To compensate for the external disturbances, fuzzy approximation errors, and nonlinear dynamics, a disturbance observer was integrated into the controller. The developed control algorithm was validated with a 2-DOFs robot manipulator in simulation. The results indicate the proposed control framework is effective and crucial for the applications in robot manipulator control.

Observer-based adaptive force–position control for nonlinear bilateral teleoperation with time delay

In this paper, a novel observer-based force control scheme is proposed to guarantee the position and force tracking in nonlinear teleoperation systems, subject to constant communication time delay. Stability of the teleoperation system is analytically proved using the Lyapunov stability theorem. Enough condition for asymptotic convergence of the force and position are derived. The transparency is also improved by force tracking capability of the system. The teleoperation controller is then applied to a pair of planar 2-DOF robots, connected via a communication channel with constant time delay. The experimental and simulation results demonstrate the effectiveness of the proposed control algorithm, in force estimation and dealing with time delay.

Teleoperation Control Design with Virtual Force Feedback for the Cable-Driven Hyper-Redundant Continuum Manipulator

Hyper-redundant continuum manipulators present dexterous kinematic skills in complicated tasks and demonstrate promising potential in underground exploration, intra-cavity inspection, surgery, etc. However, the hyper-redundancy, which endows much dexterity and flexibility, brings a huge challenge to the kinematics solution and control of the continuum manipulators. Due to the pseudoinverse calculation of high-order Jacobian matrix or iteration, many inverse kinematic solution approaches of continuum manipulators are very time-consuming, which extremely limit their applicability in real-time control. Additionally, it is often difficult for the manipulators to perform the tasks well in complex scenarios due to lack of human intervention. Therefore, in this paper, a simplified kinematics model of a typical hyper-redundant manipulator is proposed based on its unique geometry relationships, where the mapping relationships between the actuators’ rotation and the end-effector’s position are derived through the analysis of its driving subsystem and motion subsystem, in particular the joint modules. To perform the tasks of manipulators with the help of operators, a teleoperation control scheme with modified wave transmission structure is designed to achieve the guaranteed stability and improved transparency, and the leader’s trajectory and generated force feedback are the transmitted signals in the communication channel. Specifically, a virtual force feedback generation algorithm is developed in the teleoperation control scheme via the processing tracking errors, which can improve the operators’ assistance and perception during the teleoperation process. The practical experiments with comparative wave variable structures in two different sets are implemented to verify the effectiveness of proposed kinematics model and control scheme.

Evaluation of the feasibility of an error-minimized approach to powered wheelchair skills training using shared control

Background Powered wheelchairs promote participation for people with mobility limitations. For older adults with cognitive impairment, existing training methods may not address learning needs, leading to difficulty with powered wheelchair skills. Error-minimized training, facilitated by shared control technology, may provide learning opportunities more suited to this population. Objective The objective of this study was to evaluate the feasibility of an error-minimized approach to powered wheelchair skills training using shared control in residential care. Feasibility indicators were hypothesized a priori to be feasible for use in a definitive RCT. Methods A 2 × 2 factorial RCT compared an error-minimized powered wheelchair skills training program (Co-pilot) to a control intervention at two doses (6 sessions vs. 12 sessions). Data were collected on the feasibility of study processes (e.g., recruitment), resources (e.g., participant time), management (e.g., technology reliability), and training outcomes (e.g., adverse events, clinical outcomes). Results Twenty-five older adults with cognitive impairment participated in the study. Technical issues were encountered in 14.5% of training sessions. Participants receiving 6 sessions of training adhered better to the treatment than those receiving 12 sessions. All participants learned the skills required for PWC use with minor errors, regardless of the training method or dose. Co-pilot participants and trainers reported feelings of safety and training benefits with the use of shared control technology. Conclusions Individuals with mild to moderate cognitive impairment are able to learn the skills required to drive a powered wheelchair in as few as six training sessions. Further evaluation of the Co-pilot training program is required. IMPLICATIONS FOR REHABILITATION Shared control teleoperation technology may be used to augment learning in older adults with cognitive impairments. Evaluation of the feasibility of use of novel rehabilitation technologies is critical prior to engaging in large-scale clinical research. Individuals with cognitive impairment are able to learn the required skills for operation of a powered wheelchair.

Chattering‐free fixed‐time sliding mode control for bilateral teleoperation under unknown time‐varying delay via disturbance and state observers

AbstractThis paper proposes a new chattering‐free Fixed‐time Observer‐Based Sliding Mode Control (FOBSMC) scheme for synchronization of two bilateral teleoperations under unknown time‐varying delay. The system is subjected to unknown disturbances which are estimated in a fixed time by utilizing the Disturbance Observer‐based SMC scheme. The fixed‐time State Observer‐based SMC is designed to estimate the unmeasured velocity state, while the position state is assumed to be available. Fixed‐time stability method is used to provide the system convergence time independently of initial states. A FOBSMC scheme is first designed for master system of bilateral teleoperation under unknown time‐varying delay to track the environmental torque by the human operating torque in the presence of time delay in the communication channel, as well as estimating the unknown disturbances and unmeasured states. A FOBSMC scheme is designed to synchronize slave system to master system in the presence of time delay in the communication channel, as well as estimating disturbances and states. The stability analysis is performed for closed‐loop system of combined controller‐observer by using Lyapunov theory. The efficacy of the proposed method to fulfill a good transparency and stability under time‐varying communication delays is tested by performing numerical simulation in Simulink/MATLAB.

Teleoperation control scheme for magnetically actuated microrobots with haptic guidance

An external magnetic field can be used in remotely controlling magnetic microrobots, making them promising candidates for diverse biomedical applications, including cell manipulation and therapy. This paper presents a teleoperation scheme to control magnetically actuated microrobots. The system was developed to allow human operators to control the motion for magnetically actuated microrobots and feel their interactions with the environment. The potential applications of the presented system will be in targeted drug delivery, micro-assembly, and biopsy procedures. A haptic interface constituted the core of the teleoperation system. It was used to provide the operator with force feedback to control the microrobots. In particular, virtual interaction forces were computed and transmitted to the human operators to guide them in performing path following tasks. The operating field of the microrobots was haptically rendered to avoid contacts with obstacles. Finally, a basic set of experimental trials were conducted, demonstrating that the average path tracking error was reduced by 67% when haptic feedback was used.

Disturbance observer enhanced variable gain controller for robot teleoperation with motion capture using wearable armbands

Disturbance observer (DOB) based controller performs well in estimating and compensating for perturbation when the external or internal unknown disturbance is slowly time varying. However, to some extent, robot manipulators usually work in complex environment with high-frequency disturbance. Thereby, to enhance tracking performance in a teleoperation system, only traditional DOB technique is insufficient. In this paper, for the purpose of constructing a feasible teleoperation scheme, we develop a novel controller that contains a variable gain scheme to deal with fast-time varying perturbation, whose gain is adjusted linearly according to human surface electromyographic signals collected from Myo wearable armband. In addition, for tracking the motion of operator’s arm, we derive five-joint-angle data of a moving human arm through two groups of quaternions generated from the armbands. Besides, the radial basis function neural networks and the disturbance observer-based control (DOBC) approaches are fused together into the proposed controller to compensate the unknown dynamics uncertainties of the slave robot as well as environmental perturbation. Experiments and simulations are conducted to demonstrated the effectiveness of the proposed strategy.

Robust control design for teleoperation of multiple mobile manipulators under time delays

SummaryThis article addresses a novel multilateral teleoperation control scheme for single‐master multiple slave systems, which can be extended to n masters and n slaves without the loss of generality, where the master is a m degrees of freedom (DOF) manipulator arm and the slaves are m DOF mobile manipulators. The human operator operates the master robot to remotely control the slaves handling a target object. The master position signal is transmitted to the slave side to generate a desired object trajectory as well as the reference mobile base velocity. An adaptive robust controller is designed for the slaves to follow the desired trajectory from the master, which not only provides the excellent trajectory tracking but also optimize the internal force distribution of the object. A null space controller is designed for the mobile platforms of the mobile manipulators to achieve the velocity consensus while achieving the main task of object transportation. The novel control design uses the transmission of the environmental force feedback over the communication channel by the estimated parameters of the environment, which helps retain the stability of the overall system. The environmental force is predicted on the master side based on the estimated environmental parameters. The proposed control design can simultaneously achieve the objectives of stability, synchronization, and optimal internal force distribution. The simulation results of a single‐master and three slaves teleoperation system validate the efficacy of the proposed control algorithm.

Observer-based H∞ Control for Synchronization in Delayed Neural Networks Under Multiple Disturbances

In this paper, the problem on master-slave synchronization is investigated for a class of delayed neural networks (DNNs) subject to multiple variable disturbances, where the synchronization can be widely utilized in many engineering fields such as teleoperation control, secure communication, and so on. Initially, the unknown disturbances are assumed to be generated by two external systems, in which the additional ones are required to belong to the space of ℒ2 [0, +∞). Then based on disturbance-observer-based-control method, H∞ control technique, and composite controller, a sufficient condition on designing the observer gains and controller ones are established to ensure that the overall closed-loop system is asymptotically stable with H∞ control performance. Furthermore, by using matrix transformation technique, the derived condition is converted into the linear matrix inequality (LMI) forms, which can be easily checked and present much less conservatism. Finally, some simulations and comparisons in an example are given to illustrate the effectiveness of our control methods.

Design and Development of Portable LoRa-based Teleoperation Controller

Design and Development of Portable LoRa-based Teleoperation Controller

Route Planning of Teleoperation Mobile Robot based on the Virtual Reality Technology

Mobile teleoperation robot is one of the effective methods to help operators to work in complex environments. However, the time delay by distance is a key factor that restricts its application. To solve this problem, the motion trajectory of the robot is simulated using virtual reality technology and the obtained optimization data are applied to control teleoperation robot.

Combined perception, control, and learning for teleoperation: key technologies, applications, and challenges

Teleoperation provides a promising way for human–robot collaboration in the unknown or unstructured environments to perform a cooperative task. It enables humans to complete a task at a remote side and combines both the human's intelligence and the robots’ capabilities in a collaborative task. Therefore, it is necessary to conduct cross researches in terms of robotics, artificial intelligence, sensors, and mechatronics. This study covers the state-of-the-arts research in terms of perception, control, and learning. In this review, key technologies about information perception mechanism, control algorithms, human–robot interface, and robot learning strategies for teleoperation are introduced. Then, a comprehensive survey is summarised in representative applications in teleoperation, and the existing challenges and potential directions of the development are discussed.

Cover Image

SummaryThe cover image is based on the Research Article High‐gain nonlinear observer‐based impedance control for deformable object cooperative teleoperation with nonlinear contact model by Zhenyu Lu et al., https://doi.org/10.1002/rnc.4880. image

Surgeon-Centered Analysis of Robot-Assisted Needle Driving Under Different Force Feedback Conditions.

Robotic assisted minimally invasive surgery (RAMIS) systems present many advantages to the surgeon and patient over open and standard laparoscopic surgery. However, haptic feedback, which is crucial for the success of many surgical procedures, is still an open challenge in RAMIS. Understanding the way that haptic feedback affects performance and learning can be useful in the development of haptic feedback algorithms and teleoperation control systems. In this study, we examined the performance and learning of inexperienced participants under different haptic feedback conditions in a task of surgical needle driving via a soft homogeneous deformable object—an artificial tissue. We designed an experimental setup to characterize their movement trajectories and the forces that they applied on the artificial tissue. Participants first performed the task in an open condition, with a standard surgical needle holder, followed by teleoperation in one of three feedback conditions: (1) no haptic feedback, (2) haptic feedback based on position exchange, and (3) haptic feedback based on direct recording from a force sensor, and then again with the open needle holder. To quantify the effect of different force feedback conditions on the quality of needle driving, we developed novel metrics that assess the kinematics of needle driving and the tissue interaction forces, and we combined our novel metrics with classical metrics. We analyzed the final teleoperated performance in each condition, the improvement during teleoperation, and the aftereffect of teleoperation on the performance when using the open needle driver. We found that there is no significant difference in the final performance and in the aftereffect between the 3 conditions. Only the two conditions with force feedback presented statistically significant improvement during teleoperation in several of the metrics, but when we compared directly between the improvements in the three different feedback conditions none of the effects reached statistical significance. We discuss possible explanations for the relative similarity in performance. We conclude that we developed several new metrics for the quality of surgical needle driving, but even with these detailed metrics, the advantage of state of the art force feedback methods to tasks that require interaction with homogeneous soft tissue is questionable.

High‐gain nonlinear observer‐based impedance control for deformable object cooperative teleoperation with nonlinear contact model

SummaryUnmeasurable object deformation and local communication time delays between the slave robots influence the manipulation effect for multirobot multioperator teleoperation. In this article, a distributed control method based on high‐gain nonlinear observer, interactive identification, and impedance control is proposed for this problem. First, we use Hunt‐Crossley contact model and deduce the desired synchronizing object state in cooperative teleoperation. Second, an impedance item expressed by the internal position errors is presented to decrease object position tracking errors. For the unmeasurable object deformation, an interactive identification method is proposed for estimating unknown variables. Third, we consider both varying communication time delays and local time delays in the slave side. Two mirror high‐gain nonlinear observers are designed for estimating other slave robots' real‐time state. Finally, we build the system controllers and prove the stability of the closed‐loop system and the boundless of estimating errors using Lyapunov functions. Comparable simulation results executed by the physical system present that the position and internal force tracking errors of the object decrease in the designated cooperative tasks.

A Novel Multilateral Control Design for Delayed Nonlinear Teleoperation System With RBFNN-based Environments

Abstract In this paper, a novel multilateral control design for nonlinear teleoperation system is proposed to improve the capability of multiple robots to coordinate efficiently and precisely in the remote environments under time-varying delays and various uncertainties. The environment is modeled with a general form of force under radial basis function neural network(RBFNN)-based identification and reconstruction to avoid the passivity issue in the traditional teleoperation control and provide the human operators with good sensing of environments. The desired trajectory producers and RBFNN-based sliding mode controllers are designed separately to achieve the good tracking of master/slave robots, and the coordinated distribution algorithm is designed to obtain the control input uS,i for each slave robot. Therefore, the global stability, good transparency performance with both position tracking and force feedback, and good cooperative performance can be achieved simultaneously for delayed nonlinear teleoperation system. The real platform experiment is carried out on a 2-master-2-slave teleoperation system to verify the effectiveness of proposed control design.

Review of Control Methods for Upper Limb Telerehabilitation With Robotic Exoskeletons

Given the escalating unmet demand for physical rehabilitation due to the growing global aging population and the effects of the coronavirus COVID-19 including increased incidents of stroke, hospital bed shortages, and clinics closures, robotic telerehabilitation is an emerging, timely, and crucial technology. Rehabilitating the upper limbs of affected patients is of upmost importance for restoring physical function and lighten the societal burden due to disabilities. So far, the majority of the research in robotic telerehabilitation for upper limbs has been performed with end-effector-type assistive robots; however, the use of robotic exoskeletons has significant and distinctive benefits. Although there are surveys written about control methods for upper limb robotic exoskeletons and other surveys written about bilateral teleoperation control methods, there are no surveys written specifically about telerehabilitation control methods for upper limbs using robotic exoskeletons. As a result, this article reviews the state-of-the-art control strategies including various advanced linear and nonlinear control approaches for upper limb rehabilitation robotic exoskeletons, bilateral teleoperation, and several state-of-the-art telerehabilitation applications with upper limb robotic exoskeletons. The benefits, drawbacks, challenges, and future directions of existing methodologies are extensively discussed. This article offers a comprehensive overview and insight for new researchers in the area of telerehabilitation robotic exoskeletons.