Teleoperation Control Research Articles

Tele-Impedance control of a virtual avatar based on EMG and M-IMU sensors: a proof-of-concept

The broad spread of cooperative robots into many application domains has resulted in a demand for intuitive and effective solutions for teleoperated control. A relevant role in teleoperation has been assumed by impedance controllers, that allow the increase of stability and accuracy during interaction. This paper aims to test a teleoperation method based on an impedance controller, namely tele-impedance control, that is usable in unstructured environments since it relies only on wearable sensors. The proposed solution maps the joint stiffness and position of the human user, computed through six EMG and two M-IMU sensors, into the remote system to be teleoperated. We developed a 2-DoFs virtual task involving virtual physical interactions to compare the performance of our solution with the one of a traditional position-based controller. The study has been conducted on five healthy participants, who experienced both controllers in two different sessions. The tele-impedance approach has proved to be less physically demanding and more intuitive than the position-based one. Experimental data also allow us to investigate the strategy employed by the volunteers in the case of remote interactions, while using the two controllers. Of note, even though with the position controller the variation of subject impedance has no effect on the virtual arm, participants still tend to regulate both impedance and position of their own arm.

A novel affordable user interface for robotic surgery training: design, development and usability study.

The use of robotic systems in the surgical domain has become groundbreaking for patients and surgeons in the last decades. While the annual number of robotic surgical procedures continues to increase rapidly, it is essential to provide the surgeon with innovative training courses along with the standard specialization path. To this end, simulators play a fundamental role. Currently, the high cost of the leading VR simulators limits their accessibility to educational institutions. The challenge lies in balancing high-fidelity simulation with cost-effectiveness; however, few cost-effective options exist for robotic surgery training. This paper proposes the design, development and user-centered usability study of an affordable user interface to control a surgical robot simulator. It consists of a cart equipped with two haptic interfaces, a VR visor and two pedals. The simulations were created using Unity, which offers versatility for expanding the simulator to more complex scenes. An intuitive teleoperation control of the simulated robotic instruments is achieved through a high-level control strategy. Its affordability and resemblance to real surgeon consoles make it ideal for implementing robotic surgery training programs in medical schools, enhancing accessibility to a broader audience. This is demonstrated by the results of an usability study involving expert surgeons who use surgical robots regularly, expert surgeons without robotic surgery experience, and a control group. The results of the study, which was based on a traditional Peg-board exercise and Camera Control task, demonstrate the simulator's high usability and intuitive control across diverse user groups, including those with limited experience. This offers evidence that this affordable system is a promising solution for expanding robotic surgery training.

Distributed Observer-Based Prescribed Performance Control for Multi-Robot Deformable Object Cooperative Teleoperation

Distributed Observer-Based Prescribed Performance Control for Multi-Robot Deformable Object Cooperative Teleoperation

Teleoperation Control of Autonomous Underwater Vehicle Toward Human on the Loop: Needs, Analyses, and Solutions

Teleoperation Control of Autonomous Underwater Vehicle Toward Human on the Loop: Needs, Analyses, and Solutions

Adaptive Impedance Control for Teleoperation with Event-Triggered Controller

Uncertainty in the master–slave model is one of the primary factors affecting the transparency of teleoperation systems, and congestion in the master–slave communication network also greatly influences the performance of the teleoperation system. This paper proposes a combined framework of adaptive and impedance control to address the uncertainty in the master–slave model and achieve smooth operation at the slave end. Building upon this linear model, an event-triggered mechanism is designed using Lyapunov functions, with dynamic online adjustment of the triggering threshold parameters. Following the completion of the aforementioned research, control objectives are established to validate the performance of the teleoperation control system proposed in this paper. Finally, simulation verification is conducted in the Matlab/Simulink environment.

Online Bilateral Predictive Control for Time-Delayed Teleoperation of Snake-like Robots

Controlling the teleoperation of snake-like robots is challenging due to complex nonlinear dynamics and communication delays. This research proposes an online bilateral predictive control architecture to address these issues. This control structure is established by predicting environment force and the user’s future motion. The former uses a model-mediated approach by creating a virtual environment on the master side and the latter adopts an artificial neural network (ANN) for online operator’s motion prediction. The slave controller utilizes transmitted data from ANN to generate required backbone lengths, which are then transformed into the slave's local bending and torsional degrees of freedom through the inverse kinematics of the robot. Motion prediction is examined in two scenarios: when the ANN predicts the trained motions, and when it predicts a different motion. Simulation studies demonstrate that the proposed online bilateral predictive teleoperation structure successfully achieves real-time position synchronization and force feedback, by effectively bypassing communication delays.

Cooperative vs. Teleoperation Control of the Steady Hand Eye Robot with Adaptive Sclera Force Control: A Comparative Study.

A surgeon's physiological hand tremor can significantly impact the outcome of delicate and precise retinal surgery, such as retinal vein cannulation (RVC) and epiretinal membrane peeling. Robot-assisted eye surgery technology provides ophthalmologists with advanced capabilities such as hand tremor cancellation, hand motion scaling, and safety constraints that enable them to perform these otherwise challenging and high-risk surgeries with high precision and safety. Steady-Hand Eye Robot (SHER) with cooperative control mode can filter out surgeon's hand tremor, yet another important safety feature, that is, minimizing the contact force between the surgical instrument and sclera surface for avoiding tissue damage cannot be met in this control mode. Also, other capabilities, such as hand motion scaling and haptic feedback, require a teleoperation control framework. In this work, for the first time, we implemented a teleoperation control mode incorporated with an adaptive sclera force control algorithm using a PHANTOM Omni haptic device and a force-sensing surgical instrument equipped with Fiber Bragg Grating (FBG) sensors attached to the SHER 2.1 end-effector. This adaptive sclera force control algorithm allows the robot to dynamically minimize the tool-sclera contact force. Moreover, for the first time, we compared the performance of the proposed adaptive teleoperation mode with the cooperative mode by conducting a vessel-following experiment inside an eye phantom under a microscope.

Imperceptible liquid metal based tattoo for Human-Machine interface on hairy skin

Biopotential signals recorded by skin surface electrodes reflect the state of the body and corresponding organ functions. These signals are commonly utilized in clinical diagnosis, health monitoring, sports training, and human–computer interaction. However, the utilization of conventional electrodes on hairy skin causes a number of issues, including suboptimal skin compliance, obstructed sweat secretion and impaired tactile perception, making it difficult to obtain long-term and high-fidelity biopotential recordings. Here, we propose an imperceptible spray-on electrode tattoo (SE-tattoo) with sweat-permeability, high adhesiveness, high conductivity, and fast curing ability (<20 s). The water-based conductive ink is sprayed directly on the hairy skin to form a conformable conductive tattoo that neither affects nor is affected by the skin hair. We demonstrate it for heart health monitoring, sports training, electro-tactile display and close-loop teleoperation control. The SE-tattoo offers a reliable and versatile solution to biopotential recordings and serves as a promising platform for bidirectional human–machine interaction.One-Sentence Summary: Imperceptible sweat-permeable SE-tattoo enables high-fidelity biopotential recordings for bidirectional human–machine interaction.

Development of a flexible endoscopic robot with autonomous tracking control ability using machine vision and deep learning

Abstract. A flexible endoscopic robot is designed to solve the problem that it is difficult for auxiliary doctors to maintain a stable visual field in traditional endoscopic surgery. Based on geometric derivation, a motion control method under the constraint of the remote center motion (RCM) of the robot system is established, and a set of circular trajectories are planned for it. The RCM error of the robot during operation and the actual trajectory of the robot end in three-dimensional space are obtained through the motion capture system. The end of the robot is controlled by the heterogeneous primary–secondary teleoperation control algorithm based on position increments. Finally, the RTMDet deep learning object detection algorithm was selected to identify and locate surgical instruments through comparative experiments, and the autonomous tracking control was completed based on visual guidance. In the process of autonomous tracking, the RCM error was less than 1 mm, which met the actual surgical requirements.

GestureMoRo: an algorithm for autonomous mobile robot teleoperation based on gesture recognition

Gestures are a common way people communicate. Gesture-based teleoperation control systems tend to be simple to operate and suitable for most people’s daily use. This paper employed a LeapMotion sensor to develop a mobile robot control system based on gesture recognition, which mainly established connections through a client/server structure. The principles of gesture recognition in the system were studied and the relevant self-investigated algorithms—GestureMoRo, for the association between gestures and mobile robots were designed. Moreover, in order to avoid the unstably fluctuated movement of the mobile robot caused by palm shaking, the Gaussian filter algorithm was used to smooth and denoise the collected gesture data, which effectively improved the robustness and stability of the mobile robot’s locomotion. Finally, the teleoperation control strategy of the gesture to the WATER2 mobile robot was realized, and the effectiveness and practicability of the designed system were verified through multiple experiments.

A Hybrid Method Combining Data-Driven and Model-Based Algorithms for External Force-Sensing and Haptics Control of Cable-Pulley-Driven Surgical Robotic Manipulator

Abstract The implementation of the high-precision tracking control and external force-sensing ability of a manipulator is important for achieving refined surgical robot operation. In this paper, a hybrid method based on data-driven and model-based algorithms is proposed for the manipulator of a cable-pulley-driven surgical robot. This method integrates an artificial neural network and a dynamic model rotation angle estimation, and a full closed-loop control architecture is further constructed. The algorithm compensates for the hysteresis of the joint angle and effectively improves the tracking control precision. Based on the architecture, the external force estimator (EFE) using a joint torque disturbance observer and the force interaction teleoperation control strategy using a direct force feedback framework (DFF) are implemented. In the force loading experiment, it was shown that the EFE performs well for static and dynamic force estimation, and the teleoperated haptic control experiment showed that the DFF-EFE-based system has a high position-tracking accuracy with real-time external force-sensing ability.

A Framework for the Evaluation of Human Machine Interfaces of Robot-Assisted Colonoscopy.

This article presents a framework to compare and evaluate different HMIs for robotic colonoscopy, with the objective of identifying the optimal HMI that minimises the clinician's effort and maximises the clinical outcomes. The framework comprises a 1) a virtual simulator (clinically validated), 2) wearable sensors measuring the cognitive load, 3) a data collection unit of metrics correlated to the clinical performance, and 4) questionnaires exploring the users' impressions and perceived stress. The framework was tested with 42 clinicians investigating the optimal device for tele-operated control of robotic colonoscopes. Two control devices were selected and compared: a haptic serial-kinematic device and a standard videogame joypad. The haptic device was preferred by the endoscopists, but the joypad enabled better clinical performance and reduced cognitive and physical load. The framework can be used to evaluate different aspects of a HMI, both hardware and software, and determine the optimal HMI that can reduce the burden on clinicians while improving the clinical outcome. The findings of this study, and of future studies performed with this framework, can inform the design and development of HMIs for intraluminal robots, leading to improved clinical performance, reduced physical and mental stress for clinicians, and ultimately better patient outcomes.

Research on vision control methods based on multi-sensor fusion

This paper focuses on the application scenarios of electric power live line working robots, addressing tasks such as live wire stripping and live wire connection/disconnection. It utilizes Mixed Reality (MR) technology for rapid environment modeling and visual signal acquisition. Through the research on robot contact force sensing, real-time force feedback, and obstacle avoidance through the fusion of robot motion paths and environmental data, it establishes a novel electric power live line working robot system centered around dexterous dual arms at the operation end, force feedback teleoperation controllers, multi-sensor fusion acquisition systems, and robot motion planning and control technology. Through prototyping, a demonstration application is realized, showcasing a semi-autonomous electric power live line working robot system that employs remote operation combined with sensor-based MR for task execution. This teleoperation and Mixed Reality-based semi-autonomous electrified operation approach is poised to play a significant role in advancing and demonstrating the use of robots in the field of electrical grid operations.

Lie Group-Based User Motion Refinement Control for Teleoperation of a Constrained Robot Arm

Lie Group-Based User Motion Refinement Control for Teleoperation of a Constrained Robot Arm

Hierarchical Prescribed-Performance Nonlinear Control for Teleoperation of Networked Slave Robots with Coupling Constraints

Hierarchical Prescribed-Performance Nonlinear Control for Teleoperation of Networked Slave Robots with Coupling Constraints

REAL-TIME TELEOPERATION CONTROL SYSTEM FOR AUTONOMOUS VEHICLE

In recent years, there has been a significant surge of interest among autonomous vehicle (AV) companies and researchers in integrating teleoperation as a fallback method. However, it’s crucial to acknowledge the limited focus on real-time teleoperation control, particularly in addressing issues related to signal latency and uncertainties in the vehicle’s nonlinear dynamics. To address this gap, we’ve developed a teleoperation system that empowers operators to control the vehicle remotely directly. This system utilizes the wave variable approach for communication to mitigate signal latency effects. It utilizes Radial Basis Function Networks (RBFN) to handle uncertainties in the vehicle’s nonlinear dynamics. We conducted simulations to assess the effectiveness of our approach, revealing enhanced capability for the AV to adhere to operators’ desired trajectories.

Mixed-Reality-Based Teleoperation Grasping Control

Abstract Traditionally, teleoperation means that the system sends a series of signal commands from the master while the slave manipulator receives and realizes the desired control operations. For the purpose of implementing more dexterous and complex tasks, we propose a novel framework with dual-hand master teleoperation systems under time-varied delays. In this paper, we emphasize studying the bilateral grasping teleoperation control, as the time delay causes a communication outage. Combining a wave-variable structure with a four-channel framework, an event-trigger-based bilateral sliding mode teleoperation control and an adaptive neural network are designed to effectively achieve master-slave trajectory tracking. In the virtual 3D environment, we created a mixed-reality interface based on dual-hand master teleoperation control that effectively responded to the two Omni manipulators' position transformation of the virtual manipulator. The time delay between the real slave force feedback and the virtual interface is addressed by designed event-trigger-based control in order to efficiently reduce the impact of time communication outage. The system's stability is analyzed and robot experiments are performed. From the experimental results, the telepresence platform innovatively applied virtual force feedback to reveal the soft target grasping and to accurately estimate the interactive force, enabling sensorless force feedback control.

Performance comparison of bilateral teleoperation control of direct control, PID control, and fuzzy logic control for water tank system

Bilateral teleoperation has attracted significant research and application interests in a wide range of areas. The first and main use for bilateral teleoperation was to handle the dangerous and remote distance tasks such as space exploration and nuclear materials manipulation. In the past few years, bilateral teleoperation has found its way into other applications as a result of the development of control technologies and the latest breakthrough in artificial intelligence and machine learning. In this paper, the goal is to model bilateral teleoperation for direct control, PID control, and Fuzzy Logic control for the water tank system. The goal is to create a simulation for master-slave communication where the time delay is minimized to the optimal and accepted values. The experimental results obtained from the simulation show a fairly high accuracy in terms of all three ways of control modes, which highlights the effectiveness of the proposed system in the paper.

Assisted teleoperation control of robotic endoscope with visual feedback for nasotracheal intubation

Endoscopic operation is one of the most difficult parts of nasotracheal intubation (NTI), a surgical procedure used to secure the airway of a patient. Due to the requirement of eye–hand coordination, manual operation of a flexible endoscope is challenging, even for an experienced surgeon. To enhance intubation efficiency, this paper developed a master–slave robotic nasotracheal intubation system (RNIS) for endoscopic operation. Movements in three degrees of freedom of the endoscope are controlled by the RNIS. An assisted teleoperation control strategy is designed to assist the operator in remotely controlling the pose, i.e., position and orientation, of the endoscope tip via a joystick. To ensure the efficiency of intubation, visual feedback assistance is hereby proposed, which fine-tunes the orientation of the endoscope tip as needed. The proposed system and methods are experimentally validated on a motion simulator and a phantom. The results demonstrated that the master–slave RNIS can successfully insert the flexible endoscope into the trachea of a phantom through the nasal cavity.

Use of a predictive display for the provision of flight safety in space missions requiring high-precision manual control

Theoretical and experimental studies were performed with the goal of determining the best kind of information presented on a predictive display to provide the highest accuracy in space mission execution, taking into account the requirement for reduced propellant consumption. The following space missions were studied: docking performed in manual and teleoperator control modes and lunar landing. By using a theoretical technique, algorithms for the flight path marker in manual and teleoperator control modes were developed. For the landing mission, a technique for determining the program trajectory and its calculation in real time was proposed. The experiments performed using a ground-based simulator demonstrated the high effectiveness of a predictive display and allowed to select the best variant of information to be presented on the display.