Teleoperation System Research Articles

Exploratory study on time-delayed excavator teleoperation in virtual lunar construction simulation: Task performance and operator behavior

Building sustainable habitats on the moon has been planned for decades. However, applying fully automated construction systems is still challenging in altered environments. Teleoperation, which is the remote control of the machine, can serve as an intermediate phase before achieving fully autonomous systems. Since the teleoperation between operators on the earth-ground and robots on the lunar surface introduces inevitable communication time delays under a deep space network system, it is important to understand its impact on task performance and operator behaviors in teleoperated construction tasks. This paper develops a simulated lunar environment for excavator teleoperation systems in virtual reality to examine task performance and operator behaviors in time delay conditions. The outcomes indicate that time delays significantly degrade task performance, and the operators modify their control strategies to cope with the time delay conditions. The findings will contribute to understanding human behaviors in time-delayed teleoperation of lunar construction tasks.

Brain functional connectivity under teleoperation latency: a fNIRS study

IntroductionLong-distance robot teleoperation faces high latencies that pose cognitive challenges to human operators. Latency between command, execution, and feedback in teleoperation can impair performance and affect operators’ mental state. The neural underpinnings of these effects are not well understood.MethodsThis study aims to understand the cognitive impact of latency in teleoperation and the related mitigation methods, using functional Near-Infrared Spectroscopy (fNIRS) to analyze functional connectivity. A human subject experiment (n = 41) of a simulated remote robot manipulation task was performed. Three conditions were tested: no latency, with visual and haptic latency, with visual latency and no haptic latency. fNIRS and performance data were recorded and analyzed.ResultsThe presence of latency in teleoperation significantly increased functional connectivity within and between prefrontal and motor cortexes. Maintaining visual latency while providing real-time haptic feedback reduced the average functional connectivity in all cortical networks and showed a significantly different connectivity ratio within prefrontal and motor cortical networks. The performance results showed the worst performance in the all-delayed condition and best performance in no latency condition, which echoes the neural activity patterns.ConclusionThe study provides neurological evidence that latency in teleoperation increases cognitive load, anxiety, and challenges in motion planning and control. Real-time haptic feedback, however, positively influences neural pathways related to cognition, decision-making, and sensorimotor processes. This research can inform the design of ergonomic teleoperation systems that mitigate the effects of latency.

A Robotic Teleoperation System with Integrated Augmented Reality and Digital Twin Technologies for Disassembling End-of-Life Batteries

Disassembly is a key step in remanufacturing, especially for end-of-life (EoL) products such as electric vehicle (EV) batteries, which are challenging to dismantle due to uncertainties in their condition and potential risks of fire, fumes, explosions, and electrical shock. To address these challenges, this paper presents a robotic teleoperation system that leverages augmented reality (AR) and digital twin (DT) technologies to enable a human operator to work away from the danger zone. By integrating AR and DTs, the system not only provides a real-time visual representation of the robot’s status but also enables remote control via gesture recognition. A bidirectional communication framework established within the system synchronises the virtual robot with its physical counterpart in an AR environment, which enhances the operator’s understanding of both the robot and task statuses. In the event of anomalies, the operator can interact with the virtual robot through intuitive gestures based on information displayed on the AR interface, thereby improving decision-making efficiency and operational safety. The application of this system is demonstrated through a case study involving the disassembly of a busbar from an EoL EV battery. Furthermore, the performance of the system in terms of task completion time and operator workload was evaluated and compared with that of AR-based control methods without informational cues and ‘smartpad’ controls. The findings indicate that the proposed system reduces operation time and enhances user experience, delivering its broad application potential in complex industrial settings.

Optimisation of synchronisation control parameters for enhanced bilateral teleoperation system

Abstract Bilateral teleoperation systems encounter challenges in achieving synchronisation between master and slave robots due to communication time delays. This paper addresses the instability caused by these delays and proposes a solution through advanced control algorithms. Nonlinear optimisation algorithms might only sometimes deliver solutions in the allotted time, particularly when handling complicated, high-dimensional issues or when optimisation iterations are extensive. The study first develops a comprehensive mathematical model encompassing the dynamics and communication intricacies of both master and slave sides in teleoperation. By recognising the limitations of existing proportional-derivative controllers in compensating for communication errors, a sequential quadratic programming-proportional-integral-derivative (SQP-PID) controller is introduced. This controller accumulates and rectifies synchronisation delay errors, ensuring precise control without steady-state deviations. The proposed SQP-PID controller stands out for its ability to handle steady-state errors effectively, offering swift response and maintaining stability. Leveraging the SQP optimisation algorithm, it intelligently tunes the parameters, minimising synchronisation errors. The approach capitalises on the simplicity, performance, and robustness of the SQP-PID controller, providing a promising avenue for enhancing bilateral teleoperation systems’ accuracy and stability, maintaining initial discrepancy with a best fitness value of 0.98 % in varied operating conditions.

A virtual reality-based immersive teleoperation system for remote human-robot collaborative manufacturing

In recent years, the cutting-edge technologies in smart manufacturing have presented promising opportunities for the utilization of human-robot collaborative teleoperation in personalized manufacturing tasks. To effectively leverage the creative capabilities of humans while benefiting from the efficiency and stability of robots, the provision of an intuitive teleoperation interface assumes paramount importance. However, current teleoperation systems still face limitations in terms of intuitive operability. In this study, we present a virtual reality-based teleoperation system that offers operators a more intuitive interaction platform for robot control, thereby facilitating personalized manufacturing processes. The overall system framework design, as well as the main components are elaborated in detail. Furthermore, an evaluative case study based on the battery disassembly task is conducted to assess the performance of the proposed system. The results demonstrate that the proposed teleoperation system exhibits improved intuitiveness.

Robust non-monotonic Lyapunov based stability and stabilization methods for continuous-time systems: Applied on bilateral teleoperation system

This study introduces a Non-monotonic Lyapunov (NML) framework aimed at stability evaluation and controller design for continuous-time systems, particularly under conditions of uncertainty. Conventional Lyapunov techniques often exhibit a conservative nature, particularly in the context of uncertain systems, which necessitates the development of less conservative alternatives like NML. The NML methodology distinguishes itself by not imposing strict monotonicity requirements for demonstrating the decrease of a Lyapunov functional. Consequently, this paper derives new stability and stabilization criteria framed as matrix inequalities applicable to a specific class of uncertain systems. The practical applicability of the introduced approach is illustrated through controller design for uncertain systems, exemplified by a nonlinear bilateral teleoperation model. Assorted demonstrative examples and simulation outcomes support the findings, underscoring the NML approach’s efficaciousness.

Adaptive control of bilateral teleoperators with convergent parameter estimation and guaranteed compliance

Although adaptive control has been studied for teleoperation systems in the past decades, the convergence of parameter estimation is generally difficult to claim in the existing methods, and thus only specific scenarios (e.g., free motion) and partial operation performance (e.g., synchronization) have been addressed. In this paper, we found that the convergence of parameter estimation is crucial for bilateral teleoperation systems to retain the guaranteed operation performances, including compliance in the tracking motion and transparency in the contact motion. Accordingly, this paper suggests an alternative adaptive control for bilateral teleoperation systems, where a new adaptive law with the estimation errors as leakage terms is developed to obtain finite-time convergence of the parameter estimation. This convergent property allows to compensate for the undesired dynamics (e.g., gravity) in the closed-loop system so as to achieve ideal compliance and transparency. The control system’s stability and performances under constant time delays are analyzed for three scenarios: free motion, tracking motion, and contact motion. The quantitative relationship between the joint velocities and the human torques in the tracking motion is also studied, and the transparency is addressed in the contact motion. Simulations and experiments on a Baxter robot all showcase the effectiveness and superiority of the developed approach.

Using a Guidance Virtual Fixture on a Soft Robot to Improve Ureteroscopy Procedures in a Phantom

Manipulating a flexible ureteroscope is difficult, due to its bendable body and hand–eye coordination problems, especially when exploring the lower pole of the kidney. Though robotic interventions have been adopted in various clinical scenarios, they are rarely used in ureteroscopy. This study proposes a teleoperation system consists of a soft robotic endoscope together with a Guidance Virtual Fixture (GVF) to help users explore the kidney’s lower pole. The soft robotic arm was a cable-driven, 3D-printed design with a helicoid structure. GVF was dynamically constructed using video streams from an endoscopic camera. With a haptic controller, GVF can provide haptic feedback to guide the users in following a trajectory. In the user study, participants were asked to follow trajectories when the soft robotic arm was in a retroflex posture. The results suggest that the GVF can reduce errors in the trajectory tracking tasks when the users receive the proper training and gain more experience. Based on the NASA Task Load Index questionnaires, most participants preferred having the GVF when manipulating the robotic arm. In conclusion, the results demonstrate the benefits and potential of using a robotic arm with a GVF. More research is needed to investigate the effectiveness of the GVFs and the robotic endoscope in ureteroscopic procedures.

Fixed-time control of teleoperation systems based on adaptive event-triggered communication and force estimators

Abstract A fixed-time control strategy based on adaptive event-triggered communication and force estimators is proposed for a class of teleoperation systems with time-varying delays and limited bandwidth. Two force estimators are designed to estimate the operator force and environment force instead of force sensors. With the position, velocity, force estimate signals, and triggering error, an adaptive event-triggered scheme is designed, which automatically adjusts the triggering thresholds to reduce the access frequency of the communication network. With the state information transmitted at the moment of event triggering while considering the time-varying delays, a fixed-time sliding mode controller is designed to achieve the position and force tracking. The stability of the system and the convergence of tracking error within a fixed time are mathematically proved. Experimental results indicate that the control strategy can significantly reduce the information transmission, enhance the bandwidth utilization, and ensure the convergence of tracking error within a fixed time for teleoperation systems.

Virtual Teleoperation System for Mobile Manipulator Robots Focused on Object Transport and Manipulation

This work describes the development of a tool for the teleoperation of robots. The tool is developed in a virtual environment using the Unity graphics engine. For the development of the application, a kinematic model and a dynamic model of a mobile manipulator are used. The mobile manipulator robot consists of an omnidirectional platform and an anthropomorphic robotic arm with 4 degrees of freedom (4DOF). The model is essential to emulate the movements of the robot and to facilitate the immersion in the virtual environment. In addition, the control algorithms are established and developed in MATLAB 2020 software, which improves the acquisition of knowledge to teleoperate robots and execute tasks of manipulation and transport of objects. This methodology offers a cheaper and safer alternative to real physical systems, as it reduces both the costs and risks associated with using a real robot for training.

Sense of Embodiment Supports Motor Learning and Adaptation in Tele-robotics

In this paper, we transition from the theoretical and experimental groundwork of manipulating and measuring Sense of Embodiment (SoE) to addressing a fundamental question: What is the purpose of optimizing the SoE in a teleoperation system? This exploration centers on investigating the potential positive effects of SoE on motor adaptation, the acceleration of motor learning, and the potential enhancement of task performance. The paper delves into this investigation by focusing on two critical research questions: 1) what is the effect of SoE on task performance in a perceptual-motor task? 2) What is the effect of SoE on the asymptote of the learning curve in a perceptual-motor task? Drawing insights from the existing literature, the hypothesis emerges that enhancing SoE yields positive effects, not only on task performance (H1) but also on the overall embodiment experience (H2). An additional layer of exploration is introduced through an exploratory research question: Are these results consistent across diverse scenarios and tasks? The study design encompasses two distinct user studies, each set in different applications and featuring various avatars, yet all anchored in similar tasks, specifically a modified peg-in-hole task: 1) in the first experiment, participants operated a robotic arm with a human-like hand as end-effector, and they were required to perform a classic peg-in-hole task; 2) in user study 2, the task is transformed into a variation that we called “peg-on-button”, wherein participants use a robotic arm with a gripper as the end-effector to press a lit button. In both studies, a consistent pattern emerges: a setup that fosters embodiment has a positive impact on motor learning and adaptation, resulting in improved task performance. A supportive setup also reduces the perception of the surrogate as a mere mediator between the operator and the remote environment, especially when contrasted with a setup that suppresses embodiment. The positive effects on motor learning and task performance advocate for the incorporation of embodiment-supportive designs in teleoperational setups. However, the nuanced relationship between SoE and long-term task performance prompts a call for further exploration and consideration of various factors influencing teleoperation outcomes across diverse scenarios and tasks.

Enabling intuitive and effective micromanipulation: A wearable exoskeleton-integrated macro-to-micro teleoperation system with a 3D electrothermal microgripper

In this article, we present a novel teleoperation system for dexterous micromanipulation with a 3D three-fingered electrothermal microgripper. A lightweight wearable exoskeleton hand is designed and employed as the primary device, integrating rotational potentiometers as angle sensors, which are embedded in a closed-loop kinematic chain for detecting flexion/extension and adduction/abduction angles of motion. The measured angles are subsequently translated into exoskeleton hand-fingertip positions utilized as the primary inputs. A 3D electrothermal microgripper based tele-micro manipulation system is realized. The displacement of the exoskeleton fingertips is harnessed to govern the actions of the microgripper via an effective position incremental control method. Furthermore, the system's capabilities are exemplified through intricate micromanipulations performed on soft zebrafish embryos. The micromanipulations encompass gripping and rotational maneuvers. The outcomes of empirical experimentation clearly demonstrate the suitability of the macro-micro teleoperation system, which incorporates an exoskeleton hand for controlling a microgripper in 3D micromanipulation. The system improves operator comfort and maneuvering efficiency. Even for untrained users, the tasks can be accomplished with ease in an intuitive and effective way.

Adaptive event-triggered security control for networked teleoperation systems.

An adaptive event-triggered security control method is proposed for networked robotic teleoperation systems subject to time-varying delays and false data injection (FDI) attacks. An event-triggered scheme is designed via the position and velocity signals of the master and slave robots, where the triggering thresholds can change adaptively with the system states. The position and velocity triggered signals and the feedback error between the operator and the environment forces are utilized to detect whether the transmitted signals suffered from FDI attacks. Then, a switching controller is designed, which is able to adopt the corresponding control strategy according to the corresponding detection result. The stability of the system and the convergence of the tracking errors are proved through Lyapunov functions, and the proposed method is validated through practical experiments. The proposed method is able to adaptively adjust the triggering frequency and effectively reduce the transmitted data, thus saving the network resources. Meanwhile, it can ensure the stability of networked teleoperation systems under time-varying delays and FDI attacks, as well as the position and force tracking performance.

Dynamic Bandwidth Allocation for Collaborative Multi-Robot Systems Based on Task Execution Measures

Multi-robot systems (MRSs) is a growing field of research that focuses on the collaboration of multiple robots to achieve a common global objective. Managing these systems poses several challenges, including coordination, task allocation, and communication. Among these challenges, a major area of focus is devising an effective communication scheme that ensures robots’ cooperation and adapts to varying conditions during task execution. In this paper, we develop a novel communication management framework tailored for MRSs, specifically addressing dynamic bandwidth distribution in networked teleoperated robotic systems. The algorithm is combined with semi-autonomous formation control based on the Artificial Potential Fields (APF) algorithm, which allows each individual robot to avoid local obstacles autonomously and tries to maintain a desired formation with its neighbors, while the operator is in charge of high-level control only. Common Dynamic Bandwidth Allocation (DBA) algorithms allocate bandwidth to different units based on network conditions and requirements. On the other hand, our proposed DBA scheme dynamically distributes the available bandwidth on communication streams based on factors related to task execution and system performance. In specific, bandwidth is allocated in a way that adapts to changes occurring in the system’s environment and its internal state, including the effect of the autonomous action taken by the path planner on the MRS and the performance of the controller of each individual robot. By addressing the limitations of existing approaches through shaping the communication behavior of the MRS based on performance measures, our proposed algorithm offers a promising solution for improving the performance and efficiency of MRSs. The proposed scheme is tested through simulations on a group of six unmanned aerial vehicles (UAVs) in the Robot Operating System (ROS)-Gazebo simulation environment. The obtained results show the scheme’s capability for enhancing the robotic system’s performance while significantly reducing bandwidth consumption. Experimental testing on two mobile robots further demonstrates the effectiveness of the proposed scheme.

Adaptive Iterative Learning Tracking Control for Nonlinear Teleoperators with Input Saturation

Addressing input saturation, external disturbances, and uncertain system parameters, this paper investigates the position tracking control problem for bilateral teleoperation systems with a time delay communication channel. Based on a composite energy function, we propose an adaptive iterative learning control (AILC) method to achieve the objective of position tracking under the alignment condition. This extends the existing research on the control of nonlinear teleoperation systems with time delay. The saturation constraint property of the Softsign function ensures that no state of the system exceeds its constraints. The controller learns to simultaneously deal with the uncertainty of system parameters online, reject external disturbances, and eliminate positional errors along the time and iteration axes. All signals in the system for any constant time delay are proved to be bounded. Ultimately, the performance of the proposed controller is further verified through numerical simulations.

Human Reliability Assessment of Space Teleoperation Based on ISM-BN

Space teleoperation systems, as complex giant systems, feature performance-influencing factors that are interrelated. Accurately describing the dependence between these factors is crucial for constructing a human factor reliability assessment (HRA) model. Moreover, data scarcity has consistently been a challenge in space HRA. There are primarily two types of data in this domain: expert judgment data and empirical data (simulation data, actual reports), each with complementary effects. The expert judgment data, although subjective, are readily accessible, while empirical data provide robust objectivity but are difficult to obtain. Addressing these challenges, this paper constructs an HRA model for space teleoperation that combines Interpretive Structural Modeling (ISM) with a two-stage Bayesian update method. This model reflects the dependencies between factors and accommodates multisource data (expert judgment and experimental data). With more empirical data, the model can be continuously updated and refined to yield increasingly accurate evaluations of human error probability (HEP). The validity of the model was verified through the analysis of 52 space incidents using the N-K model. The study provides a methodological foundation for HRA in other space missions.

Internet of robotic things with a local LoRa network for teleoperation of an agricultural mobile robot using a digital shadow

In unstructured agricultural fields where autonomous navigation is challenging and demands additional safety, the operator’s experience and knowledge are essential for supervising operations and making decisions beyond the robot’s autonomous capabilities. Local networks with long-range wireless communication combined with digital twin concepts are promising solutions that can be used for robot teleoperation. The purpose of this study was to demonstrate the feasibility of supervising a mobile robot inside berry orchards using a digital shadow from a long-range distance (between 300 and 3000 m), with the primary objective of assisting the robot in navigating in complex situations such as row-end turning. This involved creating a virtual representation of the robot that mirrors its state and actions, allowing the remote operator to monitor and guide the robot effectively. The system comprised a GPS-based navigation controller with collision avoidance sensors, two sets of LoRa transmitters and repeaters, a simulation environment with a digital shadow of the robot, and a graphical user interface for the remote operator. Information about the digital shadow’s state, including location, orientation, and distances to obstacles, was received as a message by the LoRa gateway and was used to update the path for the actual robot that interfaced with the Robot Operating System (ROS). The main research hypothesis aimed to test the quality of the LoRa communication link between the robot and the operator, as well as the robustness of the robot’s control system, with an emphasis on the architecture, communication link, and situation awareness creation. Preliminary results showed that depending on the environment, the average packet loss was 12% at distances of approximately 2300 m. Our results highlight some of the core technical challenges that need to be addressed for an effective teleoperation system, including latency, stability, and the limited range of wireless communication. Future works involves evaluating the performance and reliability of the proposed method under different field conditions and scenarios, as well as considering the use of the 5G network for a significant improvement in data transmission speed, navigation efficiency, and visual feedback. Upon successful implementation, this study has the potential to enhance the efficiency and safety of robot navigation, providing a practical solution for remote supervision in challenging environments.

Novel adaptive robust L1‐based controllers for teleoperation systems with uncertainties and time delays

Novel adaptive robust L₁‐based controllers for teleoperation systems with uncertainties and time delays

Master–slaves synchronization of teleoperation systems with the Try‐Once‐Discard protocol under event‐triggered communication

AbstractIn this paper, a class of single‐master‐multiple‐slaves (SMMS) bilateral teleoperation systems with the limitation of communication bandwidth is investigated. The signals are exchanged between the master and the slaves through delayed communication. Besides, to avoid data collisions, the Try‐Once‐Discard (TOD) scheduling protocol is utilized to decide which slave gets the network access at each triggered instant. A control algorithm with the event‐triggered mechanism is proposed to ensure the master–slaves synchronization. Furthermore, the closed‐loop system's stability criterion associated with the controller gains, the varying time delays' bounds, and the maximum transmission interval is established on the basis of the Lyapunov–Krasovskii analysis. Finally, the presented algorithm's performance is demonstrated by the simulation and experiment results.

Time delay compensated disturbance observer-based sliding mode slave controller and neural network model for bilateral teleoperation system

Time delay compensated disturbance observer-based sliding mode slave controller and neural network model for bilateral teleoperation system