Teleoperation System Research Articles

The experience of a tele-operated avatar being touched increases operator's sense of discomfort.

Recent advancements in tele-operated avatars, both on-screen and robotic, have expanded opportunities for human interaction that exceed spatial and physical limitations. While numerous studies have enhanced operator control and improved the impression left on remote users, one area remains underexplored: the experience of operators during touch interactions between an avatar and a remote interlocutor. Touch interactions have become commonplace with avatars, especially those displayed on or integrated with touchscreen interfaces. Although the need for avatars to exhibit human-like touch responses has been recognized as beneficial for maintaining positive impressions on remote users, the sensations and experiences of the operators behind these avatars during such interactions remain largely uninvestigated. This study examines the sensations felt by an operator when their tele-operated avatar is touched remotely. Our findings reveal that operators can perceive a sensation of discomfort when their on-screen avatar is touched. This feeling is intensified when the touch is visualized and the avatar reacts to it. Although these autonomous responses may enhance the human-like perceptions of remote users, they might also lead to operator discomfort. This situation underscores the importance of designing avatars that address the experiences of both remote users and operators. We address this issue by proposing a tele-operated avatar system that minimizes unwarranted touch interactions from unfamiliar interlocutors based on social intimacy.

The Adaptive Bilateral Control of Underwater Manipulator Teleoperation System with Uncertain Parameters and External Disturbance

A novel self-adaptive bilateral control strategy is introduced to manage uncertainties inherent in the teleoperation of an underwater manipulator system effectively. In response to uncertainties stemming from both the mathematical model and external disturbances, our approach offers innovative solutions. Firstly, to address uncertainties in the master model parameters, we propose a reference adaptive impedance control based on a nominal model. This control strategy dynamically adjusts the reference position of the desired model, leveraging adaptive control laws to compensate for model uncertainties. Additionally, to tackle uncertainties specific to the slave manipulator, we employ adaptive compensation using radial basis function (RBF) networks. Our unique combination of sliding mode variable structure controllers and robust adaptive controllers aims to mitigate approximation errors, ensuring precise tracking of the master manipulator’s position by the slave manipulator. By employing Lyapunov function analysis, we demonstrate the system’s superior tracking performance and global stability, with assured asymptotic convergence for force–position tracking. Through comprehensive experimentation, our results showcase the exceptional force–position tracking capabilities of the overall control system, even under challenging conditions of model uncertainties and external disturbances. Moreover, our system exhibits remarkable stability, reliability, and robustness, underscoring the effectiveness of our proposed adaptive control approach.

Sensorless force estimation of teleoperation system based on multilayer depth Extreme Learning Machine

The force feedback technology has a crucial impact on the precise control of teleoperation system. Robots in fields such as minimally invasive surgery and nuclear waste cannot integrate force sensors to obtain force feedback due to their end size and harsh working environment. In order to obtain end-effector force feedback in sensorless situation, a sensorless force estimation model based on artificial neural network is defined in this paper. For improving the model prediction ability, a new Multi-layer Depth Extreme Learning Machine (MDELM) is proposed. Firstly, the network structure of the extreme learning machine is redesigned, and the smoothing function and Gauss-Laplacian function are constructed as the feature extraction layer and enhancement layer of the MDELM model. Then, to further improve the prediction performance of MDELM, an improved Sooty Tern Optimization Algorithm (STOA) is introduced to optimize the parameters of the model. The results show that the proposed force estimator outperforms the existing model with an MAE of less than 0.205, which is at least 18.65% lower than the existing model. In addition, when the acceleration of the remote operating system is a fixed constant or the information is not available，a sensorless force feedback system can be constructed by combining motor torque, joint position and joint velocity parameter. In general, this study provides an effective force estimation solution for teleoperation system without force sensor.

Applications of 3D Reconstruction in Virtual Reality-Based Teleoperation: A Review in the Mining Industry

Although multiview platforms have enhanced work efficiency in mining teleoperation systems, they also induce “cognitive tunneling” and depth-detection issues for operators. These issues inadvertently focus their attention on a restricted central view. Fully immersive virtual reality (VR) has recently attracted the attention of specialists in the mining industry to address these issues. Nevertheless, developing VR teleoperation systems remains a formidable challenge, particularly in achieving a realistic 3D model of the environment. This study investigates the existing gap in fully immersive teleoperation systems within the mining industry, aiming to identify the most optimal methods for their development and ensure operator’s safety. To achieve this purpose, a literature search is employed to identify and extract information from the most relevant sources. The most advanced teleoperation systems are examined by focusing on their visualization types. Then, various 3D reconstruction techniques applicable to mining VR teleoperation are investigated, and their data acquisition methods, sensor technologies, and algorithms are analyzed. Ultimately, the study discusses challenges associated with 3D reconstruction techniques for mining teleoperation. The findings demonstrated that the real-time 3D reconstruction of underground mining environments primarily involves depth-based techniques. In contrast, point cloud generation techniques can mostly be employed for 3D reconstruction in open-pit mining operations.

Extraperitoneal robot assisted laparoscopic prostatectomy with Versius system: single centre experience.

Versius Surgical System (CMR Surgical, Cambridge, UK) is a novel tele-operated robotic surgical system designed to assist surgeons for minimally invasive surgery which is gaining momentum in the world of robotic surgery. We describe our single centre experience with Versius and report the advantages and challenges posed by this new robotic system in a series of 53 extraperitoneal robotic assisted laparoscopic prostatectomies (eRALP) for prostate cancer (PCa). Data of 53 eRALP performed with Versius in our centre were collected and analysed, Descriptive statistics were used to report our results. In 16 months we performed 53 eRALP: 18 (34%) with PLND, 33 (62%) nerve sparingcases. Mean setup time was 15 min, mean console time was 100 min and mean operative time was 130 min. We observed a substantial reduction of console time and set-up time after only 5 procedures. In the first 4 procedures, the dissection of the neurovascular bundle was performed laparoscopically, to switch back to robotic assisted approach afterwards. No major system failures were observed. No major intra-operative and post-operative complications occurred. Mean follow-up time was 9 months (range 3-15 months); no patients experienced biochemical recurrence or metastatic progression over this period, 8 (15%) patients had adjuvant radiotherapy based on unfavourable pathology report (positive surgical margins or positive limphnodes). This represents to our knowledge the largest extraperitoneal RALP case series with Versius, and it aims to provide solid clinical proof of the safety, effectiveness and versatility of this innovative system. In our experience, this platform represents a good option for every urologic surgeon who wants to start a robotic programme and it appears particularly suitable for urologists with a large laparoscopic expertise.

A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision.

Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-to-end hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-to-image translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.

The nonlinear model reference adaptive impedance control for underwater manipulator operation objects in bilateral teleoperation system

The bilateral teleoperation system is susceptible to model parameter uncertainty and unknown disturbances in both the master and slave manipulators, resulting in instability and inaccuracies in the force and position tracking performance. To address these issues, a novel nonlinear model reference adaptive impedance controller has been designed to achieve coordinated force and position synchronization of dual manipulators. The adaptive control laws, based on sliding mode functions, have been designed to compensate for the uncertainty of the manipulator model. Furthermore, an adaptive estimation law has been employed to appraise the unknown upper bound of external disturbances. This ensures that the closed-loop model parameters of the dual manipulator converge to the reference impedance model respectively. Simultaneously, it enables the position error between the reference model response and the end-effector task space position to asymptotically converge to zero. To verify the effectiveness of the proposed controller, simulations have been conducted on the MATLAB platform and experiments on a single degree of freedom teleoperation system have been performed. The results demonstrate that the controller exhibits strong robustness and has the capability of force-position tracking ability.

The Influence of the Operator’s Perception on the Energy Demand for a Hydraulic Manipulator with a Large Working Area

The efficient operation of hydraulic manipulators with expansive working areas is crucial in various applications such as the construction industry, the rescue service, and the military. These machines are characterized by having more capabilities than humans, and they perform tasks that are not repeated in the same environment. For this reason, they are most often controlled by a human in a teleoperation system. This research investigates the influence of the operator’s perception on the energy demand of such manipulators. Specifically, the research focused on assessing how the intuitive control systems, such as primary–secondary solutions, impact the energy consumption. Understanding the relation between the operator’s perception and the energy demand is essential for optimizing manipulator design and operation. Experimental research was conducted to analyze the velocity and acceleration of the manipulator’s effector, which is controlled by human operators under different movement ranges and size ratios. The obtained test results allow for the assessment of the dynamic loads, velocity, and energy consumption of the movement of a manipulator with a large working area due to the limitations resulting from the operator’s perception.

Verification of a virtual lunar regolith simulant

Introduction: Physical regolith simulants are valuable tools for developing In-Situ Resource Utilisation hardware. However, using virtual models of regolith instead can reduce costs, limit exposure to hazardous materials, and offer a practical method of testing the effects of reduced gravity.Methods: We verify a virtual model of regolith as macroparticles against physical tests. Using space partitioning techniques to identify neighbouring particles, we present a scalable model of regolith, in which the computation time increases roughly proportionally with the number of particles. We evaluated the performance of this virtual simulant vs. a physical simulant (Exolith LMS-1) by comparing the flow rate through funnels of various diameters, and the resultant angle of repose of material on both large (500 g) and small (16 g) scale tests.Results: For large scale tests, the flow rates were within the predicted range for macroparticles with radii 3–7 mm, with the greatest accuracy achieved for radii 4–5 mm. However, the macroparticles blocked the simulated funnels more easily than in the physical trials, due to their high cohesion. The angle of repose was not accurately represented by this model for either of the tests.Discussion: The high efficiency of this model makes it best suited for applications which require large scale approximations of regolith with real-time execution, such as virtual training for robot operators or providing visual and haptic feedback in model-mediated teleoperation systems. The results of this model in reduced gravity could be further verified against data from upcoming lunar missions in future work.

Experimental Evaluation of Manipulator Teleoperation System Based on Trajectory Planning for Obstacle Removal Task in Nuclear Plant Decommissioning

In the teleoperation system of a dual-arm manipulator developed to retrieve fuel debris and reactor interior structures at the Fukushima Daiichi Nuclear Power Plant, we used software in which an obstacle avoidance function (trajectory planning) was implemented to conduct a mock test simulating obstacle removal operations in narrow spaces. The test results confirmed the validity of the obstacle avoidance function, the executability of a series of necessary tasks, and the improved operability. In addition, issues were identified using the test data.

Global terminal sliding‐mode control for the synchronization problem of uncertain bilateral teleoperation system with time‐varying delays

AbstractIn this paper, an adaptive global terminal sliding‐mode control is developed for a nonlinear bilateral teleoperation system with time‐varying delays, external disturbances, and uncertainties. By selecting a time‐varying sliding‐mode manifold, a finite‐time sliding‐mode control algorithm is proposed to guarantee the suppression of the reaching phase associated with the conventional sliding‐mode control. This implies that the robustness is guaranteed throughout to entire response of the system. Meanwhile, the chattering phenomenon and the energy consumption are significantly reduced. Furthermore, in order to enhance the tracking performance in practical situations, adaptive tuning laws are developed to estimate the unknown upper bounds of the lumped uncertainties. Hence, the proposed approach has several merits, namely, fast response, accurate synchronization, and strong robustness with respect to large uncertainties and disturbances. The stability of the system in closed‐loop and the boundedness on the internal dynamics are proved by Lyapunov stability theory. Illustrative simulations were conducted to show the effectiveness of the proposed controller.

Practical Preassigned Fixed-Time Fuzzy Control for Teleoperation System Under Scheduled Shared-Control Framework

Practical Preassigned Fixed-Time Fuzzy Control for Teleoperation System Under Scheduled Shared-Control Framework

Adaptive Practical Predefined-Time Control for Uncertain Teleoperation Systems With Input Saturation and Output Error Constraints

In this paper, the synchronization tracking control issue is investigated for uncertain teleoperation systems with input saturation and output error constraints. To this end, an adaptive practical predefined-time control scheme integrating backstepping recursive design, predefined time theory, prescribed performance function, and the anti-saturation auxiliary system, is developed for the first time. A fixed-time extended state observer is utilized to eliminate the negative influence of the lumped uncertainty, and the anti-saturation auxiliary system is constructed to address the input saturation. In contrast to finite/fixed-time controllers, the upper bound of convergence time can be obtained in advance by adjusting a single control parameter. The results indicate that the error signals can converge into a small region of the zero domain within a user-defined time and that the output error never violates the prescribed performance boundary. Simulations and experiments are performed on a teleoperation platform made up of two Phantom Touch robots to verify the effectiveness and practicality of the developed controller.

Fundamental Study on Force-Projecting Bilateral Control for Pneumatically Driven Follower Device

This paper proposes the application of force-projecting bilateral control to a master-follower teleoperation system with pneumatic drive on the follower side and evaluates its effectiveness. The proposed method directly projects the operating force on the master side to the driving force on the follower side, eliminating the need for both position control and external force detection on the follower side, thereby solving the problem of low rigidity and response delay of a pneumatic servo system and providing highly stable sensor-less force presentation against variable environments. In this study, dynamic response analyses of a 1-DOF master-follower system were performed by numerical simulation using a linear system model, followed by experimental verification by implementing an actual system with an external force estimator. The results showed that the proposed force-projecting bilateral control has significantly higher positioning rigidity and better force control stability than the conventional force-reflecting bilateral control. A theoretical consideration was also given using the equivalent transformation of force transfer functions to provide evidence of high stability.

Adaptive synchronization of multilateral teleoperation system with DoS attacks

Adaptive synchronization of multilateral teleoperation system with DoS attacks

Enhancing autonomy in social robots through a real-time teleoperation system

<p>The utilization of socially assistive robots (SARs) has shown great potential in various applications. However, there are practical challenges that hinder their widespread adoption. Recent studies have explored the use of teleoperation systems as a solution to overcome these challenges by streamlining data collection and communication with SARs. This paper introduces a cloud-based teleoperation system for robotics that enables real-time control of the robot platform, allowing remote users to view video streams and manipulate the robot system. The proposed teleoperation system was implemented in a cloud environment and tested with two different social robot platforms. The experimental results demonstrate the system's quick response and stable communication performance, meeting the requirements for remote-controlled teleoperation. Consequently, employing the proposed mobile-based teleoperation system in social robot applications enhances operator productivity, expedites operations, and reduces task completion time.</p>

Practical Approaches Towards Transparent and Stable Bilateral Teleoperation Under Time-Varying Network Delay

AbstractIn real-life teleoperation scenarios, the presence of time-varying network delays, particularly in wireless networks, poses significant challenges in maintaining stability in a bilateral teleoperation system. Various approaches have been proposed in the past to address stability concerns; however, these often come at the expense of system transparency. Nevertheless, increasing transparency is crucial in a teleoperation system to enable precise and safe operations, as well as to provide real-time decision-making capabilities for the operator. This paper presents our comprehensive approaches to maximize teleoperation transparency by minimizing system impedance, enhance the wave variable method to handle time-varying network delays, and alleviate non-smooth effects caused by network jitters in bilateral teleoperation. The proposed methodologies take into account the real-world challenges and considerations to ensure the practical applicability and effectiveness of the teleoperation system. Throughout these implementations, passivity analysis is employed to ensure system stability, guaranteeing a reliable and safe teleoperation experience. The proposed approaches were successfully validated in Team Northeastern’s Avatar telepresence system, which achieved the 3rd place in ANA Avatar XPRIZE challenge.

Development and user evaluation of an immersive light field system for space exploration

This paper presents the developmental work and user evaluation results of an immersive light field system built for the European Space Agency’s (ESA) project called “Light field-enhanced immersive teleoperation system for space station and ground control.” The main aim of the project is to evaluate the usefulness and feasibility of light fields in space exploration, and compare it to other types of immersive content, such as 360° photos and point clouds. In the course of the project, light field data were captured with a robotically controlled camera and processed into a suitable format. The light field authoring process was performed, and a light field renderer capable of displaying immersive panoramic or planar light fields on modern virtual reality hardware was developed. The planetary surface points of interest (POIs) were modeled in the laboratory environment, and three distinct test use cases utilizing them were developed. The user evaluation was held in the European Astronaut Centre (EAC) in the summer of 2023, involving prospective end-users of various backgrounds. During the evaluation, questionnaires, interviews, and observation were used for data collection. At the end of the paper, the evaluation results, as well as a discussion about lessons learned and possible improvements to the light field system, are presented.

Experimental Use Validation of the Master Hybrid Haptic Device Dedicated to Remote Center-of-Motion Tasks

The main objective of this paper is to discuss the experimental validation of a tele-operation system for remote center-of-motion tasks, such as laparoscopic surgery. This validation is based on the use of an extra sensor placed on the master manipulator. The tele-operation system is composed of a new hybrid haptic device (nHH) intended to be used as a master manipulator controlling a collaborative robot, used as a slave surgical robot. The resolution of the forward kinematic model (FKM) of the master device is performed experimentally thanks to the use of an extra sensor. The IMU, as the extra sensor, is installed on the serial part of the nHH device to measure the orientation and is enabled to solve the FKM of the parallel part of the nHH device. The use of an extra sensor reduces the calculation time, improves the accuracy of the KFM, and makes it suitable for real-time applications. The preliminary validation of the force feedback in the nHH workspace is validated. Experiments were conducted on the master–slave platform to validate the proposed approach. The results are promising, which proves that the nHH device presents a suitable performance for the desired task.

Robust passivity-based nonlinear controller design for bilateral teleoperation system under variable time delay and variable load disturbance

Abstract This research focuses on implementing a robust passivity-based nonlinear control method for bilateral/teleoperation systems. The key challenge is addressing communication pathways between the master and slave, control delays, and load disturbances, which can lead to instability and reduced transparency. To tackle these issues, the proposed controller incorporates a second-order super-twisting sliding-mode observer to counteract communication and control delays. A sliding mode assist disturbance observer compensates for load torque variations. The approach aims to ensure stability and transparency by handling time-varying delays. The system model comprises two interconnected direct-drive motors, simulating robotic configurations without a physical robot. The nonlinear controller framework simplifies the complex bilateral control problem, significantly improving stability and transparency performance. Computer simulations with step and sinusoidal inputs demonstrate the effectiveness of the approach, providing a satisfactory level of accuracy and transparency between estimated and actual slave positions, even with varying delays and load variations. The research contributes to control engineering by offering a robust method to enhance bilateral system performance, ensuring stable and transparent communication between the master and slave, particularly suitable for real-time internet-based bilateral control systems.