Robot Teleoperation Research Articles

Real-time Video Prediction Using GANs With Guidance Information for Time-delayed Robot Teleoperation

Real-time Video Prediction Using GANs With Guidance Information for Time-delayed Robot Teleoperation

An Incremental Broad-Learning-System-Based Approach for Tremor Attenuation for Robot Tele-Operation.

The existence of the physiological tremor of the human hand significantly affects the application of tele-operation systems in performing high-precision tasks, such as tele-surgery, and currently, the process of effectively eliminating the physiological tremor has been an important yet challenging research topic in the tele-operation robot field. Some scholars propose using deep learning algorithms to solve this problem, but a large number of hyperparameters lead to a slow training speed. Later, the support-vector-machine-based methods have been applied to solve the problem, thereby effectively canceling tremors. However, these methods may lose the prediction accuracy, because learning energy cannot be accurately assigned. Therefore, in this paper, we propose a broad-learning-system-based tremor filter, which integrates a series of incremental learning algorithms to achieve fast remodeling and reach the desired performance. Note that the broad-learning-system-based filter has a fast learning rate while ensuring the accuracy due to its simple and novel network structure. Unlike other algorithms, it uses incremental learning algorithms to constantly update network parameters during training, and it stops learning when the error converges to zero. By focusing on the control performance of the slave robot, a sliding mode control approach has been used to improve the performance of closed-loop systems. In simulation experiments, the results demonstrated the feasibility of our proposed method.

Time Delay Mitigation in Aerial Telerobotic Operations Using Heterogeneous Stereo-Vision Systems

This paper investigates the use of a heterogeneous stereo-vision system to mitigate the effects of time delays in a drone-based visual interface presented to a human operator. Time delays in the display for a telerobotic interface refer to the time difference between the operator’s input action and the corresponding visible outcome. In human/machine interfaces, time delays can arise due to computation, telecommunication, and mechanical limitations. These delays can degrade the performance of the human/machine system. A heterogeneous stereo-vision predictive algorithm is presented that can reduce the negative effects of time delays in the operator’s display. The heterogeneous stereo-vision system consists of an omnidirectional camera and a pan/tilt/zoom camera. Two predictive display setups were developed that modify the delayed video imagery that would otherwise be presented to the operator in a way that provides an almost immediate visual response to the operator’s control actions. The usability of the system is determined through human performance testing with and without the predictive algorithms. The results indicate that the predictive algorithm allows more efficient, accurate, and user-friendly operation.

Teleoperation of Humanoid Robots: A Survey

Teleoperation of humanoid robots enables the integration of the cognitive skills and domain expertise of humans with the physical capabilities of humanoid robots. The operational versatility of humanoid robots makes them the ideal platform for a wide range of applications when teleoperating in a remote environment. However, the complexity of humanoid robots imposes challenges for teleoperation, particularly in unstructured dynamic environments with limited communication. Many advancements have been achieved in the last decades in this area, but a comprehensive overview is still missing. This survey article gives an extensive overview of humanoid robot teleoperation, presenting the general architecture of a teleoperation system and analyzing the different components. We also discuss different aspects of the topic, including technological and methodological advances, as well as potential applications.

Highly Stretchable and Sensitive Strain Sensor Based on Porous Materials and Rhombic‐Mesh Structures for Robot Teleoperation

AbstractWearable sensors for human motion capture offer a promising human–robot interface to control robots in the teleoperation scenario, where robots could function as the second body of human operators to fulfill tasks remotely and accurately. In this paper, a novel strain sensor based on a soft polyurethane (PU) sponge and carbon nanotubes (CNT) is designed for motion capture of human joints. The unique 3D porous microstructure of the PU‐CNT sponge provides the sensor with high sensitivity. To bridge the gap between the high sensitivity and high stretchability of the strain sensor, a rhombic‐mesh structure with optimized geometric parameters, in conjunction with a pre‐compression design, is proposed for strain sensor prototyping, which endows the sensor with an extra elongation rate during the stretching process. The proposed PU‐CNT strain sensor manifests promising sensing performance with a stretchability of up to 300% and a maximum gauge factor of 3893, together with long‐term durability, low detection limit, and fast response capacity. Finally, the validation of the strain sensor is carried out by deploying the sensor on a human elbow to realize the teleoperation of a robot arm, which could be monitored through the digital twin model of the robot in a real‐time manner.

Analog Twin Framework for Human and AI Supervisory Control and Teleoperation of Robots

Analog Twin Framework for Human and AI Supervisory Control and Teleoperation of Robots

Evaluating the Effects of Virtual Reality Environment Learning on Subsequent Robot Teleoperation in an Unfamiliar Building.

Using a map in an unfamiliar environment requires identifying correspondences between elements of the map's allocentric representation and elements in egocentric views. Aligning the map with the environment can be challenging. Virtual reality (VR) allows learning about unfamiliar environments in a sequence of egocentric views that correspond closely to the perspectives and views that are experienced in the actual environment. We compared three methods to prepare for localization and navigation tasks performed by teleoperating a robot in an office building: studying a floor plan of the building and two forms of VR exploration. One group of participants studied a building plan, a second group explored a faithful VR reconstruction of the building from a normal-sized avatar's perspective, and a third group explored the VR from a giant-sized avatar's perspective. All methods contained marked checkpoints. The subsequent tasks were identical for all groups. The self-localization task required indication of the approximate location of the robot in the environment. The navigation task required navigation between checkpoints. Participants took less time to learn with the giant VR perspective and with the floorplan than with the normal VR perspective. Both VR learning methods significantly outperformed the floorplan in the orientation task. Navigation was performed quicker after learning in the giant perspective compared to the normal perspective and the building plan. We conclude that the normal perspective and especially the giant perspective in VR are viable options for preparing for teleoperation in unfamiliar environments when a virtual model of the environment is available.

Modified Bilateral Active Estimation Model: A Learning-Based Solution to the Time Delay Problem in Robotic Tele-Control

The ubiquitous presence of three types of delay in robotic teleoperation systems, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , computation delay, transmission delay, and mechanical delay, is the major factor of the system degradation. It is noticeable that the transmission latency over the communication network shows a periodic trend due to the network flux changing. Accordingly, in our previous work, a neural network-based open-loop approach named Bilateral Active Estimation Model (BAEM) was proposed to compensate for the upcoming transmission delay in a unilateral teleoperation system by sending predicted trajectories as commands. In this paper, a modified version of BAEM (m-BAEM) is proposed to compensate for all these three types of delay explicitly, and a real-time robotic teleoperation system based on Robot Operating System 2 (ROS 2) framework is built to evaluate the performance of the m-BAEM in constant and varying delay scenarios with both pre-defined and human-input trajectories. The results of pre-defined trajectories present the satisfactory performance of the m-BAEM even in the presence of transmission delay up to 1000 miliseconds with large variations. The main limitation of the m-BAEM is that it is yet unable to handle unknown trajectories.

A humanoid robot teleoperation approach based on waist–arm coordination

PurposeIn the current teleoperation system of humanoid robots, the control between arms and the control between the waist and arms are individual and lack coordinated motion. This paper aims to solve the above problem and proposes a teleoperation control approach for a humanoid robot based on waist–arm coordination (WAC).Design/methodology/approachThe teleoperation approach based on WAC comprises dual-arm coordination (DAC) and WAC. The DAC method realizes the coordinated motion of both arms through one hand by establishing a mapping relationship between a single hand controller and the manipulated object; the WAC method realizes the coordinated motion of both arms and waist by calculating the inverse kinematic input of robotic arms based on the desired velocity of the waist and the end of both arms. An integrated teleoperation control framework provides interfaces for the above methods, and users can switch control modes online to adapt to different tasks.FindingsAfter conducting experiments on the dual-arm humanoid robot through the teleoperation control framework, it was found that the DAC method can save 27.2% of the operation time and reduce 99.9% of the posture change of the manipulated object compared with the commonly used individual control. The WAC method can accomplish a task that cannot be done by individual control. The experiments proved the improvement of both methods in terms of operation efficiency, operation stability and operation capability compared with individual control.Originality/valueThe DAC method better maintains the constraints of both arms and the manipulated object. The WAC method better maintains the constraints of the manipulated object itself. Meanwhile, the teleoperation framework integrates the proposed methods and enriches the teleoperation modes and control means.

Teleoperation of Soft Robots with Real‐Time Fingertip Haptic Feedback Using Small Batteries

AbstractSoft robots are promising candidates for robust grasping and manipulation of different types of objects; however, their development has been a scientific challenge because of the lack of compact, yet precise, controllable soft actuators. In this work, innovative haptic interactive systems based on sensor‐integrated actuators powered by small batteries are reported. A liquid‐metal strain sensor with a versatile form factor and robust sensing performance is incorporated into a fast‐switching high‐force ionic polymer actuator. The structure of the sensor‐integrated actuator is rationally designed through encapsulation technology to achieve the best sensitivity and actuation while maintaining compactness, allowing for accurate, real‐time estimation and tracking of the actuation process. In a closed loop between the user and sensor‐integrated actuator, the motion and manipulation dynamics of a soft gripper made of sensor‐integrated actuators can be controlled by hand motions during a task. By combining with a thimble‐type haptic feedback device, the states of the gripper can be transmitted back to the user in real‐time via the proprioception of the integrated sensor. The approach opens a prospective avenue for the development of future robotic technologies without requiring camera‐based sensors for versatile, wearable, and scalable haptic systems.

Parallel Dual-Hand Detection by Using Hand and Body Features for Robot Teleoperation

Visual hand-based robot teleoperation provides a powerful guarantee for robots to complete complex tasks. However, detection and distinction of dual hands on images are difficult because of the small differences between left and right hands. To solve this problem, a parallel dual-hand detection and distinction method that combines the features of hands with the relationship features between the dual hands and body pose is proposed to achieve robust and accurate dual-hand detection. This parallel dual-hand detection method includes a hand detection module, a body pose estimation module, and a fusion module. In the hand detection module, a hand detector that realizes fast and accurate hand detection by detecting the center and corner points of hands is designed. In the body pose estimation module, a body pose estimator with dual-hand positions is proposed. The fusion module is designed to fuse hand detection and dual-hand estimation results to achieve distinction between left and right hands. Finally, the parallel dual-hand detection method is applied to a bimanual robot teleoperation system by using a designed dual-hand teleoperation framework. The proposed parallel dual-hand detection method can achieve 98.54% mAP of hand detection with 18 frames per second on a custom dual-hand detection dataset, and the bimanual robot teleoperation method can achieve 95.4% average accuracy for teleoperation tasks. Experimental results show the high accuracy and speed of our proposed parallel dual-hand detection method and its practicability in bimanual robot teleoperation.

Design of a Novel Haptic Joystick for the Teleoperation of Continuum-Mechanism-Based Medical Robots

Continuum robots are increasingly used in medical applications and the master–slave-based architectures are still the most important mode of operation in human–machine interaction. However, the existing master control devices are not fully suitable for either the mechanical mechanism or the control method. This study proposes a brand-new, four-degree-of-freedom haptic joystick whose main control stick could rotate around a fixed point. The rotational inertia is reduced by mounting all powertrain components on the base plane. Based on the design, kinematic and static models are proposed for position perception and force output analysis, while at the same time gravity compensation is also performed to calibrate the system. Using a continuum-mechanism-based trans-esophageal ultrasound robot as the test platform, a master–slave teleoperation scheme with position–velocity mapping and variable impedance control is proposed to integrate the speed regulation on the master side and the force perception on the slave side. The experimental results show that the main accuracy of the design is within 1.6°. The workspace of the control sticks is −60° to 110° in pitch angle, −40° to 40° in yaw angle, −180° to 180° in roll angle, and −90° to 90° in translation angle. The standard deviation of force output is within 8% of the full range, and the mean absolute error is 1.36°/s for speed control and 0.055 N for force feedback. Based on this evidence, it is believed that the proposed haptic joystick is a good addition to the existing work in the field with well-developed and effective features to enable the teleoperation of continuum robots for medical applications.

Perception-Motion Coupling in Active Telepresence: Human Behavior and Teleoperation Interface Design

Teleoperation enables complex robot platforms to perform tasks beyond the scope of the current state-of-the-art robot autonomy by imparting human intelligence and critical thinking to these operations. For seamless control of robot platforms, it is essential to facilitate optimal situational awareness of the workspace for the operator through active telepresence cameras. However, the control of these active telepresence cameras adds an additional degree of complexity to the task of teleoperation. In this paper we present our results from the user study that investigates: (1) how the teleoperator learns or adapts to performing the tasks via active cameras modeled after camera placements on the TRINA humanoid robot; (2) the perception-action coupling operators implement to control active telepresence cameras, and (3) the camera preferences for performing the tasks. These findings from the human motion analysis and post-study survey will help us determine desired design features for robot teleoperation interfaces and assistive autonomy.

Collision Risk Assessment and Operation Assistant Strategy for Teleoperation System

Teleoperation robots remain superior to fully automated robots in complicated and unstructured environments (e.g., in-orbit assembly). However, the collision risk is also greatly increased in these environments. Therefore, the teleoperation robot should possess the capability of collision risk perception and be configured with security assistance strategy to improve safety and efficiency. With this objective in mind, this paper proposes a collision risk assessment system based on fuzzy theory, which comprehensively considers the effects of shortest distance, operation speed, and delay time. The introduction of fuzzy theory makes the risk assessment results more accurate. Furthermore, this paper also proposes a creative discrete expandable bounding box method to make the calculation of the nearest distance between complicated obstacles and the robot more efficient and faster. Secondly, to improve the safety and efficiency of teleoperation, this paper proposes a set of teleoperation assistance strategies for teleoperation robots based on collision risk. The strategies include partial view highlighting, variation in the motion mapping ratio, and haptic risk warning, all of which are achieved in the graphical interactive interface. Finally, this paper verifies the proposed collision risk model and the operation assistant strategy through experiments. The results show that the proposed fuzzy collision risk model has a correct trend with risk factors, and the proposed operation assistance strategies can effectively reduce the robot collision risk and improve the safety of teleoperation robots. In conclusion, this research contributes to the collision risk assessment and obstacle avoidance assistance strategy of teleoperation robots.

Encouraging prosocial behavior from older adults through robot teleoperation: A feasibility study

IntroductionIn Japan, the social climate surrounding older adults has gotten worse as a result of the spread of COVID-19 and the growing isolation of older adults who are increasingly unable to engage in prosocial behavior through work and volunteering. This is detrimental to the physical and mental well-being of older adults. The purpose of this study is to look into robot teleoperation for older adults as a viable way to deal with these issues and overcome the barriers preventing older adults from engaging in prosocial behavior.Materials and methodsWe designed and tested a remote-control approach for dialogue agents that is appropriate for older adults as well as evaluating their impressions in a real-world setting. Twelve older adults participated in experiments in two separate locations, a children's center and the city ward office, where they could remotely teleoperate a robot and have conversations with the visitors. In the city ward office, the older adults had a conversation with the visitors and gave them information and trivia quizzes about the city. In the children's center, older adults had conversations with children regarding their age, family, their likes, and dislikes. A questionnaire and interview were set up after the experiments to understand their impressions of the system and to clarify how older adults feel about certain issues regarding remote-controlled work, starting a new job, social interaction, to what extent have older adults been affected by the pandemic, how and in what ways has it affected their involvement in society, and whether teleoperating a robot can be a suitable approach to encourage prosocial behavior from them through volunteer work and social engagement.ResultsThe results show that older adults have a strong desire to engage in volunteer work, but are hampered mainly by physical isolation resulting from COVID-19 restrictions and their declining physical and mental health. Their impressions of the teleoperation system were highly positive, as they enjoyed having conversations with children through the robot. With this teleoperation system, older adults were able to remote control a robot by themselves without major issues. It made interaction simpler as conversing with children through a robot added a layer of anonymity that allowed older adults to express themselves freely without worrying about how they are perceived by others in public.DiscussionOlder adults were able to successfully engage in prosocial behavior through remote-controlling a robot. The system seems to be effective at easing the physical barriers preventing older adults from engaging in volunteer work, which have worsened since the spread of COVID-19.

Latency mitigation using applied HMMs for mixed reality-enhanced intuitive teleoperation in intelligent robotic welding

Latency mitigation is crucial to increasing operational success, ease of use, and product quality in telemanipulation tasks when remotely guiding complex robotic systems. Hardware limitations have created a gap in performance optimization due to large teleoperation delays, which machine learning techniques could fill with lower time, improved performance, and reduced operating costs. Hidden Markov models (HMMs), in particular, have been explored to alleviate the issue due to their relative ease of use. A mixed reality-enhanced intuitive teleoperation framework for immersive and intuitive telerobotic welding is presented. The proposed system implements an HMM generative algorithm to learn and predict human-welder motion to enable a low-cost solution, combining smoothing and forecasting techniques to minimize robotic teleoperation time delay. The predicted welding motion system is simple to implement, can be used as a general solution to solve time delays, and is accurate. More specifically, it provides a 66% RMSE reduction compared to the application without HMM, which may be further optimized by up to 38%. Experiments show the HMM generative algorithm lets humans conduct tele-robot-assisted welding with better performance.

Teleoperation by seamless transitions in real and virtual world environments

This study investigates operability and acceptability issues in the teleoperation of robots. Prior studies have proposed efficient approaches to increase human perceptual ability and robot autonomy but with reduced operability and acceptance. We propose a novel teleoperation method that overcomes the weaknesses of existing approaches while inheriting their strengths. The key feature of our method is switching the teleoperated robot world from real to virtual. The user study results showed that the proposed method offered an improved user experience compared to the conventional methods, while task efficiency was equivalent in all methods. The contributions of this paper include the proposal of the teleoperation method by seamless switching between real and virtual space, the proposal of an image transformation method and visual effect to achieve seamless switching, and verification of the practicality of the proposed system through experiments on actual mobile robots.

Toward Multi-Service Edge-Intelligence Paradigm: Temporal-Adaptive Prediction for Time-Critical Control over Wireless

Time-critical control applications typically pose stringent connectivity requirements for communication networks. The imperfections associated with the wireless medium such as packet losses, synchronization errors, and varying delays have a detrimental effect on performance of real-time control, often with safety implications. This paper introduces multi-service edge-intelligence as a new paradigm for realizing time-critical control over wireless. It presents the concept of multi-service edge-intelligence which revolves around tight integration of wireless access, edge-computing and machine learning techniques, in order to provide stability guarantees under wireless imperfections. The paper articulates some of the key system design aspects of multi-service edge-intelligence. It also presents a temporal-adaptive prediction technique to cope with dynamically changing wireless environments. It provides performance results in a robotic teleoperation scenario. Finally, it discusses some open research and design challenges for multi-service edge-intelligence.

A Novel Multi-Modal Teleoperation of a Humanoid Assistive Robot with Real-Time Motion Mimic

This research shows the development of a teleoperation system with an assistive robot (NAO) through a Kinect V2 sensor, a set of Meta Quest virtual reality glasses, and Nintendo Switch controllers (Joycons), with the use of the Robot Operating System (ROS) framework to implement the communication between devices. In this paper, two interchangeable operating models are proposed. An exclusive controller is used to control the robot's movement to perform assignments that require long-distance travel. Another teleoperation protocol uses the skeleton joints information readings by the Kinect sensor, the orientation of the Meta Quest, and the button press and thumbstick movements of the Joycons to control the arm joints and head of the assistive robot, and its movement in a limited area. They give image feedback to the operator in the VR glasses in a first-person perspective and retrieve the user's voice to be spoken by the assistive robot. Results are promising and can be used for educational and therapeutic purposes.

An Efficient RGB-D Hand Gesture Detection Framework for Dexterous Robot Hand-Arm Teleoperation System

Aiming at the problems of accurate and fast hand gesture detection and teleoperation mapping in the hand-based visual teleoperation of dexterous robots, an efficient hand gesture detection framework based on deep learning is proposed in this article. It can achieve an accurate and fast hand gesture detection and teleoperation of dexterous robots based on an anchor-free network architecture by using an RGB-D camera. First, an RGB-D early-fusion method based on the HSV space is proposed, effectively reducing background interference and enhancing hand information. Second, a hand gesture classification network (HandClasNet) is proposed to realize hand detection and localization by detecting the center and corner points of hands, and a HandClasNet is proposed to realize gesture recognition by using a parallel EfficientNet structure. Then, a dexterous robot hand-arm teleoperation system based on the hand gesture detection framework is designed to realize the hand-based teleoperation of a dexterous robot. Our method achieves high accuracy with fast speed on public and custom hand datasets and outperforms some state-of-the-art methods. In addition, the application of the proposed method in the hand-based teleoperation system can control the grasping of various objects by a dexterous hand-arm system in real time and accurately, which verifies the efficiency of our method.