Teleoperation System Research Articles

Sliding-mode control for teleoperation system with uncertain kinematics and dynamics: An observer-based approach

This study concentrates on the tracking control of teleoperation system subjected to robot uncertainties. The coupling of kinematic and dynamic uncertainties poses a challenge to construct the teleoperation controller. To overcome this difficulty, an observer-based approach is designed to ensure position tracking while compensating for the unfavorable effects arising from the uncertainties. First, two sliding-mode observers together with a novel power reaching law are constructed, upon which, the uncertainties will be estimated in finite time. Next, a controller is proposed to solve the finite-time convergence of the tracking errors. The settling time and the stability of the closed-loop system are derived by Lyapunov’s direct method. Simulation results are presented to testify the tracking performance of the suggested control.

Emulation Framework for Haptic Data Transmission Using Real-Time Transport Protocol

The Tactile Internet (TI) can be regarded as the next evolution in the world of communication. With its envisioned purpose and potential in shaping up the economy, industry and society, this paradigm aims to bring a new dimension to life by enabling humans to interact with machines remotely and in real-time with haptic and kinesthetic feedback. However, to translate this into reality, Tactile Internet will need to meet the stringent requirements of extremely low latency in conjunction with ultra-high reliability, availability, and security. This poses a challenge on the available communication systems to achieve a round-trip delay within 1 to 10 milliseconds time bound that enables the timely delivery of critical tactile and haptic sensations.
 This paper aims to evaluate the Real-Time Transport Protocol (RTP) through an emulation framework. It integrates containerization using Linux-based Docker Containers with NS-3 Network Simulator to conceptualize a haptic teleoperation system. The framework is then used to test the protocol’s feasibility for delivering texture haptic data between master and slave domains in accordance with the end-to-end delay requirements specified by IEEE 1918.1 standards. The results have shown that the timely provision of haptic data is achievable by obtaining an average round-trip delay of 17.8493 ms from the emulation experiment. As such, the results satisfy the expected IEEE 1918.1 standards constraints for medium-dynamic environment use cases.

Teleoperation of Humanoid Robots: A Survey

Teleoperation of humanoid robots enables the integration of the cognitive skills and domain expertise of humans with the physical capabilities of humanoid robots. The operational versatility of humanoid robots makes them the ideal platform for a wide range of applications when teleoperating in a remote environment. However, the complexity of humanoid robots imposes challenges for teleoperation, particularly in unstructured dynamic environments with limited communication. Many advancements have been achieved in the last decades in this area, but a comprehensive overview is still missing. This survey article gives an extensive overview of humanoid robot teleoperation, presenting the general architecture of a teleoperation system and analyzing the different components. We also discuss different aspects of the topic, including technological and methodological advances, as well as potential applications.

Adaptive Event-Triggered Control for Nonlinear Systems With Asymmetric State Constraints: A Prescribed-Time Approach

Finite/Fixed-time control yields a promising tool to optimize a system&#x0027;s settling time, but lacks the ability to separately define the settling time and the convergence domain (known as <i>practically prescribed-time stability</i>, PPTS). We provide a sufficient condition for PPTS based on a new piecewise exponential function, which decouples the settling time and convergence domain into separately user-defined parameters. We propose an adaptive event-triggered prescribed-time control scheme for nonlinear systems with asymmetric output constraints, using an exponential-type barrier Lyapunov function. We show that this PPTS control scheme can guarantee tracking error convergence performance, while restricting the output state according to the prescribed asymmetric constraints. Compared with traditional finite/fixed-time control, the proposed methodology yields separately user-defined settling time and convergence domain without the prior information on disturbance. Moreover, asymmetric state constraints can be handled in the control structure through bias state transformation, which offers an intuitive analysis technique for general constraint issues. Simulation and experiment results on a heterogeneous teleoperation system demonstrate the merits of the proposed control scheme.

ROV teleoperation via human body motion mapping: Design and experiment

Remotely Operated Vehicles (ROV) are widely used in subsea engineering such as inspection, construction and maintenance of underwater facilities. Currently, ROV controls are primarily based on control kiosks with the camera display and joystick control devices on a surface vessel. It sets a high barrier for ROV teleoperation and results in prolonged personnel on board (POB) time. This paper proposes a human body motion and hand gesture control method in Virtual Reality (VR) for ROV teleoperation in navigation and stabilization tasks. Specifically, a whole-body haptic suit converts ROV sensor data of hydrodynamic forces into haptic feedback of different magnitudes, then body-carried sensors capture and model human body motions and map them into the gesture controls of a remote ROV. As a result, a simulated haptomotor loop is accomplished, facilitating automatic and spontaneous motor controls of ROV in an intuitive way. A human subject experiment (N = 30) was performed to test the effectiveness of the proposed ROV control method. The result shows that with the proposed method, the navigation and stabilization control precision is improved, along with reduced mental load and perceived benefits. The findings will inspire the design of novel ROV teleoperation systems that would lower the professional barrier and increase broader participation in the subsea engineering of tomorrow.

A Lightweight Exoskeleton Force Feedback Glove

The wearable force feedback glove provides a promising solution for enhancing immersion during teleoperation. In this study, a lightweight five-finger exoskeleton force feedback glove (EFFG) was designed, enabling driving force detection and flexible force feedback. This wireless prototype weighs only 278 g. The glove features a bionic structure and optimized linkage length to ensure operator safety while providing extensive coverage of the finger working space. Moreover, a detailed illustration of the kinematic and dynamic analyses, as well as the circuit structure, was presented. With this prototype as the basis, an isomorphic teleoperation system is designed to achieve force feedback during teleoperation. Concurrently, a driving force-based impedance controller was proposed to enable smooth and precise force feedback. Finally, the performance of the EFFG prototype was evaluated in both unconstrained and constrained environments, demonstrating that the proposed glove is lightweight, capable of detecting driving force, and provides flexible force feedback.

An Adaptive Hand Exoskeleton for Teleoperation System

Teleoperation can assist people to complete various complex tasks in inaccessible or high-risk environments, in which a wearable hand exoskeleton is one of the key devices. Adequate adaptability would be available to enable the master hand exoskeleton to capture the motion of human fingers and reproduce the contact force between the slave hand and its object. This paper presents a novel finger exoskeleton based on the cascading four-link closed-loop kinematic chain. Each finger has an independent closed-loop kinematic chain, and the angle sensors are used to obtain the finger motion including the flexion/extension and the adduction/abduction. The cable tension is changed by the servo motor to transmit the contact force to the fingers in real time. Based on the finger exoskeleton, an adaptive hand exoskeleton is consequently developed. In addition, the hand exoskeleton is tested in a master–slave system. The experiment results show that the adaptive hand exoskeleton can be worn without any mechanical constraints, and the slave hand can follow the motions of each human finger. The accuracy and the real-time capability of the force reproduction are validated. The proposed adaptive hand exoskeleton can be employed as the master hand to remotely control the humanoid five-fingered dexterous slave hand, thus, enabling the teleoperation system to complete complex dexterous manipulation tasks.

Unified Human Intention Recognition and Heuristic-Based Trajectory Generation for Haptic Teleoperation of Non-Holonomic Vehicles

In this paper, a novel bilateral shared control approach is proposed to address the issue of strong dependence on the human, and the resulting burden of manipulation, in classical haptic teleoperation systems for vehicle navigation. A Hidden Markov Model (HMM) is utilized to handle the Human Intention Recognition (HIR), according to the force input by the human—including the HMM solution, i.e., Baum–Welch algorithm, and HMM decoding, i.e., Viterbi algorithm—and the communication delay in teleoperation systems is added to generate a temporary goal. Afterwards, a heuristic and sampling method for online generation of splicing trajectory based on the goal is proposed innovatively, ensuring the vehicle can move feasibly after the change in human intention is detected. Once the trajectory is available, the vehicle velocity is then converted to joystick position information as the haptic cue of the human, which enhances the telepresence. The shared teleoperation control framework is verified in the simulation environment, where its excellent performance in the complex environment is evaluated, and its feasibility is confirmed. The experimental results show that the proposed method can achieve simple and efficient navigation in a complex environment, and can also give a certain situational awareness to the human.

Modified Bilateral Active Estimation Model: A Learning-Based Solution to the Time Delay Problem in Robotic Tele-Control

The ubiquitous presence of three types of delay in robotic teleoperation systems, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , computation delay, transmission delay, and mechanical delay, is the major factor of the system degradation. It is noticeable that the transmission latency over the communication network shows a periodic trend due to the network flux changing. Accordingly, in our previous work, a neural network-based open-loop approach named Bilateral Active Estimation Model (BAEM) was proposed to compensate for the upcoming transmission delay in a unilateral teleoperation system by sending predicted trajectories as commands. In this paper, a modified version of BAEM (m-BAEM) is proposed to compensate for all these three types of delay explicitly, and a real-time robotic teleoperation system based on Robot Operating System 2 (ROS 2) framework is built to evaluate the performance of the m-BAEM in constant and varying delay scenarios with both pre-defined and human-input trajectories. The results of pre-defined trajectories present the satisfactory performance of the m-BAEM even in the presence of transmission delay up to 1000 miliseconds with large variations. The main limitation of the m-BAEM is that it is yet unable to handle unknown trajectories.

Fixed-time adaptive neural network synchronization control for teleoperation system with position error constraints and time-varying delay

Fixed-time adaptive neural network synchronization control for teleoperation system with position error constraints and time-varying delay

Ultrasound assessments of organs and blood vessels before and after 40days isolation in a cavern (deep time experiment 2021).

Introduction: Spaceflight simulation studies like confinement in small volume habitat with limited physical activity have reported even after 60days an abnormal arterial wall adaptation with increase thickness or stiffness. The purpose of the current study was to determine the effects on blood vessel and organ structure of 40days of isolation in a huge habitat with intensive physical activity. Method: Data were collected from 14 individuals (7 male) who isolated in a cavern for 40-days while performing normal daily activities without time references. Ultrasound assessments were performed pre- and post-isolation using a teleoperated system with eight different acoustic windows to obtain 19 measurements on 12 different organ/vascular structures which included the common carotid artery, femoral artery, tibial artery, jugular vein, portal vein, bile duct, kidney, pancreas, abdominal aorta, cervical and lumbar vertebral distance, and Achilles tendon. Results: Common carotid artery measures, including the intima media thickness, stiffness index, and the index of reflectivity measured from the radiofrequency signal, were not changed with isolation. Similarly, no differences were found for femoral artery measurements or measurements of any of the other organs/vessels assessed. There were no sex differences for any of the assessments. Discussion: Results from this study indicate a lack of physiological effects of 40-days of isolation in a cavern, contrary to what observed in previous 60days confinement. This suggests a potential protective effect of sustained physical activity, or reduced environmental stress inside the huge volume of the confined facility.

A medical assistive robot for tele-healthcare during the COVID-19 pandemic: development and usability study in an isolation ward.

The coronavirus disease 2019 (COVID-19) pandemic is affecting the mental and emotional well-being of patients, family members, and healthcare workers. Patients in the isolation ward may have psychological problems due to long-term hospitalization, the development of the epidemic, and the inability to meet their families. The medical assistive robot (MAR), acting as an intermediary of communication, can be deployed to address mental pressures. CareDo, a MAR with telepresence and teleoperation functions, is developed in this work for remote healthcare. This study aims to investigate its practical performance in the isolation ward during the pandemic. Two systems were integrated into the CareDo robot. For the telepresence system, web real-time communications solution is used for the multi-user chat system and the convolutional neural network is used for expression recognition. For the teleoperation system, an incremental motion mapping method is used for operating the robot remotely. This study was finally conducted at the First Affiliated Hospital, Zhejiang University for clinical trials. During the clinical trials in First Affiliated Hospital, Zhejiang University, tasks such as video chatting, emotion detection, and medical supplies delivery are performed through this robot. Seven voice commands are set for performing system wakeup, video chatting, and system exiting. Statistical durations from 1 second to 3 seconds of common commands are set to improve voice command detection. The facial expression was recorded 152 times for a patient in one day for the psychological intervention. The recognition accuracy reaches 95% and 92.8% for happy and neutral expressions respectively. Patients and healthcare workers can use this MAR in the isolation ward for tele-healthcare during the COVID-19 pandemic. It can be a useful approach to break the chains of virus transmission, and also an effective way for remote psychological intervention.

A humanoid robot teleoperation approach based on waist–arm coordination

PurposeIn the current teleoperation system of humanoid robots, the control between arms and the control between the waist and arms are individual and lack coordinated motion. This paper aims to solve the above problem and proposes a teleoperation control approach for a humanoid robot based on waist–arm coordination (WAC).Design/methodology/approachThe teleoperation approach based on WAC comprises dual-arm coordination (DAC) and WAC. The DAC method realizes the coordinated motion of both arms through one hand by establishing a mapping relationship between a single hand controller and the manipulated object; the WAC method realizes the coordinated motion of both arms and waist by calculating the inverse kinematic input of robotic arms based on the desired velocity of the waist and the end of both arms. An integrated teleoperation control framework provides interfaces for the above methods, and users can switch control modes online to adapt to different tasks.FindingsAfter conducting experiments on the dual-arm humanoid robot through the teleoperation control framework, it was found that the DAC method can save 27.2% of the operation time and reduce 99.9% of the posture change of the manipulated object compared with the commonly used individual control. The WAC method can accomplish a task that cannot be done by individual control. The experiments proved the improvement of both methods in terms of operation efficiency, operation stability and operation capability compared with individual control.Originality/valueThe DAC method better maintains the constraints of both arms and the manipulated object. The WAC method better maintains the constraints of the manipulated object itself. Meanwhile, the teleoperation framework integrates the proposed methods and enriches the teleoperation modes and control means.

A Mixed-Reality-Based Unknown Space Navigation Method of a Flexible Manipulator.

A hyper-redundant flexible manipulator is characterized by high degree(s) of freedom (DoF), flexibility, and environmental adaptability. It has been used for missions in complex and unknown spaces, such as debris rescue and pipeline inspection, where the manipulator is not intelligent enough to face complex situations. Therefore, human intervention is required to assist in decision-making and control. In this paper, we designed an interactive navigation method based on mixed reality (MR) of a hyper-redundant flexible manipulator in an unknown space. A novel teleoperation system frame is put forward. An MR-based interface was developed to provide a virtual model of the remote workspace and virtual interactive interface, allowing the operator to observe the real-time situation from a third perspective and issue commands to the manipulator. As for environmental modeling, a simultaneous localization and mapping (SLAM) algorithm based on an RGB-D camera is applied. Additionally, a path-finding and obstacle avoidance method based on artificial potential field (APF) is introduced to ensure that the manipulator can move automatically under the artificial command in the remote space without collision. The results of the simulations and experiments validate that the system exhibits good real-time performance, accuracy, security, and user-friendliness.

An Electrostatically Actuated Gecko Adhesive Clutch

AbstractThis work presents a new variation on electrostatic clutches that uses gecko‐inspired adhesives instead of friction for its braking force. As a result, it requires no power or normal pressure to remain engaged or disengaged. It requires only a brief pulse of voltage to switch states. In some applications, this capability is desirable for safety reasons. As an illustration, the clutch is incorporated into the needle‐driving axis of a magnetic resonance compatible teleoperated robotic system. Adding the clutch has no effect on imaging quality and provides a fail‐safe brake to prevent the needle axis from dropping in the event of a power failure. As a second application, the clutch is integrated into a force‐controlled robotic gripper where it allows the motor to be turned off while maintaining a static grasping force. In both applications, the 20 ms response time of the clutch prototypes is advantageous to prevent any motion immediately after receiving a braking command. This work additionally presents details on the design and manufacturing process of the gecko‐inspired clutch, including a new, non‐uniform profile for the microscopic adhesive features. The fabricated prototypes are thin (305 µm per layer) and flexible. They provide a controllable, adhesive braking force of 60 kPa per layer. Multiple layers can be assembled to increase the braking force.

Parallel Dual-Hand Detection by Using Hand and Body Features for Robot Teleoperation

Visual hand-based robot teleoperation provides a powerful guarantee for robots to complete complex tasks. However, detection and distinction of dual hands on images are difficult because of the small differences between left and right hands. To solve this problem, a parallel dual-hand detection and distinction method that combines the features of hands with the relationship features between the dual hands and body pose is proposed to achieve robust and accurate dual-hand detection. This parallel dual-hand detection method includes a hand detection module, a body pose estimation module, and a fusion module. In the hand detection module, a hand detector that realizes fast and accurate hand detection by detecting the center and corner points of hands is designed. In the body pose estimation module, a body pose estimator with dual-hand positions is proposed. The fusion module is designed to fuse hand detection and dual-hand estimation results to achieve distinction between left and right hands. Finally, the parallel dual-hand detection method is applied to a bimanual robot teleoperation system by using a designed dual-hand teleoperation framework. The proposed parallel dual-hand detection method can achieve 98.54% mAP of hand detection with 18 frames per second on a custom dual-hand detection dataset, and the bimanual robot teleoperation method can achieve 95.4% average accuracy for teleoperation tasks. Experimental results show the high accuracy and speed of our proposed parallel dual-hand detection method and its practicability in bimanual robot teleoperation.

Design and Evaluation of an Intuitive Haptic Teleoperation Control System for 6-DoF Industrial Manipulators

Industrial robots are capable of performing automated tasks repeatedly, reliably and accurately. However, in some scenarios, human-in-the-loop control is required. In this case, having an intuitive system for moving the robot within the working environment is crucial. Additionally, the operator should be aided by sensory feedback to obtain a user-friendly robot control system. Haptic feedback is one way of achieving such a system. This paper designs and assesses an intuitive teleoperation system for controlling an industrial 6-DoF robotic manipulator using a Geomagic Touch haptic interface. The system utilises both virtual environment-induced and physical sensor-induced haptic feedback to provide the user with both a higher amount of environmental awareness and additional safety while manoeuvering the robot within its working area. Different tests show that the system is capable of fully stopping the manipulator without colliding with the environment, and preventing it from entering singularity states with Cartesian end effector velocities of up to 0.25 m/s. Additionally, an operator is capable of executing low-tolerance end effector positioning tasks (∼0.5 mm) with high-frequency control of the robot (∼100 Hz). Fourteen inexperienced volunteers were asked to perform a typical object removal and writing task to gauge the intuitiveness of the system. It was found that when repeating the same test for a second time, the participants performed 22.2% faster on average. The results for the second attempt also became significantly more consistent between participants, as the inter quartile range dropped by 82.7% (from 52 s on the first attempt to 9 s on the second).

After a Decade of Teleimpedance: A Survey

Despite the significant progress made in making robots more intelligent and autonomous, today, teleoperation remains a dominant robot control paradigm for the execution of complex and highly unpredictable tasks. Attempts have been made to make teleoperation systems stable, easy to use, and efficient in terms of physical interactions between the follower remote robot and the environment. In particular, the emergence of torque-controlled robots has permitted to regulate the interaction forces from a distance through direct force or impedance control, enabling them to engage in complex interaction tasks. Exploiting this feature, the concept of teleimpedance control was introduced as an alternative method to bilateral force-reflecting teleoperation. The aim was to create a feed-froward yet contact-efficient teleoperation by enriching the leader commands with desired impedance profiles while executing a task. Since then, the teleimpedance concept has found its way into a wide range of interface and controller designs, as well as application domains. Accordingly, after a decade of research progress, this survey aims to provide: first, a convenient introduction of the concept to new researchers in the field, second, consolidate the existing state-of-the-art for active researchers, third, and discuss the pros and cons of different methods in terms of interface and force feedback to provide guidelines for different applications and future developments.

Collision Risk Assessment and Operation Assistant Strategy for Teleoperation System

Teleoperation robots remain superior to fully automated robots in complicated and unstructured environments (e.g., in-orbit assembly). However, the collision risk is also greatly increased in these environments. Therefore, the teleoperation robot should possess the capability of collision risk perception and be configured with security assistance strategy to improve safety and efficiency. With this objective in mind, this paper proposes a collision risk assessment system based on fuzzy theory, which comprehensively considers the effects of shortest distance, operation speed, and delay time. The introduction of fuzzy theory makes the risk assessment results more accurate. Furthermore, this paper also proposes a creative discrete expandable bounding box method to make the calculation of the nearest distance between complicated obstacles and the robot more efficient and faster. Secondly, to improve the safety and efficiency of teleoperation, this paper proposes a set of teleoperation assistance strategies for teleoperation robots based on collision risk. The strategies include partial view highlighting, variation in the motion mapping ratio, and haptic risk warning, all of which are achieved in the graphical interactive interface. Finally, this paper verifies the proposed collision risk model and the operation assistant strategy through experiments. The results show that the proposed fuzzy collision risk model has a correct trend with risk factors, and the proposed operation assistance strategies can effectively reduce the robot collision risk and improve the safety of teleoperation robots. In conclusion, this research contributes to the collision risk assessment and obstacle avoidance assistance strategy of teleoperation robots.

Encouraging prosocial behavior from older adults through robot teleoperation: A feasibility study

IntroductionIn Japan, the social climate surrounding older adults has gotten worse as a result of the spread of COVID-19 and the growing isolation of older adults who are increasingly unable to engage in prosocial behavior through work and volunteering. This is detrimental to the physical and mental well-being of older adults. The purpose of this study is to look into robot teleoperation for older adults as a viable way to deal with these issues and overcome the barriers preventing older adults from engaging in prosocial behavior.Materials and methodsWe designed and tested a remote-control approach for dialogue agents that is appropriate for older adults as well as evaluating their impressions in a real-world setting. Twelve older adults participated in experiments in two separate locations, a children's center and the city ward office, where they could remotely teleoperate a robot and have conversations with the visitors. In the city ward office, the older adults had a conversation with the visitors and gave them information and trivia quizzes about the city. In the children's center, older adults had conversations with children regarding their age, family, their likes, and dislikes. A questionnaire and interview were set up after the experiments to understand their impressions of the system and to clarify how older adults feel about certain issues regarding remote-controlled work, starting a new job, social interaction, to what extent have older adults been affected by the pandemic, how and in what ways has it affected their involvement in society, and whether teleoperating a robot can be a suitable approach to encourage prosocial behavior from them through volunteer work and social engagement.ResultsThe results show that older adults have a strong desire to engage in volunteer work, but are hampered mainly by physical isolation resulting from COVID-19 restrictions and their declining physical and mental health. Their impressions of the teleoperation system were highly positive, as they enjoyed having conversations with children through the robot. With this teleoperation system, older adults were able to remote control a robot by themselves without major issues. It made interaction simpler as conversing with children through a robot added a layer of anonymity that allowed older adults to express themselves freely without worrying about how they are perceived by others in public.DiscussionOlder adults were able to successfully engage in prosocial behavior through remote-controlling a robot. The system seems to be effective at easing the physical barriers preventing older adults from engaging in volunteer work, which have worsened since the spread of COVID-19.