Telerobotic System Research Articles

Secured energy efficient machine -to-machine communication for telerobotic system

Machine-to-Machine (M2M) communication refers to applications that run on intelligent devices or machines. It allows smart devices to communicate directly with each other with little or no human intervention.In a telerobotic system, medical devices interact with each other and exchange information automatically to provide visual assistance to physicians during surgery. Wireless communication provides attackers with numerous options to gain access to sensitive data and carry out malicious operations.A small communication delay in the telerobotic system can have a significant impact on the performance of medical equipment, potentially putting the patient's life in danger. Additionally, it can slow down the entire system in an operating room. Here, an energy-efficient and secured M2M network is presented to make the system robust.The system is designed with Event-Driven Duty Cycling (EDDC) routing protocol for enhancing the energy of medical sensor nodes. A Denial-of-Service (DoS) attack is performed and network performance is measured through Quality of Service (QoS) parameters. The results of the proposed routing protocol are compared with those of the LEACH and ECFU.The EDDC protocol outperforms even when attacks are launched.

Review of Advanced Medical Telerobots

The advent of telerobotic systems has revolutionized various aspects of the industry and human life. This technology is designed to augment human sensorimotor capabilities to extend them beyond natural competence. Classic examples are space and underwater applications when distance and access are the two major physical barriers to be combated with this technology. In modern examples, telerobotic systems have been used in several clinical applications, including teleoperated surgery and telerehabilitation. In this regard, there has been a significant amount of research and development due to the major benefits in terms of medical outcomes. Recently telerobotic systems are combined with advanced artificial intelligence modules to better share the agency with the operator and open new doors of medical automation. In this review paper, we have provided a comprehensive analysis of the literature considering various topologies of telerobotic systems in the medical domain while shedding light on different levels of autonomy for this technology, starting from direct control, going up to command-tracking autonomous telerobots. Existing challenges, including instrumentation, transparency, autonomy, stochastic communication delays, and stability, in addition to the current direction of research related to benefit in telemedicine and medical automation, and future vision of this technology, are discussed in this review paper.

Operation Status of Teleoperator Based Shared Control Telerobotic System

Telerobotic system is a typical human-machine system. The operation results not only depend on the performance of machine, but also the status of teleoperators (SoT). However, existing telerobotic systems scarcely consider the impact of teleoperators. This paper proposes a method for the online identification of the SoT and incorporates it to a shared control telerobotic system. First, some mental indicators are obtained based on Electroencephalogram during teleoperations. The relationship between the SoT and mental indicators is then established by a neural network. The online SoT identification is further implemented on a mobile telerobotic system. Second, a SoT based shared control framework is proposed in telerobotic system. The SoT is designed to dynamically adjust the control weight of the shared controller. Finally, comparative experiments are performed between a sensor based shared control method and the SoT based shared control method. The result validates the effectiveness of the proposed SoT based shared control method in telerobotic system.

Remote Assessment of Post-Stroke Elbow Function Using Internet-Based Telerobotics: A Proof-of-Concept Study.

Purpose: Upper limb hemiparesis is the most common impairment in stroke survivors, and adequate assessment is crucial for setting the rehabilitation strategy and monitoring the effect of treatment. However, adequate timely assessments are difficult due to the limited accessibility to clinics for stroke survivors. We designed this study to investigate whether teleassessments for motor impairments of the spastic elbow (i.e., passive range of motion (PROM), muscle strength, and spasticity) are feasible in stroke survivors.Methods: To implement a telerobotic system for remote assessment with physical interaction, we constructed a system with a master robot interacting with a doctor (assessor) and a slave robot interacting with the elbow of a subject with stroke. The master robot is operated by the doctor, where the torque and the speed are transferred to the slave robot via the Internet, and the reaction of the patient's elbow to the slave robot's movement is measured with a torque sensor, then finally transferred back to the master robot. An intercontinental remote assessment, which is considered one of the worst possible scenarios, was used as a clinical test to strictly check the feasibility. For the clinical tests, the examiner for the teleassessment was located at a lab in the National Institutes of Health (NIH, Bethesda, MD, USA) while the stroke patients were located at Seoul National University Bundang Hospital (Bundang, Kyeonggido, South Korea).Results: In total, 12 stroke patients' elbows (age range, 28–74; M:F = 6:6) were tested. For the PROM, the absolute difference between two assessments (in-person vs. remote) was 5.98 ± 3.51° on average (range, 0–11.2). The agreements for the strength and the spasticity of elbow flexor between in-person and remote assessments were substantial (k = 0.643) and fair (k = 0.308), respectively. No adverse events were observed during or immediately after the telerobotic assessment.Conclusions: Internet-based telerobotic remote assessment for motor impairment of spastic elbow in stroke using our system is feasible even in the worst setting, with too long of a distance and a delayed communication network.

Exploring The Effect of Visual Information Degradation on Human Perception and Performance In A Human-Telerobot System

Human-robotic interaction system are increasingly becoming integrated into industrial, commercial and emergency service agencies. It is critical that human operators understand and trust automation when these systems support and even make important decisions. The following study focused on human-in-loop telerobotic system performing a reconnaissance operation. Twenty-four subjects were divided into groups based on level of automation (Low-Level Automation (LLA), and High-Level Automation (HLA)). Results indicated a significant difference between low and high word level of control in hit rate when permanent error occurred. In the LLA group, the type of error had a significant effect on the hit rate. In general, the high level of automation was better than the low level of automation, especially if it was more reliable, suggesting that subjects in the HLA group could rely on the automatic implementation to perform the task more effectively and more accurately.

COVID-19 Pandemic Spurs Medical Telerobotic Systems: A Survey of Applications Requiring Physiological Organ Motion Compensation.

The coronavirus disease 2019 (COVID-19) pandemic has resulted in public health interventions such as physical distancing restrictions to limit the spread and transmission of the novel coronavirus, causing significant effects on the delivery of physical healthcare procedures worldwide. The unprecedented pandemic spurs strong demand for intelligent robotic systems in healthcare. In particular, medical telerobotic systems can play a positive role in the provision of telemedicine to both COVID-19 and non-COVID-19 patients. Different from typical studies on medical teleoperation that consider problems such as time delay and information loss in long-distance communication, this survey addresses the consequences of physiological organ motion when using teleoperation systems to create physical distancing between clinicians and patients in the COVID-19 era. We focus on the control-theoretic approaches that have been developed to address inherent robot control issues associated with organ motion. The state-of-the-art telerobotic systems and their applications in COVID-19 healthcare delivery are reviewed, and possible future directions are outlined.

Stability Analysis and Performance Evaluation of Delayed Bilateral Telerobotic Systems over a Lossy Communication Channel

The focus of this paper is to address a novel control technique for stability and transparency analysis of bilateral telerobotic systems in the presence of data loss and time delay in the communication channel. Different control strategies have been reported to compensate the effects of time delay in the communication channel; however, most of them result in poor performance under data loss. First, a model for data loss is proposed using a finite series representation of a set of periodic continuous pulses. To improve the performance and data reconstruction, a holder circuits is also introduced. The passivity of the overall system is provided via the wave variable technique based on the proposed model for the data loss. The stability analysis of the system is then derived using the Lyapunov theorem under the time delay and the data loss. Finally, experimental results are given to illustrate the capability of the proposed control technique.

Development of Robot-Assisted Untact Swab Sampling System for Upper Respiratory Disease

We propose a new telerobotic system for untact swab sampling to prevent the infection of medical staff during upper respiratory sample collection. The system consists of a slave robot and two master devices. The slave robot is designed to move a swab in 6 degrees of freedom within the facial area and to insert and remove the swab under remote control by an operator at the master site based on magnified imaging of the patient’s facial area. The insertion and removal of the swab into and from the nostril are implemented by means of a swab insertion unit; as the counterpart to this unit, the master system also includes a swab insertion device to control the swab insertion unit remotely. A force sensor installed on the swab holder enables monitoring of the force generated when the swab touches the target. In experiments, a virtual specimen was installed on the posterior nasopharynx wall of a life-size head phantom model. The nasopharyngeal swab samplings of the phantom model were successfully performed thanks to the force monitoring capability of the proposed telerobotic system, showing that this system is suitable for remote upper respiratory sample collection.

Perception of Tactile Directionality via Artificial Fingerpad Deformation and Convolutional Neural Networks.

Humans can perceive tactile directionality with angular perception thresholds of 14-40° via fingerpad skin displacement. Using deformable, artificial tactile sensors, the ability to perceive tactile directionality was developed for a robotic system to aid in object manipulation tasks. Two convolutional neural networks (CNNs) were trained on tactile images created from fingerpad deformation measurements during perturbations to a handheld object. A primary CNN regression model provided a point estimate of tactile directionality over a range of grip forces, perturbation angles, and perturbation speeds. A secondary CNN model provided a variance estimate that was used to determine uncertainty about the point estimate. A 5-fold cross-validation was performed to evaluate model performance. The primary CNN produced tactile directionality point estimates with an error rate of 4.3% for a 20° angular resolution and was benchmarked against an open-source force estimation network. The model was implemented in real-time for interactions with an external agent and the environment with different object shapes and widths. The perception of tactile directionality could be used to enhance the situational awareness of human operators of telerobotic systems and to develop decision-making algorithms for context-appropriate responses by semi-autonomous robots.

Shared mixed reality-bilateral telerobotic system

This study proposes a new shared mixed reality (MR)-bilateral telerobotic system. The main contribution of this study is to combine MR teleoperation and bilateral teleoperation, which takes advantage of the two types of teleoperation and compensates for each other’s drawbacks. With this combination, the proposed system can address the asymmetry issues in bilateral teleoperation, such as kinematic redundancy and workspace inequality, and provide force feedback, which is lacking in MR teleoperation. In addition, this system effectively supports long-distance movements and fine movements. In this system, a new MR interface is developed to provide the operator with an immersive visual feedback of the workspace, in which a useful virtual controller known as an interaction proxy—is designed. Compared with previous virtual reality-based teleoperation systems, this interaction proxy can freely decouple the operator from the control loop, such that the operational burden can be substantially alleviated. Additionally, the force feedback provided by the bilateral teleoperation gives the operator an advanced perception about the remote workspace and can improve task performance. Experiments on multiple pick-and-place tasks are provided to demonstrate the feasibility and effectiveness of the proposed system.

Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction

Robotic teleoperation, i.e., manipulating remote robotic systems at a distance, has gained its popularity in various industrial applications, including construction operations. The key to a successful teleoperation robot system is the delicate design of the human-robot interface that helps strengthen the human operator’s situational awareness. Traditional human-robot interface for robotic teleoperation is usually based on imagery data (e.g., video streaming), causing the limited field of view (FOV) and increased cognitive burden for processing additional spatial information. As a result, 3D scene reconstruction methods based on point cloud models captured by scanning technologies (e.g., depth camera and LiDAR) have been explored to provide immersive and intuitive feedback to the human operator. Despite the added benefits of applying reconstructed 3D scenes in telerobotic systems, challenges still present. Most 3D reconstruction methods utilize raw point cloud data due to the difficulty of real-time model rendering. The significant size of point cloud data makes the processing and transfer between robots and human operators difficult and slow. In addition, most reconstructed point cloud models do not contain physical properties such as weight and colliders. A more enriched control mechanism based on physics engine simulations is impossible. This paper presents an intelligent robot teleoperation interface that collects, processes, transfers, and reconstructs the immersive scene model of the workspace in Virtual Reality (VR) and enables intuitive robot controls accordingly. The proposed system, Telerobotic Operation based on Auto-reconstructed Remote Scene (TOARS), utilizes a deep learning algorithm to automatically detect objects in the captured scene, along with their physical properties, based on the point cloud data. The processed information is then transferred to the game engine where rendered virtual objects replace the original point cloud models in the VR environment. TOARS is expected to significantly improve the efficiency of 3D scene reconstruction and situational awareness of human operators in robotic teleoperation.

Assist-as-needed policy for movement therapy using telerobotics-mediated therapist supervision

In this paper, a new impedance-based teleoperation strategy is proposed for assist-as-needed tele-rehabilitation via a multi-DOF telerobotic system having patient–master and therapist–slave interactions. Unlike a regular teleoperation system and as the main contribution of this work to minimize the therapist’s movements, the therapist’s hand only follows the patient’s deviation from the target trajectory. Also it provides a better perception of the patient’s problems in motor control to the therapist The admissible deviation of the patient’s limb from a reference target trajectory is governed by an impedance model responding to both patient’s and therapist’s interaction forces. As the other benefit of this framework, two sources of assistance to the patient are delivered through the master robot: (1) the adjustable impedance model, and (2) the force applied by the therapist to the slave robot. The assistive impedance model is beneficial to reduce magnitudes of the required force from the therapist and decrease his/her intervention. This results in delaying and declining the therapist’s muscle fatigue in time-consuming movement therapies. Bilateral nonlinear control laws with two types of adaptation laws are designed for the nonlinear teleoperation system. The Lyapunov stability proof of the teleoperation system and the stability of the impedance model enhance the patient’s and therapist’s safety even in the presence of modeling uncertainties of the multi-DOF telerobotic system. The performance of the proposed bilateral impedance-based strategy is experimentally investigated using different impedance parameters adjusted based on the patient’s characteristics (e.g., involuntary tremor) and disabilities (e.g., insufficient actuation force). The experiments are performed by a healthy person (as the therapist), a mechanical test bed and a volunteer (simulating the patients’ characteristics). A new force–position mapping from Cartesian to Normal–Tangential (N–T) coordinates is utilized between the master and slave workspaces and compared with typical Cartesian to Cartesian projection.

Robotics, Smart Wearable Technologies, and Autonomous Intelligent Systems for Healthcare During the COVID‐19 Pandemic: An Analysis of the State of the Art and Future Vision

Herein, how robotic and autonomous systems and smart wearables complement and support healthcare delivery and the healthcare staff during the COVID‐19 pandemic are presented. For instance, robotic and telerobotic systems significantly reduce the risk of infectious disease transmission to frontline healthcare workers by making it possible to triage, evaluate, monitor, and treat patients from a safe distance. Various examples of where the medical, engineering, and science communities come together to aid the healthcare system, healthcare workers, and society during the current crisis are presented. The goal is to encourage an interdisciplinary dialog so that ethical, practical, and beneficial technological solutions are found to effectively tackle this and similar crises.

Evaluation of haptic devices and end-users: Novel performance metrics in tele-robotic microsurgery.

Here, we present performance evaluation methodology that distinguishes the performance of a haptic device from end-user skill level in a tele-robotic system. A pick-&-place experiment was designed and eight participants micromanipulated cotton strips, similar to maneuvers performed during microsurgery. Using three nonredundant haptic devices: neuroArmPLUS HD , a custom developed master manipulator, and two commercially available products, sigma.7 and HD2 , several features including the speed, effort, consistency, hand/gimbal agility, and force characteristics were measured and recorded for each participant and device. The participants showed variable skill level. For consistency, hand/gimbal agility and force characteristics, they performed significantly better when using neuroArmPLUS HD prototype. Based on the experimental data, performance metrics for both the device and the end-users were established. The integrated performance metrics allows independent evaluation of both the user and haptic device, thereby quantifying human-machine interactions.

Applications of Light-Weight Wearable Devices to Online Programming of Industrial Dual-Arm Robots

Dexterous manipulation of dual-arm robots in unstructured environments is very useful. Programming a dual-arm industrial robot to efficiently complete dexterous tasks, however, is especially challenging due to the complexity of its inverse kinematics, motion planning, dual-arm coordination with self-collision avoidance, and so on. This paper presents a systematic solution to accurately manipulate a dual-arm industrial robot on-site via light-weight wearable devices. In the developed system, the human operator directly drives the robot through the human arms motions tracked by the combination of inertial measurement units and handheld joystick controllers. A proper motion retargeting method with self-collision avoidance is used to enable the user to manipulate the robot directly through intuitive arm motions within a comfortable range and ensure the task manipulation with safety in unstructured environments. The developed system has been tested with various tasks, such as the manipulation of objects of different shapes, dexterous turn-over, and dual-arm coordination. Compared with the existing telerobotic systems, the developed system with simultaneous 14 degree-of-freedom teleoperation directly driven by light-weight wearable devices is able to handle more dexterous and accurate manipulation tasks with the capability of fast deployment and self-collision awareness. Such a solution could pave the way for online dual-arm robot programming on efficient manipulation skills transfer in the future.

Communication of Hazards in Mixed-Reality Telerobotic Systems: The Usage of Naturalistic Avoidance Cues in Driving Tasks.

This study investigates the effect of naturalistic visual cues on human avoidance behavior for a potential use in telerobotic user interfaces incorporating mixed-reality environments (e.g., augmented reality). Telerobotic systems used in hazardous environments require interfaces that draw operators' attention to potential dangers. Existing means of hazard notification can often distract or induce stress in operators. In the design and implementation of such interfaces, visual semiotics plays a critical role in creating more effective interfaces. Naturalistic visual cues such as Aposematism or Kindchenschema have proven effective to communicate danger or caution in nature, but the application of these cues in visual systems have yet to be thoroughly investigated. A study was conducted where 40 volunteering participants were asked to control a remote vehicle in a simulated environment. The environment contained a set of neutral and visually augmented obstacles that were designed to provoke avoidance behavior. The use of visual cues triggered greater avoidance behaviors in participants compared to neutral obstacles. The distance of avoidance was correlated with the type of cue present, with obstacles augmented by Aposematism (Cue A) having a greater participant-obstacle distance than Kindchenschema (Cue K). This study shows the potential for the incorporation of naturalistic visual cues as a means to designate warning or caution in telerobotic environments. The findings can offer practical guidelines for the design of visual cues in telerobotic interfaces. The further incorporation of such cues may reduce operator stress and the amount of human errors in telerobotic operations.

Toward Enhanced Teleoperation Through Embodiment.

Telerobotics aims to transfer human manipulation skills and dexterity over an arbitrary distance and at an arbitrary scale to a remote workplace. A telerobotic system that is transparent enables a natural and intuitive interaction. We postulate that embodiment (with three sub-components: sense of ownership, agency, and self-location) of the robotic system leads to optimal perceptual transparency and increases task performance. However, this has not yet been investigated directly. We reason along four premises and present findings from the literature that substantiate each of them: (1) the brain can embody non-bodily objects (e.g., robotic hands), (2) embodiment can be elicited with mediated sensorimotor interaction, (3) embodiment is robust against inconsistencies between the robotic system and the operator's body, and (4) embodiment positively correlates to dexterous task performance. We use the predictive encoding theory as a framework to interpret and discuss the results reported in the literature. Numerous previous studies have shown that it is possible to induce embodiment over a wide range of virtual and real extracorporeal objects (including artificial limbs, avatars, and android robots) through mediated sensorimotor interaction. Also, embodiment can occur for non-human morphologies including for elongated arms and a tail. In accordance with the predictive encoding theory, none of the sensory modalities is critical in establishing ownership, and discrepancies in multisensory signals do not necessarily lead to loss of embodiment. However, large discrepancies in terms of multisensory synchrony or visual likeness can prohibit embodiment from occurring. The literature provides less extensive support for the link between embodiment and (dexterous) task performance. However, data gathered with prosthetic hands do indicate a positive correlation. We conclude that all four premises are supported by direct or indirect evidence in the literature, suggesting that embodiment of a remote manipulator may improve dexterous performance in telerobotics. This warrants further implementation testing of embodiment in telerobotics. We formulate a first set of guidelines to apply embodiment in telerobotics and identify some important research topics.

Teleoperation of Collaborative Robot for Remote Dementia Care in Home Environments.

As a senile chronic, progressive and currently incurable disease, dementia has an enormous impact on society and life quality of the elderly. The development of teleoperation technology has changed the traditional way of care delivery and brought a variety of novel applications for dementia care. In this paper, a telerobotic system is presented which gives the caregivers the capability of assisting dementia elderly remotely. The proposed system is composed of a dual-arm collaborative robot (YuMi) and a wearable motion capture device. The communication architecture is achieved by the robot operation system (ROS). The position-orientation data of the operator’s hand are obtained and used to control the YuMi robot. Besides, a path-constrained mapping method is designed for motion trajectory tracking between the robot and the operator in the progress of teleoperation. Meanwhile, corresponding experiments are conducted to verify the performance of the trajectory tracking using the path-constrained mapping method. Results show that the position tracking deviation between the trajectory of the operator and the robot measured by dynamic time warping distance is 1.05 mm at the sampling frequency of 7.5 Hz. Moreover, the practicability of the proposed system was verified by teleoperating the YuMi robot to pick up a medicine bottle and further demonstrated by assisting an elderly woman in picking up a cup remotely. The proposed telerobotic system has potential utility for improving the life quality of dementia elderly and the care effect of their caregivers.

Adaptive Fault-Tolerant Prescribed-Time Control for Teleoperation Systems With Position Error Constraints

In this article, we present an adaptive prescribed-time control method for a class of nonlinear telerobotic systems with actuator faults and position error constraints. Extended from prescribed-time stability , practically prescribed-time stability (PPTS) is proposed for the first time aiming at stability analysis and control synthesis of nonlinear systems with disturbance and uncertainty. We show that, under the control scheme in the framework of PPTS, the system states are guaranteed to converge to a user-defined set (physically realizable) within user-defined settling time (physically realizable). Based on PPTS, an adaptive fault-tolerant controller is developed by integrating a novel exponential-type barrier Lyapunov function. Rigorous stability analysis based on back-stepping approach proves that, under the proposed control strategy, synchronization errors converge to a user-defined residual-set within predefined settling time and never exceed the prescribed range. Universal performance indexes, including the settling time, residual-set, accuracy, and overshoot, can be user-defined and only dependent on fewer user-defined parameters. Simulation results illustrate the effectiveness of the developed control scheme.

Time delay prediction for space telerobot system with a modified sparse multivariate linear regression method

Compared with general telerobot system, space telerobot system (STS) faces with more serious and complex time delay. In time delay prediction, the traditional sparse multivariate linear regressive (SMLR), auto regressive (AR), neural network (NN) and cubic polynomial model based (CBMB) approaches share lower efficiency or precision, when time delay is serious and complex. To solve this problem, time delay prediction for STS is discussed in this paper. Through analysis, time delay in STS can be divided to three parts: a) communication time delay; b) transmission time delay; c) processing time delay. Then, detailed varying rule and accurate statistics analysis are given for each parts of time delay. Based on the previous work, a modified SMLR is proposed and realizes the prediction of time delay. Compared with previous SMLR, AR, NN and CBMB approaches, the modified SMLR method shares more higher prediction efficiency or precision. Finally, a simulation example is given and the simulation results show the superiorities.