Telerobotic Research Articles

Robotic Manipulator‐Assisted Omnidirectional Augmented Reality for Endoluminal Intervention Telepresence

Robotic telemedicine can provide timely treatment to critical patients in geographically remote locations. However, owing to the lack of depth information, occlusion of instrument, and view direction limitations, visual feedback from the patient to the clinician is typically unintuitive, which affects surgical safety. Herein, an omnidirectional augmented reality (AR)‐assisted robotic telepresence for interventional medicine is developed. A monocular camera is used as an AR device with a manipulator, where virtual key anatomies and instruments are superimposed on video images using multiobject hand–eye calibration, iterative closest point, and images superposition algorithms. The view direction of the camera can be changed via the manipulator, which allows the key anatomies and instruments to be viewed from different directions. Structure‐from‐motion and multi‐view stereo algorithms are used to reconstruct the scene on the patient's side, thus providing a virtual reality (VR) interactive interface for the manipulator's safe teleoperation. Two different phantom experiments are conducted to validate the effectiveness of the proposed method. Finally, an ex vivo experiment involving a porcine lumen is performed, in which the operator can view the interventional flexible robot. The proposed method integrates omnidirectional AR and VR with robotic telepresence, which can provide intuitive visual feedback to clinicians.

Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

There is an increasingly urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations, concomitant with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices. The potential high value of medical telerobotic applications was also evident during the recent coronavirus pandemic and will grow in future. Robotic teleoperation satisfies the demands of the scenarios in which human access carries measurable risk, but human intelligence is required. An effective teleoperation system not only enables intuitive human-robot interaction (HRI) but ensures the robot can also be operated in a way that allows the operator to experience the “feel” of the robot working on the remote side, gaining a “sense of presence”. Extended reality (XR) technology integrates real-world information with computer-generated graphics and has the potential to enhance the effectiveness and performance of HRI by providing depth perception and enabling judgment and decision making while operating the robot in a dynamic environment. This review examines novel approaches to the development and evaluation of an XR-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions. It presents a strong review of XR-enhanced telerobotics for remote robotic applications; a particular focus of the review includes the use of integrated 2D/3D mixed reality with haptic interfaces to perform intuitive remote operations to remove humans from dangerous conditions. This review also covers primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can better control or interact with real robotic platforms using these devices and systems to extend the user’s reality and provide a more intuitive interface. The objective of this article is to present recent, relevant, common, and accessible frameworks implemented in research articles published on XR-enhanced telerobotics for industrial applications. Finally, we present and classify the application context of the reviewed articles in two groups: mixed reality–enhanced robotic telemanipulation and mixed reality–enhanced robotic tele-welding. The review thus addresses all elements in the state of the art for these systems and ends with recommended research areas and targets. The application range of these systems and the resulting recommendations is readily extensible to other application areas, such as remote robotic surgery in telemedicine, where surgeons are scarce and need is high, and other potentially high-risk/high-need scenarios.

DEMO in-vessel equatorial coils for power-exhaust and fast plasma control

This paper addresses the issue of magnetic control of the DEMO device through the equatorial in-vessel coils. While vertical stabilization and radial control are typical applications of equatorial coils in plasma control, the cumbersome compatibility of in-vessel divertor coils with the DEMO divertor structures and remote handling draws attention to the possibility of using the in-vessel equatorial coils also for active plasma wobbling, as an alternative to traditional strike-point sweeping. In this work, the technical specifications of the DEMO equatorial in-vessel coils and the design requirements of the vertical stabilization and radial control systems are assessed. Finally, an optimized strategy for efficient power-exhaust control through active wobbling in DEMO is proposed.

Design and experimental verification of an electro-hydraulic dual system with reliable resistance to γ-irradiation

Remote handling constitutes a critical technological facet in the domain of fusion reactors. However, exposure to γ rays inflicts irreversible deleterious effects on both electronic assemblies and structural materials. In the present investigation, we meticulously engineered an electro-hydraulic dual system tailored for remote handling applications within fusion reactors. Subsequently, the system was subjected to γ irradiation employing 60Co as the radioactive source. After exposure to a dose of 4 MGy, the electro-hydraulic dual joint system demonstrates sustained functional integrity. In contrast, isolated electric and hydraulic joints manifests minor oil seepage after exposure to a radiation dose of 5 MGy. Additionally, the effectiveness of the selected O-ring sealing material, ethylene propylene diene monomer (EPDM), exhibits a significant decrement upon irradiation to a dose of 1–3 MGy. The observed γ-radiation resilience is attributed to strategic material selection, specifically in the choice of O-ring, as well as metal reinforcement. The findings furnish valuable insights for the development of remote handling systems capable of operating under γ irradiation and strenuous mechanical conditions.

Unpacking older adults’ experiences using mobile telepresence robots: A systematic mixed-studies review

This review aimed to explore the experiences of mobile telepresence robots (MTRs) among older adults (OAs) using a systematic mixed-studies approach. Ten electronic databases were searched for published and unpublished studies from inception to January 2022. The mixed methods assessment tool was used to assess the quality of the studies. A results-based convergent synthesis was employed to integrate quantitative and qualitative results. A total of 3,816 records were retrieved, and 43 studies were included, involving 1,568 participants across 19 countries. More than half of the selected studies had high methodological quality. After integrating findings from seven qualitative subthemes and eight quantitative categories, four themes emerged: (1) enjoyable experience, (2) willingness and acceptance, (3) promoting social connectedness, and (4) improved psychosocial and physiological well-being. The MTRs show promising benefits for OAs and their families. Future research should improve the design and functionalities of MTRs to customize different OAs.

Robots2school: telepresence-mediated learning in the hybrid classroom – experiences in education support for children during cancer treatment: a qualitative study

Children with cancer experience recurring hospitalizations and isolation during treatment, which affect their school attendance. This study explores experiences of children with cancer, their classmates, and teachers with using the telepresence robot as a learning mediator in the hybrid classroom during treatment periods. 31 children with cancer (aged 7–17 years), 30 teachers, and 118 classmates participated in interviews and 19 h of participant observations were undertaken in nine classrooms. The Agential Realism Theory and Situational Analysis framed the data analysis. There was a single overarching theme, “Telepresence robot didactic,“ and five sub-themes (Telepresence mediated learning, school-home collaboration, hybrid robot teaching, intra-actions in class, and inclusive spatiality). This study advocates the complexity of telepresence robot didactics, emphasizing that numerous human and other factors must intra-act and work simultaneously to achieve optimal learning conditions for children during cancer treatment. This includes considerations such as modality availability for the remote child; the teacher’s understanding of telepresence robot didactic and hybrid learning; the classmate’s ability to involve the remote child in groupwork; the child’s own treatment protocol, the robot’s functionalities, and spatiality in the class. Strategies for use and the systematic surveillance of telepresence robots are needed to ensure that children during cancer treatment do not lag in academic achievement. This study proposes that children with cancer can continue participating in class while hospitalized or isolated and consequently reduce social and academic setbacks.

Remote maintenance scenario of the caesium ovens and beam source of the ITER heating neutral beam

Maintenance of the Heating Neutral Beam (HNB) injectors that are located in the ITER Neutral Beam (NB) cell needs Remote Handling (RH) operations because the NB cell is a radiation controlled area. This paper describes the remote maintenance sequences of the caesium ovens and the Beam Source (BS) of the ITER HNB by using the Neutral Beam Remote Handling System (NBRHS). Caesium oven maintenance is a regular maintenance that needs to be performed every Long-Term Maintenance (LTM) period as caesium is a consumable to produce caesium vapor to enhance negative ion production. The BS maintenance is a corrective maintenance to do if it fails during the lifetime of ITER. Maintenance of the caesium ovens and the BS involves many RH operations that require collaboration of the NBRHS subsystems and RH tools. Moreover, the maintenance operation needs to be done without spreading contamination from the HNB vessel to the NB cell. The developed maintenance scenario relies on the current NB cell environment and the existing interfaces in the plant system.

Experimental Assessment of Human-Robot Teaming for Multi-Step Remote Manipulation with Expert Operators

Remote robot manipulation with human control enables applications where safety and environmental constraints are adverse to humans (e.g. underwater, space robotics and disaster response) or the complexity of the task demands human-level cognition and dexterity (e.g. robotic surgery and manufacturing). These systems typically use direct teleoperation at the motion level, and are usually limited to low-DOF arms and 2D perception. Improving dexterity and situational awareness demands new interaction and planning workflows. We explore the use of human-robot teaming through teleautonomy with assisted planning for remote control of a dual-arm dexterous robot for multi-step manipulation, and conduct a within-subjects experimental assessment (n=12 expert users) to compare it with direct teleoperation with an imitation controller with 2D and 3D perception, as well as teleoperation through a teleautonomy interface. The proposed assisted planning approach achieves task times comparable with direct teleoperation while improving other objective and subjective metrics, including re-grasps, collisions, and TLX workload. Assisted planning in the teleautonomy interface achieves faster task execution, and removes a significant interaction with the operator’s expertise level, resulting in a performance equalizer across users. Our study protocol, metrics and models for statistical analysis might also serve as a general benchmarking framework in teleoperation domains. Accompanying video and reference R code: https://people.csail.mit.edu/cdarpino/THRIteleop/

Design and research of bilateral teleoperation for CFETR tile assembly using haptic feedback

Bilateral teleoperation is a control strategy used in remote handling (RH) to enable an operator to control a robot or system while perceiving the interactive force in the environment. To enhance the immersive operating experience by reducing time delays, a control system based on OROCOS and DDS has been presented to rapidly establish the bilateral teleoperation system with real-time performance. Additionally, a bilateral control algorithm combining closed-loop pose tracking and force feedback is proposed to improve the pose tracking accuracy of the slave robot and enhance the sense of telepresence. As a part of the pre-study on the tile assembly conducted by CMOR in Chinese Fusion Engineering Test Reactor (CFETR), a lightweight experimental setup has been constructed based on the maintenance scenario in EAST. Relative experiments have been conducted to validate the accuracy and effectiveness of the bilateral teleoperation system. The results demonstrate low latency and smooth control during the bilateral teleoperation process.

Toward AI-Enabled Commercial Telepresence Robots to Combine Home Care Needs and Affordability

Toward AI-Enabled Commercial Telepresence Robots to Combine Home Care Needs and Affordability

Design and Implementation of Engine Order Telegraph With RS485 Communication for Ship Maneuvering

EOT (Engine Order Telegraph) is the embodiment of a machine-human interface that is controlled by the captain of the ship for safe and sustainable ship maneuvers as a basic part of a connected remote propulsion system. Remote control systems on modern ships usually have a control transfer system that allows control to be transferred between locations. Remote control is usually possible from two locations: the platform and the Engine Control Room (ECR). Some ships do not have a remote control handle on the ECR. When in bridge control mode, the bridge handle directly controls the engine set point. When in Engine Control Room mode, the bridge handle sends a telegraph signal to the ECR and the ECR handle controls the control system set point. In local control, the remote control system is inactive and the bridge handle sends a telegraph signal to the local control position and the machine is operated by its manual control in the engine room. In this study, an EOT device will be realized using the Arduino Mega2560 microcontroller and data communication using the RS485 serial from the bridge to the engine control room.

Thermoresponsive double-network hydrogel/elastomer hybrid bilayer laminates with high interfacial toughness and controllable shape deformations

Thermoresponsive hydrogel-based actuators have shown great potential in tissue engineering, biosensors and soft robots, while the preparation of temperature-driven bilayer hydrogel actuators with rapid responseness and recovery properties remains a challenge. Inspired by the layered heterostructure of human skin, we developed a Janus bilayer structure actuator consisting of thermoresponsive double network (TDN) hydrogel and polydimethylsiloxane (PDMS) elastomer with tough interface adhesion. Specifically, PNaAMPS/P(NIPAM-co-AAm) TDN hydrogel via photocopolymerization of N-isopropyl acrylamide (NIPAM) and acrylamide (AAm) and reinforced by microgel poly(2-acrylamide-2-methylpropane sulfonate sodium salt) (PNaAMPS) was in situ grafted onto benzophenone-activated PDMS surface, exhibiting strong interface adhesion (628 Jm-2 adhesion energy). The thermoresponsive shape deformations of the double-layer structure actuator were controllable by adjusting hydrogel/elastomer thickness ratio and introduction of metal ions, taking advantage of the different swelling and shrinkage characteristics of the hydrogel layer and the elastomer layer. The TDN/PDMS bilayer structure actuator, which integrates the "Janus" structure, thermal responsiveness and photothermal conversion, provides new ideas for the application and remote control of soft robots and soft actuators.

Design and implementation of distributed middleware for CRAFT remote handling system

The software system of remote handling control system (RHCS) for the comprehensive research facility for fusion technology (CRAFT) consists of multiple subsystems and various device controllers running on different hosts or servers. A high-performance CRAFT remote handling (RH) middleware (CRMW) is designed to address the high real-time requirements for task response and data transmission in this distributed scenario, as well as the scalability of the RHCS. CRMW adopts a low-coupling, high-cohesion, and component-based architectural design, comprising a component-based software development framework and a real-time communication port. The robot framework allows encapsulating devices or functional modules in the RHCS as reusable components, while the communication port provides low-latency and stable communication quality. It supports various communication modes such as publish/subscribe, server/client, and multiple transmission protocols, while also offering distributed components like remote procedure call (RPC). Corresponding experimental results demonstrate that CRMW can deliver low-latency and stable task response and data transmission, fully meeting the requirements for task execution and system management in the RHCS.

Using Footpad Sculpturing to Enhance the Maneuverability and Speed of a Robotic Marangoni Surfer.

From insects to arachnids to bacteria, the surfaces of lakes and ponds are teaming with life. Many modes of locomotion are employed by these organisms to navigate along the air-water interface, including the use of lipid-laden excretions that can locally change the surface tension of the water and induce a Marangoni flow. In this paper, we improved the speed and maneuverability of a miniature remote-controlled robot that mimics insect locomotion using an onboard tank of isopropyl alcohol and a series of servomotors to control both the rate and location of alcohol release to both propel and steer the robot across the water. Here, we studied the effect of a series of design changes to the foam rubber footpads, which float the robot and are integral in efficiently converting the alcohol-induced surface tension gradients into propulsive forces and effective maneuvering. Two designs were studied: a two-footpad design and a single-footpad design. In the case of two footpads, the gap between the two footpads was varied to investigate its impact on straight-line speed, propulsion efficiency, and maneuverability. An optimal design was found with a small but finite gap between the two pads of 7.5 mm. In the second design, a single footpad without a central gap was studied. This footpad had a rectangular cut-out in the rear to capture the alcohol. Footpads with wider and shallower cut-outs were found to optimize efficiency. This observation was reinforced by the predictions of a simple theoretical mechanical model. Overall, the optimized single-footpad robot outperformed the two-footpad robot, producing a 30% improvement in speed and a 400% improvement in maneuverability.

Research on the Dynamic Control Method of CFETR Multi-Purpose Overload Robot

The CFETR multi-purpose overload robot (CMOR) is a key subsystem of the remote handling system of the China fusion engineering test reactor (CFETR). This paper first establishes the kinematic and dynamic models of CMOR and analyzes the working process in the vacuum chamber. Based on the uncertainty of rigid-flexible coupling, a CMOR adaptive robust sliding mode controller (ARSMC) is designed based on the Hamilton-Jacobi equation to enhance the robustness of the control system. In addition, to compensate the influence of non-geometric factors on position accuracy, an error compensation method is designed. Based on the matrix differentiation method, the CMOR coupling parameter errors are decoupled, and then the gridded workspace principle is used to identify the parameter errors and improve the motion control accuracy. Finally, the CMOR rigid-flexible coupling simulation system is established by ADAMS-MATLAB/Simulink to analyze the dynamic control effect of ARSMC. The simulation results show that the CMOR end position error exceeds 0.1 m for single joint motion. The average value of CMOR end position error is less than 0.025 m after compensation, and the absolute error value is reduced by 4 times, improves the dynamic control accuracy of CMOR.

Task-oriented arm training for stroke patients based on remote handling technology concepts: A feasibility study.

Improving arm-hand skill performance is a major therapeutic target in stroke rehabilitation. Arm-hand rehabilitation may be enriched in content and variation by using technology-assisted training. Especially for people with a severely affected arm, technology-assisted training offers more challenging training possibilities. The aim of this study was to explore the feasibility of ReHab-TOAT, a "Remote Handling Based Task-Oriented Arm Training" approach featuring enriched haptic feedback aimed at improving daily activities and participation. Five subacute or chronic stroke patients suffering moderate to severe arm-hand impairments and five rehabilitation therapists participated. All participants received 2 ReHab-TOAT sessions. Outcome measure was a bespoke feasibility questionnaire on user experiences and satisfaction regarding 'motivation', 'individualization of training', 'potential training effects', and 'implementation in rehabilitation' of patients and therapists. Both patients and therapists experienced ReHab-TOAT as being feasible. They found ReHab-TOAT very motivating and challenging. All patients perceived an added value of ReHab-TOAT and would continue the training. Small improvements regarding exercise variability were suggested. ReHab-TOAT seems to be a feasible and very promising training approach for arm-hand rehabilitation of stroke patients with a moderately or severely affected arm. Further research is necessary to investigate potential training effects of ReHab-TOAT.

Multi-site domestication: taming technologies across multiple institutional settings

ABSTRACT This article advances domestication theory by developing the concept of multi-site domestication. Whereas domestication theory traditionally focuses on the ‘taming’ of technologies at a single site (most often, the household), the concept of multi-site domestication captures how technologies often require different taming processes across multiple institutional settings. In this article, we apply the concept to understand the multi-site domestication of AV1: a communication solution for children who are homebound because of chronic illness or disabilities, which creates a communicative bridge from an app on the homebound student’s phone/tablet and to a ‘telepresence robot’ that is placed physically in the classroom, where it is meant to function as the homebound student’s proxy. Using data from a larger qualitative study of the implementation of AV1 in Norway, the article shows how the ‘traditional’ domestication processes of appropriation, objectification, incorporation, and conversion play out and are complicated when domestication occurs across settings with different and at times opposing norms, rules, values, and logics. In charting these multi-site dynamics, the article updates domestication theory for an age of increasingly intertwined technologies, thus helping future studies to look beyond single sites and appreciate more complex taming processes.

The social robot? Analyzing whether and how the telepresence robot AV1 affords socialization

Telepresence robots are increasingly used in schools as a way of including students who are unable to be physically present in the classroom with other students. The use of such robots is intended not just to help students follow their education but also to serve a social purpose. However, the extent to which the robots actually afford socializing needs to be explored further. This article analyzes how, to what extent, for whom, and under what circumstances the telepresence robot AV1 affords social contact for the heterogenous group of homebound Norwegian upper secondary school students. Building on Jenny Davis’ mechanisms and conditions framework of affordances, we focus on how AV1 affords for different students in specific circumstances. Our analysis draws on interviews with 11 upper secondary school students in Norway and finds that individual traits and circumstances such as health issues and social networks are important aspects when assessing whether a technology affords socializing. Based on our findings, we argue for expanding the mechanisms and conditions framework to include not just its current focus on perception, dexterity, and cultural and institutional legitimacy, but also the users’ emotions.

Preliminary architecture of the DTT remote handling test and training facility

The realization of a specific REMote HAndling Test and Training Facility (REMHAT) is necessary for the development and optimization of the Remote Handling (RH) system, to demonstrate the fulfilment of all its design requirements, for testing and validation of the RH maintenance procedures, for the qualification and acceptance testing of RH equipment and tooling. Furthermore, it provides training and certification to the RH maintenance operators, by making available to operators the knowledge, instructions and experiencing of the Virtual Simulator platform, the control room devices, the robots and tooling to be used for RH tasks. In this work, the preliminary architecture of the DTT REMHAT facility is described, with a focus on the added value of the adoption of Virtual Reality (VR) technologies enabling real-time alignment of virtual robots’ configuration with the real ones (Digital Twin). A first approach to the facility design is proposed, starting from the technical requirements: Environmental, Design, Physical, Functional, Operational, Human Factors and Product assurance and safety. The proposed logical architecture consists of four Sub-Systems (S/S): building and auxiliaries, mock-ups, robots and Control Room. In the present work, a particular attention is paid on the last one. The idea is to create a digital twin of the RH equipment mock-ups to be coupled with software of real-time structural simulation, in order to have at disposition and show in VR environment real-time additional information about the real environment, like robots’ deflections and/or vibrations, clearances and possible collisions detection. These data could be crucial for the operator’s decision-making process about a sudden intervention on the RH system, further to their employment for future studies and optimization. Therefore, for each subsystem of the facility, the components identification and a description of the software and hardware aspects are provided. In conclusion, an overview of the critical issues of such complex system is presented, in order to prepare the ground for future research works.

Cost-effectiveness of remote robotic mechanical thrombectomy in acute ischemic stroke.

Clinical outcomes following endovascular thrombectomy (EVT) for acute ischemic stroke (AIS) treatment are highly time sensitive. Remote robotic (RR)-EVT systems may be capable of mitigating time delays in patient transfer from a primary stroke center (PSC) to a comprehensive/thrombectomy-capable stroke center. However, health economic evidence is needed to assess the costs and benefits of an RR-EVT system. Therefore, the authors of this study aimed to determine whether performing RR-EVT in suspected AIS patients at a PSC as opposed to standard of care might translate to cost-effectiveness over a lifetime. An economic evaluation study was performed from a US healthcare perspective, combining decision analysis and Markov modeling methods over a lifetime horizon to evaluate the cost-effectiveness of RR-EVT in suspected AIS patients at a PSC compared to the standard-of-care approach. Total expected costs and quality-adjusted life-years (QALYs) were estimated. In the cost-effectiveness analysis, RR-EVT yielded greater effectiveness per patient (4.05 vs 3.88 QALYs) and lower costs (US$321,269 vs US$321,397) than the standard-of-care approach. Owing to these lower costs and greater health benefits, RR-EVT was the dominant cost-effective strategy. After initiation of an RR-EVT system, the average costs per year were similar (or slightly reduced), according to this simulation. Sensitivity analyses revealed that RR-EVT remains cost-effective in a wide variety of time delays and cost assumptions. In a one-way sensitivity analysis, RR-EVT remained the most cost-effective strategy when time delays were greater than 2.5 minutes, its complication rate did not exceed 37%, and costs were lower than $54,081. When the cost of the RR-EVT strategy ranged from $19,340 to $54,081 and its complication rate varied from 15% to 37%, the RR-EVT strategy remained the most cost-effective throughout the two ranges. RR-EVT was also the most cost-effective strategy even when its cost doubled (to approximately $40,000) and time delays exceeded 20 minutes. In a probabilistic sensitivity analysis, RR-EVT was the long-term cost-effective strategy in 89.8% of iterations at a willingness-to-pay threshold of $100,000/QALY. This analysis suggests that RR-EVT as an innovative solution to expedite EVT is cost-effective. An RR-EVT system could potentially extend access to care in underserved communities and rural areas, as well as improve care for socioeconomically disadvantaged populations affected by health inequities.