Teleoperation Applications Research Articles

Safety, Efficiency, and Mental Workload in Simulated Teledriving of a Vehicle as Functions of Camera Viewpoint.

Teleoperation services are expected to operate on-road and often in urban areas. In current teleoperation applications, teleoperators gain a higher viewpoint of the environment from a camera on the vehicle's roof. However, it is unclear how this viewpoint compares to a conventional viewpoint in terms of safety, efficiency, and mental workload. In the current study, teleoperators (n = 148) performed driving tasks in a simulated urban environment with a conventional viewpoint (i.e., the simulated camera was positioned inside the vehicle at the height of a driver's eyes) and a higher viewpoint (the simulated camera was positioned on the vehicle roof). The tasks required negotiating road geometry and other road users. At the end of the session, participants completed the NASA-TLX questionnaire. Results showed that participants completed most tasks faster with the higher viewpoint and reported lower frustration and mental demand. The camera position did not affect collision rates nor the probability of hard braking and steering events. We conclude that a viewpoint from the vehicle roof may improve teleoperation efficiency without compromising driving safety, while also lowering the teleoperators' mental workload.

Optimizing link-length fitting between an operator and a robot with compensation for habitual movements

Teleoperation has become increasingly important for the real-world application of robots. However, in current teleoperation applications, operators need to visually recognize any physical deviations between the robot and themselves and correct its operation accordingly. Even when humans move their bodies, achieving high positional accuracy can be difficult for them, which can limit their speed of movement and place a heavy burden on them. In this study, we proposed a parameter optimization method for feedforward compensation of the link-length deviation and operator's habitual body movements in leader–follower control in 3D space. To optimize the parameters, we used Digital Annealer developed by Fujitsu Ltd, which could rapidly solve the combinatorial optimization problem. The objective function minimized the difference between the hand positions of the robot and targets. Simulations verified that the proposed method could reduce hand positional differences.

Fabrication and Characterization of Pneumatic Unit Cell Actuators

In the realm of virtual and augmented reality (VR/AR) and teleoperation applications, haptic feedback plays a role in enhancing task performance. One of the main goals of this study is to simplify haptic device hardware while improving its capacity to provide various stimuli at different intensities. In response to these challenges, this research introduces the Pneumatic Unit Cell (PUC), a soft pneumatically driven device—a hollow silicone cylinder with the ability to provide both static-pressure and vibrotactile feedback. Furthermore, the Pneumatic Unit Cell’s design simplicity has the potential for scalability, modularity, and the flexibility to mount the device on any part of the human body. The focus of the current paper is to study PUCs as actuators and lay the foundation for future perceptual studies. The characterization studies encompass the fabrication and verification of the fabrication accuracy through dimensional measurements, characterizing PUCs under static-pressure conditions (measuring the free deflection and blocking force) and frequency conditions (measuring the free deflection). In the static-pressure conditions, we applied pressures ranging from 0 to 40 kPa to measure the free deflection and from 0 to 30 kPa to measure the blocking force. In the frequency conditions, we applied pressures of 10, 20, and 30 kPa with inflation/deflation rates varying between 0.5 Hz and 100 Hz. The measurements of free deflection under static-pressure conditions revealed that 0.9 mm and 1.2 mm PUCs exhibit a linear increase in free deflection with an increase in inflation pressure. The results of free-deflection measurements under the frequency conditions indicate a direct relationship between the free-deflection magnitude and applied pressure. The results also demonstrate an inverse relationship to the frequency of inflation/deflation. The characterization results demonstrate a broad range of free deflection observed under both static-pressure and frequency conditions, encouraging the potential application of Pneumatic Unit Cell actuators as haptic devices.

Mixed reality human teleoperation with device-agnostic remote ultrasound: Communication and user interaction

For many applications, remote guidance and telerobotics provide great advantages. For example, tele-ultrasound can bring much-needed expert healthcare to isolated communities. However, existing tele-guidance methods have serious limitations including either low precision for video conference-based systems, or high complexity and cost for telerobotics. A new concept called human teleoperation leverages mixed reality, haptics, and high-speed communication to provide tele-guidance that gives an expert nearly-direct remote control without requiring a robot. This paper provides an overview of the human teleoperation concept and its application to tele-ultrasound. The concept and its impact are discussed. A new approach to remote streaming and control of point-of-care ultrasound systems independent of their manufacturer is described, as is a high-speed communication system for the HoloLens 2 that is compatible with ResearchMode API sensor stream access. Details of these systems are shown in supplementary video demonstrations. Novel interaction methods enabled by HoloLens 2-based pose tracking are also introduced and tests of the communication and user interaction are presented. The results show continued improvement of the system compared to previous work in instrumentation, HCI, and communication. The system thus has good potential for tele-ultrasound, as well as possible other applications of human teleoperation including remote maintenance, inspection, and training. The remote ultrasound streaming and control application is made available open source.

Bilateral Teleoperation with a Shared Design of Master and Slave Devices for Robotic Excavators in Agricultural Applications

The main objective of this study is to develop a shared design of master and slave devices for bilateral teleoperation mechanisms used for robotic excavators in agricultural applications. In robot teleoperation research, many potential applications within controlled and hazardous environments come to light. Robots, with their capacity for remote control by human operators through master devices, often employ the master-slave teleoperation approach. This strategy finds frequent use across manufacturing, construction, and agriculture industries. The master-slave system necessitates two interdependent components that collaboratively steer the robot in real time. However, challenges arise when manipulating the robot arm proves challenging due to structural differences between the master device (such as a joystick) and the slave device (the robot arm). A stable operational framework must be established for the robot to function optimally. This teleoperation system employs a master device to govern the actions of the slave device, a dynamic that heavily influences the operational complexity. Hence, the focal point of this study is to enhance the master-slave algorithm for teleoperation applications that rely on controlling robot arm movements. Despite the differing dimensions of the master and slave devices, they both share a common structure. The kinematic model bridging these components must be intelligible to ensure user-friendliness, facilitating effortless robot control. Calculating the robot arm's end effector movement and positioning involves employing the forward kinematics of the arm, determined through Denavit-Hartenberg parameters and transformation matrices. By mitigating communication delays between the master and slave devices using a technique centered around the robot arm's end effector position, the effectiveness of teleoperation can be significantly improved. Our designed robot arm attains 80% to 100% precision across joints. In summary, streamlining the robot arm's structure and minimizing delays offers a route to bolstering both stability and efficiency in robotic movement.

Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

There is an increasingly urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations, concomitant with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices. The potential high value of medical telerobotic applications was also evident during the recent coronavirus pandemic and will grow in future. Robotic teleoperation satisfies the demands of the scenarios in which human access carries measurable risk, but human intelligence is required. An effective teleoperation system not only enables intuitive human-robot interaction (HRI) but ensures the robot can also be operated in a way that allows the operator to experience the “feel” of the robot working on the remote side, gaining a “sense of presence”. Extended reality (XR) technology integrates real-world information with computer-generated graphics and has the potential to enhance the effectiveness and performance of HRI by providing depth perception and enabling judgment and decision making while operating the robot in a dynamic environment. This review examines novel approaches to the development and evaluation of an XR-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions. It presents a strong review of XR-enhanced telerobotics for remote robotic applications; a particular focus of the review includes the use of integrated 2D/3D mixed reality with haptic interfaces to perform intuitive remote operations to remove humans from dangerous conditions. This review also covers primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can better control or interact with real robotic platforms using these devices and systems to extend the user’s reality and provide a more intuitive interface. The objective of this article is to present recent, relevant, common, and accessible frameworks implemented in research articles published on XR-enhanced telerobotics for industrial applications. Finally, we present and classify the application context of the reviewed articles in two groups: mixed reality–enhanced robotic telemanipulation and mixed reality–enhanced robotic tele-welding. The review thus addresses all elements in the state of the art for these systems and ends with recommended research areas and targets. The application range of these systems and the resulting recommendations is readily extensible to other application areas, such as remote robotic surgery in telemedicine, where surgeons are scarce and need is high, and other potentially high-risk/high-need scenarios.

A Mechanical Hand-Tracking System With Tactile Feedback Designed for Telemanipulation.

In this paper, we present a mechanical hand-tracking system with tactile feedback designed for fine manipulation in teleoperation scenarios. Alternative tracking methods based on artificial vision and data gloves have become an asset for virtual reality interaction. Yet, occlusions, lack of precision, and the absence of effective haptic feedback beyond vibrotactile still appear as a limit for teleoperation applications. In this work, we propose a methodology to design a linkage mechanism for hand pose tracking purposes, preserving complete finger mobility. Presentation of the method is followed by design and implementation of a working prototype, and by evaluation of the tracking accuracy using optical markers. Moreover, a teleoperation experiment involving a dexterous robotic arm and hand was proposed to ten participants. It investigated the effectiveness and repeatability of the hand tracking with combined haptic feedback during a proposed pick and place manipulation tasks.

Designing Gaze-Based Interactions for Teleoperation: Eye Stick and Eye Click

Perspective-taking and attentional switching are some of the ergonomic challenges that existing teleoperation human-machine interface designs need to address. This study developed two gaze interaction methods, the Eye Stick and the Eye Click, which were based on the joystick metaphor and the navigation metaphor, respectively, to be used in exocentric perspective teleoperation scenarios. We conducted two user studies to test the task performance and the subjective experience of the gaze interaction methods in a virtual ground vehicle teleoperation task. The results showed that compared with a traditional joystick design, the Eye Stick led to a shorter driving distance and the Eye Click led to less task time, and the gaze interaction methods had performance advantages in more difficult mazes. After multiple task sessions, the participants reported that the gaze interaction methods and the traditional joystick were similar in terms of task workload, perceived learnability, and satisfaction; however, the perceived usability of the Eye Stick was not as good as the Eye Click and the traditional joystick. In conclusion, both the Eye Stick and the Eye Click are feasible and promising gaze interaction methods for teleoperation applications with task performance advantages; however, more research is needed to optimize their user experience design.

A 3D Hand Attitude Estimation Method for Fixed Hand Posture Based on Dual-View RGB Images

This work provides a 3D hand attitude estimation approach for fixed hand posture based on a CNN and LightGBM for dual-view RGB images to facilitate the application of hand posture teleoperation. First, using dual-view cameras and an IMU sensor, we provide a simple method for building 3D hand posture datasets. This method can quickly acquire dual-view 2D hand image sets and automatically append the appropriate three-axis attitude angle labels. Then, combining ensemble learning, which has strong regression fitting capabilities, with deep learning, which has excellent automatic feature extraction capabilities, we present an integrated hand attitude CNN regression model. This model uses a Bayesian optimization based LightGBM in the ensemble learning algorithm to produce 3D hand attitude regression and two CNNs to extract dual-view hand image features. Finally, a mapping from dual-view 2D images to 3D hand attitude angles is established using a training approach for feature integration, and a comparative experiment is run on the test set. The results of the experiments demonstrate that the suggested method may successfully solve the hand self-occlusion issue and accomplish 3D hand attitude estimation using only two normal RGB cameras.

Coevolution of internal representations in physical human-robot orchestration – models of the surgeon and the robot in robotic surgery

In teleoperated Robot-Assisted Minimally-Invasive Surgery (RAMIS), a surgeon controls the movements of instruments inside the patient’s body via a pair of robotic joysticks. RAMIS has transformed many surgical disciplines, but its full potential is still to be realized. In this chapter we propose a pathway towards overcoming several bottlenecks that are related to transparency and stability of the teleoperation channels that mediate RAMIS. We describe the traditional system centered and the more recent human-centred approaches to teleoperation, and the special considerations for RAMIS as an application of teleoperation. However, the human-centered approach is still one sided view focusing on the surgeon but neglecting the learning capabilities of robotic systems. Hence, we consider a more general idea of physical human-robot orchestration with coevolution of mutual internal representations – of the human and the robot, and discuss it in comparison to human-human collaboration over teleoperated channels.

The impact of data sonification in virtual reality robot teleoperation

Virtual Reality (VR) is being increasingly used to provide a more intuitive and embodied approach to robotic teleoperation, giving operators a sense of presence in the remote environment. Prior research has shown that presence can be enhanced when additional sensory cues such as sound are introduced. Data sonification is the use of non-speech audio to convey information and, in the context of VR robot teleoperation, it has the potential to 1) improve task performance by enhancing an operator’s sense of presence and 2) reduce task load by spreading data between sensory modalities. Here we present a novel study methodology to investigate how the design of data sonification impacts on these important metrics and other key measures of user experience, such as stress. We examine a nuclear decommissioning application of robotic teleoperation where the benefits of VR in terms of spatial reasoning and task performance are desirable. However, as the operational environment is hazardous, a sense of presence may not be desirable as it can lead to heightened operator stress. We conduct a study in which we compare the effects of diegetic sounds (literal and established sonifications) with abstract sounds (non-established sonifications). Our findings show that the diegetic sounds decrease workload, whereas abstract sounds increase workload, and are more stressful. Additionally, and contrary to expectations, sonification does not impact presence. These findings have implications for the design of sonic environments in virtual reality.

A Teleoperation Framework Based on Heterogeneous Matching for Hydraulic Manipulator

In recent years, hydraulic manipulators have been more and more widely used in the field of industrial automation because of their high power-to-weight ratio. Compared with autonomous control, bilateral teleoperation is a more reliable way to operate hydraulic manipulators on harsh occasions. However, teleoperation application in the hydraulic field is still limited due to operation accuracy and master–slave heterogeneous problem. In this paper, a teleoperation framework based on master–slave heterogeneous matching and nonlinear precise motion control for hydraulic manipulator is proposed, including workspace mapping algorithm, velocity-limited interpolator, inverse kinematics solver, trajectory planner, model-based controller, and feedback torque generator. The high-precision teleoperation of the hydraulic manipulator in the situation of master–slave heterogeneity is finally realized based on the designed teleoperation framework. The specialized experiment is set up, and the results also show that the method proposed in this paper can satisfy the teleoperation requirements of the hydraulic manipulator with the maximum operating accuracy up to 0.02 m under the condition of master–slave heterogeneity, which can greatly improve the applicability of hydraulic manipulator teleoperation in the case of master–slave heterogeneity.

Haptic Feedback and Force-Based Teleoperation in Surgical Robotics

This article presents an overview of the current state of research and application of haptic (primarily kinesthetic) feedback and force-based teleoperation in the context of surgical robotics. Telerobotic surgery provides an approach for transferring the sensorimotor skills of a surgeon through a robotic platform to perform surgical intervention inside a patient’s body. Integration of advanced sensing and haptic technologies in telerobotic surgery can help to enhance the sensory awareness and motor accuracy of the surgeon, thereby leading to improved surgical procedures and outcomes for patients. The primary mode of sensory feedback has been through 3-D visual observation using stereo endoscopes. However, until recently, the sense of touch, i.e., haptics, has been missing in the commercial telesurgery robots approved for use in the operating room despite over two decades of research and development in the field of haptics for teleoperated systems (“telehaptics”). Research has shown that high-fidelity force feedback can enhance the performance of telesurgery and potential outcomes by enabling the surgeon to have a more natural feel of interaction between surgical tools and tissue as normally experienced during open surgery. Interaction forces, such as those generated during palpation of tissue, insertion of a needle, unintentional (and potentially unsafe) exertion of force by a tool, suture breakage, needle slippage, or tool interaction, are replaced by indirect (virtual) sensations, termed visual haptics, which provides an alternative to sensory compensation. Although there is a significant amount of literature supporting this benefit, there are still several important technical challenges in introducing haptics in telesurgery, including instrumentation, fidelity (transparency), stability, and modalities for force reflection, e.g., direct or indirect. This article examines these challenges and discusses recent work on haptics-based teleoperated surgical robotic systems.

A Telepresence-Guaranteed Control Scheme for Teleoperation Applications of Transferring Weight-Unknown Objects

Currently, most teleoperation work is focusing on scenarios where slave robots interact with unknown environments. However, in some fields such as medical robots or rescue robots, the other typical teleoperation application is precise object transportation. Generally, the object's weight is unknown yet essential for both accurate control of the slave robot and intuitive perception of the human operator. However, due to high cost and limited installation space, it is unreliable to employ a force sensor to directly measure the weight. Therefore, in this paper, a control scheme free of force sensor is proposed for teleoperation robots to transfer a weight-unknown object accurately. In this scheme, the workspace mapping between master and slave robot is firstly established, based on which, the operator can generate command trajectory on-line by operating the master robot. Then, a slave controller is designed to follow the master command closely and estimate the object's weight rapidly, accurately and robust to unmodeled uncertainties. Finally, for the sake of telepresence, a master controller is designed to generate force feedback to reproduce the estimated weight of the object. In the end, comparative experiments show that the proposed scheme can achieve better control accuracy and telepresence, with accurate force feedback generated in only 500 ms.

Object Detection and Distance Measurement in Teleoperation

In recent years, teleoperation has experienced rapid development. Numerous teleoperation applications in diverse areas have been reported. Among all teleoperation-related components, computer vision (CV) is treated as one of the must-have technologies, because it allows users to observe remote scenarios. In addition, CV can further help the user to identify and track the desired targets from complex scenes. It has been proven that efficient CV methods can significantly improve the operation accuracy and relieve user’s physical and mental fatigue. Therefore, furthering understanding about CV techniques and reviewing the latest research outcomes is necessary for teleoperation designers. In this context, this review article was composed.

Still Not Solved: A Call for Renewed Focus on User-Centered Teleoperation Interfaces.

Teleoperation is one of the oldest applications of human-robot interaction, yet decades later, robots are still difficult to control in a variety of situations, especially when used by non-expert robot operators. That difficulty has relegated teleoperation to mostly expert-level use cases, though everyday jobs and lives could benefit from teleoperated robots by enabling people to get tasks done remotely. Research has made great progress by improving the capabilities of robots, and exploring a variety of interfaces to improve operator performance, but many non-expert applications of teleoperation are limited by the operator’s ability to understand and control the robot effectively. We discuss the state of the art of user-centered research for teleoperation interfaces along with challenges teleoperation researchers face and discuss how an increased focus on human-centered teleoperation research can help push teleoperation into more everyday situations.

Toward Standard Guidelines to Design the Sense of Embodiment in Teleoperation Applications: A Review and Toolbox

ABSTRACT We present a literature review and a toolbox to help the reader find the best method to design for and assess Sense of Embodiment (SoE) in several application scenarios. The main examples are based on teleoperation applications, due the challenges that these applications present. The three embodiment components that we consider to describe SoE are sense of ownership, sense of agency, and sense of self-location. We relate each embodiment component to the most often used assessment measures, test tasks, and application scenarios. The toolbox is built to efficiently design, test, and assess an embodiment experience, following seven concrete steps. We provide four main contributions: 1) a literature review of the assessment measures and strategies used to measure SoE; 2) a systematic categorization of SoE measures; 3) A categorization of the main test tasks used in SoE assessment; and 4) a toolbox consisting of seven steps as guidance to design SoE. We included several examples and tables to guide the user step by step through the design of an embodiment experience.

Teleoperation and Visualization Interfaces for Remote Intervention in Space

Approaches to robotic manufacturing, assembly, and servicing of in-space assets range from autonomous operation to direct teleoperation, with many forms of semi-autonomous teleoperation in between. Because most approaches require one or more human operators at some level, it is important to explore the control and visualization interfaces available to those operators, taking into account the challenges due to significant telemetry time delay. We consider one motivating application of remote teleoperation, which is ground-based control of a robot on-orbit for satellite servicing. This paper presents a model-based architecture that: 1) improves visualization and situation awareness, 2) enables more effective human/robot interaction and control, and 3) detects task failures based on anomalous sensor feedback. We illustrate elements of the architecture by drawing on 10 years of our research in this area. The paper further reports the results of several multi-user experiments to evaluate the model-based architecture, on ground-based test platforms, for satellite servicing tasks subject to round-trip communication latencies of several seconds. The most significant performance gains were obtained by enhancing the operators’ situation awareness via improved visualization and by enabling them to precisely specify intended motion. In contrast, changes to the control interface, including model-mediated control or an immersive 3D environment, often reduced the reported task load but did not significantly improve task performance. Considering the challenges of fully autonomous intervention, we expect that some form of teleoperation will continue to be necessary for robotic in-situ servicing, assembly, and manufacturing tasks for the foreseeable future. We propose that effective teleoperation can be enabled by modeling the remote environment, providing operators with a fused view of the real environment and virtual model, and incorporating interfaces and control strategies that enable interactive planning, precise operation, and prompt detection of errors.

A novel hand exoskeleton to enhance fingers motion for tele-operation of a robot gripper with force feedback☆

This research describes a robotic system's design, optimization, and implementation processes for tele-operation with haptic feedback using the Hand Exoskeleton of New technologies research center (HEXON) as a novel haptic mechanism and the HEXON robotic gripper to work collaboratively with the exoskeleton. In particular, a force-sensitive robotic gripper is connected to a hand exoskeleton, mimics the user's hand movements, and transfers haptic feedback through the exoskeleton. The two-finger exoskeleton's design process consists of a multi-criteria optimization procedure to simultaneously maximize applied force to the finger and workspace of the attached exoskeleton, optimizing both the Perpendicular Impact Force (PIF) and the Global Isotropy Index (GII), considering worst-case collision avoidance. Due to the tele-operation application and using a robotic arm as a slave in the remote scenario, the gripper can be used as an end-effector for a palletizing 3-DOF robotic arm, and movement of the gripper can be achieved by using a visual sensor to pinpoint the user's hand location in 3D space. With the aim of the stress analysis software, the exoskeleton and gripper designs were analyzed to evaluate the system strength against applied forces. Finally, based on the achieved results, the gripper and two-fingered exoskeleton prototypes were fabricated. Simulations and experimental results are reported, showing the potential of the HEXON in haptics and tele-operation applications, and the exoskeleton is used to operate a robotic arm and gripper while the user can perceive the objects virtually.

A hybrid position–rate teleoperation system

Position resolution is a major problem in teleoperation applications with significant disparity between master and slave workspace. While rate mode control is suitable for slave free motion operations, it poses significant stability and performance challenges for task manipulation. In this paper, we propose a hybrid control scheme that offers seamless transition between position and rate control modes based on the environment location information collected from a range sensor. The system incorporates the strengths of position and rate control modes while masking their shortcomings. Experiments to determine the viability of this method are carried out on a single degree-of-freedom teleoperation test-bed.