Teleoperation Interface Research Articles

A virtual reality-based immersive teleoperation system for remote human-robot collaborative manufacturing

In recent years, the cutting-edge technologies in smart manufacturing have presented promising opportunities for the utilization of human-robot collaborative teleoperation in personalized manufacturing tasks. To effectively leverage the creative capabilities of humans while benefiting from the efficiency and stability of robots, the provision of an intuitive teleoperation interface assumes paramount importance. However, current teleoperation systems still face limitations in terms of intuitive operability. In this study, we present a virtual reality-based teleoperation system that offers operators a more intuitive interaction platform for robot control, thereby facilitating personalized manufacturing processes. The overall system framework design, as well as the main components are elaborated in detail. Furthermore, an evaluative case study based on the battery disassembly task is conducted to assess the performance of the proposed system. The results demonstrate that the proposed teleoperation system exhibits improved intuitiveness.

Design of multi-modal feedback channel of human–robot cognitive interface for teleoperation in manufacturing

AbstractTeleoperation, which is a specific mode of human–robot collaboration enabling a human operator to provide instructions and monitor the actions of the robot remotely, has proved beneficial for application to hazardous and unstructured manufacturing environments. Despite the design of a command channel from human operators to robots, most existing studies on teleoperation fail to focus on the design of the feedback channel from the robot to the human operator, which plays a crucial role in reducing the cognitive load, particularly in precise and concentrated manufacturing tasks. This paper focuses on designing a feedback channel for the cognitive interface between a human operator and a robot considering human cognition. Current studies on human–robot cognitive interfaces in robot teleoperation are extensively surveyed. Further, the modalities of human cognition that foster understanding and transparency during teleoperation are identified. In addition, the human–robot cognitive interface, which utilizes the proposed multi-modal feedback channel, is developed on a teleoperated robotic grasping system as a case study. Finally, a series of experiments based on different modal feedback channels are conducted to demonstrate the effectiveness of enhancing the performance of the teleoperated grasping of fragile products and reducing the cognitive load via the objective aspects of experimental results and the subjective aspects of operator feedback.

Opportunities and Challenges in Using Tangible, Teleoperated Voice Agents in Kid-Driven Moments in Play Among Families with Neurodivergent Children

Child-agent interaction research has largely focused on automated voice agents, although privacy concerns and quality of interactions serve as a rationale for using teleoperated agents in some settings. Research on neurodivergent children's interactions with voice agents has largely focused on therapy and classroom contexts as opposed to home contexts. We conducted a remote case study via Zoom observing six families with neurodivergent children playing with a tangible, parent-operated voice agent at home for up to eight weeks. We used quantitative measures of agent utilization, qualitative analysis of video recordings, and parent survey responses to identify implications for teleoperation interface design for playful voice agents. Based on our analysis, we argue joint media engagement principles are applicable to a broader set of play activities, and teleoperated agents present affordances and challenges distinct from those of automated agents.

Robotic Manipulator‐Assisted Omnidirectional Augmented Reality for Endoluminal Intervention Telepresence

Robotic telemedicine can provide timely treatment to critical patients in geographically remote locations. However, owing to the lack of depth information, occlusion of instrument, and view direction limitations, visual feedback from the patient to the clinician is typically unintuitive, which affects surgical safety. Herein, an omnidirectional augmented reality (AR)‐assisted robotic telepresence for interventional medicine is developed. A monocular camera is used as an AR device with a manipulator, where virtual key anatomies and instruments are superimposed on video images using multiobject hand–eye calibration, iterative closest point, and images superposition algorithms. The view direction of the camera can be changed via the manipulator, which allows the key anatomies and instruments to be viewed from different directions. Structure‐from‐motion and multi‐view stereo algorithms are used to reconstruct the scene on the patient's side, thus providing a virtual reality (VR) interactive interface for the manipulator's safe teleoperation. Two different phantom experiments are conducted to validate the effectiveness of the proposed method. Finally, an ex vivo experiment involving a porcine lumen is performed, in which the operator can view the interventional flexible robot. The proposed method integrates omnidirectional AR and VR with robotic telepresence, which can provide intuitive visual feedback to clinicians.

Perception-Motion Coupling in Active Telepresence: Human Behavior and Teleoperation Interface Design

Teleoperation enables complex robot platforms to perform tasks beyond the scope of the current state-of-the-art robot autonomy by imparting human intelligence and critical thinking to these operations. For seamless control of robot platforms, it is essential to facilitate optimal situational awareness of the workspace for the operator through active telepresence cameras. However, the control of these active telepresence cameras adds an additional degree of complexity to the task of teleoperation. In this paper we present our results from the user study that investigates: (1) how the teleoperator learns or adapts to performing the tasks via active cameras modeled after camera placements on the TRINA humanoid robot; (2) the perception-action coupling operators implement to control active telepresence cameras, and (3) the camera preferences for performing the tasks. These findings from the human motion analysis and post-study survey will help us determine desired design features for robot teleoperation interfaces and assistive autonomy.

Enhanced Teleoperation Interfaces for Multi-Second Latency Conditions: System Design and Evaluation

Adding human cognitive skills to planetary exploration through remote teleoperation can lead to more valuable scientific data acquisition. Still, even small amounts of latency can significantly affect real-time operations, often leading to compromised robot safety, goal overshoot, and high levels of cognitive workload. Thus, novel operational strategies are necessary to cope with these effects. This paper proposes three augmented teleoperation interfaces that allow the user to operate a robot subject to 3 seconds of latency: (1) Avatar-Aided Interface (AAI), a semi-autonomous approach based on a virtual element; (2) Predictive Interface (PI), an approach with direct control and predictive elements; and (3) Hybrid Interface (HI), where operators can easily switch between PI and AAI. We conducted a systematic within-subject experiment to evaluate the proposed interfaces in a realistic virtual environment with frequent traction losses. The user study compared AAI and PI to a Control Interface (CI), which did not display any augmented elements. The main results of this comparison showed that: (1) AAI led to a significant reduction in workload and a significant increase in usability and robot safety; (2) the use of the PI caused a significant increase in path length, indicating that operators overshoot their goals more often with this approach; (3) PI and AAI had lower reported effort; and (4) AAI is more flexible and effortless than PI and CI. Finally, during traction loss periods, PI and AAI had shortcomings that led to confusion from the operator, showing the need to integrate uncertainty measures in future interface design.

Hand Gesture-based Teleoperation Control of a Mecanum-wheeled Mobile Robot

The aim of this study is to design an interface for the teleoperation control of a Mecanum-wheeled Mobile Robot (MMR). To make a cost-effective and non-invasive teleoperation interface approach, a hand gesture recognition strategy via a webcam is adopted. In this study, seven hand gestures are recognized on a desktop PC and then the corresponding velocity commands are sent to the MMR via WiFi. The SqueezeNet architecture is employed for the hand gesture recognition and MMR teleoperation task after various Convolutional Neural Network (CNN) models are compared. The comparison considers some aspects, i.e., accuracy, memory usage, and recognition speed, to find the most feasible method for real-time hand gesture recognition and teleoperation control. Experimental results are presented to show the excellence of the proposed SqueezeNet architecture.

A Novel 6-DOF Force-Sensed Human-Robot Interface for an Intuitive Teleoperation

The teleoperation of a 6 degrees-of-freedom (DOF) manipulator is one of the basic methods to extend people’s capabilities in the wide variety of applications. The master interface based on the force/torque (FT) sensor could provide the full-dimension intuitive teleoperation of a 6-DOF robot since it has the ability to trigger 6-DOF command input. However, due to the force coupling, noise disturbance and unlimited input signals of the FT sensor, this force-sensed interface could not be widely used in practice. In this paper, we present an intuitive teleoperation method based on the FT sensor to overcome these challenges. In this method, the input signals from the force-sensed joystick were filtered and then processed to the force commands by force limit algorithm, with the merits of anti-interference, output limitation, and online velocity adjustment. Furthermore, based on the admittance control and position controller, the manipulator could be teleoperated by the force commands. Three experiments were conducted on our self-designed robotic system. The result of the first experiment shows that the interfered force from the force coupling could be effectively suppressed with the limitation of the input force through force limit algorithm. Then, a parameter was introduced in the other two experiments to adjust the velocity online practically with force limit algorithm. The proposed method could give a practical solution to the intuitive teleoperation based on the FT sensor.

Performance evaluation of a tactile and kinesthetic finger feedback system for teleoperation

Teleoperation during a catastrophic event requires an interface that can perform under frequently changing circumstances caused by unpredictable and dangerous conditions. Thus, teleoperation interfaces are under active development to provide both visual and haptic feedback to the fingers. However, studies of teleoperation systems with finger haptic feedback based on force profiles are difficult to conduct because of interface limitations. Therefore, in this paper, we introduce an intuitive teleoperation interface, an anthropomorphic teleoperated robot, and a hand-wearable force-feedback system that provides various feedbacks to the fingers. We combined these systems to compare and evaluated the performance of tactile and kinesthetic finger feedback using two experiments: maintaining appropriate grip force for variably fragile objects and following a force trajectory that changed in real time. Ten subjects participated in the experiments. The results were analyzed using repeated measures analysis of variance. Feedback factors differed significantly. Provision of force feedback to the user’s finger was most effective in both teleoperation experiments.

Intuitive, Efficient and Ergonomic Tele-Nursing Robot Interfaces: Design Evaluation and Evolution

Tele-nursing robots provide a safe approach for patient-caring in quarantine areas. For effective nurse–robot collaboration, ergonomic teleoperation and intuitive interfaces with low physical and cognitive workload must be developed. We propose a framework to evaluate the control interfaces to iteratively develop an intuitive, efficient, and ergonomic teleoperation interface. The framework is a hierarchical procedure that incorporates general to specific assessment and its role in design evolution. We first present pre-defined objective and subjective metrics used to evaluate three representative contemporary teleoperation interfaces. The results indicate that teleoperation via human motion mapping outperforms the gamepad and stylus interfaces. The tradeoff with using motion mapping as a teleoperation interface is the non-trivial physical fatigue. To understand the impact of heavy physical demand during motion mapping teleoperation, we propose an objective assessment of physical workload in teleoperation using electromyography. We find that physical fatigue happens in the actions that involve precise manipulation and steady posture maintenance. We further implemented teleoperation assistance in the form of shared autonomy to eliminate the fatigue-causing component in robot teleoperation via motion mapping. The experimental results show that the autonomous feature effectively reduces the physical effort while improving the efficiency and accuracy of the teleoperation interface.

Augmented Reality-Based Interface for Bimanual Robot Teleoperation

Teleoperation of bimanual robots is being used to carry out complex tasks such as surgeries in medicine. Despite the technological advances, current interfaces are not natural to the users, who spend long periods of time in learning how to use these interfaces. In order to mitigate this issue, this work proposes a novel augmented reality-based interface for teleoperating bimanual robots. The proposed interface is more natural to the user and reduces the interface learning process. A full description of the proposed interface is detailed in the paper, whereas its effectiveness is shown experimentally using two industrial robot manipulators. Moreover, the drawbacks and limitations of the classic teleoperation interface using joysticks are analyzed in order to highlight the benefits of the proposed augmented reality-based interface approach.

Quantitative Physical Ergonomics Assessment of Teleoperation Interfaces

Human factors and ergonomics are the essential constituents of teleoperation interfaces, which can significantly affect the human operator’s performance. Thus, a quantitative evaluation of these elements and the ability to establish reliable comparison bases for different teleoperation interfaces are the keys to select the most suitable one for a particular application. However, most of the works on teleoperation have so far focused on the stability analysis and the transparency improvement of these systems and do not cover the important usability aspects. In this article, we propose a foundation to build a general framework for the analysis of human factors and ergonomics in employing diverse teleoperation interfaces. The proposed framework will go beyond the traditional subjective analyses of usability by complementing it with online measurements of human body configurations. As a result, multiple quantitative metrics, such as joints’ usage, range of motion comfort, center of mass divergence, and posture comfort, are introduced. To demonstrate the potential of the proposed framework, two different teleoperation interfaces are considered, and real-world experiments with 11 participants performing a simulated industrial remote pick-and-place task are conducted. The quantitative results of this analysis are provided, and compared with subjective questionnaires, illustrating the effectiveness of the proposed framework.

Still Not Solved: A Call for Renewed Focus on User-Centered Teleoperation Interfaces.

Teleoperation is one of the oldest applications of human-robot interaction, yet decades later, robots are still difficult to control in a variety of situations, especially when used by non-expert robot operators. That difficulty has relegated teleoperation to mostly expert-level use cases, though everyday jobs and lives could benefit from teleoperated robots by enabling people to get tasks done remotely. Research has made great progress by improving the capabilities of robots, and exploring a variety of interfaces to improve operator performance, but many non-expert applications of teleoperation are limited by the operator’s ability to understand and control the robot effectively. We discuss the state of the art of user-centered research for teleoperation interfaces along with challenges teleoperation researchers face and discuss how an increased focus on human-centered teleoperation research can help push teleoperation into more everyday situations.

A Brain-Computer Interface for Teleoperation of a Semiautonomous Mobile Robotic Assistive System Using SLAM

The proposed assistive hybrid brain-computer interface (BCI) semiautonomous mobile robotic arm demonstrates a design that is (1) adaptable by observing environmental changes with sensors and deploying alternate solutions and (2) versatile by receiving commands from the user’s brainwave signals through a noninvasive electroencephalogram cap. Composed of three integrated subsystems, a hybrid BCI controller, an omnidirectional mobile base, and a robotic arm, the proposed robot has commands mapped to the user’s brainwaves related to a set of specific physical or mental tasks. The implementation of sensors and the camera systems enable both the mobile base and the arm to be semiautonomous. The mobile base’s SLAM algorithm has obstacle avoidance capability and path planning to assist the robot maneuver safely. The robot arm calculates and deploys the necessary joint movement to pick up or drop off a desired object selected by the user via a brainwave controlled cursor on a camera feed. Validation, testing, and implementation of the subsystems were conducted using Gazebo. Communication between the BCI controller and the subsystems is tested independently. A loop of prerecorded brainwave data related to each specific task is used to ensure that the mobile base command is executed; the same prerecorded file is used to move the robot arm cursor and initiate a pick-up or drop-off action. A final system test is conducted where the BCI controller input moves the cursor and selects a goal point. Successful virtual demonstrations of the assistive robotic arm show the feasibility of restoring movement capability and autonomy for a disabled user.

Teleoperation and Visualization Interfaces for Remote Intervention in Space

Approaches to robotic manufacturing, assembly, and servicing of in-space assets range from autonomous operation to direct teleoperation, with many forms of semi-autonomous teleoperation in between. Because most approaches require one or more human operators at some level, it is important to explore the control and visualization interfaces available to those operators, taking into account the challenges due to significant telemetry time delay. We consider one motivating application of remote teleoperation, which is ground-based control of a robot on-orbit for satellite servicing. This paper presents a model-based architecture that: 1) improves visualization and situation awareness, 2) enables more effective human/robot interaction and control, and 3) detects task failures based on anomalous sensor feedback. We illustrate elements of the architecture by drawing on 10 years of our research in this area. The paper further reports the results of several multi-user experiments to evaluate the model-based architecture, on ground-based test platforms, for satellite servicing tasks subject to round-trip communication latencies of several seconds. The most significant performance gains were obtained by enhancing the operators’ situation awareness via improved visualization and by enabling them to precisely specify intended motion. In contrast, changes to the control interface, including model-mediated control or an immersive 3D environment, often reduced the reported task load but did not significantly improve task performance. Considering the challenges of fully autonomous intervention, we expect that some form of teleoperation will continue to be necessary for robotic in-situ servicing, assembly, and manufacturing tasks for the foreseeable future. We propose that effective teleoperation can be enabled by modeling the remote environment, providing operators with a fused view of the real environment and virtual model, and incorporating interfaces and control strategies that enable interactive planning, precise operation, and prompt detection of errors.

A Mixed-Initiative Formation Control Strategy for Multiple Quadrotors

In this paper, we present a mixed-initiative motion control strategy for multiple quadrotor aerial vehicles. The proposed approach incorporates formation specifications and motion-planning commands as well as inputs by a human operator. More specifically, we consider a leader–follower aerial robotic system, which autonomously attains a specific geometrical formation, by regulating the distances among neighboring agents while avoiding inter-robot collisions. The desired formation is realized by a decentralized prescribed performance control strategy, resulting in a low computational complexity implementation with guaranteed robustness and accurate formation establishment. The multi-robot system is safely guided towards goal configurations, by employing a properly defined navigation function that provides appropriate motion commands to the leading vehicle, which is the only one that has knowledge of the workspace and the goal configurations. Additionally, the overall framework incorporates human commands that dictate the motion of the leader via a teleoperation interface. The resulting mixed-initiative control system has analytically guaranteed stability and convergence properties. A realistic simulation study, considering a team of five quadrotors operating in a cluttered environment, was carried out to demonstrate the performance of the proposed strategy.

Dynamics modeling of human–machine control interface for underwater teleoperation – ERRATUM

In the original version of this paper the name of the first author was incorrectly published as Muscolo Giovanni Gerardo. The correct name is Giovanni Gerardo Muscolo. The Publisher apologises for this error.

Teleoperation of Highly Automated Vehicles in Public Transport: User-Centered Design of a Human-Machine Interface for Remote-Operation and Its Expert Usability Evaluation

Paving the way to future mobility, teleoperation of vehicles promises a reachable solution to effectively use the benefits of automated driving as long as fully automated vehicles (SAE 5) are not entirely feasible. Safety and reliability are assured by a human operator who remotely observes the vehicle and takes over control in cases of disturbances that exceed the vehicle automation’s skills. In order to integrate the vehicle’s automation and human remote-operation, we developed a novel user-centered human-machine interface (HMI) for teleoperation. It is tailored to the remote-operation of a highly automated shuttle (SAE 4) by a public transport control center and based on a systematic analysis of scenarios, of which detailed requirements were derived. Subsequently, a paper-pencil prototype was generated and refined until a click-dummy emerged. This click-dummy was evaluated by twelve control center professionals. The experts were presented the prototype in regular mode and were then asked to solve three scenarios with disturbances in the system. Using structured interview and questionnaire methodology, the prototype was evaluated regarding its usability, situation awareness, acceptance, and perceived workload. Results support our HMI design for teleoperation of a highly automated shuttle, especially regarding usability, acceptance, and workload. Participant ratings and comments indicated particularly high satisfaction with the interaction design to resolve disturbances and the presentation of camera images. Participants’ feedbacks provide valuable information for a refined HMI design as well as for further research.

Soft Robotic Manipulation System Capable of Stiffness Variation and Dexterous Operation for Safe Human–Machine Interactions

AbstractSoft robots have attracted great attention in the past decades owing to their unique flexibility and adaptability for accomplishing tasks via simple control strategies, as well as their inherent safety for interactions with humans and environments. Here, a soft robotic manipulation system capable of stiffness variation and dexterous operations through a remotely controlled manner is reported. The smart manipulation system consists of a soft omnidirectional arm, a dexterous multimaterial gripper, and a self‐powered human–machine interface (HMI) for teleoperation. The cable‐driven soft arm is made of soft elastomers and embedded with low melting point alloy as a stiffness‐tuning mechanism. The self‐powered HMI patches are designed based on the triboelectric nanogenerator that utilizes a sliding mode of tribo‐layers made of copper and polytetrafluoroethylene. The novel soft manipulation system has great potential for soft and remote manipulation and human machine interactions in a variety of applications from elderly care to surgical operation to agriculture harvesting.

Electromyography for Teleoperated Tasks in Weightlessness

The cooperation between robots and astronauts will become a core element of future space missions. This is accompanied by the demand for suitable input devices. An interface based on electromyography (EMG) represents a small, light, and wearable device to generate a continuous three-dimensional (3D) control signal from voluntarily muscle activation of the operator's arm. We analyzed the influence of microgravity on task performance during a two-dimensional (2D) task on a screen. Six subjects performed aiming and tracking tasks in parabolic flights. Three different levels of fixation-fixed feet using foot straps, semi-free by using a foot rail, and free-floating feet-are tested to investigate how much user fixation is required to operate via the interface. The user study showed that weightlessness affects the usage of the interface only to a small extent. Success rates between 89${\%}$ and 96${\%}$ are reached within all conditions during microgravity. A significant effect between 0 and 1G could not be identified for the test series of fixed and semi-free feet, while free-floating feet showed significantly worse results in fine and gross motion times in 0G compared to ground tests (with success rates of 92${\%}$ for 0G and 99${\%}$ for 1G). Further adaptation to the altered proprioception may be needed here. Hence, foot rails as already mounted in the International Space Station (ISS) would be sufficient to use the interface in weightlessness. Low impact of microgravity, high success rates, and an easy handling of the system, indicates a high potential of an EMG-based interface for teleoperation in space.