Proposed Teleoperation System Research Articles

A virtual reality-based immersive teleoperation system for remote human-robot collaborative manufacturing

In recent years, the cutting-edge technologies in smart manufacturing have presented promising opportunities for the utilization of human-robot collaborative teleoperation in personalized manufacturing tasks. To effectively leverage the creative capabilities of humans while benefiting from the efficiency and stability of robots, the provision of an intuitive teleoperation interface assumes paramount importance. However, current teleoperation systems still face limitations in terms of intuitive operability. In this study, we present a virtual reality-based teleoperation system that offers operators a more intuitive interaction platform for robot control, thereby facilitating personalized manufacturing processes. The overall system framework design, as well as the main components are elaborated in detail. Furthermore, an evaluative case study based on the battery disassembly task is conducted to assess the performance of the proposed system. The results demonstrate that the proposed teleoperation system exhibits improved intuitiveness.

A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision.

Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-to-end hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-to-image translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.

Enhancing autonomy in social robots through a real-time teleoperation system

<p>The utilization of socially assistive robots (SARs) has shown great potential in various applications. However, there are practical challenges that hinder their widespread adoption. Recent studies have explored the use of teleoperation systems as a solution to overcome these challenges by streamlining data collection and communication with SARs. This paper introduces a cloud-based teleoperation system for robotics that enables real-time control of the robot platform, allowing remote users to view video streams and manipulate the robot system. The proposed teleoperation system was implemented in a cloud environment and tested with two different social robot platforms. The experimental results demonstrate the system's quick response and stable communication performance, meeting the requirements for remote-controlled teleoperation. Consequently, employing the proposed mobile-based teleoperation system in social robot applications enhances operator productivity, expedites operations, and reduces task completion time.</p>

Teleoperated Surgical Robot with Adaptive Interactive Control Architecture for Tissue Identification.

The remote perception of teleoperated surgical robotics has been a critical issue for surgeons in fulfilling their remote manipulation tasks. In this article, an adaptive teleoperation control framework is proposed. It provides a physical human-robot interaction interface to enhance the ability of the operator to intuitively perceive the material properties of remote objects. The recursive least square (RLS) is adopted to estimate the required human hand stiffness that the operator can achieve to compensate for the contact force. Based on the estimated stiffness, a force feedback controller is designed to avoid the induced motion and to convey the haptic information of the slave side. The passivity of the proposed teleoperation system is ensured by the virtual energy tank. A stable contact test validated that the proposed method achieved stable contact between the slave robot and the hard environment while ensuring the transparency of the force feedback. A series of human subject experiments was conducted to empirically verify that the proposed teleoperation framework can provide a more smooth, dexterous, and intuitive user experience with a more accurate perception of the mechanical property of the interacted material on the slave side, compared to the baseline method. After the experiment, the design idea about the force feedback controller of the bilateral teleoperation is discussed.

Toward Unified Adaptive Teleoperation Based on Damping ZNN for Robot Manipulators With Unknown Kinematics

Traditional teleoperation systems are usually designed for a single class of slave robots and require the specific kinematic model. This leads to the problems of high computational cost, poor robustness, and poor portability in traditional teleoperation systems. In this paper, a unified adaptive teleoperation system is proposed for different kinds of slave robots. To this end, a closed-loop control system based on damping zeroing neural network (DampZNN) is proposed to achieve the unified kinematic control of both the redundant robot and the continuum robot with unknown kinematics. The control system is constructed based on two DampZNNs, which are utilized to estimate the unknown kinematic model and to solve the inverse kinematics problem respectively. By combining the DampZNN control system with teleoperation technology, a unified adaptive teleoperation system is proposed for human operators to teleoperate both the redundant slave robot and the continuum slave robot easily. The efficacy of the proposed teleoperation system is verified by experiments based on different slave robots and the advantages of the proposed teleoperation system are revealed by comparative studies.

Augmented reality-assisted gesture-based teleoperated system for robot motion planning

Purpose The study proposed a human–robot interaction (HRI) framework to enable operators to communicate remotely with robots in a simple and intuitive way. The study focused on the situation when operators with no programming skills have to accomplish teleoperated tasks dealing with randomly localized different-sized objects in an unstructured environment. The purpose of this study is to reduce stress on operators, increase accuracy and reduce the time of task accomplishment. The special application of the proposed system is in the radioactive isotope production factories. The following approach combined the reactivity of the operator’s direct control with the powerful tools of vision-based object classification and localization. Design/methodology/approach Perceptive real-time gesture control predicated on a Kinect sensor is formulated by information fusion between human intuitiveness and an augmented reality-based vision algorithm. Objects are localized using a developed feature-based vision algorithm, where the homography is estimated and Perspective-n-Point problem is solved. The 3D object position and orientation are stored in the robot end-effector memory for the last mission adjusting and waiting for a gesture control signal to autonomously pick/place an object. Object classification process is done using a one-shot Siamese neural network (NN) to train a proposed deep NN; other well-known models are also used in a comparison. The system was contextualized in one of the nuclear industry applications: radioactive isotope production and its validation were performed through a user study where 10 participants of different backgrounds are involved. Findings The system was contextualized in one of the nuclear industry applications: radioactive isotope production and its validation were performed through a user study where 10 participants of different backgrounds are involved. The results revealed the effectiveness of the proposed teleoperation system and demonstrate its potential for use by robotics non-experienced users to effectively accomplish remote robot tasks. Social implications The proposed system reduces risk and increases level of safety when applied in hazardous environment such as the nuclear one. Originality/value The contribution and uniqueness of the presented study are represented in the development of a well-integrated HRI system that can tackle the four aforementioned circumstances in an effective and user-friendly way. High operator–robot reactivity is kept by using the direct control method, while a lot of cognitive stress is removed using elective/flapped autonomous mode to manipulate randomly localized different configuration objects. This necessitates building an effective deep learning algorithm (in comparison to well-known methods) to recognize objects in different conditions: illumination levels, shadows and different postures.

A Compliant Teleoperation System by Utilizing Haptic Guidance and Series Elastic Actuator-Based Gripper

Abstract This article introduces a haptic-guided teleoperation framework using a series elastic actuator (SEA)-based compliant gripper. The proposed teleoperation system involves one local haptic device, one layer of virtual reality, and one remote industrial manipulator with a compliant gripper. The overall compliance of the teleoperation is distributed between the software and hardware components of the bilateral loop. On the one hand, the implemented haptic guidance is based on an elementary coupling model uniquely defined and established for all interactions among the user, remote manipulator, and virtual interface. On the other hand, the manipulator operating in the remote environment is equipped with a novel compliant gripper based on series elastic actuators, providing passive compliance at the interactions with the environment. Introducing the gripper into the haptic loop is expected to compensate for disturbances due to inaccurate modeling and/or unmodeled dynamics of the remote environment and external effects. The teleoperation system is implemented for manipulation and tracking tasks and tested with different users. Experimental results show that the haptic guidance and the compliant gripper together significantly improve the teleoperation performances in terms of transparency.

An Impedance-Control Based Teleoperation System for Live-Line Maintenance Robot

Robotic operation is an effective way to upgrade the live-line maintenance safety, efficiency, and quality. This paper proposes an impedance control-based teleoperation system to improve the adaptability of live-line maintenance robot in outdoors environment. The key technology of this system is utilizing three virtual spring-damper systems to model the elastic connection between the end-effector of slave manipulator and the environment, between the end-effector of slave manipulator and the counterpart of master haptic device, and between the end-effector of master haptic device and its base, respectively. Experiment results show that, under control of our proposed teleoperation system, the slave manipulator is able to track the motion of master haptic device and the robot is able to complete a set of complex action to peel the coat off the cable.

Towards the development of an intuitive teleoperation system for human support robot using a VR device

This paper presents our development of an intuitive teleoperation system for Toyota's Human Support Robot. While this robot has been widely used as an autonomous agent in the field of service robotics, we explore a possibility of this robot for the use of remote assistance of daily activities in home environments. For rapid prototyping an affordable teleoperation system, we adopt a commercially available VR device as a user interface and develop a Robot Operating System (ROS)-based control framework building on the software application programming interfaces (APIs). In particular, we implement a singularity robust inverse kinematics algorithm to achieve a coordinated whole-body motion of the robot for safe and intuitive operation. We empirically evaluate the practical feasibility of the proposed teleoperation system through various test tasks. We discuss the potential and limitations of the current development for its future improvement.

Event‐triggered control for adaptive bilateral teleoperators with communication delays

Bilateral teleoperation systems have been widely applied for human operators to manipulate slave robots in accomplishing various tasks within remote environments. However, most teleoperation systems developed previously have heavily relied on continuous communication or a network with higher sampling rates. In this study, the authors propose novel control schemes to guarantee stability and tracking performance of bilateral teleoperation by considering an event-based transmission. Based on local output signals and the proposed triggering mechanisms, the master and slave robots transmit signals over a communication network only when the triggering conditions are satisfied. An event-triggered coordinating controller is presented to guarantee stability and position tracking for non-linear teleoperators. Input-to-state stability of the proposed teleoperation system with bounded external forces is studied with guaranteed static force reflection. Subsequently, a transformation-based framework is addressed with the event-triggered coordinating control to cope with bilateral teleoperation systems under time delays. The Zeno behaviour is demonstrated to be excluded in the proposed event-triggered teleoperation systems. Numerical examples and experiments are presented to discuss the efficacy of the proposed event-triggered teleoperation systems.

Teleoperation system for a mobile robot arm with visual servomechanism based on turning radius determination using angle information of image

This paper considers the development of a teleoperation system for a mobile robot arm with visual servo mechanism. The robot to be operated mainly consists of a mobile robot and a robot arm with USB camera. The robot is teleoperated according to the control command which is generated by combining the manual operation with the autonomous control via the vision system based on the USB camera images. In this paper, a control method of determining the turning radius of the mobile robot is proposed based on the angle information of image corresponding to the error of a target position from the reference one. To evaluate the proposed teleoperation system, experiments of pressing a button were conducted by using an actual robot. It was verified from the experimental results that sufficiently high success rate of teleoperation could be obtained by the proposed method.

TeMoto: Intuitive Multi-Range Telerobotic System with Natural Gestural and Verbal Instruction Interface

Teleoperated mobile robots, equipped with object manipulation capabilities, provide safe means for executing dangerous tasks in hazardous environments without putting humans at risk. However, mainly due to a communication delay, complex operator interfaces and insufficient Situational Awareness (SA), the task productivity of telerobots remains inferior to human workers. This paper addresses the shortcomings of telerobots by proposing a combined approach of (i) a scalable and intuitive operator interface with gestural and verbal input, (ii) improved Situational Awareness (SA) through sensor fusion according to documented best practices, (iii) integrated virtual fixtures for task simplification and minimizing the operator’s cognitive burden and (iv) integrated semiautonomous behaviors that further reduce cognitive burden and negate the impact of communication delays, execution latency and/or failures. The proposed teleoperation system, TeMoto, is implemented using ROS (Robot Operating System) to ensure hardware agnosticism, extensibility and community access. The operator’s command interface consists of a Leap Motion Controller for hand tracking, Griffin PowerMate USB as turn knob for scaling and a microphone for speech input. TeMoto is evaluated on multiple robots including two mobile manipulator platforms. In addition to standard, task-specific evaluation techniques (completion time, user studies, number of steps, etc.)—which are platform and task dependent and thus difficult to scale—this paper presents additional metrics for evaluating the user interface including task-independent criteria for measuring generalized (i) task completion efficiency and (ii) operator context switching.

Teleoperation of robot arm with position measurement via angle-pixel characteristic and visual supporting function

In this paper, a teleoperation system of a robot arm with position measurement function and visual supporting function is developed. The working robot arm is remotely controlled by the manual operation of the human operator and the autonomous control via visual servo. The visual servo employs the template matching technique. The position measurement is realized using a stereo camera based on the angle-pixel characteristic. The visual supporting function to give the human operator useful information about the teleoperation is also provided. The usefulness of the proposed teleoperation system is confirmed through experiments using an industrial articulated robot arm.

S1510107 逐次最適化法を用いた移動ロボットの視点操作インタフェース

Camera view centered mobile robot control is proposed to provide intuitive operation feeling. The proposed controller only requires the motion of the view point. If an operator indicates a reference in the camera coordinate frame by a joystick, the controller generates the optimal control inputs for the robot to move the view point to the reference considering the hardware constraints. The generalized model of the non-holonomic mobile robot with a camera is introduced to generate the discrete velocity input array for controlling the motion of the robot. Based on the model, the model based trajectory planning is achieved by QP (quadratic programing) problem with constraints. The proposed method was implemented to the mobile robot simulator so that the effectiveness is evaluated by comparing to the conventional controller. The experimental result shows the effectiveness of the proposed teleoperation system.

Complete Bilateral Teleoperation System for a Rotorcraft UAV with Time-Varying Delay

This paper proposes the use of simple proportional plus damping injection (P+d) controllers for delayed bilateral teleoperation of a rotorcraft UAV. The proposed control scheme involves P+d remote and local controllers, considers master and slave dynamics, and takes into account asymmetric time-varying delays. The stability of the proposed teleoperation system is analyzed using Lyapunov-Krasovskii functionals and delay-dependent stability criteria are obtained under linear-matrix-inequalities conditions. The performance of the teleoperation scheme is tested driving a virtual nonlinear 6DOF dynamic model of a minihelicopter in a human-in-the-loop simulation.

Low-cost 3D-supported interactive control

Devices for stereoscopic vision are gaining increasing diffusion, but their usage is mostly oriented toward entertainment. A prototype based on consumer devices and open software to achieve low-cost 3D-video-supported interactive control is presented here. This research could stimulate the study and implementation of low-cost general-purpose systems that could be used on a wide spectrum of applications, including remote operation, education, training, and surveillance. There are two related Web extras that provide supplemental material. The first video [http://youtu.be/5zApDO3wSq4] shows the building blocks of the proposed teleoperation system based on real-time 3D video feedback. The second video [http://youtu.be/1Olg1oV2fIs] shows the experiment. A smartphone camera was mounted on a radio-controlled toy car, and users were asked to perform an alignment task using the 3D vision system and the car remote controller.

Task-space bilateral teleoperation systems for heterogeneous robots with time-varying delays

SUMMARYThis paper proposes control algorithms for heterogeneous teleoperation systems to guarantee stability and tracking performance in the presence of time-varying communication delays. Because robotic manipulators, in most applications of bilateral teleoperation systems, interact with a human operator and remote environment on the end-effector, the control system is developed in the task-space. When the dynamic parameters of the robots are unknown and the communication network is subject to time-varying delay, the developed controller can ensure stability and task-space position tracking. Additionally, if the robotic systems are influenced by human and environmental forces, the presented teleoperation control system is demonstrated to be stable and all signals are proven to be ultimately bounded. By employing the redundancy of the slave robot for sub-task control, the proposed teleoperation system can autonomously achieve additional missions in the remote environment. Numerical examples utilizing a redundant planar robot are addressed to validate the proposed task-space teleoperators with time-varying delay.

Vehicle teleoperation using 3D maps and GPS time synchronization.

In conventional vehicle teleoperation systems, using low-bandwidth, high-delay transmission links causes a serious problem for remote control of the vehicles. To solve this problem, a proposed teleoperation system employs 3D maps and GPS time synchronization. Two GPS receivers measure the transmission delay, which the system uses to estimate the vehicle's location and orientation. Field experiments show that the 3D-map-based interface lets users easily comprehend the remote environment while navigating a vehicle. The experiments also show that taking communication delays into account improves maneuverability.

Control of semi-autonomous teleoperation system with time delays

In teleoperation systems operating in complex environments, due to the cognitive limitations of the human operator and lack of complete information about the remote environment, safety and performance of such systems can potentially be comprised. In order to ensure the safety and enhance the efficiency of complex teleoperation systems operating in cluttered environments, in this paper we investigate a semi-autonomous control framework for bilateral teleoperation. The semi-autonomous teleoperation system is composed of heterogeneous master and slave robots, where the slave robot is assumed to be a redundant manipulator. Considering robots with different configurations, and in the presence of dynamic uncertainties and asymmetric communication delays, we first develop a control algorithm to ensure position and velocity tracking in the task space. Additionally in the absence of dynamic uncertainty, and in the presence of human operator and environmental forces, all signals of the proposed teleoperation system are proven to be ultimately bounded. The redundancy of the slave robot is then utilized for achieving autonomous sub-task control, such as singularity avoidance, joint limits, and collision avoidance. The control algorithms for the proposed semi-autonomous teleoperation system are validated through numerical simulations on a non-redundant master and a redundant slave robot.

재구성된 그래픽 모델을 이용한 원격제어

일반적으로 마스터/슬레이브 원격제어 시스템에서 작업자는 원격에서 슬레이브를 제어하기 위하여 카메라 영상 같은 시각 정보를 이용하는 경우가 많은데, 많은 데이터 양으로 인하여 원격지의 영상 정보에 지연이 발생할 수 있고 카메라 위치에 따라 영상 정보가 제한되거나 불완전하여 제어가 어려운 경우가 생길 수 있다. 카메라 영상을 이용한 원격제어 시스템의 이러한 문제점을 해결하기 위하여 본 논문에서는 영상 정보 대신에 3차원 재구성 그래픽 모델을 이용한 원격제어 시스템을 제안한다. 제안된 원격제어시스템은 로봇 제어 모듈, 힘반향 조이스틱을 이용한 마스터 모듈, 그래픽유저인터페이스모듈로 이루어져 있는데, 여기서 그래픽유저인터페이스모듈은 원격지에서 전달된 적은 양의 센싱 데이터를 이용하여 3차원으로 재구성된 그래픽 모델을 작업자에게 제공해 준다. 제안된 원격제어 시스템은 펙인홀(peg-in-hole) 조립 작업을 이용하여 그 효과가 검증되었다. In typical master/slave teleoperation systems, a human operator generally manipulates the master to control the slave through the visual information like camera image. However, the operator may get into trouble due to the limited visual information depending on the camera positions and the delay on the visual information because of low communication bandwidth. To cope with this inherit problem in the camera-based teleoperation system, this paper presents a teleoperation system using a reconstructed graphic model instead of the camera image. The proposed teleoperation system consists of a robot control module, a master module using a force-reflective joystick, and a graphic user interface (GUI) module. The graphic user interface module provides the operator with a 3D model reconstructed using a small set of sensing data received from the remote site. The proposed teleoperation system is evaluated through a peg-in-hole assembly task.