Direct Teleoperation Research Articles

Towards the use of control systems with variable autonomy levels in tasks of extreme robotics

Increasing the level of robot autonomy in scenarios of extreme robotics allows, on the one hand, to reduce the load on the operator due to the sharing of labor between man and machine, and on the other, to increase the safety and efficiency of performing complex and critical tasks, such as search and rescue operations, monitoring of hazard-ous environments, disposal or neutralization of dangerous objects and many others. Such robots are characterized by the predominant use of teleoperation. However, recently there appeared a trend towards increasing of robot’s autonomy levels even for those types of robots that worked exclusively in direct teleoperation mode of control [1]. This article focuses on the consideration of control strategies for ground-based mobile robots with variable levels of autonomy when performing various types of work in extreme situations.

Preference-Based People-Aware Navigation for Telepresence Robots

AbstractThis work proposes an innovative people-aware navigation for telepresence robots in a populated environment based on the estimated inclination of people to interact and the context information. The main novelty of the proposed people-aware shared intelligence is the ability to fuse the remote operator’s commands with the probability of person-robot interaction—from both the operator driving the robot and the people around it—and translate it into semi-autonomous approaching and avoiding behaviors that are not coded a priori but rather dynamically emerge according to the current context-awareness. Experiments involved 45 healthy participants who evaluated the proposed approach on a real robot. Three conditions have been tested: (a) the new people-aware shared intelligence; (b) a shared intelligence system integrated with the standard ROS social navigation layers and; (c) a direct teleoperation (i.e., no robot’s intelligence). Results from our people-aware shared intelligence system have shown that the robot’s social behaviors were in line with the expectations of the participants in terms of comfort, naturalness, and sociability and coherent with the findings from previous studies. Furthermore, the proposed system has facilitated the social interaction between the remote operator and the surrounding people, making the robot more proactive and without affecting navigation performance.

Experimental Assessment of Human-Robot Teaming for Multi-Step Remote Manipulation with Expert Operators

Remote robot manipulation with human control enables applications where safety and environmental constraints are adverse to humans (e.g. underwater, space robotics and disaster response) or the complexity of the task demands human-level cognition and dexterity (e.g. robotic surgery and manufacturing). These systems typically use direct teleoperation at the motion level, and are usually limited to low-DOF arms and 2D perception. Improving dexterity and situational awareness demands new interaction and planning workflows. We explore the use of human-robot teaming through teleautonomy with assisted planning for remote control of a dual-arm dexterous robot for multi-step manipulation, and conduct a within-subjects experimental assessment (n=12 expert users) to compare it with direct teleoperation with an imitation controller with 2D and 3D perception, as well as teleoperation through a teleautonomy interface. The proposed assisted planning approach achieves task times comparable with direct teleoperation while improving other objective and subjective metrics, including re-grasps, collisions, and TLX workload. Assisted planning in the teleautonomy interface achieves faster task execution, and removes a significant interaction with the operator’s expertise level, resulting in a performance equalizer across users. Our study protocol, metrics and models for statistical analysis might also serve as a general benchmarking framework in teleoperation domains. Accompanying video and reference R code: https://people.csail.mit.edu/cdarpino/THRIteleop/

A Target-Guided Telemanipulation Architecture for Assisted Grasping

Teleoperation offers the possibility to combine human intelligence with robot power and endurance, making it a perfect solution for hostile-for-human environments. Nonetheless, when it concerns prolonged and repetitive operations, classical teleoperation interfaces that replicate one-to-one human commands can be quite demanding in terms of physical and mental efforts. Shared-autonomy approaches, which foresee the combination of direct teleoperation with autonomous control, can help the operator in overcoming such issues without sacrificing the task execution efficiency. Following this philosophy, in this letter we propose a novel technique to assist the operators in reaching and manipulation of objects with one or two arms, combining local visual perception and operator’s action monitoring. In more details, the reaching intention of an operator towards a target object is detected, which is used to adapt robot trajectories autonomously towards a successful grasping pose. In addition, based on the detected size of a target object, single- or dual-arm coordinated movements are autonomously generated without the need for additional human interventions. The experimental results on thirteen participants confirm the potential of the proposed framework in terms of success rate and operator effort.

Exploring planet geology through force-feedback telemanipulation from orbit.

Current space exploration roadmaps envision exploring the surface geology of celestial bodies with robots for both scientific research and in situ resource utilization. In such unstructured, poorly lit, complex, and remote environments, automation is not always possible, and some tasks, such as geological sampling, require direct teleoperation aided by force-feedback (FF). The operator would be on an orbiting spacecraft, and poor bandwidth, high latency, and packet loss from orbit to ground mean that safe, stable, and transparent interaction is a substantial technical challenge. For this scenario, a control method was developed that ensures stability at high delay without reduction in speed or loss of positioning accuracy. At the same time, a new level of safety is achieved not only through FF itself but also through an intrinsic property of the approach preventing hard impacts. On the basis of this method, a tele-exploration scenario was simulated in the Analog-1 experiment with an astronaut on the International Space Station (ISS) using a 6-degree-of-freedom (DoF) FF capable haptic input device to control a mobile robot with manipulator on Earth to collect rock samples. The 6-DoF FF telemanipulation from space was performed at a round-trip communication delay constantly between 770 and 850 milliseconds and an average packet loss of 1.27%. This experiment showcases the feasibility of a complete space exploration scenario via haptic telemanipulation under spaceflight conditions. The results underline the benefits of this control method for safe and accurate interactions and of haptic feedback in general.

Vision-Based Assisted Tele-Operation of a Dual-Arm Hydraulically Actuated Robot for Pipe Cutting and Grasping in Nuclear Environments

This article investigates visual servoing for a hydraulically actuated dual-arm robot, in which the user selects the object of interest from an on-screen image, whilst the computer control system implements via feedback control the required position and orientation of the manipulators. To improve on the current joystick direct tele-operation commonly used as standard in the nuclear industry, which is slow and requires extensive operator training, the proposed assisted tele-operation makes use of a single camera mounted on the robot. Focusing on pipe cutting as an example, the new system ensures that one manipulator automatically grasps the user-selected pipe, and appropriately positions the second for a cutting operation. Initial laboratory testing (using a plastic pipe) shows the efficacy of the approach for positioning the manipulators, and suggests that for both experienced and inexperienced users, the task is completed significantly faster than via tele-operation.

Egocentric teleoperation approach

In this paper, an egocentric teleoperation approach to overcome human cognitive and operation limits for multi body motions (robot motions) is proposed. This approach minimizes the degrees of freedom that a human operator has to operate by providing a virtual geometric constraint on the work path;the slave robot autonomously maintains the geometric constraint. The experimental results of an operation of target fitting showed significantly less duration time and work load than the direct teleoperation.

Gyro-based odometry associated with steering characteristics for wheeled mobile robot in rough terrain

Mobile robot used for planetary exploration performs several scientific missions over long distance travel and needs to have a high degree of autonomous mobility system because the communication delay from the Earth impedes its direct teleoperation. Localization of a mobile robot is of particular importance on the autonomous mobility. Classical localization methods such as wheel/visual odometry have been widely investigated and demonstrated, but they possess a well-known trade-off between computational cost and localization accuracy. This paper proposes an accurate gyro-based odometry method for a wheeled mobile robot in rough terrain. The robot in rough terrain is often subject to large wheel slip or vehicle sideslip related with its steering maneuver, and those slips degrade the localization accuracy. The basic approach of the proposed method is to exploit odometry data for the robot distance traveled as well as gyroscope data for the robot heading calculation; however each data-set is weighted in accordance with steering characteristics of a robot in rough terrain. The usefulness of the proposed method is examined through field experiments using a wheeled mobile robot testbed in Martian analog site. The experimental result confirms that the proposed method accurately estimates the robot trajectory.

Three-layer intelligence of planetary exploration wheeled mobile robots: Robint, virtint, and humint

The great success of the Sojourner rover in the Mars Pathfinder mission set off a global upsurge of planetary exploration with autonomous wheeled mobile robots (WMRs), or rovers. Planetary WMRs are among the most intelligent space systems that combine robotic intelligence (robint), virtual intelligence (virtint), and human intelligence (humint) synergetically. This article extends the architecture of the three-layer intelligence stemming from successful Mars rovers and related technologies in order to support the R&D of future tele-operated robotic systems. Double-layer human-machine interfaces are suggested to support the integration of humint from scientists and engineers through supervisory (Mars rovers) or three-dimensional (3D) predictive direct tele-operation (lunar rovers). The concept of multilevel autonomy to realize robint, in particular, the Coupled-Layer Architecture for Robotic Autonomy developed for Mars rovers, is introduced. The challenging issues of intelligent perception (proprioception and exteroception), navigation, and motion control of rovers are discussed, where the terrains’ mechanical properties and wheel-terrain interaction mechanics are considered to be key. Double-level virtual simulation architecture to realize virtint is proposed. Key technologies of virtint are summarized: virtual planetary terrain modeling, virtual intelligent rover, and wheel-terrain interaction mechanics. This generalized three-layer intelligence framework is also applicable to other systems that require human intervention, such as space robotic arms, robonauts, unmanned deep-sea vehicles, and rescue robots, particularly when there is considerable time delay.

A Generalized Visual Aid System for Teleoperation Applied to Satellite Servicing

This paper presents the latest results of a newly developed visual aid system for direct teleoperation. This method is extended to visual control to make an efficient teleoperation system by combining direct teleoperation and automatic control. On the one hand, an operator can conduct direct teleoperation with 3D graphic prediction simulation established by the VR technique. In order to remove inconsistencies between the virtual and real environments, a practical model-matching method is investigated. On the other hand, to realize real-time visual servoing control, a particular object recognition and pose estimation algorithm based on polygonal approximation is investigated to ensure a low computational cost for image processing. To avoid undesired forces involved in contact operation, 3D visual servoing incorporating a compliant control based on impedance control is developed. Finally, in a representative laboratory environment, a typical satellite servicing experiment is carried out based on this combined system. Experimental results demonstrate the feasibility and the effectiveness of the proposed method.

Experimental Evaluation of Feedback Modalities for Five Teleoperation Tasks

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> A distributed telerobotic system is proposed based on a master-arm station that is interconnected by a computer network to a slave-arm station. The distributed telerobotic system is evaluated using a set of teleoperated experiments: 1) peg-in-hole insertion; 2) assembly of a small water pump; 3) operating drawers; 4) pouring of water; and 5) wire wrapping. Direct teleoperation is evaluated using the following schemes: 1) stereo vision; 2) vision and force feedback (VFF); and 3) vision with active compliance (VAC). Space indexing and scaling tools are also used. Operator hand is logically mapped to a remote tool both in position and force. The operator feels the forces that were exerted on the tool as they were exerted on the hand. Extensive experimental analysis showed that mapping of operator hand motion and force feedback (FF) to a convenient tool point reduces the operator mental load and task time due to highly coordinated motion. Stereo vision may solely be used at the cost of large peak forces and extended task time. VFF has nearly equal task time compared to VAC but with a noticeable increase in contact forces. For a large majority of cases, the contact-based tasks that were done using VAC resulted in the least task times and the least contact forces. VAC is superior to VFF, which is better than V. In other words, there is an enormous gain in stability if one removes the bilateral FF channel in teleoperation and relies on a slave-arm active compliance. </para>

Direct Mobile Robot Teleoperation via Internet

As networks such as the Internet go global, they become an important tool for everyone from engineers to the general public. This makes network-related technology an up-and-coming field. Conventional techniques should be applied for teleservice and teleoperation in global and general-purpose networks. Robots are a promising example of physical agent potentially useful via such networks in the near future. We describe direct teleoperation of a mobile robot via the Internet and experiments involving a real system and network.

Special Issue on Mobile Robot

Mobility, or locomotion, is as important a function for robots as manipulation. A robot can enlarge its work space by locomotion. It can also recognize its environment well with its sensors by moving around and by observing its surroundings from various directions. Much researches has been done on mobile robots and the research appears to be mature. Research activity on robot mobility is still very active; for example, 22% of the sessions at ICRA'98 - the International Conference on Robotics and Automation - and 24% of the sessions at IROS'98 - the International Conference on Intelligent Robots and Systems - dealt with issues directly related to mobile robots. One of the main reasons may be that intelligent mobile robots are thought to be the closest position to autonomous robot applications. This special issue focuses on a variety of mobile robot research from mobile mechanisms, localization, and navigation to remote control through networks. The first paper, entitled ""Control of an Omnidirectional Vehicle with Multiple Modular Steerable Drive Wheels,"" by M. Hashimoto et al., deals with locomotion mechanisms. They propose an omnidirectional mobile mechanism consisting of modular steerable drive wheels. The omnidirectional function of mobile mechanisms will be an important part of the human-friendly robot in the near future to realize flexible movements in indoor environments. The next three papers focus on audiovisual sensing to localize and navigate a robot. The second paper, entitled ""High-Speed Measurement of Normal Wall Direction by Ultrasonic Sensor,"" by A. Ohya et al., proposes a method to measure the normal direction of walls by ultrasonic array sensor. The third paper, entitled ""Self-Position Detection System Using a Visual-Sensor for Mobile Robots,"" is written by T. Tanaka et al. In their method, the position of the robot is decided by measuring marks such as name plates and fire alarm lamps by visual sensor. In the fourth paper, entitled ""Development of Ultra-Wide-Angle Laser Range Sensor and Navigation of a Mobile Robot in a Corridor Environment,"" written by Y Ando et al., a very wide view-angle sensor is realized using 5 laser fan beam projectors and 3 CCD cameras. The next three papers discussing navigation problems. The fifth paper, entitled ""Autonomous Navigation of an Intelligent Vehicle Using 1-Dimensional Optical Flow,"" by M. Yamada and K. Nakazawa, discusses navigation based on visual feedback. In this work, navigation is realized by general and qualitative knowledge of the environment. The sixth paper, entitled ""Development of Sensor-Based Navigation for Mobile Robots Using Target Direction Sensor,"" by M. Yamamoto et al., proposes a new sensor-based navigation algorithm in an unknown obstacle environment. The seventh paper, entitled ""Navigation Based on Vision and DGPS Information for Mobile Robots,"" S. Kotani et al., describes a navigation system for an autonomous mobile robot in an outdoor environment. The unique point of their paper is the utilization of landmarks and a differential global positioning system to determine robot position and orientation. The last paper deals with the relationship between the mobile robot and computer networks. The paper, entitled ""Direct Mobile Robot Teleoperation via Internet,"" by K. Kawabata et al., proposes direct teleoperation of a mobile robot via the Internet. Such network-based robotics will be an important field in robotics application. We sincerely thank all of the contributors to this special issue for their cooperation from the planning stage to the review process. Many thanks also go to the reviewers for their excellent work. We will be most happy if this issue aids readers in understanding recent trends in mobile robot research and furthers interest in this research field.

Robots for lunar exploration: Present and future

Direct teleoperation of a rover on the moon has proven to be feasible but very demanding in telemetry data rate. Several modes are proposed to control the rover with lower communication data rates and the necessary equipments to operate the vehicle in these modes are identified. The impact of the equipment's mass and consumption on the minimal size of a rover is discussed for several applications.

Behavior control for robotic exploration of planetary surfaces

This paper describes a series of robots developed at JPL to demonstrate the feasibility of using a behavior-control approach to control small robots on planetary surfaces. The round-trip light-time delay makes direct teleoperation of a mobile robot on a planetary surface impossible. Planetary rovers must therefore possess a certain degree of autonomy. However, small robots can only support small computers (due mostly to power, not size constraints). Behavior control provides a means of autonomous control that requires very little computation. The robots described in this paper all used 8-bit, 1-MIP microprocessors with as little as 4 k and no more than 40 k of memory, and extremely simple sensors. Despite these limitations they reliably perform both autonomous navigation and manipulation in both indoor and outdoor rough-terrain environments.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Teleprogramming: Toward Delay-Invariant Remote Manipulation

This paper addresses the problem of teleoperation in the presence of communication delays. Delays occur with earth-based teleoperation in space and with surface-based teleoperation undersea using untethered submersibles and acoustic communication links. The delay in obtaining position and force feedback from the remote slave arms makes direct teleoperation infeasible. We are proposing a control methodology, called teleprogramming, which draws on the experience in the development of supervisory control techniques and robotics over the last three decades and introduces a number of new ideas in operator-model interaction as well as the nature and content of the information being sent to the slave robot. A teleprogramming system allows the operator to kinesthetically, as well as visually, interact with a graphic simulation of the remote environment and to interactively, online teleprogram the remote manipulator through a sequence of elementary robot instructions. A key feature and contribution of this work is the fact that these instructions are generated automatically, in real time, based on the operator's interaction with the simulated environment. The slave robot executes these commands delayed in time and, should an error occur, allows the operator to specify the necessary corrective actions and continue with the task. We will in this paper introduce the overall teleprogramming control concept, describe its main components, and report on the preliminary results using our experimental teleprogramming system.