Haptic Teleoperation Research Articles

Model-Mediated Teleoperation: An Operator-Centric Perspective

Abstract Haptic teleoperation is a means of dexterously manipulating environments wherein a human operator can be situated several hundreds of kilometers away from the site of manipulation. Time delays between the local and remote sites pose problems for the user experience. Model-mediated teleoperation addresses these problems, but assumes a sufficiently representative model of the operator and of the environment. When applied to complex environments, this model becomes difficult to work with. A solution to this difficulty lies in the fact that human perceptual capabilities are limited. So too, possibly, is the amount of feedback needed in order to accomplish tasks effectively. In this work, we review information on human perception of the environmental properties involved in environmental perception (inertia, friction, roughness, compliance, and geometry). Moreover, we detail how these properties can be modeled. Finally, we propose some open questions that are relevant in understanding the link between the environment presented to the operator and their effectiveness at accomplishing tasks.

Attentiveness Map Estimation for Haptic Teleoperation of Mobile Robot Obstacle Avoidance and Approach

Attentiveness Map Estimation for Haptic Teleoperation of Mobile Robot Obstacle Avoidance and Approach

E-BTS: Event-Based Tactile Sensor for Haptic Teleoperation in Augmented Reality

E-BTS: Event-Based Tactile Sensor for Haptic Teleoperation in Augmented Reality

Emulation Framework for Haptic Data Transmission Using Real-Time Transport Protocol

The Tactile Internet (TI) can be regarded as the next evolution in the world of communication. With its envisioned purpose and potential in shaping up the economy, industry and society, this paradigm aims to bring a new dimension to life by enabling humans to interact with machines remotely and in real-time with haptic and kinesthetic feedback. However, to translate this into reality, Tactile Internet will need to meet the stringent requirements of extremely low latency in conjunction with ultra-high reliability, availability, and security. This poses a challenge on the available communication systems to achieve a round-trip delay within 1 to 10 milliseconds time bound that enables the timely delivery of critical tactile and haptic sensations.
 This paper aims to evaluate the Real-Time Transport Protocol (RTP) through an emulation framework. It integrates containerization using Linux-based Docker Containers with NS-3 Network Simulator to conceptualize a haptic teleoperation system. The framework is then used to test the protocol’s feasibility for delivering texture haptic data between master and slave domains in accordance with the end-to-end delay requirements specified by IEEE 1918.1 standards. The results have shown that the timely provision of haptic data is achievable by obtaining an average round-trip delay of 17.8493 ms from the emulation experiment. As such, the results satisfy the expected IEEE 1918.1 standards constraints for medium-dynamic environment use cases.

Unified Human Intention Recognition and Heuristic-Based Trajectory Generation for Haptic Teleoperation of Non-Holonomic Vehicles

In this paper, a novel bilateral shared control approach is proposed to address the issue of strong dependence on the human, and the resulting burden of manipulation, in classical haptic teleoperation systems for vehicle navigation. A Hidden Markov Model (HMM) is utilized to handle the Human Intention Recognition (HIR), according to the force input by the human—including the HMM solution, i.e., Baum–Welch algorithm, and HMM decoding, i.e., Viterbi algorithm—and the communication delay in teleoperation systems is added to generate a temporary goal. Afterwards, a heuristic and sampling method for online generation of splicing trajectory based on the goal is proposed innovatively, ensuring the vehicle can move feasibly after the change in human intention is detected. Once the trajectory is available, the vehicle velocity is then converted to joystick position information as the haptic cue of the human, which enhances the telepresence. The shared teleoperation control framework is verified in the simulation environment, where its excellent performance in the complex environment is evaluated, and its feasibility is confirmed. The experimental results show that the proposed method can achieve simple and efficient navigation in a complex environment, and can also give a certain situational awareness to the human.

Design and Evaluation of an Intuitive Haptic Teleoperation Control System for 6-DoF Industrial Manipulators

Industrial robots are capable of performing automated tasks repeatedly, reliably and accurately. However, in some scenarios, human-in-the-loop control is required. In this case, having an intuitive system for moving the robot within the working environment is crucial. Additionally, the operator should be aided by sensory feedback to obtain a user-friendly robot control system. Haptic feedback is one way of achieving such a system. This paper designs and assesses an intuitive teleoperation system for controlling an industrial 6-DoF robotic manipulator using a Geomagic Touch haptic interface. The system utilises both virtual environment-induced and physical sensor-induced haptic feedback to provide the user with both a higher amount of environmental awareness and additional safety while manoeuvering the robot within its working area. Different tests show that the system is capable of fully stopping the manipulator without colliding with the environment, and preventing it from entering singularity states with Cartesian end effector velocities of up to 0.25 m/s. Additionally, an operator is capable of executing low-tolerance end effector positioning tasks (∼0.5 mm) with high-frequency control of the robot (∼100 Hz). Fourteen inexperienced volunteers were asked to perform a typical object removal and writing task to gauge the intuitiveness of the system. It was found that when repeating the same test for a second time, the participants performed 22.2% faster on average. The results for the second attempt also became significantly more consistent between participants, as the inter quartile range dropped by 82.7% (from 52 s on the first attempt to 9 s on the second).

RoboDoc: Smart Robot Design Dealing with Contagious Patients for Essential Vitals Amid COVID-19 Pandemic

The COVID-19 pandemic took valuable lives all around the world. The virus was so contagious and lethal that some of the doctors who worked with COVID-19 patients either were seriously infected or died, even after using personal protective equipment. Therefore, the challenge was not only to help communities recover from the pandemic, but also to protect the healthcare staff/professionals. In this regard, this paper presents a comprehensive design of a customized pseudo-humanoid robot to specifically deal with contagious patients by taking basic vitals through a healthcare staff member from a remote location amid the COVID-19 pandemic. The proposed design consists of two portions: (1) a complete design of mechanical, electrical/electronic, mechatronic, control, and communication parts along with complete assembly to make a complete multitask-performing robot that interacts with patients to take vitals, termed as RoboDoc, and (2) the design of the healthcare staff side (master/operator side) control of a joystick mechanism with haptic feedback. The proposed RoboDoc design can be majorly divided into three parts: (1) the locomotion part is composed of two-wheeled DC motors on a rover base and two omni wheels to support the movements of the robot; (2) the interaction part consists of a single degree-of-freedom (s-DOF) neck to have communication with different heights of patients and (3) two anthropomorphic arms with three degrees-of-freedom (3-DOF). These parts help RoboDoc to reach to patient’s location and take all of the vitals using relevant devices such as an IR temperature thermometer, pulse oximeter, and electronic stethoscope for taking live auscultations from the lungs and heart of the patient. The mechanical design was created using solid works, and the electronic control design was made via proteus 8.9. For haptic teleoperation, an XBOX 360 controller based on wireless communication is used at the master/operator side. For the convenience of the healthcare staff (operator), an interactive desktop-based GUI was developed for live monitoring of all the vital signs of patients. For the remote conversation between the healthcare staff and the patient, a tablet is mounted (that also serves as the robot’s face), and that tablet is controlled via a mobile application. For visual aid, a DSLR camera is integrated and controlled remotely, which helps the doctor monitor the patient’s location as well as examine the patient’s throat. Finally, successful experimental results of basic vitals of the remote patient such as temperature sensing, pulse oximeter, and heart rate (using haptic feedback) were obtained to show the significance of the proposed cost-effective RoboDoc design.

Active tactile sensing of small insect force by a soft microfinger toward microfinger-insect interactions

Human–robot interaction technology has contributed to improving sociality for humanoid robots. At scales far from human scales, a microrobot can interact with an environment in a small world. Microsensors have been applied to measurement of forces by flying or walking insects. Meanwhile, most previous works focused on the measurement of the behavior of insects. Here, we propose microrobot-insect interactions by soft microfingers integrated with artificial muscle actuators and tactile sensors, which has been developed for a haptic teleoperation robot system. A soft pneumatic balloon actuator acts as the artificial muscle, and a flexible strain sensor using a liquid metal provides tactile sensing. Force interaction between a pill bug and the microfinger could be accomplished. The microfinger (12 mm × 3 mm × 490 μm) can move and touch an insect, and it can detect reaction force from an insect. The measured reaction force from the legs of a pill bug as a representative insect was less than 10 mN. This paper presents a microfinger as an end effector for the active sensing of reaction force from a small insect. We anticipate that our results will lead to further evaluation of small living things as well as technology development for human–environment interaction.

Dual-User Haptic Teleoperation of Complementary Motions of a Redundant Wheeled Mobile Manipulator Considering Task Priority

With the increasing applications of wheeled mobile manipulators (WMMs), consisting of a mobile platform (MP) and a manipulator, in diverse fields, new challenges have arisen in achieving multiple tasks such as obstacle avoidance in a constrained environment during the end-effector (EE) operation. A WMM is usually redundant due to the combination of the MP and the manipulator, making multitask control possible via employing its null space. Dual-user/two-handed teleoperation of a WMM is desirable for tasks where it is important to simultaneously control the poses of both the MP and the EE. The existing teleoperation approaches for WMMs are mostly executed at the kinematic level, without considering the nonlinear rigid-body dynamics of the WMMs. In this article, a task-priority-based dual-user teleoperation framework for a WMM is implemented to perform tasks in a constrained environment. It can simultaneously manipulate the MP and the EE, the overground obstacles are avoided by telecontrolling the MP using the WMM’s null space. Any residual redundancy can be further employed for other tasks such as singularity avoidance. The stability of the entire teleoperation design is rigorously proved even with arbitrary time delays. Experiments with a dual-user teleoperation system, consisting of two local robots and an omnidirectional WMM, are conducted to verify the proposed approach’s feasibility and effectiveness.

Haptic Teleoperation of Impact Hammers in Underground Mining

Impact hammers are used to reduce the size of blasted ore in mining operations. In underground mines, tele-operated impact hammers are used to reduce the size of boulders placed on the orepass’s grizzlies. An impact hammer consists of a hydraulic arm with 4 degrees of freedom, powered with a hydraulic impact hammer as an end-effector. The tele-operation of impact hammers is difficult due to the latency of communications, the poor visibility of the environment, and the used 2D interfaces. This may result in a collision with the hammer and the infrastructure, idle strokes, and non-optimal operation. To address these issues, this paper proposes the haptic tele-operation of impact hammers. The proposed haptic tele-operation system is based on a 3D model of the environment, which is used to estimate repulsion forces that are transferred to the operator via a haptic device, so that the hammer does not collide with the structures of the mine. The system also allows identifying the oversized boulders deposited on the grizzly and notifying the operator every time the orepass is blocked, as well as providing different 3D views of the environment. A proof of concept is presented using a scaled setup, where it is validated that the use of the proposed system allows for providing a better and more efficient tele-operation experience.

Low Frequency Transparency Analyses of Passivity-based Approaches for Two-channel Haptic Teleoperation Architectures

Low Frequency Transparency Analyses of Passivity-based Approaches for Two-channel Haptic Teleoperation Architectures

Optimal adaptive computed torque control for haptic-teleoperation system with uncertain dynamics

This study aimed to eliminate dynamic uncertainty, one of the main problems of haptic teleoperation robotic systems. The optimal adaptive computed torque control method was used to overcome this problem. As is known, excellent stability and transparency are required in teleoperation systems. However, dynamic uncertainty that causes stability problems in the control of these systems also causes poor performance. In conventional adaptive computed torque control methods, updating the parameters of the system is generally discussed, but updating the control coefficients of vital importance in the control of the system is not considered. In the proposed method, an adaptation rule has been created to update uncertain parameters. In addition, the gray wolf optimization algorithm, one of the current optimization algorithms, has been proposed and applied to obtain the control coefficients of the system. The position tracking stability of the system was examined by using Lyapunov stability analysis method. As a result, both simulation and real-time optimal adaptive computational torque control method were used and bilateral position and force control was performed and the performance results of the system are obtained graphically and examined. Optimal adaptive computed torque control method obtained using the gray wolf optimization algorithm was used first in the literature search and success results have been obtained. In this regard, the authors have the idea that this work is an innovative aspect of both simulation and real time with the optimal adaptive computed torque control method.

A unifying framework for transparency optimized controller design in multilateral teleoperation with time delays

Multilateral teleoperation is an umbrella term for various N-robot haptic teleoperation schemes and contains bilateral teleoperation as a special case with N=2. While transparency of bilateral teleoperation systems can be analyzed through hybrid matrices and transparent controllers can be designed with four channel/Lawrence architectures, such a general analysis and design framework has not yet been established for multilateral teleoperation systems. Here, a framework is proposed to achieve transparency in multi-user shared control multilateral teleoperation systems even in the presence of time delays. First, a novel control law unifying a wide range of dominance factor, control and communication architectures is proposed. This control law enables the adaptation of all bilateral linear Lawrence architectures to multilateral teleoperation. Second, a novel transparency analysis approach is proposed and it is shown that TOPF architecture is the only Lawrence architecture that can potentially achieve ideal transparency in multilateral teleoperation under time delays. Third, based on proposed L2 stability criteria, a trade-off between stability robustness and transparency under varying time delays is demonstrated. Effectiveness of the proposed framework is validated with numerical analyses and experiments.

Single-master/multi-slave haptic teleoperation: a force control scheme

PurposeThis paper aims to present a novel modular design framework for the haptic teleoperation of single-master/multiple-slave (SM/MS) systems with cooperating manipulators.Design/methodology/approachThe user commands the remote-leader robot and the slave remote robot follows the leader in a leader–follower formation. The remote-slave is purely force-controlled. A virtual model of the remote environment is introduced between the local and remote environments through simulation software. Locally generated motion inputs are transmitted to the remote environment through the virtual model. A haptic coupling is designed in the virtual environment and the haptic feedback is transmitted to the user along with the forces measured in the remote environment. The controllers proposed in this work are experimentally evaluated with experienced and inexperienced users.FindingsThe proposed haptic interaction model contributes to the total force feedback and smoothens the high-frequency signals occurring at the physical interaction in the remote environment. Experimental results show that the implemented controllers including the proposed haptic interaction improve the teleoperation performances in terms of trajectory tracking. Furthermore, pure force control of the remote-slave is shown to enhance the robustness of the teleoperation against external disturbances. Satisfactory teleoperation performances are observed with both experienced and inexperienced users.Originality/valueThe proposed SM/MS teleoperation system involves a multi-purpose virtual simulator and a purely force-controlled remote-slave manipulator in a modular cooperative configuration. The uniquely defined structure of the proposed haptic coupling is used in modeling the interaction between the local and remote manipulators on the one hand, and between cooperating remote manipulators on the other.

Haptic Teleoperation of Flexible Needles Combining 3D Ultrasound Guidance and Needle Tip Force Feedback

We present a haptic teleoperation system capable of steering flexible needles under ultrasound imaging toward a target. With respect to similar works, this approach enables intuitive control of the needle motion while providing the user with 3D navigation and needle tip cutting force using a combination of kinesthetic and vibrotactile haptic feedback. The needle is tracked during the insertion using a 3D ultrasound probe. A friction estimation algorithm extracts salient information about the cutting force at the needle tip from a force sensor placed at the needle base. A grounded haptic interface enables natural 6-DoF control of the needle motion while providing kinesthetic feedback, and a wearable cutaneous interface on the forearm provides distributed vibrotactile sensations. We carried out a human subject study to validate the insertion system in a gelatine phantom and compare seven different feedback techniques. The best performance was registered when providing navigation cues through kinesthetic feedback and needle tip cutting force through cutaneous vibrotactile feedback. In this modality, results showed an 87% accuracy improvement with respect to providing no haptic feedback at all.

Dual-Master/Single-Slave Haptic Teleoperation System for Semiautonomous Bilateral Control of Hexapod Robot Subject to Deformable Rough Terrain

The increasing application requirements of multilegged walking robots in outdoor environments pose new challenges regarding the design of their teleoperation systems. Some of these challenges arise from the multiple degrees of freedom of the telerobotic system and nonpassive exogenous disturbance. Herein, a novel control system based on a dual-master/single-slave bilateral haptic teleoperation framework using a semiautonomous strategy for hexapod robots walking on deformable rough terrains is proposed. In this teleoperation system, the body velocities and postures of the hexapod robot are determined according to the positions of two haptic master robots. The proposed teleoperator includes a time-domain passivity control approach to compensate for the system’s potential nonpassivity induced by the contact slippage between the foot and the ground. Furthermore, a posture-level bilateral controller is designed to overcome the unpredictable posture vibration. Information about the velocity loss and posture error is displayed to the human operator in the form of haptic force. In the underlying controller of the slave robot, a foot-force optimization algorithm is developed to improve the local autonomy of the teleoperation system. Furthermore, the stability of the system is demonstrated by its passivity. Experimental results indicate that the proposed controllers can provide a stable and transparent bilateral haptic teleoperation system for a hexapod robot under environmental perturbations.

Conceptualisation of Human-on-the-Loop Haptic Teleoperation With Fully Autonomous Self-Driving Vehicles in the Urban Environment

The automotive industry aims to deploy commercial level-5 fully autonomous self-driving vehicles (FA-SDVs) in a diverse range of benefit-driven concepts on city roads in the years to come. In all future visions of operating networks of FA-SDVs, humans are expected to intervene with some kind of remote supervisory role. Recent advances in cyber-physical systems (CPS) within the concept of Internet of Everything (IoE) using tactile internet (TI) teleport us to teleoperate remote objects within the cyber-world. Human-on-the-loop (HOTL) haptic teleoperation with an extension of human control and sensing capability by coupling with artificial sensors and actuators with an increased sense of real-time driving in the remote vehicle can help overcome the challenging tasks when the new driver - artificial intelligence (AI) agent - encounters an unorthodox situation that can't be addressed by the autonomous capabilities. This paper analyses HOTL real-time haptic delay-sensitive teleoperation with FA-SDVs, in the aspects of human-vehicle teamwork by establishing two similar remote parallel worlds --- real-world vehicle time-varying environment and cyber-world emulation of this environment, i.e., digital twins (DTs) --- in which a human telesupervisor (HTS), as a biological agent, can be immersed within a reasonable timescale with no cybersickness enabling omnipresence and a bidirectional flow of energy and information. The experiments conducted as a proof of concept of HOTL haptic teleoperation shows promising results and the potential of benefiting from the proposed framework.

Haptic teleoperation of a multirotor aerial robot using path planning with human intention estimation

Classical haptic teleoperation systems heavily rely on operators’ intelligence and efforts in aerial robot navigation tasks, thereby posing significantly users’ workloads. In this paper, a novel shared control scheme is presented facilitating a multirotor aerial robot haptic teleoperation system that exhibits autonomous navigation capability. A hidden Markov model filter is proposed to identify the intention state of operator based on human inputs from haptic master device, which is subsequently adopted to derive goal position for a heuristic sampling based local path planner. The human inputs are considered as commanded velocity for a trajectory servo controller to drive the robot along the planned path. In addition, vehicle velocity is perceived by the user via haptic feedback on master device to enhance situation awareness and navigation safety of the user. An experimental study was conducted in a simulated and a physical environment, and the results verify the effectiveness of the novel scheme in safe navigation of aerial robots. A user study was carried out between a classical haptic teleoperation system and the proposed approach in the identical simulated complex environment. The flight data and task load index (TLX) are acquired and analyzed. Compared with the conventional haptic teleoperation scheme, the proposed scheme exhibits superior performance in safe and fast navigation of the multirotor vehicle, and is also of low task and cognitive loads.

Modeling and Emulating a Physiotherapist's Role in Robot‐Assisted Rehabilitation

In home‐based rehabilitation, one possible approach is haptic teleoperation in which a hospital‐based therapist is haptically linked and tele‐presented to a home‐based patient to effectively simulate traditional in‐hospital therapies over a distance. In this context, this article proposes a learn‐and‐replay (LAR) paradigm that consists of two phases: a therapist‐in‐loop (interactive) phase where the therapist interacts through the haptic teleoperation loop with the patient to perform the cooperative therapy task, and a therapist‐out‐of‐loop (standalone) phase where the therapist's task is played by the patient‐side robot in future repetitions. During the interactive phase, the therapist demonstrates impedance during cooperating with the patient. During the standalone phase, the patient‐side robot is automatically controlled to mimic the therapist's demonstrated impedance which is learned in the interactive phase. The direct force reflection (DFR) architecture is utilized as the control method for the bilateral telerehabilitation system. Case studies involving 1‐degree‐of‐freedom and 2‐degree‐of‐freedom cooperative manipulation tasks are tested for proof of concept. The results show that the impedance of the therapist's arm can be replicated by the patient‐side robot for both tasks and proposed LAR telerehabilitation paradigm that assists the therapist in the rehabilitation procedure to take care of other tasks or attend to other patients.

Haptic Teleoperation of UAVs Through Control Barrier Functions

This article presents a novel approach to haptic teleoperation. Specifically, we use control barrier functions (CBFs) to generate force feedback to help human operators safely fly quadrotor UAVs. CBFs take a control signal as input and output a control signal that is as close as possible to the initial control signal, while also meeting specified safety constraints. In the proposed method, we generate haptic force feedback based on the difference between a command issued by the human operator and the safe command returned by a CBF. In this way, if the user issues an unsafe control command, the haptic feedback will help guide the user towards the safe input command that is closest to their current command. We conducted a within-subject user study, in which 12 participants flew a simulated UAV in a virtual hallway environment. Participants completed the task with our proposed CBF-based haptic feedback, no haptic feedback, and haptic feedback generated via parametric risk fields, which is a state-of-the-art method described in the literature. The results of this study show that CBF-based haptic feedback can improve a human operator's ability to safely fly a UAV and reduce the operator's perceived workload, without sacrificing task efficiency.