Haptic Teleoperation System Research Articles

Emulation Framework for Haptic Data Transmission Using Real-Time Transport Protocol

The Tactile Internet (TI) can be regarded as the next evolution in the world of communication. With its envisioned purpose and potential in shaping up the economy, industry and society, this paradigm aims to bring a new dimension to life by enabling humans to interact with machines remotely and in real-time with haptic and kinesthetic feedback. However, to translate this into reality, Tactile Internet will need to meet the stringent requirements of extremely low latency in conjunction with ultra-high reliability, availability, and security. This poses a challenge on the available communication systems to achieve a round-trip delay within 1 to 10 milliseconds time bound that enables the timely delivery of critical tactile and haptic sensations.
 This paper aims to evaluate the Real-Time Transport Protocol (RTP) through an emulation framework. It integrates containerization using Linux-based Docker Containers with NS-3 Network Simulator to conceptualize a haptic teleoperation system. The framework is then used to test the protocol’s feasibility for delivering texture haptic data between master and slave domains in accordance with the end-to-end delay requirements specified by IEEE 1918.1 standards. The results have shown that the timely provision of haptic data is achievable by obtaining an average round-trip delay of 17.8493 ms from the emulation experiment. As such, the results satisfy the expected IEEE 1918.1 standards constraints for medium-dynamic environment use cases.

Unified Human Intention Recognition and Heuristic-Based Trajectory Generation for Haptic Teleoperation of Non-Holonomic Vehicles

In this paper, a novel bilateral shared control approach is proposed to address the issue of strong dependence on the human, and the resulting burden of manipulation, in classical haptic teleoperation systems for vehicle navigation. A Hidden Markov Model (HMM) is utilized to handle the Human Intention Recognition (HIR), according to the force input by the human—including the HMM solution, i.e., Baum–Welch algorithm, and HMM decoding, i.e., Viterbi algorithm—and the communication delay in teleoperation systems is added to generate a temporary goal. Afterwards, a heuristic and sampling method for online generation of splicing trajectory based on the goal is proposed innovatively, ensuring the vehicle can move feasibly after the change in human intention is detected. Once the trajectory is available, the vehicle velocity is then converted to joystick position information as the haptic cue of the human, which enhances the telepresence. The shared teleoperation control framework is verified in the simulation environment, where its excellent performance in the complex environment is evaluated, and its feasibility is confirmed. The experimental results show that the proposed method can achieve simple and efficient navigation in a complex environment, and can also give a certain situational awareness to the human.

Haptic Teleoperation of Impact Hammers in Underground Mining

Impact hammers are used to reduce the size of blasted ore in mining operations. In underground mines, tele-operated impact hammers are used to reduce the size of boulders placed on the orepass’s grizzlies. An impact hammer consists of a hydraulic arm with 4 degrees of freedom, powered with a hydraulic impact hammer as an end-effector. The tele-operation of impact hammers is difficult due to the latency of communications, the poor visibility of the environment, and the used 2D interfaces. This may result in a collision with the hammer and the infrastructure, idle strokes, and non-optimal operation. To address these issues, this paper proposes the haptic tele-operation of impact hammers. The proposed haptic tele-operation system is based on a 3D model of the environment, which is used to estimate repulsion forces that are transferred to the operator via a haptic device, so that the hammer does not collide with the structures of the mine. The system also allows identifying the oversized boulders deposited on the grizzly and notifying the operator every time the orepass is blocked, as well as providing different 3D views of the environment. A proof of concept is presented using a scaled setup, where it is validated that the use of the proposed system allows for providing a better and more efficient tele-operation experience.

Optimal adaptive computed torque control for haptic-teleoperation system with uncertain dynamics

This study aimed to eliminate dynamic uncertainty, one of the main problems of haptic teleoperation robotic systems. The optimal adaptive computed torque control method was used to overcome this problem. As is known, excellent stability and transparency are required in teleoperation systems. However, dynamic uncertainty that causes stability problems in the control of these systems also causes poor performance. In conventional adaptive computed torque control methods, updating the parameters of the system is generally discussed, but updating the control coefficients of vital importance in the control of the system is not considered. In the proposed method, an adaptation rule has been created to update uncertain parameters. In addition, the gray wolf optimization algorithm, one of the current optimization algorithms, has been proposed and applied to obtain the control coefficients of the system. The position tracking stability of the system was examined by using Lyapunov stability analysis method. As a result, both simulation and real-time optimal adaptive computational torque control method were used and bilateral position and force control was performed and the performance results of the system are obtained graphically and examined. Optimal adaptive computed torque control method obtained using the gray wolf optimization algorithm was used first in the literature search and success results have been obtained. In this regard, the authors have the idea that this work is an innovative aspect of both simulation and real time with the optimal adaptive computed torque control method.

Haptic Teleoperation of Flexible Needles Combining 3D Ultrasound Guidance and Needle Tip Force Feedback

We present a haptic teleoperation system capable of steering flexible needles under ultrasound imaging toward a target. With respect to similar works, this approach enables intuitive control of the needle motion while providing the user with 3D navigation and needle tip cutting force using a combination of kinesthetic and vibrotactile haptic feedback. The needle is tracked during the insertion using a 3D ultrasound probe. A friction estimation algorithm extracts salient information about the cutting force at the needle tip from a force sensor placed at the needle base. A grounded haptic interface enables natural 6-DoF control of the needle motion while providing kinesthetic feedback, and a wearable cutaneous interface on the forearm provides distributed vibrotactile sensations. We carried out a human subject study to validate the insertion system in a gelatine phantom and compare seven different feedback techniques. The best performance was registered when providing navigation cues through kinesthetic feedback and needle tip cutting force through cutaneous vibrotactile feedback. In this modality, results showed an 87% accuracy improvement with respect to providing no haptic feedback at all.

Dual-Master/Single-Slave Haptic Teleoperation System for Semiautonomous Bilateral Control of Hexapod Robot Subject to Deformable Rough Terrain

The increasing application requirements of multilegged walking robots in outdoor environments pose new challenges regarding the design of their teleoperation systems. Some of these challenges arise from the multiple degrees of freedom of the telerobotic system and nonpassive exogenous disturbance. Herein, a novel control system based on a dual-master/single-slave bilateral haptic teleoperation framework using a semiautonomous strategy for hexapod robots walking on deformable rough terrains is proposed. In this teleoperation system, the body velocities and postures of the hexapod robot are determined according to the positions of two haptic master robots. The proposed teleoperator includes a time-domain passivity control approach to compensate for the system’s potential nonpassivity induced by the contact slippage between the foot and the ground. Furthermore, a posture-level bilateral controller is designed to overcome the unpredictable posture vibration. Information about the velocity loss and posture error is displayed to the human operator in the form of haptic force. In the underlying controller of the slave robot, a foot-force optimization algorithm is developed to improve the local autonomy of the teleoperation system. Furthermore, the stability of the system is demonstrated by its passivity. Experimental results indicate that the proposed controllers can provide a stable and transparent bilateral haptic teleoperation system for a hexapod robot under environmental perturbations.

Haptic teleoperation of a multirotor aerial robot using path planning with human intention estimation

Classical haptic teleoperation systems heavily rely on operators’ intelligence and efforts in aerial robot navigation tasks, thereby posing significantly users’ workloads. In this paper, a novel shared control scheme is presented facilitating a multirotor aerial robot haptic teleoperation system that exhibits autonomous navigation capability. A hidden Markov model filter is proposed to identify the intention state of operator based on human inputs from haptic master device, which is subsequently adopted to derive goal position for a heuristic sampling based local path planner. The human inputs are considered as commanded velocity for a trajectory servo controller to drive the robot along the planned path. In addition, vehicle velocity is perceived by the user via haptic feedback on master device to enhance situation awareness and navigation safety of the user. An experimental study was conducted in a simulated and a physical environment, and the results verify the effectiveness of the novel scheme in safe navigation of aerial robots. A user study was carried out between a classical haptic teleoperation system and the proposed approach in the identical simulated complex environment. The flight data and task load index (TLX) are acquired and analyzed. Compared with the conventional haptic teleoperation scheme, the proposed scheme exhibits superior performance in safe and fast navigation of the multirotor vehicle, and is also of low task and cognitive loads.

Peg-in-a-hole by haptic teleoperation

This paper presents the experimental study on the application of the peg-in-a-hole task in the proposed haptic teleoperation system. Motion references from the user towards the remote manipulator and force feedback signals from the remote environment towards the user are transmitted through a virtual reality layer where the remote operation is simulated in real-tme. Various physical interactions occuring in the bilatral loop are modelled and controlled through the introduction of virtual spring-damper elements. Modeling and controller design of the teleoperation system is made modular and straightforward with the use of identical virtual dynamics. The design approach combines the introduction of the middle layer of virtual reality and modeling of all interactons by identical virtual spring-dampers. The haptic teleoperation system is experimentally evaluated with experienced and unexperienced users in the bilateral loop. Experimental results show that stable peg-in-a-hole operation is achieved in presence of bounded time-delays in all communication channels between the user, virtual reality layer and remote environment. Improvements of the teleoperation performances in terms of position tracking and total operation time are also observed in experiments.

Adaptive Control of a Robot-Assisted Tele-Surgery in Interaction With Hybrid Tissues

In a haptic teleoperation system, which interacts with unknown and hybrid environments, it is important to achieve stability and transparency. In medical usages, the utilization of knowledge on the tissues behavior in a controller design can improve the performance of the surgery in a robot-assisted telesurgery. Simultaneous interaction with hard and soft tissues makes it difficult to achieve stability and transparency. To deal with this difficulty, two controller schemes are designed. At first, a nonlinear mathematical model (inspired by the Hunt-Crossley (HC) model), which has the properties of soft and hard tissues, is combined with the slave side dynamic. In the second approach, the reaction force applied by hybrid tissues during the transition between tissues of different properties is modeled as an unknown force acting on the slave side. In a four-channel (4-CH) architecture, nonlinear adaptive controllers are designed without any knowledge about the parameters of the master, the slave robot, and the environment. For both control schemes, Lyapunov candidate functions provide a way to ensure the stability and transparency in the presence of uncertainties. The testbed comprises two Novint Falcon robots functioning as master and slave robots. Moreover, the experiments are performed on various objects, including a soft cube, a hard cube, and a phantom tissue. This paper rigorously evaluates the performances of the proposed methods, comparing them with each other and other previous schemes. Experimental and numerical results demonstrate the effectiveness of the proposed control schemes.

A novel bilateral haptic teleoperation approach for hexapod robot walking and manipulating with legs

Teleoperation control of the multiple degrees-of-freedom (DOF) robot requires a strategy for improving its controllability and adaptability. A novel bilateral haptic teleoperation approach (multi-master single-slave) is proposed for a hexapod robot in which the first master control the primary task of the slave, e.g. the linear/steering motion; meanwhile, the second master can perform a minor task, e.g. to interact with the environment by leg. The designed teleoperation system consists of two parts: body-level and leg-level. In body-level subsystem, the linear/angular velocity of the slave body follow the first master‘s position, and the velocity loss caused by the system disturbances is fed back to the operator in the form of haptic force. In leg-level subsystem, a modified four-channel (4CH) teleoperation control architecture is proposed for the manipulable leg, which can be regarded as a manipulator, so as to guarantee the performance of the position/force tracking in the subsystem subject to parametric uncertainties, environmental perturbation and unmeasurable interaction force. Additionally, the stability of the multi-DOF bilateral haptic teleoperation system are verified via absolute stability criterion and Lyapunov theorem, respectively. Experiments of the proposed approach demonstrate that it can result in stable and transparent bilateral teleoperation with haptic force feedback.

Increasing stability in model-mediated teleoperation approach by reducing model jump effect

Model-mediated teleoperation is a predictive control approach for controlling haptic teleoperation systems whereby the environment force is virtually located on master side in order to increase the stability and transparency of the system. This promising approach, however, results in new challenges. One pivotal challenge is the model jump effect, which stems from the delay in correct creation of the virtual environment. Previous works have endeavored to reduce this effect; however, they either led to transparency decrease or assumed simplified environment models. In this paper, we propose a control approach for this aim based on the idea of decoupling. This means that when a new environment has been identified, the operation is interrupted and no signal is transmitted between master and slave sides. During this time, both sides are controlled by their own sliding mode controllers until the system reaches stability. The main advantage of this method is its independence from environment type, which makes it usable for different kinds of applications. To verify the effectiveness of the proposed approach simulation tests are conducted. The results show the system is stable in interaction with hard and soft environments in presence of large time delays in communication channels.

A new impedance force control of a haptic teleoperation system for improved transparency

In this study, a new impedance force control model of a haptic device for teleoperation is developed and analyzed. A new contact force model for the haptic device and the human hand is provided. Movements of the human hand give additional force disturbances to the force control system. Disturbance force model by haptic dynamics and hand movements is fully analyzed and included in the control system design. An adaptive control scheme is proposed to improve system transparency by achieving good force tracking performance, while simultaneously compensating human hand disturbances and sensor noises. A separate reference model for every DOF is proposed to satisfy tracking performances. Adaptive control gains cover force tracking performance and compensates for human hand disturbances while providing robustness to sensor noises.

Stable Nonlinear Trilateral Impedance Control for Dual-User Haptic Teleoperation Systems With Communication Delays

A new nonlinear adaptive impedance-based trilateral controller is proposed to ensure the absolute stability of multi-degrees-of-freedom (DOFs) dual-user haptic teleoperation systems subjected to communication delays. Using this strategy, reference impedance models are realized for the trilateral teleoperation system represented by a three-port network to facilitate cooperation of two human operators in order to perform a remote physical task. For this purpose, an impedance model defines the desired haptic interaction between the two human operators, while another impedance model specifies the desired behavior of the slave robot in terms of tracking the mater robots' trajectories during interaction with the remote environment. It is shown that different performance goals such as position synchronization and force reflection can be achieved via different adjustments to the impedance parameters. The sufficient conditions for the trilateral haptic system's absolute stability are investigated in terms of the impedance models' parameters. Accordingly, guidelines for modification of the impedance parameters are obtained to guarantee the absolute stability of the trilateral haptic system in the presence of communication time delays. A trilateral nonlinear version of the model reference adaptive impedance control (MRAIC) scheme is developed for implementing the proposed reference impedance models on the masters and the slave. The convergence of robots' trajectories to desired responses and the robustness against modeling uncertainties are ensured using the proposed controller as proven by the Lyapunov stability theorem. The proposed impedance-based control strategy is evaluated experimentally by employing a nonlinear multi-DOFs teleoperated trilateral haptic system with and without communication delays.

Nonlinear trilateral teleoperation stability analysis subjected to time-varying delays

A trilateral teleoperation system facilitates the collaboration of two users to share control of a single robot in a remote environment. While various applications of shared-control trilateral haptic teleoperation systems have recently emerged, they have mostly been studied in the context of single-DOF, LTI robotic systems. On the other hand, robotic manipulators with multiple degrees of freedom (DOF) and therefore nonlinear dynamics have recently found many applications such as in robotic-assisted surgery and therapy, space exploration and navigation systems. In this paper, considering the full nonlinear dynamical models of multi-DOF robots, stability analysis of a dual-user haptic teleoperation system is considered over a communication network subjected to asymmetrical time varying delays and through a dominance factor suitable for trainer–trainee applications. Stability in free motion and contact motion and asymptotic position tracking of the trilateral haptic teleoperation system in free motion are proven via Lyapunov stability analysis and Barbalat's lemma where operators and the environment are assumed to be passive. Simulation and experimental results concerning robot position tracking and user-perceived forces for three 2-DOF robots and experimental analysis of user-perceived stiffnesses for three 3-DOF robots validate the theoretical findings pertaining to the system stability and demonstrate the efficiency of the proposed controller.

Design of bilateral control for force feedback in surgical robot

In robot assisted laparoscopic surgery using a commercialized robot, such as the da Vinci, surgeons have encountered problems, such as organ impairment, because they depend only on visual information. To solve these problems, a haptic system is required. Force feedback and a bilateral control system are essential for realizing a haptic teleoperation system. The use of a force/torque sensor for force feedback is difficult in a surgical robot instrument, but previous studies confirmed that the reaction force loaded on a surgical robot instrument can be estimated using a sliding perturbation observer (SPO)-based reaction force estimation method. This paper suggests a bilateral controller applying SPO based force estimation method to realize the possibility of a haptic function in a laparoscopic surgery robot system. In the suggested bilateral controller, the master control uses impedance control and the slave control uses a sliding mode control (SMC). A torque and force sensorless teleoperation system can be implemented using the suggested bilateral control structure through an experimental evaluation.

Stability of cooperative teleoperation using haptic devices with complementary degrees of freedom

In bilateral teleoperation of a dexterous task, to take full advantage of the human's intelligence, experience and sensory inputs, a possibility is to engage multiple human arms through multiple masters (haptic devices) in controlling a single-slave robot with high degrees-of-freedom (DOF); the total DOFs of the masters will be equal to the DOFs of the slave. A multi-master/single-slave cooperative haptic teleoperation system with w DOFs can be modelled as a two-port network where each port (terminal) connects to a termination defined by w inputs and w outputs. The stability analysis of such a system is not trivial because of dynamic coupling across the different DOFs of the robots, the human operators and the physical or virtual environments. The unknown dynamics of the users and the environments exacerbate the problem. The authors present a novel, straightforward and convenient frequency-domain method for stability analysis of this system. As a case study, two 1-DOF and 2-DOF master haptic devices are considered to teleoperate a 3-DOF slave robot. It is qualitatively discussed how such a trilateral haptic teleoperation system may result in better task performance by splitting the various DOFs of a dexterous task between two arms of a human or two humans. Simulation and experimental results demonstrate the validity of the stability analysis framework.

복강경 수술 로봇의 힘 반향을 위한 임피던스 모델 기반의 양방향 제어

LAS (Laparoscopy Assisted Surgery) has been substituted alternatively for traditional open surgery. However, when using a commercialized robot assisted laparoscopic such as Da Vinci, surgeons have encountered some problems due to having to depend only on information by visual feedback. To solve this problem, a haptic function is required. In order to realize the haptic teleoperation system, a force feedback and bilateral control system are needed. Previous research showed that the perturbation value estimated by a SPO (Sliding Perturbation Observer) followed a reaction force that loaded on the surgical robot instrument. Thus, in this paper, the force feedback problem of surgical robots is solved through the reaction force estimation method. This paper then introduces the possibility of the haptic function realization of a laparoscopic surgery robot using a bilateral control system. For bilateral control, the master uses an impedance control and the slave uses a SMC (Sliding Mode Control). The experiment results show that a torque and force sensorless teleoperation system can be implemented using a bilateral control structure.

A method for passivity analysis of multilateral haptic systems

This paper presents a novel criterion to study the stability of multilateral teleoperation systems based on passivity. Such systems (modeled as n-port networks) have recently found interesting applications in cooperative haptic teleoperation and haptic-assisted training. The criterion provides researchers with an analytical, closed-form, necessary, and sufficient condition useful for both analysis and design of multilateral haptic teleoperation systems. The paper shows that when the proposed conditions reduce to the well-known Raisbeck’s passivity criterion for 2-port networks. The proposed conditions are used to study the passivity (and consequently the stability) of a dual-user haptic system for control of a single teleoperated robot. Simulations and experiments are performed to further test the validity of the proposed criterion.

Absolute stability of a class of trilateral haptic systems.

Trilateral haptic systems can be modeled as three-port networks. We present a criterion for absolute stability of a general class of three-port networks. Traditionally, existing (i.e., Llewellyn's) criteria have facilitated the stability analysis of bilateral haptic systems modeled as two-port networks. If the same criteria were to be used for stability analysis of a three-port network, its third port termination would need to be assumed known for it to reduce to a two-port network. This is restrictive because, for absolute stability, all three terminations of the three-port network must be allowed to be arbitrary (while passive). Extending Llewellyn's criterion, we present closed-form necessary and sufficient conditions for absolute stability of a general class of three-port networks. We first find a symmetrization condition under which a general asymmetric impedance (or admittance) matrix Z3 × 3 has a symmetric equivalent Zeq from a network stability perspective. Then, via the equivalence of passivity and absolute stability for reciprocal networks, an absolute stability condition for the original nonreciprocal network is derived. To demonstrate the convenience and utility of using this criterion for both analysis and design, it is applied to the problem of designing stabilizing controllers for dual-user haptic teleoperation systems, with simulations and experiments validating the criterion.

Passivity and Absolute Stability Analysesof Trilateral Haptic Collaborative Systems

Trilateral haptic systems can be modeled as three-port networks. Analysis of coupled stability of a three-port network can be accomplished in either the passivity or the absolute stability frameworks assuming all three ports are connected to passive but otherwise unknown terminations. This paper first introduces our recent results in terms of extending Raisbecks passivity criterion and Llewellyns absolute stability criterion to general three-port networks --- both criteria are founded on the properties of a positive-real Hermitian matrix. Next, we show that the absolute stability criterion is less conservative than the passivity criterion. Then, to show how the two criteria may be utilized at the system design stage, we apply them to the problem of designing controllers for a dual-user haptic teleoperation system and a triple-user collaborative haptic virtual environment. Using the two criteria, controllers are then designed and compared in terms of conservatism in simulations and experiments.