Robotic Telemanipulation Research Articles

Robotic telemanipulation with EMG-driven strategy-assisted shared control method

Robotic telemanipulation with EMG-driven strategy-assisted shared control method

Integrating Virtual, Mixed, and Augmented Reality into Remote Robotic Applications: A Brief Review of Extended Reality-Enhanced Robotic Systems for Intuitive Telemanipulation and Telemanufacturing Tasks in Hazardous Conditions

There is an increasingly urgent need for humans to interactively control robotic systems to perform increasingly precise remote operations, concomitant with the rapid development of space exploration, deep-sea discovery, nuclear rehabilitation and management, and robotic-assisted medical devices. The potential high value of medical telerobotic applications was also evident during the recent coronavirus pandemic and will grow in future. Robotic teleoperation satisfies the demands of the scenarios in which human access carries measurable risk, but human intelligence is required. An effective teleoperation system not only enables intuitive human-robot interaction (HRI) but ensures the robot can also be operated in a way that allows the operator to experience the “feel” of the robot working on the remote side, gaining a “sense of presence”. Extended reality (XR) technology integrates real-world information with computer-generated graphics and has the potential to enhance the effectiveness and performance of HRI by providing depth perception and enabling judgment and decision making while operating the robot in a dynamic environment. This review examines novel approaches to the development and evaluation of an XR-enhanced telerobotic platform for intuitive remote teleoperation applications in dangerous and difficult working conditions. It presents a strong review of XR-enhanced telerobotics for remote robotic applications; a particular focus of the review includes the use of integrated 2D/3D mixed reality with haptic interfaces to perform intuitive remote operations to remove humans from dangerous conditions. This review also covers primary studies proposing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) solutions where humans can better control or interact with real robotic platforms using these devices and systems to extend the user’s reality and provide a more intuitive interface. The objective of this article is to present recent, relevant, common, and accessible frameworks implemented in research articles published on XR-enhanced telerobotics for industrial applications. Finally, we present and classify the application context of the reviewed articles in two groups: mixed reality–enhanced robotic telemanipulation and mixed reality–enhanced robotic tele-welding. The review thus addresses all elements in the state of the art for these systems and ends with recommended research areas and targets. The application range of these systems and the resulting recommendations is readily extensible to other application areas, such as remote robotic surgery in telemedicine, where surgeons are scarce and need is high, and other potentially high-risk/high-need scenarios.

Haptic-guided grasping to minimise torque effort during robotic telemanipulation

Haptic-guided grasping to minimise torque effort during robotic telemanipulation

Control Evaluation of Antagonistic Series Elastic Actuation for a Robotic Endoscope Joint

Haptic feedback is typically missing during telemanipulation of surgical robots in minimally invasive surgeries, i.e., surgeons cannot feel the interaction forces between the instruments and tissues. Instead, surgeons have to solely rely on visual feedback, which increases complexity of guiding the instruments and poses the safety threat of unperceivable contacts outside the field of view. We propose a novel series elastic actuation design for articulated robotic endoscopes to overcome these limitations and evaluate an according device with one joint. Similar to the actuation of human fingers, the joint is driven by antagonistic tendons. Springs are integrated in the transmission between the tendons and the motors outside of the endoscope shaft. We estimated the joint angle and thereby the endoscope shape, measured spring deflection, estimated tendon forces from that deflection, and implemented force control for the endoscope joint. Zero torque control and impedance control were evaluated under application of both a continuous force and an impact force to the endoscope tip. The springs reduced impact forces at the tip of the endoscope through their inherent compliance. At the same time, feeding back the estimated force resulted in a stable tendon force control and a tunable endoscope joint control: Zero torque control effectively reduced the external forces, while the endoscope joint showed the expected stiffness in impedance control. These results show that antagonistic series elastic actuation is a promising concept for endoscope joint actuation and that it can lead towards safer robot–tissue interactions in surgical robotics.

On EMG Based Dexterous Robotic Telemanipulation: Assessing Machine Learning Techniques, Feature Extraction Methods, and Shared Control Schemes

Electromyography (EMG) signals are commonly used for the development of Muscle Machine Interfaces. EMG-based solutions provide intuitive and often hand-free control in a wide range of applications that range from the decoding of human intention in classification tasks to the continuous decoding of human motion employing regression models. In this work, we compare various machine learning and feature extraction methods for the creation of EMG based control frameworks for dexterous robotic telemanipulation. Various models are needed that can decode dexterous, in-hand manipulation motions and perform hand gesture classification in real-time. Three different machine learning methods and eight different time-domain features were evaluated and compared. The performance of the models was evaluated in terms of accuracy and time required to predict a data sample. The model that presented the best performance and prediction time trade-off was used for executing in real-time a telemanipulation task with the New Dexterity Autonomous Robotic Assistance (ARoA) platform (a humanoid robot). Various experiments have been conducted to experimentally validate the efficiency of the proposed methods. The robotic system is shown to successfully complete a series of tasks autonomously as well as to efficiently execute tasks in a shared control manner.

Haptic User Interface of a Cable-Driven Input Device to Control the End Effector of a Surgical Telemanipulation System

Abstract In robotic telemanipulation for minimally-invasive surgery, lack of haptic sensation and non-congruent movement of input device and manipulator are major drawbacks. Input devices based on cable-driven parallel mechanisms have the potential to be a stiff alternative to input devices based on rigid parallel or serial kinematics by offering low inertia and a scalable workspace. In this paper, the haptic user interface of a cable-driven input device and its technical specifications are presented and assessed. The haptic user interface allows to intuitively control the gripping movement of the manipulator’s end effector by providing a two-finger precision grasp. By design, the interface allows to command input angles between 0° and 45°. Furthermore, interaction forces from the manipulator’s end effector can be displayed to the user’s twofinger grasp in a range from 0 N to 6 N with a frequency bandwidth of 17 Hz.

Caring About the Human Operator: Haptic Shared Control for Enhanced User Comfort in Robotic Telemanipulation.

Haptic shared control enables a human operator and an autonomous controller to share the control of a robotic system using haptic active constraints. It has been used in robotic teleoperation for different purposes, such as navigating along paths minimizing the torques requested to the manipulator or avoiding possibly dangerous areas of the workspace. However, few works have focused on using these ideas to account for the user's comfort. In this article, we present an innovative haptic-enabled shared control approach aimed at minimizing the user's workload during a teleoperated manipulation task. Using an inverse kinematic model of the human arm and the rapid upper limb assessment (RULA) metric, the proposed approach estimates the current user's comfort online. From this measure and an a priori knowledge of the task, we then generate dynamic active constraints guiding the users towards a successful completion of the task, along directions that improve their posture and increase their comfort. Studies with human subjects show the effectiveness of the proposed approach, yielding a 30% perceived reduction of the workload with respect to using standard guided human-in-the-loop teleoperation.

A Haptic Shared-Control Architecture for Guided Multi-Target Robotic Grasping.

Although robotic telemanipulation has always been a key technology for the nuclear industry, little advancement has been seen over the last decades. Despite complex remote handling requirements, simple mechanically linked master-slave manipulators still dominate the field. Nonetheless, there is a pressing need for more effective robotic solutions able to significantly speed up the decommissioning of legacy radioactive waste. This paper describes a novel haptic shared-control approach for assisting a human operator in the sort and segregation of different objects in a cluttered and unknown environment. A three-dimensional scan of the scene is used to generate a set of potential grasp candidates on the objects at hand. These grasp candidates are then used to generate guiding haptic cues, which assist the operator in approaching and grasping the objects. The haptic feedback is designed to be smooth and continuous as the user switches from a grasp candidate to the next one, or from one object to another one, avoiding any discontinuity or abrupt changes. To validate our approach, we carried out two human-subject studies, enrolling 15 participants. We registered an average improvement of 20.8%, 20.1%, and 32.5% in terms of completion time, linear trajectory, and perceived effectiveness, respectively, between the proposed approach and standard teleoperation.

Haptic-Based Shared-Control Methods for a Dual-Arm System

We propose novel haptic guidance methods for a dual-arm telerobotic manipulation system, which are able to deal with several different constraints, such as collisions, joint limits, and singularities. We combine the haptic guidance with shared-control algorithms for autonomous orientation control and collision avoidance meant to further simplify the execution of grasping tasks. The stability of the overall system in various control modalities is presented and analyzed via passivity arguments. In addition, a human subject study is carried out to assess the effectiveness and applicability of the proposed control approaches both in simulated and real scenarios. Results show that the proposed haptic-enabled shared-control methods significantly improve the performance of grasping tasks with respect to the use of classic teleoperation with neither haptic guidance nor shared control.

Inverse Kinematics Control Methods for Redundant Snakelike Robot Teleoperation During Minimally Invasive Surgery

The real-time teleoperation or telemanipulation of redundant snakelike robots for minimally invasive surgery in a master-slave configuration is a complex problem. There are many possible mappings between a master's standard 6 degrees of freedom (DOF) and a redundant slave robot, typically with n ≫ 6 DOF. This letter introduces a snakelike robot for ear, nose, and throat surgery. The robot's architecture is comprised of n = 26 joint variables. Six different control methods were investigated. The methods are compared through simulation with a user study. Each participant performed the same task using each of the six different control methods. Based on the metrics selected, the sparse pseudo-L 0 and our proposed approach performed better in terms of intuitiveness, real-time capabilities, and overall occupied volume.

Bilateral Control Between Electric and Hydraulic Actuators Using Linearization of Hydraulic Actuators

In this study, bilateral control between electric and hydraulic actuators is investigated. Bilateral control realizes telemanipulation of remote robots with the sense of touch. We focus on the system of local (master) robots, manipulated by operators that are driven by electric actuators, and slave robots, which work in remote places that are driven by hydraulic actuators. To obtain bilateral control in this situation, we find that combination of three key techniques is necessary. The first key is oblique coordinate control, which can decouple position and force controllers of bilateral control. The second key is linearization of hydraulic actuators as third-order systems including oil compression by using disturbance observers. In the absence of linearization, the design becomes practically impossible because of the need to apply complicated models. Since the master robots are second-order systems, and the slave robots are linearized as third-order systems, oblique coordinate control cannot be implemented. Then, the last key is the agreement of system order of electric and hydraulic actuators by using a pseudodifferentiator. Two conventional controllers and the proposed method are experimentally and analytically compared. The proposed method shows the best tracking performance and is the most stable in contact control among the three controllers.

Telemanipulation of cooperative robots: a case of study

ABSTRACTThis article addresses the problem of dexterous robotic grasping by means of a telemanipulation system composed of a single master and two slave robot manipulators. The slave robots are analysed as a cooperative system where it is assumed that the robots can push but not pull the object. In order to achieve a stable rigid grasp, a centralised adaptive position-force control algorithm for the slave robots is proposed. On the other hand, a linear velocity observer for the master robot is developed to avoid numerical differentiation. A set of experiments with different human operators were carried out to show the good performance and capabilities of the proposed control-observer algorithm. In addition, the dynamic model and closed-loop dynamics of the telemanipulation is presented.

Steerable Robot-assisted Micromanipulation in the Middle Ear: Preliminary Feasibility Evaluation.

The use of a robotic manipulator with a dexterously orientable gripper will expand the ability of middle ear surgeons to perform precise tasks and access otherwise challenging anatomic regions. Middle ear surgery presents unique challenges because of the constrained operative space and limited access to certain anatomic regions. A custom-designed robot with a sideways-reaching gripper was used to evaluate feasibility of manipulation tasks in different middle-ear anatomical zones. Reachable workspace within the middle ear, accuracy of free-space path following, and tool steadiness were compared between robotic telemanipulation and manual control. Preliminarily assessments of the robot's clinical utility included: 1) touching the round window niche, Eustachian tube orifice, and sinus tympani; 2) placing a stapes prosthesis; 3) removal of mockup diseased tissue in the sinus tympani. The reachable workspace in the middle ear was considerably greater with the robot as compared with manual manipulation using a Rosen needle. In a simple path-tracing task outside the ear, robotic telemanipulation was associated with significantly reduced error. Within the middle ear, the robot contributed to steadier movement, but longer task completion time. The gripper successfully placed a 4.5 mm piston prosthesis, accessed the round window niche, Eustachian tube orifice, and removed mockup disease from the sinus tympani. This study demonstrates that robotic assistance using steerable tools allows surgeons to access challenging anatomic regions of the middle ear. Coordinated and accurate manipulation is evidenced by motion analyses and completion of feasibility tasks within the middle ear.

Human stochastic closed-loop behavior for master-slave teleoperation using multi-leap-motion sensor

Teleoperation has a wide range of applications that have been under development over the past two decades. Previous researches have focused on the control design of teleoperation machine systems to deal with obstacles such as time-delayed stability and transparency. Recent researches have shown that the inclusion of human closed-loop dynamics in control design can improve the performance of robot telemanipulation. The complexity of human behavior arises from the uncertainty of both human physiology and psychology; hence, the investigation can benefit from empirical studies. This study develops a type of statistical learning method to model and evaluate human stochastic closed-loop behavior, which is considered as a hand motion during the direct incremental control process of master-slave teleoperation. The hand trajectory is empirically considered as having a binary linear regression relationship with the error and error rate between the demanded and simulated teleoperator trajectories, while random movements with zero error and error rate are discovered. Hand movement tracking is achieved using a multi-leap-motion sensor (MLM), which is a markerless and natural infrared vision-based manner for motion capture. The established behavior model and statistical learning results reveal certain human properties of operational activities including visual perception, decision making, and robot telemanipulation. The properties indicate some probable system enhancements for future work.

Minimally Invasive Mitral Valve Surgery: An Update

Minimally invasive mitral valve surgery (MIMVS) was introduced in the mid-1990s to refer to a variety of surgical techniques that avoid full sternotomy through smaller or alternative chest wall incisions, as an attempt to reduce complications, but at the same time preserve outcomes of the full sternotomy approach. In this review, different aspects of MIMVS are discussed in relation to its approaches (right parasternal incision, lower ministernotomy, right anterolateral minithoracotomy and left posterior minithoracotomy) as well as strategies (mini-incisions, video-assistance, video direction with robotic assistance and telemanipulation) passing through more than 2 decades of continuous evolution. In the current practice MIMVS shows similar outcome to conventional surgery with even more superior results regarding blood loss, ICU and hospital stay, as well as functional recovery. The accumulating experience with MIMVS encouraged surgeons to extend the application of these techniques to high-risk patients, redo surgeries, concomitant double or triple valve procedures as well as combined coronary artery and mitral valve diseases in a hybrid approach, reducing the need for full median sternotomy. In addition there is an emerging trend of transcatheter valve implantation in the mitral position with small reports of valve-in valve or valve-in- ring implantation as well as valve replacement in case of severe MAC. This new trend may establish itself in the future as a modality in treating native mitral valve diseases in high risk patients. Therefore it is recommended for cardiac centers to build up a program for MIMVS in order to fulfill the recent requirements of cardiac surgery.

Cardiac ultrasonography over 4G wireless networks using a tele-operated robot.

This Letter proposes an end-to-end mobile tele-echography platform using a portable robot for remote cardiac ultrasonography. Performance evaluation investigates the capacity of long-term evolution (LTE) wireless networks to facilitate responsive robot tele-manipulation and real-time ultrasound video streaming that qualifies for clinical practice. Within this context, a thorough video coding standards comparison for cardiac ultrasound applications is performed, using a data set of ten ultrasound videos. Both objective and subjective (clinical) video quality assessment demonstrate that H.264/AVC and high efficiency video coding standards can achieve diagnostically-lossless video quality at bitrates well within the LTE supported data rates. Most importantly, reduced latencies experienced throughout the live tele-echography sessions allow the medical expert to remotely operate the robot in a responsive manner, using the wirelessly communicated cardiac ultrasound video to reach a diagnosis. Based on preliminary results documented in this Letter, the proposed robotised tele-echography platform can provide for reliable, remote diagnosis, achieving comparable quality of experience levels with in-hospital ultrasound examinations.

Continuous Prediction of Finger Movements Using Force Myography

The tracking and prediction of hand and finger movements along with gesture recognition is an active subject of research due to its vast applications in many fields, such as prosthetic control and robotic telemanipulation of rehabilitation and assistive devices. The common challenge in all of these fields is developing a control interface for operating an automated assistive device in a more precise and human-like manner. In this paper, we aim to study the possibility of using a force-sensing resistors (FSRs) strap for the continuous prediction of finger movements, specifically those of the thumb and middle and index fingers. An array of 8 FSRs is used to construct a wearable band that wraps around the forearm. Ten healthy volunteers participated in this study. The subjects were asked to perform 3 representative grasping motions. The relative displacements of the tips of the thumb and middle and index fingers with respect to a reference marker placed on the hand (slightly above the wrist) was captured using a camera and colored marker system. An epsilon-based function of support vector regression with a radial basis function kernel is employed to train a regression model using the force myography (FMG) signal acquired from the forearm with the displacement of each fingertip. The results demonstrate the feasibility of using an FSR band for continuously predicting the displacement of the fingertips with an average squared correlation coefficient of 0.96 for the thumb and middle and index fingers, spanning three types of grasp.

Robotic total endoscopic sutureless aortic valve replacement: proof of concept for a future surgical setting.

Sutureless valves have recently enabled closed chest aortic valve replacement. This paper evaluates the feasibility of a robotic telemanipulation during thoracoscopic sutureless aortic valve implantation in cadavers. Cadavers were prepared with a five thoracosopic trocar setting, with a transthoracic clamp inserted in the first right intercostal space and the optics inserted in the second. Seven sutureless valve implantations were scheduled using 5 Sorin Perceval and 2 Medtronic 3f Enable valves. In all cases performance of pericardial suspension, aortotomy and root exposure required less than 20 min. Native valve excision and sutureless bioprosthesis implantation was technically feasible in all cases. A satisfactory prosthesis sealing was ascertained visually and the absence of paravalvular leakages was assessed with a nerve hook test around the prosthetic flange. Closed chest and robot assisted sutureless valve implantation is feasible in cadavers. Robotic technology can enhance reproducibility of the technique. Copyright © 2015 John Wiley & Sons, Ltd.

HAPTIC HUMAN INTERFACES FOR ROBOTIC TELEMANIPULATION

Recent investigation in haptic man-robot interaction suggests that there are ultimately only two topical tactile feedback generation modalities for haptic human interfaces. These allow the human operator to handle either (i) temporary virtual reality-based material replicas of the local geometric and/or force profile at the contact areas of an unlimited set of generic objects that could virtually be handled during the manipulation, or (ii) permanent material replicas of a limited set of typical objects. In this paper, the two modalities are analyzed and examples of tactile human interfaces developed by the authors for telerobotic blind tactile exploration of objects, and for telerobotic hapto-visual stylus-style tool manipulation are presented to illustrate the proposed approaches. The necessary modelling of the elastic properties of 3D objects from experimental tactile and range imaging data is also presented using a neural network architecture that becomes an important component of the haptic interface.

Angiographic evaluation of graft patency in robotic-assisted coronary artery bypass surgery: 8 year follow-up

Robotic telemanipulation systems have emerged as facilitating tools that enhance minimally invasive cardiac surgery.The purpose of this study was to evaluate graft patency by a combination of invasive and non-invasive coronary angiography methods in robotic-assisted coronary artery surgery(CABG) for optimal quality control. Between April 2004 and February 2012, patients who had robotic-assisted CABG were called to have cardiac catheterization or multislice computed tomographic angiography to evaluate graft patency. One hundred patients out of a total 250 cases were followed for graft patency for a mean period of 60.3 ± 23.8 (range 12-94) months. Mean operative time and left internal mammary artery harvest time was 165.96 ± 19.5 and 41.74 ± 5.9 (range 30-55) min. Postoperative graft patency rate was 94% and freedom from target vessel re-interventions was 98%. Robotic-assisted CABG can be accomplished with low morbidity, mortality and re-intervention rates. It is a safe procedure in selected patients and produces excellent mid-term graft patency.