Telemanipulation Framework Research Articles

Hand-Guiding Gesture-Based Telemanipulation with the Gesture Mode Classification and State Estimation Using Wearable IMU Sensors

This study proposes a telemanipulation framework with two wearable IMU sensors without human skeletal kinematics. First, the states (intensity and direction) of spatial hand-guiding gestures are separately estimated through the proposed state estimator, and the states are also combined with the gesture’s mode (linear, angular, and via) obtained with the bi-directional LSTM-based mode classifier. The spatial pose of the 6-DOF manipulator’s end-effector (EEF) can be controlled by combining the spatial linear and angular motions based on integrating the gesture’s mode and state. To validate the significance of the proposed method, the teleoperation of the EEF to the designated target poses was conducted in the motion-capture space. As a result, it was confirmed that the mode could be classified with 84.5% accuracy in real time, even during the operator’s dynamic movement; the direction could be estimated with an error of less than 1 degree; and the intensity could be successfully estimated with the gesture speed estimator and finely tuned with the scaling factor. Finally, it was confirmed that a subject could place the EEF within the average range of 83 mm and 2.56 degrees in the target pose with only less than ten consecutive hand-guiding gestures and visual inspection in the first trial.

A Shared Autonomy Reconfigurable Control Framework for Telemanipulation of Multi-Arm Systems

Teleoperation is a widely adopted strategy to control robotic manipulators executing complex tasks that require highly dexterous movements and critical high-level intelligence. Classical teleoperation schemes are based on either joystick control, or on more intuitive interfaces which map directly the user arm motions into one robot arm's motions. These approaches have limits when the execution of a given task requires reconfigurable multiple robotic arm systems. Indeed, the simultaneous teleoperation of two or more robot arms could extend the workspace of the manipulation cell, or increase its total payload, or afford other advantages. In different phases of a reconfigurable multi-arm system, each robot could act as an independent arm, or as one of a pair of cooperating arms, or as one of the fingers of a virtual, large robot hand. This manuscript proposes a novel telemanipulation framework that enables both the individual and combined control of any number of robotic arms. Thanks to the designed control architecture, the human operator can intuitively choose the proposed control modalities and the manipulators that make the task convenient to execute through the user interface. Moreover, through the tele-impedance paradigm, the system can address complex tasks that require physical interaction by letting the robot mimic the arm impedance and position references of the human operator. The proposed framework is validated with 8 subjects controlling 4 Franka Emika Panda robots with 7-DoFs to execute a telemanipulation task. Qualitative results of the experiments show us the promising applicability of our framework.

Evaluation of Hybrid Control and Palpation Assistance for Situational Awareness in Telemanipulated Task Execution

The use of intelligent feedback modalities to control and react to interaction forces during surgical procedures is an important factor in enabling safe and precise surgery. We explore the use of a model-mediated telemanipulation framework to enhance a user’s situational awareness using assistive virtual fixtures and semi-automated task execution for safe and intuitive environment interaction during robotic laparoscopic surgery. The framework allows stiffness mapping with semi-autonomous excitation, hybrid position-force control, and model updates during soft geometry contact. A 24-person study was carried out at 3 sites in simulated ablation and palpation of phantom anatomy. Compared to methods lacking intelligent feedback and guidance, the proposed framework improved task execution metrics (force regulation, completion time, path-following error) and reduced user effort.

Synergy-Based Bilateral Port: A Universal Control Module for Tele-Manipulation Frameworks Using Asymmetric Master-Slave Systems.

Endowing tele-manipulation frameworks with the capability to accommodate a variety of robotic hands is key to achieving high performances through permitting to flexibly interchange the end-effector according to the task considered. This requires the development of control policies that not only cope with asymmetric master–slave systems but also whose high-level components are designed in a unified space in abstraction from the devices specifics. To address this dual challenge, a novel synergy port is developed that resolves the kinematic, sensing, and actuation asymmetries of the considered system through generating motion and force feedback references in the hardware-independent hand postural synergy space. It builds upon the concept of the Cartesian-based synergy matrix, which is introduced as a tool mapping the fingertips Cartesian space to the directions oriented along the grasp principal components. To assess the effectiveness of the proposed approach, the synergy port has been integrated into the control system of a highly asymmetric tele-manipulation framework, in which the 3-finger hand exoskeleton HEXOTRAC is used as a master device to control the SoftHand, a robotic hand whose transmission system relies on a single motor to drive all joints along a soft synergistic path. The platform is further enriched with the vision-based motion capture system Optitrack to monitor the 6D trajectory of the user’s wrist, which is used to control the robotic arm on which the SoftHand is mounted. Experiments have been conducted with the humanoid robot COMAN and the KUKA LWR robotic manipulator. Results indicate that this bilateral interface is highly intuitive and allows users with no prior experience to reach, grasp, and transport a variety of objects exhibiting very different shapes and impedances. In addition, the hardware and control solutions proved capable of accommodating users with different hand kinematics. Finally, the proposed control framework offers a universal, flexible, and intuitive interface allowing for the performance of effective tele-manipulations.

Force-Controlled Exploration for Updating Virtual Fixture Geometry in Model-Mediated Telemanipulation

This paper proposes an approach for using force-controlled exploration data to update and register an a priori virtual fixture geometry to a corresponding deformed and displaced physical environment. An approach for safe exploration implementing hybrid motion/force control is presented on the slave robot side. During exploration, the shape and the local surface normals of the environment are estimated and saved in an exploration data set. The geometric data collected during this exploration scan are used to deform and register the a priori environment model to the exploration data set. The environment registration is achieved using a deformable registration based on the coherent point drift method. The task-description of the high-level assistive telemanipulation law, called a virtual fixture (VF), is then deformed and registered in the new environment. The new model is updated and used within a model-mediated telemanipulation framework. The approach is experimentally validated using a da-Vinci research kit (dVRK) master interface, a dVRK patient side manipulator, and a Cartesian stage robot. Experiments demonstrate that the updated VF and the updated model allow the users to improve their path following performance and to shorten their completion time when the updated path following VF is applied. The approach presented has direct bearing on a multitude of surgical applications including force-controlled ablation.

Multicontact Bilateral Telemanipulation With Kinematic Asymmetries

We propose a novel bilateral telemanipulation framework to tame master and slave devices having different structures. This condition applies to multicontact teleoperation scenarios where the number of contact points on the slave side and the number of interaction points on the master side are different. An example is a master device interacting with the thumb and the index fingertips of the human operator, and as slave device, a robotic arm with a multifingered robotic hand. In case of a manipulation task, it is not straightforward to transmit motion commands and reflect forces from the interaction with the environment. A general telemanipulation framework, that does not consider the specific kinematics of the devices involved, is needed. The main idea of this study is to take advantage of a virtual object as a mediator between the master and slave side. The arising forward and backward mapping algorithms are able to relate the motions and the exerted forces of very dissimilar systems. The approach has been evaluated in a case study consisting of two haptic interfaces used both to track the index and thumb motions and to render forces on the master side and a robotic arm with a multifingered hand as end effector on the slave side. The results presented in this paper can be extended to cooperative grasping scenarios where multiple robots telemanipulate the same object.

Choice of handedness and automated suturing for anthropomorphic dual-arm surgical robots

SUMMARYLaparoscopic and Single Port Access Surgery (SPAS) present unique dexterity challenges related to dual-arm operations in confined spaces and tele-manipulation of highly dexterous surgical slaves. In an effort to reduce tele-manipulation burden, new paradigms for semi-automating surgical tasks are needed. This paper presents a new minimal constraint suturing and automated choice of handedness for anthropomorphic dual-arm robots. The automated choice of handedness supports surgeons during tele-manipulation of complex robotic slaves where dexterity and workspace constraints are difficult to learn. This criterion is also used to support automated dual-arm rendezvous for quicker suture exchange during dual-arm suturing. The minimal constraint algorithm presented in this paper allows surgeons to operate within a shared-control tele-manipulation framework whereby the surgeon controls the needle insertion speed and the robot controls the needle orientation while respecting a minimalistic set of tissue constraints. This framework is evaluated on a novel insertable robotic end-effectors platform for SPAS. A simulation study demonstrates the effectiveness of the automated choice of handedness criterion through a study of dexterity limitations of each arm. Additional simulations show the proposed algorithm for automated rendezvous and suture exchange.