Hand Object Research Articles

In-Hand Object Classification and Pose Estimation With Sim-to-Real Tactile Transfer for Robotic Manipulation

In-Hand Object Classification and Pose Estimation With Sim-to-Real Tactile Transfer for Robotic Manipulation

Validations of various in-hand object manipulation strategies employing a novel tactile sensor developed for an under-actuated robot hand.

Prisma Hand II is an under-actuated prosthetic hand developed at the University of Naples, Federico II to study in-hand manipulations during grasping activities. 3 motors equipped on the robotic hand drive 19 joints using elastic tendons. The operations of the hand are achieved by combining tactile hand sensing with under-actuation capabilities. The hand has the potential to be employed in both industrial and prosthetic applications due to its dexterous motion capabilities. However, currently there are no commercially available tactile sensors with compatible dimensions suitable for the prosthetic hand. Hence, in this work, we develop a novel tactile sensor designed based on an opto-electronic technology for the Prisma Hand II. The optimised dimensions of the proposed sensor made it possible to be integrated with the fingertips of the prosthetic hand. The output voltage obtained from the novel tactile sensor is used to determine optimum grasping forces and torques during in-hand manipulation tasks employing Neural Networks (NNs). The grasping force values obtained using a Convolutional Neural Network (CNN) and an Artificial Neural Network (ANN) are compared based on Mean Square Error (MSE) values to find out a better training network for the tasks. The tactile sensing capabilities of the proposed novel sensing method are presented and compared in simulation studies and experimental validations using various hand manipulation tasks. The developed tactile sensor is found to be showcasing a better performance compared to previous version of the sensor used in the hand.

AngoraPy: A Python toolkit for modeling anthropomorphic goal-driven sensorimotor systems.

Goal-driven deep learning increasingly supplements classical modeling approaches in computational neuroscience. The strength of deep neural networks as models of the brain lies in their ability to autonomously learn the connectivity required to solve complex and ecologically valid tasks, obviating the need for hand-engineered or hypothesis-driven connectivity patterns. Consequently, goal-driven models can generate hypotheses about the neurocomputations underlying cortical processing that are grounded in macro- and mesoscopic anatomical properties of the network's biological counterpart. Whereas, goal-driven modeling is already becoming prevalent in the neuroscience of perception, its application to the sensorimotor domain is currently hampered by the complexity of the methods required to train models comprising the closed sensation-action loop. This paper describes AngoraPy, a Python library that mitigates this obstacle by providing researchers with the tools necessary to train complex recurrent convolutional neural networks that model the human sensorimotor system. To make the technical details of this toolkit more approachable, an illustrative example that trains a recurrent toy model on in-hand object manipulation accompanies the theoretical remarks. An extensive benchmark on various classical, 3D robotic, and anthropomorphic control tasks demonstrates AngoraPy's general applicability to a wide range of tasks. Together with its ability to adaptively handle custom architectures, the flexibility of this toolkit demonstrates its power for goal-driven sensorimotor modeling.

Tracking Object’s Pose via Dynamic Tactile Interaction

It is a challenging task to localize and track an in-hand object in robotic domain. Researchers were mainly using the vision as major modality for extracting object’s pose. The vision approaches are fragile when the object is occluded by the robotic arm and hand. To this end, we propose a tactile-based DTI-Tracker (tracking object’s pose via Dynamic Tactile Interaction) approach and formalize the object’s tracking as a filter problem. An Extended Kalman Filter (EKF) is used to estimate the in-hand object pose exploiting the high spatial resolution tactile feedback. Given the initial estimation error, the proposed approach rapidly converges the estimation result to the real pose and the statistic evaluation shows the robustness of the proposed approach. We evaluate this method in physics simulation and real multi-fingered grasping setup while the object is static and movable. The proposed method is a potential tool to foster future research on dexterous manipulation using multifingered robotic hand.

Visual dexterity: In-hand reorientation of novel and complex object shapes.

In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in less structured environments, which remain beyond the reach of current robots. Prior works built reorientation systems assuming one or many of the following conditions: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasi-static manipulation, simulation-only results, the need for specialized and costly sensor suites, and other constraints that make the system infeasible for real-world deployment. We present a general object reorientation controller that does not make these assumptions. It uses readings from a single commodity depth camera to dynamically reorient complex and new object shapes by any rotation in real time, with the median reorientation time being close to 7 seconds. The controller was trained using reinforcement learning in simulation and evaluated in the real world on new object shapes not used for training, including the most challenging scenario of reorienting objects held in the air by a downward-facing hand that must counteract gravity during reorientation. Our hardware platform only used open-source components that cost less than 5000 dollars. Although we demonstrate the ability to overcome assumptions in prior work, there is ample scope for improving absolute performance. For instance, the challenging duck-shaped object not used for training was dropped in 56% of the trials. When it was not dropped, our controller reoriented the object within 0.4 radians (23°) 75% of the time.

Intrinsically Motivated Multi-Goal Reinforcement Learning Using Robotics Environment Integrated with OpenAI Gym

Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failed experience to a successful one by relabelling the goals. In open-ended and changing environments, agents face a wide range of potential tasks that might not come with associated reward functions. Such autonomous learning agents must set their own tasks and build their own curriculum through an intrinsically motivated exploration. Because some tasks might prove easy and some impossible, agents must actively select which task to practice at any given moment, to maximize their overall mastery on the set of learnable tasks. The purpose of this technical report is two-fold. First, it introduces a suite of challenging continuous control tasks (integrated with OpenAI Gym) based on currently existing robotics hardware. The tasks include pushing, sliding and pick & place with a Fetch robotic arm as well as in-hand object manipulation with a Shadow Dexterous Hand. All tasks have sparse binary rewards and follow a Multi-Goal Reinforcement Learning (RL) framework in which an agent is told what to do using an additional input. The second part of the paper presents a set of concrete research ideas for improving RL algorithms, most of which are related to Multi-Goal RL and Hindsight Experience Replay. The Fetch environments are based on the 7-DoF Fetch robotics arm,2 which has a two-fingered parallel gripper. Agents focus on achievable tasks first and focus back on tasks that are being forgotten. Experiments conducted in a new multi-task multi-goal robotic environment show that our algorithm benefits from these two ideas and demonstrate properties of robustness to distracting tasks, forgetting and changes in body properties

Bio-Inspired Proprioceptive Touch of a Soft Finger with Inner-Finger Kinesthetic Perception.

In-hand object pose estimation is challenging for humans and robots due to occlusion caused by the hand and object. This paper proposes a soft finger that integrates inner vision with kinesthetic sensing to estimate object pose inspired by human fingers. The soft finger has a flexible skeleton and skin that adapts to different objects, and the skeleton deformations during interaction provide contact information obtained by the image from the inner camera. The proposed framework is an end-to-end method that uses raw images from soft fingers to estimate in-hand object pose. It consists of an encoder for kinesthetic information processing and an object pose and category estimator. The framework was tested on seven objects, achieving an impressive error of 2.02 mm and 11.34 degrees for pose error and 99.05% for classification.

Precise geometry and pose measurement of in-hand objects with simple features using a multi-camera system

In precision pick-and-place tasks, robotic grasp errors can cause the failure of subsequent placement steps. Therefore, in-hand object pose measurement has attracted the attention of the intelligent manufacturing industry. However, existing algorithms have limited accuracy and insufficient practicality. Moreover, they often assume that the actual object geometry is identical to the CAD model. To alleviate the problems in terms of accuracy, practicality, and uncritical assumptions, this paper proposes an in-hand measurement solution based on multi-view reconstruction for objects with simple features. In this concept, multiple monocular cameras are deployed at the robot end to calculate the object’s actual geometry and 6D pose by reconstructing its key geometric features. An in-hand pose compensation strategy and a robot terminal system integrated with manipulation and measurement are developed. To verify the solution’s effectiveness, several experiments were carried out. First, grasp errors caused by multiple sources are intuitively demonstrated in a simulation environment. The in-hand measurement is validated to compensate for grasp errors through comparison experiments. Then, robots can rearrange various irregularly piled objects in real-world experiments based on the developed terminal system. Results demonstrate that maximum remaining position errors are within 2 mm with maximum angle errors of less than 1°. The maximum error compensation rate reaches 95%, which is superior to existing in-hand measurement algorithms. It can be observed that the proposed solution is valid for multiple objects. Geometry measurement can help robots calculate the in-hand object pose and identify the object’s category and availability. Therefore, the proposed measurement solution seems accurate and universal for robot applications in smart manufacturing industries, enhancing the accuracy and flexibility of robot manipulation.

Towards robust grasping: An analysis of in-hand object motion with FBG optical fibers as force sensing technology

To enable or allow for advanced manipulation scenarios in the envisaged future in robotic manipulation, the introduction of tactile or force-sensing technology is essential. This work introduces a study on a novel integration of fiber sensing elements inside the fingers of robotics grippers that allow obtaining accurate interaction forces during object manipulation. The study focuses on in-hand object motion and analyzes the objects’ dynamic behaviors during grasping or contact phases. The proposed approach classifies the behavior of the gripper-object interaction in real-time by looking at the measured forces in the orthogonal direction of the gripper’s fingers after a filtering stage. The methodology is validated theoretically and evaluated with real experiments using objects presenting different shapes, sizes, and surface properties.

Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation.

Dexterous robotic manipulation tasks depend on estimating the state of in-hand objects, particularly their orientation. Although cameras have been traditionally used to estimate the object's pose, tactile sensors have recently been studied due to their robustness against occlusions. This paper explores tactile data's temporal information for estimating the orientation of grasped objects. The data from a compliant tactile sensor were collected using different time-window sample sizes and evaluated using neural networks with long short-term memory (LSTM) layers. Our results suggest that using a window of sensor readings improved angle estimation compared to previous works. The best window size of 40 samples achieved an average of 0.0375 for the mean absolute error (MAE) in radians, 0.0030 for the mean squared error (MSE), 0.9074 for the coefficient of determination (R2), and 0.9094 for the explained variance score (EXP), with no enhancement for larger window sizes. This work illustrates the benefits of temporal information for pose estimation and analyzes the performance behavior with varying window sizes, which can be a basis for future robotic tactile research. Moreover, it can complement underactuated designs and visual pose estimation methods.

Curvature Tracking of a Two-Segmented Soft Finger Using an Adaptive Sliding-Mode Controller

The emergence of soft robotics in the last decade has brought about the exponential usage of elastomer bending robots in numerous fields of expertise from prosthetic hands to industrial grippers. While considerable attention has been drawn to static grasp, a substantial gap can be recognized in the development of soft grippers and anthropomorphic hands with the ability to manipulate objects dynamically, an ability that is referred to as in-hand object manipulation. In order to fill this gap, efficient and practical approaches should be investigated for the fabrication, modeling, and control of soft robots that are computationally efficient and capable of yielding repeatable results. Accordingly, in this article, qualified approaches for the dynamic modeling and control of a soft bending finger are introduced that can be efficiently implemented for planar in-hand manipulation applications in soft robotics. In this regard, a novel 3-D-printed mold is designed to cast a uniformly fabricated soft finger consisting of two independently actuated bending segments. After deriving the dynamic model of the soft finger and identifying its dynamic parameters, an adaptive sliding-mode controller is proposed to guarantee the convergence of the bending angles to desired trajectories under dynamic uncertainties and external disturbances. In the experimental validations, the root-mean-square errors of the proposed controller are 0.0323 and 0.0303 rad/m in free motion for each segment, while these numbers are 0.0328 and 0.0226 rad/m for constrained motion. For a comparative evaluation of the proposed controller, the results of the proportional–integral–differential controller are demonstrated under similar circumstances.

EMG-driven shared human-robot compliant control for in-hand object manipulation in hand prostheses

Objective. The limited functionality of hand prostheses remains one of the main reasons behind the lack of its wide adoption by amputees. Indeed, while commercial prostheses can perform a reasonable number of grasps, they are often inadequate for manipulating the object once in hand. This lack of dexterity drastically restricts the utility of prosthetic hands. We aim at investigating a novel shared control strategy that combines autonomous control of forces exerted by a robotic hand with electromyographic (EMG) decoding to perform robust in-hand object manipulation. Approach. We conduct a three-day long longitudinal study with eight healthy subjects controlling a 16-degrees-of-freedom robotic hand to insert objects in boxes of various orientations. EMG decoding from forearm muscles enables subjects to move, proportionally and simultaneously, the fingers of the robotic hand. The desired object rotation is inferred using two EMG electrodes placed on the shoulder that record the activity of muscles responsible for elevation and depression. During the object interaction phase, the autonomous controller stabilizes and rotates the object to achieve the desired pose. In this study, we compare an incremental and a proportional shoulder-decoding method in combination with two state machine interfaces offering different levels of assistance. Main results. Results indicate that robotic assistance reduces the number of failures by and, when combined with an incremental shoulder EMG decoding, leads to faster task completion time (median = 16.9 s), compared to other control conditions. Training to use the assistive device is fast. After one session of practice, all subjects managed to achieve tasks with less failures. Significance. Shared control approaches that give some authority to an autonomous controller on-board the prosthesis are an alternative to control schemes relying on EMG decoding alone. This may improve the dexterity and versatility of robotic prosthetic hands for people with trans-radial amputation. By delegating control of forces to the prosthesis’ on-board control, one speeds up reaction time and improves the precision of force control. Such a shared control mechanism may enable amputees to perform fine insertion tasks solely using their prosthetic hands. This may restore some of the functionality of the disabled arm.

In-Hand Object Localization Using a Novel High-Resolution Visuotactile Sensor

In-hand object localization and manipulation has always been a challenging task in robotic community. In this article, we address this problem by vision-based tactile sensing with high-spatial resolution. Specifically, we design a novel tactile sensor based on stereo vision, named GelStereo, which can perceive tactile point cloud with high-spatial resolution ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt;\!1$</tex-math></inline-formula> mm). A tactile-based in-hand object localization pipeline composed of saliency detection and probabilistic point-set registration algorithms of the perceived contact point cloud is presented. Furthermore, extensive qualitative and quantitative analyses of perceived tactile point cloud and in-hand localization and insertion experiments of small parts are performed on our robot platform. The experimental results verify the accuracy and robustness of the tactile point cloud sensed by the novel GelStereo tactile sensor and the proposed in-hand object localization pipeline. This novel high-resolution visuotactile sensing technology has predictable application potential in the field of dexterous robotic manipulation.

VisuoTactile 6D Pose Estimation of an In-Hand Object Using Vision and Tactile Sensor Data

Knowledge of the 6D pose of an object can benefit in-hand object manipulation. Existing 6D pose estimation methods use vision data. In-hand 6D object pose estimation is challenging because of heavy occlusion produced by the robot’s grippers, which can have an adverse effect on methods that rely on vision data only. Many robots are equipped with tactile sensors at their fingertips that could be used to complement vision data. In this letter, we present a method that uses both tactile and vision data to estimate the pose of an object grasped in a robot’s hand.The main challenges of this research include 1) lack of standard representation for tactile sensor data, 2) fusion of sensor data from heterogeneous sources—vision and tactile, and 3) a need for large training datasets. To address these challenges, first, we propose use of point clouds to represent object surfaces that are in contact with the tactile sensor. Second, we present a network architecture based on pixel-wise dense fusion to fuse vision and tactile data to estimate the 6D pose of an object. Third, we extend NVIDIA’s Deep Learning Dataset Synthesizer to produce synthetic photo-realistic vision data and the corresponding tactile point clouds for 11 objects from the YCB Object and Model Set in Unreal Engine 4. We present results of simulated experiments suggesting that using tactile data in addition to vision data improves the 6D pose estimate of an in-hand object. We also present qualitative results of experiments in which we deploy our network on real physical robots showing successful transfer of a network trained on synthetic data to a real system.

Maneuver and formation control of a pair of soft pneumatic bending fingers by a cooperative strategy for planar in-hand object manipulation

In this paper, we seek performing dynamic in-hand object manipulation with two-segmented soft fingers in 2-D plane. Thereby, after the dynamic behavior of the soft fingers was defined, we evaluate the influence of external tip force on the finger. It is shown that the tip position error, under the effect of tip force, can be represented by uncertainties in kinematic parameters of the finger. As a result, we design a sliding-adaptive controller with compensation of kinematic uncertainties to fulfill our expectation that is controlling the tip position of the finger in vertical plane while interacting with the environment. Finally, inspired by the way that human fingers manipulate an object, a cooperative control system for manipulation is proposed which incorporates the states and tip positions of the fingers in its feedback signal. Combination of this controller with the adaptive control system yields more promising results than does the PID controller. In order to evaluate performance of the proposed controller, Simulation results for manipulation of a cubic rigid object are demonstrated.

A Robust Controller for Stable 3D Pinching Using Tactile Sensing

This letter proposes a controller for stable grasping of unknown-shaped objects by two robotic fingers with tactile fingertips. The grasp is stabilised by rolling the fingertips on the contact surface and applying a desired grasping force to reach an equilibrium state. The validation is both in simulation and on a fully-actuated robot hand (the Shadow Modular Grasper) fitted with custom-built optical tactile sensors (based on the BRL TacTip). The controller requires the orientations of the contact surfaces, which are estimated by regressing a deep convolutional neural network over the tactile images. Overall, the grasp system is demonstrated to achieve stable equilibrium poses on various objects ranging in shape and softness, with the system being robust to perturbations and measurement errors. This approach also has promise to extend beyond grasping to stable in-hand object manipulation with multiple fingers.

Status Identification and Object In-Hand Reorientation Using Force/Torque Sensors

Object in-hand reorientation is widely known to be crucial for robotic dexterous manipulation, and the stable reorientation of the in-hand object with the two-fingered gripper is challenging. In this paper, the object reorientation is completed using a commercial two-fingered gripper with integrated force/torque sensors. The identification models for the object in-hand status (slipping status and pivoting status) are built based on the friction theory. In-hand status identification provides an important guarantee for the object in-hand reorientation. Moreover, we propose a control framework for the object in-hand reorientation under the action of gravity. This framework is to make the object slip or pivot by continuously controlling the gripping force. The force control law is related to the object's kinematics parameters, and the gripping force is modified according to the object in-hand status and kinematics parameters. Finally, the effectiveness of the proposed method is validated by testing object slipping and object pivoting in experiments. Results show that the gripping point and pose of the in-hand object are adjusted through object slipping and object pivoting.

A Differentiable Extended Kalman Filter for Object Tracking Under Sliding Regime.

Tactile sensing represents a valuable source of information in robotics for perception of the state of objects and their properties. Modern soft tactile sensors allow perceiving orthogonal forces and, in some cases, relative motions along the surface of the object. Detecting and measuring this kind of lateral motion is fundamental to react to possibly uncontrolled slipping and sliding of the object being manipulated. Object slip detection and prediction have been extensively studied in the robotic community leading to solutions with good accuracy and suitable for closed-loop grip stabilization. However, algorithms for object perception, such as in-hand object pose estimation and tracking algorithms, often assume no relative motion between the object and the hand and rarely consider the problem of tracking the pose of the object subjected to slipping and sliding motions. In this work, we propose a differentiable Extended Kalman filter that can be trained to track the position and the velocity of an object under translational sliding regime from tactile observations alone. Experiments with several objects, carried out on the iCub humanoid robot platform, show that the proposed approach allows achieving an average position tracking error in the order of 0.6 cm, and that the provided estimate of the object state can be used to take control decisions using tactile feedback alone. A video of the experiments is available as Supplementary Material.

Using Tactile Sensing to Improve the Sample Efficiency and Performance of Deep Deterministic Policy Gradients for Simulated In-Hand Manipulation Tasks.

Deep Reinforcement Learning techniques demonstrate advances in the domain of robotics. One of the limiting factors is a large number of interaction samples usually required for training in simulated and real-world environments. In this work, we demonstrate for a set of simulated dexterous in-hand object manipulation tasks that tactile information can substantially increase sample efficiency for training (by up to more than threefold). We also observe an improvement in performance (up to 46%) after adding tactile information. To examine the role of tactile-sensor parameters in these improvements, we included experiments with varied sensor-measurement accuracy (ground truth continuous values, noisy continuous values, Boolean values), and varied spatial resolution of the tactile sensors (927 sensors, 92 sensors, and 16 pooled sensor areas in the hand). To facilitate further studies and comparisons, we make these touch-sensor extensions available as a part of the OpenAI Gym Shadow-Dexterous-Hand robotics environments.

Robot gripper with high speed, in-hand object manipulation capabilities

This paper discusses the design and implementation of a high-speed electromechanical robotic gripper for grasping and manipulation of objects. The novelty of the developed robot end-effector lies in the ability to perform rapid and precision in-hand manipulation of various parts. Unlike existing solutions, the end-effector is capable to grasp, reorient and release objects, enabling high productivity in sophisticated industrial feeding and packaging operations. The motivation for the gripper design is inspired by the industrial needs of handling products with complex geometric characteristics, while a use case originating from the consumer goods industry was used for assessing the gripper's performance in an industrial relevant environment.© 2020 The Authors, Published by Elsevier B.V.Peer review under the responsibility of the scientific committee of CIRP