In-hand Object Manipulation Research Articles

Survey of learning-based approaches for robotic in-hand manipulation

Human dexterity is an invaluable capability for precise manipulation of objects in complex tasks. The capability of robots to similarly grasp and perform in-hand manipulation of objects is critical for their use in the ever changing human environment, and for their ability to replace manpower. In recent decades, significant effort has been put in order to enable in-hand manipulation capabilities to robotic systems. Initial robotic manipulators followed carefully programmed paths, while later attempts provided a solution based on analytical modeling of motion and contact. However, these have failed to provide practical solutions due to inability to cope with complex environments and uncertainties. Therefore, the effort has shifted to learning-based approaches where data is collected from the real world or through a simulation, during repeated attempts to complete various tasks. The vast majority of learning approaches focused on learning data-based models that describe the system to some extent or Reinforcement Learning (RL). RL, in particular, has seen growing interest due to the remarkable ability to generate solutions to problems with minimal human guidance. In this survey paper, we track the developments of learning approaches for in-hand manipulations and, explore the challenges and opportunities. This survey is designed both as an introduction for novices in the field with a glossary of terms as well as a guide of novel advances for advanced practitioners.

Validations of various in-hand object manipulation strategies employing a novel tactile sensor developed for an under-actuated robot hand.

Prisma Hand II is an under-actuated prosthetic hand developed at the University of Naples, Federico II to study in-hand manipulations during grasping activities. 3 motors equipped on the robotic hand drive 19 joints using elastic tendons. The operations of the hand are achieved by combining tactile hand sensing with under-actuation capabilities. The hand has the potential to be employed in both industrial and prosthetic applications due to its dexterous motion capabilities. However, currently there are no commercially available tactile sensors with compatible dimensions suitable for the prosthetic hand. Hence, in this work, we develop a novel tactile sensor designed based on an opto-electronic technology for the Prisma Hand II. The optimised dimensions of the proposed sensor made it possible to be integrated with the fingertips of the prosthetic hand. The output voltage obtained from the novel tactile sensor is used to determine optimum grasping forces and torques during in-hand manipulation tasks employing Neural Networks (NNs). The grasping force values obtained using a Convolutional Neural Network (CNN) and an Artificial Neural Network (ANN) are compared based on Mean Square Error (MSE) values to find out a better training network for the tasks. The tactile sensing capabilities of the proposed novel sensing method are presented and compared in simulation studies and experimental validations using various hand manipulation tasks. The developed tactile sensor is found to be showcasing a better performance compared to previous version of the sensor used in the hand.

AngoraPy: A Python toolkit for modeling anthropomorphic goal-driven sensorimotor systems.

Goal-driven deep learning increasingly supplements classical modeling approaches in computational neuroscience. The strength of deep neural networks as models of the brain lies in their ability to autonomously learn the connectivity required to solve complex and ecologically valid tasks, obviating the need for hand-engineered or hypothesis-driven connectivity patterns. Consequently, goal-driven models can generate hypotheses about the neurocomputations underlying cortical processing that are grounded in macro- and mesoscopic anatomical properties of the network's biological counterpart. Whereas, goal-driven modeling is already becoming prevalent in the neuroscience of perception, its application to the sensorimotor domain is currently hampered by the complexity of the methods required to train models comprising the closed sensation-action loop. This paper describes AngoraPy, a Python library that mitigates this obstacle by providing researchers with the tools necessary to train complex recurrent convolutional neural networks that model the human sensorimotor system. To make the technical details of this toolkit more approachable, an illustrative example that trains a recurrent toy model on in-hand object manipulation accompanies the theoretical remarks. An extensive benchmark on various classical, 3D robotic, and anthropomorphic control tasks demonstrates AngoraPy's general applicability to a wide range of tasks. Together with its ability to adaptively handle custom architectures, the flexibility of this toolkit demonstrates its power for goal-driven sensorimotor modeling.

Intrinsically Motivated Multi-Goal Reinforcement Learning Using Robotics Environment Integrated with OpenAI Gym

Sparse reward is one of the most challenging problems in reinforcement learning (RL). Hindsight Experience Replay (HER) attempts to address this issue by converting a failed experience to a successful one by relabelling the goals. In open-ended and changing environments, agents face a wide range of potential tasks that might not come with associated reward functions. Such autonomous learning agents must set their own tasks and build their own curriculum through an intrinsically motivated exploration. Because some tasks might prove easy and some impossible, agents must actively select which task to practice at any given moment, to maximize their overall mastery on the set of learnable tasks. The purpose of this technical report is two-fold. First, it introduces a suite of challenging continuous control tasks (integrated with OpenAI Gym) based on currently existing robotics hardware. The tasks include pushing, sliding and pick & place with a Fetch robotic arm as well as in-hand object manipulation with a Shadow Dexterous Hand. All tasks have sparse binary rewards and follow a Multi-Goal Reinforcement Learning (RL) framework in which an agent is told what to do using an additional input. The second part of the paper presents a set of concrete research ideas for improving RL algorithms, most of which are related to Multi-Goal RL and Hindsight Experience Replay. The Fetch environments are based on the 7-DoF Fetch robotics arm,2 which has a two-fingered parallel gripper. Agents focus on achievable tasks first and focus back on tasks that are being forgotten. Experiments conducted in a new multi-task multi-goal robotic environment show that our algorithm benefits from these two ideas and demonstrate properties of robustness to distracting tasks, forgetting and changes in body properties

Curvature Tracking of a Two-Segmented Soft Finger Using an Adaptive Sliding-Mode Controller

The emergence of soft robotics in the last decade has brought about the exponential usage of elastomer bending robots in numerous fields of expertise from prosthetic hands to industrial grippers. While considerable attention has been drawn to static grasp, a substantial gap can be recognized in the development of soft grippers and anthropomorphic hands with the ability to manipulate objects dynamically, an ability that is referred to as in-hand object manipulation. In order to fill this gap, efficient and practical approaches should be investigated for the fabrication, modeling, and control of soft robots that are computationally efficient and capable of yielding repeatable results. Accordingly, in this article, qualified approaches for the dynamic modeling and control of a soft bending finger are introduced that can be efficiently implemented for planar in-hand manipulation applications in soft robotics. In this regard, a novel 3-D-printed mold is designed to cast a uniformly fabricated soft finger consisting of two independently actuated bending segments. After deriving the dynamic model of the soft finger and identifying its dynamic parameters, an adaptive sliding-mode controller is proposed to guarantee the convergence of the bending angles to desired trajectories under dynamic uncertainties and external disturbances. In the experimental validations, the root-mean-square errors of the proposed controller are 0.0323 and 0.0303 rad/m in free motion for each segment, while these numbers are 0.0328 and 0.0226 rad/m for constrained motion. For a comparative evaluation of the proposed controller, the results of the proportional–integral–differential controller are demonstrated under similar circumstances.

EMG-driven shared human-robot compliant control for in-hand object manipulation in hand prostheses

Objective. The limited functionality of hand prostheses remains one of the main reasons behind the lack of its wide adoption by amputees. Indeed, while commercial prostheses can perform a reasonable number of grasps, they are often inadequate for manipulating the object once in hand. This lack of dexterity drastically restricts the utility of prosthetic hands. We aim at investigating a novel shared control strategy that combines autonomous control of forces exerted by a robotic hand with electromyographic (EMG) decoding to perform robust in-hand object manipulation. Approach. We conduct a three-day long longitudinal study with eight healthy subjects controlling a 16-degrees-of-freedom robotic hand to insert objects in boxes of various orientations. EMG decoding from forearm muscles enables subjects to move, proportionally and simultaneously, the fingers of the robotic hand. The desired object rotation is inferred using two EMG electrodes placed on the shoulder that record the activity of muscles responsible for elevation and depression. During the object interaction phase, the autonomous controller stabilizes and rotates the object to achieve the desired pose. In this study, we compare an incremental and a proportional shoulder-decoding method in combination with two state machine interfaces offering different levels of assistance. Main results. Results indicate that robotic assistance reduces the number of failures by and, when combined with an incremental shoulder EMG decoding, leads to faster task completion time (median = 16.9 s), compared to other control conditions. Training to use the assistive device is fast. After one session of practice, all subjects managed to achieve tasks with less failures. Significance. Shared control approaches that give some authority to an autonomous controller on-board the prosthesis are an alternative to control schemes relying on EMG decoding alone. This may improve the dexterity and versatility of robotic prosthetic hands for people with trans-radial amputation. By delegating control of forces to the prosthesis’ on-board control, one speeds up reaction time and improves the precision of force control. Such a shared control mechanism may enable amputees to perform fine insertion tasks solely using their prosthetic hands. This may restore some of the functionality of the disabled arm.

Tactile and mechanical investigation of screen printed specimens with puff effect

The subject of this paper was the investigation of the tactile and mechanical characteristics of printed specimens achieved using the manual technique of screen printing. The specimens are printed using ink enriched with a puff base. The puff base gives the print a three-dimensional shape and surface characteristics. This paper aims to investigate whether such prints can be used to improve the ergonomic characteristics of a product that undergoes in-hand manipulation. To determine the possibility of using a screen printing technique with a puff effect for ergonomic purposes, two experiments were performed. The first experiment involved subjective investigation of the tactile properties of the prints which are important since the end-users are people. The second experiment involved laboratory testing of the resistance of prints to mechanical rubbing (colour rendering) which is important since the prints should be able to endure a lot of in-hand manipulation. The specimens were printed using the manual screen printing technique on four different textile substrates. Apart from the substrate, the amount of added puff substance in ink and the screen printing mesh count varied. After testing the mechanical resistance to rubbing, colour differences were calculated. Based on the results obtained, resistance to mechanical effect was confirmed, and it was determined which prints have the best resistance and tactile features. Further investigations will be focused on investigating the same type of printing on different materials, and discovering how can prints with puff effect contribute to in-hand object manipulation.

VisuoTactile 6D Pose Estimation of an In-Hand Object Using Vision and Tactile Sensor Data

Knowledge of the 6D pose of an object can benefit in-hand object manipulation. Existing 6D pose estimation methods use vision data. In-hand 6D object pose estimation is challenging because of heavy occlusion produced by the robot’s grippers, which can have an adverse effect on methods that rely on vision data only. Many robots are equipped with tactile sensors at their fingertips that could be used to complement vision data. In this letter, we present a method that uses both tactile and vision data to estimate the pose of an object grasped in a robot’s hand.The main challenges of this research include 1) lack of standard representation for tactile sensor data, 2) fusion of sensor data from heterogeneous sources—vision and tactile, and 3) a need for large training datasets. To address these challenges, first, we propose use of point clouds to represent object surfaces that are in contact with the tactile sensor. Second, we present a network architecture based on pixel-wise dense fusion to fuse vision and tactile data to estimate the 6D pose of an object. Third, we extend NVIDIA’s Deep Learning Dataset Synthesizer to produce synthetic photo-realistic vision data and the corresponding tactile point clouds for 11 objects from the YCB Object and Model Set in Unreal Engine 4. We present results of simulated experiments suggesting that using tactile data in addition to vision data improves the 6D pose estimate of an in-hand object. We also present qualitative results of experiments in which we deploy our network on real physical robots showing successful transfer of a network trained on synthetic data to a real system.

Maneuver and formation control of a pair of soft pneumatic bending fingers by a cooperative strategy for planar in-hand object manipulation

In this paper, we seek performing dynamic in-hand object manipulation with two-segmented soft fingers in 2-D plane. Thereby, after the dynamic behavior of the soft fingers was defined, we evaluate the influence of external tip force on the finger. It is shown that the tip position error, under the effect of tip force, can be represented by uncertainties in kinematic parameters of the finger. As a result, we design a sliding-adaptive controller with compensation of kinematic uncertainties to fulfill our expectation that is controlling the tip position of the finger in vertical plane while interacting with the environment. Finally, inspired by the way that human fingers manipulate an object, a cooperative control system for manipulation is proposed which incorporates the states and tip positions of the fingers in its feedback signal. Combination of this controller with the adaptive control system yields more promising results than does the PID controller. In order to evaluate performance of the proposed controller, Simulation results for manipulation of a cubic rigid object are demonstrated.

A Robust Controller for Stable 3D Pinching Using Tactile Sensing

This letter proposes a controller for stable grasping of unknown-shaped objects by two robotic fingers with tactile fingertips. The grasp is stabilised by rolling the fingertips on the contact surface and applying a desired grasping force to reach an equilibrium state. The validation is both in simulation and on a fully-actuated robot hand (the Shadow Modular Grasper) fitted with custom-built optical tactile sensors (based on the BRL TacTip). The controller requires the orientations of the contact surfaces, which are estimated by regressing a deep convolutional neural network over the tactile images. Overall, the grasp system is demonstrated to achieve stable equilibrium poses on various objects ranging in shape and softness, with the system being robust to perturbations and measurement errors. This approach also has promise to extend beyond grasping to stable in-hand object manipulation with multiple fingers.

Algorithmic Selection of Sliding–Sticking Contacts in Robotic In-hand Manipulation

Abstract The paper describes a kinematic method for robotic in-hand manipulation of objects. The method focuses on repositioning the object using a combination of sticking and sliding robotic contacts. Two fingertips with sliding contacts are fixed in space while the remaining two fingertips actively manipulate the object without a change in the point of contact with the object. When sliding over two fixed contacts, the object is constrained to a “three-parameter twist space” if it is not programmed to rotate about the line joining the two fixed contacts. A gradient-descent-based kinematic algorithm is developed to project the desired twist to the allowable twist space, generating a movement sequence of robotic fingertips. The transition from fixed support vis-á-vis the sticking contacts for manipulating the object also emerges from the algorithm.

Using Tactile Sensing to Improve the Sample Efficiency and Performance of Deep Deterministic Policy Gradients for Simulated In-Hand Manipulation Tasks.

Deep Reinforcement Learning techniques demonstrate advances in the domain of robotics. One of the limiting factors is a large number of interaction samples usually required for training in simulated and real-world environments. In this work, we demonstrate for a set of simulated dexterous in-hand object manipulation tasks that tactile information can substantially increase sample efficiency for training (by up to more than threefold). We also observe an improvement in performance (up to 46%) after adding tactile information. To examine the role of tactile-sensor parameters in these improvements, we included experiments with varied sensor-measurement accuracy (ground truth continuous values, noisy continuous values, Boolean values), and varied spatial resolution of the tactile sensors (927 sensors, 92 sensors, and 16 pooled sensor areas in the hand). To facilitate further studies and comparisons, we make these touch-sensor extensions available as a part of the OpenAI Gym Shadow-Dexterous-Hand robotics environments.

Electromyography-Based Decoding of Dexterous, In-Hand Manipulation of Objects: Comparing Task Execution in Real World and Virtual Reality

The increased use of Virtual and Augmented Reality based systems necessitates the development of more intuitive and unobtrusive means of interfacing. Over the last years, Electromyography (EMG) based interfaces have been employed for interaction with robotic and computer applications, but no studies have been carried out to investigate the continuous decoding of the effects of human motion (e.g., manipulated object behavior) in simulated and virtual environments. In this work, we compare the object motion decoding accuracy of an EMG based learning framework for two different dexterous manipulation scenarios: i) for simulated objects handled by a teleoperated model of a hand within a virtual environment and ii) for real, everyday life objects manipulated by the human hand. To do that, we utilize EMG activations from 16 muscle sites (9 on the hand and 7 on the forearm). The object motion decoding is formulated as a regression problem using the Random Forests methodology. A 5-fold cross validation procedure is used for model assessment purposes and the feature variable importance values are calculated for each model. The decoding accuracy for the real world is considerably higher than the virtual world. Each of the objects examined had a single manipulation motion that offered the highest estimation accuracy across both worlds. This study also shows that it is feasible to decode the object motions using just the myoelectric activations of the muscles of the forearm and the hand. This is particularly surprising since simulations lacked haptic feedback and the ability to account for other dynamic phenomena like friction and contact rolling.

Robot gripper with high speed, in-hand object manipulation capabilities

This paper discusses the design and implementation of a high-speed electromechanical robotic gripper for grasping and manipulation of objects. The novelty of the developed robot end-effector lies in the ability to perform rapid and precision in-hand manipulation of various parts. Unlike existing solutions, the end-effector is capable to grasp, reorient and release objects, enabling high productivity in sophisticated industrial feeding and packaging operations. The motivation for the gripper design is inspired by the industrial needs of handling products with complex geometric characteristics, while a use case originating from the consumer goods industry was used for assessing the gripper's performance in an industrial relevant environment.© 2020 The Authors, Published by Elsevier B.V.Peer review under the responsibility of the scientific committee of CIRP

Object-Level Impedance Control for Dexterous In-Hand Manipulation

This work presents a novel object-level control framework for the dexterous in-hand manipulation of objects with torque-controlled robotic hands. The proposed impedance-based controller realizes the compliant 6-DOF positioning of a grasped object. Enabled by the in-hand localization, the slippage of contacts is avoided by actively maintaining the desired grasp configuration. The internal forces on the object are determined by solving a quadratic optimization problem, which explicitly considers friction constraints on the contacts and allows to gradually shift the load between fingers. The proposed framework is capable of dealing with dynamic changes in the grasp configuration, which makes it applicable for the control of the object during grasp acquisition or the reconfiguration of fingers (i.e. finger gaiting). A nullspace controller avoids joint limits and singularities. Experiments with the DLR robot David demonstrate the efficiency and performance of the controller in various grasping scenarios.

The unexpected importance of the fifth digit during stone tool production

Unique anatomical features of the human hand facilitate our ability to proficiently and forcefully perform precision grips and in-hand manipulation of objects. Extensive research has been conducted into the role of digits one to three during these manual behaviours, and the origin of the highly derived first digit anatomy that facilitates these capabilities. Stone tool production has long been thought a key influence in this regard. Despite previous research stressing the unique derived morphology of the human fifth digit little work has investigated why humans alone display these features. Here we examine the recruitment frequency, loading magnitude, and loading distribution of all digits on the non-dominant hand of skilled flintknappers during four technologically distinct types of Lower Palaeolithic stone tool production. Our data reveal the fifth digit to be heavily and frequently recruited during all studied behaviours. It occasionally incurred pressures, and was used in frequencies, greater or equal to those of the thumb, and frequently the same or greater than those of the index finger. The fifth digit therefore appears key to >2 million years of stone tool production activities, a behaviour that likely contributed to the derived anatomy observed in the modern human fifth ray.

Morphology Specific Stepwise Learning of In-Hand Manipulation With a Four-Fingered Hand

In past research, in-hand object manipulation for various sized and shaped objects has been achieved. However, the network had to be trained for each different motion. Training data takes time to acquire and increases the hardware load, thereby increasing the cost for training data. Four-fingered in-hand manipulation is especially difficult as a high number of joints need to be controlled in synchrony. This paper presents a method that reduces the required training data for in-hand manipulation with the idea of pretraining and mutual finger motions. The Allegro Hand is used with soft fingertips and integrated 6-axis F/T sensors to evaluate the proposed method. To make the network more versatile, the training data included objects of various sizes and shapes. When pretraining the network, one shot learning suffices to learn a new task; mutual finger motions can be exploited to use three-fingered pretraining data for four-fingered manipulation. Both data-sharing and weight-sharing were used and show similar results. Crucially, pretraining data from fingers with the same kinematic chain has to be used, showing the importance of morphology specific learning. Moreover, objects with untrained sizes and shapes could be manipulated.

Usability of Multiviewpoint Images for Spatial Interaction in Projection-Based Display Systems

In a common application scenario, large screen projection-based stereoscopic display systems are not used by a single user alone, but are shared by a small group of people. Using multiviewpoint images for multiuser interaction does not require special hardware and scales transparently with the number of colocated users in a system. We present a qualitative and quantitative study comparing usability and interaction performance for multiviewpoint images to non-head-tracked and head-tracked interaction for ray-casting selection and in-hand object manipulation. Results show that while direct first-person interaction in projection-based displays without head-tracking is difficult or even completely impractical, interaction with multiviewpoint images can produce similar or even better performance than fully head-tracked interaction. For ray-casting selection, interaction with multiviewpoint images is actually up to 10 percent faster than head-tracked interaction. For in-hand object manipulation in a simple docking task, multiviewpoint interaction performs only about 6 percent slower than fully head-tracked interaction.