Dexterous Object Manipulation Research Articles

Grasp context-dependent uncertainty alters the relative contribution of anticipatory and feedback-based mechanisms in object manipulation

Predictive control within dexterous object manipulation while allowing for the choice of contact points has been shown to employ a predominantly feedback-based force modulation. The anticipation is thought to be facilitated through the internal representation of the object dynamics being integrated and updated on a trial-to-trial basis with the feedback of contact locations on the object. This is as opposed to the classically studied memory representation-based fingertip force control for grasping with pre-selected contact locations. We designed a study to examine this grasp context-dependent asymmetry in sensorimotor integration by introducing binary uncertainty about the grasp type before movement initiation within the framework of motor planning. An inverted T-shaped instrumented object was presented to 24 participants as the manipulandum, and they were asked to reach, grasp, and lift it while minimising the peak roll. We dissociated the planning and the execution phases by pseudo-randomly manipulating the availability of visual contact cues on the object after movement onset. We analysed both derived as well as direct kinetic and kinematic measures of the grasp during the loading phase to understand the anticipatory coordination. Our findings suggest that uncertainty about the grasp context during movement preparation resulted in a shift towards feedback-based mechanisms for grasp force modulation despite the persistence of visual cues.

VR-HandNet: A Visually and Physically Plausible Hand Manipulation System in Virtual Reality.

This study aims to allow users to perform dexterous hand manipulation of objects in virtual environments with hand-held VR controllers. To this end, the VR controller is mapped to the virtual hand and the hand motions are dynamically synthesized when the virtual hand approaches an object. At each frame, given the information about the virtual hand, VR controller input, and hand-object spatial relations, the deep neural network determines the desired joint orientations of the virtual hand model in the next frame. The desired orientations are then converted into a set of torques acting on hand joints and applied to a physics simulation to determine the hand pose at the next frame. The deep neural network, named VR-HandNet, is trained with a reinforcement learning-based approach. Therefore, it can produce physically plausible hand motion since the trial-and-error training process can learn how the interaction between hand and object is performed under the environment that is simulated by a physics engine. Furthermore, we adopted an imitation learning paradigm to increase visual plausibility by mimicking the reference motion datasets. Through the ablation studies, we validated the proposed method is effectively constructed and successfully serves our design goal. A live demo is demonstrated in the supplementary video.

Dexterous manipulation: differential sensitivity of manipulation and grasp forces to task requirements.

How humans coordinate digit forces to perform dexterous manipulation is not well understood. This gap is due to the use of tasks devoid of dexterity requirements and/or the use of analytical techniques that cannot isolate the roles that digit forces play in preventing object slip and controlling object position and orientation (pose). In our recent work, we used a dexterous manipulation task and decomposed digit forces into FG, the internal force that prevents object slip, and FM, the force responsible for object pose control. Unlike FG, FM was modulated from object lift onset to hold, suggesting their different sensitivity to sensory feedback acquired during object lift. However, the extent to which FG and FM can be controlled independently remains to be determined. Importantly, how FG and FM change as a function of object property is mathematically indeterminate and therefore requires active modulation. To address this gap, we systematically changed either object mass or external torque. The FM normal component responsible for object orientation control was modulated to changes in object torque but not mass. In contrast, FG was distinctly modulated to changes in object mass and torque. These findings point to a differential sensitivity of FG and FM to task requirements and provide novel insights into the neural control of dexterous manipulation. Importantly, our results indicate that the proposed digit force decomposition has the potential to capture important differences in how sensory inputs are processed and integrated to simultaneously ensure grasp stability and dexterous object pose control.NEW & NOTEWORTHY Successful dexterous object manipulation requires simultaneous prevention of object slip and object pose control. How these two task goals are attained can be investigated by decomposing digit forces into grasp and manipulation forces, respectively. We found that these forces were characterized by differential sensitivity to changes in object properties (mass and torque). This finding suggests the involvement of distinct sensorimotor mechanisms that, combined, simultaneously ensure grasp stability and dexterous control of object pose.

Tactile Sensing and Grasping Through Thin‐Shell Buckling

Soft and lightweight grippers have greatly enhanced the performance of robotic manipulators in handling complex objects with varying shape, texture, and stiffness. However, the combination of universal grasping with passive sensing capabilities still presents challenges. To overcome this limitation, a fluidic soft gripper is introduced based on the buckling of soft, thin hemispherical shells. Leveraging a single fluidic pressure input, the soft gripper can grasp slippery and delicate objects while passively providing information on this physical interaction. Guided by analytical, numerical, and experimental tools, the novel grasping principle of this mechanics‐based soft gripper is explored. First, the buckling behavior of a free hemisphere is characterized as a function of its geometric parameters. Inspired by the free hemisphere's two‐lobe mode shape ideal for grasping purposes, it is demonstrated that the gripper can perform dexterous manipulation and gentle gripping of fragile objects in confined spaces and underwater environments. Last, the soft gripper's embedded capability of detecting contact, grasping, and release conditions during the interaction with an unknown object is proved. This simple buckling‐based soft gripper opens new avenues for the design of adaptive gripper morphologies with tactile sensing capabilities for applications ranging from medical and agricultural robotics to space and underwater exploration.

Learning-Based Planner for Unknown Object Dexterous Manipulation Using ANFIS

Dexterous manipulation of unknown objects performed by robots equipped with mechanical hands represents a critical challenge. The difficulties arise from the absence of a precise model of the manipulated objects, unpredictable environments, and limited sensing capabilities of the mechanical hands compared to human hands. This paper introduces a data-driven approach that provides a learning-based planner for dexterous manipulation employing an Adaptive Neuro-Fuzzy Inference System (ANFIS) fed by data obtained from an analytical manipulation planner. ANFIS captures the complex relationships between inputs and optimal manipulation parameters. Moreover, during a training phase, it is able to fine-tune itself on the basis of its experiences. The proposed planner enables a robot to interact with objects of various shapes, sizes, and material properties while providing an adaptive solution for increasing robotic dexterity. The planner is validated in a real-world environment, applying an Allegro anthropomorphic robotic hand. A link to a video of the experiment is provided in the paper.

Spike timing-based coding in neuromimetic tactile system enables dynamic object classification.

Rapid processing of tactile information is essential to human haptic exploration and dexterous object manipulation. Conventional electronic skins generate frames of tactile signals upon interaction with objects. Unfortunately, they are generally ill-suited for efficient coding of temporal information and rapid feature extraction. In this work, we report a neuromorphic tactile system that uses spike timing, especially the first-spike timing, to code dynamic tactile information about touch and grasp. This strategy enables the system to seamlessly code highly dynamic information with millisecond temporal resolution on par with the biological nervous system, yielding dynamic extraction of tactile features. Upon interaction with objects, the system rapidly classifies them in the initial phase of touch and grasp, thus paving the way to fast tactile feedback desired for neuro-robotics and neuro-prosthetics.

Open-Source Instrumented Object to Study Dexterous Object Manipulation.

Humans use tactile feedback to perform skillful manipulation. When tactile sensory feedback is unavailable, for instance, if the fingers are anesthetized, dexterity is severely impaired. Imaging the deformation of the finger pad skin when in contact with a transparent plate provides information about the tactile feedback received by the central nervous system. Indeed, skin deformations are transduced into neural signals by the mechanoreceptors of the finger pad skin. Understanding how this feedback is used for active object manipulation would improve our understanding of human dexterity. In this paper, we present a new device for imaging the skin of the finger pad of one finger during manipulation performed with a precision grip. The device's mass (300 g) makes it easy to use during unconstrained dexterous manipulation. Using this device, we reproduced the experiment performed in Delhaye et al. (2021) We extracted the strains aligned with the object's movement, i.e., the vertical strains in the ulnar and radial parts of the fingerpad, to see how correlated they were with the grip force (GF) adaptation. Interestingly, parts of our results differed from those in Delhaye et al. (2021) due to weight and inertia differences between the devices, with average GF across participants differing significantly. Our results highlight a large variability in the behavior of the skin across participants, with generally low correlations between strain and GF adjustments, suggesting that skin deformations are not the primary driver of GF adaptation in this manipulation scenario.

Soft capacitive tactile sensing array with high-robustness for three-axis pressure measurement

Tactile sensing with capability to measure and differentiate synergistic normal and shear pressures real-timely is crucial for possible dexterous manipulation of objects in robotic applications. Here in this paper, the measuring principle, as well as fabrication and testing of a soft tactile sensing array for three-axis pressure measurement are proposed. It is enabled by a floating electrode structure rather than a typical electrode-dielectric-electrode sandwiched prototype that the interconnects and sensing electrodes are realized using flexible printed circuit (FPC) meanwhile the floating electrodes are easily fabricated on a soft polyethylene terephthalate (PET) film. The transformation of embedding fragile interconnects into soft PDMS sensor body under excessive bending and deformation to a FPC layer of increased robustness and less compression drastically reduce the possibility of loss of function, and guarantee the robustness of the designed tactile sensing array. Moreover, hierarchically porous PDMS composite as the dielectric inlaid between floating electrode layer and sensing electrode layer results in a soft tactile sensing array with enhanced sensitivity (0.04 kPa−1 under normal pressure and 0.10 kPa−1 under shear pressures), large pressure sensing range (>300 kPa under normal pressure and > 62 kPa under shear pressures), minimal hysteresis, excellent repeatability, and fast response time (less than 60 ms) attributing to the synergistic effect of uniformly distributed macro and micro pores. The constructed soft tactile sensing array would provide essential sensing feedback for controlling an artificial arm in various tasks, illustrating its great potential application in robotics.

Visual information following object grasp supports digit position variability and swift anticipatory force control.

Anticipatory force control underlying dexterous manipulation has historically been understood to rely on visual object properties and on sensorimotor memories associated with previous experiences with similar objects. However, it is becoming increasingly recognized that anticipatory force control also relies on how an object is grasped. Experiments that allow unconstrained grasp contact points when preventing tilting an object with an off-centered mass show trial-to-trial variations in digit position and subsequent scaling of lift forces, all before feedback of object properties becomes available. Here, we manipulated the availability of visual information before reach onset and after grasp contact (with no vision during the reach) to determine the contribution and timing of visual information processing to the scaling of fingertip forces during dexterous manipulation at flexible contact points. Results showed that anticipatory force control was similarly successful, quantified as an appropriate compensatory torque at lift onset that counters the external torque of an object with a left and right center of mass, irrespective of the timing and availability of visual information. However, the way in which anticipatory force control was achieved varied depending on the availability of visual information. Visual information following grasp contact was associated with greater use of an asymmetric thumb and index finger grasp configuration to generate a compensatory torque and digit position variability, together with faster fingertip force scaling and sensorimotor learning. This result supports the hypothesis that visual information at a critical and functionally relevant time point following grasp contact supports variable and swift digit-based force control for dexterous object manipulation.NEW & NOTEWORTHY Humans excel in dexterous object manipulation by precisely coordinating grasp points and fingertip forces, highlighted in scenarios requiring countering object torques in advance, e.g., lifting a teacup without spilling will demand a unique digit force pattern based on the grip configuration at lift onset. Here, we show that visual information following grasp contact, a critical and functionally relevant time point, supports digit position variability and swift anticipatory force control to achieve a dexterous motor goal.

How Tactile Afferents in the Human Fingerpad Encode Tangential Torques Associated with Manipulation: Are Monkeys Better than Us?

Dexterous object manipulation depends critically on information about forces normal and tangential to the fingerpads, and also on torque associated with object orientation at grip surfaces. We investigated how torque information is encoded by human tactile afferents in the fingerpads and compared them to 97 afferents recorded in monkeys (n = 3; 2 females) in our previous study. Human data included slowly-adapting Type-II (SA-II) afferents, which are absent in the glabrous skin of monkeys. Torques of different magnitudes (3.5-7.5 mNm) were applied in clockwise and anticlockwise directions to a standard central site on the fingerpads of 34 human subjects (19 females). Torques were superimposed on a 2, 3, or 4 N background normal force. Unitary recordings were made from fast-adapting Type-I (FA-I, n = 39), and slowly-adapting Type-I (SA-I, n = 31) and Type-II (SA-II, n = 13) afferents supplying the fingerpads via microelectrodes inserted into the median nerve. All three afferent types encoded torque magnitude and direction, with torque sensitivity being higher with smaller normal forces. SA-I afferent responses to static torque were inferior to dynamic stimuli in humans, while in monkeys the opposite was true. In humans this might be compensated by the addition of sustained SA-II afferent input, and their capacity to increase or decrease firing rates with direction of rotation. We conclude that the discrimination capacity of individual afferents of each type was inferior in humans than monkeys which could be because of differences in fingertip tissue compliance and skin friction.SIGNIFICANCE STATEMENT We investigated how individual human tactile nerve fibers encode rotational forces (torques) and compared them to their monkey counterparts. Human hands, but not monkey hands, are innervated by a tactile neuron type (SA-II afferents) specialized to encode directional skin strain yet, so far, torque encoding has only been studied in monkeys. We find that human SA-I afferents were generally less sensitive and less able to discriminate torque magnitude and direction than their monkey counterparts, especially during the static phase of torque loading. However, this shortfall in humans could be compensated by SA-II afferent input. This indicates that variation in afferent types might complement each other signaling different stimulus features possibly providing computational advantage to discriminate stimuli.

DotView: A Low-Cost Compact Tactile Sensor for Pressure, Shear, and Torsion Estimation

Tactile sensation is one of the most important sensing methods of humans, which helps humans complete complex decisions and actions by itself or in cooperation with other sensations. Similarly, tactile sensors are also crucial for robots to interact with the environment and fulfill dexterous object manipulation. In this study, we present DotView, a compact, low-cost, and high-performance tactile sensor that utilizes a biomimetic micro-structured soft layer to generate rich, high-resolution tactile images captured by a commercialized capacitive sensing array, which exhibits sensitive and robust responses under pressure, shear and torsion applied by different indenters. Furthermore, we utilize a neural network to estimate multiaxial contact forces and torques, and achieve good accuracy in different contact geometries. Finally, we demonstrate the capability of DotView in a dexterous manipulation scenario by performing the task of guiding a tool through a maze with only tactile feedback.

Laser-Actuated Multi-Fingered Hand for Dexterous Manipulation of Micro-Objects

Dexterous robotic manipulation refers to the coordination of multiple fingers or manipulators to grasp and manipulate a target object. In past decades, various achievements have been established in dexterous robotic manipulation in the physical world. However, such dexterous ability is rarely found in nano/micro manipulation, which is mainly due to the difficulty in developing actuators and sensors at a nano/micro-scale, as well as the dependence on the physical properties of objects. As a result, only limited and relatively simple micro-manipulation tasks such as pushing and picking have been realized so far. Here, we demonstrate a platform to achieve dexterous manipulation in the micro-world by developing a micro multi-fingered hand that is actuated by optical traps. The system consists of several micro laser-actuated fingers which are coordinated to function as a robotic micro-hand. Each micro-finger is configured with three degrees of freedom in a 2-dimensional space, and a system composed of coordinated micro-fingers can then be utilized for grasping, rotating, moving, and even levering and rolling of objects in micron scale. Thus, the proposed approach in this paper establishes a foundation in achieving dexterous robotic manipulation of objects at micron scale through the coordination of multiple micro fingers.

Dexterous Object Manipulation with an Anthropomorphic Robot Hand via Natural Hand Pose Transformer and Deep Reinforcement Learning

Dexterous object manipulation using anthropomorphic robot hands is of great interest for natural object manipulations across the areas of healthcare, smart homes, and smart factories. Deep reinforcement learning (DRL) is a particularly promising approach to solving dexterous manipulation tasks with five-fingered robot hands. Yet, controlling an anthropomorphic robot hand via DRL in order to obtain natural, human-like object manipulation with high dexterity remains a challenging task in the current robotic field. Previous studies have utilized some predefined human hand poses to control the robot hand’s movements for successful object-grasping. However, the hand poses derived from these grasping taxonomies are limited to a partial range of adaptability that could be performed by the robot hand. In this work, we propose a combinatory approach of a deep transformer network which produces a wider range of natural hand poses to configure the robot hand’s movements, and an adaptive DRL to control the movements of an anthropomorphic robot hand according to these natural hand poses. The transformer network learns and infers the natural robot hand poses according to the object affordance. Then, DRL trains a policy using the transformer output to grasp and relocate the object to the designated target location. Our proposed transformer-based DRL (T-DRL) has been tested using various objects, such as an apple, a banana, a light bulb, a camera, a hammer, and a bottle. Additionally, its performance is compared with a baseline DRL model via natural policy gradient (NPG). The results demonstrate that our T-DRL achieved an average manipulation success rate of 90.1% for object manipulation and outperformed NPG by 24.8%.

A self-adjusting and modular supervisory control algorithm for planar dexterous manipulation

Many applications require precise handling and manipulation of delicate objects. In some cases, the object must be transported to a new location following a strict travel path including time-related constraints. This paper presents a self-adjusting modular control algorithm for dexterous manipulation of planar objects using multiple manipulators with precise path and timing deliveries. The popular caging approach is simple, and usually effective when manipulating objects with multiple devices but can fail following complex paths with orientation adjustments under time-critical tracking requirements. The proposed approach exploits the dynamics of the object in real-time using tracking control and allocates the force that needs to be applied by each manipulator based on their current position around the object to maximize their capability to push in the direction of the contact angle. The new algorithm is self-adjusting and modular; It can adjust its force allocation according to configuration changes during operation, and manipulators execute the same algorithm regardless of their number. The advantages of the new approach are successfully demonstrated both with simulations and testbed experiments, including orientation tracking, which is not typically featured with the caging approach. Conditions to check when the new algorithm is most effective are also analyzed. The closed-loop stability and performance of the new algorithm are also studied and necessary conditions are identified.

Attracting Conductive Nonmagnetic Objects With Rotating Magnetic Dipole Fields

Recent research has shown that eddy currents induced by rotating magnetic dipole fields can produce forces and torques useful for dexterous manipulation of conductive nonmagnetic objects. This control paradigm shows promise for application in the remediation of space debris. However, the resulting force from each field source always includes a repulsive component, suggesting that the object should be surrounded by field sources to some degree to ensure the object does not leave the dexterous workspace during manipulation. In this letter, we describe the ability of multiple field sources working together in a coordinated fashion to attract conductive nonmagnetic objects that are not surrounded, reminiscent of a tractor beam. This enables an object to be pulled into a dexterous manipulation workspace. We explicitly describe the nominal attractive force on spherical objects, serving as an approximation for other object geometries, as a function of the properties of the object and the field-generation system. We perform experimental verification using a water-based microgravity simulator.

Learning of Dexterous Object Manipulation in a Virtual Reality Environment.

Humans have unrivalled abilities to perform dexterous object manipulation. This requires the sensorimotor system to quickly adapt to environmental changes and predictively counter act the external disturbances. Many studies have focused on the anticipatory control of digits with real-world experiments. However, examining manipulation using virtual reality with haptic devices expands the possibilities of investigation. In this work, participants grasped and lifted an inverted T-shaped object in a virtual reality setup. The graspable surface of the object was either constrained to a small area or unconstrained. The position of the object's center of mass changed between blocks, and the participants were asked to minimize the rotation of the object during the lift. Our results show that, consistent with the results of real-world experiments, participants gradually learn to adjust the digit positions and forces to predictively compensate for the torque due to the shifted center of mass prior to liftoff. The only major difference found was that the length of trials needed during the adaptation phase to each condition increased from 3 in real-world to 5 in virtual environment.

Full Soft Capacitive Omnidirectional Tactile Sensor Based on Micro-Spines Electrode and Hemispheric Dielectric Structure.

Flourishing in recent years, intelligent electronics is desirably pursued in many fields including bio-symbiotic, human physiology regulatory, robot operation, and human–computer interaction. To support this appealing vision, human-like tactile perception is urgently necessary for dexterous object manipulation. In particular, the real-time force perception with strength and orientation simultaneously is critical for intelligent electronic skin. However, it is still very challenging to achieve directional tactile sensing that has eminent properties, and at the same time, has the feasibility for scale expansion. Here, a fully soft capacitive omnidirectional tactile (ODT) sensor was developed based on the structure of MWCNTs coated stripe electrode and Ecoflex hemisphere array dielectric. The theoretical analysis of this structure was conducted for omnidirectional force detection by finite element simulation. Combined with the micro-spine and the hemispheric hills dielectric structure, this sensing structure could achieve omnidirectional detection with high sensitivity (0.306 ± 0.001 kPa−1 under 10 kPa) and a wide response range (2.55 Pa to 160 kPa). Moreover, to overcome the inherent disunity in flexible sensor units due to nano-materials and polymer, machine learning approaches were introduced as a prospective technical routing to recognize various loading angles and finally performed more than 99% recognition accuracy. The practical validity of the design was demonstrated by the detection of human motion, physiological activities, and gripping of a cup, which was evident to have great potential for tactile e-skin for digital medical and soft robotics.

Mapping the Integration of Sensory Information across Fingers in Human Sensorimotor Cortex.

The integration of somatosensory signals across fingers is essential for dexterous object manipulation. Previous experiments suggest that this integration occurs in neural populations in the primary somatosensory cortex (S1). However, the integration process has not been fully characterized, as previous studies have mainly used 2-finger stimulation paradigms. Here, we addressed this gap by stimulating all 31 single- and multifinger combinations. We measured population-wide activity patterns evoked during finger stimulation in human S1 and primary motor cortex (M1) using 7T fMRI in female and male participants. Using multivariate fMRI analyses, we found clear evidence of unique nonlinear interactions between fingers. In Brodmann area (BA) 3b, interactions predominantly occurred between pairs of neighboring fingers. In BA 2, however, we found equally strong interactions between spatially distant fingers, as well as interactions between finger triplets and quadruplets. We additionally observed strong interactions in the hand area of M1. In both M1 and S1, these nonlinear interactions did not reflect a general suppression of overall activity, suggesting instead that the interactions we observed reflect rich, nonlinear integration of sensory inputs from the fingers. We suggest that this nonlinear finger integration allows for a highly flexible mapping from finger sensory inputs to motor responses that facilitates dexterous object manipulation.SIGNIFICANCE STATEMENT Processing of somatosensory information in primary somatosensory cortex (S1) is essential for dexterous object manipulation. To successfully handle an object, the sensorimotor system needs to detect complex patterns of haptic information, which requires the nonlinear integration of sensory inputs across multiple fingers. Using multivariate fMRI analyses, we characterized brain activity patterns evoked by stimulating all single- and multifinger combinations. We report that progressively stronger multifinger interactions emerge in posterior S1 and in the primary motor cortex (M1), with interactions arising between inputs from neighboring and spatially distant fingers. Our results suggest that S1 and M1 provide the neural substrate necessary to support a flexible mapping from sensory inputs to motor responses of the hand.

Design of a 3D-Printed Soft Robotic Hand With Integrated Distributed Tactile Sensing

Humans rely on distributed tactile sensing in their hands to achieve robust and dexterous manipulation of delicate objects. Soft robotic hands have received increased attention in recent years due to their adaptability to unknown objects and safe interactions with the environment. However, the integration of distributed sensing in soft robotic hands is lacking. This is largely due to the complexity in the integration of soft sensing solutions with the hands. This letter proposes a novel soft robotic hand that incorporates an active palm and distributed pneumatic tactile sensing in both the fingers and the palm. Multi-material 3D printing allows the tactile sensors to be directly printed on the hand, whereas conventional tactile approaches require the sensors to be attached as part of multiple fabrication procedures. Active degrees of freedom are introduced in the palm to achieve increased dexterity. The proposed hand successfully performed 32 of the 33 Feix taxonomy grasps and all 11 Kapandji thumb opposition poses.

Grip Force is Adjusted at a Level That Maintains an Upper Bound on Partial Slip Across Friction Conditions During Object Manipulation.

Dexterous manipulation of objects heavily relies on the feedback provided by the tactile afferents innervating the fingertips. Previous studies have suggested that humans might take advantage of partial slip, localized loss of grip between the skin and the object, to gauge the stability of a contact and react appropriately when it is compromised, that is, when slippage is about to happen. To test this hypothesis, we asked participants to perform point-to-point movements using a manipulandum. Through optical imaging, the device monitored partial slip at the contact interface, and at the same time, the forces exerted by the fingers. The level of friction of the contact material was changed every five trials. We found that the level of grip force was systematically adjusted to the level of friction, and thus partial slip was limited to an amount similar across friction conditions. We suggest that partial slip is a key signal for dexterous manipulation and that the grip force is regulated to continuously maintain an upper bound on partial slip across friction conditions.