Graspable Objects Research Articles

T-TD3: A Reinforcement Learning Framework for Stable Grasping of Deformable Objects Using Tactile Prior

T-TD3: A Reinforcement Learning Framework for Stable Grasping of Deformable Objects Using Tactile Prior

NFL: Normal Field Learning for 6-DoF Grasping of Transparent Objects

NFL: Normal Field Learning for 6-DoF Grasping of Transparent Objects

Visual sensitivity at the service of action control in posterior parietal cortex.

The posterior parietal cortex (PPC) serves as a crucial hub for the integration of sensory with motor cues related to voluntary actions. Visual input is used in different ways along the dorsomedial and the dorsolateral visual pathways. Here we focus on the dorsomedial pathway and recognize a visual representation at the service of action control. Employing different experimental paradigms applied to behaving monkeys while single neural activity is recorded from the medial PPC (area V6A), we show how plastic visual representation can be, matching the different contexts in which the same object is proposed. We also present data on the exchange between vision and arm actions and highlight how this rich interplay can be used to weight different sensory inputs in order to monitor and correct arm actions online. Indeed, neural activity during reaching or reach-to-grasp actions can be excited or inhibited by visual information, suggesting that the visual perception of action, rather than object recognition, is the most effective factor for area V6A. Also, three-dimensional object shape is encoded dynamically by the neural population, according to the behavioral context of the monkey. Along this line, mirror neuron discharges in V6A indicate the plasticity of visual representation of the graspable objects, that changes according to the context and peaks when the object is the target of one's own action. In other words, object encoding in V6A is a visual encoding for action.

Application of Object Grasping Using Dual-Arm Autonomous Mobile Robot—Path Planning by Spline Curve and Object Recognition by YOLO—

In the trash-collection challenge of the Nakanoshima Robot Challenge, an autonomous robot must collect trash (bottles, cans, and bentos) scattered in a defined area within a time limit. A method for collecting the trash is to use machine learning to recognize the objects, move to the target location, and grasp the objects. An autonomous robot can achieve the target position and posture by rotating on the spot at the starting point, moving in a straight line, and rotating on the spot at the destination, but the rotation requires stopping and starting. To achieve faster movement, we implemented a smooth movement approach without sequential stops using a spline curve. When using the training data previously generated by the authors in their laboratory for object recognition, the robot could not correctly recognize objects in the environment of the robot competition, where strong sunlight shines through glass, because of the varying brightness and darkness. To solve this problem, we added our newly generated training data to YOLO, an image-recognition algorithm based on deep learning, and performed machine learning to achieve object recognition under various conditions.

Human-Inspired Non-prehensile Manipulation Strategies: Design, Implementation, and Evaluation

Grasping of objects is not always feasible for robot manipulators, e.g., due to their geometric properties. Non-prehensile manipulation strategies can enable manipulators to successfully move these objects around. We analyze human-inspired gripper configurations for pushing small or heavy objects and propose closed-loop pushing strategies based on force-torque measurements as well as open-loop strategies to push small objects. In a thorough evaluation on a KUKA LWR4+ manipulator arm and in simulation, we discuss the effects of the different designs and strategies.

An Inchworm-Snake Inspired Flexible Robotic Manipulator With Multisection SMA Actuators for Object Grasping

Aiming at achieving the object grasping task, an inchworm-snake inspired Flexible Robotic Manipulator (FRM) is presented in this paper. By using both flexible and rigid materials in the robot structure, the FRM has the stability and control accuracy of a rigid robot as well as the compliant behavior of a soft robot. The piecewise-linear driving characteristic of the Shape Memory Alloy (SMA) actuator enables the accurate establishment of the FRM's kinematic model and model-based control method. Deep learning-based eye-in-hand binocular visual perception is designed to obtain the state feedback without relying on external sensors, making it suitable to perform tasks in confined and unstructured environment. A model-based controller with the deformation planning is introduced with the aim of achieving accurate control in object grasping. The experiments demonstrate that the FRM possesses the capability of performing the object grasping with the proposed methods.

Curricula for teaching end-users to kinesthetically program collaborative robots.

Non-expert users can now program robots using various end-user robot programming methods, which have widened the use of robots and lowered barriers preventing robot use by laypeople. Kinesthetic teaching is a common form of end-user robot programming, allowing users to forgo writing code by physically guiding the robot to demonstrate behaviors. Although it can be more accessible than writing code, kinesthetic teaching is difficult in practice because of users' unfamiliarity with kinematics or limitations of robots and programming interfaces. Developing good kinesthetic demonstrations requires physical and cognitive skills, such as the ability to plan effective grasps for different task objects and constraints, to overcome programming difficulties. How to help users learn these skills remains a largely unexplored question, with users conventionally learning through self-guided practice. Our study compares how self-guided practice compares with curriculum-based training in building users' programming proficiency. While we found no significant differences between study participants who learned through practice compared to participants who learned through our curriculum, our study reveals insights into factors contributing to end-user robot programmers' confidence and success during programming and how learning interventions may contribute to such factors. Our work paves the way for further research on how to best structure training interventions for end-user robot programmers.

Comparison of synergy patterns between the right and left hand while performing postures and object grasps

The human hand, with many degrees of freedom, serves as an excellent tool for dexterous manipulation. Previous research has demonstrated that there exists a lower-dimensional subspace that synergistically controls the full hand kinematics. The elements of this subspace, also called synergies, have been viewed as the strategy developed by the CNS in the control of finger movements. Considering that the control of fingers is lateralized to the contralateral hemisphere, how the synergies differ for the control of the dominant and the non-dominant hand has not been widely addressed. In this paper, hand kinematics was recorded using electromagnetic tracking system sensors as participants made various postures and object grasps with their dominant hand and non-dominant hand separately. Synergies that explain 90% of variance in data of both hands were analyzed for similarity at the individual level as well as at the population level. The results showed no differences in synergies between the hands at both these levels. PC scores and cross-reconstruction errors were analyzed to further support the prevalence of similarity between the synergies of the hands. Future work is proposed, and implications of the results to the treatment and diagnosis of neuromotor disorders are discussed.

The functional organization of skilled actions in the adextral and atypical brain

When planning functional grasps of tools, right-handed individuals (dextrals) show mostly left-lateralized neural activity in the praxis representation network (PRN), regardless of the used hand. Here we studied whether or not similar cerebral asymmetries are evident in non-righthanded individuals (adextrals). Sixty two participants, 28 righthanders and 34 non-righthanders (21 lefthanders, 13 mixedhanders), planned functional grasps of tools vs. grasps of control objects, and subsequently performed their pantomimed executions, in an event-related functional magnetic resonance imaging (fMRI) project. Both hands were tested, separately in two different sessions, counterbalanced across participants. After accounting for non-functional components of the prospective grasp, planning functional grasps of tools was associated with greater engagement of the same, left-hemisphere occipito-temporal, parietal and frontal areas of PRN, regardless of hand and handedness. Only when the analyses involved signal changes referenced to resting baseline intervals, differences between adextrals and dextrals emerged. Whereas in the left hemisphere the neural activity was equivalent in both groups (except for the occipito-temporo-parietal junction), its increases in the right occipito-temporal cortex, medial intraparietal sulcus (area MIP), the supramarginal gyrus (area PFt/PF), and middle frontal gyrus (area p9-46v) were significantly greater in adextrals. The inverse contrast was empty. Notably, when individuals with atypical and typical hemispheric phenotypes were directly compared, planning functional (vs. control) grasps invoked, instead, significant clusters located nearly exclusively in the left hemisphere of the typical phenotype. Previous studies interpret similar right-sided vs. left-sided increases in neural activity for skilled actions as handedness dependent, i.e., located in the hemisphere dominant for manual skills. Yet, none of the effects observed here can be purely handedness dependent because there were mixed-handed individuals among adextrals, and numerous mixed-handed and left-handed individuals possess the typical phenotype. Thus, our results clearly show that hand dominance has limited power in driving the cerebral organization of motor cognitive functions.

A new tilted aerial robotic platform: Modeling and control

Aerial Manipulators are considered in modern applications due to their essential features that have the maneuverability of Unmanned Aerial Vehicles (UAVs) and abilities of robotics arms. One of the primary challenges encountered by aerial manipulators is their inability to maintain a consistent orientation though the duration of flight. This limits the application of aerial manipulators for movement of fragile and liquid objects that should not change their orientation during the flight. To solve this issue, this study introduced a new Aerial Robot Platform (AR-P) which can decouple all motions by configuring tilting rotors into the structure of the AR-P. The AR-P consists of a Tilted-Rotor Quadcopter (T-RQ) and three linked Aerial Robot Arm (AR-A) that have a gripper to grasp objects. The AR-P with this new mechanism of tilting parts overcomes the problem of manipulation limitations by adjusting the direction of the thrusted rotors according to the exerted forces of the grasped object. The new dynamics model of the AR-P is derived using the Lagrange-d'Alembert approach. A special Multi Input Multi Output (MIMO) Proportional Derivative (PD) controller based on Computed Torque Control (CTC) is designed for the developed AR-P. The method of nonlinear least squares is applied to obtain the optimal gains of the over-looped PD controller. The simulation results showed the ability of the developed CTC to track the required joint angle of each the T-RQ and each link of the AR-A with minimizing steady state errors and without overshooting. The implemented controller showed its robustness against external disturbances in form of wind conditions. On one hand, the controller is tested considering traditional aerial manipulator. On the other hand, the controller is tested by our new tilted aerial robot platform. The comparison results of the flight scenario between the new tilted aerial manipulator and the traditional one approved the advantage of the proposed new tilted AR-P to transport the grasped objects in fixed orientation.

Fabrication of Origami Soft Gripper Using On-Fabric 3D Printing

In this work, we have presented a soft encapsulating gripper for gentle grasps. This was enabled by a series of soft origami patterns, such as the Yoshimura pattern, which was directly printed on fabric. The proposed gripper features a deformable body that enables safe interaction with its surroundings, gentle grasps of delicate and fragile objects, and encapsulated structures allowing for noninvasive enclosing. The gripper was fabricated by a direct 3D printing of soft materials on fabric. This allowed for the stiffness adjustment of gripper components and a simple fabrication process. We evaluated the grasping performance of the proposed gripper with several delicate and ultra-gentle objects. It was concluded that the proposed gripper could manipulate delicate objects from fruits to silicone jellyfishes and, therefore, have considerable potential for use as improved soft encapsulating grippers in agriculture and engineering fields.

A Dual-Modal Hybrid Gripper with Wide Tunable Contact Stiffness Range and High Compliance for Adaptive and Wide-Range Grasping Objects with Diverse Fragilities.

The difficulties of traditional rigid/soft grippers in meeting the increasing performance expectations (e.g., high grasping adaptability and wide graspable objects range) of a single robotic gripper have given birth to numerous soft-rigid coupling grippers with promising performance. However, it is still hard for these hybrid grippers to adaptively grasp various objects with diverse fragilities intact, such as incense ash and orange, due to their limited contact stiffness adjustable range and compliance. To solve these challenging issues, herein, we propose a dual-modal hybrid gripper, whose fingers contain a detachable elastomer-coated flexible sheet that is restrained by a moving frame as a teardrop shape. The gripper's two modes switched by controlling the moving frame position can selectively highlight the low contact stiffness and excellent compliance of the teardrop-shaped flexible sheets and the high contact stiffness of the moving frames. Moreover, the contact stiffness of the teardrop-shaped sheets can be wide-range adjusted by online controlling the moving frame position and offline replacing the sheets with different thicknesses. The compliance of the teardrop-shaped sheets also proves to be excellent compared with an Ecoflex 10 fingertip with the same profile. Such a gripper with wide-range tunable contact stiffness and high compliance demonstrates excellent grasping adaptability (e.g., it can safely grasp several fragile strawberries with a maximum size difference of 18 mm, a strawberry with a left/right offset of 3 cm, and a strawberry in two different lying poses) and wide-range graspable objects (from 0.1 g super fragile cigarette ashes to 5.1 kg dumbbell).

Grasping of Unknown Objects with an Autonomous Manipulator: State of the Art, Problems and Prospects

Grasping of Unknown Objects with an Autonomous Manipulator: State of the Art, Problems and Prospects

Syntax matters in shaping sensorimotor activation driven by nouns.

Existing evidence has shown that adjectives modulate the grasp-compatibility effect elicited by object nouns. The aim of the present study was to investigate the role of syntax on the sensorimotor activation elicited by nouns in a grasp-compatibility task. We assessed two languages with different syntactic rules, Italian in Experiment 1 and English in Experiment 2. In both experiments, an adjective-noun pair was shown on the screen. The adjective was always in a pre-nominal position and denoted either a disadvantageous quality of the object graspability (e.g., sharp) or the object colour (e.g., reddish). Participants had to categorize the object nouns as natural or artifact, performing a precision or a power reach-to-grasp movement. On different trials, the grasp response was compatible or incompatible with the grip typically used to manipulate the object indicated by the noun. In Experiment 1 (Italian language) the adjective-noun order violated the syntactic order and no difference emerged between reaction times on compatible and incompatible trials (no grasp compatibility effect). In Experiment 2(English language), the adjective-noun order followed the syntactic rule. Results showed a grasp-compatibility effect when a colour adjective was presented before a natural object noun. When a disadvantageous adjective preceded an artifact or a natural object noun, an inverted grasp-compatibility effect emerged with slower responses on compatible than incompatible trials. Taken together, these findings suggest that adjectives can shape the sensorimotor activation elicited by nouns of graspable objects only when the syntax is correct. Results are discussed with respect to embodied cognition theories.

Too much to handle? Interference from distractors with similar affordances on target selection for handled objects.

The existence of handle affordances has been classically demonstrated using the Stimulus-Response Compatibility paradigm, with shorter response times when the orientation of the object handle and the response hand are compatible in comparison to incompatible. Yet the activation of handle affordances from visual objects has been investigated in very simple situations involving single stimulus and motor response. As natural perceptual scenes are usually composed of multiple objects that could activate multiple affordances, the consequence of multiple affordance activation on the perception and processing of a given object of the scene requires more investigation. The aim of this study was to determine the impact of distractor affordances on the processing of a target object in situations involving several familiar graspable objects. In two online experiments, 229 participants had to select a target object (the kitchen utensil or the tool) in a visual scene displaying a pair of objects. They performed left key presses when the target was on the left and right key presses when the target was on the right. Target handle orientation and response side could be compatible or incompatible. Critically, target and distractor objects had similar or dissimilar handle affordances, with handles oriented for left- or right-hand grasps. Results from the two experiments showed slower response times when target and distractor objects had similar handle affordances in comparison to dissimilar affordances, when participants performed right hand responses and when target orientation and response were compatible. Thus, affordance similarity between objects may interfere rather than facilitate object processing and slow down target selection. These findings are in line with models of affordance and object selection assuming automatic inhibition of distractors' affordances for appropriate object interaction.

An Affordable Telerobotic System Architecture for Grasp Training and Object Grasping for Human–Machine Interaction

Abstract Due to mobility impairment, a person might rely on wheelchairs, canes, and crutches for assistance but could face challenges when performing tasks such as grasping and manipulating objects due to limitations in reach and capability. To overcome these challenges, a multidegree-of-freedom robotic arm with an anthropomorphic robotic hand (ARH) could be used. In this research, we propose an architecture and then implement it toward the development of an assistive system to assist a person with object grasping. The architecture interlinks three functional modules to provide three operation modes to calibrate the system, train a user on how to execute a grasp, synthesize grasps, and execute a grasp. The developed system consists of a user input and feedback glove capable of capturing user inputs and providing grasp-related vibrotactile feedback, a coppeliasim virtual environment emulating the motions of the ARH, and an underactuated ARH capable of executing grasps while sensing grasp contact locations. The operation of the developed system is evaluated to determine the ability of a person to operate it and perform a grasp using two control methods; using a synthesized grasp or under real-time continuous control. The successful evaluation validates the architecture and the developed system to provide the ability to perform a grasp. The results of the evaluation provide confidence in expanding the system capabilities and using it to develop a database of grasp trajectories of objects with different geometries.

Universally Grasping Objects with Granular-Tendon Finger: Principle and Design.

Nowadays, achieving the stable grasping of objects in robotics requires an increased emphasis on soft interactions. This research introduces a novel gripper design to achieve a more universal object grasping. The key feature of this gripper design was a hybrid mechanism that leveraged the soft structure provided by multiple granular pouches attached to the finger skeletons. To evaluate the performance of the gripper, a series of experiments were conducted using fifteen distinct types of objects, including cylinders, U-shaped brackets, M3 bolts, tape, pyramids, big pyramids, oranges, cakes, coffee sachets, spheres, drink sachets, shelves, pulley gears, aluminium profiles, and flat brackets. Our experimental results demonstrated that our gripper design achieved high success rates in gripping objects weighing less than 210 g. One notable advantage of the granular-tendon gripper was its ability to generate soft interactions during the grasping process while having a skeleton support to provide strength. This characteristic enabled the gripper to adapt effectively to various objects, regardless of their shape and material properties. Consequently, this work presented a promising solution for manipulating a wide range of objects with both stability and soft interaction capabilities, regardless of their individual characteristics.

A Learning‐Based Sensor Array for Untethered Soft Prosthetic Hand Aiming at Restoring Tactile Sensation

Endowing tactile feedback for prosthetic hands is profound for upper‐limb amputees. However, existing prosthetic hands are generally not in possession of the embedded sensory feedback. Herein, a flexible tactile sensor array which can be integrated into an untethered soft prosthetic hand to achieve static and dynamic discrimination tasks is presented. The flexible piezoresistive sensory arrays with 25 sensor units which can be arranged on five fingers of the soft prosthetic hand are fabricated. According to the collected large‐scale tactile dataset (including pressure distribution and pressure magnitude) during different grasping tasks, a learning‐based classification model that can reveal the correspondences between tactile information and object attributes while interacting with touched objects is developed. To transfer tactile information extracted from tactile sensor arrays, a wearable vibrotactile feedback band with a spatial coding feedback strategy is implemented by selectively activating vibrotactile motors located on the skin of the upper arm. In a set of tests performed by an individual with transradial amputation and eight able‐bodied subjects, the soft prosthetic hand integrated with tactile sensor arrays can help the users regain finger tactile sensation, discriminate grasped objects, and achieve real‐time dynamic rolling detection.

Grasping Unstructured Objects with Full Convolutional Network in Clutter

Grasping objects in cluttered environments remains a significant challenge in robotics, particularly when dealing with novel objects that have not been previously encountered. This paper proposes a novel approach to address the problem of robustly learning object grasping in cluttered scenes, focusing on scenarios where the objects are unstructured and randomly placed. We present a unique Deep Q-learning (DQN) framework combined with a full convolutional network suitable for the end-to-end grasping of multiple adhesive objects in a cluttered environment. Our method combines the depth information of objects with reinforcement learning to obtain an adaptive grasping strategy to enable a robot to learn and generalize grasping skills for novel objects in the real world. The experimental results demonstrate that our method significantly improves the grasping performance on novel objects compared to conventional grasping techniques. Our system demonstrates remarkable adaptability and robustness in cluttered scenes, effectively grasping a diverse array of objects that were previously unseen. This research contributes to the advancement of robotics with potential applications, including, but not limited to, redundant manipulators, dual-arm robots, continuum robots, and soft robots.

Manipulable objects can potentiate pointing and unusual grasping besides habitual grasping behaviors.

People are faster to perform a precision grip when they see a cherry (i.e., a small graspable object) than to perform a power grip, and the reverse holds true when they see an apple (i.e., a large graspable object). This potentiation effect supports that object representations could include motor components that would be simulated when a graspable object is seen. However, the nature of these motor components remains unclear. The embodied account posits that seeing an object only potentiates the most frequent actions associated with it (i.e., usual actions). In contrast, the size-coding account posits that seeing an object potentiates any actions associated to spatial codes compatible with those associated to the objects. We conducted three experiments to disentangle these two alternative accounts. We especially varied the nature of the responses while participants saw either large or small graspable objects. Our results showed a potentiation effect when participants performed the usual grasping actions (Experiment 1: power and precision grip) but also when they performed unusual grasping actions (Experiment 2: grasping between the thumb and little finger) and even when they had to perform non-grasping actions (Experiment 3: pointing actions). By supporting the size-coding account, our contribution underlines the need for a better understanding of the nature of the motor components of object representations and for using a proper control condition (i.e., pointing action) before arguing that the embodied account convincingly explains the potentiation effect of grasping behaviors.