Autonomous Object Manipulation Research Articles

Deep Learning-Based Ensemble Approach for Autonomous Object Manipulation with an Anthropomorphic Soft Robot Hand

Autonomous object manipulation is a challenging task in robotics because it requires an essential understanding of the object’s parameters such as position, 3D shape, grasping (i.e., touching) areas, and orientation. This work presents an autonomous object manipulation system using an anthropomorphic soft robot hand with deep learning (DL) vision intelligence for object detection, 3D shape reconstruction, and object grasping area generation. Object detection is performed using Faster-RCNN and an RGB-D sensor to produce a partial depth view of the objects randomly located in the working space. Three-dimensional object shape reconstruction is performed using U-Net based on 3D convolutions with bottle-neck layers and skip connections generating a complete 3D shape of the object from the sensed single-depth view. Then, the grasping position and orientation are computed based on the reconstructed 3D object information (e.g., object shape and size) using U-Net based on 3D convolutions and Principal Component Analysis (PCA), respectively. The proposed autonomous object manipulation system is evaluated by grasping and relocating twelve objects not included in the training database, achieving an average of 95% successful object grasping and 93% object relocations.

Breathable Metal-Organic Framework Enhanced Humidity-Responsive Nanofiber Actuator with Autonomous Triboelectric Perceptivity.

Autonomous object manipulation and perception with environmental factor-triggered and self-powered actuation is one of the most attractive directions for developing next-generation soft robotics with a smart human-machine-environment interface. Humidity, as a sustainable energy source ubiquitous in the surrounding environment, can be used for triggering smart grippers. In this work, it is proposed that by contacts between the gripper and objects upon humidity-induced actuation, real-time distinguishable triboelectric signals can be generated to realize the humidity-driven object manipulation and identification. Herein, a thermo-modified electrospun polyvinylpyrrolidone/poly(acrylic acid)/MIL-88A (T-PPM) nanofibrous film with micro-to-nano cross-scale porosity is developed, and a bilayer humidity-responsive actuator (T-HRA) was designed, mimicking the tamariskoid spikemoss to enhance the humidity-driven actuation. The breathing effect of MIL-88A and hierarchical porous structure of the T-PPM facilitate moisture diffusion and offer huge actuation (2.41 cm-1) with a fast response (0.084 cm-1 s-1). For autonomous object manipulation perception, T-PPM was verified as a tribo-positive material located between paper and silk. Accordingly, the T-HRA was demonstrated as a smart soft gripper that generates a different electric signal upon contact with objects of different material. This work proposes a concept of soft robots that are interactive with the environment for both autonomous object manipulation and information acquisition.

An Embedded Framework for Fully Autonomous Object Manipulation in Robotic-Empowered Assisted Living

Most of the humanoid social robots currently diffused are designed only for verbal and animated interactions with users, and despite being equipped with two upper arms for interactive animation, they lack object manipulation capabilities. In this paper, we propose the MONOCULAR (eMbeddable autONomous ObjeCt manipULAtion Routines) framework, which implements a set of routines to add manipulation functionalities to social robots by exploiting the functional data fusion of two RGB cameras and a 3D depth sensor placed in the head frame. The framework is designed to: (i) localize specific objects to be manipulated via RGB cameras; (ii) define the characteristics of the shelf on which they are placed; and (iii) autonomously adapt approach and manipulation routines to avoid collisions and maximize grabbing accuracy. To localize the item on the shelf, MONOCULAR exploits an embeddable version of the You Only Look Once (YOLO) object detector. The RGB camera outcomes are also used to estimate the height of the shelf using an edge-detecting algorithm. Based on the item's position and the estimated shelf height, MONOCULAR is designed to select between two possible routines that dynamically optimize the approach and object manipulation parameters according to the real-time analysis of RGB and 3D sensor frames. These two routines are optimized for a central or lateral approach to objects on a shelf. The MONOCULAR procedures are designed to be fully automatic, intrinsically protecting sensitive users' data and stored home or hospital maps. MONOCULAR was optimized for Pepper by SoftBank Robotics. To characterize the proposed system, a case study in which Pepper is used as a drug delivery operator is proposed. The case study is divided into: (i) pharmaceutical package search; (ii) object approach and manipulation; and (iii) delivery operations. Experimental data showed that object manipulation routines for laterally placed objects achieves a best grabbing success rate of 96%, while the routine for centrally placed objects can reach 97% for a wide range of different shelf heights. Finally, a proof of concept is proposed here to demonstrate the applicability of the MONOCULAR framework in a real-life scenario.

Operation Planning and Closed-Loop Control of a Soft Robotic Table for Simultaneous Multiple-Object Manipulation

This article presents the design, development, and validation of a closed-loop control system for a soft robotic table. The soft table can provide independent manipulation to multiple objects simultaneously, which is an advantage over traditional devices such as robotic arms and belt conveyor systems. The table also provides intrinsic soft handling for delicate objects. In order to execute automation tasks, a vision feedback system and an operation controller are developed and incorporated into the existing table prototype to form a closed-loop control. The vision feedback system implements image processing techniques to identify the object location from captured images. The operation controller is integrated from three sets of algorithms, including movement planning, actuator selection, and actuation planning. Two sets of experiments were conducted to validate the closed-loop system and the capability of the soft table. The first set validated the manipulation task of transporting an object to a desired location with autonomous planning and execution. The second set validated the table’s ability to manipulate multiple objects simultaneously and independently. Videos of both experiments have been supplied. Note to Practitioners —A soft robotic table is developed for the manipulation of delicate objects on a surface. The table is intended for automated sorting tasks in an industrial environment. Multiple objects placed on the table surface can be simultaneously and independently transported and orientated on the $XY$ horizontal plane. The deformable surface of the table provides soft handling for delicate objects. These are the advantages over existing manipulators such as robotic arms and belt conveyor systems. Based on previously conducted experiments, it was found that the open-loop control is ineffective to execute automation tasks due to the variability in object movements with soft interactions. To solve this problem, a closed-loop control system for autonomous object manipulation is developed and its detailed design and implementation are presented in this article. A vision feedback system and a controller are developed to form the proposed closed-loop control. Two sets of experiments involving common manipulation tasks were conducted. The control system is proven to be effective at handling the uncertainties in object movements and autonomous object manipulation by the soft robotic table is validated. The performance of this control system can be improved with better predictions of both the deformations on the table surface and the resultant object movements. Embedding deformable sensors into the table surface is a potential approach to achieve such improvement and this can be investigated in future works.

Semantic SLAM for an AUV using object recognition from point clouds

This paper presents a navigation and mapping system for an autonomous underwater vehicle (AUV) while operating near a man-made underwater environment. The objective is to recognize objects (or object parts) and use these as landmarks for simultaneous localization and mapping (SLAM). This approach is intended as the first step towards autonomous object manipulation, to be carried out at a later stage. The approach contains two main components: Object recognition from range data, and feature-based semantic SLAM. For the first component we propose an automatic method for the recognition and location of 3D objects using 3D point clouds as input, extracted from a laser scanner. Since it is common in inspection maintenance and repair (IMR) applications to have access to the 3D models of the objects of interest, the proposed method assumes a priori knowledge of the 3D models of these objects. In typical man-made environments, such objects can be distinct components of a structure, such as valves, pipes and wet-mateable connectors. The object recognition is based on recently proposed global descriptors for point clouds, that allow a compact description of the object shape, which is independent of the object view point. Once an object is recognized, its pose with respect to the AUV is determined using an ICP-based method. The second component of the approach is a feature based SLAM algorithm that uses the recognized objects as landmarks to improve the AUV navigation. The paper presents preliminary results obtained with the Girona 500 AUV, equipped with a fast laser scanner recently developed at the University of Girona. Tests conducted in a controlled environment (water tank) illustrate the suitability of the approach.

What, Where and How? Introducing pose manifolds for industrial object manipulation

In this paper we propose a novel method for object grasping that aims to unify robot vision techniques for efficiently accomplishing the demanding task of autonomous object manipulation. Through ontological concepts, we establish three mutually complementary processes that lead to an integrated grasping system able to answer conjunctive queries such as “What”, “Where” and “How”? For each query, the appropriate module provides the necessary output based on ontological formalities. The “What” is handled by a state of the art object recognition framework. A novel 6 DoF object pose estimation technique, which entails a bunch-based architecture and a manifold modeling method, answers the “Where”. Last, “How” is addressed by an ontology-based semantic categorization enabling the sufficient mapping between visual stimuli and motor commands.

Using an improved SIFT algorithm and fuzzy closed-loop control strategy for object recognition in cluttered scenes.

Partial occlusions, large pose variations, and extreme ambient illumination conditions generally cause the performance degradation of object recognition systems. Therefore, this paper presents a novel approach for fast and robust object recognition in cluttered scenes based on an improved scale invariant feature transform (SIFT) algorithm and a fuzzy closed-loop control method. First, a fast SIFT algorithm is proposed by classifying SIFT features into several clusters based on several attributes computed from the sub-orientation histogram (SOH), in the feature matching phase only features that share nearly the same corresponding attributes are compared. Second, a feature matching step is performed following a prioritized order based on the scale factor, which is calculated between the object image and the target object image, guaranteeing robust feature matching. Finally, a fuzzy closed-loop control strategy is applied to increase the accuracy of the object recognition and is essential for autonomous object manipulation process. Compared to the original SIFT algorithm for object recognition, the result of the proposed method shows that the number of SIFT features extracted from an object has a significant increase, and the computing speed of the object recognition processes increases by more than 40%. The experimental results confirmed that the proposed method performs effectively and accurately in cluttered scenes.

An open-source multi-DOF articulated robotic educational platform for autonomous object manipulation

This research presents an autonomous robotic framework for academic, vocational and training purpose. The platform is centred on a 6 Degree Of Freedom (DOF) serial robotic arm . The kinematic and dynamic models of the robot have been derived to facilitate controller design . An on-board camera to scan the arm workspace permits autonomous applications development. The sensory system consists of position feedback from each joint of the robot and a force sensor mounted at the arm gripper . External devices can be interfaced with the platform through digital and analog I/O ports of the robot controller . To enhance the learning outcome for beginners, higher level commands have been provided. Advanced users can tailor the platform by exploiting the open-source custom-developed hardware and software architectures . The efficacy of the proposed platform has been demonstrated by implementing two experiments; autonomous sorting of objects and controller design. The proposed platform finds its potential to teach technical courses (like Robotics, Control, Electronics, Image-processing and Computer vision) and to implement and validate advanced algorithms for object manipulation and grasping, trajectory generation , path planning, etc. It can also be employed in an industrial environment to test various strategies prior to their execution on actual manipulators. • The proposed platform is centred on a 6 DOF serial robotic arm. • An on-board camera permits development of autonomous applications. Other feedbacks include force and position sensors data. • Derived kinematic and dynamic models of the robot offer readily implementation of the control strategies. • The efficacy of the platform has been demonstrated by implementing two common industrial tasks; ‘pick and place’ and ‘sorting’. • The open-source platform finds potential in training and teaching courses on Robotics, Control, Mechatronics and Image-processing.

Robust Stereo-Vision Based 3D Object Reconstruction for the Assistive Robot FRIEND

A key requirement of assistive robot vision is the robust 3D object reconstruction in complex environments for reliable autonomous object manipulation. In this paper the idea is presen ...

1P2-S-039 QRコードを用いた自律的物体マニピュレーションに関する研究(ネットワーク/メディアロボティクス1,生活を支援するロボメカ技術のメガインテグレーション)

1P2-S-039 QRコードを用いた自律的物体マニピュレーションに関する研究(ネットワーク/メディアロボティクス1,生活を支援するロボメカ技術のメガインテグレーション)