Autonomous Robotic Manipulation Research Articles

YOLOPose V2: Understanding and improving transformer-based 6D pose estimation

6D object pose estimation is a crucial prerequisite for autonomous robot manipulation applications. The state-of-the-art models for pose estimation are convolutional neural network (CNN)-based. Lately, Transformers, an architecture originally proposed for natural language processing, is achieving state-of-the-art results in many computer vision tasks as well. Equipped with the multi-head self-attention mechanism, Transformers enable simple single-stage end-to-end architectures for learning object detection and 6D object pose estimation jointly. In this work, we propose YOLOPose (short form for You Only Look Once Pose estimation), a Transformer-based multi-object 6D pose estimation method based on keypoint regression and an improved variant of the YOLOPose model. In contrast to the standard heatmaps for predicting keypoints in an image, we directly regress the keypoints. Additionally, we employ a learnable orientation estimation module to predict the orientation from the keypoints. Along with a separate translation estimation module, our model is end-to-end differentiable. Our method is suitable for real-time applications and achieves results comparable to state-of-the-art methods. We analyze the role of object queries in our architecture and reveal that the object queries specialize in detecting objects in specific image regions. Furthermore, we quantify the accuracy trade-off of using datasets of smaller sizes to train our model.

Online robotics technology course design by balancing workload and affect

This paper describes our course design approach that successfully transformed an undergraduate Robotics Technology course from in-person teaching to online guided by the TASKS model. Our course redesign process includes identifying conflicts, generating solutions, self-evaluation, and analyses of design solutions. We carefully balanced between Workload and students’ Affect, by designing comparable workloads as before, as well as maintaining students’ motivation similarly at the in-person level. Transforming this course consisting of lecture sessions and lab sessions with hardware and software elements yielded a set of course activities and teaching practices applicable to online teaching of other courses. When physical robots become unavailable, simulation projects were designed as alternatives and simulated versions of those physical projects used in face-to-face classroom. These simulation projects are in the areas of autonomous mobile robots, robotic manipulator, and advanced robotic control on MATLAB-ROS, respectively. Comparisons with past in-person results confirm that effective learning has been achieved remotely, having maintained student’s performance and motivation.

Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation.

Multistep tasks, such as block stacking or parts (dis)assembly, are complex for autonomous robotic manipulation. A robotic system for such tasks would need to hierarchically combine motion control at a lower level and symbolic planning at a higher level. Recently, reinforcement learning (RL)-based methods have been shown to handle robotic motion control with better flexibility and generalizability. However, these methods have limited capability to handle such complex tasks involving planning and control with many intermediate steps over a long time horizon. First, current RL systems cannot achieve varied outcomes by planning over intermediate steps (e.g., stacking blocks in different orders). Second, the exploration efficiency of learning multistep tasks is low, especially when rewards are sparse. To address these limitations, we develop a unified hierarchical reinforcement learning framework, named Universal Option Framework (UOF), to enable the agent to learn varied outcomes in multistep tasks. To improve learning efficiency, we train both symbolic planning and kinematic control policies in parallel, aided by two proposed techniques: 1) an auto-adjusting exploration strategy (AAES) at the low level to stabilize the parallel training, and 2) abstract demonstrations at the high level to accelerate convergence. To evaluate its performance, we performed experiments on various multistep block-stacking tasks with blocks of different shapes and combinations and with different degrees of freedom for robot control. The results demonstrate that our method can accomplish multistep manipulation tasks more efficiently and stably, and with significantly less memory consumption.

Autonomous Robotic Manipulation: Real-Time, Deep-Learning Approach for Grasping of Unknown Objects

Recent advancement in vision-based robotics and deep-learning techniques has enabled the use of intelligent systems in a wider range of applications requiring object manipulation. Finding a robust solution for object grasping and autonomous manipulation became the focus of many engineers and is still one of the most demanding problems in modern robotics. This paper presents a full grasping pipeline proposing a real-time data-driven deep-learning approach for robotic grasping of unknown objects using MATLAB and convolutional neural networks. The proposed approach employs RGB-D image data acquired from an eye-in-hand camera centering the object of interest in the field of view using visual servoing. Our approach aims at reducing propagation errors and eliminating the need for complex hand tracking algorithm, image segmentation, or 3D reconstruction. The proposed approach is able to efficiently generate reliable multi-view object grasps regardless of the geometric complexity and physical properties of the object in question. The proposed system architecture enables simple and effective path generation and a real-time tracking control. In addition, our system is modular, reliable, and accurate in both end effector path generation and control. We experimentally justify the efficacy and effectiveness of our overall system on the Barrett Whole Arm Manipulator.

Structure‐aware multiple salient region detection and localization for autonomous robotic manipulation

This paper proposes a multiple salient region detection and localization approach for unstructured industrial robot work environments with arbitrarily located and orientated objects. Different from the existing, the authors' novel technique to detect multiple salient regions performs locally adaptive center-surround operations on proto-object partitions obtained through color consistency and spatial proximity analysis. The multi-scale center-surround operations are done by masks that are local structure-aware yielding regions with precise and accurate boundaries as required for robotic manipulation. First, experiments to evaluate the multiple salient region detection performance are carried out using four standard databases having images with multiple salient objects. Quantitative result analysis using F-measure, shuffled F-measure, shuffled AUC and MAE, and subjective result inspection suggests that the proposed approach is in general better at collectively detecting multiple salient regions than the state-of-the-art, including those based on deep learning. Then, real-life experiments involving robotic manipulation are carried out to demonstrate the utility of the multiple salient region detection method. For robotic manipulation, object localization is improved after salient region detection by employing a fast shadow detection algorithm proposed based on hue analysis, and recognition through existing matching techniques is applied only at the localized salient regions. The benefit of the novel multiple salient region detection approach in the robotic manipulation system is shown using localization and pose estimation accuracy, rates of detection and recognition, positional and angular errors, and processing speed.

A neural autonomous robotic manipulator with three degrees of freedom

This study focuses on the development of a robotic arm with three degrees of freedom designed for different applications, especially in industry, where the main contribution is the use of a low-cost robotic arm manipulator, which with the proper adjustments, mainly of speed, aims to add the concept of collaborative robotics in its most advanced versions. This study employs some classical principles of robotics, such as acquisition, image processing, and object manipulation. In this context, artificial neural networks were used for two essential functionalities: camera calibration and inverse kinematics resolution. From the results obtained with the use of the artificial neural networks (ANNs), it was possible to control the trajectory of the servo motors with the aid of software support. Also, by employing ANNs and using computer vision techniques, the robotic manipulator achieves the objectives of manipulating objects in space with a relatively acceptable error rate, considering the low-cost proposal applied. This manipulator, at first, has a didactic purpose, but the concepts and applications can be used in different areas of the industry.

High-Precision Mobile Robotic Manipulator for Reconfigurable Manufacturing Systems

The manufacturing industry has identified a new megatrend of mass customization, which is one of the essential goals of Industry 4.0. This megatrend requires the realization of manufacturing that can respond quickly and flexibly to various changing production requirements and ensure the achievement of various quality criteria. However, the manufacturing cannot be realized by conventional manufacturing systems in which reconfigurations need to be performed by skilled engineers. This paper proposes a new reconfigurable manufacturing system concept based on an ultra-flexible transfer system. Particularly, an autonomous mobile robotic manipulator, consisting of a high-performance automated guided vehicle module and a collaborative robotic manipulator module, represents a key component of the system concept. In this context, the focus is on the cooperative control between the modules of the autonomous mobile manipulator, which is essential for high-precision processes (e.g., machining, assembly, measurement, inspection), and its wide operating area. The experimental results confirm that the proposed cooperative control improves the positioning performance of the autonomous mobile manipulator, including the time required for positioning and the positioning accuracy.

Special Issue on New Technologies for Robotic Manipulators and Their Industrial Applications

The industrial robot is more precisely an “automatically controlled, reprogrammable, multipurpose manipulator, programmable in three or more axes, which can be either fixed in place or mobile” (ISO 8373:2012). According to the International Federation of Robotics, by 2018, more than 400,000 new units were being installed annually, and the global average robot density in the manufacturing industry was 99 robots per 10,000 employees. More than 30% of all installed robots were in the automotive industry, the biggest customer for robots. Research on measuring and calibrating, modeling, programming and controlling, and integrating systems has been conducted to give robotic manipulators a wider variety of industrial applications. This special issue covers technical and academic efforts related to new technologies that improve the accuracy and facilitate the implementation of robotic manipulators in industrial applications. The first paper, by Ibaraki et al., outlines technical issues and future research directions for the implementation of model-based numerical compensation schemes for industrial robots. The random forest method is used by Kato et al. to construct a calibration model for positioning errors and identify industrial robots’ positioning errors. A procedure for the quasi-static compliance calibration of serial articulated industrial manipulators is proposed by Theissen et al. A review of the kinematic modeling theory and a derived algorithm to identify error sources for a six-axis industrial robot are presented by Alam et al. Nagao et al. derive a forward kinematics model and identify the kinematics parameters for the calibration of a robot-type machine tool. A novel trajectory generation algorithm, including a corner smoothing method for high-speed and high-accuracy machining by industrial robots, is proposed by Tajima et al. Sato et al. study the vibration characteristics of an industrial robot and derive a mathematical model that represents the dynamic behavior of the system. In the context of smart manufacturing, a multilayer quality inspection framework including a measurement instrument and a robot manipulator is introduced by Azamfirei et al. To support mass customization and the development of reconfigurable manufacturing systems, Inoue et al. propose an autonomous mobile robotic manipulator. Yonemoto and Suwa present an adaptive manipulation procedure to establish an automated scheduling technique that flexibly responds to unforeseen events, such as machine failures. Sasatake et al. introduce a learning system that is based on a method for calculating the similarity between tools, and they test it on a robot system for doing housework. Finally, for better knowledge of the key challenges that manufacturers experience in implementing collaborative industrial robots, an industrial survey is conducted by Andersson et al. The editors sincerely appreciate the contributions of all the authors as well as the work of the reviewers. We are confident that this special issue will further encourage research and engineering work to increase our understanding and knowledge of robotic manipulators and their industrial applications.

Grasp Stability Prediction for a Dexterous Robotic Hand Combining Depth Vision and Haptic Bayesian Exploration.

Grasp stability prediction of unknown objects is crucial to enable autonomous robotic manipulation in an unstructured environment. Even if prior information about the object is available, real-time local exploration might be necessary to mitigate object modelling inaccuracies. This paper presents an approach to predict safe grasps of unknown objects using depth vision and a dexterous robot hand equipped with tactile feedback. Our approach does not assume any prior knowledge about the objects. First, an object pose estimation is obtained from RGB-D sensing; then, the object is explored haptically to maximise a given grasp metric. We compare two probabilistic methods (i.e. standard and unscented Bayesian Optimisation) against random exploration (i.e. uniform grid search). Our experimental results demonstrate that these probabilistic methods can provide confident predictions after a limited number of exploratory observations, and that unscented Bayesian Optimisation can find safer grasps, taking into account the uncertainty in robot sensing and grasp execution.

Modeling Method of Autonomous Robot Manipulator Based on D-H Algorithm

With the continuous development of science and technology, robotics is widely used in various fields. In recent years, more and more research studies have been done on the control of autonomous robotic manipulators. How to quickly, accurately, and smoothly grasp objects has always been a difficult point of research. As the robot’s executive mechanism, the robot arm plays an important role in whether the robot can complete a specific task. Therefore, the research on the robot arm is also the main topic in the development of robot technology. The control theory, kinematics, and human-computer interaction of robotic arms are the focus of the research in the field of robotic arms. Based on the above background, the research content of this paper is the research on the modeling method of autonomous robotic manipulator based on D-H algorithm. This paper uses D-H modeling method to model a four-degree-of-freedom robotic arm and gives the forward kinematics equation of the robotic arm. The inverse solution of the manipulator was given by the method and the geometric method, and the joint variable values were calculated. Finally, through experimental simulation, the experimental results show that the inverse solution of the end position of the machine by the geometric method is in the range of 2∼4 mm, and the inverse solution of the end position of the machine by the algebraic method is in the range of 6∼14 mm. It is more accurate to find the inverse solution of the geometrical method of the manipulator than the algebraic method.

Real-time deep learning approach to visual servo control and grasp detection for autonomous robotic manipulation

Robots still cannot perform everyday manipulation tasks, such as grasping, with the same dexterity as humans do. In order to explore the potential of supervised deep learning for robotic grasping in unstructured and dynamic environments, this work addresses the visual perception phase involved in the task. This phase involves the processing of visual data to obtain the location of the object to be grasped, its pose and the points at which the robot’s grippers must make contact to ensure a stable grasp. For this, the Cornell Grasping Dataset (CGD) is used to train a Convolutional Neural Network (CNN) that is able to consider these three stages simultaneously. In other words, having an image of the robot’s workspace, containing a certain object, the network predicts a grasp rectangle that symbolizes the position, orientation and opening of the robot’s parallel grippers the instant before its closing. In addition to this network, which runs in real-time, another network is designed, so that it is possible to deal with situations in which the object moves in the environment. Therefore, the second convolutional network is trained to perform a visual servo control, ensuring that the object remains in the robot’s field of view. This network predicts the proportional values of the linear and angular velocities that the camera must have to ensure the object is in the image processed by the grasp network. The dataset used for training was automatically generated by a Kinova Gen3 robotic manipulator with seven Degrees of Freedom (DoF). The robot is also used to evaluate the applicability in real-time and obtain practical results from the designed algorithms. Moreover, the offline results obtained through test sets are also analyzed and discussed regarding their efficiency and processing speed. The developed controller is able to achieve a millimeter accuracy in the final position considering a target object seen for the first time. To the best of our knowledge, we have not found in the literature other works that achieve such precision with a controller learned from scratch. Thus, this work presents a new system for autonomous robotic manipulation, with the ability to generalize to different objects and with high processing speed, which allows its application in real robotic systems.

A method for autonomous robotic manipulation through exploratory interactions with uncertain environments

Expanding robot autonomy can deliver functional flexibility and enable fast deployment of robots in challenging and unstructured environments. In this direction, significant advances have been recently made in visual-perception driven autonomy, which is mainly due to the availability of rich sensory data-sets. However, current robots’ physical interaction autonomy levels still remain at a basic level. Towards providing a systematic approach to this problem, this paper presents a new context-aware and adaptive method that allows a robotic platform to interact with unknown environments. In particular, a multi-axes self-tuning impedance controller is introduced to regulate quasi-static parameters of the robot based on previous experience in interacting with similar environments and the real-time sensory data. The proposed method is also capable of differentiating internal and external disruptions, and responding to them accordingly and appropriately. An agricultural experiment with different deformable material is presented to validate robot interaction autonomy improvements, and the capability of the proposed methodology in detecting and responding to unexpected events (e.g., faults).

Vision-Based Measurement and Prediction of Object Trajectory for Robotic Manipulation in Dynamic and Uncertain Scenarios

Vision-based measurement and prediction (VMP) are the very important and challenging part for autonomous robotic manipulation, especially in dynamic and uncertain scenarios. However, due to the potential limitations of visual measurement in such an environment such as occlusion, lighting, and hardware limitations, it is not easy to acquire the accurate positions of an object as the observations. Moreover, manipulating a dynamic object with unknown or uncertain motion rules usually requires an accurate prediction of motion trajectory at the desired moment, which dramatically increases the difficulty. To address the problem, we propose a time granularity-based vision prediction framework whose core is an integrated prediction model based on multiple [i.e., long short-term memory (LSTM)] neural networks. At first, we use the vision sensor to acquire raw measurements and adopt the preprocessing method (e.g., data completion, error compensation, and filtering) to turn raw measurements into the standard trajectory data. Then, we devise a novel integration strategy based on time granularity boost (TG-Boost) to select appropriate base predictors and further utilize these history trajectory data to construct the high-precision prediction model. Finally, we use the simulation and a series of dynamic manipulation experiments to validate the proposed methodology. The results also show that our method outperforms the state-of-the-art prediction algorithms in terms of prediction accuracy, success rate, and robustness.

3D Object Recognition and Pose Estimation From Point Cloud Using Stably Observed Point Pair Feature

Recognition and pose estimation from 3D free-form objects is a key step for autonomous robotic manipulation. Recently, the point pair features (PPF) voting approach has been shown to be effective for simultaneous object recognition and pose estimation. However, the global model descriptor (e.g., PPF and its variants) that contained some unnecessary point pair features decreases the recognition performance and increases computational efficiency. To address this issue, in this paper, we introduce a novel strategy for building a global model descriptor using stably observed point pairs. The stably observed point pairs are calculated from the partial view point clouds which are rendered by the virtual camera from various viewpoints. The global model descriptor is extracted from the stably observed point pairs and then stored in a hash table. Experiments on several datasets show that our proposed method reduces redundant point pair features and achieves better compromise of speed vs accuracy.

Design and Implementation of an Autonomous Robot Manipulator for Pick & Place Planning

This paper focuses on advancement of an autonomous robot manipulator for pick and place applications using Artificial Intelligence Technique. The mobile robot carries a four Degree of Freedom (DOF) robotic arm for picking a particular object from a given initial position to the required target position in an unknown static environment, while tackling obstacles and implementing path planning algorithm using artificial intelligence technique. The proposed artificial intelligence technique employs the optimization strategy, based on a novel meta-heuristic approach for mobile robot path planning. The mobile manipulator involves the use of infrared, ultrasonic sensors for detecting obstacles, microcontroller for artificial intelligence software and geared motors & servo motors for motion locomotion. The effectiveness of the method was tested and verified by via simulation mode on four different trajectories. From the simulation results, it was found that approximately path length and elapsed time of triangular shape, conical shape, cubical shape&S- shapeobstacles environments are 70.22, 70.92, 71.29 & 70.36 pixels and 117.93, 73.94, 122.86 & 117.87 seconds respectively.

SilhoNet: An RGB Method for 6D Object Pose Estimation

Autonomous robot manipulation involves estimating the translation and orientation of the object to be manipulated as a 6-degree-of-freedom (6D) pose. Methods using RGB-D data have shown great success in solving this problem. However, there are situations where cost constraints or the working environment may limit the use of RGB-D sensors. When limited to monocular camera data only, the problem of object pose estimation is very challenging. In this work, we introduce a novel method called SilhoNet that predicts 6D object pose from monocular images. We use a Convolutional Neural Network (CNN) pipeline that takes in Region of Interest (ROI) proposals to simultaneously predict an intermediate silhouette representation for objects with an associated occlusion mask and a 3D translation vector. The 3D orientation is then regressed from the predicted silhouettes. We show that our method achieves better overall performance on the YCB-Video dataset than two state-of-the art networks for 6D pose estimation from monocular image input.

Continuous Visual World Modeling for Autonomous Robot Manipulation

Service robots need to handle a variety of everyday manipulation tasks to accomplish household chores such as cooking and cleaning. Successful execution of these tasks is highly dependent on how reliable the robot perceives its environment through noisy sensing. We present a visual world modeling system for service robots to generate and maintain accurate models of their environments for continuous scenarios. This system is designed to provide a generic platform for both humanoid and ground manipulation robots using different types of vision sensors and algorithms. In our particular implementation, visual data processed by different perception algorithms are used for building and continuously updating a world model of the environment. We evaluate our system on a variety of object manipulation scenarios and show that the system produces consistent perception outcomes suitable for different manipulation tasks.

Virtual Training for a Real Application: Accurate Object-Robot Relative Localization Without Calibration

Localizing an object accurately with respect to a robot is a key step for autonomous robotic manipulation. In this work, we propose to tackle this task knowing only 3D models of the robot and object in the particular case where the scene is viewed from uncalibrated cameras -- a situation which would be typical in an uncontrolled environment, e.g., on a construction site. We demonstrate that this localization can be performed very accurately, with millimetric errors, without using a single real image for training, a strong advantage since acquiring representative training data is a long and expensive process. Our approach relies on a classification Convolutional Neural Network (CNN) trained using hundreds of thousands of synthetically rendered scenes with randomized parameters. To evaluate our approach quantitatively and make it comparable to alternative approaches, we build a new rich dataset of real robot images with accurately localized blocks.

Robust shape estimation for 3D deformable object manipulation

Existing shape estimation methods for deformable object manipulation suffer from the drawbacks of being off-line, model dependent, noise-sensitive or occlusion-sensitive, and thus are not appropriate for manipulation tasks requiring high precision. In this paper, we present a real-time shape estimation approach for autonomous robotic manipulation of 3D deformable objects. Our method fulfills all the requirements necessary for the high-quality deformable object manipulation in terms of being real-time, model-free and robust to noise and occlusion. These advantages are accomplished using a joint tracking and reconstruction framework, in which we track the object deformation by aligning a reference shape model with the stream input from the RGB-D camera, and simultaneously upgrade the reference shape model according to the newly captured RGB-D data. We have evaluated the quality and robustness of our real-time shape estimation pipeline on a set of deformable manipulation tasks implemented on physical robots. Videos are available at https://lifeisfantastic.github.io/DeformShapeEst/

RGB-D object detection and semantic segmentation for autonomous manipulation in clutter

Autonomous robotic manipulation in clutter is challenging. A large variety of objects must be perceived in complex scenes, where they are partially occluded and embedded among many distractors, often in restricted spaces. To tackle these challenges, we developed a deep-learning approach that combines object detection and semantic segmentation. The manipulation scenes are captured with RGB-D cameras, for which we developed a depth fusion method. Employing pretrained features makes learning from small annotated robotic datasets possible. We evaluate our approach on two challenging datasets: one captured for the Amazon Picking Challenge 2016, where our team NimbRo came in second in the Stowing and third in the Picking task; and one captured in disaster-response scenarios. The experiments show that object detection and semantic segmentation complement each other and can be combined to yield reliable object perception.