Programming By Demonstration System Research Articles

Industrial robot programming by demonstration using stereoscopic vision and inertial sensing

PurposeThe purpose of this paper is to present a programming by demonstration (PbD) system based on 3D stereoscopic vision and inertial sensing that provides a cost-effective pose tracking system, even during error-prone situations, such as camera occlusions.Design/methodology/approachThe proposed PbD system is based on the 6D Mimic innovative solution, whose six degrees of freedom marker hardware had to be revised and restructured to accommodate an IMU sensor. Additionally, a new software pipeline was designed to include this new sensing device, seeking the improvement of the overall system’s robustness in stereoscopic vision occlusion situations.FindingsThe IMU component and the new software pipeline allow the 6D Mimic system to successfully maintain the pose tracking when the main tracking tool, i.e. the stereoscopic vision, fails. Therefore, the system improves in terms of reliability, robustness, and accuracy which were verified by real experiments.Practical implicationsBased on this proposal, the 6D Mimic system reaches a reliable and low-cost PbD methodology. Therefore, the robot can accurately replicate, on an industrial scale, the artisan level performance of highly skilled shop-floor operators.Originality/valueTo the best of the authors’ knowledge, the sensor fusion between stereoscopic images and IMU applied to robot PbD is a novel approach. The system is entirely designed aiming to reduce costs and taking advantage of an offline processing step for data analysis, filtering and fusion, enhancing the reliability of the PbD system.

A spatial information inference method for programming by demonstration of assembly tasks by integrating visual observation with CAD model

PurposeIn robot programming by demonstration (PbD) of small parts assembly tasks, the accuracy of parts poses estimated by vision-based techniques in demonstration stage is far from enough to ensure a successful execution. This paper aims to develop an inference method to improve the accuracy of poses and assembly relations between parts by integrating visual observation with computer-aided design (CAD) model.Design/methodology/approachIn this paper, the authors propose a spatial information inference method called probabilistic assembly graph with optional CAD model, shorted as PAGC*, to achieve this task. Then an assembly relation extraction method from CAD model is designed, where different assembly relation descriptions in CAD model are summarized into two fundamental relations that are colinear and coplanar. The relation similarity, distance similarity and rotation similarity are adopted as the similar part matching criterions between the CAD model and the observation. The knowledge of part in CAD is used to correct that of the corresponding part in observation. The likelihood maximization estimation is used to infer the accurate poses and assembly relations based on the probabilistic assembly graph.FindingsIn the experiments, both simulated data and real-world data are applied to evaluate the performance of the PAGC* model. The experimental results show the superiority of PAGC* in accuracy compared with assembly graph (AG) and probabilistic assembly graph without CAD model (PAG).Originality/valueThe paper provides a new approach to get the accurate pose of each part in demonstration stage of the robot PbD system. By integrating information from visual observation with prior knowledge from CAD model, PAGC* ensures the success in execution stage of the PbD system.

MASD: A Multimodal Assembly Skill Decoding System for Robot Programming by Demonstration

Programming by demonstration (PBD) transforms the robot programming from the code level to automated interface between robot and human, promoting the flexibility of robotized automation. In this paper, we focus on programming the industrial robot for assembly tasks by parsing the human demonstration into a series of assembly skills and compiling the skill to the robot executables. To achieve this goal, an identification system using multimodal information to recognize the assembly skill, called MASD, is proposed including: 1) an initial learning stage using a hierarchical model to recognize the action by considering the features from action-object effect, gesture, and trajectory and 2) a retrospective thinking stage using a segmentation method to cut the continuous demonstrations into multiple assembly skills optimally. Using MASD, the demonstration of assembly tasks can be explained with high accuracy in real time, driving a hypothesis that a PBD system on the top of MASD can be extended to more realistic assembly tasks beyond pure positional moving and picking. In experiments, the skill identification module is used to recognize the five kinds of assembly skills in demonstrations of both single and multiple assembly skills, and outperforms the comparative action identification methods. Besides integrated with the MASD, the PBD system can generate the program based on the demonstration and successfully enable an ABB industrial robotic arm simulator to assemble a flashlight and a switch, verifying the initial hypothesis. Note to Practitioners —In the conventional robotized automation, the key role of the robot mainly owes to its capacity for repeating a wide variety of tasks with high speed and accuracy in long term, with a cost of days to months of programming for deployment. On the other hand, the new trend of customization brings the new characteristics: production in short cycle and small volume. This irreversible momentum urges the robot to switch from task to task efficiently. The biggest bottleneck here is the tedious programming, which also has high prerequisites for most practitioners in manufacturing. This situation motivates the development of a PBD system that can understand the assembly skills performed by the human experts in the demonstration and accordingly generate the program for robot’s execution of the taught task. In this paper, we present a skill decoding system to parse the observational raw demonstration into symbolic sequences, which is the crucial bridge to enable the automatic programming. The system achieves high performance in recognition and is tailored for the PBD in assembly tasks by considering both advantages and disadvantages in the background of assembly, such as controllable environment and limited computational resources. It is particularly useful for assembly tasks with modularized actions based on a set of standard parts. At the perspective of industrial application, the PBD upon the proposed system is a promising solution to improve the flexibility of manufacture, which is expected to be true in midterm but an important step toward this goal.

Advances in Robot Programming by Demonstration

Robot Programming by Demonstration (PbD) has been dealt with in the literature as a promising way to teach robots new skills in an intuitive way. In this paper we describe our current work in the field toward the implementation of PbD system which allows robots to learn continuously from human observation, build generalized representations of human demonstration and apply such representations to new situations.

Robust trajectory learning and approximation for robot programming by demonstration

Trajectory learning is a fundamental component in a robot Programming by Demonstration (PbD) system, where often the very purpose of the demonstration is to teach complex manipulation patterns. However, human demonstrations are inevitably noisy and inconsistent. This paper highlights the trajectory learning component of a PbD system for manipulation tasks encompassing the ability to cluster, select, and approximate human demonstrated trajectories. The proposed technique provides some advantages with respect to alternative approaches and is suitable for learning from both individual and multiple user demonstrations.

Graphical representation of programs in a demonstrational visual shell—an empirical evaluation

An open question in the area of Programming by Demonstration (PBD) is how to best represent the inferred program. Without a way to view, edit, and share programs, PBD systems will never reach their full potential. We designed and implemented two graphical representation languages for a PBD desktop similar to the Apple Macintosh Finder. Although a user study showed that both languages enabled nonprogrammers to generate and comprehend programs, the study also revealed that the language that more closely reflected the desktop domain doubled users' abilities to accurately generate programs. Trends suggest that the same language was easier for users to comprehend. These findings suggest that it is possible for a PBD system to enable nonprogrammers to construct programs and that the form of the representation can impact the PBD system's effectiveness. A paper-and-pencil evaluation of the two versions of the PBD desktop prior to the study supported these finding and provided interesting feedback on the interaction between usability evaluations and user studies. In particular, the comparison of the paper-and-pencil evaluation with the empirical evaluation suggested that nonempirical evaluation techniques can provide guidance into how to interpret empirical data and, in particular, that PBD systems need to provide support for programming-strategy selection in order to be successful.

Robot programming by Demonstration (RPD): Supporting the induction by human interaction

Programming by Demonstration (PbD) is a programming method that allows software developers to add new functionalities to a system by simply showing them in the form of few examples. In the robotics domain it has the potential to reduce the amount of time required for programming and also to make programming more “natural”. Just imagine the task of assembling a torch by a manipulator. Wouldn't it be nice to just assemble the torch with one's own hands, watched by video and laser cameras and maybe wearing data gloves, i. e. sensors that provide the data to automatically generate the necessary robot program for the assembly task? And wouldn't it be even nicer to demonstrate the task with few different torches, but achieving an assembly function for all possible variants of them? In order to realize such a PbD environment, at least two major problems have to be solved. First, the sensor data have to be transformed to high-level situation-action descriptors, a task that is not yet solved in general. Second, if a generalization is required, induction algorithms must be applied to the recorded and transformed traces, aiming to find the most general user-intended function from only few examples. In this article we will concentrate only on the second problem. The described experimental environment consists of an industrial robot (PUMA 260b), a 6D teach mouse input device, and some sensors. Various data can be recorded during a demonstration for further processing in the PbD system running on a workstation. The objective is to explore the possibilities of integrating learning and clustering algorithms for automated robot programming. In particular it is investigated how human interaction within system- as well as user-initiated dialogs can support the induction component.