Robot Programming Framework Research Articles

A versatile interaction framework for robot programming based on hand gestures and poses

This paper proposes a framework for industrial and collaborative robot programming based on the integration of hand gestures and poses. The framework allows operators to control the robot via both End-Effector (EE) and joint movements and to transfer compound shapes accurately to the robot. Seventeen hand gestures, which cover the position and orientation controls of the robotic EE and other auxiliary operations, are designed according to cognitive psychology. Gestures are classified by a deep neural network, which is pre-trained for two-hand pose estimation and fine-tuned on a custom dataset, achieving a test accuracy of 99%. The index finger’s pointing direction and the hand’s orientation are extracted via 3D hand pose estimation to indicate the robotic EE’s moving direction and orientation, respectively. The number of stretched fingers is detected via two-hand pose estimation to represent decimal digits for selecting robot joints and inputting numbers. Finally, we integrate these three manners seamlessly to form a programming framework. We conducted two interaction experiments. The reaction time of the proposed hand gestures in indicating randomly given instructions is significantly less than that of other gesture sets, such as American Sign Language (ASL). The accuracy of our method in compound shape reconstruction is much better than that of hand movement trajectory-based methods, and the operating time is comparable with that of teach pendants.

Using an Intelligent Robot to Introduce Elementary School Students to Computational Thinking and Artificial Intelligence

In 2018 I received a SIGCSE Special Projects award to fund creation of computational thinking and artificial intelligence curriculum materials for students in grades 3-5 using my Calypso for Cozmo robot programming framework (https://Calypso.software) and the Cozmo robot from Anki. The work was done in collaboration with the New Brighton Area School District in New Brighton, Pennsylvania. The project succeeded, then failed, and may now be reborn again.

Experimental Evaluation Of Intuitive Programming Of Robot Interaction Behaviour During Kinesthetic Teaching Using sEMG And Cutaneous Feedback

Modern applications related to service or production can nowadays benefit from the introduction of collaborative robots already available on the market and endowed with several advanced features such as precise torque control or safe physical interaction with operators. More importantly, collaborative robots allow operators to teach end-effector trajectories by means of physical interaction – known as kinesthetic teaching – which is one of the most intuitive programming-by-demonstration techniques. However, important functionalities provided by modern collaborative robots, like the possibility of performing smooth interactions, cannot be programmed intuitively with the available framework of kinesthetic teaching. In the present study, we propose and experimentally evaluate a robot programming framework for the simultaneous teaching of both trajectories by means of kinesthetic teaching, and robot interaction behavior by means of impedance shaping along the trajectory exploiting a wearable interface. Specifically, the wearable interface is designed to not affect the free motion of the operator, necessary to perform kinesthetic teaching, and it is based on the usage of surface electromyography (sEMG) and vibrotactile stimulation. In this way, we propose an intuitive robot programming framework for an offline robot trajectory and interaction behavior programming, according to which the operator will be able to plan interactions with the environment and humans. In this article, we report a preliminary experimental evaluation of the proposed system, in which an operator will teach a 7-degrees-of-freedom manipulator the execution of a task on a robotic wiring test-bed. In the experiment, the programming of requested compliance levels during the kinesthetic teaching of a trajectory is performed, and the reported results show that the provided wearable interface is successfully exploited by the operator. Finally, the experiment also demonstrates that the offline intuitive programming of trajectories and impedance levels can be exploited online for human-robot co-work.

IRoPro: An interactive Robot Programming Framework

The great diversity of end-user tasks ranging from manufacturing environments to personal homes makes pre-programming robots for general purpose applications extremely challenging. In fact, teaching robots new actions from scratch that can be reused for previously unseen tasks remains a difficult challenge and is generally left up to robotics experts. In this work, we present iRoPro, an interactive Robot Programming framework that allows end-users with little to no technical background to teach a robot new reusable actions. We combine Programming by Demonstration and Automated Planning techniques to allow the user to construct the robot's knowledge base by teaching new actions by kinesthetic demonstration. The actions are generalised and reused with a task planner to solve previously unseen problems defined by the user. We implement iRoPro as an end-to-end system on a Baxter Research Robot to simultaneously teach low- and high-level actions by demonstration that the user can customise via a Graphical User Interface to adapt to their specific use case. To evaluate the feasibility of our approach, we first conducted pre-design experiments to better understand the user's adoption of involved concepts and the proposed robot programming process. We compare results with post-design experiments, where we conducted a user study to validate the usability of our approach with real end-users. Overall, we showed that users with different programming levels and educational backgrounds can easily learn and use iRoPro and its robot programming process.

A digital twin-based frame work for task planning and robot programming in HRC

This paper presents a digital twin-based framework for complex assembly tasks in human-robot collaborative assembly system. In this research, the system is developed to support the planning, decision, and implementation of human-robot collaboration. Virtual simulation is applied from task planning to robot control, including assembly task allocation and scheduling, robot trajectory planning and programming. Online optimization is conducted for real-time control based the status of operators and robots from the real production system. An industrial case is presented to validate the performance of the proposed digital twin framework and collaborative assembly system.

A framework for building mobile single and multi-robot applications

The complexity of robot software systems calls for the use of a well-conceived architecture together with programming tools to support it. One common feature of robot architectures is the modular decomposition of systems into simpler and largely independent components. These components implement primitive actions and report events about their state. The robot programming framework proposed here includes a tool (RoboGraph) to program and coordinate the activity (tasks) of these middleware modules. Project developers use the same task programming IDE (RoboGraph) on two different levels. The first is to program tasks that must be executed autonomously by one robot and the second is to program tasks that can include several robots and building elements. Tasks are described using a Signal Interpreted Petri Net (SIPN) editor and stored in an xml file. A dispatcher loads these files and executes the different Petri nets as needed. A monitor that shows the state of all the running nets is very useful for debugging and tracing purposes. The whole system has been used in several applications: A tour-guide robot (GuideBot), a multi-robot surveillance project (WatchBot) and a hospital food and laundry transportation system based on mobile robots.

Structures of visual servos

The paper classifies visual servos and using this classification derives their possible structures. It also analyzes the influence of robot–model calibration and robot–camera calibration on the derived control structures. The systematic derivation of possible structures of visual servos enabled the creation of software patterns within the MRROC++ (Zieliński et al., 2007) [29] robot programming framework facilitating their implementation. Those considerations are preceded by a brief introduction defining the principal concepts used in the paper. The paper is concluded by presenting the use of the derived structures in the experiments with grasping static and tracking moving objects. The paper highlights the advantages and drawbacks of the presented structures.

Motion Generation in the MRROC++ Robot Programming Framework

In this paper we present a formal approach to robot motion specification. This motion specification takes into account three elementary behaviors that suffice to define any robot interaction with the environment, i.e. free motion, exerting generalized forces and the transition between both of these behaviors. These behaviors provide a foundation for general motion generation taking into account any sensors, any effectors and the capability to exchange information between embodied agents. This specification can be used both for the definition of robot tasks and implementation of robot control software, hence both of those aspects are presented in this paper. This formal approach was used for the implementation of the MRROC++ robot programming framework. Two-handed manipulation of a Rubik’s cube is used as an exemplary task. Extensive experimentation both with the presented formalism and the MRROC++ framework showed that the imposed formal rigor eliminates many errors at the software specification phase, produces well-structured control software and significantly speeds up and simplifies its implementation. These advantages are mainly due to the fact that the proposed formal specification tool is derived from operational semantics used in computer science for the definition of programming languages, thus a close relationship between abstract definition and the implementation of the control system resulted.

Rubik's cube as a benchmark validating MRROC++ as an implementation tool for service robot control systems

PurposeThis paper seeks to develop universal software (a programming framework) enabling the implementation of service robot controllers. The software should distinguish the hardware‐oriented part of the system from the task‐oriented one. Moreover, force, vision as well as other sensors should be taken into account. Multi‐effector systems have to be considered.Design/methodology/approachThe robot programming framework MRROC++ has been implemented as a hierarchical structure composed of processes, potentially consisting of threads. All of the software is written in an object‐oriented manner using C++ and is supervised by a QNX real‐time operating system. The framework has been verified on several systems executing diverse tasks. Here, a Rubik's cube puzzle‐solving system, consisting of two arms and utilizing force control and visual servos, is presented.FindingsThe presented framework is well suited to tasks requiring two‐handed manipulation with force sensing, visual servoing and online construction of plans of actions. The Rubik's cube puzzle is a reasonable initial benchmark for validation of fundamental service robot capabilities. It requires force sensing and sight coupled with two‐handed manipulation and logical reasoning, as do the majority of service tasks. Owing to the use of force sensing during manipulation, jamming of the faces has always been avoided; however, visual servoing could only cope with slow handing over of the cube due to the volume of computations associated with vision processing.Research limitations/implicationsThe proposed software structure does not limit the implementation of service robot controllers. However, some of the specific algorithms used for the solution of the benchmark task (i.e. Rubik's cube puzzle) need to be less time‐consuming.Practical implicationsThe MRROC++ robot programming framework can be applied to the implementation of diverse robot controllers executing complex service tasks.Originality/valueA demanding benchmark task for service robots has been formulated. This task, as well as many others, has been used to validate the MRROC++ robot programming framework which significantly facilitates the implementation of diverse robot systems.

MERMAID – MULTIPLE-ROBOT MIDDLEWARE FOR INTELLIGENT DECISION-MAKING

This paper describes the basic concepts and features of MeRMaID (Multiple-Robot Middleware for Intelligent Decision-making), a robot programming framework whose goal is to provide a simplified and systematic high-level behavior programming environment for multi-robot teams. MeRMaID constrains, on purpose, some of the programmer's options, and can accept plans described by state machines, Petri nets and other types of decision-making algorithms, including fuzzy-logic decision-making and rule-based systems. Its current version enables concurrent multi-platform programming, modularity (for flexible module replacement and easy module edition/modification), and independence from robot hardware (since it includes an Hardware Abstraction Layer).

A prototype robot for polishing and milling large objects

The paper describes a prototype robot which due to its serial‐parallel structure exhibits, high stiffness and has a large work envelope. These features make this robot suitable for relatively high precision machining operations on large workpieces. The conroller for this robot was based on MRROC++, which is a robot programming framework. Thus the controller could be tailored to the tasks at hand, including the capability of in‐program switching of kinematic model parameters. To obtain those parameters for different locations in the work‐space a calibration procedure using linear measurement guides has been devised.

Miro: Middleware for Autonomous Mobile Robots

Implementing software for autonomous mobile robots is a non-trivial task, because such robots incorporate several sensor systems and actuators that must be controlled simultaneously by a heterogeneous ensemble of networked computers and microcontrollers. Additionally, the use of modern software engineering technologies like object-oriented and distributed programming and client/server architectures is essential in order to maintain program code effectively. In this paper, we present Miro, a new CORBA-based robot programming framework which allows a rapid development of reliable and safe software on heterogeneous computer networks and supports the mixed use of several programming languages.