Robot Task Execution Research Articles

Autonomous Robot Task Execution in Flexible Manufacturing: Integrating PDDL and Behavior Trees in ARIAC 2023.

The Agile Robotics for Industrial Automation Competition (ARIAC) was established to advance flexible manufacturing, aiming to increase the agility of robotic assembly systems in unstructured and dynamic industrial environments. ARIAC 2023 introduced eight agility challenges involving faulty parts, flipped parts, faulty grippers, robot malfunctions, sensor blackouts, high-priority orders, insufficient parts, and human safety. Given the unpredictability of these scenarios, it is impractical to develop a specific strategy for each possible situation. To address these issues, this paper presents a hierarchical framework for autonomous robotic task generation and execution in dynamic scenarios. The framework is divided into a task level and an execution level. Initially, an immediate task management strategy is adopted at the task level, which reasonably decomposes dynamic tasks and allocates short-term tasks to the floor robot and ceiling robot. Later, at the execution level, each robot is designed with an agent architecture that combines PDDL planning with the quick response of behavior trees. Finally, the effectiveness and practicality of the proposed framework were thoroughly validated in ARIAC 2023.

Proposing a Computational Modeling Framework for Generating Masonry Wall Units, Enhancing the Information Within a BIM

In recent decades, the construction industry has undergone a technological shift incorporating innovative technologies, such as robotics. However, information requirements must be met to integrate robotics further. Currently, building information models (BIM) contain substantial project information that can be leveraged for robots to create construction tasks, but for some building systems, the level of development (LOD) is inadequate to support these new requirements. Therefore, this study proposes a framework to increase the LOD of building systems by considering location information (X, Y, Z), orientation, material type, and component I.D. The computational modeler, Dynamo, is leveraged to increase the model’s LOD, extract information, and facilitate robotic task execution in the future. A case study is presented for multiple masonry room configurations developed in Autodesk Revit, where masonry units are generated and placed into design locations based on the geometry of the wall system. The case study used concrete masonry units (CMU) and standard brick. The number of partial-sized and full-sized blocks for each configuration was recorded, along with the computational time required to generate the units. It was observed that room configurations with more openings had longer computational times when compared to rooms constructed from the same material. After running the script, the model is reviewed to ensure accuracy and prevent overlaps or gaps in the model. The workflow provides insight into the methods used to interpret model geometry and extract information.

A novel redundant cooperative control strategy for robotic pollination

This paper proposes a novel Redundant Cooperative Control (RCC) strategy to address the pollination sequence planning problem for a mobile pollination robot in the greenhouse. Firstly, the RCC strategy divides the robot’s workspace into subspaces based on the distribution of flowers through clustering. Subsequently, the RCC strategy plans the traveling salesman problem within the visual field of these subspaces, seeking a compromise solution for an optimal path. Finally, the RCC strategy utilizes the redundant motion capabilities of the mobile chassis and robotic arm to accomplish the pollination task during the movement. The RCC strategy exploits the available redundancy in both the agricultural robot’s mobile platform and end-effector, carefully balancing computational complexity and path quality. To validate the effectiveness of the RCC strategy, we designed a series of experiments conducted on our pollination robot platform, measuring different strategies. The results demonstrate that the RCC strategy achieves an average pollination rate of 7.5 s per flower, representing a 36.4% improvement compared to the most common intermittence strategy. Furthermore, the RCC strategy can be extended to tasks such as harvesting and pruning, significantly enhancing the efficiency of robot task execution even in industrial and service sectors.

Enhancing Robot Task Planning and Execution through Multi-Layer Large Language Models.

Large language models have found utility in the domain of robot task planning and task decomposition. Nevertheless, the direct application of these models for instructing robots in task execution is not without its challenges. Limitations arise in handling more intricate tasks, encountering difficulties in effective interaction with the environment, and facing constraints in the practical executability of machine control instructions directly generated by such models. In response to these challenges, this research advocates for the implementation of a multi-layer large language model to augment a robot's proficiency in handling complex tasks. The proposed model facilitates a meticulous layer-by-layer decomposition of tasks through the integration of multiple large language models, with the overarching goal of enhancing the accuracy of task planning. Within the task decomposition process, a visual language model is introduced as a sensor for environment perception. The outcomes of this perception process are subsequently assimilated into the large language model, thereby amalgamating the task objectives with environmental information. This integration, in turn, results in the generation of robot motion planning tailored to the specific characteristics of the current environment. Furthermore, to enhance the executability of task planning outputs from the large language model, a semantic alignment method is introduced. This method aligns task planning descriptions with the functional requirements of robot motion, thereby refining the overall compatibility and coherence of the generated instructions. To validate the efficacy of the proposed approach, an experimental platform is established utilizing an intelligent unmanned vehicle. This platform serves as a means to empirically verify the proficiency of the multi-layer large language model in addressing the intricate challenges associated with both robot task planning and execution.

Robot task programming in complex task scenarios based on spatio-temporal constraint calculus

PurposeRobots face fundamental challenges in achieving reliable and stable operations for complex home service scenarios. This is one of the crucial topics of robotics methods to imitate human beings’ advanced cognitive characteristics and apply them to solve complex tasks. The purpose of this study is to enable robots to have the ability to understand the scene and task process in complex scenes and to provide a reference method for robot task programming in complex scenes.Design/methodology/approachThis paper constructs a task modeling method for robots in complex environments based on the characteristics of the perception-motor memory model of human cognition. In the aspect of episodic memory construction, the task execution process is included in the category of qualitative spatio-temporal calculus. The topology interaction of objects in a task scenario is used to define scene attributes. The task process can be regarded as changing scene attributes on a time scale. The qualitative spatio-temporal activity graphs are used to analyze the change process of the object state with time during the robot task execution. The tasks are divided according to the different values of scene attributes at different times during task execution. Based on this, in procedural memory, an object-centered motion model is developed by analyzing the changes in the relationship between objects in the scene episode by analyzing the scene changes before and after the robot performs the actions. Finally, the task execution process of the robot is constructed by alternately reconstructing episodic memory and procedural memory.FindingsTo verify the applicability of the proposed model, a scenario where the robot combines the object (one of the most common tasks in-home service) is set up. The proposed method can obtain the landscape of robot tasks in a complex environment.Originality/valueThe robot can achieve high-level task programming through the alternating interpretation of scenarios and actions. The proposed model differs from traditional methods based on geometric or physical feature information. However, it focuses on the spatial relationship of objects, which is more similar to the cognitive mechanism of human understanding of the environment.

Robot Symbolic Motion Planning and Task Execution Based on Mixed Reality Operation

Robot Symbolic Motion Planning and Task Execution Based on Mixed Reality Operation

Reactive Task Adaptation of a Dynamic System With External Disturbances Based on Invariance Control and Movement Primitives

This article proposes a general control framework for robot physical contact tasks. To satisfy the tradeoff between safety and performance in the interaction between robots and the environment, multipriority control is regarded as a preemptive strategy for constraint management when unknown disturbances exist. The equality and inequality constraints related to robot safety are enforced by invariance control, while reference profile tracking is accommodated by dynamic movement primitives without violating the higher priority constraints. With this framework, we complete the unification of force control and motion control in robot task execution. The robot thus acquires the capability for strong disturbance rejection and transferrable intelligence between similar tasks. At the same time, a variant linear–quadratic regulator (LQR) is integrated into the framework, which enables the robot to achieve exponential convergence of the tracking error. The proposed approach is tested and evaluated with two types of physical contact tasks, showing a superior control effect and faster convergence than the existing methods.

BIM-based semantic building world modeling for robot task planning and execution in built environments

From its internal representation of a building, or, its building world, a robot can retrieve a building's information for task planning and execution. However, a building is a complex construct composed of thousands of sub-elements, therefore manually programming the complete lifecycle information in a robot can be impractical. Our study presents the creation of a semantic building world from a building information model (BIM) that can be used for robot operations. To describe static and dynamic elements, we create a Universal Robot Description Format (URDF) building world using the Industry Foundation Classes (IFC) that a robot can directly query information for task planning and execution. As the case study demonstrates, our study bridges the gap between BIM's lifecycle information and robot operations. By allowing robots to acquire information about operating environments, the proposed methodologies provide the foundation for studies attempting task robotization in different types of facilities throughout their lifecycles.

OCRA – An ontology for collaborative robotics and adaptation

Industrial collaborative robots will be used in unstructured scenarios and a large variety of tasks in the near future. These robots shall collaborate with humans, who will add uncertainty and safety constraints to the execution of industrial robotic tasks. Hence, trustworthy collaborative robots must be able to reason about their collaboration’s requirements (e.g., safety), as well as the adaptation of their plans due to unexpected situations. A common approach to reasoning is to represent the knowledge of interest using logic-based formalisms, such as ontologies. However, there is not an established ontology defining notions such as collaboration or adaptation yet. In this article, we propose an Ontology for Collaborative Robotics and Adaptation (OCRA), which is built around two main notions: collaboration, and plan adaptation. OCRA ensures a reliable human-robot collaboration, since robots can formalize, and reason about their plan adaptations and collaborations in unstructured collaborative robotic scenarios. Furthermore, our ontology enhances the reusability of the domain’s terminology, allowing robots to represent their knowledge about different collaborative and adaptive situations. We validate our formal model, first, by demonstrating that a robot may answer a set of competency questions using OCRA. Second, by studying the formalization’s performance in limit cases that include instances with incongruent and incomplete axioms. For both validations, the example use case consists in a human and a robot collaborating on the filling of a tray.

Hybrid offline and online task planning for service robot using object-level semantic map and probabilistic inference

To make service robots plan a whole task execution sequence and deal with task failure intelligently in the face of object occlusion and incomplete home environment information, a hybrid offline and online task planning strategy is proposed. We establish object-level semantic map and object location probabilistic relations between dynamic and static objects. Semantic mapping helps to obtain semantic locations of static objects, and the probabilistic relationship between dynamic and static objects can obtain semantic locations of dynamic objects through probabilistic reasoning. Probabilistic planning domain definition language (PPDDL) can generate offline action sequences, while partially observable Markov decision process (POMDP) can generate online action sequences. Therefore, a hybrid task planner that can receive semantic location information is constructed to generate offline and online action sequences, and realizes the dynamic switching of the two kinds of sequences through the designed planner switching mechanism. In order to improve the robustness and intelligence of robot task planning, a task replanning mechanism considering the position relationship between robot and candidate static objects is designed. Experimental results in real environment and simulation environment show that this approach can effectively increase the intelligence of task planning and improve the robustness and efficiency of robot task execution.

Hybrid Path Planning Model for Multiple Robots Considering Obstacle Avoidance

To solve the problem that the single robot task execution capability is not enough to meet the whole handling task demand under complex conditions, the hybrid path planning models such as multi-robot path planning and formation cooperative control considering obstacle avoidance are studied. Firstly, for the robot global path finding problem, on the basis of the construction for a robot working environment model based on the geometric map model building method, an improved particle swarm algorithm-based global path planning model is proposed to solve the problems of low robot path planning solution efficiency and easy to fall into local optimal solutions. Secondly, for the multi-robot cooperative formation control and obstacle avoidance and inter-robot collision avoidance problems, a multi-robot formation local path planning model based on the improved artificial potential field method is constructed, a simulated annealing algorithm is introduced to optimize the traditional artificial potential field method, and a multi-robot formation control strategy, robot obstacle avoidance, and inter-robot collision avoidance methods are designed in combination with the pilot-following method to improve the robot formation path exploration The proposed method can improve the path exploration capability and handling efficiency of robot formation. Finally, the global path planning model of the robot based on the improved particle swarm algorithm is simulated and analyzed using Matlab 7.0 to verify the outstanding performance of the model in pathfinding capability. Then the local path planning model of multi-robot formation based on the improved artificial potential field is simulated and analyzed to verify the improved algorithm has good path planning as well as obstacle avoidance performance. The hybrid path planning model is applied to a real case and simulated, and the results show that the improved algorithm improves the exploration capability of the robot formation, effectively avoids obstacles, and verifies its reliability and superiority in the hybrid path planning process.

Actions Selection During a Mobile Robot Navigation for the Autonomous Recharging Problem

The use of mobile robots has increased for its application in various areas such as supply chains, factories, cleaning, disinfection, medical assistance, search, and exploration. It is a fact that most of these robots, if not all, use batteries to power themselves. During a mobile robot task execution, the battery's electric charge tends to deplete as a function of the energy load demands, which would cause the robot to shut down if the discharge is critical, leaving its task inconclusive. Therefore, it is of utmost importance that the robot learns when to charge its batteries, avoiding turning off. This work shows a reactive navigation scheme for a mobile robot that integrates a module for battery-level monitoring. A robot moves from a starting point to a destination according to the battery level. During the navigation, the robot decides when to change the course toward a battery charging station. This paper presents a rules-based reinforcement learning architecture with three entries; these entries correspond to the robot's battery level, the distance to the destination, and the distance to the battery charging station. According to the simulations, the robot learns to select an appropriate action to accomplish its task.

Status Recognition Using Pre-Trained YOLOv5 for Sustainable Human-Robot Collaboration (HRC) System in Mold Assembly

Molds are still assembled manually because of frequent demand changes and the requirement for comprehensive knowledge related to their high flexibility and adaptability in operation. We propose the application of human-robot collaboration (HRC) systems to improve manual mold assembly. In the existing HRC systems, humans control the execution of robot tasks, and this causes delays in the operation. Therefore, we propose a status recognition system to enable the early execution of robot tasks without human control during the HRC mold assembly operation. First, we decompose the mold assembly operation into task and sub-tasks, and define the actions representing the status of sub-tasks. Second, we develop status recognition based on parts, tools, and actions using a pre-trained YOLOv5 model, a one-stage object detection model. We compared four YOLOv5 models with and without a freezing backbone. The YOLOv5l model without a freezing backbone gave the optimal performance with a mean average precision (mAP) value of 84.8% and an inference time of 0.271 s. Given the success of the status recognition, we simulated the mold assembly operations in the HRC environment and reduced the assembly time by 7.84%. This study improves the sustainability of the mold assembly from the point of view of human safety, with reductions in human workload and assembly time.

ViMantic, a distributed robotic architecture for semantic mapping in indoor environments

Semantic maps augment traditional representations of robot workspaces, typically based on their geometry and/or topology, with meta-information about the properties, relations and functionalities of their composing elements. A piece of such information could be: fridges are appliances typically found in kitchens and employed to keep food in good condition. Thereby, semantic maps allow for the execution of high-level robotic tasks in an efficient way, e.g. “Hey robot, Store the leftover salad”. This paper presents ViMantic, a novel semantic mapping architecture for the building and maintenance of such maps, which brings together a number of features as demanded by modern mobile robotic systems, including: (i) a formal model, based on ontologies, which defines the semantics of the problem at hand and establishes mechanisms for its manipulation; (ii) techniques for processing sensory information and automatically populating maps with, for example, objects detected by cutting-edge CNNs; (iii) distributed execution capabilities through a client–server design, making the knowledge in the maps accessible and extendable to other robots/agents; (iv) a user interface that allows for the visualization and interaction with relevant parts of the maps through a virtual environment; (v) public availability, hence being ready to use in robotic platforms. The suitability of ViMantic has been assessed using Robot@Home, a vast repository of data collected by a robot in different houses. The experiments carried out consider different scenarios with one or multiple robots, from where we have extracted satisfactory results regarding automatic population, execution times, and required size in memory of the resultant semantic maps.

Building the Foundation of Robot Explanation Generation Using Behavior Trees

As autonomous robots continue to be deployed near people, robots need to be able to explain their actions. In this article, we focus on organizing and representing complex tasks in a way that makes them readily explainable. Many actions consist of sub-actions, each of which may have several sub-actions of their own, and the robot must be able to represent these complex actions before it can explain them. To generate explanations for robot behavior, we propose using Behavior Trees (BTs), which are a powerful and rich tool for robot task specification and execution. However, for BTs to be used for robot explanations, their free-form, static structure must be adapted. In this work, we add structure to previously free-form BTs by framing them as a set of semantic sets {goal, subgoals, steps, actions} and subsequently build explanation generation algorithms that answer questions seeking causal information about robot behavior. We make BTs less static with an algorithm that inserts a subgoal that satisfies all dependencies. We evaluate our BTs for robot explanation generation in two domains: a kitting task to assemble a gearbox, and a taxi simulation. Code for the behavior trees (in XML) and all the algorithms is available at github.com/uml-robotics/robot-explanation-BTs.

Compensation for Undefined Behaviors During Robot Task Execution by Switching Controllers Depending on Embedded Dynamics in RNN

Robotic applications require both correct task performance and compensation for undefined behaviors. Although deep learning is a promising approach to perform complex tasks, the response to undefined behaviors that are not reflected in the training dataset remains challenging. In a human-robot collaborative task, the robot may adopt an unexpected posture due to collisions and other unexpected events. Therefore, robots should be able to recover from disturbances for completing the execution of the intended task. We propose a compensation method for undefined behaviors by switching between two controllers. Specifically, the proposed method switches between learning-based and model-based controllers depending on the internal representation of a recurrent neural network that learns task dynamics. We applied the proposed method to a pick-and-place task and evaluated the compensation for undefined behaviors. Experimental results from simulations and on a real robot demonstrate the effectiveness and high performance of the proposed method.

A Multimodal Approach to Human Safety in Collaborative Robotic Workcells

This article investigates the problem of controlling the speed of robots in collaborative workcells for automated manufacturing. The solution is tailored to robotic cells for cooperative assembly of aircraft fuselage panels, where only structural elements are present and robots and humans can share the same workspace, but no physical contact is allowed, unless it happens at zero robot speed. The proposed approach addresses the problem of satisfying the minimal set of requirements of an industrial human–robot collaboration (HRC) task: precision and reliability of human detection and tracking in the shared workspace; correct robot task execution with minimum cycle time while assuring safety for human operators. These requirements are often conflicting with each other. The former does not only concern with safety only but also with the need of avoiding unnecessary robot stops or slowdowns in case of false-positive human detection. The latter, according to the current regulations, concerns with the need of computing the minimum protective separation distance between the human operator and the robots by adjusting their speed when dangerous situations happen. This article proposes a novel fuzzy inference approach to control robot speed enforcing safety while maximizing the level of productivity of the robot minimizing cycle time as well. The approach is supported by a sensor fusion algorithm that merges the images acquired from different depth sensors with those obtained from a thermal camera, by using a machine learning approach. The methodology is experimentally validated in two experiments: the first one at a lab-scale and the second one performed on a full-scale robotic workcell for cooperative assembly of aeronautical structural parts. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article discusses a way to handle human safety specifications versus production requirements in collaborative robotized assembly systems. State-of-the-art (SoA) approaches cover only a few aspects of both human detection and robot speed scaling. The present research work proposes a complete pipeline that starts from a robust human tracking algorithm and scales the robot speed in real time. An innovative multimodal perception system composed of two depth cameras and a thermal camera monitors the collaborative workspace. The speed scaling algorithm is optimized to take on different human behaviors during less risky situations or more dangerous ones to guarantee both operator safety and minimum production time with the aim of better profitability and efficiency for collaborative workstations. The algorithm estimates the operator intention for real-time computation of the minimum protective distance according to the current safety regulations. The robot speed is smoothly changed for the psychological advantages of operators, both in the case of single and multiple workers. The result is a complete system, easily implementable on a standard industrial workcell.

Quantifying Hypothesis Space Misspecification in Learning From Human–Robot Demonstrations and Physical Corrections

Human input has enabled autonomous systems to improve their capabilities and achieve complex behaviors that are otherwise challenging to generate automatically. Recent work focuses on how robots can use such input - like demonstrations or corrections - to learn intended objectives. These techniques assume that the human's desired objective already exists within the robot's hypothesis space. In reality, this assumption is often inaccurate: there will always be situations where the person might care about aspects of the task that the robot does not know about. Without this knowledge, the robot cannot infer the correct objective. Hence, when the robot's hypothesis space is misspecified, even methods that keep track of uncertainty over the objective fail because they reason about which hypothesis might be correct, and not whether any of the hypotheses are correct. In this paper, we posit that the robot should reason explicitly about how well it can explain human inputs given its hypothesis space and use that situational confidence to inform how it should incorporate human input. We demonstrate our method on a 7 degree-of-freedom robot manipulator in learning from two important types of human input: demonstrations of manipulation tasks, and physical corrections during the robot's task execution.

Probabilistic planning with formal performance guarantees for mobile service robots

We present a framework for mobile service robot task planning and execution, based on the use of probabilistic verification techniques for the generation of optimal policies with attached formal performance guarantees. Our approach is based on a Markov decision process model of the robot in its environment, encompassing a topological map where nodes represent relevant locations in the environment, and a range of tasks that can be executed in different locations. The navigation in the topological map is modeled stochastically for a specific time of day. This is done by using spatio-temporal models that provide, for a given time of day, the probability of successfully navigating between two topological nodes, and the expected time to do so. We then present a methodology to generate cost optimal policies for tasks specified in co-safe linear temporal logic. Our key contribution is to address scenarios in which the task may not be achievable with probability one. We introduce a task progression function and present an approach to generate policies that are formally guaranteed to, in decreasing order of priority: maximize the probability of finishing the task; maximize progress towards completion, if this is not possible; and minimize the expected time or cost required. We illustrate and evaluate our approach with a scalability evaluation in a simulated scenario, and report on its implementation in a robot performing service tasks in an office environment for long periods of time.

Shadow Space Modeling and Task Planning for Collision-free Cooperation of Dual Manipulators for Planar Task

Planning the manipulators’ task sequences is necessary for proper execution of specific robotic tasks. With multiple manipulators for cooperative tasks, planning becomes difficult owing to possible collisions between component manipulators. This paper proposes a shadow space approach that utilizes the swept area of a robot during its motion, for collision-free task planning of dual manipulators. A collision can be detected and prevented by computing the intersection of two manipulators’ shadow spaces. Then, a novel genetic algorithm with mutually exclusive chromosomes is proposed to search optimal collision-free task sequences for dual manipulators. For a pragmatic example of placing pemnuts onto the back-chassis of a liquid crystal device (LCD) panel, experiments and simulation show that the proposed method can determine a nearly optimal sequence, which allows two manipulators to move cooperatively without collisions, even though they share a common workspace.