Autonomous Mobile Manipulator Research Articles

Digital Twin and Deep Reinforcement Learning-Driven Robotic Automation System for Confined Workspaces: A Nozzle Dam Replacement Case Study in Nuclear Power Plants

Robotic automation has emerged as a leading solution for replacing human workers in dirty, dangerous, and demanding industries to ensure the safety of human workers. However, practical implementation of this technology remains limited, requiring substantial effort and costs. This study addresses the challenges specific to nuclear power plants, characterized by hazardous environments and physically demanding tasks such as nozzle dam replacement in confined workspaces. We propose a digital twin and deep-reinforcement-learning-driven robotic automation system with an autonomous mobile manipulator. The study follows a four-step process. First, we establish a simplified testbed for a nozzle dam replacement task and implement a high-fidelity digital twin model of the real-world testbed. Second, we employ a hybrid visual perception system that combines deep object pose estimation and an iterative closest point algorithm to enhance the accuracy of the six-dimensional pose estimation. Third, we use a deep-reinforcement-learning method, particularly the proximal policy optimization algorithm with inverse reachability map, and a centroidal waypoint strategy, to improve the controllability of an autonomous mobile manipulator. Finally, we conduct pre-performed simulations of the nozzle dam replacement in the digital twin and evaluate the system on a robot in the real-world testbed. The nozzle dam replacement with precise object pose estimation, navigation, target object grasping, and collision-free motion generation was successful. The robotic automation system achieved a 92.0%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$92.0\\%$$\\end{document} success rate in the digital twin. Our proposed method can improve the efficiency and reliability of robotic automation systems for extreme workspaces and other perilous environments.

Whole Body Control of an Autonomous Mobile Manipulator Using Series Elastic Actuators

The flexible joint robot has superior attractive features because of its high mobility, high load ratiohigh torque fidelity, robustness for external disturbance, task adaptability, and safety. In this paper, an autonomous mobile manipulator driven by the designed series elastic actuators (SEAs) is developed. A complete dynamic model of nonholonomic mobile manipulator including a mobile platform and manipulator with joint flexibility operating simultaneously is proposed. An integrated whole body trajectory control framework is proposed for such robot to perform mobile manipulation tasks. Considering the nonholonomic and holonomic constraints in the mobile manipulation, the whole-body dynamics is formulated and reduced. To address the highly nonlinear of the dynamics and model uncertainty, a novel integral Lyapunov function (ILF)-based adaptive neural network control for task tracking under uncertainties of the flexible joint robot model is proposed. Compared with existing methods, the proposed method provides an alternative for controlling flexible joint robots. The feasibility of the proposed method is verified by the extensive trajectory tracking experiment results in our developed flexible joint manipulator.

A Tree-Based World Model for Reducing System Complexity in Autonomous Mobile Manipulation

A Tree-Based World Model for Reducing System Complexity in Autonomous Mobile Manipulation

A framework to integrate mobile manipulators as cyber–physical systems into existing production systems in the context of industry 4.0

With the emergence of the Cyber–Physical Systems (CPSs) along with the fourth industrial revolution, the discussion of distributed architecture and coordination and extensive communication between all components of a system was raised. On the other hand, Modern industrial manufacturing relies heavily on flexible production. Autonomous mobile manipulators (MMs) have the potential to improve the flexibility of existing manufacturing environments. With the advantages associated with this technology, there is an upsurge in the demand for MMs. Thus, adding a new MM as a new CPS into a system following industry 4.0 requires a framework to help taking all the advantages from it. In this article, we proposed a framework to integrating MMs as CPSs into an existing production system. This integration requires comprehensive knowledge of the existing production process and control, which is a view to improve the performance of the existing production systems via new functionalities due to MMs. The next step of the framework is to make new MM operations compatible with the existing production control system. For this purpose, we define different parametric blocks to perform new operations and thus contribute to making the production line more autonomous. Considering all of these issues simultaneously is necessary for an efficient integration in smart production context. For a successful integration, we presented four principal dimensions of integration: Physical integration, communicational integration, functional and operational integration, and digital integration. Furthermore, the framework has been applied to a research production line called Platform 4.0 at Arts et Métiers. An MM from OMRON Company (called MoMa) has been integrated into the existing production system. Results from this real-world demonstration show that MoMa is capable of successfully increasing flexibility, autonomy, and efficiency of a production system using command signals from Manufacturing Execution System (MES).

A speech-enabled virtual assistant for efficient human–robot interaction in industrial environments

This paper presents a natural language-enabled virtual assistant (VA), named Max, developed to support flexible and scalable human–robot interactions (HRI) with industrial robots. Regardless of the numerous natural language interfaces already proposed for intuitive HRI on the industrial shop floor, most of those interfaces remain tightly bound with a specific robotic system. Besides, the lack of a natural and efficient human–robot communication protocol hinders the user experience. Therefore three key elements characterize the proposed framework. First, a Client–Server style architecture is introduced so Max can provide a centralized solution for managing and controlling various types of robots deployed on the shop floor. Second, inspired by human–human communication, two conversation strategies, lexical-semantic and general diversion strategies, are used to guide Max’s response generation. These conversation strategies were embedded to improve the operator’s engagement with the manufacturing tasks. Third, we fine-tuned the state-of-the-art (SOTA) pre-trained model, Bidirectional Encoder Representations from Transformers (BERT), to support a highly accurate prediction of requested intents from the operator and robot services. Multiple experiments were conducted using the latest iteration of our autonomous industrial mobile manipulator, “Little Helper (LH)”, to validate Max’s performance in a real manufacturing environment.

A Review on the Prospects of Mobile Manipulators for Smart Maintenance of Railway Track

Inspection and repair interventions play vital roles in the asset management of railways. Autonomous mobile manipulators possess considerable potential to replace humans in many hazardous railway track maintenance tasks with high efficiency. This paper investigates the prospects of the use of mobile manipulators in track maintenance tasks. The current state of railway track inspection and repair technologies is initially reviewed, revealing that very few mobile manipulators are in the railways. Of note, the technologies are analytically scrutinized to ascertain advantages, unique capabilities, and potential use in the deployment of mobile manipulators for inspection and repair tasks across various industries. Most mobile manipulators in maintenance use ground robots, while other applications use aerial, underwater, or space robots. Power transmission lines, the nuclear industry, and space are the most extensive application areas. Clearly, the railways infrastructure managers can benefit from the adaptation of best practices from these diversified designs and their broad deployment, leading to enhanced human safety and optimized asset digitalization. A case study is presented to show the potential use of mobile manipulators in railway track maintenance tasks. Moreover, the benefits of the mobile manipulator are discussed based on previous research. Finally, challenges and requirements are reviewed to provide insights into future research.

Whole-Body Control of an Autonomous Mobile Manipulator Using Model Predictive Control and Adaptive Fuzzy Technique

Whole-body control (WBC) has emerged as an important framework in manipulation for mobile manipulators. However, most existing WBC frameworks require known dynamics. Considering whole-body manipulation and optimization with unknown dynamics, this article presents the WBC of a nonholonomic mobile manipulator using model predictive control (MPC) and fuzzy logic system. First, by constructing a dynamics-based feedback linearized robotic multi-input-multi-output (MIMO) system, an MPC-based WBC strategy is proposed for mobile manipulator. Such a strategy can provide the optimal control inputs with the specified optimization index and constraints. Thereafter, a primal-dual neural network effectively addresses the constrained quadratic programming (QP) problem over a finite receding horizon brought by the MPC. Then, in order to convert the intermediate control signals into the optimal control torques that can be executed by actuators, an adaptive FLS is employed to approximate the unknown dynamics. The novel elements of the current design control approach refer to the dynamics-based feedback linearized robotic MIMO system and the combination of an MPC module with an adaptive fuzzy controller. Finally, the trajectory tracking experiments performed on a mobile dual-arm robot demonstrate the effectiveness of the proposed method.

Human-Multirobot Collaborative Mobile Manipulation: The Omnid Mocobots

The Omnid human-collaborative mobile manipulators are an experimental platform for testing control architectures for autonomous and human-collaborative multirobot mobile manipulation. An Omnid consists of a mecanum-wheel omnidirectional mobile base and a series-elastic Delta-type parallel manipulator, and it is a specific implementation of a broader class of mobile collaborative robots ("mocobots") suitable for safe human co-manipulation of delicate, flexible, and articulated payloads. Key features of mocobots include passive compliance, for the safety of the human and the payload, and high-fidelity end-effector force control independent of the potentially imprecise motions of the mobile base. We describe general considerations for the design of teams of mocobots; the design of the Omnids in light of these considerations; manipulator and mobile base controllers to achieve useful multirobot collaborative behaviors; and initial experiments in human-multirobot collaborative mobile manipulation of large, unwieldy payloads. For these experiments, the only communication among the humans and Omnids is mechanical, through the payload.

Integrated Task Sensing and Whole Body Control for Mobile Manipulation With Series Elastic Actuators

In this paper, an integrated framework consisting of the sensing, navigation and control is proposed for an autonomous mobile manipulator driven by series elastic actuators (SEAs) to preform mobile manipulation tasks in unknown environments. First, ORB-SLAM2 technique is combined into the environment sensing by extracting the ORB features, automatic initialization, repositioning and loop detection for real-time posture estimation. Then, the navigation function is designed for generating collision-free trajectory in an environment with obstacles. To realize kinematic and dynamic control of the mobile manipulator with the developed SEA joints, the whole body dynamics is considered and described. And to handle dynamic uncertainties and the SEA inherent saturation limits, a novel adaptive neural network control considering the whole body dynamics is proposed. Without knowing the exact parameters of the whole body model, the designed controller merely requires the position and velocity of the actuators and links, which can make the tracking errors converge to zero and keep all signals uniformly bounded in the closed-loop system. The performance and efficiency of the proposed method are verified by extensive experiments. Note to Practitioners—This paper is motivated by issues of manipulation control of autonomous unmanned system. Traditional manipulation frameworks focus either on sensing or control by assuming that the environment is known, which would result in lacking of autonomy for a specified task. Since mobile manipulation tasks often consist of nonholonomic and holonomic constraints for wheeled mobile manipulators with differential steering. In addition, most current works for whole body control are based on the condition that robot dynamic parameters are known beforehand. Therefore, it is necessary to establish an enhanced framework to simultaneously deal with these problems. In this paper, an integrated navigation and control framework is proposed. To make the mobile manipulator work in the unknown environment, task sensing and whole body control for mobile manipulators are also developed. The framework is partitioned into the task sensing, navigation and control, where the mobile manipulator can fulfill the mobile manipulation tasks in the unstructured environments.

A Framework to Explore Proximate Human-Robot Coordination

Proximate human-robot teaming (pxHRT) is a complex subspace within human-robot interaction. Studies in this space involve a range of equipment and methods, including the ability to sense people and robots precisely. Research in this area draws from a wide variety of other fields, from human-human interaction to control theory, making the study design complex, particularly for those outside the field of HRI. In this paper, we introduce a framework that helps researchers consider tradeoffs across various task contexts, platforms, sensors, and analysis methods; metrics frequently used in the field; and common challenges researchers may face. We demonstrate the use of the framework via a case study which employs an autonomous mobile manipulator continuously engaging in shared workspace, handover, and co-manipulation tasks with people, and explores the effect of cognitive workload on pxHRT dynamics. We also demonstrate the utility of the framework in a case study with two groups of researchers new to pxHRT. With this framework, we hope to enable researchers, especially those outside HRI, to more thoroughly consider these complex components within their studies, more easily design experiments, and more fully explore research questions within the space of pxHRT.

Assembly of Compliant Structures with Autonomous Industrial Mobile Manipulators (AIMM) Using an End Effector with Active Deformation Compensation for the Assembly of Flaps

Composite structures in aeroplanes are often thin-walled and lightweight, resulting in significant compliance, which presents a handling and assembly challenge due to the associated part deformations. In order to counteract these deformations, the parts are held in their specified geometry using stiff and correspondingly heavy fixtures or jigs. Mobile industrial robots are very versatile and widely used in industrial volume production, but they are limited in their payload capacity. High-rate production of large aerospace modules requires highly automated flexible assembly processes. The approach presented in this paper is to combine mobile units with lightweight assembly jigs that have the capability of deformation compensation. The subject of the study is a high-rate assembly process for flap modules using an Autonomous Industrial Mobile Manipulator (AIMM) and a lightweight end effector. The end effector has a shape compensation function, implemented by an integrated Stewart platform, which enables the compensation of manufacturing tolerances as well as gravity effects. The compensation function is used in a closed loop and counteracts shape deviations by appropriate fixture shape adjustments. The paper reports on the conceptual design of the assembly scenario, the design of the end effector, its realization and the successful experimental demonstration at 1:1 scale.

DeepNet-Based 3D Visual Servoing Robotic Manipulation

The fourth industrial revolution (industry 4.0) demands high-autonomy and intelligence robotic manipulators. The goal is to accomplish autonomous manipulation tasks without human interventions. However, visual pose estimation of target object in 3D space is one of the critical challenges for robot-object interaction. Incorporating the estimated pose into an autonomous manipulation control scheme is another challenge. In this paper, a deep-ConvNet algorithm is developed for object pose estimation. Then, it is integrated into a 3D visual servoing to achieve a long-range mobile manipulation task using a single camera setup. The proposed system integrates (1) deep-ConvNet training using only synthetic single images, (2) 6DOF object pose estimation as sensing feedback, and (3) autonomous long-range mobile manipulation control. The developed system consists of two main steps. First, a perception network trains on synthetic datasets and then efficiently generalizes to real-life environment without postrefinements. Second, the execution step takes the estimated pose to generate continuous translational and orientational joint velocities. The proposed system has been experimentally verified and discussed using the Husky mobile base and 6DOF UR5 manipulator. Experimental findings from simulations and real-world settings showed the efficiency of using synthetic datasets in mobile manipulation task.

Area-Coverage Planning for Spray-based Surface Disinfection with a Mobile Manipulator

The use of robots has significantly increased to fight highly contagious diseases like SARS-COV-2, Ebola, MERS, and others. One of the important applications of robots to fight such infectious diseases is disinfection. Manual disinfection can be a time-consuming, risky, labor-intensive, and mundane, and humans may fail to disinfect critical areas due to the resulting fatigue. Autonomous or semi-autonomous mobile manipulators mounted with a spray nozzle at the end-effector can be very effective in spraying disinfectant liquid for deep disinfection of objects and surfaces. In this paper, we present an area-coverage planning algorithm to compute a path that the nozzle follows to disinfect surfaces represented by their point clouds. We project the point cloud on a plane and produce a polygon on which we generate multiple spray paths using our branch and bound-based tree search area-coverage algorithm such that the spray paths cover the entire area of the polygon. An appropriate spray path is chosen using a robot capability map-based selection criterion. We generate mobile manipulator trajectories using successive refinement-based parametric optimization so that the paths for the nozzle are followed accurately. Thereafter, we need to make sure that the joint velocities of the mobile manipulator are regulated appropriately such that each point on the surface receives enough disinfectant spray. To this end, we compute the time intervals between the robot path waypoints such that enough disinfectant liquid is sprayed on all points of the point cloud that results in thorough disinfection of the surface, and the particular robot path is executed in the minimum possible time. We have implemented the area-coverage planning and mobile manipulator motion planning on five test scenarios in simulation using our ADAMMS-SD (Agile Dexterous Autonomous Mobile Manipulation System for Surface Disinfection) robot. We benchmark our spray path generation algorithm with three competing methods by showing that the generated paths are significantly more efficient in terms of area coverage and reducing disinfectant wastage. We also show the time interval computation between successive waypoints results in thorough disinfection of surfaces.

A Robot-Assisted Large-Scale Inspection of Wind Turbine Blades in Manufacturing Using an Autonomous Mobile Manipulator

Wind energy represents the dominant share of renewable energies. The rotor blades of a wind turbine are typically made from composite material, which withstands high forces during rotation. The huge dimensions of the rotor blades complicate the inspection processes in manufacturing. The automation of inspection processes has a great potential to increase the overall productivity and to create a consistent reliable database for each individual rotor blade. The focus of this paper is set on the process of rotor blade inspection automation by utilizing an autonomous mobile manipulator. The main innovations include a novel path planning strategy for zone-based navigation, which enables an intuitive right-hand or left-hand driving behavior in a shared human–robot workspace. In addition, we introduce a new method for surface orthogonal motion planning in connection with large-scale structures. An overall execution strategy controls the navigation and manipulation processes of the long-running inspection task. The implemented concepts are evaluated in simulation and applied in a real-use case including the tip of a rotor blade form.

High-Precision Mobile Robotic Manipulator for Reconfigurable Manufacturing Systems

The manufacturing industry has identified a new megatrend of mass customization, which is one of the essential goals of Industry 4.0. This megatrend requires the realization of manufacturing that can respond quickly and flexibly to various changing production requirements and ensure the achievement of various quality criteria. However, the manufacturing cannot be realized by conventional manufacturing systems in which reconfigurations need to be performed by skilled engineers. This paper proposes a new reconfigurable manufacturing system concept based on an ultra-flexible transfer system. Particularly, an autonomous mobile robotic manipulator, consisting of a high-performance automated guided vehicle module and a collaborative robotic manipulator module, represents a key component of the system concept. In this context, the focus is on the cooperative control between the modules of the autonomous mobile manipulator, which is essential for high-precision processes (e.g., machining, assembly, measurement, inspection), and its wide operating area. The experimental results confirm that the proposed cooperative control improves the positioning performance of the autonomous mobile manipulator, including the time required for positioning and the positioning accuracy.

Vision-Based Control of a Mobile Manipulator With an Adaptable-Passive Suspension for Unstructured Environments

Abstract The kinematic design, development, and navigation control of a new autonomous mobile manipulator for unstructured terrain is presented in this work. An innovative suspension system is designed based on the kinematic synthesis of an adaptable, passive mechanism. This novel suspension can compensate for irregularities in the terrain by using two pairs of bogies joined by a crank-slider mechanism and facilitates the control of the robotic platform using video cameras. The mobile robot is also equipped with a robotic manipulator, of which a synthesis, simulation, and experimental validation are presented. Additionally, manipulation is accomplished during motion on rough terrain. The proposed mobile robot has been fabricated using additive manufacturing (AM) techniques. A vision-based control approach, from here on named mobile linear-camera space manipulation (MLCSM), for mobile manipulators has been synthesized and implemented to conduct experimental tests. This mobile manipulator has been designed to traverse uneven terrain so that the loading platform is kept close to horizontal while crossing obstacles up to one-third of the size of its wheels. This feature allows for the onboard cameras to stay oriented toward the target; it also allows for any device mounted on the payload platform to remain nearly horizontal during the task. The developed control approach allows us to estimate the position and orientation of the manipulator’s end effector and update its trajectory along the path toward the target. The experiments show a final precision for engagement of a pallet within +/−2.5 mm in position and +/−2 deg in orientation

A Real Application of an Autonomous Industrial Mobile Manipulator within Industrial Context

In modern industry there are still a large number of low added-value processes that can be automated or semi-automated with safe cooperation between robot and human operators. The European SHERLOCK project aims to integrate an autonomous industrial mobile manipulator (AIMM) to perform cooperative tasks between a robot and a human. To be able to do this, AIMMs need to have a variety of advanced cognitive skills like autonomous navigation, smart perception and task management. In this paper, we report the project’s tackle in a paradigmatic industrial application combining accurate autonomous navigation with deep learning-based 3D perception for pose estimation to locate and manipulate different industrial objects in an unstructured environment. The proposed method presents a combination of different technologies fused in an AIMM that achieve the proposed objective with a success rate of 83.33% in tests carried out in a real environment.

Robust and adaptive door operation with a mobile robot

The ability to deal with articulated objects is very important for robots assisting humans. In this work, a framework to robustly and adaptively operate common doors, using an autonomous mobile manipulator, is proposed. To push forward the state-of-the-art in robustness and speed performance, we devise a novel algorithm that fuses a convolutional neural network with efficient point cloud processing. This advancement enables real-time grasping pose estimation for multiple handles from RGB-D images, providing a speed up improvement for assistive human-centered applications. In addition, we propose a versatile Bayesian framework that endows the robot with the ability to infer the door kinematic model from observations of its motion and learn from previous experiences or human demonstrations. Combining these algorithms with a Task Space Region motion planner, we achieve an efficient door operation regardless of the kinematic model. We validate our framework with real-world experiments using the Toyota Human Support Robot.

An autonomous mobile manipulator to build outdoor structures consisting of heterogeneous brick patterns

This paper describes the implementation of an autonomous mobile manipulator to build outdoor structures consisting of heterogeneous brick patterns, finding applications in different industrial automation, manufacturing and civil construction scenarios. This system was developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020, to demonstrate the novel real-world application of constructing structures using teams of robots, showcasing once again the role of major scientific competitions in advancing the state of the art towards exploring solutions to open problems. The paper presents in detail the hardware and software architectures of the developed mobile manipulator, integrating different research results and developments into a functional complex robot system, while proposing methods to detect, approach and manipulate differently sized/colored bricks to build a wall of predefined pattern given to the robot just before the building task starts.Article HighlightsFull system description of an autonomous mobile manipulator for construction tasks tested in a realistic settingAlgorithms for detection, localization, picking and placement of heterogeneous building blocks to form large structuresVersatile service robot capable of smooth adaptation to other functions developed through scientific robot competitions

A model predictive approach for online mobile manipulation of non-holonomic objects using learned dynamics

Assistive robots designed for physical interaction with objects will play an important role in assisting with mobility and fall prevention in healthcare facilities. Autonomous mobile manipulation presents a hurdle prior to safely using robots in real-life applications. In this article, we introduce a mobile manipulation framework based on model predictive control using learned dynamics models of objects. We focus on the specific problem of manipulating legged objects such as those commonly found in healthcare environments and personal dwellings (e.g., walkers, tables, chairs). We describe a probabilistic method for autonomous learning of an approximate dynamics model for these objects. In this method, we learn dynamic parameters using a small dataset consisting of force and motion data from interactions between the robot and object. Moreover, we account for multiple manipulation strategies by formulating manipulation planning as a mixed-integer convex optimization. The proposed framework considers the hybrid control system composed of (i) choosing which leg to grasp and (ii) control of continuous applied forces for manipulation. We formalize our algorithm based on model predictive control to compensate for modeling errors and find an optimal path to manipulate the object from one configuration to another. We present results for several objects with various wheel configurations. Simulation and physical experiments show that the obtained dynamics models are sufficiently accurate for safe and collision-free manipulation. When combined with the proposed manipulation planning algorithm, the robot successfully moves the object to the desired pose while avoiding any collision.