Autonomous Mobile Manipulator Research Articles

A model predictive approach for online mobile manipulation of non-holonomic objects using learned dynamics

Assistive robots designed for physical interaction with objects will play an important role in assisting with mobility and fall prevention in healthcare facilities. Autonomous mobile manipulation presents a hurdle prior to safely using robots in real-life applications. In this article, we introduce a mobile manipulation framework based on model predictive control using learned dynamics models of objects. We focus on the specific problem of manipulating legged objects such as those commonly found in healthcare environments and personal dwellings (e.g., walkers, tables, chairs). We describe a probabilistic method for autonomous learning of an approximate dynamics model for these objects. In this method, we learn dynamic parameters using a small dataset consisting of force and motion data from interactions between the robot and object. Moreover, we account for multiple manipulation strategies by formulating manipulation planning as a mixed-integer convex optimization. The proposed framework considers the hybrid control system composed of (i) choosing which leg to grasp and (ii) control of continuous applied forces for manipulation. We formalize our algorithm based on model predictive control to compensate for modeling errors and find an optimal path to manipulate the object from one configuration to another. We present results for several objects with various wheel configurations. Simulation and physical experiments show that the obtained dynamics models are sufficiently accurate for safe and collision-free manipulation. When combined with the proposed manipulation planning algorithm, the robot successfully moves the object to the desired pose while avoiding any collision.

Whole-Body Control of an Autonomous Mobile Manipulator Using Series Elastic Actuators

The flexible-joint robot has superior attractive features because of its high mobility, high load ratio, high torque fidelity, robustness for external disturbance, task adaptability, and safety. In this article, an autonomous mobile manipulator driven by the designed series elastic actuators is developed. A complete dynamic model of a nonholonomic mobile manipulator including a mobile platform and a manipulator with joint flexibility operating simultaneously is proposed. An integrated whole-body trajectory control framework is proposed for such robot to perform mobile manipulation tasks. Considering the nonholonomic and holonomic constraints in the mobile manipulation, the whole-body dynamics is formulated and reduced. To address the highly nonlinear of the dynamics and model uncertainty, a novel integral Lyapunov function based adaptive neural network control for task tracking under uncertainties of the flexible-joint robot model is proposed. Compared with existing methods, the proposed method provides an alternative for controlling flexible-joint robots. The feasibility of the proposed method is verified by the extensive trajectory tracking experiment results in our developed flexible joint manipulator.

On-Line Precision Calibration of Mobile Manipulators Based on the Multi-Level Measurement Strategy

Autonomous industrial mobile manipulator (AIMM), which is composed of a mobile platform and a robot arm of six degrees of freedom (DoFs), can move autonomously and perform dexterous tasks independently in the industrial environment. A novel on-line calibration system for AIMM in real manufacturing workshop is proposed in this paper. Compared to calibration scenarios of fixed-base robot, we focus on the large-space autonomous calibration for AIMM. To cope with the issue of high precision calibration in large space, we propose a multi-level guided-based measurement strategy, which includes the low-cost vision positioning and guidance system, and the high-precision distributed base station (BS) measurement system. The corresponding experiment proves that the total uncertainty of measurement system is less than 0.1 mm. Based on the measured data of end-effector installed at the end of robot arm, the AIMM can perform on-line calibration tasks anywhere automatically without taking into account the expensive devices for the high-precision location of mobile platform and the downtime for the robotic off-line calibration. The mean positioning error of AIMM after calibration reduces 93% based on the calibration experiment.

OMNIVIL-An Autonomous Mobile Manipulator for Flexible Production.

Flexible production is a key element in modern industrial manufacturing. Autonomous mobile manipulators can be used to execute various tasks: from logistics, to pick and place, or handling. Therefore, autonomous robotic systems can even increase the flexibility of existing production environments. However, the application of robotic systems is challenging due to their complexity and safety concerns. This paper addresses the design and implementation of the autonomous mobile manipulator OMNIVIL. A holonomic kinematic design provides high maneuverability and the implemented sensor setup with the underlying localization strategies are robust against typical static and dynamic uncertainties in industrial environments. For a safe and efficient human–robot collaboration (HRC), a novel workspace monitoring system (WMS) is developed to detect human co-workers and other objects in the workspace. The multilayer sensor setup and the parallel data analyzing capability provide superior accuracy and reliability. An intuitive zone-based navigation concept is implemented, based on the workspace monitoring system. Preventive behaviors are predefined for a conflict-free interaction with human co-workers. A workspace analyzing tool is implemented for adaptive manipulation, which significantly simplifies the determination of suitable platform positions for a manipulation task.

Trajectory via-point generation for autonomous mobile manipulation using 3D LiDAR data

In this work, an approach to generating a set of via points for use in manipulator trajectory path planning is presented. The approach was developed for use on a robotic underground mining system, particularly for the task of autonomous application of a sprayable concrete called shotcrete. A LiDAR (light detection and ranging) scanner on a nodding head produces point clouds that are used as the input for the via-point selection algorithm. The algorithm generates a set of position and orientation via points that the manipulator must follow to perform the shotcreting task. The developed algorithm has been successfully tested on an autonomous mobile-manipulator system in a scaled mock-up of an underground mine. The main advantage of this algorithm is the ability to generate via points for any section of an underground mine in any position relative to the robot.

Robot Errors in Proximate HRI

Advancements within human–robot interaction generate increasing opportunities for proximate, goal-directed joint action (GDJA). However, robot errors are common and researchers must determine how to mitigate them. In this article, we examine how expectations for robot functionality affect people’s perceptions of robot reliability and trust for a robot that makes errors. Here 35 participants ( n = 35) performed a collaborative banner-hanging task with an autonomous mobile manipulator (Toyota HSR). Each participant received either a low- or high-functionality framing for the robot. We then measured how participants perceived the robot’s reliability and trust prior to, during, and after interaction. Functionality framing changed how robot errors affected participant experiences of robot behavior. People with low expectations experienced positive changes in reliability and trust after interacting with the robot, while those with high expectations experienced a negative change in reliability and no change in trust. The low-expectation group also showed greater trust recovery following the robot’s first error compared to the high group. Our findings inform human–robot teaming through: (1) identifying robot presentation factors that can be employed to facilitate trust calibration and (2) establishing the effects of framing, functionality, and the interactions between them to improve dynamic models of human–robot teaming.

A novel coordinated motion planner based on capability map for autonomous mobile manipulator

With the development of the robotic technology, Autonomous Mobile Manipulator (AMM) is increasingly used in more applications. Reasonable motion planning for AMM to maintain high manipulation capability is the prerequisite for the success of the mobile manipulation task. In this paper, the Capability Map (CM) of AMM that gives the distribution of the manipulability in cartesian space is first built. Then given the path of the end effector, we design a novel path planner for the mobile robot by querying CM online so that AMM keeps high manipulability. Moreover, a task-priority coordinated motion controller is developed to control the mobile robot and the manipulator to track their trajectories. In this controller, the trajectory of the manipulator is used as the primary task, and the trajectory of the mobile robot is treated as the constrained task. Simulation results show that the path of the mobile robot can be found online, and AMM follows the trajectories well.

Multirepresentation, Multiheuristic A* search‐based motion planning for a free‐floating underwater vehicle‐manipulator system in unknown environment

AbstractA key challenge in autonomous mobile manipulation is the ability to determine, in real time, how to safely execute complex tasks when placed in unknown or changing world. Addressing this issue for Intervention Autonomous Underwater Vehicles (I‐AUVs), operating in potentially unstructured environment is becoming essential. Our research focuses on using motion planning to increase the I‐AUVs autonomy, and on addressing three major challenges: (a) producing consistent deterministic trajectories, (b) addressing the high dimensionality of the system and its impact on the real‐time response, and (c) coordinating the motion between the floating vehicle and the arm. The latter challenge is of high importance to achieve the accuracy required for manipulation, especially considering the floating nature of the AUV and the control challenges that come with it. In this study, for the first time, we demonstrate experimental results performing manipulation in unknown environment. The Multirepresentation, Multiheuristic A* (MR‐MHA*) search‐based planner, previously tested only in simulation and in a known a priori environment, is now extended to control Girona500 I‐AUV performing a Valve‐Turning intervention in a water tank. To this aim, the AUV was upgraded with an in‐house‐developed laser scanner to gather three‐dimensional (3D) point clouds for building, in real time, an occupancy grid map (octomap) of the environment. The MR‐MHA* motion planner used this octomap to plan, in real time, collision‐free trajectories. To achieve the accuracy required to complete the task, a vision‐based navigation method was employed. In addition, to reinforce the safety, accounting for the localization uncertainty, a cost function was introduced to keep minimum clearance in the planning. Moreover a visual‐servoing method had to be implemented to complete the last step of the manipulation with the desired accuracy. Lastly, we further analyzed the approach performance from both loose‐coupling and clearance perspectives. Our results show the success and efficiency of the approach to meet the desired behavior, as well as the ability to adapt to unknown environments.

Hybrid Maps Enhanced Localization System for Mobile Manipulator in Harsh Manufacturing Workshop

With excellent mobility and flexibility, mobile manipulators have great potential for loading and unloading tasks of numerical control machine tools (CNC) in manufacturing workshops. However, because of the rough and oily ground, dynamic obstacles and the convex plate of a CNC, harsh manufacturing workshop poses a huge challenge to the localization system of an autonomous mobile manipulator. To address the above problem, this paper presents a hybrid maps enhanced localization system which mainly consists of a global localization method and a pose tracking method. Hybrid maps including hybrid grid map, multi-resolution likelihood fields (MLFs) and hybrid point map are constructed to efficaciously model the harsh environment and to improve localization performance. Our global localization method employs the convex hull sampling to spares dense Lidar data and the MLFs based branch and bound (BnB) search to speed up global search. To achieve real-time localization reliably and accurately, our pose tracking method seamlessly combines the BnB search and the adaptive Monte Carlo localization, and the Iterative Closest Point (ICP) based scan matching using the hybrid point map is adopted for higher accuracy. In addition, a distance filter improved by unscented transform is integrated into the pose tracking process to mitigate the influence of dynamic obstacles. The developed localization system is evaluated through different experiments including two weeks of loading and unloading tasks in a real manufacturing scenario, resulting in superior localization performance.

Adaptive motion generation using imitation learning and highly compliant end effector for autonomous cleaning

Recent demographic trends in super aging societies, such as Japan, is leading to severe worker shortage. Service robots can play a promising role to augment human workers for performing various household and assistive tasks. Toilet cleanup is one such challenging task that involves performing complaint motion planning in a constrained toilet setting. In this study, we propose an end-to-end robotic framework to perform various tasks related to toilet cleanup. Our key contributions include the design of a complaint and multipurpose end-effector, an adaptive motion generation algorithm, and an autonomous mobile manipulator capable of garbage detection, garbage disposal and liquid removal. We evaluate the performance of our framework with the competition setting used for toilet cleanup in the Future Convenience Store Challenge at the World Robot Summit 2018. We demonstrate that our proposed framework is capable of successfully completing all the tasks of the competition within the time limit.

Whole-Body MPC for a Dynamically Stable Mobile Manipulator

Autonomous mobile manipulation offers a dual advantage of mobility provided by a mobile platform and dexterity afforded by the manipulator. In this paper, we present a whole-body optimal control framework to jointly solve the problems of manipulation, balancing and interaction as one optimization problem for an inherently unstable robot. The optimization is performed using a Model Predictive Control (MPC) approach; the optimal control problem is transcribed at the end-effector space, treating the position and orientation tasks in the MPC planner, and skillfully planning for end-effector contact forces. The proposed formulation evaluates how the control decisions aimed at end-effector tracking and environment interaction will affect the balance of the system in the future. We showcase the advantages of the proposed MPC approach on the example of a ball-balancing robot with a robotic manipulator and validate our controller in hardware experiments for tasks such as end-effector pose tracking and door opening.

Dexterous Grasping by Manipulability Selection for Mobile Manipulator With Visual Guidance

Industry 4.0 demands the heavy usage of robotic mobile manipulators with high autonomy and intelligence. The goal is to accomplish dexterous manipulation tasks without prior knowledge of the object status in unstructured environments. It is important for the mobile manipulator to recognize and detect the objects, determine manipulation pose, and adjust its pose in the workspace fast and accurately. In this research, we developed a stereo vision algorithm for the object pose estimation using point cloud data from multiple stereo vision systems. An improved iterative closest point algorithm method is developed for the pose estimation. With the pose input, algorithms and several criteria are studied for the robot to select and adjust its pose by maximizing its manipulability on a given manipulation task. The performance of each technical module and the complete robotic system is finally shown by the virtual robot in the simulator and real robot in experiments. This study demonstrates a setup of autonomous mobile manipulator for various flexible manufacturing and logistical scenarios.

Deployment of an autonomous mobile manipulator at MBZIRC

AbstractThis paper introduces a custom mobile manipulator and presents the planning and control architecture to execute autonomous missions including navigation and manipulation. The developed system combines a commercially available ground vehicle with a custom‐designed robot arm. A typical mission requires the robot to find and navigate to its site of action, identify the suitable manipulation tools, and manipulate the correct objects according to the task specification. Our contribution includes a complete software pipeline for autonomous map building, localization, and navigation. We demonstrate stable localization on vast outdoor areas without GPS support. Our exploration strategy coupled with a path planner reliably scans the environment while avoiding obstacles until it finds the manipulation site. The position of the target location is estimated from laser data within a 5 cm radius, and the subsequent positioning procedure of the mobile base achieves a precision of ±4 cm. The developed manipulator uses series elastic actuation, making the system mechanically robust and compliant. Thanks to the precise joint torque controllability, a low end‐effector impedance can be realized to passively overcome positioning inaccuracies during manipulation. Our solution incorporates a support vector machine to detect the manipulation object as required for visual servoing. We show how a valve stem and standard wrenches can be identified with a detection rate higher than 90%. The paper finally discusses the performance and lessons learned while participating at the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017, where we showed the successful autonomous operation in a partially unknown environment.

Motion planning survey for autonomous mobile manipulators underwater manipulator case study

Sampling-based, search-based, and optimization-based motion planners are just some of the different approaches developed for motion planning problems. Given the wide variety of application tackled by autonomous mobile manipulators, the question “which planner to choose” may be tough. In this paper, we review the state of the art of the most common approaches, and present a set of benchmarks with the aim to provide not only a theoretical review but also a qualitative/quantitative comparison of the algorithms. Our purpose is to provide an insight and analyze their performance with respect to different metrics. The results are based on an Underwater Vehicle Manipulator System UVMS, although they can be extended to terrestrial and aerial robots as well. The paper uses these results to formalize a set of guidelines for the selection process of the most appropriate approach, for a given problem/requirements.

Toward fully autonomous mobile manipulation for industrial environments

This work presents a concept for autonomous mobile manipulation in industrial environments. Utilizing autonomy enables an unskilled human worker to easily configure a complex robotics system in a setup phase before carrying out fetch and carry operations in the execution phase. In order to perform the given tasks in real industrial production sites, we propose a robotic system consisting of a mobile platform, a torque-controlled manipulator, and an additional sensor head. Multiple sensors are attached which allow for perception of the environment and the objects to be manipulated. This is essential for coping with uncertainties in real-world application. In order to provide an easy-to-use and flexible system, we present a modular software concept which is handled and organized by a hierarchical flow control depending on the given task and environmental requirements. The presented concept for autonomous mobile manipulation is implemented exemplary for industrial manipulation tasks and proven by real-world application in a water pump production site. Furthermore, the concept has also been applied to other robotic systems and other domains for planetary exploration with a rover.

A Skill-based Robot Co-worker for Industrial Maintenance Tasks

This paper investigates the concept of a sensor based robot co-worker working in flexible industrial environments together with and alongside human operators. In this particular work, a realisation of a robot co-worker scenario is developed in order to demonstrate the implementation of a robot co-worker from the starting point of an autonomous industrial mobile manipulator. The cobot is applied on the industrially relevant task of screwing by the use of a skill-based approach. The technical work on the human-robot interface and the screwing skill is described.

Condition Monitoring and Cloud-based Energy Analysis for Autonomous Mobile Manipulation - Smart Factory Concept with LUHbots

In this paper, a smart factory concept for autonomous mobile robots is presented. The main purpose is to increase productivity of the transport in machine-floor. It is based on advanced methods for failure handling and prevention, leading to increased robustness, less downtime and less effort in maintenance [1], [2]. Therefore, condition data and states of the robot are collected by Robot Operation System (ROS) and transferred to a factory hub (server). The collected data, e.g. voltages, currents, set points, velocities and accelerations are used to identify important system parameters, e.g. moving masses and friction parameters to enable the proposed smart factory concept. Further aim is to let the factory hub control a group of mobile robots using a self-organizing algorithm for different tasks.Due to the increasing customization of products causing smaller lot sizes [3], manufacturers of mobile robotic production systems have developed a diversity of flexible robots [4], [5], [6], [7], [8]. Mobile robots inside the production line allow for collecting and evaluation of system-inherent data e.g. handling and transportation time, wheel friction, workpieces mass, center of gravity and energy consumption during trajectory execution.In general, mobile robots are electrically driven. Hence, an estimation of the battery state is essential in order to automatically plan charging cycles and to organize and optimize the cooperation behavior of a group of mobile robots. In this proposed approach, mobile robots are equipped with a measurement system and connected via Bluetooth to a factory hub, providing monitoring, analyzing and planning tools. The battery states of all robots are considered in the process planning.The robots are based on the KUKA youBot, equipped with a soft gripper and a RealSense camera. A condition monitoring system measures the energy consumption of all components and transfers the information to the factory hub. The state of charge limits the number of executable operations. Therefore, in a first step the power consumption of all individual consumers is captured, e.g. EC-Maxxon base motors, PC, gripper, camera and five-axis arm. Experimental results show, that the youBot requires 46 W in standstill plus the drive power depending on the movement. Here, the results for mobile manipulation in industrial scenarios during preparation for the RoboCup@Work 2016 will be presented. The transfer of raw measurement data to the hub is shown, as well as the proposed algorithms allowing for range prediction and optimized set point generation. The concept provides excellent capability in data collection, analysis of existing production and production planning.

Mobile Manipulation: Toward Smart Manufacturing [TC Spotlight

The goal of autonomous mobile manipulation is the execution of complex manipulation tasks in both structured and unstructured environments, possibly in cooperation or close interaction with human beings. This entails challenges in different fields: perception, navigation, task and path planning, control, error recovery, and human-robot interaction. Each field is an area of research on its own, but the special challenge in mobile manipulation is to obtain an integrated system that can combine a large variety of hardware and software components to increase the tasks that the robot can perform, while decreasing the dependency on a priori information and increasing the awareness of the robot to its current situation. Challenges in the field arise from the high-dimensional spaces associated to systems with high number of actuators and sensors, uncertainty in perception and execution of the tasks, generality of the solutions required to manipulate objects in the real world, and engineering and scientific complexity of the systems required for these tasks.

NimbRo Explorer: Semiautonomous Exploration and Mobile Manipulation in Rough Terrain

Fully autonomous exploration and mobile manipulation in rough terrain are still beyond the state of the art—robotics challenges and competitions are held to facilitate and benchmark research in this direction. One example is the 2013 DLR SpaceBot Cup, for which we developed an integrated robot system to semiautonomously perform planetary exploration and manipulation tasks. Our robot explores, maps, and navigates in previously unknown, uneven terrain using a three‐dimensional laser scanner and an omnidirectional RGB‐D camera. We developed manipulation capabilities for object retrieval and pick‐and‐place tasks. Many parts of the mission can be performed autonomously. In addition, we developed teleoperation interfaces on different levels of shared autonomy, which allow for specifying missions, monitoring mission progress, and on‐the‐fly reconfiguration. To handle network communication interruptions and latencies between robot and operator station, we implemented a robust network layer for the middleware Robot Operating System (ROS). The integrated system has been demonstrated at the 2013 DLR SpaceBot Cup. In addition, we conducted systematic experiments to evaluate the performance of our approaches.

Decision making logic for flexible assembly lines reconfiguration

This paper presents a framework for the control of autonomous production units for real time reconfiguration of the shop floor. The approach considers a series of highly flexible production solutions including a) autonomous mobile manipulators, which are capable of repositioning themselves on the line, b) flexible grippers with multiple connection points that allow the exchange of both parts and grippers between robots. The paper focuses on providing a decision making mechanism for the generation of efficient shop floor reconfiguration plans that can efficiently exploit these functionalities. The developments have been tested in a case study in the automotive industry and most specifically, in the Body in White (BIW) assembly stage. Simulation models have been used for the evaluation of the control logic performance and the results indicate that a significant increase in the overall resource utilization can be achieved. We examine a decision making mechanism for the generation of efficient shop floor reconfiguration plans.The mechanism exploits the use of autonomous and flexible manufacturing units such as autonomous mobile manipulators, cooperating robots and flexible grippers with multiple connection points.An intelligent search algorithm was introduced to efficiently (in terms of computational time and power) derive reconfiguration plans.A multi criteria decision making method is introduced in for the evaluation of the reconfiguration plans, accounting for Reconfiguration time, Average Resource utilization and Annual production volume.Evaluation of the decision making logic performance through simulation in an automotive assembly case study.