Mobile Manipulator Research Articles

Hybrid Sampling-Based Path Planning for Mobile Manipulators Performing Pick and Place Tasks in Narrow Spaces

A mobile manipulator is capable of traversing a vast area while performing manipulation tasks in confined spaces. However, the high degree of freedom presents a challenge for path planning. In this paper, a hybrid sampling-based path planning method is proposed for mobile manipulators performing pick and place tasks in confined spaces. This method employs a random sampling approach, yet differs from the traditional RRT method. Firstly, a sampling-based configuration generation method for mobile manipulators is proposed, with the objective of generating a valid, collision-free configuration with the end-effector at the desired pose. A path for the end-effector corresponding to the goal configuration is then planned using the RRT method. Secondly, an area-restricted approach that samples in the vicinity of the previous configuration is introduced to generate the next valid configuration. Subsequently, a cost computation rule is devised to identify the optimal subsequent configuration utilizing the trajectory of the end-effector as a guiding principle. Finally, the obtained path is smoothed. Simulations demonstrate that the proposed hybrid sample-based method is an effective solution to the path planning problem for mobile manipulators performing pick and place tasks in narrow spaces.

Robust Shared Control With Stable Contact Servoing for Enhanced Object Transportation by Telerobotic Bimanual Mobile Manipulators

ABSTRACTBimanual mobile manipulation significantly enhances the capabilities of field robots that interact with diverse dynamic environments and expands the range of possible manipulation actions. This paper introduces a robust shared control architecture for remote manipulation of the Collaborative Dual‐arm Robot Manipulator (CURI), aimed at stable object transportation. Given that stable contact is pivotal for successful object transportation, our focus is on stabilizing the contact between the robot end‐effectors and the object using a contact servoing strategy. To maintain stable contact, it is essential that the contact wrenches consistently stay within the predefined friction cones throughout the task. We propose a novel shared control strategy to stabilize the contact using the contact parameterization model. This model formulates the contact stability manifold using a set of unconstrained reparameterized variables through a bijective mapping function, ensuring that the controller derived from these unconstrained reparameterized variables consistently maintains contact stability. The efficacy of the proposed approach is convincingly demonstrated through a series of experiments involving representative bimanual transportation tasks within a logistics context. Both theoretical analysis and experimental validations confirm that our proposed control strategy not only ensures the stable transportation of objects by CURI but also substantially enhances the overall robustness of the shared control system.

Variable impedance control on contact-rich manipulation of a collaborative industrial mobile manipulator: An imitation learning approach

Variable impedance control (VIC) endows robots with the ability to adjust their compliance, enhancing safety and adaptability in contact-rich tasks. However, determining suitable variable impedance parameters for specific tasks remains challenging. To address this challenge, this paper proposes an imitation learning-based VIC policy that employs observations integrated with RGBD and force/torque (F/T) data enabling a collaborative mobile manipulator to execute contact-rich tasks by learning from human demonstrations. The VIC policy is learned through training the robot in a customized simulation environment, utilizing an inverse reinforcement learning (IRL) algorithm. High-dimensional demonstration data is encoded by integrating a 16-layer convolutional neural network (CNN) into the IRL environment. To minimize the sim-to-real gap, contact dynamic parameters in the training environment are calibrated. Then, the learning-based VIC policy is comprehensively trained in the customized environment and its transferability is validated through an industrial production case involving a high precision peg-in-hole task using a collaborative mobile manipulator. The training and testing results indicate that the proposed imitation learning-based VIC policy ensures robust performance for contact-rich tasks.

Distributed Neural Adaptive Impedance Control for Cooperative Manipulation With Unknown Objects.

Existing cooperative manipulation methods for multiple manipulator systems usually assume that the grasp matrix and the desired trajectory of each manipulator are known in advance. In this work, distributed neural adaptive impedance control (AIC) strategies integrating fully distributed observers are proposed to remove both limitations. Specifically, two fully distributed finite-time observers are designed to estimate the actual and ideal states of the reference point without using global information. The estimates of the grasp matrix and the desired trajectory of each end-effector (EE) are then obtained by kinematic constraints and the estimates of the reference point's states. At the controller development, a distributed adaptive impedance model is established to achieve an adaptive trade-off between tracking performance and compliance. Then, distributed neural network (NN)-based tracking control strategies are developed to asymptotically realize the desired adaptive impedance dynamics in the presence of uncertainties. Additionally, a virtual energy tank (EK) is employed to interact with the impedance system to correct the adaptive impedance laws for system passivity. A simulation for four mobile manipulators tightly cooperative transport an unknown object is carried out to demonstrate the established results.

A Human-Centered View of Continual Learning: Understanding Interactions, Teaching Patterns, and Perceptions of Human Users Toward a Continual Learning Robot in Repeated Interactions

Continual learning (CL) has emerged as an important avenue of research in recent years, at the intersection of Machine Learning (ML) and Human–Robot Interaction (HRI), to allow robots to continually learn in their environments over long-term interactions with humans. Most research in CL, however, has been robot-centered to develop CL algorithms that can quickly learn new information on systematically collected static datasets. In this article, we take a human-centered approach to CL, to understand how humans interact with, teach, and perceive CL robots over the long term, and if there are variations in their teaching styles. We developed a socially guided CL system that integrates CL models for object recognition with a mobile manipulator robot and allows humans to directly teach and test the robot in real time over multiple sessions. We conducted an in-person study with 60 participants who interacted with the CL robot in 300 sessions with 5 sessions per participant. In this between-participant study, we used three different CL models deployed on a mobile manipulator robot. An extensive qualitative and quantitative analysis of the data collected in the study shows that there is significant variation among the teaching styles of individual users indicating the need for personalized adaptation to their distinct teaching styles. Our analysis shows that the constrained experimental setups that have been widely used to test most CL models are not adequate, as real users interact with and teach CL robots in a variety of ways. Finally, our analysis shows that although users have concerns about CL robots being deployed in our daily lives, they mention that with further improvements CL robots could assist older adults and people with disabilities in their homes.

Towards full autonomy in mobile robot navigation and manipulation

AbstractRobot autonomy is a key for automatizing industrial production plants, facilitating responsive search and rescue (SAR), and for interacting in urban settings. This work presents mobile systems capable of carrying out these tasks and summarises experience from practical deployments in three different domains for navigation. For industry, a mobile manipulator is presented, which is used for dynamic precise docking and object handling. For SAR, a semi-autonomous system is presented which is used for sampling hazardous materials, manipulating tools, and for localizing nuclear radiation sources. To improve urban logistics, a mobile robot autonomously navigates complex outdoor environments using semantic mapping techniques. Future work will explore further advancements that strive toward increased flexibility, reasoning and universal applicability of mobile robots. The goal is to reduce training, onboarding and tool changing times to reduce manual interfacing times.

MoMa: An assistive mobile manipulator with a webcam-based gaze control system

Mobile Manipulators (MoMa) is a category of mobile robots designed to assist people with motor disabilities to perform object retrieval tasks using a webcam-based gaze control system. Using off-the-shelf components such as reproducible acrylic and 3D-printed plates, and a webcam for eye tracking, MoMa serves as an inexpensive, open-source, and customizable solution in assistive robotics. The robotic system consists of a mobile base that can move forward and backward, as well as turn in place; and a 2-axis cartesian arm equipped with a claw gripper that opens and closes. The simple movement of the robot also allows for a simple control method and graphical user interface (GUI). The user receives information about what is in front of the robot through a mounted camera, and, by looking at parts of the screen that correspond to controls, has their gaze predicted by a convolutional neural network and sends commands wirelessly. The performance of the entire system has been validated through testing of the gaze prediction model, the integration of the control system, as well as its task completion capabilities. All the design, construction and software files are freely available online under the CC BY 4.0 license at https://doi.org/10.17632/k7yfn6wdv7.2.

Hybrid Orientation and Position Collaborative Motion Generation Scheme for a Multiple Mobile Redundant Manipulator System Synthesized by a Recurrent Neural Network.

To enable distributed multiple mobile manipulator systems to complete collaborative tasks safely and stably, this article investigates and presents a motion generation scheme that considers both orientation and position coordination based on a distributed recurrent neural network. Moreover, physical limits are also considered. Specifically, the orientation and position coordination constraints and physical limits are modeled separately as equality and inequality constraints with coupled variables. Subsequently, a motion generation scheme for multiple mobile manipulators based on quadratic programming is established. Finally, a distributed linear variational inequality-based primal-dual neural network is constructed to solve the motion generation scheme and obtain the motion trajectories of all the mobile manipulators. The simulation results demonstrate that the hybrid orientation and position collaboration motion generation scheme effectively addresses the position and orientation coordination problem for multiple mobile manipulator systems. Compared to other schemes, the proposed scheme based on a distributed computing structure greatly enhances the stability of the system. Additionally, the proposed approach introduces orientation coordination and physical limits, which increases the practicality of the system.

Language-Grounded Dynamic Scene Graphs for Interactive Object Search With Mobile Manipulation

Language-Grounded Dynamic Scene Graphs for Interactive Object Search With Mobile Manipulation

Control barrier function based visual servoing for Mobile Manipulator Systems under functional limitations

This paper proposes a new control strategy for Mobile Manipulator Systems (MMSs) that integrates image-based visual servoing (IBVS) to address operational limitations and safety constraints. The proposed approach based on the concept of control barrier functions (CBFs), provides a solution to address various operational challenges including visibility constraints, manipulator joint limits, predefined system velocity bounds, and system dynamic uncertainties. The proposed control strategy is a two-tiered structure, wherein the first level, a CBF-IBVS controller calculates control commands, taking into account the Field of View (FoV) constraints. By leveraging null space techniques, these commands are transposed to the joint-level configuration of the MMS, while considering system operational limits. Subsequently, in the second level, a CBF velocity controller employed for the entire MMS undertakes the tracking of the commands at the joint level, ensuring compliance with the predefined system’s velocity limitations as well as the safety of the whole combined system dynamics. The proposed control strategy offers superior transient and steady-state responses and heightened resilience to disturbances and modeling uncertainties. Furthermore, due to its low computational complexity, it can be easily implemented on an onboard computing system, facilitating real-time operation. The proposed strategy’s effectiveness is illustrated via simulation outcomes, which reveal enhanced performance and system safety compared to conventional IBVS methods. The results indicate that the proposed approach is effective in addressing the challenging operational limitations and safety constraints of mobile manipulator systems, making it suitable for practical applications.

Path Planning Based on Artificial Potential Field with an Enhanced Virtual Hill Algorithm

The artificial potential field algorithm has been widely applied to mobile robots and robotic arms due to its advantage of enabling simple and efficient path planning in unknown environments. However, solving the local minimum problem is an essential task and is still being studied. Among current methods, the technique using the virtual hill concept is reliable and suitable for real-time path planning because it does not create a new local minimum and provides lower complexity. However, in the previous study, the shape of the obstacles was not considered in determining the robot’s direction at the moment it is trapped in a local minimum. For this reason, longer or blocked paths are sometimes selected. In this study, we propose an enhanced virtual hill algorithm to reduce errors in selecting the driving direction and improve the efficiency of robot movement. In the local minimum area, a dead-end algorithm is also proposed that allows the robot to return without entering deeply when it encounters a dead end.

Non-Orthogonal Serret–Frenet Parametrization Applied to Path Following of B-Spline Curves by a Mobile Manipulator

Non-Orthogonal Serret–Frenet Parametrization Applied to Path Following of B-Spline Curves by a Mobile Manipulator

Working space planning for wheeled robot overturning stability

Abstract Mobile manipulators have been widely used in the field of building construction. Typical features of construction robots include crossing various rugged terrains in construction sites, carrying heavy payloads, and performing tasks in a large workspace, which may lead to unstable posture or overturning. The tipping of the mobile manipulator, especially when carrying a payload, will cause danger to the surrounding environment and staff, leading to the termination of the task. The purpose of this study is to establish a real-time tipping prediction system, which can be used to avoid tipping of mobile manipulators. By employing the CAD model of the robot and parameters such as the center of gravity and direction, we can ascertain and assess the instability of the robot and the overturning algorithm, thus providing early warning and averting any potential danger.

Virtual Teleoperation System for Mobile Manipulator Robots Focused on Object Transport and Manipulation

This work describes the development of a tool for the teleoperation of robots. The tool is developed in a virtual environment using the Unity graphics engine. For the development of the application, a kinematic model and a dynamic model of a mobile manipulator are used. The mobile manipulator robot consists of an omnidirectional platform and an anthropomorphic robotic arm with 4 degrees of freedom (4DOF). The model is essential to emulate the movements of the robot and to facilitate the immersion in the virtual environment. In addition, the control algorithms are established and developed in MATLAB 2020 software, which improves the acquisition of knowledge to teleoperate robots and execute tasks of manipulation and transport of objects. This methodology offers a cheaper and safer alternative to real physical systems, as it reduces both the costs and risks associated with using a real robot for training.

Robotic Valve Turning with a Wheeled Mobile Manipulator via Hybrid Passive/Active Compliance.

This paper addresses the problems of valve-turning operation in rescue environments where a wheeled mobile manipulator (WMM) is employed, including the possible occurrence of large internal forces. Rather than attempting to obtain the exact position of the valve, this paper presents a solution to two main problems in robotic valve-turning operations: the radial position deviation between the rotation axes of the tool and the valve handle, which may cause large radial forces, and the possible axial displacement of the valve handle as the valve turns, which may lead to large axial forces. For the former problem, we designed a compliant end-effector with a tolerance of approximately 3.5° (angle) and 9.7 mm (position), and provided a hybrid passive/active compliance method. For the latter problem, a passivity-based force tracking algorithm was employed. Combining the custom-built compliant end-effector and the passivity-based control method can significantly reduce both the radial and the axial forces. Additionally, for valves with different installation types and WMMs with different configurations, we analyzed the minimum required number of actuators for valve turning. Simulation and experimental results are presented to show the effectiveness of the proposed approach.

Assistance in Picking Up and Delivering Objects for Individuals with Reduced Mobility Using the TIAGo Robot

Individuals with reduced mobility, including the growing elderly demographic and those with spinal cord injuries, often face significant challenges in daily activities, leading to a dependence on assistance. To enhance their independence, we propose a robotic system that facilitates greater autonomy. Our approach involves a functional assistive robotic implementation for picking, placing, and delivering containers using the TIAGo mobile manipulator robot. We developed software and routines for detecting containers marked with an ArUco code and manipulating them using the MoveIt library. Subsequently, the robot navigates to specific points of interest within a room to deliver the container to the user or another designated location. This assistance task is commanded through a user interface based on a web application that can be accessed from the personal phones of patients. The functionality of the system was validated through testing. Additionally, a series of user trials were conducted, yielding positive feedback on the performance and the demonstration. Insights gained from user feedback will be incorporated into future improvements to the system.

Task and motion planning using mixed integer linear programming for solving fetch-and-carry tasks by a mobile manipulator

This manuscript describes a TAsk and Motion Planning (TAMP) method for mobile manipulators. We focus on fetch-and-carry tasks, and we aim to simultaneously generate both a sequence of actions and a motion sequence for each action. The number of factors to be considered in solving such a problem, and the interactions among them are complex due to the multifaceted characteristics of mobile manipulation. As a result, straightforwardly performing simultaneous TAMP may require a significant amount of processing time. Therefore, we reduce the complexity of the problem by formulating fetch-and-carry planning for a Mixed Integer Linear Programming (MILP) problem. The proposed method can obtain the sequence of actions and the movement of the mobile manipulator in less than one second in many cases. The effectiveness of the proposed method is verified in environments in which delivery objects and obstacles are placed in various patterns, using a robot with an omnidirectional mobile platform and a serial link manipulator.

Task-Dependent Comfort Zone, a Base Placement Strategy for Mobile Manipulators Based on Manipulability Measures

The present contribution introduces the task-dependent comfort zone as a base placement strategy for mobile manipulators using different manipulability measures. Four different manipulability measures depending on end-effector velocities, forces, stiffness, and accelerations are considered. By evaluating a discrete subspace of the manipulator workspace with these manipulability measures and using image-processing algorithms, a suitable goal position for the autonomous mobile manipulator was defined within the comfort zone. This always ensures a certain manipulator manipulablity value with a lower limit with respect to the maximum possible manipulability in the discrete subspace. Results are shown for three different mobile manipulators using the velocity-dependent manipulability measure in a simulation.

PRF: A Program Reuse Framework for Automated Programming by Learning from Existing Robot Programs

This paper explores the problem of automated robot program generation from limited historical data when neither accurate geometric environmental models nor online vision feedback are available. The Program Reuse Framework (PRF) is developed, which uses expert-defined motion classes, a novel data structure introduced in this work, to learn affordances, workspaces, and skills from historical data. Historical data comprise raw robot joint trajectories and descriptions of the robot task being completed. Given new tasks, motion classes are then used again to formulate an optimization problem capable of generating new open-loop, skill-based programs to complete the tasks. To cope with a lack of geometric models, a technique to learn safe workspaces from demonstrations is developed, allowing the risk of new programs to be estimated before execution. A new learnable motion primitive for redundant manipulators is introduced, called a redundancy dynamical movement primitive, which enables new end-effector goals to be reached while mimicking the whole-arm behavior of a demonstration. A mobile manipulator part transportation task is used throughout to illustrate each step of the framework.

Multiple Self-Powered Sensor-Integrated Mobile Manipulator for Intelligent Environment Detection.

A multiple self-powered sensor-integrated mobile manipulator (MSIMM) system was proposed to address challenges in existing exploration devices, such as the need for a constant energy supply, limited variety of sensed information, and difficult human-computer interfaces. The MSIMM system integrates triboelectric nanogenerator (TENG)-based self-powered sensors, a bionic manipulator, and wireless gesture control, enhancing sensor data usability through machine learning. Specifically, the system includes a tracked vehicle platform carrying the manipulator and electronics, including a storage battery and a microcontroller unit (MCU). An integrated sensor glove and terminal application (APP) enable intuitive manipulator control, improving human-computer interaction. The system responds to and analyzes various environmental stimuli, including the droplet and fall height, temperature, pressure, material type, angles, angular velocity direction, and acceleration amplitude and direction. The manipulator, fabricated using 3D printing technology, integrates multiple sensors that generate electrical signals through the triboelectric effect of mechanical motion. These signals are classified using convolutional neural networks for accurate environmental monitoring. Our database shows signal recognition and classification accuracy exceeding 94%, with specific accuracies of 100% for pressure sensors, 99.55% for angle sensors, and 98.66, 95.91, 96.27, and 94.13% for material, droplet, temperature, and acceleration sensors, respectively.