Mobile Manipulator Research Articles

MSOA: A modular service-oriented architecture to integrate mobile manipulators as cyber-physical systems

MSOA: A modular service-oriented architecture to integrate mobile manipulators as cyber-physical systems

Modeling Positions and Orientations of Cantaloupe Flowers for Automatic Pollination

An automatic system for cantaloupe flower pollination in greenhouses is proposed to meet the requirements of automatic pollination. The system consists of a mobile platform, robotic manipulator, and camera that reaches the flowers to detect and recognise their external features. The main task of the vision system is to detect the position and orientation of the flower in Cartesian coordinates, allowing the manipulator to reach the pose and perform pollination. A comprehensive method to ensure the accuracy of the pollination process is proposed that accurately determines the position and orientation of cantaloupe flowers in real environments. The vision system is used to capture images, detect the flower, and recognise its state according to its external features, such as size, colour, and shape, thereby providing appropriate nozzle access during pollination. The proposed approach begins with a segmentation method designed to precisely locate and segment the target cantaloupe flowers. Subsequently, a mathematical model is used to determine the key points that are important for establishing the growth orientation of each flower. Finally, an inverse-projection method is employed to convert the position of the flower from a two-dimensional (2D) image into a three-dimensional (3D) space, providing the necessary position for the pollination robot. The experimental process is conducted in a laboratory and proves the efficacy of the cantaloupe flower segmentation method, yielding precision, recall, and F1 scores of 87.91%, 90.76%, and 89.31%, respectively. Furthermore, the accuracy of the growth-orientation prediction method reaches approximately 86.7%. Notably, positional errors in 3D space predominantly fall within the allowable range, resulting in a successful pollination rate of up to 83.1%.

Visual Servoing Architecture of Mobile Manipulators for Precise Industrial Operations on Moving Objects

Although the use of articulated robots and AGVs is common in many industrial sectors such as automotive or aeronautics, the use of mobile manipulators is not widespread nowadays. Even so, the majority of applications separate the navigation and manipulation tasks, avoiding simultaneous movements of the platform and arm. The capability to use mobile manipulators to perform operations on moving objects would open the door to new applications such as the riveting or screwing of parts transported by conveyor belts or AGVs. This paper presents a novel position-based visual servoing (PBVS) architecture for mobile manipulators for precise industrial operations on moving parts. The proposed architecture includes a state machine to guide the process through the different phases of the task to ensure its correct execution. The approach has been validated in an industrial environment for screw-fastening operations, obtaining promising results and metrics.

Robust Hybrid Visual Servoing of Omnidirectional Mobile Manipulator With Kinematic Uncertainties Using a Single Camera.

A robust hybrid visual servoing (HVS) method for a single camera-mounted omnidirectional mobile manipulator (OMM) with kinematic uncertainties caused by slipping is proposed herein. Most existing studies pertaining to the visual servoing of mobile manipulators do not consider the kinematic uncertainties and the singularity of the manipulator that can occur during actual operations; furthermore, they required external sensors other than a single camera. In this study, the kinematics of an OMM are modeled considering kinematic uncertainties. Accordingly, an integral sliding-mode observer (ISMO) is designed to estimate the kinematic uncertainties. Subsequently, an integral sliding-mode control (ISMC) law is proposed to achieve robust visual servoing using the estimates of the ISMO. Additionally, an ISMO-ISMC-based HVS method is proposed to address the singularity issue of the manipulator; this method guarantees both robustness and finite-time stability in the presence of kinematic uncertainties. Overall, the entire visual servoing task is performed using only a single camera attached to the end effector without any other external sensors, unlike in previous studies. The stability and performance of the proposed method are verified numerically and experimentally in a slippery environment that generates kinematic uncertainties.

Enhancing human–robot collaborative transportation through obstacle-aware vibrotactile warning and virtual fixtures

Transporting large and heavy objects can benefit from Human–Robot Collaboration (HRC), increasing the contribution of robots to our daily tasks and addressing challenges arising from labor shortages. This strategy typically positions the human collaborator as the leader, with the robot assuming the follower role. However, when transporting large objects, the operator’s situational awareness can be compromised as the objects may occlude different parts of the environment, weakening the human leader’s decision-making capacity and leading to failure due to collision. This paper proposes a situational awareness framework for collaborative transportation to face this challenge. The framework integrates a multi-modal haptic-based Obstacle Feedback Module with two units. The first unit consists of a warning module that alerts the operator through a haptic belt with four vibrotactile devices that provide feedback about the location and proximity of the obstacles. The second unit implements virtual fixtures as hard constraints for mobility. The warning feedback and the virtual fixtures act online based on the information given by two Lidars mounted on a mobile manipulator to detect the obstacles in the surroundings. By enhancing the operator’s awareness of the environment, the proposed module improves the safety of the human–robot team in collaborative transportation scenarios by preventing collisions. Experiments with 16 non-expert subjects in four feedback modalities during four scenarios report an objective evaluation thanks to quantitative metrics and subjective evaluations based on user-level experiences. The results reveal the strengths and weaknesses of the implemented feedback modalities while providing solid evidence of the increased situational awareness of the operator when the two haptic units are employed.

The dynamics of vertical coalescence of acoustically levitated droplets

Mobility manipulation of liquid droplets is an important task of digital imicrofluidics. Acoustic levitation has revolutionised the contactless manipulation of liquid droplets for various applications. Acoustic levitation technique can be effectively used to manipulate droplets to obtain their coalescence. This paper reports a unique, versatile, and material-independent approach for the vertical coalescence of the droplets suspended in an acoustic levitator. The acoustic power of the levitator is carefully engineered to obtain vertical coalescence of two liquid droplets. Water, 20% and 40% glycerol–water solutions are used as the working liquids. The results of the experiments revealed three outcomes during the coalescence. The outcomes are analysed and discussed.

Trajectory Tracking Control of Mobile Manipulator Based on Improved Sliding Mode Control Algorithm

Research on trajectory tracking control for climbing welding robots holds significant importance in the field of automated welding. However, existing trajectory tracking methods suffer from issues such as jitter and slow speed. In this paper, an improved sliding mode control strategy is proposed based on the self-designed wall-climbing welding mobile manipulator. Firstly, a new adaptive sliding mode control strategy is proposed for the mobile platform based on the kinematic model. By introducing a new approach law, the controller is designed when the distance between the center of mass is unknown. Secondly, regarding the manipulator, we analyze simplified dynamic equations, extract uncertain components, and utilize a CNN for compensation. This compensation strategy is integrated into the sliding mode control law, achieving precise control over the manipulator and effectively resolving issues like slow tracking speeds, large errors, and chattering. The stability of the robot control system is proved by the Lyapunov function. Through simulation analysis and experimental validation, the proposed control method is confirmed to be feasible and superior.

Position control of input-delayed mobile manipulator

Position control of input-delayed mobile manipulator

Bimanual Telemanipulation Framework Utilising Multiple Optically Localised Cooperative Mobile Manipulators

Bimanual manipulation is valuable for its potential to provide robots in the field with increased capabilities when interacting with environments, as well as broadening the number of possible manipulation actions available. However, for a robot to perform bimanual manipulation, the system must have a capable control framework to localise and generate trajectories and commands for each sub-system to allow for successful cooperative manipulation as well as sufficient control over each individual sub-system. The proposed method suggests using multiple mobile manipulator platforms coupled through the use of an optical tracking localisation method to act as a single bimanual manipulation system. The framework’s performance relies on the accuracy of the localisation. As commands are primarily high-level, it is possible to use any number and combination of mobile manipulators and fixed manipulators within this framework. We demonstrate the functionality of this system through tests in a Pybullet simulation environment using two different omnidirectional mobile manipulators, as well a real-life experiment using two quadrupedal manipulators.

An Object Deformation-Agnostic Framework for Human–Robot Collaborative Transportation

In this study, an adaptive object deformability-agnostic human-robot collaborative transportation framework is presented. The proposed framework enables to combine the haptic information transferred through the object with the human kinematic information obtained from a motion capture system to generate reactive whole-body motions on a mobile collaborative robot. Furthermore, it allows rotating the objects in an intuitive and accurate way during co-transportation based on an algorithm that detects the human rotation intention using the torso and hand movements. First, we validate the framework with the two extremities of the object deformability range (i.e., purely rigid aluminum rod and highly deformable rope) by utilizing a mobile manipulator which consists of an Omni-directional mobile base and a collaborative robotic arm. Next, its performance is compared with an admittance controller during a co-carry task of a partially deformable object in a 12-subjects user study. Quantitative and qualitative results of this experiment show that the proposed framework can effectively handle the transportation of objects regardless of their deformability and provides intuitive assistance to human partners. Finally, we have demonstrated the potential of our framework in a different scenario, where the human and the robot co-transport a manikin using a deformable sheet. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Transportation of objects which requires the cooperation of multiple partners, is a common task in industrial settings such as factories and warehouses. The existing human-robot collaboration solutions for this task have focused only on purely rigid objects, although deformable objects need to be carried frequently in real-world applications. In this paper, we introduce a human-robot collaborative transportation framework that can handle objects with different deformability ranging from purely rigid to highly deformable. In particular, the proposed framework generates whole-body movements on a mobile collaborative robot by combining of the haptic information transmitted through the object and the human motion information obtained from a motion capture system. Moreover, the framework includes an intuitive way to rotate the object during the execution based on human hand and torso motion. The results of the experiments where objects with various deformability characteristics were transported in collaboration with a mobile manipulator demonstrated the high potential of the proposed approach in a laboratory setting. In the future, we plan to employ a less expensive vision-based human motion tracking system instead of the IMU-based system used in this study. With this change, we will be able to eliminate the need for wearable sensors from the framework presented, which would enhance its usability in real-world scenarios.

A Biobjective Scheme for Kinematic Control of Mobile Robotic Arms With Manipulability Optimization

A Biobjective Scheme for Kinematic Control of Mobile Robotic Arms With Manipulability Optimization

Exploring Saliency for Learning Sensory-Motor Contingencies in Loco-Manipulation Tasks

The objective of this paper is to propose a framework for a robot to learn multiple Sensory-Motor Contingencies from human demonstrations and reproduce them. Sensory-Motor Contingencies are a concept that describes intelligent behavior of animals and humans in relation to their environment. They have been used to design control and planning algorithms for robots capable of interacting and adapting autonomously. However, enabling a robot to autonomously develop Sensory-Motor Contingencies is challenging due to the complexity of action and perception signals. This framework leverages tools from Learning from Demonstrations to have the robot memorize various sensory phases and corresponding motor actions through an attention mechanism. This generates a metric in the perception space, used by the robot to determine which sensory-motor memory is contingent to the current context. The robot generalizes the memorized actions to adapt them to the present perception. This process creates a discrete lattice of continuous Sensory-Motor Contingencies that can control a robot in loco-manipulation tasks. Experiments on a 7-dof collaborative robotic arm with a gripper, and on a mobile manipulator demonstrate the functionality and versatility of the framework.

Development of 3D printed mecanum mobile manipulator and task-space control

Development of 3D printed mecanum mobile manipulator and task-space control

Adaptive robust motion/force control of constrained mobile manipulators using RBF neural network

Adaptive robust motion/force control of constrained mobile manipulators using RBF neural network

A novel redundant cooperative control strategy for robotic pollination

This paper proposes a novel Redundant Cooperative Control (RCC) strategy to address the pollination sequence planning problem for a mobile pollination robot in the greenhouse. Firstly, the RCC strategy divides the robot’s workspace into subspaces based on the distribution of flowers through clustering. Subsequently, the RCC strategy plans the traveling salesman problem within the visual field of these subspaces, seeking a compromise solution for an optimal path. Finally, the RCC strategy utilizes the redundant motion capabilities of the mobile chassis and robotic arm to accomplish the pollination task during the movement. The RCC strategy exploits the available redundancy in both the agricultural robot’s mobile platform and end-effector, carefully balancing computational complexity and path quality. To validate the effectiveness of the RCC strategy, we designed a series of experiments conducted on our pollination robot platform, measuring different strategies. The results demonstrate that the RCC strategy achieves an average pollination rate of 7.5 s per flower, representing a 36.4% improvement compared to the most common intermittence strategy. Furthermore, the RCC strategy can be extended to tasks such as harvesting and pruning, significantly enhancing the efficiency of robot task execution even in industrial and service sectors.

Digital Twin and Deep Reinforcement Learning-Driven Robotic Automation System for Confined Workspaces: A Nozzle Dam Replacement Case Study in Nuclear Power Plants

Robotic automation has emerged as a leading solution for replacing human workers in dirty, dangerous, and demanding industries to ensure the safety of human workers. However, practical implementation of this technology remains limited, requiring substantial effort and costs. This study addresses the challenges specific to nuclear power plants, characterized by hazardous environments and physically demanding tasks such as nozzle dam replacement in confined workspaces. We propose a digital twin and deep-reinforcement-learning-driven robotic automation system with an autonomous mobile manipulator. The study follows a four-step process. First, we establish a simplified testbed for a nozzle dam replacement task and implement a high-fidelity digital twin model of the real-world testbed. Second, we employ a hybrid visual perception system that combines deep object pose estimation and an iterative closest point algorithm to enhance the accuracy of the six-dimensional pose estimation. Third, we use a deep-reinforcement-learning method, particularly the proximal policy optimization algorithm with inverse reachability map, and a centroidal waypoint strategy, to improve the controllability of an autonomous mobile manipulator. Finally, we conduct pre-performed simulations of the nozzle dam replacement in the digital twin and evaluate the system on a robot in the real-world testbed. The nozzle dam replacement with precise object pose estimation, navigation, target object grasping, and collision-free motion generation was successful. The robotic automation system achieved a 92.0%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$92.0\\%$$\\end{document} success rate in the digital twin. Our proposed method can improve the efficiency and reliability of robotic automation systems for extreme workspaces and other perilous environments.

Study of the Influence of the Mecanum-Wheel Design and Contact Forces on the Control Accuracy of Mobile Robot-Manipulator KUKA YouBot

Study of the Influence of the Mecanum-Wheel Design and Contact Forces on the Control Accuracy of Mobile Robot-Manipulator KUKA YouBot

Reactive Base Control for On-the-Move Mobile Manipulation in Dynamic Environments

Reactive Base Control for On-the-Move Mobile Manipulation in Dynamic Environments

Toward coordinated planning and hierarchical optimization control for highly redundant mobile manipulator

This paper represents a constraint planning and optimization control scheme for a highly redundant mobile manipulator considering a complex indoor environment. Compared with the traditional optimization solution of a redundant manipulator, infinity norm and slack variable are additionally introduced and leveraged by the optimization algorithm. The former takes into account the joint limits effectively by considering individual joint velocities and the latter relaxes the equality constraint by decreasing the infeasible solution area. By using derived kinematic equations, the tracking control problem is expressed as an optimization problem and converted into a new quadratic programming (QP) problem. To address the optimization problem, the two-timescale recurrent neural networks optimization scheme is proposed and tested with a 9 DOFs nonholonomic mobile-based manipulator. Additionally, the BI2RRT∗ path-planning algorithm incorporates path planning in the complex environment where different obstacles are positioned. To test and evaluate the proposed optimization scheme, both predefined and generated paths are tested in the Neurorobotics Platform (NRP) 22https://neurorobotics.net.which is open access and open source integrative simulation framework powered by Gazebo and developed by our team.

A heavy-load wall-climbing robot for bridge concrete structures inspection

Purpose The purpose of this paper is to present a wall-climbing robot platform for heavy-load with negative pressure adsorption, which could be equipped with a six-degree of freedom (DOF) collaborative robot (Cobot) and detection device for inspecting the overwater part of concrete bridge towers/piers for large bridges. Design/methodology/approach By analyzing the shortcomings of existing wall-climbing robots in detecting concrete structures, a wall-climbing mobile manipulator (WCMM), which could be compatible with various detection devices, is proposed for detecting the concrete towers/piers of the Hong Kong-Zhuhai-Macao Bridge. The factors affecting the load capacity are obtained by analyzing the antislip and antioverturning conditions of the wall-climbing robot platform on the wall surface. Design strategies for each part of the structure of the wall-climbing robot are provided based on the influencing factors. By deriving the equivalent adsorption force equation, analyzed the influencing factors of equivalent adsorption force and provided schemes that could enhance the load capacity of the wall-climbing robot. Findings The adsorption test verifies the maximum negative pressure that the fan module could provide to the adsorption chamber. The load capacity test verifies it is feasible to achieve the expected bearing requirements of the wall-climbing robot. The motion tests prove that the developed climbing robot vehicle could move freely on the surface of the concrete structure after being equipped with a six-DOF Cobot. Practical implications The development of the heavy-load wall-climbing robot enables the Cobot to be installed and equipped on the wall-climbing robot, forming the WCMM, making them compatible with carrying various devices and expanding the application of the wall-climbing robot. Originality/value A heavy-load wall-climbing robot using negative pressure adsorption has been developed. The wall-climbing robot platform could carry a six-DOF Cobot, making it compatible with various detection devices for the inspection of concrete structures of large bridges. The WCMM could be expanded to detect the concretes with similar structures. The research and development process of the heavy-load wall-climbing robot could inspire the design of other negative-pressure wall-climbing robots.