Autonomous Robot Operating Research Articles

Research Progress on Autonomous Operation of Industrial Robots Based on Big Data and Machine Vision

Research Progress on Autonomous Operation of Industrial Robots Based on Big Data and Machine Vision

Bausteinkasten zur Einführung flexibler mobiler Robotik

Abstract Autonomous mobile robots are a key factor in maintaining the competitiveness of companies in Germany. However, their implementation and operation in brownfield environments pose various challenges for small and medium-sized enterprises. The FlexTools research project aims to help these companies with a modular toolbox to facilitate the customization and operation of autonomous mobile robots.

Dynamic pickup and delivery problem for autonomous delivery robots in an airport terminal

Autonomous delivery robot (ADR) operation for short-range delivery purposes is becoming an increasingly popular mode of service. Assigning orders to robots is critical in ensuring high quality in these automated delivery operations. This paper considers a dynamic pickup and delivery problem using autonomous robots (DPDP-AR), where a fleet of ADRs picks up desired items from stores and delivers them to customers. The arrival time of orders is uncertain, with a hard delivery deadline for each order. This study considered the battery consumption of ADRs, which has been less extensively covered in previous research, and devised a battery recharging strategy for the operation. To handle this stochasticity and dynamism in the problem, we developed a reassignment algorithm that reschedules previously assigned orders at the arrival of a new order. Additionally, as periods of high demand can be estimated, peak time management of the battery is proposed to enhance ADR utilization at peak periods. Computational experiments were performed on real-world ADR food delivery service instances in an airport terminal. Test instances on various demand scenarios demonstrated improvement in service quality when devised policies were used compared to current practice. We substantiated that the proposed algorithms found efficient solutions within short computation times, validating their applicability to real-world operations.

Enhancing Autonomous Driving Robot Systems with Edge Computing and LDM Platforms

The efficient operation and interaction of autonomous robots play crucial roles in various fields, e.g., security, environmental monitoring, and disaster response. For these purposes, processing large volumes of sensor data and sharing data between robots is essential; however, processing such large data in an on-device environment for robots results in substantial computational resource demands, causing high battery consumption and heat issues. Thus, this study addresses challenges related to processing large volumes of sensor data and the lack of dynamic object information sharing among autonomous robots and other mobility systems. To this end, we propose an Edge-Driving Robotics Platform (EDRP) and a Local Dynamic Map Platform (LDMP) based on 5G mobile edge computing and Kubernetes. The proposed EDRP implements the functions of autonomous robots based on a microservice architecture and offloads these functions to an edge cloud computing environment. The LDMP collects and shares information about dynamic objects based on the ETSI TR 103 324 standard, ensuring cooperation among robots in a cluster and compatibility with various Cooperative-Intelligent Transport System (C-ITS) components. The feasibility of operating a large-scale autonomous robot offloading system was verified in experimental scenarios involving robot autonomy, dynamic object collection, and distribution by integrating real-world robots with an edge computing–based offloading platform. Experimental results confirmed the potential of dynamic object collection and dynamic object information sharing with C-ITS environment components based on LDMP.

Autonomous navigation method based on RGB‐D camera for a crop phenotyping robot

AbstractPhenotyping robots have the potential to obtain crop phenotypic traits on a large scale with high throughput. Autonomous navigation technology for phenotyping robots can significantly improve the efficiency of phenotypic traits collection. This study developed an autonomous navigation method utilizing an RGB‐D camera, specifically designed for phenotyping robots in field environments. The PP‐LiteSeg semantic segmentation model was employed due to its real‐time and accurate segmentation capabilities, enabling the distinction of crop areas in images captured by the RGB‐D camera. Navigation feature points were extracted from these segmented areas, with their three‐dimensional coordinates determined from pixel and depth information, facilitating the computation of angle deviation (α) and lateral deviation (d). Fuzzy controllers were designed with α and d as inputs for real‐time deviation correction during the walking of phenotyping robot. Additionally, the method includes end‐of‐row recognition and row spacing calculation, based on both visible and depth data, enabling automatic turning and row transition. The experimental results showed that the adopted PP‐LiteSeg semantic segmentation model had a testing accuracy of 95.379% and a mean intersection over union of 90.615%. The robot's navigation demonstrated an average walking deviation of 1.33 cm, with a maximum of 3.82 cm. Additionally, the average error in row spacing measurement was 2.71 cm, while the success rate of row transition at the end of the row was 100%. These findings indicate that the proposed method provides effective support for the autonomous operation of phenotyping robots.

Road Anomaly Detection with Unknown Scenes Using DifferNet-Based Automatic Labeling Segmentation

Obstacle avoidance is essential for the effective operation of autonomous mobile robots, enabling them to detect and navigate around obstacles in their environment. While deep learning provides significant benefits for autonomous navigation, it typically requires large, accurately labeled datasets, making the data’s preparation and processing time-consuming and labor-intensive. To address this challenge, this study introduces a transfer learning (TL)-based automatic labeling segmentation (ALS) framework. This framework utilizes a pretrained attention-based network, DifferNet, to efficiently perform semantic segmentation tasks on new, unlabeled datasets. DifferNet leverages prior knowledge from the Cityscapes dataset to identify high-entropy areas as road obstacles by analyzing differences between the input and resynthesized images. The resulting road anomaly map was refined using depth information to produce a robust drivable area and map of road anomalies. Several off-the-shelf RGB-D semantic segmentation neural networks were trained using pseudo-labels generated by the ALS framework, with validation conducted on the GMRPD dataset. Experimental results demonstrated that the proposed ALS framework achieved mean precision, mean recall, and mean intersection over union (IoU) rates of 80.31%, 84.42%, and 71.99%, respectively. The ALS framework, through the use of transfer learning and the DifferNet network, offers an efficient solution for semantic segmentation of new, unlabeled datasets, underscoring its potential for improving obstacle avoidance in autonomous mobile robots.

Construction of Three-Dimensional Semantic Maps of Unstructured Lawn Scenes Based on Deep Learning

Traditional automatic gardening pruning robots generally employ electronic fences for the delineation of working boundaries. In order to quickly determine the working area of a robot, we combined an improved DeepLabv3+ semantic segmentation model with a simultaneous localization and mapping (SLAM) system to construct a three-dimensional (3D) semantic map. To reduce the computational cost of its future deployment in resource-constrained mobile robots, we replaced the backbone network of DeepLabv3+, ResNet50, with MobileNetV2 to decrease the number of network parameters and improve recognition speed. In addition, we introduced an efficient channel attention network attention mechanism to enhance the accuracy of the neural network, forming an improved Multiclass MobileNetV2 ECA DeepLabv3+ (MM-ED) network model. Through the integration of this model with the SLAM system, the entire framework was able to generate a 3D semantic point cloud map of a lawn working area and convert it into octree and occupancy grid maps, providing technical support for future autonomous robot operation and navigation. We created a lawn dataset containing 7500 images, using our own annotated images as ground truth. This dataset was employed for experimental purposes. Experimental results showed that the proposed MM-ED network model achieved 91.07% and 94.71% for MIoU and MPA metrics, respectively. Using a GTX 3060 Laptop GPU, the frames per second rate reached 27.69, demonstrating superior recognition performance compared to similar semantic segmentation architectures and better adaptation to SLAM systems.

Obstacle Avoidance and Path Planning Methods for Autonomous Navigation of Mobile Robot.

Path planning creates the shortest path from the source to the destination based on sensory information obtained from the environment. Within path planning, obstacle avoidance is a crucial task in robotics, as the autonomous operation of robots needs to reach their destination without collisions. Obstacle avoidance algorithms play a key role in robotics and autonomous vehicles. These algorithms enable robots to navigate their environment efficiently, minimizing the risk of collisions and safely avoiding obstacles. This article provides an overview of key obstacle avoidance algorithms, including classic techniques such as the Bug algorithm and Dijkstra's algorithm, and newer developments like genetic algorithms and approaches based on neural networks. It analyzes in detail the advantages, limitations, and application areas of these algorithms and highlights current research directions in obstacle avoidance robotics. This article aims to provide comprehensive insight into the current state and prospects of obstacle avoidance algorithms in robotics applications. It also mentions the use of predictive methods and deep learning strategies.

Developments of a centimeter-level precise muometric wireless navigation system (MuWNS-V) and its first demonstration using directional information from tracking detectors

Various positioning techniques such as Wi-Fi positioning system have been proposed to use in situations where satellite navigation is unavailable. One such system, the muometric positioning system (muPS), was invented for navigation which operates in locations where even radio waves cannot reach such as underwater or underground. muPS takes advantage of a key feature of its probe, cosmic-ray muons, which travel straightforwardly at almost a speed of light in vacuum regardless of the matter they traverse. Similar to other positioning techniques, muPS is a technique to determine the position of a client’s muPS receiver within the coordinate defined by reference detectors. This can be achieved either by using time-of-flight (ToF) or angle of arrival (AoA) measurements. The latter configuration (AoA), called the Vector-muPS has recently been invented and the present paper describes the developments of the first prototype of a vector muometric wireless navigation system (MuWNS-V) with this new vector-muPS concept and its demonstration. With MuWNS-V, the reference tracker and the receiver ran wirelessly with fully independent readout systems, and a positioning accuracy of 3.9 cm (RMS) has been achieved. We also evaluated the outcome of measuring continuous indoor localization of a moving receiver with this prototype. Our results indicated that further improvements in positioning accuracy will be attainable by acquiring higher angular resolution of the reference trackers. It is anticipated that “sub-cm level” navigation will be possible for muPS which could be applied to many situations such as future autonomous mobile robot operations.

Energy- and Safety-Aware Operation of Battery-Powered Autonomous Robots in Agriculture

Energy- and Safety-Aware Operation of Battery-Powered Autonomous Robots in Agriculture

Autonomous Dynamic Hitch-Hiking Control of a Bionic Robotic Remora

Remora suckerfish and its hitch-hiking behavior bring enormous inspiration into the engineering field. In this paper, a bioinspired hitch-hiking behavior is accomplished automatically by a robotic remora, which creates possibilities for prolonging its endurance and multi-robot cooperation. First, the definition of the hitch-hiking task and the mechatronic design of robotic remora are introduced. Then, aiming at the hitch-hiking task, the LED-marker-based underwater visual localization method and planar state synchronization controller are developed. The localization method includes a complete framework from LED detection to marker pose optimization, which considers the refraction correction to improve the localization precision in water. The synchronization controller is decomposed into the lateral and longitudinal sub-controllers to overcome the challenges caused by underactuated dynamic. Besides, a finite state machine is designed to model the state and action transition during the hitch-hiking task. Extensive experimental results demonstrate the effectiveness of the proposed method. The autonomous hitch-hiking task toward a moving host is successfully implemented by a robotic fish for the first time. Such results may offer valuable insights into the future autonomous operation of underwater robots.

Uncertainty analysis of autonomous delivery robot operations for last-mile logistics in European cities

Although autonomous delivery robots (ADRs) are widely anticipated to significantly enhance the efficiency of last-mile logistics operations in dense urban environments in the coming years, their impact on logistics service providers’ supply chains has yet to be accurately assessed on a large scale. The primary objective of this article is to quantify the uncertainty in the cost of ADR operations as a function of the stochastic behavior of given input variables. First, ADR operations are modeled using the continuous approximation methodology. The mathematical formulations proposed in this article relate certain ADR and service area input parameters to provide an estimate of the carrier’s last-mile operating costs. An uncertainty analysis based on the Monte Carlo approach is then performed. The numerical results indicate that the implementation of a two-echelon delivery scheme with heavy-duty vehicles cooperating with ADRs through an urban logistics micro-hub would, on average, reduce last-mile operating costs by more than 10% in small and medium-sized European cities, given adequate operational conditions.

A Method for Industrial Robots to Grasp and Detect Parts of Instrument under 3D Visual Guidance

<p>Guiding industrial robots to complete grasping tasks through machine vision is an important part of achieving autonomous robot operation. This article explores the control method of industrial robots under 3D vision, focusing on the feature that two-dimensional vision can only perform color and pose recognition but lacks depth recognition. Firstly, a high-precision point cloud registration calibration matrix solution method is proposed. Then, an improved recognition model is designed to address the issue of how vision guides robots to grasp and detect. This model integrates feature extraction and object detection modules, and describes the parameters of each module. Finally, the effectiveness of the proposed method is verified in the assembly scene of gas instruments. Finally, experimental results show that, the method proposed in this article can limit the grasping accuracy to within 2 millimeters in guiding robots to grasp detection scenes, achieving the expected effect.</p> <p> </p>

Digitization of Accounting in the Innovative Management of Autonomous Robotic Transport

The digitization of economic processes is advancing across all sectors, contributing to the development of Industry 5.0. A key element of this fifth industrial revolution is the activation of robotic economic activity. Recently, advancements in autonomous robotic transport have been implemented in practice. However, both the practical application of unmanned vehicles and scientific developments in this field have shown low efficiency in the implementation of projects for the autonomous transportation of goods and passengers. This inefficiency stems from insufficient attention to the accounting and management aspects of autonomous robotic transport operations. The scientific and practical novelty of this study lies in improving accounting and management practices in the context of digitalization, specifically by addressing the fundamental transformations in economic processes caused by the use of autonomous vehicles. The key organizational factors influencing accounting for robotic transport operations include the type of transported objects, fuel and energy resource consumption, human involvement, the capacity and number of goods (or passengers) transported at one time, continuous operation, maintainability, software update capabilities, autonomous interaction with other transport means, and communication and information sharing with customers of transport services. A method for digitizing the accounting of fuel and energy costs, personnel wages, social activity deductions, depreciation, operational costs, and other costs related to the functioning of autonomous robotic transport has been developed. This method leverages IoT data and considers the organizational prerequisites mentioned. The use of two-dimensional calculation units, such as "kilogramkilometre" and "passenger-kilometre" units, for the digitalization of cost calculations for passenger and cargo transportation via autonomous robotic transport has been proposed. Additionally, the procedure for determining the cost of transport services for end users and the formation of information arrays for the innovative management of transport enterprises has been refined. The elimination of organizational restrictions in managing autonomous transport operations, alongside the need for information synchronization between transport enterprises and other business entities within the information ecosystem of a smart city, highlights future research prospects in this area.

Visual Navigation and Obstacle Avoidance Control for Agricultural Robots via LiDAR and Camera

Obstacle avoidance control and navigation in unstructured agricultural environments are key to the safe operation of autonomous robots, especially for agricultural machinery, where cost and stability should be taken into account. In this paper, we designed a navigation and obstacle avoidance system for agricultural robots based on LiDAR and a vision camera. The improved clustering algorithm is used to quickly and accurately analyze the obstacle information collected by LiDAR in real time. Also, the convex hull algorithm is combined with the rotating calipers algorithm to obtain the maximum diameter of the convex polygon of the clustered data. Obstacle avoidance paths and course control methods are developed based on the danger zones of obstacles. Moreover, by performing color space analysis and feature analysis on the complex orchard environment images, the optimal H-component of HSV color space is selected to obtain the ideal vision-guided trajectory images based on mean filtering and corrosion treatment. Finally, the proposed algorithm is integrated into the Three-Wheeled Mobile Differential Robot (TWMDR) platform to carry out obstacle avoidance experiments, and the results show the effectiveness and robustness of the proposed algorithm. The research conclusion can achieve satisfactory results in precise obstacle avoidance and intelligent navigation control of agricultural robots.

Path Planning of a Mobile Delivery Robot Operating in a Multi-Story Building Based on a Predefined Navigation Tree.

Planning the path of a mobile robot that must transport and deliver small packages inside a multi-story building is a problem that requires a combination of spatial and operational information, such as the location of origin and destination points and how to interact with elevators. This paper presents a solution to this problem, which has been formulated under the following assumptions: (1) the map of the building's floors is available; (2) the position of all origin and destination points is known; (3) the mobile robot has sensors to self-localize on the floors; (4) the building is equipped with remotely controlled elevators; and (5) all doors expected in a delivery route will be open. We start by defining a static navigation tree describing the weighted paths in a multi-story building. We then proceed to describe how this navigation tree can be used to plan the route of a mobile robot and estimate the total length of any delivery route using Dijkstra's algorithm. Finally, we show simulated routing results that demonstrate the effectiveness of this proposal when applied to an autonomous delivery robot operating in a multi-story building.

Actuator-level motion and contact episode learning and classification using adaptive resonance theory

Several methods exist to detect and distinguish collisions of robotic systems with their environment, since this information is a critical dependency of many tasks. These methods are prevalently based on thresholds in combination with filters, models, or offline trained machine learning models. To improve the adaptation and thereby enable a more autonomous operation of robots in new environments, this work evaluates the applicability of an incremental learning approach. The method addresses online learning and recognition of motion and contact episodes of robotic systems from proprioceptive sensor data using machine learning. The objective is to learn new category templates representing previously encountered situations of the actuators and improve them based on newly gathered similar data. This is achieved using an artificial neural network based on adaptive resonance theory (ART). The input samples from the robot’s actuator measurements are preprocessed into frequency spectra. This enables the ART neural network to learn incrementally recurring episodic patterns from these preprocessed data. An evaluation based on preliminary experimental data from a grasping motion of a humanoid robot’s arm encountering contacts is presented and suggests that this is a promising approach.

LD-SLAM: A Robust and Accurate GNSS-Aided Multi-Map Method for Long-Distance Visual SLAM

Continuous, robust, and precise localization is pivotal in enabling the autonomous operation of robots and aircraft in intricate environments, particularly in the absence of GNSS (global navigation satellite system) signals. However, commonly employed approaches, such as visual odometry and inertial navigation systems, encounter hindrances in achieving effective navigation and positioning due to issues of error accumulation. Additionally, the challenge of managing extensive map creation and exploration arises when deploying these systems on unmanned aerial vehicle terminals. This study introduces an innovative system capable of conducting long-range and multi-map visual SLAM (simultaneous localization and mapping) using monocular cameras equipped with pinhole and fisheye lens models. We formulate a graph optimization model integrating GNSS data and graphical information through multi-sensor fusion navigation and positioning technology. We propose partitioning SLAM maps based on map health status to augment accuracy and resilience in large-scale map generation. We introduce a multi-map matching and fusion algorithm leveraging geographical positioning and visual data to address excessive discrete mapping, leading to resource wastage and reduced map-switching efficiency. Furthermore, a multi-map-based visual SLAM online localization algorithm is presented, adeptly managing and coordinating distinct geographical maps in different temporal and spatial domains. We employ a quadcopter to establish a testing system and generate an aerial image dataset spanning several kilometers. Our experiments exhibit the framework’s noteworthy robustness and accuracy in long-distance navigation. For instance, our GNSS-assisted multi-map SLAM achieves an average accuracy of 1.5 m within a 20 km range during unmanned aerial vehicle flights.

Underwater Color-Cast Image Enhancement by Noise Suppression and Block Effect Elimination

Efficient underwater visual environment perception is the key to realizing the autonomous operation of underwater robots. Because of the complex and diverse underwater environment, the underwater images not only have different degrees of color cast but also produce a lot of noise. Due to the existence of noise in the underwater image and the blocking effect in the process of enhancing the image, the enhanced underwater image is still rough. Therefore, an underwater color-cast image enhancement method based on noise suppression and block effect elimination is proposed in this paper. Firstly, an automatic white balance algorithm for brightness and color balance is designed to correct the color deviation of underwater images and effectively restore the brightness and color of underwater images. Secondly, aiming at the problem of a large amount of noise in underwater images, a noise suppression algorithm for heat conduction matrix in the wavelet domain is proposed, which suppresses image noise and improves the contrast and edge detail information of underwater images. Thirdly, for the block effect existing in the process of enhancing the underwater color-cast image, a block effect elimination algorithm based on compressed domain boundary average is proposed, which eliminates the block effect in the enhancement process and balances the bright area and dark area in the image. Lastly, multi-scale image fusion is performed on the images after color correction, noise suppression, and block effect elimination, and finally, the underwater enhanced image with rich features is obtained. The results show that the proposed method is superior to other algorithms in color correction, contrast, and visibility. It also shows that the proposed method corrects the underwater color-cast image to a certain extent and effectively suppresses the noise and block effect of the underwater image, which provides theoretical support for underwater visual environment perception technology.

Semantic OcTree Mapping and Shannon Mutual Information Computation for Robot Exploration

Autonomous robot operation in unstructured and unknown environments requires efficient techniques for mapping and exploration using streaming range and visual observations. Information-based exploration techniques, such as Cauchy–Schwarz quadratic mutual information and fast Shannon mutual information, have successfully achieved active binary occupancy mapping with range measurements. However, as we envision robots performing complex tasks specified with semantically meaningful concepts, it is necessary to capture semantics in the measurements, map representation, and exploration objective. This work presents semantic octree mapping and Shannon mutual information computation for robot exploration. We develop a Bayesian multiclass mapping algorithm based on an octree data structure, where each voxel maintains a categorical distribution over semantic classes. We derive a closed-form efficiently computable lower bound of the Shannon mutual information between a multiclass octomap and a set of range-category measurements using semantic run-length encoding of the sensor rays. The bound allows rapid evaluation of many potential robot trajectories for autonomous exploration and mapping. We compare our method against state-of-the-art exploration techniques and apply it in a variety of simulated and real-world experiments.