Semi-structured Environment Research Articles

Six-Axis Robotic Gripping Control Algorithm Based on Deep Learning

Due to several complex factors such as the type, size, and shape of target object, vision-assisted robot grasping technology still faces serious challenges. In this research, a deep learning-based robot hand vision grasping algorithm was developed considering semi-structural environmental constraints. The proposed algorithm could build a deep learning network on the basis of the desired object, perform object recognition, category classification and position judgment, and complete robot hand-grasping tasks. The obtained experimental results demonstrated that the proposed algorithm effectively solved the problem of recognizing and classifying multi-category objects in a semi-structured environment, improving recognition rate and grasping rate and reducing collision rate.

Extended residual learning with one-shot imitation learning for robotic assembly in semi-structured environment.

Robotic assembly tasks require precise manipulation and coordination, often necessitating advanced learning techniques to achieve efficient and effective performance. While residual reinforcement learning with a base policy has shown promise in this domain, existing base policy approaches often rely on hand-designed full-state features and policies or extensive demonstrations, limiting their applicability in semi-structured environments. In this study, we propose an innovative Object-Embodiment-Centric Imitation and Residual Reinforcement Learning (OEC-IRRL) approach that leverages an object-embodiment-centric (OEC) task representation to integrate vision models with imitation and residual learning. By utilizing a single demonstration and minimizing interactions with the environment, our method aims to enhance learning efficiency and effectiveness. The proposed method involves three key steps: creating an object-embodiment-centric task representation, employing imitation learning for a base policy using via-point movement primitives for generalization to different settings, and utilizing residual RL for uncertainty-aware policy refinement during the assembly phase. Through a series of comprehensive experiments, we investigate the impact of the OEC task representation on base and residual policy learning and demonstrate the effectiveness of the method in semi-structured environments. Our results indicate that the approach, requiring only a single demonstration and less than 1.2 h of interaction, improves success rates by 46% and reduces assembly time by 25%. This research presents a promising avenue for robotic assembly tasks, providing a viable solution without the need for specialized expertise or custom fixtures.

Analytical Formalism for Data Representation and Object Detection with 2D LiDAR: Application in Mobile Robotics.

In mobile robotics, LASER scanners have a wide spectrum of indoor and outdoor applications, both in structured and unstructured environments, due to their accuracy and precision. Most works that use this sensor have their own data representation and their own case-specific modeling strategies, and no common formalism is adopted. To address this issue, this manuscript presents an analytical approach for the identification and localization of objects using 2D LiDARs. Our main contribution lies in formally defining LASER sensor measurements and their representation, the identification of objects, their main properties, and their location in a scene. We validate our proposal with experiments in generic semi-structured environments common in autonomous navigation, and we demonstrate its feasibility in multiple object detection and identification, strictly following its analytical representation. Finally, our proposal further encourages and facilitates the design, modeling, and implementation of other applications that use LASER scanners as a distance sensor.

A novel perception and semantic mapping method for robot autonomy in orchards

Agricultural robots must navigate challenging dynamic and semi-structured environments. Recently, environmental modelling using LiDAR-based SLAM has shown promise in providing highly accurate geometry. However, how this chaotic environmental information can be used to achieve effective robot automation in the agricultural sector remains unexplored. In this study, we propose a novel semantic mapping and navigation framework for achieving robotic autonomy in orchards. It consists of two main components: a semantic processing module and a navigation module. First, we present a novel 3D detection network architecture, 3D-ODN, which can accurately process object instance information from point clouds. Second, we develop a framework to construct the visibility map by incorporating semantic information and terrain analysis. By combining these two critical components, our framework is evaluated in a number of key horticultural production scenarios, including a robotic system for in-situ phenotyping and daily monitoring, and a selective harvesting system in apple orchards. The experimental results show that our method can ensure high accuracy in understanding the environment and enable reliable robot autonomy in agricultural environments.

Optimization-based local planner for a nonholonomic autonomous mobile robot in semi-structured environments

This paper proposes an optimization-based adaptive local planner to address the aggressive motion planning problem of an autonomous mobile robot in realistic semi-structured scenarios. The goal is to enable the robot to perform autonomous navigation tasks at the limit of its speed capability to maximize the efficiency, while ensuring safety and feasibility. The proposed approach constitutes an integrated scheme that leverages real-time hierarchical collision avoidance constraints reformulation and handles the heterogeneous constraints using a unified nonlinear optimization. An aggressiveness adaption method is employed to cope with disturbances instead of relying on excessive obstacle inflation, resulting in reduced conservatism while enhancing safety. The efficacy of the proposed method is demonstrated through simulation and experimental results. Compared with previous work, the proposed approach provides significant improvements in navigation success rate without sacrificing the task efficiency.

Когнитивная эргономика в образовательных системах: специфика взаимосвязей структуры «студент-учебная информация»

The paper reveals the essence of the “cognitive system” concept, considered in cognitive ergonomics as a multi-level system that ensures implementing all the main cognitive functions and stages of the learning process. The specificity of cognitive analysis is shown as a tool for studying unstable and semi-structured environment, which provides understanding the existing problems, identifying contradictions and a qualitative analysis of ongoing processes. The relevance of applying cognitive analysis principles to find approaches to the quality of the interaction pattern “human – information” in educational systems in teaching and professional activities is indicated. The results of the conducted cognitive analysis aimed at studying the specifics of the existing ergonomic relationships in the cognitive system “student – educational information” are analysed and discussed. Conclusions are drawn and recommendations are made on taking into account the identified relationships in the specified system when designing the learning process at all levels of education to develop a typology of individual profiles of cognitive systems “student – educational information” and search for an optimal strategy that ensures the effectiveness of the pedagogical interaction between the teacher and the student and the dynamics of the cognitive system development. The paper indicates the necessity for analysing and modelling educational systems based on the principles of cognitive management in the future.

Когнитивная эргономика в образовательных системах: специфика взаимосвязей структуры «студент-учебная информация»

The paper reveals the essence of the “cognitive system” concept, considered in cognitive ergonomics as a multi-level system that ensures implementing all the main cognitive functions and stages of the learning process. The specificity of cognitive analysis is shown as a tool for studying unstable and semi-structured environment, which provides understanding the existing problems, identifying contradictions and a qualitative analysis of ongoing processes. The relevance of applying cognitive analysis principles to find approaches to the quality of the interaction pattern “human – information” in educational systems in teaching and professional activities is indicated. The results of the conducted cognitive analysis aimed at studying the specifics of the existing ergonomic relationships in the cognitive system “student – educational information” are analysed and discussed. Conclusions are drawn and recommendations are made on taking into account the identified relationships in the specified system when designing the learning process at all levels of education to develop a typology of individual profiles of cognitive systems “student – educational information” and search for an optimal strategy that ensures the effectiveness of the pedagogical interaction between the teacher and the student and the dynamics of the cognitive system development. The paper indicates the necessity for analysing and modelling educational systems based on the principles of cognitive management in the future.

Collaborative robot dynamics with physical human–robot interaction and parameter identification with PINN

Collaborative robots are increasingly being used in dynamic and semi-structured environments because of their ability to perform physical Human–Robot Interaction (pHRI) to ensure safety. Therefore, it is crucial to model the dynamics of collaborative robots during pHRI to gain valuable insights into the system’s behavior when in contact with humans. In this work, a generic dynamic model is proposed for the quasi-static contact phase of pHRI, which considers the interaction dynamics and the complete structural dynamics of the collaborative robot. Moreover, a hybrid physics-informed neural network (PINN) is proposed, which utilizes a recurrent neural network (RNN) and the Runge–Kutta method to identify the joint dynamic parameters without complicated regressor construction. Experiments are conducted using a UR3e collaborative robot, and the PINN is trained using the acquired data. The results demonstrate the effectiveness of the PINN in identifying joint dynamics without prior knowledge, and the dynamic simulation of pHRI is consistent with the experimental results. The proposed model and PINN-based identification approach have the potential to improve safety and productivity in industrial environments by facilitating the control of pHRI.

Human in the Loop: What the Avatar XPrize Revealed About the Future of Telepresence Robots

It's still an achievement for autonomous robots to merely survive in the real world, which is a long way from any kind of useful generalized autonomy. Under some fairly specific constraints, autonomous robots are starting to find a few valuable niches in semistructured environments, like offices and hospitals and warehouses. But when it comes to the unstructured nature of disaster areas or human interaction, or really any situation that requires innovation and creativity, autonomous robots are often at a loss.

Localization of Mobile Manipulator in Vineyards for Autonomous Task Execution

Although robotic systems have found their place in agriculture, there are still many challenges, especially in the area of localization in semi-structured environments. A robotic system has been developed and tested to perform various tasks in the steep vineyards of the Mediterranean region. In this paper, we describe a method for vine trunk localization, based solely on the visual recognition of vine trunks by neural networks fed by an RGB camera. Assuming that the height of the first wire in the vineyard is known, the proposed method is used to determine the location of vines in the immediate vicinity of the all-terrain mobile manipulator—ATMM-VIV—needed for spraying and bud suckering. The experiment was conducted in a slightly inclined vineyard to evaluate the proposed localization method.

Topological Navigation for Autonomous Underwater Vehicles in Confined Semi-Structured Environments.

In this work, we present the design, implementation, and simulation of a topology-based navigation system for the UX-series robots, a spherical underwater vehicle designed to explore and map flooded underground mines. The objective of the robot is to navigate autonomously in the 3D network of tunnels of a semi-structured but unknown environment in order to gather geoscientific data. We start from the assumption that a topological map has been generated by a low-level perception and SLAM module in the form of a labeled graph. However, the map is subject to uncertainties and reconstruction errors that the navigation system must address. First, a distance metric is defined to compute node-matching operations. This metric is then used to enable the robot to find its position on the map and navigate it. To assess the effectiveness of the proposed approach, extensive simulations have been carried out with different randomly generated topologies and various noise rates.

A Dual Forward–Backward Algorithm to Solve Convex Model Predictive Control for Obstacle Avoidance in a Logistics Scenario

In recent years, the logistics sector expanded significantly, leading to the birth of smart warehouses. In this context, a key role is represented by autonomous mobile robots, whose main challenge is to find collision-free paths in their working environment in real-time. Model Predictive Control Algorithms combined with global path planners, such as the A* algorithm, show great potential in providing efficient navigation for collision avoidance problems. This paper proposes a Dual Forward–Backward Algorithm to find the solution to a Model Predictive Control problem in which the task of driving a mobile robotic platform into a bi-dimensional semi-structured environment is formulated in a convex optimisation framework.

Autonomous Navigation of a Forestry Robot Equipped with a Scanning Laser

This abstract is an overview of our research project entitled “Innovative Forest Plantation”, currently in progress. The aim of this project is to automate traditionally manual tasks for poplar plantations in the first years after planting, in particular mechanical weeding without the use of herbicides. The poplar forest is considered as a semi-structured environment where the dense canopy prevents the use of GPS signals and laser sensors are often preferred to localize the vehicle. In this paper, we focus on one of the main functionalities: autonomous navigation, which consists in detecting and locating trees to move safely in such complex environment. Autonomous navigation requires both a precise and robust mapping and localization solution. In this context, Simultaneous Localization and Mapping (SLAM) is very well-suited solution. The constructed map can be reliably used to plan semantic paths of the mobile robot in order treat specifically each tree. Simulations conducted on Gazebo and Robot Operation System (ROS) have proven that the robot could navigate autonomously in a poplar forest.

Autonomous driving using imitation learning with look ahead point for semi structured environments.

Semi-structured environments are difficult for autonomous driving because there are numerous unknown obstacles in drivable area without lanes, and its width and curvature considerably change. In such environments, searching for a path on a real-time is difficult, and localization data are inaccurate, reducing path tracking accuracy. Instead, alternative methods that reactively avoid obstacles in real-time using candidate paths or an artificial potential field have been studied. However, these require heuristics to identify specific parameters for handling various environments and are vulnerable to inaccurate input data. To address these limitations, this study proposes a method in which a vehicle drives toward drivable area using vision and deep learning. The proposed imitation learning method learns the look-ahead point where the vehicle should reach on a vision-based occupancy grid map to obtain a safe policy with a clear state action pattern relationship. Furthermore, using this point, the data aggregation (DAgger) algorithm with weighted loss function is proposed, which imitates expert behavior more accurately, especially in unsafe or near-collision situations. Experimental results in actual semi-structured environments demonstrated the limitations of general model-based methods and the effectiveness of the proposed imitation learning method. Moreover, simulation experiments showed that DAgger with the weight obtains a safer policy than existing DAgger algorithms.

Simultaneous Astronaut Accompanying and Visual Navigation in Semi-Structured and Dynamic Intravehicular Environment

The application of intravehicular robotic assistants (IRA) can save valuable working hours for astronauts in space stations. There are various types of IRA, such as an accompanying drone working in microgravity and a dexterous humanoid robot for collaborative operations. In either case, the ability to navigate and work along with human astronauts lays the foundation for their deployment. To address this problem, this paper proposes the framework of simultaneous astronaut accompanying and visual navigation. The framework contains a customized astronaut detector, an intravehicular navigation system, and a probabilistic model for astronaut visual tracking and motion prediction. The customized detector is designed to be lightweight and has achieved superior performance (AP@0.5 of 99.36%) for astronaut detection in diverse postures and orientations during intravehicular activities. A map-based visual navigation method is proposed for accurate and 6DoF localization (1~2 cm, 0.5°) in semi-structured environments. To ensure the robustness of navigation in dynamic scenes, feature points within the detected bounding boxes are filtered out. The probabilistic model is formulated based on the map-based navigation system and the customized astronaut detector. Both trajectory correlation and geometric similarity clues are incorporated into the model for stable visual tracking and trajectory estimation of the astronaut. The overall framework enables the robotic assistant to track and distinguish the served astronaut efficiently during intravehicular activities and to provide foresighted service while in locomotion. The overall performance and superiority of the proposed framework are verified through extensive ground experiments in a space-station mockup.

Structural Optimisation and Design of a Cable-Driven Hyper-Redundant Manipulator for Confined Semi-Structured Environments.

Structural optimisation of robotic manipulators is critical for any manipulator used in confined semi-structured environments, such as in agriculture. Many robotic manipulators utilised in semi-structured environments retain the same characteristics and dimensions as those used in fully-structured industrial environments, which have been proven to experience low dexterity and singularity issues in challenging environments due to their structural limitations. When implemented in environments other than fully-structured industrial environments, conventional manipulators are liable to singularity, joint limits and workspace obstacles. This makes them inapplicable in confined semi-structured environments, as they lack the flexibility to operate dexterously in such challenging environments. In this paper, structural optimisation of a hyper-redundant cable-driven manipulator is proposed to improve its performance in semi-structured and challenging confined spaces, such as in agricultural settings. The optimisation of the manipulator design is performed in terms of its manipulability and kinematics. The lengths of the links and the joint angles are optimised to minimise any error between the actual and desired position/orientation of the end-effector in a confined semi-structured task space, as well as to provide optimal flexibility for the manipulators to generate different joint configurations for obstacle avoidance in confined environments. The results of the optimisation suggest that the use of a redundant manipulator with rigid short links can result in performance with higher dexterity in confined, semi-structured environments, such as agricultural greenhouses.

E 3 MoP: Efficient Motion Planning Based on Heuristic-Guided Motion Primitives Pruning and Path Optimization With Sparse-Banded Structure

To solve the autonomous navigation problem in complex environments, an efficient motion planning approach is newly presented in this paper. Considering the challenges from large-scale, partially unknown complex environments, a three-layer motion planning framework is elaborately designed, including global path planning, local path optimization, and time-optimal velocity planning. Compared with existing approaches, the novelty of this work is twofold: 1) a novel heuristic-guided pruning strategy of motion primitives is proposed and fully integrated into the state lattice-based global path planner to further improve the computational efficiency of graph search, and 2) a new soft-constrained local path optimization approach is proposed, wherein the sparse-banded system structure of the underlying optimization problem is fully exploited to efficiently solve the problem. We validate the safety, smoothness, flexibility, and efficiency of our approach in various complex simulation scenarios and challenging real-world tasks. It is shown that the computational efficiency is improved by 66.21% in the global planning stage and the motion efficiency of the robot is improved by 22.87% compared with the recent quintic Bézier curve-based state space sampling approach. We name the proposed motion planning framework E <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathbf {^{3}} $ </tex-math></inline-formula> MoP, where the number 3 not only means our approach is a three-layer framework but also means the proposed approach is efficient in three stages. Note to Practitioners—This paper is motivated by the challenges of motion planning problems of mobile robots. A three-layer motion planning framework is proposed by combining global path planning, local path optimization, and time-optimal velocity planning. For mobile robot navigation applications in semi-structured environments, optimization-based local planners are recommended. Extensive simulation and experimental results show the effectiveness of the proposed motion planning framework. However, due to the non-convexity of the path optimization formulation, the proposed local planner may get stuck in local optima. In future research, we will concentrate on extending the proposed local path optimization approach with the theory of homology classes to maintain several homotopically distinct local paths and seek global optima.

UAV-Supported Route Planning for UGVs in Semi-Deterministic Agricultural Environments

Automated agricultural operations must be planned and organized to reduce risk and failure potential while optimizing productivity and efficiency. However, the diversity of natural outdoor environments and the varied data types and volumes required to represent an agricultural setting comprise critical challenges for the deployment of fully automated agricultural operations. In this regard, this study develops an integrated system for enabling an unmanned aerial vehicle (UAV) supported route planning system for unmanned ground vehicles (UGVs) in the semi-structured environment of orchards. The research focus is on the underpinning planning system components (i.e., world representation or map generation or perception and path planning). In particular, the system comprises a digital platform that receives as input a geotagged depiction of an orchard, which is obtained by a UAV. The pre-processed data define the agri-field’s tracks that are transformed into a grid-based map capturing accessible areas. The grid map is then used to generate a topological path planning solution. Subsequently, the solution is translated into a sequence of coordinates that define the calculated optimal path for the UGV to traverse. The applicability of the developed system was validated in routing scenarios in a walnuts’ orchard using a UGV. The contribution of the proposed system entails noise reduction techniques for the accurate representation of a semi-deterministic agricultural environment for enabling accuracy in the route planning of utilized automated machinery.

Computer Vision in Self-Steering Tractors

Automatic navigation of agricultural machinery is an important aspect of Smart Farming. Intelligent agricultural machinery applications increasingly rely on machine vision algorithms to guarantee enhanced in-field navigation accuracy by precisely locating the crop lines and mapping the navigation routes of vehicles in real-time. This work presents an overview of vision-based tractor systems. More specifically, this work deals with (1) the system architecture, (2) the safety of usage, (3) the most commonly faced navigation errors, (4) the navigation control system of tractors and presents (5) state-of-the-art image processing algorithms for in-field navigation route mapping. In recent research, stereovision systems emerge as superior to monocular systems for real-time in-field navigation, demonstrating higher stability and control accuracy, especially in extensive crops such as cotton, sunflower, maize, etc. A detailed overview is provided for each topic with illustrative examples that focus on specific agricultural applications. Several computer vision algorithms based on different optical sensors have been developed for autonomous navigation in structured or semi-structured environments, such as orchards, yet are affected by illumination variations. The usage of multispectral imaging can overcome the encountered limitations of noise in images and successfully extract navigation paths in orchards by using a combination of the trees’ foliage with the background of the sky. Concisely, this work reviews the current status of self-steering agricultural vehicles and presents all basic guidelines for adapting computer vision in autonomous in-field navigation.

Aircraft Parking Trajectory Planning in Semistructured Environment Based on Kinodynamic Safety RRT ∗

To improve the safety and effectiveness of autonomous towing aircraft aboard the carrier deck, this study proposes a velocity-restricted path planner algorithm named as kinodynamic safety optimal rapidly exploring random tree (KS-RRT ∗ ) to plan a near time-optimal path. First, a speed map is introduced to assign different maximum allowable velocity for the sampling points in the workspace, and the traverse time is calculated along the kinodynamic connection of two sampling points. Then the near time-optimal path in the tree-structured search map can be obtained by the rewiring procedures, instead of a distance-optimal path in the original RRT ∗ algorithm. In order to enhance the planner’s performance, goal biasing scheme and fast collision checking technique are adopted in the algorithm. Since the sampling-based methods are sensitive to their parameters, simulation experiments are first conducted to determine the optimal input settings for the specific problem. The effectiveness of the proposed algorithm is validated in several common aircraft parking scenarios. Comparing with standard RRT ∗ and human heuristic driving, KS-RRT ∗ demonstrates a higher success rate, as well as shorter computation and trajectory time. In conclusion, KS-RRT ∗ algorithm is suitable to generate a near time-optimal safe path for autonomous high density parking in semistructured environment.