Vision-based Navigation Research Articles

Interaction-aware control for robotic vegetation override in off-road environments

Robotic systems tasked with completing off-road economic, military, or humanitarian missions often encounter environmental objects when traversing unstructured terrains. Certain objects (e.g. safety cones) must be avoided to ensure operational integrity, but others (e.g. small vegetation) can be interacted with (e.g. overridden/pushed) safely. Pure object-avoidance assumptions in conventional robotic system navigation policies may lead to inefficient (slow) or overly-cautious (immobilized) traversal behaviors in off-road terrains. To address this gap in system performance, we draw inspiration from existing hybrid dynamic system control literature. We have designed a nonlinear trajectory optimization controller that utilizes vegetation-interaction models as a jump map in the dynamics constraint. In contrast to purely vision-based navigation policies which classify the traversability of obstacles, the allowable subset of objects with which the vehicle can safely interact is now characterized by a data-driven collision model and the existence of a dynamically-feasible trajectory which satisfies the contact constraints. The controller’s capabilities are demonstrated on a full-sized autonomous utility task vehicle where objects including posts and trees of up to 25.4 [mm] and 81.8 [mm] diameter are overridden.

Path Planning and Motion Control of Robot Dog Through Rough Terrain Based on Vision Navigation.

This article delineates the enhancement of an autonomous navigation and obstacle avoidance system for a quadruped robot dog. Part one of this paper presents the integration of a sophisticated multi-level dynamic control framework, utilizing Model Predictive Control (MPC) and Whole-Body Control (WBC) from MIT Cheetah. The system employs an Intel RealSense D435i depth camera for depth vision-based navigation, which enables high-fidelity 3D environmental mapping and real-time path planning. A significant innovation is the customization of the EGO-Planner to optimize trajectory planning in dynamically changing terrains, coupled with the implementation of a multi-body dynamics model that significantly improves the robot's stability and maneuverability across various surfaces. The experimental results show that the RGB-D system exhibits superior velocity stability and trajectory accuracy to the SLAM system, with a 20% reduction in the cumulative velocity error and a 10% improvement in path tracking precision. The experimental results also show that the RGB-D system achieves smoother navigation, requiring 15% fewer iterations for path planning, and a 30% faster success rate recovery in challenging environments. The successful application of these technologies in simulated urban disaster scenarios suggests promising future applications in emergency response and complex urban environments. Part two of this paper presents the development of a robust path planning algorithm for a robot dog on a rough terrain based on attached binocular vision navigation. We use a commercial-of-the-shelf (COTS) robot dog. An optical CCD binocular vision dynamic tracking system is used to provide environment information. Likewise, the pose and posture of the robot dog are obtained from the robot's own sensors, and a kinematics model is established. Then, a binocular vision tracking method is developed to determine the optimal path, provide a proposal (commands to actuators) of the position and posture of the bionic robot, and achieve stable motion on tough terrains. The terrain is assumed to be a gentle uneven terrain to begin with and subsequently proceeds to a more rough surface. This work consists of four steps: (1) pose and position data are acquired from the robot dog's own inertial sensors, (2) terrain and environment information is input from onboard cameras, (3) information is fused (integrated), and (4) path planning and motion control proposals are made. Ultimately, this work provides a robust framework for future developments in the vision-based navigation and control of quadruped robots, offering potential solutions for navigating complex and dynamic terrains.

Advanced Edge AI Navigation and Assistance System for Blind User

- Despite substantial advancements in visual assistance technology, many existing systems are limited by sensor capabilities, computational resources, and power consumption. Traditional computer vision algorithms struggle to perform complex tasks required for real-time navigation. This paper presents the design and implementation of an advanced computer vision-based navigation system specifically tailored for Blender users, aimed at facilitating independent navigation in diverse environments .By leveraging edge Artificial Intelligence (AI) and deep learning methodologies, the proposed system achieves real-time object detection, person recognition, and environmental awareness, addressing the critical challenges posed by conventional systems. The use of cost-effective, low-power mobile computing platforms, such as smart depth sensors like the OpenCV AI Kit-Depth (OAK-D), enables efficient processing while ensuring portability. This system not only enhances the ability to identify and navigate obstacles but also incorporates additional functionalities, including reading written notices aloud and responding to traffic signals. Key design considerations, such as training data collection, computational efficiency, and portability, have been meticulously addressed to ensure reliable performance in real-world scenarios. The incorporation of an AI-driven voice interface facilitates user-friendly interaction, making this system an innovative and unobtrusive visual assistance device.

Convolutional-Neural-Network-Based Autonomous Navigation of Hera Mission Around Didymos

The European Space Agency (ESA)’s Hera mission requires autonomous visual-based navigation in order to safely orbit around the target binary asteroid system Didymos and its moon Dimorphos in 2027. Nevertheless, the performance of optical-based navigation systems depends on the intrinsic properties of the image, such as high Sun phase angles, the presence of other bodies, and, especially, the irregular shape of the target. Therefore, to improve the navigation performance, thermal and/or range measurements from additional onboard instruments are usually needed to complement optical measurements. However, this work addresses these challenges by developing a fully visual-based autonomous navigation system using a convolutional-neural-network (CNN)-based image processing (IP) algorithm and applying it to the detailed characterization phase of the proximity operation of the mission. The images taken by the onboard camera are processed by the CNN-based IP algorithm that estimates the position of the geometrical centers of Didymos and Dimorphos, the range from Didymos, and the associated covariances. The results show that the developed algorithm can be used for a fully visual-based navigation for the position of the Hera spacecraft around the target with good robustness and accuracy.

Research on Automatic Recharging Technology for Automated Guided Vehicles Based on Multi-Sensor Fusion

Automated guided vehicles (AGVs) play a critical role in indoor environments, where battery endurance and reliable recharging are essential. This study proposes a multi-sensor fusion approach that integrates LiDAR, depth cameras, and infrared sensors to address challenges in autonomous navigation and automatic recharging. The proposed system overcomes the limitations of LiDAR’s blind spots in near-field detection and the restricted range of vision-based navigation. By combining LiDAR for precise long-distance measurements, depth cameras for enhanced close-range visual positioning, and infrared sensors for accurate docking, the AGV’s ability to locate and autonomously connect to charging stations is significantly improved. Experimental results show a 25% increase in docking success rate (from 70% with LiDAR-only to 95%) and a 70% decrease in docking error (from 10 cm to 3 cm). These improvements demonstrate the effectiveness of the proposed sensor fusion method, ensuring more reliable, efficient, and precise operations for AGVs in complex indoor environments.

果园环境中巡检机器人的 SLAM 和路径规划方法研究

Orchard robots play a crucial role in agricultural production. Autonomous navigation serves as the foundation for orchard robots and eco-unmanned farms. Accurate sensing and localization are prerequisites for achieving autonomous navigation. However, current vision-based navigation solutions are sensitive to environmental factors, such as light, weather, and background, which can affect positioning accuracy. Therefore, they are unsuitable for outdoor navigation applications. LIDAR provides accurate distance measurements and is suitable for a wide range of environments. Its immunity to interference is not affected by light, colour, weather, or other factors, making it suitable for low objects and complex orchard scenes. Therefore, LiDAR navigation is more suitable for orchard environments. In complex orchard environments, tree branches and foliage can cause Global Positioning System (GNSS) accuracy to degrade, resulting in signal loss. Therefore, the major challenge that needs to be addressed is generating navigation paths and locating the position of orchard robots. In this paper, an improved method for Simultaneous Localization and Mapping (SLAM) and A-star algorithm is proposed. The SLAM and path planning method designed in this study effectively solves the problems of insufficient smoothness and large curvature fluctuation of the path planned in the complex orchard environment, and improves the detection efficiency of the robot. The experimental results indicate that the method can consistently and accurately fulfil the robot's detection needs in intricate orchard environments.

Autonomous Vision-Based Navigation and Stability Augmentation Control of a Biomimetic Robotic Hammerhead Shark

The application potential of robotic fish embedded with intelligent visual navigation algorithms in underwater autonomous operation is on full display recently. However, the existing visual navigation methods are limited by the underwater visual conditions and the motion characteristics of robotic fish. To this end, this paper proposes a novel autonomous navigation framework integrated with visual stabilization control. In practice, a stereo vision-based navigation network is proposed to generate the guidance law. On this basis, a biomimetic robotic hammerhead shark with a controllable cephalofoil is developed, and a nonlinear model predictive controller for cephalofoil stabilization relying on the dynamic model is elaborately designed. Extensive simulations and underwater experiments are conducted to validate the effectiveness and superiority of the proposed methods, which significantly enhance exploration efficiency and reduce image jitter by 26.02% compared to the traditional methods. The obtained results provide a new idea for underwater robots to autonomously explore the ocean. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the problem of vision-based underwater autonomous navigation for a biomimetic robotic fish that possesses underwater visual stability and good maneuverability. The existing visual navigation networks usually generate unexpected navigation instructions when dealing with complex or ambiguous underwater scenes. Additionally, image jitter caused by the rhythmic motion of robotic fish can lead to navigation failure. This paper suggests an integrated navigation framework that includes a biomimetic platform design, a visual stabilization controller, and an intelligent underwater navigation network. Specifically, a novel sphyrnidae-inspired robotic shark is designed as a new platform with superior motion performance. To enhance underwater visual stability, a nonlinear model predictive control-based visual stabilization controller is proposed. Furthermore, a deep stereo attention navigation network based on a parallax attention mechanism is proposed to improve the generalization of vision-based autonomous navigation. A series of underwater search experiments on the robotic shark demonstrate the effectiveness and superiority of the proposed navigation framework. Hopefully, our proposed methods can provide valuable guidance and support for universal underwater robot navigation to accomplish practical marine tasks, such as underwater rescue, resource exploitation, biological observation, and so on.

Autonomous Vision-Based Algorithm for Interplanetary Navigation

The surge of deep-space probes makes it unsustainable to navigate them with standard radiometric tracking. Autonomous interplanetary satellites represent a solution to this problem. In this work, a vision-based navigation algorithm is built by combining an orbit determination method with an image processing pipeline suitable for interplanetary transfers of autonomous platforms. To increase the computational efficiency of the algorithm, an extended Kalman filter is selected as state estimator, fed by the positions of the planets extracted from deep-space images. An enhancement of the estimation accuracy is performed by applying an optimal strategy to select the best pair of planets to track. Moreover, a novel analytical measurement model for deep-space navigation is developed providing a first-order approximation of the light-aberration and light-time effects. Algorithm performance is tested on a high-fidelity, Earth–Mars interplanetary transfer, showing the algorithm applicability for deep-space navigation.

MONET: The Minor Body Generator Tool at DART Lab.

Minor bodies exhibit considerable variability in shape and surface morphology, posing challenges for spacecraft operations, which are further compounded by highly non-linear dynamics and limited communication windows with Earth. Additionally, uncertainties persist in the shape and surface morphology of minor bodies due to errors in ground-based estimation techniques. The growing need for autonomy underscores the importance of robust image processing and visual-based navigation methods. To address this demand, it is essential to conduct tests on a variety of body shapes and with different surface morphological features. This work introduces the procedural Minor bOdy geNErator Tool (MONET), implemented using an open-source 3D computer graphics software. The starting point of MONET is the three-dimensional mesh of a generic minor body, which is procedurally modified by introducing craters, boulders, and surface roughness, resulting in a photorealistic model. MONET offers the flexibility to generate a diverse range of shapes and surface morphological features, aiding in the recreation of various minor bodies. Users can fine-tune relevant parameters to create the desired conditions based on the specific application requirements. The tool offers the capability to generate two default families of models: rubble-pile, characterized by numerous different-sized boulders, and comet-like, reflecting the typical morphology of comets. MONET serves as a valuable resource for researchers and engineers involved in minor body exploration missions and related projects, providing insights into the adaptability and effectiveness of guidance and navigation techniques across a wide range of morphological scenarios.

Autonomous landing of a quadrotor on a moving platform using motion capture system

This paper investigates the challenging problem of the autonomous landing of a quadrotor on a moving platform in a non-cooperative environment. The limited sensing ability of quadrotors often hampers their utilization for autonomous landing, especially in GPS-denied areas. The performance of motion capture systems (MCSs) in many application areas is the motivation to utilize them for the autonomous take-off and landing of the quadrotor in this research. An autonomous closed-loop vision-based navigation, tracking, and control system is proposed for quadrotors to perform landing based upon Model Predictive Control (MPC) by utilizing multi-objective functions. The entire process is posed as a constrained tracking problem to minimize energy consumption and ensure smooth maneuvers. The proposed approach is fully autonomous from take-off to landing; whereas, the movements of the landing platform are pre-defined but still unknown to the quadrotor. The landing performance of the quadrotor is tested and evaluated for three different movement patterns: static, square-shaped, and circular-shaped. Through experimental results, the pose error between the quadrotor and the platform is measured and found to be less than 30 cm.

Advancements in AI Driven Perception Systems for Agricultural Robotics

Robots have become increasingly important over the years. Ever- increasing demands for production, labour fatigue, reduced labour and environmental safety have brought robotics to the forefront of scientific advances. The same approach is used with agro-robots, where similar solutions are feasible. help farmers farm faster, safer and more profitably This paper evaluates the prevailing level of mind-based vision in agricultural robots and field applications, specifically weed identification, crop identification, classification, disease detection, vision- based navigation, harvesting, and propagation. This paper analyzes the present level of mind-based vision in agricultural robots in field applications, specifically weed recognition, crop identification, phenotyping, disease detection, vision-based navigation, harvesting, and propagation. The survey demonstrated an extensive curiosity in drawing vision-based solutions.in agricultural robotics, where the most popular RGB cameras and sensors optionally can have promising outcomes, and no particular algorithm leads all others. Instead, artificial intelligence gives specific benefits for performing certain issues associated with agriculture.

PVSPE: A pyramid vision multitask transformer network for spacecraft pose estimation

Spacecraft pose estimation (SPE) plays a vital role in the relative navigation system for on-orbit servicing and active debris removal. Current deep learning-based methods have made great achievements on object pose estimation. However, towards the challenging onboard SPE missions, most existing Convolutional Neural Network (CNN) methods failed to capture remote vision attention, leading to the reduction of accuracy and robustness. In this paper, we presented an end-to-end multi-task Pyramid Transformer SPE network (PVSPE) consisting of two novel feature extraction modules: EnhancedPVT (EnPVT) and SlimGFPN. The EnPVT module is designed to combine global spatial and channel attention, while the Slim GFPN module can fuse features more effectively. Matrix Fisher and multivariate Gaussian distributions are further employed to model the uncertainty of pose regression to increase its accuracy. Extensive experiments are carried out on challenging SPEED + and SHIRT datasets, to validate the performances on pose estimation and vision-based navigation, respectively. The results show that the proposed PVSPE model achieved high accuracy for SPE on the SPEED + dataset even under different scales and severe illumination, demonstrating its robustness and high generalization. Leveraging the insightful uncertainty model of PVSPE, the vision-based navigation pipeline, combined with Kalman filters, accurately estimated the satellite pose under challenging rendezvous scenarios on the SHIRT dataset, with degree-level attitude errors and centimeter-level translation accuracy at steady-state.

Robotic Odor Source Localization via Vision and Olfaction Fusion Navigation Algorithm.

Robotic odor source localization (OSL) is a technology that enables mobile robots or autonomous vehicles to find an odor source in unknown environments. An effective navigation algorithm that guides the robot to approach the odor source is the key to successfully locating the odor source. While traditional OSL approaches primarily utilize an olfaction-only strategy, guiding robots to find the odor source by tracing emitted odor plumes, our work introduces a fusion navigation algorithm that combines both vision and olfaction-based techniques. This hybrid approach addresses challenges such as turbulent airflow, which disrupts olfaction sensing, and physical obstacles inside the search area, which may impede vision detection. In this work, we propose a hierarchical control mechanism that dynamically shifts the robot's search behavior among four strategies: crosswind maneuver, Obstacle-Avoid Navigation, Vision-Based Navigation, and Olfaction-Based Navigation. Our methodology includes a custom-trained deep-learning model for visual target detection and a moth-inspired algorithm for Olfaction-Based Navigation. To assess the effectiveness of our approach, we implemented the proposed algorithm on a mobile robot in a search environment with obstacles. Experimental results demonstrate that our Vision and Olfaction Fusion algorithm significantly outperforms vision-only and olfaction-only methods, reducing average search time by 54% and 30%, respectively.

Model-aided and vision-based navigation for an aerial robot in real-time application

Model-aided and vision-based navigation for an aerial robot in real-time application

Qualification of Avionic Software Based on Machine Learning: Challenges and Key Enabling Domains

Advances in machine learning (ML) open the way to innovating functions in the avionic domain, such as navigation/surveillance assistance (e.g., vision-based navigation, obstacle sensing, virtual sensing), speech-to-text applications, autonomous flight, predictive maintenance, and cockpit assistance. Current standards and practices, which were defined and refined over decades with classical programming in mind, do not, however, support this new development paradigm. This paper provides an overview of the main challenges raised by the use of ML in the demonstration of compliance with regulatory requirements (i.e., software qualification) and an overview of literature relevant to these challenges, with particular focus on the issues of robustness, provability, and explainability of ML results.

Angle Robustness Unmanned Aerial Vehicle Navigation in GNSS-Denied Scenarios

Due to the inability to receive signals from the Global Navigation Satellite System (GNSS) in extreme conditions, achieving accurate and robust navigation for Unmanned Aerial Vehicles (UAVs) is a challenging task. Recently emerged, vision-based navigation has been a promising and feasible alternative to GNSS-based navigation. However, existing vision-based techniques are inadequate in addressing flight deviation caused by environmental disturbances and inaccurate position predictions in practical settings. In this paper, we present a novel angle robustness navigation paradigm to deal with flight deviation in point-to-point navigation tasks. Additionally, we propose a model that includes the Adaptive Feature Enhance Module, Cross-knowledge Attention-guided Module and Robust Task-oriented Head Module to accurately predict direction angles for high-precision navigation. To evaluate the vision-based navigation methods, we collect a new dataset termed as UAV_AR368. Furthermore, we design the Simulation Flight Testing Instrument (SFTI) using Google Earth to simulate different flight environments, thereby reducing the expenses associated with real flight testing. Experiment results demonstrate that the proposed model outperforms the state-of-the-art by achieving improvements of 26.0% and 45.6% in the success rate of arrival under ideal and disturbed circumstances, respectively.

On-ground validation of orbital GNC: Visual navigation assessment in robotic testbed facility

AbstractCubeSats have become versatile platforms for various space missions (e.g., on-orbit servicing and debris removal) owing to their low cost and flexibility. Many space tasks involve proximity operations that require precise guidance, navigation, and control (GNC) algorithms. Vision-based navigation is attracting interest for such operations. However, extreme lighting conditions in space challenge optical techniques. The on-ground validation of such navigation systems for orbital GNC becomes crucial to ensure their reliability during space operations. These systems undergo rigorous testing within their anticipated operational parameters, including the exploration of potential edge cases. The ability of GNC algorithms to function effectively under extreme space conditions that exceed anticipated scenarios is crucial, particularly in space missions where the scope of errors is negligible. This paper presents the ground validation of a GNC algorithm designed for autonomous satellite rendezvous by leveraging hardware-in-the-loop experiments. This study focuses on two key areas. First, the rationale underlying the augmentation of the robot workspace (six-degree-of-freedom UR10e robot + linear rail) is investigated to emulate relatively longer trajectories with complete position and orientation states. Second, the control algorithm is assessed in response to uncertain pose observations from a vision-based navigation system. The results indicate increased control costs with uncertain navigation and exemplify the importance of on-ground testing for system validation before launch, particularly in extreme cases that are typically difficult to assess using software-based testing.

Tactile Perception-Based Depth and Angle Control During Robot-Assisted Bent Bone Grinding

During bent bone grinding, the robot often needs to adjust its grinding depth and angle in real-time to ensure surgical quality. Optical- or vision-based navigation methods are no longer applicable due to the interference caused by splashing cooling water in the operating field. Inspired by orthopedic surgeons, this manuscript presents a tactile perception-based method to solve this problem. The effect of grinding parameters on the cutter's current and vibration is analyzed, and current and vibration signals are further processed to predict depth and angle. Depth and angle estimation models were calibrated on homogeneous artificial bone. Grinding control experiments were carried out on femur models, and the statistical mean and standard deviation results of the measured depth show that the accuracy of depth control can be further improved by adding angle estimation and proper adjustment when depth is indirectly estimated based on sensing signals. The proposed approach relies solely on the estimation of the relative position and posture between the tool and the bone surface and is expected to improve the safety of automatic grinding with orthopedic robots.

Sustainable Vision-Based Navigation for Autonomous Electric Vehicle Charging

Sustainable Vision-Based Navigation for Autonomous Electric Vehicle Charging

Mobile Robot Obstacle Detection and Avoidance with NAV-YOLO

Intelligent robotics is gaining significance in Maintenance, Repair, and Overhaul (MRO) hangar operations, where mobile robots navigate complex and dynamic environments for Aircraft visual inspection. Aircraft hangars are usually busy and changing, with objects of varying shapes and sizes presenting harsh obstacles and conditions that can lead to potential collisions and safety hazards. This makes Obstacle detection and avoidance critical for safe and efficient robot navigation tasks. Conventional methods have been applied with computational issues, while learning-based approaches are limited in detection accuracy. This paper proposes a vision-based navigation model that integrates a pre-trained Yolov5 object detection model into a Robot Operating System (ROS) navigation stack to optimise obstacle detection and avoidance in a complex environment. The experiment is validated and evaluated in ROS-Gazebo simulation and turtlebot3 waffle-pi robot platform. The results showed that the robot can increasingly detect and avoid obstacles without colliding while navigating through different checkpoints to the target location.