Vision-based Navigation System Research Articles

Autonomous Drone Navigation Using Computer Vision

Abstract: Autonomous drone navigation with computer vision is an innovative technology that allows drones to move through intricate surroundings without needing human control. This system uses live visual information to identify objects, steer clear of obstacles, and determine routes, improving operational safety and precision. The project's main objective is to create a visionbased navigation system that combines object detection and obstacle avoidance algorithms through deep learning methods like YOLO (You Only Look Once), alongside real-time sensor fusion. The drone uses computer vision algorithms to process aerial images and automatically changes its flight path to prevent crashes. A personalized dataset of aerial images is generated and utilized for improving object detection skills in various environments. In order to guarantee practicality in real- world situations, the system undergoes validation through simulations and field tests on different terrains, focusing on its resilience in changing environments. Improvements in navigation accuracy and obstacle detection are accomplished by implementing adaptive pathplanning and integrating multiple sensors, guaranteeing the drone's efficient operation in real-life situations. The goal of this method is to enhance the drone's ability to make decisions, minimize human mistakes, and expand its range of potential uses in areas like surveillance, agriculture, and disaster response

Advanced Edge AI Navigation and Assistance System for Blind User

- Despite substantial advancements in visual assistance technology, many existing systems are limited by sensor capabilities, computational resources, and power consumption. Traditional computer vision algorithms struggle to perform complex tasks required for real-time navigation. This paper presents the design and implementation of an advanced computer vision-based navigation system specifically tailored for Blender users, aimed at facilitating independent navigation in diverse environments .By leveraging edge Artificial Intelligence (AI) and deep learning methodologies, the proposed system achieves real-time object detection, person recognition, and environmental awareness, addressing the critical challenges posed by conventional systems. The use of cost-effective, low-power mobile computing platforms, such as smart depth sensors like the OpenCV AI Kit-Depth (OAK-D), enables efficient processing while ensuring portability. This system not only enhances the ability to identify and navigate obstacles but also incorporates additional functionalities, including reading written notices aloud and responding to traffic signals. Key design considerations, such as training data collection, computational efficiency, and portability, have been meticulously addressed to ensure reliable performance in real-world scenarios. The incorporation of an AI-driven voice interface facilitates user-friendly interaction, making this system an innovative and unobtrusive visual assistance device.

Autonomous landing of a quadrotor on a moving platform using motion capture system

This paper investigates the challenging problem of the autonomous landing of a quadrotor on a moving platform in a non-cooperative environment. The limited sensing ability of quadrotors often hampers their utilization for autonomous landing, especially in GPS-denied areas. The performance of motion capture systems (MCSs) in many application areas is the motivation to utilize them for the autonomous take-off and landing of the quadrotor in this research. An autonomous closed-loop vision-based navigation, tracking, and control system is proposed for quadrotors to perform landing based upon Model Predictive Control (MPC) by utilizing multi-objective functions. The entire process is posed as a constrained tracking problem to minimize energy consumption and ensure smooth maneuvers. The proposed approach is fully autonomous from take-off to landing; whereas, the movements of the landing platform are pre-defined but still unknown to the quadrotor. The landing performance of the quadrotor is tested and evaluated for three different movement patterns: static, square-shaped, and circular-shaped. Through experimental results, the pose error between the quadrotor and the platform is measured and found to be less than 30 cm.

On-ground validation of orbital GNC: Visual navigation assessment in robotic testbed facility

AbstractCubeSats have become versatile platforms for various space missions (e.g., on-orbit servicing and debris removal) owing to their low cost and flexibility. Many space tasks involve proximity operations that require precise guidance, navigation, and control (GNC) algorithms. Vision-based navigation is attracting interest for such operations. However, extreme lighting conditions in space challenge optical techniques. The on-ground validation of such navigation systems for orbital GNC becomes crucial to ensure their reliability during space operations. These systems undergo rigorous testing within their anticipated operational parameters, including the exploration of potential edge cases. The ability of GNC algorithms to function effectively under extreme space conditions that exceed anticipated scenarios is crucial, particularly in space missions where the scope of errors is negligible. This paper presents the ground validation of a GNC algorithm designed for autonomous satellite rendezvous by leveraging hardware-in-the-loop experiments. This study focuses on two key areas. First, the rationale underlying the augmentation of the robot workspace (six-degree-of-freedom UR10e robot + linear rail) is investigated to emulate relatively longer trajectories with complete position and orientation states. Second, the control algorithm is assessed in response to uncertain pose observations from a vision-based navigation system. The results indicate increased control costs with uncertain navigation and exemplify the importance of on-ground testing for system validation before launch, particularly in extreme cases that are typically difficult to assess using software-based testing.

Visual teach and generalise (VTAG)—Exploiting perceptual aliasing for scalable autonomous robotic navigation in horticultural environments

Nowadays, most agricultural robots rely on precise and expensive localisation, typically based on global navigation satellite systems (GNSS) and real-time kinematic (RTK) receivers. Unfortunately, the precision of GNSS localisation significantly decreases in environments where the signal paths between the receiver and the satellites are obstructed. This precision hampers deployments of these robots in, e.g., polytunnels or forests. An attractive alternative to GNSS is vision-based localisation and navigation. However, perceptual aliasing and landmark deficiency, typical for agricultural environments, cause traditional image processing techniques, such as feature matching, to fail. We propose an approach for an affordable pure vision-based navigation system which is not only robust to perceptual aliasing, but it actually exploits the repetitiveness of agricultural environments. Our system extends the classic concept of visual teach and repeat to visual teach and generalise (VTAG). Our teach and generalise method uses a deep learning-based image registration pipeline to register similar images through meaningful generalised representations obtained from different but similar areas. The proposed system uses only a low-cost uncalibrated monocular camera and the robot’s wheel odometry to produce heading corrections to traverse crop rows in polytunnels safely. We evaluate this method at our test farm and at a commercial farm on three different robotic platforms where an operator teaches only a single crop row. With all platforms, the method successfully navigates the majority of rows with most interventions required at the end of the rows, where the camera no longer has a view of any repeating landmarks such as poles, crop row tables or rows which have visually different features to that of the taught row. For one robot which was taught one row 25 m long our approach autonomously navigated the robot a total distance of over 3.5 km, reaching a teach-generalisation gain of 140.

Monocular Based Navigation System for Autonomous Ground Robots Using Multiple Deep Learning Models

In recent years, the development of ground robots with human-like perception capabilities has led to the use of multiple sensors, including cameras, lidars, and radars, along with deep learning techniques for detecting and recognizing objects and estimating distances. This paper proposes a computer vision-based navigation system that integrates object detection, segmentation, and monocular depth estimation using deep neural networks to identify predefined target objects and navigate towards them with a single monocular camera as a sensor. Our experiments include different sensitivity analyses to evaluate the impact of monocular cues on distance estimation. We show that this system can provide a ground robot with the perception capabilities needed for autonomous navigation in unknown indoor environments without the need for prior mapping or external positioning systems. This technique provides an efficient and cost-effective means of navigation, overcoming the limitations of other navigation techniques such as GPS-based and SLAM-based navigation.Graphical

A Vision-Based Navigation System With Markerless Image Registration and Position-Sensing Localization for Oral and Maxillofacial Surgery

The quality of oral and maxillofacial surgery (OMS) significantly depends on the accuracy of surgical navigation. In this article, a vision-based markerless surgical navigation system is developed to overcome the shortcomings in the currently available technologies. Registration methods both for patient and surgical instrument tracking are improved to increase the navigation performance. For patient-image registration, we propose an efficient texture-less pose estimation method using only shape information. An innovative strategy is developed to effectively reject the outliers and improve the pose accuracy, which is the first attempt at introducing geometric matching information to guide PnP calculation. For surgical instrument tracking, a position-sensing marker is used to achieve robust and convenient instrument localization with high accuracy. Experiments were conducted on the 3-D-printed maxilla and mandible models to evaluate the navigation performance. Evaluation results validate the effectiveness of the proposed pose estimation method in improving the pose accuracy for texture-less teeth. Besides, it is revealed that the position-sensing marker can be localized with high accuracy even under nonideal visibility conditions, which expands the motion range of the instrument and decreases the size of the tool. The entire system has a sufficiently small target registration error (TRE). These experimental results have verified that the proposed surgical navigation system can provide practical guidance for OMS with satisfactory accuracy.

UNav: An Infrastructure-Independent Vision-Based Navigation System for People with Blindness and Low Vision.

Vision-based localization approaches now underpin newly emerging navigation pipelines for myriad use cases, from robotics to assistive technologies. Compared to sensor-based solutions, vision-based localization does not require pre-installed sensor infrastructure, which is costly, time-consuming, and/or often infeasible at scale. Herein, we propose a novel vision-based localization pipeline for a specific use case: navigation support for end users with blindness and low vision. Given a query image taken by an end user on a mobile application, the pipeline leverages a visual place recognition (VPR) algorithm to find similar images in a reference image database of the target space. The geolocations of these similar images are utilized in a downstream task that employs a weighted-average method to estimate the end user's location. Another downstream task utilizes the perspective-n-point (PnP) algorithm to estimate the end user's direction by exploiting the 2D-3D point correspondences between the query image and the 3D environment, as extracted from matched images in the database. Additionally, this system implements Dijkstra's algorithm to calculate a shortest path based on a navigable map that includes the trip origin and destination. The topometric map used for localization and navigation is built using a customized graphical user interface that projects a 3D reconstructed sparse map, built from a sequence of images, to the corresponding a priori 2D floor plan. Sequential images used for map construction can be collected in a pre-mapping step or scavenged through public databases/citizen science. The end-to-end system can be installed on any internet-accessible device with a camera that hosts a custom mobile application. For evaluation purposes, mapping and localization were tested in a complex hospital environment. The evaluation results demonstrate that our system can achieve localization with an average error of less than 1 m without knowledge of the camera's intrinsic parameters, such as focal length.

Relative Positioning Method for UAVs Based on Multi-Source Information Fusion

To improve the accuracy and reliability of the relative positioning for unmanned aerial vehicles (UAVs), a relative positioning method based on multi-source information fusion is proposed. An integrated positioning scheme is constructed by the Beidou Navigation Satellite System (BDS) receivers, Global Positioning System (GPS) receivers, Vision-Based Navigation System (VisNav), and Inertial Navigation System (INS). The BDS pseudorange relative difference equation, the GPS relative difference equation, the relative line-of-sight vector equation, and the INS measurement equation are established, respectively. The least squared (LS) method is used to realize information fusion, and the Gauss-Newton method is used to iteratively solve the relative position results. Finally, numerical simulation and result analysis are conducted with different sensor configurations. Simulation results show that BDS/INS/GPS/VisNav relative positioning result is obviously better than that of INS/GPS, INS/VisNav, and INS/GPS/VisNav, and the proposed method reduces the relative positioning errors and has a higher accuracy and excellent robustness. The research result is suitable for application scenarios with high navigation accuracy requirement such as AAR and intelligent swarm formation control.

Design and evaluation of hedge trimmer robot

An autonomous hedge trimmer robot was developed to reduce the harmful effects of traditional hedge trimming method (gasoline-powered hedge trimmer) on operators' bodies and increase the speed and quality of the operation. The robot had five degrees of freedom (PRRRR) structure and was geometrically optimized to simplify the control strategy, enabling it to trim simple, stepped, and circular forms. The control system, consisting of a host computer, servo motors and servo drivers, and controller, guided the robot platform, manipulator, and end-effector as it approached and trimmed the hedge. By using a recognition algorithm and vision-based navigation system, the robot was able to detect the path and trim the hedge automatically. Experimental tests showed the vision-based system is accurate and fast enough to control the robot. The accuracy of hedge detection performance under various conditions was about 96% and the success rate of trimming was 88%.

Optical navigation for Lunar landing based on Convolutional Neural Network crater detector

Traditional vision-based navigation algorithms are highly affected from non-nominal conditions, which comprise illumination conditions and environmental uncertainties. Thanks to the outstanding generalization capability and flexibility, deep neural networks (and AI algorithms in general) are excellent candidates to solve the aforementioned shortcoming of navigation algorithms. The paper presents a vision-based navigation system using a Convolutional Neural Network to solve the task of pinpoint landing on the Moon using absolute navigation, namely with respect to the Mean Earth/Polar Axis reference frame. The Moon landing scenario consists in the spacecraft descent on the South Pole from a parking orbit to the powered descent phase. The architecture features an Object Detection Convolutional Neural Network (ODN) trained with supervised learning approach. The CNN is used to extract features of the observed craters that are then processed by standard image processing algorithms in order to provide pseudo-measurements that can be used by navigation filter. The craters are matched against a database containing the inertial location of the known craters. An Extended Kalman Filter with time-delayed measurements integration is developed to fuse optical and altimeter information.

Autonomous Underwater Vehicle with Vision-based Navigation System for Underwater Robot Competition

Autonomous Underwater Vehicle with Vision-based Navigation System for Underwater Robot Competition

A Vision-based Navigation System for an Agricultural Autonomous Tractor

RTK-GNSS (Real-time Kinematic-Global Navigation Satellite System) signal is necessary for the safe navigation of an autonomous tractor. However, the main problem associated with this type of precise GNSS is that the RTK correction signal might not be available at all geographical locations. Tree canopies present close to the tractors’ vicinity obstruct satellite signals and degrade the position accuracy. This can lead the autonomous tractor to travel outside of its predetermined path, which is unsafe. To avoid such situations, this study aims to develop a vision-based road detection system, which does not rely on RTK-GNSS. This system aims to detect both paved and unpaved roads in rural farm areas. It takes the RGB images obtained from an HD onboard camera and segments the road surface and road edges. The segmentation is performed by using several image filters, the sliding window method, and a set of rules that are determined manually. Finally, the lateral error obtained from the machine vision system is transmitted to the tractor's automatic navigation system. Experimental runs performed at the experimental farm of Hokkaido University showed that the lateral error calculated by this system is less than 0.2 m for unpaved roads and 0.4 m for paved roads. This shows that the system can detect both paved and unpaved roads. Future work addresses considerations to make the system more robust.

Pre-Weighted Midpoint Algorithm for Efficient Multiple-View Triangulation

Triangulation is a fundamental module in vision based navigation system, which reconstructs the observed 3D point using cameras with known intrinsic and poses. In this article, we adopt the minimization of the angular residual as the criterion of triangulation and a novel structure of multiple-view triangulation is proposed. A multiple-view triangulation problem is divided into several two-view triangulation problems and a linear algorithm is derived to find the explicit closed form solutions of these two-view triangulation problems. Then, the Cramer-Rao lower bound (CRLB) is employed to calculate their covariances, according to which these solutions are fused to obtain the final result of the multiple-view triangulation. Our proposed algorithm is proved to be efficient based on the theoretical analysis, as it is linear and non-iterative. The simulations on both synthetic and real data show that the proposed algorithm can achieve competitive accuracy within less processing time, comparing with the most state-of-art algorithms.

CamNav: a computer-vision indoor navigation system

We present CamNav, a vision-based navigation system that provides users with indoor navigation services. CamNav captures images in real time while the user is walking to recognize their current location. It does not require any installation of indoor localization devices. In this paper, we describe the techniques of our system that improve the recognition accuracy of an existing system that uses oriented FAST and rotated BRIEF (ORB) as part of its location-matching procedure. We employ multiscale local binary pattern (MSLBP) features to recognize places. We implement CamNav and conduct required experiments to compare the obtained accuracy when using ORB, the scale-invariant feature transform (SIFT), MSLBP features, and the combination of both ORB and SIFT features with MSLBP. A dataset composed of 42 classes was constructed for assessment. Each class contains 100 pictures designed for training one location and 24 pictures dedicated for testing. The evaluation results demonstrate that the place recognition accuracy while using MSLBP features is better than the accuracy when using SIFT features. The accuracy when using SIFT, MSLBP, and ORB features is 88.19%, 91.27%, and 96.33%, respectively. The overall accuracy of recognizing places increased to 93.55% and 97.52% after integrating MSLBP with SIFT with ORB, respectively.

Photometric-Planner for Visual Path Following

Robotic navigation is the aspect of cognition related to robot robust mobility. It combines perception, some knowledge on the environment and a set of goal poses to reliably control the robot during a mission that involves displacement. Vision-based autonomous navigation is an instantiation of the latter discipline where visual perception is used to control the robot and to represent the environment. This article presents a vision-based navigation system that uses its onboard camera to navigate and a visual path that represents the scene with a set of images. Being a memory-based system, the navigation is conceived as a concatenation of positioning tasks in the visual servoing scheme. The novelty on the proposed system relies on the generation of the images that compose the visual path. These images are rendered from a preobtained model of the scene. The experiments evaluate the performance of the system over three different scenes, contemplating indoor and outdoor environments, and using an Unmanned Aerial Vehicle as testbed.

A Distributed Vision-Based Navigation System for Khepera IV Mobile Robots.

This work presents the development and implementation of a distributed navigation system based on object recognition algorithms. The main goal is to introduce advanced algorithms for image processing and artificial intelligence techniques for teaching control of mobile robots. The autonomous system consists of a wheeled mobile robot with an integrated color camera. The robot navigates through a laboratory scenario where the track and several traffic signals must be detected and recognized by using the images acquired with its on-board camera. The images are sent to a computer server that performs a computer vision algorithm to recognize the objects. The computer calculates the corresponding speeds of the robot according to the object detected. The speeds are sent back to the robot, which acts to carry out the corresponding manoeuvre. Three different algorithms have been tested in simulation and a practical mobile robot laboratory. The results show an average of 84% success rate for object recognition in experiments with the real mobile robot platform.

Robust and Autonomous Stereo Visual-Inertial Navigation for Non-Holonomic Mobile Robots

Unlike micro aerial vehicles, most mobile robots have non-holonomic constraints, which makes lateral movement impossible. Consequently, the vision-based navigation systems that perform accurate visual feature initialization by moving the camera to the side to ensure a sufficient parallax of the image are degraded when applied to mobile robots. Generally, to overcome this difficulty, a motion model based on wheel encoders mounted on a mobile robot is used to predict the pose of a robot, but it is difficult to cope with errors caused by wheel slip or inaccurate wheel calibration. In this study, we propose a robust autonomous navigation system that uses only a stereo inertial sensor and does not rely on wheel-based dead reckoning. The observation model of the line feature modified with vanishing-points is applied to the visual-inertial odometry along with the point features so that a mobile robot can perform robust pose estimation during autonomous navigation. The proposed algorithm, i.e., keyframe-based autonomous visual-inertial navigation (KAVIN) supports the entire navigation system and can run onboard without an additional graphics processing unit. A series of experiments in a real environment indicated that the KAVIN system provides robust pose estimation without wheel encoders and prevents the accumulation of drift error during autonomous driving.

STEPS: An Indoor Navigation Framework for Mobile Devices.

This paper presents a vision-based navigation system designed for indoor localization. The suggested framework works as a standalone positioning system by fusing a sophisticated optical-flow pedometry with map constrains using an advanced particle filter. The presented method requires no personal calibration and works on standard smartphones with relatively low energy consumption. Field experiments on Android smartphones show that the expected error is about 1–2 m in most real-life scenarios.

Calibration and Registration of a Freehand Video-Guided Surgical Drill for Orthopaedic Trauma.

Pelvic trauma surgical procedures rely heavily on guidance with 2D fluoroscopy views for navigation in complex bone corridors. This "fluoro-hunting" paradigm results in extended radiation exposure and possible suboptimal guidewire placement from limited visualization of the fractures site with overlapped anatomy in 2D fluoroscopy. A novel computer vision-based navigation system for freehand guidewire insertion is proposed. The navigation framework is compatible with the rapid workflow in trauma surgery and bridges the gap between intraoperative fluoroscopy and preoperative CT images. The system uses a drill-mounted camera to detect and track poses of simple multimodality (optical/radiographic) markers for registration of the drill axis to fluoroscopy and, in turn, to CT. Surgical navigation is achieved with real-time display of the drill axis position on fluoroscopy views and, optionally, in 3D on the preoperative CT. The camera was corrected for lens distortion effects and calibrated for 3D pose estimation. Custom marker jigs were constructed to calibrate the drill axis and tooltip with respect to the camera frame. A testing platform for evaluation of the navigation system was developed, including a robotic arm for precise, repeatable, placement of the drill. Experiments were conducted for hand-eye calibration between the drill-mounted camera and the robot using the Park and Martin solver. Experiments using checkerboard calibration demonstrated subpixel accuracy [-0.01 ± 0.23 px] for camera distortion correction. The drill axis was calibrated using a cylindrical model and demonstrated sub-mm accuracy [0.14 ± 0.70 mm] and sub-degree angular deviation.