Vision-based Navigation System Research Articles

On-ground validation of orbital GNC: Visual navigation assessment in robotic testbed facility

AbstractCubeSats have become versatile platforms for various space missions (e.g., on-orbit servicing and debris removal) owing to their low cost and flexibility. Many space tasks involve proximity operations that require precise guidance, navigation, and control (GNC) algorithms. Vision-based navigation is attracting interest for such operations. However, extreme lighting conditions in space challenge optical techniques. The on-ground validation of such navigation systems for orbital GNC becomes crucial to ensure their reliability during space operations. These systems undergo rigorous testing within their anticipated operational parameters, including the exploration of potential edge cases. The ability of GNC algorithms to function effectively under extreme space conditions that exceed anticipated scenarios is crucial, particularly in space missions where the scope of errors is negligible. This paper presents the ground validation of a GNC algorithm designed for autonomous satellite rendezvous by leveraging hardware-in-the-loop experiments. This study focuses on two key areas. First, the rationale underlying the augmentation of the robot workspace (six-degree-of-freedom UR10e robot + linear rail) is investigated to emulate relatively longer trajectories with complete position and orientation states. Second, the control algorithm is assessed in response to uncertain pose observations from a vision-based navigation system. The results indicate increased control costs with uncertain navigation and exemplify the importance of on-ground testing for system validation before launch, particularly in extreme cases that are typically difficult to assess using software-based testing.

Visual teach and generalise (VTAG)—Exploiting perceptual aliasing for scalable autonomous robotic navigation in horticultural environments

Nowadays, most agricultural robots rely on precise and expensive localisation, typically based on global navigation satellite systems (GNSS) and real-time kinematic (RTK) receivers. Unfortunately, the precision of GNSS localisation significantly decreases in environments where the signal paths between the receiver and the satellites are obstructed. This precision hampers deployments of these robots in, e.g., polytunnels or forests. An attractive alternative to GNSS is vision-based localisation and navigation. However, perceptual aliasing and landmark deficiency, typical for agricultural environments, cause traditional image processing techniques, such as feature matching, to fail. We propose an approach for an affordable pure vision-based navigation system which is not only robust to perceptual aliasing, but it actually exploits the repetitiveness of agricultural environments. Our system extends the classic concept of visual teach and repeat to visual teach and generalise (VTAG). Our teach and generalise method uses a deep learning-based image registration pipeline to register similar images through meaningful generalised representations obtained from different but similar areas. The proposed system uses only a low-cost uncalibrated monocular camera and the robot’s wheel odometry to produce heading corrections to traverse crop rows in polytunnels safely. We evaluate this method at our test farm and at a commercial farm on three different robotic platforms where an operator teaches only a single crop row. With all platforms, the method successfully navigates the majority of rows with most interventions required at the end of the rows, where the camera no longer has a view of any repeating landmarks such as poles, crop row tables or rows which have visually different features to that of the taught row. For one robot which was taught one row 25 m long our approach autonomously navigated the robot a total distance of over 3.5 km, reaching a teach-generalisation gain of 140.

Monocular Based Navigation System for Autonomous Ground Robots Using Multiple Deep Learning Models

In recent years, the development of ground robots with human-like perception capabilities has led to the use of multiple sensors, including cameras, lidars, and radars, along with deep learning techniques for detecting and recognizing objects and estimating distances. This paper proposes a computer vision-based navigation system that integrates object detection, segmentation, and monocular depth estimation using deep neural networks to identify predefined target objects and navigate towards them with a single monocular camera as a sensor. Our experiments include different sensitivity analyses to evaluate the impact of monocular cues on distance estimation. We show that this system can provide a ground robot with the perception capabilities needed for autonomous navigation in unknown indoor environments without the need for prior mapping or external positioning systems. This technique provides an efficient and cost-effective means of navigation, overcoming the limitations of other navigation techniques such as GPS-based and SLAM-based navigation.Graphical

A Vision-Based Navigation System With Markerless Image Registration and Position-Sensing Localization for Oral and Maxillofacial Surgery

The quality of oral and maxillofacial surgery (OMS) significantly depends on the accuracy of surgical navigation. In this article, a vision-based markerless surgical navigation system is developed to overcome the shortcomings in the currently available technologies. Registration methods both for patient and surgical instrument tracking are improved to increase the navigation performance. For patient-image registration, we propose an efficient texture-less pose estimation method using only shape information. An innovative strategy is developed to effectively reject the outliers and improve the pose accuracy, which is the first attempt at introducing geometric matching information to guide PnP calculation. For surgical instrument tracking, a position-sensing marker is used to achieve robust and convenient instrument localization with high accuracy. Experiments were conducted on the 3-D-printed maxilla and mandible models to evaluate the navigation performance. Evaluation results validate the effectiveness of the proposed pose estimation method in improving the pose accuracy for texture-less teeth. Besides, it is revealed that the position-sensing marker can be localized with high accuracy even under nonideal visibility conditions, which expands the motion range of the instrument and decreases the size of the tool. The entire system has a sufficiently small target registration error (TRE). These experimental results have verified that the proposed surgical navigation system can provide practical guidance for OMS with satisfactory accuracy.

UNav: An Infrastructure-Independent Vision-Based Navigation System for People with Blindness and Low Vision.

Vision-based localization approaches now underpin newly emerging navigation pipelines for myriad use cases, from robotics to assistive technologies. Compared to sensor-based solutions, vision-based localization does not require pre-installed sensor infrastructure, which is costly, time-consuming, and/or often infeasible at scale. Herein, we propose a novel vision-based localization pipeline for a specific use case: navigation support for end users with blindness and low vision. Given a query image taken by an end user on a mobile application, the pipeline leverages a visual place recognition (VPR) algorithm to find similar images in a reference image database of the target space. The geolocations of these similar images are utilized in a downstream task that employs a weighted-average method to estimate the end user's location. Another downstream task utilizes the perspective-n-point (PnP) algorithm to estimate the end user's direction by exploiting the 2D-3D point correspondences between the query image and the 3D environment, as extracted from matched images in the database. Additionally, this system implements Dijkstra's algorithm to calculate a shortest path based on a navigable map that includes the trip origin and destination. The topometric map used for localization and navigation is built using a customized graphical user interface that projects a 3D reconstructed sparse map, built from a sequence of images, to the corresponding a priori 2D floor plan. Sequential images used for map construction can be collected in a pre-mapping step or scavenged through public databases/citizen science. The end-to-end system can be installed on any internet-accessible device with a camera that hosts a custom mobile application. For evaluation purposes, mapping and localization were tested in a complex hospital environment. The evaluation results demonstrate that our system can achieve localization with an average error of less than 1 m without knowledge of the camera's intrinsic parameters, such as focal length.

Relative Positioning Method for UAVs Based on Multi-Source Information Fusion

To improve the accuracy and reliability of the relative positioning for unmanned aerial vehicles (UAVs), a relative positioning method based on multi-source information fusion is proposed. An integrated positioning scheme is constructed by the Beidou Navigation Satellite System (BDS) receivers, Global Positioning System (GPS) receivers, Vision-Based Navigation System (VisNav), and Inertial Navigation System (INS). The BDS pseudorange relative difference equation, the GPS relative difference equation, the relative line-of-sight vector equation, and the INS measurement equation are established, respectively. The least squared (LS) method is used to realize information fusion, and the Gauss-Newton method is used to iteratively solve the relative position results. Finally, numerical simulation and result analysis are conducted with different sensor configurations. Simulation results show that BDS/INS/GPS/VisNav relative positioning result is obviously better than that of INS/GPS, INS/VisNav, and INS/GPS/VisNav, and the proposed method reduces the relative positioning errors and has a higher accuracy and excellent robustness. The research result is suitable for application scenarios with high navigation accuracy requirement such as AAR and intelligent swarm formation control.

Design and evaluation of hedge trimmer robot

An autonomous hedge trimmer robot was developed to reduce the harmful effects of traditional hedge trimming method (gasoline-powered hedge trimmer) on operators' bodies and increase the speed and quality of the operation. The robot had five degrees of freedom (PRRRR) structure and was geometrically optimized to simplify the control strategy, enabling it to trim simple, stepped, and circular forms. The control system, consisting of a host computer, servo motors and servo drivers, and controller, guided the robot platform, manipulator, and end-effector as it approached and trimmed the hedge. By using a recognition algorithm and vision-based navigation system, the robot was able to detect the path and trim the hedge automatically. Experimental tests showed the vision-based system is accurate and fast enough to control the robot. The accuracy of hedge detection performance under various conditions was about 96% and the success rate of trimming was 88%.

Optical navigation for Lunar landing based on Convolutional Neural Network crater detector

Traditional vision-based navigation algorithms are highly affected from non-nominal conditions, which comprise illumination conditions and environmental uncertainties. Thanks to the outstanding generalization capability and flexibility, deep neural networks (and AI algorithms in general) are excellent candidates to solve the aforementioned shortcoming of navigation algorithms. The paper presents a vision-based navigation system using a Convolutional Neural Network to solve the task of pinpoint landing on the Moon using absolute navigation, namely with respect to the Mean Earth/Polar Axis reference frame. The Moon landing scenario consists in the spacecraft descent on the South Pole from a parking orbit to the powered descent phase. The architecture features an Object Detection Convolutional Neural Network (ODN) trained with supervised learning approach. The CNN is used to extract features of the observed craters that are then processed by standard image processing algorithms in order to provide pseudo-measurements that can be used by navigation filter. The craters are matched against a database containing the inertial location of the known craters. An Extended Kalman Filter with time-delayed measurements integration is developed to fuse optical and altimeter information.

Autonomous Underwater Vehicle with Vision-based Navigation System for Underwater Robot Competition

Autonomous Underwater Vehicle with Vision-based Navigation System for Underwater Robot Competition

A Vision-based Navigation System for an Agricultural Autonomous Tractor

RTK-GNSS (Real-time Kinematic-Global Navigation Satellite System) signal is necessary for the safe navigation of an autonomous tractor. However, the main problem associated with this type of precise GNSS is that the RTK correction signal might not be available at all geographical locations. Tree canopies present close to the tractors’ vicinity obstruct satellite signals and degrade the position accuracy. This can lead the autonomous tractor to travel outside of its predetermined path, which is unsafe. To avoid such situations, this study aims to develop a vision-based road detection system, which does not rely on RTK-GNSS. This system aims to detect both paved and unpaved roads in rural farm areas. It takes the RGB images obtained from an HD onboard camera and segments the road surface and road edges. The segmentation is performed by using several image filters, the sliding window method, and a set of rules that are determined manually. Finally, the lateral error obtained from the machine vision system is transmitted to the tractor's automatic navigation system. Experimental runs performed at the experimental farm of Hokkaido University showed that the lateral error calculated by this system is less than 0.2 m for unpaved roads and 0.4 m for paved roads. This shows that the system can detect both paved and unpaved roads. Future work addresses considerations to make the system more robust.

Pre-Weighted Midpoint Algorithm for Efficient Multiple-View Triangulation

Triangulation is a fundamental module in vision based navigation system, which reconstructs the observed 3D point using cameras with known intrinsic and poses. In this article, we adopt the minimization of the angular residual as the criterion of triangulation and a novel structure of multiple-view triangulation is proposed. A multiple-view triangulation problem is divided into several two-view triangulation problems and a linear algorithm is derived to find the explicit closed form solutions of these two-view triangulation problems. Then, the Cramer-Rao lower bound (CRLB) is employed to calculate their covariances, according to which these solutions are fused to obtain the final result of the multiple-view triangulation. Our proposed algorithm is proved to be efficient based on the theoretical analysis, as it is linear and non-iterative. The simulations on both synthetic and real data show that the proposed algorithm can achieve competitive accuracy within less processing time, comparing with the most state-of-art algorithms.

CamNav: a computer-vision indoor navigation system

We present CamNav, a vision-based navigation system that provides users with indoor navigation services. CamNav captures images in real time while the user is walking to recognize their current location. It does not require any installation of indoor localization devices. In this paper, we describe the techniques of our system that improve the recognition accuracy of an existing system that uses oriented FAST and rotated BRIEF (ORB) as part of its location-matching procedure. We employ multiscale local binary pattern (MSLBP) features to recognize places. We implement CamNav and conduct required experiments to compare the obtained accuracy when using ORB, the scale-invariant feature transform (SIFT), MSLBP features, and the combination of both ORB and SIFT features with MSLBP. A dataset composed of 42 classes was constructed for assessment. Each class contains 100 pictures designed for training one location and 24 pictures dedicated for testing. The evaluation results demonstrate that the place recognition accuracy while using MSLBP features is better than the accuracy when using SIFT features. The accuracy when using SIFT, MSLBP, and ORB features is 88.19%, 91.27%, and 96.33%, respectively. The overall accuracy of recognizing places increased to 93.55% and 97.52% after integrating MSLBP with SIFT with ORB, respectively.

Photometric-Planner for Visual Path Following

Robotic navigation is the aspect of cognition related to robot robust mobility. It combines perception, some knowledge on the environment and a set of goal poses to reliably control the robot during a mission that involves displacement. Vision-based autonomous navigation is an instantiation of the latter discipline where visual perception is used to control the robot and to represent the environment. This article presents a vision-based navigation system that uses its onboard camera to navigate and a visual path that represents the scene with a set of images. Being a memory-based system, the navigation is conceived as a concatenation of positioning tasks in the visual servoing scheme. The novelty on the proposed system relies on the generation of the images that compose the visual path. These images are rendered from a preobtained model of the scene. The experiments evaluate the performance of the system over three different scenes, contemplating indoor and outdoor environments, and using an Unmanned Aerial Vehicle as testbed.

A Distributed Vision-Based Navigation System for Khepera IV Mobile Robots.

This work presents the development and implementation of a distributed navigation system based on object recognition algorithms. The main goal is to introduce advanced algorithms for image processing and artificial intelligence techniques for teaching control of mobile robots. The autonomous system consists of a wheeled mobile robot with an integrated color camera. The robot navigates through a laboratory scenario where the track and several traffic signals must be detected and recognized by using the images acquired with its on-board camera. The images are sent to a computer server that performs a computer vision algorithm to recognize the objects. The computer calculates the corresponding speeds of the robot according to the object detected. The speeds are sent back to the robot, which acts to carry out the corresponding manoeuvre. Three different algorithms have been tested in simulation and a practical mobile robot laboratory. The results show an average of 84% success rate for object recognition in experiments with the real mobile robot platform.

Robust and Autonomous Stereo Visual-Inertial Navigation for Non-Holonomic Mobile Robots

Unlike micro aerial vehicles, most mobile robots have non-holonomic constraints, which makes lateral movement impossible. Consequently, the vision-based navigation systems that perform accurate visual feature initialization by moving the camera to the side to ensure a sufficient parallax of the image are degraded when applied to mobile robots. Generally, to overcome this difficulty, a motion model based on wheel encoders mounted on a mobile robot is used to predict the pose of a robot, but it is difficult to cope with errors caused by wheel slip or inaccurate wheel calibration. In this study, we propose a robust autonomous navigation system that uses only a stereo inertial sensor and does not rely on wheel-based dead reckoning. The observation model of the line feature modified with vanishing-points is applied to the visual-inertial odometry along with the point features so that a mobile robot can perform robust pose estimation during autonomous navigation. The proposed algorithm, i.e., keyframe-based autonomous visual-inertial navigation (KAVIN) supports the entire navigation system and can run onboard without an additional graphics processing unit. A series of experiments in a real environment indicated that the KAVIN system provides robust pose estimation without wheel encoders and prevents the accumulation of drift error during autonomous driving.

STEPS: An Indoor Navigation Framework for Mobile Devices.

This paper presents a vision-based navigation system designed for indoor localization. The suggested framework works as a standalone positioning system by fusing a sophisticated optical-flow pedometry with map constrains using an advanced particle filter. The presented method requires no personal calibration and works on standard smartphones with relatively low energy consumption. Field experiments on Android smartphones show that the expected error is about 1–2 m in most real-life scenarios.

Calibration and Registration of a Freehand Video-Guided Surgical Drill for Orthopaedic Trauma.

Pelvic trauma surgical procedures rely heavily on guidance with 2D fluoroscopy views for navigation in complex bone corridors. This "fluoro-hunting" paradigm results in extended radiation exposure and possible suboptimal guidewire placement from limited visualization of the fractures site with overlapped anatomy in 2D fluoroscopy. A novel computer vision-based navigation system for freehand guidewire insertion is proposed. The navigation framework is compatible with the rapid workflow in trauma surgery and bridges the gap between intraoperative fluoroscopy and preoperative CT images. The system uses a drill-mounted camera to detect and track poses of simple multimodality (optical/radiographic) markers for registration of the drill axis to fluoroscopy and, in turn, to CT. Surgical navigation is achieved with real-time display of the drill axis position on fluoroscopy views and, optionally, in 3D on the preoperative CT. The camera was corrected for lens distortion effects and calibrated for 3D pose estimation. Custom marker jigs were constructed to calibrate the drill axis and tooltip with respect to the camera frame. A testing platform for evaluation of the navigation system was developed, including a robotic arm for precise, repeatable, placement of the drill. Experiments were conducted for hand-eye calibration between the drill-mounted camera and the robot using the Park and Martin solver. Experiments using checkerboard calibration demonstrated subpixel accuracy [-0.01 ± 0.23 px] for camera distortion correction. The drill axis was calibrated using a cylindrical model and demonstrated sub-mm accuracy [0.14 ± 0.70 mm] and sub-degree angular deviation.

A Perception-Aware Flatness-Based Model Predictive Controller for Fast Vision-Based Multirotor Flight

Abstract Despite the push toward fast, reliable vision-based multirotor flight, most vision-based navigation systems still rely on controllers that are perception-agnostic. Given that these controllers ignore their effect on the system’s localisation capabilities, they can produce an action that allows vision-based localisation (and consequently navigation) to fail. In this paper, we present a perception-aware flatness-based model predictive controller (MPC) that accounts for its effect on visual localisation. To achieve perception awareness, we first develop a simple geometric model that uses over 12 km of flight data from two different environments (urban and rural) to associate visual landmarks with a probability of being successfully matched. In order to ensure localisation, we integrate this model as a chance constraint in our MPC such that we are probabilistically guaranteed that the number of successfully matched visual landmarks exceeds a minimum threshold. We show how to simplify the chance constraint to a nonlinear, deterministic constraint on the position of the multirotor. With desired speeds of 10 m/s, we demonstrate in simulation (based on real-world perception data) how our proposed perception-aware MPC is able to achieve faster flight while guaranteeing localisation compared to similar perception-agnostic controllers. We illustrate how our perception-aware MPC adapts the path constraint along the path based on the perception model by accounting for camera orientation, path error and location of the visual landmarks. The result is that repeating the same geometric path but with the camera facing in opposite directions can lead to different optimal paths flown.

Improved vehicle navigation using sensor fusion of inertial, odometeric sensors with global positioning system

The primary objective of a navigation system is to continuously monitor the trajectory of a vehicle. Navigation for a land vehicle is implemented using different measurement systems, such as Inertial Navigation System (INS), radio navigation system, vision-based navigation and Global Positioning System (GPS). INS provides continuous information of position, velocity and attitude. However, its performance deteriorates with time since the errors tend to accumulate. GPS is an electromagnetic signal which is more accurate when compared to INS but cannot provide continuous and reliable position all the time. The drawbacks of these individual systems have given rise to the need for higher accuracy, integrity and robustness. This has led to the fusion of measurements from these sensors to obtain an improved performance in measuring the position, velocity of a vehicle. This paper discusses the simulation and implementation of an integrated navigation system using inertial, odometric sensors with GPS using scaled unscented Kalman filter. The method discussed involves obtaining a state transition model and a measurement model of the sensors and processing the states using scaled unscented Kalman filter to obtain better estimates of position and velocity.

Improved vehicle navigation using sensor fusion of inertial, odometeric sensors with global positioning system

The primary objective of a navigation system is to continuously monitor the trajectory of a vehicle. Navigation for a land vehicle is implemented using different measurement systems, such as Inertial Navigation System (INS), radio navigation system, vision-based navigation and Global Positioning System (GPS). INS provides continuous information of position, velocity and attitude. However, its performance deteriorates with time since the errors tend to accumulate. GPS is an electromagnetic signal which is more accurate when compared to INS but cannot provide continuous and reliable position all the time. The drawbacks of these individual systems have given rise to the need for higher accuracy, integrity and robustness. This has led to the fusion of measurements from these sensors to obtain an improved performance in measuring the position, velocity of a vehicle. This paper discusses the simulation and implementation of an integrated navigation system using inertial, odometric sensors with GPS using scaled unscented Kalman filter. The method discussed involves obtaining a state transition model and a measurement model of the sensors and processing the states using scaled unscented Kalman filter to obtain better estimates of position and velocity.