Vision-based Localization Research Articles

Vision-Based Localization and Mapping System for Navigating Autonomous Agricultural Vehicle

Vision-Based Localization and Mapping System for Navigating Autonomous Agricultural Vehicle

Vision based Indoor Localization Method via Convolution Neural Network

Existing indoor localization methods have bottleneck constraints such as multipath effect for Wi-Fi based methods, high cost for ultra-wide-band based methods and poor anti-interference for Bluetooth-based methods and so on. In order to avoid these problems, a vision-based indoor localization method is proposed. Firstly, the whole deployment environment is departed into several regions and each region is assigned to a location center. Then, in offline mode, the VGG16NET is pre-trained by ImageNet dataset and it is fine-tuned by images on a custom dataset towards indoor localization. In online mode, the fully trained and converged VGG16NET takes as input a video stream captured by the front RGB camera of a mobile robot and outputs features specific to the current location. The features are then used as input to an ArcFace classifier which outputs the current location of the mobile robot. Experimental results show that our method can estimate the location of a mobile object with imaging capability accurately in cluttered unstructured scenes without any other additional device. The localization accuracy can reach to 94.7%.

Divergent trinocular vision observers design for extended Kalman filter robot state estimation

Here, we report the design of two deterministic observers that exploit the capabilities of a home-made divergent trinocular visual sensor to sense depth data. The three-dimensional key points that the observers can measure are triangulated for visual odometry and estimated by an extended Kalman filter. This work deals with a four-wheel-drive mobile robot with four passive suspensions. The direct and inverse kinematic solutions are deduced and used for the updating and prediction models of the extended Kalman filter as feedback for the robot’s position controller. The state-estimation visual odometry results were compared with the robot’s dead-reckoning kinematics, and both are combined as a recursive position controller. One observer model design is based on the analytical geometric multi-view approach. The other observer model has fundamentals on multi-view lateral optical flow, which was reformulated as nonspatial–temporal and is modeled by an exponential function. This work presents the analytical deductions of the models and formulations. Experimental validation deals with five main aspects: multi-view correction, a geometric observer for range measurement, an optical flow observer for range measurement, dead-reckoning and visual odometry. Furthermore, comparison of positioning includes a four-wheel odometer, deterministic visual observers and the observer–extended Kalman filter, compared with a vision-based global reference localization system.

Leveraging experience for robust, adaptive nonlinear MPC on computationally constrained systems with time-varying state uncertainty

This paper presents a robust-adaptive nonlinear model predictive control (MPC) technique that leverages past experiences to achieve tractability on computationally constrained systems. We propose a robust extension of the Experience-driven Predictive Control (EPC) algorithm via a Gaussian belief propagation strategy that computes an uncertainty set, bounding the evolution of the system state in the presence of time-varying state uncertainty. This uncertainty set is used to tighten the constraints in the predictive control formulation via a chance-constrained approach, thereby providing a probabilistic guarantee of constraint satisfaction. The parameterized form of the controllers produced by EPC coupled with online uncertainty estimates ensures that this robust constraint satisfaction property persists, even as the system switches controllers and experiences variations in the uncertainty model. We validate the online performance and robust constraint satisfaction of the proposed Robust EPC algorithm through a series of trials with a simulated ground robot and three experimental platforms: (1) a small quadrotor aerial robot executing aggressive maneuvers in wind with degraded state estimates, (2) a skid-steer ground robot equipped with a laser-based localization system, and (3) a hexarotor aerial robot equipped with a vision-based localization system.

Humanoid navigation using a visual memory with obstacle avoidance

We present a complete humanoid navigation scheme based on a topological map known as visual memory (VM), which is composed by a set of key images acquired offline by means of a supervised teaching phase (human-guided). Our autonomous navigation scheme integrates the humanoid localization in the VM, a visual path planner and a path follower with obstacle avoidance. We propose a pure vision-based localization algorithm that takes advantage of the topological structure of the VM to find the key image that best fits the current image in terms of common visual information. In addition, the visual path planner benefits obstacle-free paths. The VM is updated when a new obstacle is detected with an RGB-D camera mounted on the humanoid’s head. The visual path following and obstacle avoidance problems are formulated in a unified sensor-based framework in which, a hierarchy of tasks is defined, and the transitions of consecutive and hierarchical tasks are performed smoothly to avoid instability of the humanoid. An extensive experimental evaluation using the NAO platform shows the good performance of the navigation scheme.

A hierarchical vision-based localization of rotor unmanned aerial vehicles for autonomous landing

The vision-based localization of rotor unmanned aerial vehicles for autonomous landing is challenging because of the limited detection range. In this article, to extend the vision detection and measurement range, a hierarchical vision-based localization method is proposed for unmanned aerial vehicle autonomous landing. In such a hierarchical framework, the landing is defined into three phases: “Approaching,”“Adjustment,” and “Touchdown,” in which visual artificial features at different scales can be detected from the designed object pattern for unmanned aerial vehicle pose recovery. The corresponding feature detection and pose estimation algorithms are also presented. In the end, typical simulation and field experiments have been carried out to illustrate the proposed method. The results show that our hierarchical vision-based localization has the ability to a consecutive unmanned aerial vehicle localization in a wider working range from far to near, which is significant for autonomous landing.

Low-Level Active Visual Navigation: Increasing Robustness of Vision-Based Localization Using Potential Fields

This letter proposes a low-level visual navigation algorithm to improve visual localization of a mobile robot. The algorithm, based on artificial potential fields, associates each feature in the current image frame with an attractive or neutral potential energy, with the objective of generating a control action that drives the vehicle towards the goal, while still favoring feature rich areas within a local scope, thus improving the localization performance. One key property of the proposed method is that it does not rely on mapping, and therefore it is a lightweight solution that can be deployed on miniaturized aerial robots, in which memory and computational power are major constraints. Simulations and real experimental results using a mini quadrotor equipped with a downward looking camera demonstrate that the proposed method can effectively drive the vehicle to a designated goal through a path that prevents localization failure.

A Hierarchical Vision-Based UAV Localization for an Open Landing

The localization of unmanned aerial vehicles (UAVs) for autonomous landing is challenging because the relative positions of the landing objects are almost inaccessible and the objects have nearly no transmission with UAVs. In this paper, a hierarchical vision-based localization framework for rotor UAVs is proposed for an open landing. In such a hierarchical framework, the landing is defined into three phases: “Approaching”, “Adjustment”, and “Touchdown”. Object features at different scales can be extracted from a designed Robust and Quick Response Landing Pattern (RQRLP) and the corresponding detection and localization methods are introduced for the three phases. Then a federated Extended Kalman Filter (EKF) structure is costumed and utilizes the solutions of the three phases as independent measurements to estimate the pose of the vehicle. The framework can be used to integrate the vision solutions and enables the estimation to be smooth and robust. In the end, several typical field experiments have been carried out to verify the proposed hierarchical vision framework. It can be seen that a wider localization range can be extended by the proposed framework while the precision is ensured.

Complementary Perception for Handheld SLAM

We present a novel method for mapping general three-dimensional environments, where sufficient geometric or visual information is not everywhere guaranteed and where the device motion is unconstrained as with handheld systems. The continuous-time simultaneous localization and mapping algorithm integrates a lidar, camera, and inertial measurement unit in a complementary fashion whereby all sensors contribute constraints to the optimization. The proposed algorithm is designed to expand the domain of mappable environments and therefore increase the reliability and utility of general purpose mobile mapping. A key component of the proposed algorithm is the incorporation of depth uncertainty into visual features, which is effective for noisy surfaces and allows features with and without depth estimates to be modeled in a unified manner. Results demonstrate a wider mappable domain on challenging environments compared to the state-of-the-art lidar or vision-based localization and mapping algorithms.

Where to Look? Predictive Perception With Applications to Planetary Exploration

Planetary rovers exploring the surface of Mars rely on vision-based localization and navigation algorithms to estimate their state and plan their motion during autonomous traverses. The accurate estimation of rover's motion enables safe navigation across environments with potentially hazardous terrain that would otherwise require careful human intervention. The accuracy of these vision-based localization and navigation methods are directly related to the amount of visual texture observed by the rover's sensors. This poses a challenge for Mars navigation, where texture-limited surfaces such as smooth sand is prevalent. To overcome this issue, we propose making use of a rover's ability to actively steer its visual sensors with the goal of maximizing actionable visual information content. This letter answers the question of where and when to look by presenting a method to predict the sensor trajectory that maximizes rover localization performance. This is accomplished through an online search of possible trajectories using synthetic, future camera views created from observed data. Proposed trajectories are quantified and chosen based on expected localization performance. We validate our algorithm in a high-fidelity simulation of a Mars-analogue environment and show how intelligently choosing where to look during a traverse can increase navigation accuracy compared to traditional fixed-sensor configurations.

Robust Visual Localization Across Seasons

Localization is an integral part of reliable robot navigation, and long-term autonomy requires robustness against perceptional changes in the environment during localization. In the context of vision-based localization, such changes can be caused by illumination variations, occlusion, structural development, different weather conditions, and seasons. In this paper, we present a novel approach for localizing a robot over longer periods of time using only monocular image data. We propose a novel data association approach for matching streams of incoming images to an image sequence stored in a database. Our method exploits network flows to leverage sequential information to improve the localization performance and to maintain several possible trajectories hypotheses in parallel. To compare images, we consider a semidense image description based on histogram of oriented gradients features as well as global descriptors from deep convolutional neural networks trained on ImageNet for robust localization. We perform extensive evaluations on a variety of datasets and show that our approach outperforms existing state-of-the-art approaches.

P-like controllers with collision avoidance for passive bilateral teleoperation of a UAV

PurposeThis paper aims to tele-operate the movement of an unmanned aerial vehicle (UAV) in the obstructed environment with asymmetric time-varying delays. A simple passive proportional velocity errors plus damping injection (P-like) controller is proposed to deal with the asymmetric time-varying delays in the aerial teleoperation system.Design/methodology/approachThis paper presents both theoretical and real-time experimental results of the bilateral teleoperation system of a UAV for collision avoidance over the wireless network. First, a position-velocity workspace mapping is used to solve the master-slave kinematic/dynamic dissimilarity. Second, a P-like controller is proposed to ensure the stability of the time-delayed bilateral teleoperation system with asymmetric time-varying delays. The stability is analyzed by the Lyapunov–Krasovskii function and the delay-dependent stability criteria are obtained under linear-matrix-inequalities conditions. Third, a vision-based localization is presented to calibrate the UAV’s pose and provide the relative distance for obstacle avoidance with a high accuracy. Finally, the performance of the teleoperation scheme is evaluated by both human-in-the-loop simulations and real-time experiments where a single UAV flies through the obstructed environment.FindingsExperimental results demonstrate that the teleoperation system can maintain passivity and collision avoidance can be achieved with a high accuracy for asymmetric time-varying delays. Moreover, the operator could tele-sense the force reflection to improve the maneuverability in the aerial teleoperation.Originality/valueA real-time bilateral teleoperation system of a UAV for collision avoidance is performed in the laboratory. A force and visual interface is designed to provide force and visual feedback of the slave environment to the operator.

Extended Kalman Filter for Stereo Vision-Based Localization and Mapping Applications

Image feature-based localization and mapping applications useful in field robotics are considered in this paper. Exploiting the continuity of image features and building upon the tracking algorithms that use point correspondences to provide an instantaneous localization solution, an extended Kalman filtering (EKF) approach is formulated for estimation of the rigid body motion of the camera coordinates with respect to the world coordinate system. Recent results by the authors in quantifying uncertainties associated with the feature tracking methods form the basis for deriving scene-dependent measurement error statistics that drive the optimal estimation approach. It is shown that the use of certain relative motion models between a static scene and the moving target can be recast as a recursive least squares problem and admits an efficient solution to the relative motion estimation problem that is amenable to real-time implementations on board mobile computing platforms with computational constraints. The utility of the estimation approaches developed in the paper is demonstrated using stereoscopic terrain mapping experiments carried out using mobile robots. The map uncertainties estimated by the filter are utilized to establish the registration of the local maps into the global coordinate system.

Visual Localization across Seasons Using Sequence Matching Based on Multi-Feature Combination.

Visual localization is widely used in autonomous navigation system and Advanced Driver Assistance Systems (ADAS). However, visual-based localization in seasonal changing situations is one of the most challenging topics in computer vision and the intelligent vehicle community. The difficulty of this task is related to the strong appearance changes that occur in scenes due to weather or season changes. In this paper, a place recognition based visual localization method is proposed, which realizes the localization by identifying previously visited places using the sequence matching method. It operates by matching query image sequences to an image database acquired previously (video acquired during traveling period). In this method, in order to improve matching accuracy, multi-feature is constructed by combining a global GIST descriptor and local binary feature CSLBP (Center-symmetric local binary patterns) to represent image sequence. Then, similarity measurement according to Chi-square distance is used for effective sequences matching. For experimental evaluation, the relationship between image sequence length and sequences matching performance is studied. To show its effectiveness, the proposed method is tested and evaluated in four seasons outdoor environments. The results have shown improved precision–recall performance against the state-of-the-art SeqSLAM algorithm.

The Survey of Vision-based 3D Modeling Techniques

This paper reviews the vision-based localization and map construction methods from the perspectives of VSLAM, SFM, 3DMax and Unity3D. It focuses on the key technologies and the latest research progress on each aspect, analyzes the advantages and disadvantages of each method, illustrates their implementation process and system framework, and further discusses the way to promote the combination for their complementary strength. Finally, the future opportunity of the combination of the four techniques is expected.

A survey on Visual-Based Localization: On the benefit of heterogeneous data

We are surrounded by plenty of information about our environment. From these multiple sources, numerous data could be extracted: set of images, 3D model, coloured points cloud... When classical localization devices failed (e.g. GPS sensor in cluttered environments), aforementioned data could be used within a localization framework. This is called Visual Based Localization (VBL). Due to numerous data types that can be collected from a scene, VBL encompasses a large amount of different methods. This paper presents a survey about recent methods that localize a visual acquisition system according to a known environment. We start by categorizing VBL methods into two distinct families: indirect and direct localization systems. As the localization environment is almost always dynamic, we pay special attention to methods designed to handle appearances changes occurring in a scene. Thereafter, we highlight methods exploiting heterogeneous types of data. Finally, we conclude the paper with a discussion on promising trends that could permit to a localization system to reach high precision pose estimation within an area as large as possible.

A review of feature detection and match algorithms for localization and mapping

Localization and mapping is an essential ability of a robot to keep track of its own location in an unknown environment. Among existing methods for this purpose, vision-based methods are more effective solutions for being accurate, inexpensive and versatile. Vision-based methods can generally be categorized as feature-based approaches and appearance-based approaches. The feature-based approaches prove higher performance in textured scenarios. However, their performance depend highly on the applied feature-detection algorithms. In this paper, we surveyed algorithms for feature detection, which is an essential step in achieving vision-based localization and mapping. In this pater, we present mathematical models of the algorithms one after another. To compare the performances of the algorithms, we conducted a series of experiments on their accuracy, speed, scale invariance and rotation invariance. The results of the experiments showed that ORB is the fastest algorithm in detecting and matching features, the speed of which is more than 10 times that of SURF and approximately 40 times that of SIFT. And SIFT, although with no advantage in terms of speed, shows the most correct matching pairs and proves its accuracy.

Robust and Accurate Monocular Vision-Based Localization in Outdoor Environments of Real-World Robot Challenge

This paper describes our approach to perform robust monocular camera metric localization in the dynamic environments of Tsukuba Challenge 2016. We address two issues related to vision-based navigation. First, we improved the coverage by building a custom vocabulary out of the scene and improving upon place recognition routine which is key for global localization. Second, we established possibility of lifelong localization by using previous year’s map. Experimental results show that localization coverage was higher than 90% for six different data sets taken in different years, while localization average errors were under 0.2 m. Finally, the average of coverage for data sets tested with maps taken in different years was of 75%.

Vision-based localization of the center of mass of large space debris via statistical shape analysis

The current overpopulation of artificial objects orbiting the Earth has increased the interest of the space agencies on planning missions for de-orbiting the largest inoperative satellites. Since this kind of operations involves the capture of the debris, the accurate knowledge of the position of their center of mass is a fundamental safety requirement. As ground observations are not sufficient to reach the required accuracy level, this information should be acquired in situ just before any contact between the chaser and the target. Some estimation methods in the literature rely on the usage of stereo cameras for tracking several features of the target surface. The actual positions of these features are estimated together with the location of the center of mass by state observers. The principal drawback of these methods is related to possible sudden disappearances of one or more features from the field of view of the cameras. An alternative method based on 3D Kinematic registration is presented in this paper. The method, which does not suffer of the mentioned drawback, considers a preliminary reduction of the inaccuracies in detecting features by the usage of statistical shape analysis.

Vision-based fast location of multi-bar code in any direction

The automatic location of the bar code is a key step in the bar code image recognition system. It is extremely confined that the generalization of the traditional bar code localization algorithms due to the requirements of both direction and quality of bar code, and most of them are only aimed at the single barcode localization. In this paper, we have proposed a novel multi-barcode location algorithm in arbitrary direction based on the accumulation of the linear gray value. First, the line coordinates of the barcode region is determined by the image normalized cross-correlation algorithm. Then the center line of gray value of cumulative distribution is used to analyze the barcode boundary and to determine the number of bar code within the region. Finally, the precise positioning of the barcode region is obtained. The experiments have demonstrated that our proposed method can be used to identify all the bar codes in any area, and automatically locate the bar codes in any direction.