Visual Localization System Research Articles

An RGB-D Camera Based Visual Positioning System for Assistive Navigation by a Robotic Navigation Aid

There are about 253 million people with visual impairment worldwide. Many of them use a white cane and/or a guide dog as the mobility tool for daily travel. Despite decades of efforts, electronic navigation aid that can replace white cane is still research in progress. In this paper, we propose an RGB-D camera based visual positioning system (VPS) for real-time localization of a robotic navigation aid (RNA) in an architectural floor plan for assistive navigation. The core of the system is the combination of a new 6-DOF depth-enhanced visual-inertial odometry (DVIO) method and a particle filter localization (PFL) method. DVIO estimates RNA's pose by using the data from an RGB-D camera and an inertial measurement unit (IMU). It extracts the floor plane from the camera's depth data and tightly couples the floor plane, the visual features (with and without depth data), and the IMU's inertial data in a graph optimization framework to estimate the device's 6-DOF pose. Due to the use of the floor plane and depth data from the RGB-D camera, DVIO has a better pose estimation accuracy than the conventional VIO method. To reduce the accumulated pose error of DVIO for navigation in a large indoor space, we developed the PFL method to locate RNA in the floor plan. PFL leverages geometric information of the architectural CAD drawing of an indoor space to further reduce the error of the DVIO-estimated pose. Based on VPS, an assistive navigation system is developed for the RNA prototype to assist a visually impaired person in navigating a large indoor space. Experimental results demonstrate that: 1) DVIO method achieves better pose estimation accuracy than the state-of-the-art VIO method and performs real-time pose estimation (18 Hz pose update rate) on a UP Board computer; 2) PFL reduces the DVIO-accrued pose error by 82.5% on average and allows for accurate wayfinding (endpoint position error ≤ 45 cm) in large indoor spaces.

UAV navigation based on adaptive fuzzy backstepping controller using visual odometry

ABSTRACT This study investigates the 3D trajectory tracking of a quadrotor-type unmanned aerial vehicle (UAV) using a visual localization system integrated in the feedback control loop. First, adaptive stereo visual odometry (SVO) is proposed to solve the UAV localization problem. The proposed solution mainly employs adaptive iterative closest point speeded-up robust features. Second, a vision-based trajectory tracking algorithm is implemented using a backstepping controller, whose parameters are optimized using the particle swarm optimization algorithm. Finally, to avoid the limitations of visual localization (e.g. dark environment, uniform area, and dynamic scene), the visual pose is supported by the inertial pose obtained using a fuzzy adaptive Kalman filter (FAKF). Based on the number and average depth of the estimated 3D points, the FAKF algorithm adaptively tunes the extended Kalman filter parameters. The proposed algorithms are validated using simulation and experimental data. Many scenarios are considered with different trajectories. Good performance is achieved, confirming the efficiency of the proposed approach.

Condition-Invariant Robot Localization Using Global Sequence Alignment of Deep Features.

Localization is one of the essential process in robotics, as it plays an important role in autonomous navigation, simultaneous localization, and mapping for mobile robots. As robots perform large-scale and long-term operations, identifying the same locations in a changing environment has become an important problem. In this paper, we describe a robust visual localization system under severe appearance changes. First, a robust feature extraction method based on a deep variational autoencoder is described to calculate the similarity between images. Then, a global sequence alignment is proposed to find the actual trajectory of the robot. To align sequences, local fragments are detected from the similarity matrix and connected using a rectangle chaining algorithm considering the robot’s motion constraint. Since the chained fragments provide reliable clues to find the global path, false matches on featureless structures or partial failures during the alignment could be recovered and perform accurate robot localization in changing environments. The presented experimental results demonstrated the benefits of the proposed method, which outperformed existing algorithms in long-term conditions.

On-Line Precision Calibration of Mobile Manipulators Based on the Multi-Level Measurement Strategy

Autonomous industrial mobile manipulator (AIMM), which is composed of a mobile platform and a robot arm of six degrees of freedom (DoFs), can move autonomously and perform dexterous tasks independently in the industrial environment. A novel on-line calibration system for AIMM in real manufacturing workshop is proposed in this paper. Compared to calibration scenarios of fixed-base robot, we focus on the large-space autonomous calibration for AIMM. To cope with the issue of high precision calibration in large space, we propose a multi-level guided-based measurement strategy, which includes the low-cost vision positioning and guidance system, and the high-precision distributed base station (BS) measurement system. The corresponding experiment proves that the total uncertainty of measurement system is less than 0.1 mm. Based on the measured data of end-effector installed at the end of robot arm, the AIMM can perform on-line calibration tasks anywhere automatically without taking into account the expensive devices for the high-precision location of mobile platform and the downtime for the robotic off-line calibration. The mean positioning error of AIMM after calibration reduces 93% based on the calibration experiment.

A Robust Invariant Local Feature Matching Method for Changing Scenes

The precise evaluation of camera position and orientation is a momentous procedure of most machine vision tasks, especially visual localization. Aiming at the shortcomings of local features of dealing with changing scenes and the problem of realizing a robust end‐to‐end network that worked from feature detection to matching, an invariant local feature matching method for changing scene image pairs is proposed, which is a network that integrates feature detection, descriptor constitution, and feature matching. In the feature point detection and descriptor construction stage, joint training is carried out based on a neural network. In the feature point extraction and descriptor construction stage, joint training is carried out based on a neural network. To obtain local features with solid robustness to viewpoint and illumination changes, the Vector of Locally Aggregated Descriptors based on Neural Network (NetVLAD) module is introduced to compute the degree of correlation of description vectors from one image to another counterpart. Then, to enhance the relationship between relevant local features of image pairs, the attentional graph neural network (AGNN) is introduced, and the Sinkhorn algorithm is used to match them; finally, the local feature matching results between image pairs are output. The experimental results show that, compared with the existed algorithms, the proposed method enhances the robustness of local features of varying sights, performs better in terms of homography estimation, matching precision, and recall, and when meeting the requirements of the visual localization system to the environment, the end‐to‐end network tasks can be realized.

The geometrical analysis of localization error characteristic in stereo vision systems.

The parameters of a stereo camera directly determine the accuracy of a stereo vision positioning system. In this paper, the relationship between camera parameter error and positioning error is established by using the geometric analysis method with a clear physical meaning. The error models of the relative position error and camera resolution error of the stereo camera are established. The relative position error of the stereo camera is represented by the rotation and translation of the coordinate system, and the positioning error is derived from image coordinates through the pinhole camera model. For the camera resolution error, the two pixels of the stereo camera are projected into an octahedral uncertain region in space. We use geometric methods to derive the maximum size of the octahedron in the three-axis direction. The maximum size in the three-axis direction is used to evaluate the impact of camera resolution errors. The depth of field and angle of view are used to represent the measured area. The effects of resolution error and camera parameters on the measurement error of each axis are analyzed. Finally, a large number of simulations verified our conclusion. By comparing the effects of the magnitude of the error, we can conclude that the baseline and pixel size of the camera have a greater impact on the positioning accuracy.

Visual Simultaneous Localization and Mapping with Applications to Monitoring of Underground Transportation Infrastructure

This paper presents the results of the development, design, and implementation of a visual simultaneous localization and mapping (SLAM) system for autonomous real-time localization with application to underground transportation infrastructure (UTI) such as tunnels. Localization is achieved in the absence of any global positioning system (GPS) or auxiliary system. The indoor localization system is a necessary element of a fully autonomous platform for the detection of cracks and other anomalies on the interior surfaces of tunnels and other UTI. It can be used for tagging of high-resolution sensor data obtained with low-cost prototype data acquisition platforms previously developed. Visual based SLAM has been used as the core element in an architecture employing a commercial off-the-shelf (COTS) ZED stereo camera from Stereolabs. To achieve real-time operation, an NVIDIA Jetson TX2 massively parallel graphics processing unit (GPU) was used as the core computational engine employing two different software libraries. We achieved localization at 5 frames per second (FPS) using ORBSLAM2 open-source software library, and the much lighter, but proprietary, ZED SDK was able to deliver a performance at nearly 60 FPS. To assess the accuracy of the relative localization system, we conducted several tests at 30 FPS and reported on the resulting error variances that were found to be consistently very small. Finally, we conducted several tests in a tunnel in the Los Angeles county area and confirmed the applicability of the method for monitoring UTI.

Deep-trained illumination-robust precision positioning for real-time manipulation of embedded objects

Automated manipulation guided by a precision visual positioning system is realized based on a convolutional neural network (ConvNet) trained on coordinate-explicit target images. Whereas ConvNet has exhibited superior illumination adaptability, it is still possible that high-contrast illumination effects may reduce the success rate of visually guided manipulation in the factory. In this paper, an automated training data generation scheme is proposed to enhance the performance of the visual positioning ConvNet under high-contrast shadows cast by other objects. By collecting dozens of illumination patterns, illumination templates are created by the image-to-image translation GAN (pix2pix GAN). The templates are then applied to the basis photo of the target object to spawn multiple virtual images which vastly enrich the illumination diversity of the training set for the visual positioning ConvNet. Experiments were conducted to verify the effectiveness of the pix2pix GAN illumination generation module. Experimental results also showed that the positioning accuracy augmented by the illumination module reaches above 85%, while without the illumination-augmented training, the positioning accuracy is below 15%. Experiments on automated visual manipulation with a 5-DOF manipulator also confirmed the feasibility of adopting the proposed framework for real-time operations. With the illumination-augmented training, the manipulation success rate is above 90%; without it, the success rate is less than 40%. We provide the data used for training pix2pix GAN to generate the illumination templates at https://github.com/ntutindustry40/OneShot-II .

Accurate Calibration for Large-Scale Tracking-Based Visual Measurement System

The combination of a large-field-of-view visual positioning system and a local scanning sensor affords an effective approach to the 3-D measurement of meter-scale objects and has been actively investigated by researchers. The calibration of such an integrated measurement system is essential to establish the overall measurement accuracy. This article proposes to calculate the transformation relationship between the local system and the transfer target through the determination of the scales of the combined system. The principle and implementation of this tracking-based combined measurement system are described in this article, as well as the use of the respective scales of the combined system to establish a model of the applicable geometrical relationships. The Kronecker product was used to solve the model equations, after which homogenous transformation matrix optimization was performed. An error model of the camera-based measurement system was also established based on a combination of the image distortion and the working volume. The 2-D and 3-D errors of the local and global sensors were further quantitatively simulated and visualized to demonstrate the calibration accuracy. Standard plane and complex large object measurement experiments were performed to verify the accuracy of the proposed measurement system and its use for the reconstruction of a large shape.

Error analysis based on error transfer theory and compensation strategy for LED chip visual localization systems

In the manufacturing process of LED chips, the accuracy of the LED chip visual localization system affects the quality of LED chip production directly. There are many errors that have impacts on positioning system accuracy. Therefore, the identification and compensation of the critical errors is key to efficiently improving the precision of the system. Based on this fact, an error analysis method and an error compensation strategy are proposed in this paper. The first step was to measure the relevant error sources that may affect the localization system. Then error model of localization system was established, and the validity of this error model was verified by comparing simulated and actual positioning results. In addition, the impact factors of each error source on localization system accuracy were obtained using the error transfer theory. According to the error analysis results, an efficient error compensation strategy was proposed, which could compensate the errors in order of impact factors, and judge whether the error compensation method is optimal. Finally, the experimental results proved that the proposed error analysis method was valid and the error compensation strategy could efficiently enhance the positioning accuracy to meet the industrial application requirements.

Multimodal localization: Stereo over LiDAR map

AbstractIn this paper, we present a real‐time high‐precision visual localization system for an autonomous vehicle which employs only low‐cost stereo cameras to localize the vehicle with a priori map built using a more expensive 3D LiDAR sensor. To this end, we construct two different visual maps: a sparse feature visual map for visual odometry (VO) based motion tracking, and a semidense visual map for registration with the prior LiDAR map. To register two point clouds sourced from different modalities (i.e., cameras and LiDAR), we leverage probabilistic weighted normal distributions transformation (ProW‐NDT), by particularly taking into account the uncertainty of source point clouds. The registration results are then fused via pose graph optimization to correct the VO drift. Moreover, surfels extracted from the prior LiDAR map are used to refine the sparse 3D visual features that will further improve VO‐based motion estimation. The proposed system has been tested extensively in both simulated and real‐world experiments, showing that robust, high‐precision, real‐time localization can be achieved.

Improving Visual Localization Accuracy in Dynamic Environments Based on Dynamic Region Removal

Visual localization is a fundamental capability in robotics and has been well studied for recent decades. Although many state-of-the-art algorithms have been proposed, great success usually builds on the assumption that the working environment is static. In most of the real scenes, the assumption cannot hold because there are inevitably moving objects, especially humans, which significantly degrade the localization accuracy. To address this problem, we propose a robust visual localization system building on top of a feature-based visual simultaneous localization and mapping algorithm. We design a dynamic region detection method and use it to preprocess the input frame. The detection process is achieved in a Bayesian framework which considers both the prior knowledge generated from an object detection process and observation information. After getting the detection result, feature points extracted from only the static regions will be used for further visual localization. We performed the experiments on the public TUM data set and our recorded data set, which shows the daily dynamic scenarios. Both qualitative and quantitative results are provided to show the feasibility and effectiveness of the proposed method. Note to Practitioners —This article was motivated by the visual localization problem of mobile robots in dynamic working environments. Visual localization is an important and fundamental capability in robotics. When a mobile robot is in an unknown environment, one important thing is to localize itself using the data collected from the perception sensors. In real scenes, the current localization algorithms often suffer from moving objects. The existence of moving objects, especially humans, brings a lot of noise into the localization process, which poses a big challenge and makes the robotic implementation unstable. In this article, a novel strategy is designed to handle this problem. We leverage both the object detection result and observation information in a Bayesian framework for dynamic region detection. Once the dynamic regions are determined, we discard them and feed the other regions in a state-of-the-art visual simultaneous localization and mapping system for further visual localization. The proposed strategy greatly improves the localization accuracy in dynamic working environments and guarantees the robustness for robotic implementation.

A new positioning method for remotely operated vehicle of the nuclear power plant

PurposeThe purpose of this study is to develop a new positioning method for remotely operated vehicle (ROV) in the nuclear power plant. The ROV of the nuclear power plant is developed to inspect the reactor cavity pools, the component pools and spent-fuel storage pools. To enhance the operational safety, the ability of localizing the ROV is indispensable.Design/methodology/approachTherefore, the positioning method is proposed based on the MEMS inertial measurement unit and mechanical scanning sonar in this paper. Firstly, the ROV model and on board sensors are introduced in detail. Then the sensor-based Kalman filter is deduced for attitude estimation. After that, the positioning method is proposed that divided into static positioning and dynamic positioning. The improved iterative closest point-Kalman filter is deduced to estimate the global position by the whole circle scanning sonar data in static, and the relative positioning method is proposed by the small scale scanning sonar data in dynamic.FindingsThe performance of the proposed method is verified by comparing with the visual positioning system. Finally, the effectiveness of the proposed method is proved by the experiment in the reactor simulation pool of the Daya Bay Nuclear Power Plant.Originality/valueThe research content of this manuscript is aimed at the specific application needs of nuclear power plants and has high theoretical significance and application value.

Calibration Technique of a Curved Zoom Compound Eye Imaging System.

A calibration method for the designed curved zoom compound eye is studied in order to achieve detection and positioning of spatial objects. The structure of the curved zoom compound eye is introduced. A calibration test platform is designed and built based on the image characteristics of the compound eye, which can be constructed in the large field view for the calibration target. The spot images are obtained through image processing. The center of the spot is calculated by Gauss fitting method. This method is highly simple and intuitive, and it can be used in a zoom surface compound eye without any complex procedures. Finally, the corresponding relationship between the spot center coordinates and the incident light vector of the corresponding sub-eye is established, and the calibration of the multi vision positioning system is completed.

Robust Visual Localization in Dynamic Environments Based on Sparse Motion Removal

Visual localization has been well studied in recent decades and applied in many fields as a fundamental capability in robotics. However, the success of the state of the arts usually builds on the assumption that the environment is static. In dynamic scenarios where moving objects are present, the performance of the existing visual localization systems degrades a lot due to the disturbance of the dynamic factors. To address this problem, we propose a novel sparse motion removal (SMR) model that detects the dynamic and static regions for an input frame based on a Bayesian framework. The similarity between the consecutive frames and the difference between the current frame and the reference frame are both considered to reduce the detection uncertainty. After the detection process is finished, the dynamic regions are eliminated while the static ones are fed into a feature-based visual simultaneous localization and mapping (SLAM) system for further visual localization. To verify the proposed method, both qualitative and quantitative experiments are performed and the experimental results have demonstrated that the proposed model can significantly improve the accuracy and robustness for visual localization in dynamic environments. Note to Practitioners —This article was motivated by the visual localization problem in dynamic environments. Visual localization is well applied in many robotic fields such as path planning and exploration as the basic capability for a mobile robot. In the GPS-denied environments, one robot needs to localize itself through perceiving the unknown environment based on a visual sensor. In real-world scenes, the existence of the moving objects will significantly degrade the localization accuracy, which makes the robot implementation unreliable. In this article, an SMR model is designed to handle this problem. Once receiving a frame, the proposed model divides it into dynamic and static regions through a Bayesian framework. The dynamic regions are eliminated, while the static ones are maintained and fed into a feature-based visual SLAM system for further visual localization. The proposed method greatly improves the localization accuracy in dynamic environments and guarantees the robustness for robotic implementation.

Обзор современных методов визуальной одометрии

In this paper we describe the tasks of Visual Odometry and Simultaneous Localization and Mapping systems along with their main applications. Next, we list some approaches used by the scientific community to create such systems in different time periods. We then proceed to explain in detail the more recent method based on bundle adjustment and show some of its variations for different applications. At last, we overview present-day research directions in the field of visual odometry and briefly present our work.

Automated visual positioning and precision placement of a workpiece using deep learning

An automated visual positioning system is proposed for precision placement of a workpiece on the fixture. The system includes a binocular eye-in-hand system on the end effector of the mobile manipulator and a ConvNet for detecting the relative position of the workpiece based on the holistic views observed by the CMOS cameras. We train the ConvNets with training images that are automatically generated from basis images taken at the target position and annotated with the 2D coordinates of the offset locations. The ConvNet’s superior place recognition capability renders a high success rate of coordinate detection subject to high illumination and viewpoint variations. Experimental evidence of workpiece placement confirms that the low-resolution (640 × 480 pixels) camera can obtain a translational precision of ±0.2 mm; the binocular system can control the rotational error within ± 0.1°. Within the 20 × 20-mm2 spatial tolerance of the mobile platform, the proposed system achieves a success rate of 100% in 200 workpiece placement tasks. The entire workpiece placement task can be completed in 60 s; the average elapsed time of precision positioning and placement is less than 20 s, with a total of 4 visual positioning steps.

Robust Loop Closure Detection Using Bayes Filters and CNN Features

This paper focuses on loop-closure detection (LCD) for a visual simultaneous localization and mapping (SLAM) system. We present a strategy that combines a Bayes filter and features from a pre-trained convolution neural network (CNN) to perform LCD. Rather than using features from only one layer, we fuse features from multiple layers based on spatial pyramid pooling. A flexible Bayes model is then formulated to integrate the sequential information and similarities that are computed by features at different scales. The introduction of a penalty factor and bidirectional propagation enables our approach to handle complex trajectories. We present extensive experiments on challenging datasets, and we compare our approach to state-of-the-art methods, to evaluate it. The results show that our approach can ensure remarkable performance under severe condition changes and handle trajectories that have different characteristics. We also show the advantages of Bayes filters over sequence matching in the experiments, and we analyze our feature fusion strategy by visualizing the activations of the CNN.

Programmable intelligent spaces for Industry 4.0: Indoor visual localization driving attocell networks

AbstractReal‐time and mission‐critical applications for Industry 4.0 demand fast and reliable communication. Therefore, knowing devices' location is essential, but GPS is of little use indoors, whereas electromagnetic impairments and interferences demand new approaches to ensure reliability. The challenges include real‐time feedback with end‐to‐end (E2E) low latency; high data density due to large number of IoT devices per area; and smaller communication cells, which increases the handover frequency and complexity. To tackle these issues, we introduce a programmable intelligent space (PIS) to deploy attocells, enable E2E programmability, and provide a precise computer vision localization system and networking programmability based on software‐defined networking. To validate our approach, experiments were conducted, controlling a mobile robot through a trajectory. We demonstrate the need for higher camera frame rate to achieve tighter precision, evaluating different trade‐offs on localization, bandwidth, and latency. Results have shown that PIS wireless attocell handover achieves seamlessly mobile communication, delivering packets within the deadline window, with similar performance to a no handover baseline.

Micro navigation system for smart parking area

Despite its limited accuracy, navigation systems depending on GPS are currently attracting considerable interest due to their increasing convenience in the field of transportation area. So far, most outdoor navigation applications are using GPS and the absence of a low-cost alternative with acceptable accuracy satisfying the indoor applications requirements has become one of the motivational topics to follow up. Therefore, different studies have been recently conducted in order to develop an accurate indoor localisation system. In this paper, we will propose a micro-navigation system for parking area with a visual positioning system (VPS) based on the surveillance camera network adaptation and the image processing theory. From the data provided by the VPS, a navigation tool will be presented besides. The proposed system has achieved an accuracy error of 0.08 M, and proved a remarkable improvement in terms of finding parking places time consuming. We believe that we have designed an innovative solution responding to traffic jams and pollution issues as well as stress and finding parking places time consuming.