Visual Localization Research Articles

A visual simultaneous localization and mapping approach based on scene segmentation and incremental optimization

Existing visual simultaneous localization and mapping (V-SLAM) algorithms are usually sensitive to the situation with sparse landmarks in the environment and large view transformation of camera motion, and they are liable to generate large pose errors that lead to track failures due to the decrease of the matching rate of feature points. Aiming at the above problems, this article proposes an improved V-SLAM method based on scene segmentation and incremental optimization strategy. In the front end, this article proposes a scene segmentation algorithm considering camera motion direction and angle. By segmenting the trajectory and adding camera motion direction to the tracking thread, an effective prediction model of camera motion in the scene with sparse landmarks and large view transformation is realized. In the back end, this article proposes an incremental optimization method combining segmentation information and an optimization method for tracking prediction model. By incrementally adding the state parameters and reusing the computed results, high-precision results of the camera trajectory and feature points are obtained with satisfactory computing speed. The performance of our algorithm is evaluated by two well-known datasets: TUM RGB-D and NYUDv2 RGB-D. The experimental results demonstrate that our method improves the computational efficiency by 10.2% compared with state-of-the-art V-SLAMs on the desktop platform and by 22.4% on the embedded platform, respectively. Meanwhile, the robustness of our method is better than that of ORB-SLAM2 on the TUM RGB-D dataset.

Visual Simultaneous Localization and Mapping with Applications to Monitoring of Underground Transportation Infrastructure

This paper presents the results of the development, design, and implementation of a visual simultaneous localization and mapping (SLAM) system for autonomous real-time localization with application to underground transportation infrastructure (UTI) such as tunnels. Localization is achieved in the absence of any global positioning system (GPS) or auxiliary system. The indoor localization system is a necessary element of a fully autonomous platform for the detection of cracks and other anomalies on the interior surfaces of tunnels and other UTI. It can be used for tagging of high-resolution sensor data obtained with low-cost prototype data acquisition platforms previously developed. Visual based SLAM has been used as the core element in an architecture employing a commercial off-the-shelf (COTS) ZED stereo camera from Stereolabs. To achieve real-time operation, an NVIDIA Jetson TX2 massively parallel graphics processing unit (GPU) was used as the core computational engine employing two different software libraries. We achieved localization at 5 frames per second (FPS) using ORBSLAM2 open-source software library, and the much lighter, but proprietary, ZED SDK was able to deliver a performance at nearly 60 FPS. To assess the accuracy of the relative localization system, we conducted several tests at 30 FPS and reported on the resulting error variances that were found to be consistently very small. Finally, we conducted several tests in a tunnel in the Los Angeles county area and confirmed the applicability of the method for monitoring UTI.

Image predictors of visual localization in natural scenes

Image predictors of visual localization in natural scenes

Long-Term Visual Localization Revisited.

Visual localization enables autonomous vehicles to navigate in their surroundings and augmented reality applications to link virtual to real worlds. Practical visual localization approaches need to be robust to a wide variety of viewing conditions, including day-night changes, as well as weather and seasonal variations, while providing highly accurate six degree-of-freedom (6DOF) camera pose estimates. In this paper, we extend three publicly available datasets containing images captured under a wide variety of viewing conditions, but lacking camera pose information, with ground truth pose information, making evaluation of the impact of various factors on 6DOF camera pose estimation accuracy possible. We also discuss the performance of state-of-the-art localization approaches on these datasets. Additionally, we release around half of the poses for all conditions, and keep the remaining half private as a test set, in the hopes that this will stimulate research on long-term visual localization, learned local image features, and related research areas. Our datasets are available at visuallocalization.net, where we are also hosting a benchmarking server for automatic evaluation of results on the test set. The presented state-of-the-art results are to a large degree based on submissions to our server.

Robotic intelligence in the organization of pre-medical diagnostics of the risk of tumor diseases

Based on the creation of a robotic complex, the analysis of clinical observations of 1053 patients was carried out, by means of a survey with illustrations, in which 106 (30.4 4.6%) later revealed a malignant process. In 53 patients with microscopy of pigmented nevi, other tumors and precancerous changes in visual localization, oral cavity, in which 13 (72.2% 1.1) had melanoma and other tumors. In 75 patients, based on the survey, a urease breath test was required, on the basis of which, if it was present for more than 10 units, FGS was performed, while in 12 cases, peptic ulcer in the acute stage or scarring, chronic gastritis with metaplasia zones, and in 4 (11.7 0.9), stomach cancer was detected. The survey was conducted with the participation of clinical residents and students on Open days and at the request of patients after 3 advertising exhibitions

Study of a robotic system to detect water leakage and fuel debris-System proposal and feasibility study of visual odometry providing intuitive bird\u2019s eye view-

To obtain the necessary information on fuel debris and water leakages during the decommissioning task of the Fukushima Daiichi Nuclear Power Plant, an ultrasonic-based method was proposed for future internal investigation of the primary containment vessel (PCV). In this article, we describe the rotatable winch mechanism and visual localization method, which were used to aid the investigation. We used the rotatable winch mechanism to adjust the height and orientation of the ultrasonic sensor and localized the robot with cameras to localize the sensor, to provide assisting information for data combination. We studied the feasibility of the conventional visual odometry method for application to the situation and performed localizing accuracy evaluation experiments with a mobile robotic platform prototype. The results showed that the visual odometry method could generate intuitive bird’s-eye-view maps, and provided an average error rate of 35 mm/1500 mm, which met the required maximum error rate of 100 mm/1500 mm for the grating movement. Experiments were also conducted with adjustable parameter ranges that could provide the required accuracy.

A review on absolute visual localization for UAV

Research on unmanned aerial vehicles is growing as they are becoming less expensive and more available than before. The applications span a large number of areas and include border security, search and rescue, wildlife surveying, firefighting, precision agriculture, structure inspection, surveying and mapping, aerial photography, and recreative applications. These applications can require autonomous behavior which is only possible with a precise and robust self-localization. Until recently, the favored approach to localization was based on inertial sensors and global navigation satellite systems. However, global navigation satellite systems have multiple shortcomings related to long-distance radio communications (e.g. non-line-of-sight reception, multipath, spoofing). This motivated the development of new approaches to supplement or supplant satellite navigation. Absolute visual localization is one of the two main approaches to vision-based localization. The goal is to locate the current view of the UAV in a reference satellite map or georeferenced imagery from previous flights. Various approaches were proposed in this area and this paper review most of the literature in this field since 2015. The problematic at hand is analyzed and defined. Existing approaches are reviewed in 4 categories: template matching, feature points matching, deep learning and visual odometry.

C${}^{*}$: Cross-Modal Simultaneous Tracking and Rendering for 6-DoF Monocular Camera Localization Beyond Modalities

We present a monocular camera localization technique for a three-dimensional prior map. Visual localization has been attracting considerable attention as a lightweight and widely available localization technique for any mobilities; however, it still suffers from appearance changes and a high computational cost. With a view to achieving robust and real-time visual localization, we first reduce the localization problem to alternate local tracking and occasional keyframe rendering by following a simultaneous tracking and rendering algorithm. At the same time, by using an information-theoretic metric denoted normalized information distance in the local tracking, we developed a 6-DoF localization method robust to intensity variations between modalities and varying sensor properties. We quantitatively evaluated the accuracy and robustness of our method using both synthetic and real datasets and achieved reliable and practical localization even in the case of extreme appearance changes.

GMMLoc: Structure Consistent Visual Localization With Gaussian Mixture Models

Incorporating prior structure information into the visual state estimation could generally improve the localization performance. In this letter, we aim to address the paradox between accuracy and efficiency in coupling visual factors with structure constraints. To this end, we present a cross-modality method that tracks a camera in a prior map modelled by the Gaussian Mixture Model (GMM). With the pose estimated by the front-end initially, the local visual observations and map components are associated efficiently, and the visual structure from the triangulation is refined simultaneously. By introducing the hybrid structure factors into the joint optimization, the camera poses are bundle-adjusted with the local visual structure. By evaluating our complete system, namely GMMLoc, on the public dataset, we show how our system can provide a centimeter-level localization accuracy with only trivial computational overhead. In addition, the comparative studies with the state-of-the-art vision-dominant state estimators demonstrate the competitive performance of our method.

ERF-IMCS: An Efficient and Robust Framework with Image-Based Monte Carlo Scheme for Indoor Topological Navigation

Conventional approaches to global localization and navigation mainly rely on metric maps to provide precise geometric coordinates, which may cause the problem of large-scale structural ambiguity and lack semantic information of the environment. This paper presents a scalable vision-based topological mapping and navigation method for a mobile robot to work robustly and flexibly in large-scale environment. In the vision-based topological navigation, an image-based Monte Carlo localization method is presented to realize global topological localization based on image retrieval, in which fine-tuned local region features from an object detection convolutional neural network (CNN) are adopted to perform image matching. The combination of image retrieval and Monte Carlo provide the robot with the ability to effectively avoid perceptual aliasing. Additionally, we propose an effective visual localization method, simultaneously employing the global and local CNN features of images to construct discriminative representation for environment, which makes the navigation system more robust to the interference of occlusion, translation, and illumination. Extensive experimental results demonstrate that ERF-IMCS exhibits great performance in the robustness and efficiency of navigation.

Creating Hardware Component Knowledge Bases with Training Data Generation and Multi-task Learning

Hardware component databases are vital resources in designing embedded systems. Since creating these databases requires hundreds of thousands of hours of manual data entry, they are proprietary, limited in the data they provide, and have random data entry errors. We present a machine learning based approach for creating hardware component databases directly from datasheets. Extracting data directly from datasheets is challenging because: (1) the data is relational in nature and relies on non-local context, (2) the documents are filled with technical jargon, and (3) the datasheets are PDFs, a format that decouples visual locality from locality in the document. Addressing this complexity has traditionally relied on human input, making it costly to scale. Our approach uses a rich data model, weak supervision, data augmentation, and multi-task learning to create these knowledge bases in a matter of days. We evaluate the approach on datasheets of three types of components and achieve an average quality of 77 F1 points—quality comparable to existing human-curated knowledge bases. We perform application studies that demonstrate the extraction of multiple data modalities including numerical properties and images. We show how different sources of supervision such as heuristics and human labels have distinct advantages that can be utilized together to improve knowledge base quality. Finally, we present a case study to show how this approach changes the way practitioners create hardware component knowledge bases.

A Recurrent Deep Network for Estimating the Pose of Real Indoor Images from Synthetic Image Sequences.

Recently, deep convolutional neural networks (CNN) have become popular for indoor visual localisation, where the networks learn to regress the camera pose from images directly. However, these approaches perform a 3D image-based reconstruction of the indoor spaces beforehand to determine camera poses, which is a challenge for large indoor spaces. Synthetic images derived from 3D indoor models have been used to eliminate the requirement of 3D reconstruction. A limitation of the approach is the low accuracy that occurs as a result of estimating the pose of each image frame independently. In this article, a visual localisation approach is proposed that exploits the spatio-temporal information from synthetic image sequences to improve localisation accuracy. A deep Bayesian recurrent CNN is fine-tuned using synthetic image sequences obtained from a building information model (BIM) to regress the pose of real image sequences. The results of the experiments indicate that the proposed approach estimates a smoother trajectory with smaller inter-frame error as compared to existing methods. The achievable accuracy with the proposed approach is 1.6 m, which is an improvement of approximately thirty per cent compared to the existing approaches. A Keras implementation can be found in our Github repository.

On the relative (un)importance of foveal vision during letter search in naturalistic scenes

The importance of high-acuity foveal vision to visual search can be assessed by denying foveal vision using the gaze-contingent Moving Mask technique. Foveal vision was necessary to attain normal performance when searching for a target letter in alphanumeric displays, Perception & Psychophysics, 62 (2000) 576–585. In contrast, foveal vision was not necessary to correctly locate and identify medium-sized target objects in natural scenes, Journal of Experimental Psychology: Human Perception and Performance, 40 (2014) 342–360. To explore these task differences, we used grayscale pictures of real-world scenes which included a target letter (Experiment 1: T, Experiment 2: T or L). To reduce between-scene variability with regard to target salience, we developed the Target Embedding Algorithm (T.E.A.) to place the letter in a location for which there was a median change in local contrast when inserting the letter into the scene. The presence or absence of foveal vision was crossed with four target sizes. In both experiments, search performance decreased for smaller targets, and was impaired when searching the scene without foveal vision. For correct trials, the process of target localization remained completely unimpaired by the foveal scotoma, but it took longer to accept the target. We reasoned that the size of the target may affect the importance of foveal vision to the task, but the present data remain ambiguous. In summary, the data highlight the importance of extrafoveal vision for target localization, and the importance of foveal vision for target verification during letter-in-scene search.

Visual localization under appearance change: filtering approaches

A major focus of current research on place recognition is visual localization for autonomous driving. In this scenario, as cameras will be operating continuously, it is realistic to expect videos as an input to visual localization algorithms, as opposed to the single-image querying approach used in other visual localization works. In this paper, we show that exploiting temporal continuity in the testing sequence significantly improves visual localization—qualitatively and quantitatively. Although intuitive, this idea has not been fully explored in recent works. To this end, we propose two filtering approaches to exploit the temporal smoothness of image sequences: (i) filtering on discrete domain with hidden Markov model, and (ii) filtering on continuous domain with Monte Carlo-based visual localization. Our approaches rely on local features with an encoding technique to represent an image as a single vector. The experimental results on synthetic and real datasets show that our proposed methods achieve better results than state of the art (i.e., deep learning-based pose regression approaches) for the task on visual localization under significant appearance change. Our synthetic dataset and source code are made publicly available ( https://sites.google.com/view/g2d-software/home ; https://github.com/dadung/Visual-Localization-Filtering ).

Robotic Manipulation for Specialty Crop Harvesting: A Review of Manipulator and End-Effector Technologies

This paper reviews recent developments in manipulator and end-effector technologies for the robotic harvesting of specialty crops that include fruits, vegetables, nursery crops, and nuts among others. Quantitative performance measures and general review criteria, including methods of crop detachment and end-effector sensing, are used to evaluate technologies and determine state-of-the-art in the field. Challenges affecting commercial implementation, limitations of current mechanical designs, and best practices are then presented. Results of the review show that, in general, robotic manipulation during harvesting has been limited by lack of system optimization and insufficient robustness to position error accumulated during visual localization. Inconsistent reporting practices have also hampered research and development across the field. At the conclusion of the review, some avenues of future research that could potentially lead to improvements in system performance are proposed. Some of the proposed recommendations include specific horticultural practices, the development of modular, multi-functional designs, and the incorporation of robust grasping techniques used in many of today’s robotic hands.

Improved intelligent vehicle self-localization with integration of sparse visual map and high-speed pavement visual odometry

Localization is a fundamental requirement for intelligent vehicles. Conventional localization methods usually suffer from various limitations, such as low accuracy and blocked areas for Global Positioning System, high cost for inertial navigation system or light detection and ranging, and low robustness for visual simultaneous localization and mapping or visual odometry. To overcome these problems, we propose a novel localization method integrated with a sparse visual map and a high-speed pavement visual odometry. We use a lateral-view camera to sense the sparse visual map node for accurate map-based localization. We use a down-view high-speed camera for odometry computation between two sparse visual map nodes. With a high-speed camera, it is possible to extract and track pavement features with stable resolution imaging even in high-speed movement. We also develop a data-driven motion model for the Kalman filter to fuse the localization results from the sparse map and the high-speed pavement visual odometry to enhance vehicle localization. The proposed method was tested in two different scenarios in different pavement conditions. The experimental results demonstrate that the proposed method can improve vehicle localization with low cost and high feasibility.

Improving Image Description with Auxiliary Modality for Visual Localization in Challenging Conditions

Image indexing for lifelong localization is a key component for a large panel of applications, including robot navigation, autonomous driving or cultural heritage valorization. The principal difficulty in long-term localization arises from the dynamic changes that affect outdoor environments. In this work, we propose a new approach for outdoor large scale image-based localization that can deal with challenging scenarios like cross-season, cross-weather and day/night localization. The key component of our method is a new learned global image descriptor, that can effectively benefit from scene geometry information during training. At test time, our system is capable of inferring the depth map related to the query image and use it to increase localization accuracy. We show through extensive evaluation that our method can improve localization performances, especially in challenging scenarios when the visual appearance of the scene has changed. Our method is able to leverage both visual and geometric clues from monocular images to create discriminative descriptors for cross-season localization and effective matching of images acquired at different time periods. Our method can also use weakly annotated data to localize night images across a reference dataset of daytime images. Finally we extended our method to reflectance modality and we compare multi-modal descriptors respectively based on geometry, material reflectance and a combination of both.

A 3D MAP AIDED DEEP LEARNING BASED INDOOR LOCALIZATION SYSTEM FOR SMART DEVICES

Abstract. Indoor positioning technologies represent a fast developing field of research due to the rapidly increasing need for indoor location-based services (ILBS); in particular, for applications using personal smart devices. Recently, progress in indoor mapping, including 3D modeling and semantic labeling started to offer benefits to indoor positioning algorithms; mainly, in terms of accuracy. This work presents a method for efficient and robust indoor localization, allowing to support applications in large-scale environments. To achieve high performance, the proposed concept integrates two main indoor localization techniques: Wi-Fi fingerprinting and deep learning-based visual localization using 3D map. The robustness and efficiency of technique is demonstrated with real-world experiences.

Associated Reality: A cognitive Human–Machine Layer for autonomous driving

Advanced Driver Assistance Systems (ADAS) and Automated and Autonomous Vehicles (AV) are cooperative systems and processes that use: artificial intelligence, cognitive methods, cloud technologies, cooperative vehicle-to-everything-communications (V2X), software–hardwareplatforms, sensor platforms and incipient intelligent transport infrastructures, to get self-driving systems and smart connected mobility services. This paper, to support automated driving systems (assisted, semi-autonomous and fully autonomous vehicles), introduces a cognitive layer called Associated Reality to enhance the involved information, knowledge and communication processes. The architecture defined includes an augmented Local Dynamic Map, with complementary layers, and an augmented Graph Database, with complementary semantic–cognitive relations, for the considered purpose, in cooperative human–machine and machine–machine systems. Virtual augmented landmarks are defined to improve the connectivity and intelligence of the involved spatial-information systems. Different structure landmarks and sequence landmarks (which includes regular, repetitive and periodic landmarks) are defined, categorized and used in diverse visual localization and mapping scenarios, for autonomous driving. In this paper, it is also shown, as a proof-of-concept for vehicle localization and mapping in road tunnels, the visual detection of different sequences of periodic luminaires, using YOLO v3 for the corresponding LED lights detection, or a specific alternative procedure developed with very low computational cost.

PRECISION OF VISUAL LOCALIZATION USING DYNAMIC GROUND CONTROL POINTS

Abstract. Localization is one of the first steps in navigation. Especially due to the rapid development in automated driving, a precise and reliable localization becomes essential. In this paper, we report an investigation of the usage of dynamic ground control points (GCP) in visual localization in an automotive environment. Instead of having fixed positions, dynamic GCPs move together with the camera. As a measure of quality, we employ the precision of the bundle adjustment results. In our experiments, we simulate and investigate different realistic traffic scenarios. After investigating the role of tie points, we compare an approach using dynamic GCPs to an approach with static GCPs to answer the question how a comparable precision can be reached for visual localization. We show, that in our scenario, where two dynamic GCPs move together with a camera, similar results are indeed obtained to using a number of static GCPs distributed over the whole trajectory. In another experiment, we take a closer look at sliding window bundle adjustments. Sliding windows make it possible to work with an arbitrarily large number of images and to still obtain near real-time results. We investigate this approach in combination with dynamic GCPs and vary the no. of images per window.