Visual Localization Method Research Articles

Enhanced outdoor visual localization using Py-Net voting segmentation approach.

Camera relocalization determines the position and orientation of a camera in a 3D space. Althouh methods based on scene coordinate regression yield highly accurate results in indoor scenes, they exhibit poor performance in outdoor scenarios due to their large scale and increased complexity. A visual localization method, Py-Net, is therefore proposed herein. Py-Net is based on voting segmentation and comprises a main encoder containing Py-layer and two branch decoders. The Py-layer comprises pyramid convolution and 1 × 1 convolution kernels for feature extraction across multiple levels, with fewer parameters to enhance the model's ability to extract scene information. Coordinate attention was added at the end of the encoder for feature correction, which improved the model robustness to interference. To prevent the feature loss caused by repetitive structures and low-texture images in the scene, deep over-parameterized convolution modules were incorporated into the seg and vote decoders. Landmark segmentation and voting maps were used to establish the relation between images and landmarks in 3D space, reducing anomalies and achieving high precision with a small number of landmarks. The experimental results show that, in multiple outdoor scenes, Py-Net achieves lower distance and angle errors compared to existing methods. Additionally, compared to VS-Net, which also uses a voting segmentation structure, Py-Net reduces the number of parameters by 31.85% and decreases the model size from 236MB to 170 MB.

Visual Localization Method for Unmanned Aerial Vehicles in Urban Scenes Based on Shape and Spatial Relationship Matching of Buildings

Visual Localization Method for Unmanned Aerial Vehicles in Urban Scenes Based on Shape and Spatial Relationship Matching of Buildings

Visual localization and quantitative detection method for thermal defects in cable terminals of high-speed trains based on a temperature derivative

Abstract Cable terminal defect detection plays an important role in ensuring the safe and stable operation of high-speed trains (HST). In this paper, a numerical model of the electromagnetic thermal field of overheating defects of cable terminal shielding grids and a method of detecting internal defects of cable terminals - even-order temperature derivative (EOTD) is proposed for quantitative detection of internal defective structures of vehicle-mounted cable terminals. Firstly, a numerical model of the electromagnetic thermal field of cable terminals under the condition of leakage current is constructed, through which the temperature field distribution characteristics of different defective structures are analyzed. Then, the intuitive location of the defective region is obtained by investigating the derivative characteristics of the temperature image, and the depth and intensity of the defects are quantitatively assessed by using the main side peak (MSP) distances and the MSP values extracted from the derivative curves. Finally, the simulation and experimental results achieve the identification of defect structures and the quantitative detection of defect depth and intensity, proving the effectiveness and accuracy of the proposed method.

A new visual sensing system for motion state estimation of lateral localization of intelligent vehicles

Accurate lateral localization is the basis of intelligent vehicle decision making and control and one of the core problems of autonomous driving. In the process of vehicle driving, the vehicle body vibration, road bumps, and slope change will affect the motion parameters of the camera, which influences the localization accuracy. Therefore, we investigate a vehicle visual localization method based on the motion state estimation of vehicle camera by Gaussian Bayes sphere. This method uses Gaussian spherical crown sampling and maximum likelihood search to estimate the motion state of the vehicle camera in real time. Then, the real-time homography matrix between the sampling ground plane and the pixel plane is calculated according to the results to convert the coordinate points of the lane lines to the sampling ground plane for lateral localization of vehicles. Last, the localization results are restored to the real scale by the lane line width. The road experimental results show that the average error of the proposed method for lateral localization of vehicles is 5.7 cm under different road conditions on campus roads, and that under different road conditions on urban roads is 5.9 cm. At the same time, the localization accuracy of the proposed method is tested on the slope and under different weather conditions, and good results are obtained.

RCEAU-Net: Cascade Multi-Scale Convolution and Attention-Mechanism-Based Network for Laser Beam Target Image Segmentation with Complex Background in Coal Mine.

Accurate and reliable pose estimation of boom-type roadheaders is the key to the forming quality of the tunneling face in coal mines, which is of great importance to improve tunneling efficiency and ensure the safety of coal mine production. The multi-laser-beam target-based visual localization method is an effective way to realize accurate and reliable pose estimation of a roadheader body. However, the complex background interference in coal mines brings great challenges to the stable and accurate segmentation and extraction of laser beam features, which has become the main problem faced by the long-distance visual positioning method of underground equipment. In this paper, a semantic segmentation network for underground laser beams in coal mines, RCEAU-Net, is proposed based on U-Net. The network introduces residual connections in the convolution of the encoder and decoder parts, which effectively fuses the underlying feature information and improves the gradient circulation performance of the network. At the same time, by introducing cascade multi-scale convolution in the skipping connection section, which compensates for the lack of contextual semantic information in U-Net and improves the segmentation effect of the network model on tiny laser beams at long distance. Finally, the introduction of an efficient multi-scale attention module with cross-spatial learning in the encoder enhances the feature extraction capability of the network. Furthermore, the laser beam target dataset (LBTD) is constructed based on laser beam target images collected from several coal mines, and the proposed RCEAU-Net model is then tested and verified. The experimental results show that, compared with the original U-Net, RCEAU-Net can ensure the real-time performance of laser beam segmentation while increasing the Accuracy by 0.19%, Precision by 2.53%, Recall by 22.01%, and Intersection and Union Ratio by 8.48%, which can meet the requirements of multi-laser-beam feature segmentation and extraction under complex backgrounds in coal mines, so as to further ensure the accuracy and stability of long-distance visual positioning for boom-type roadheaders and ensure the safe production in the working face.

An omnidirectional spatial monocular visual localization and tracking method for indoor unmanned aerial vehicles based on the two-axis rotary table

Aiming at the complexity and poor adaptability of the calibration process in the traditional unmanned aerial vehicles (UAV) indoor visual positioning, this paper proposes an omnidirectional spatial tracking and localization method for indoor UAV based on the two-axis rotary table. Firstly, the position of the UAV fuselage feature points in the camera coordinate system of the turntable camera is computed by the Pespective-n-Point algorithm utilizing known position information of a plurality of feature points with pixel coordinate information in the corresponding image. Pixel coordinate information is extracted by obtaining UAV body-specific feature points from rotary table camera shots. Then, UAV localization in omnidirectional space can be obtained by using the calibrated rotary axis parameters of the rotary table and the rotation angle of the rotary table and substituting them into Rodriguez’s formula to unify the UAV position information acquired by the rotary table camera at different positions into a unified coordinate system. Finally, the angle at which the rotary table should rotate is calculated from the obtained UAV pose and the spatial position of the camera optical center and the rotary axis of the rotary table. The calculated angle is fed back to the turntable as feedback information. The rotary table receiving the feedback information is rotated to a position where the UAV is located at the center of the camera image. Thereby the tracking and localization of the UAV is realized. The experimental results show that the spatial range of localization is greatly expanded with the localization accuracy reaching the level of binocular visual localization. The omnidirectional spatial tracking and localization of indoor UAV can be conveniently realized by this method.

Autonomous Dynamic Hitch-Hiking Control of a Bionic Robotic Remora

Remora suckerfish and its hitch-hiking behavior bring enormous inspiration into the engineering field. In this paper, a bioinspired hitch-hiking behavior is accomplished automatically by a robotic remora, which creates possibilities for prolonging its endurance and multi-robot cooperation. First, the definition of the hitch-hiking task and the mechatronic design of robotic remora are introduced. Then, aiming at the hitch-hiking task, the LED-marker-based underwater visual localization method and planar state synchronization controller are developed. The localization method includes a complete framework from LED detection to marker pose optimization, which considers the refraction correction to improve the localization precision in water. The synchronization controller is decomposed into the lateral and longitudinal sub-controllers to overcome the challenges caused by underactuated dynamic. Besides, a finite state machine is designed to model the state and action transition during the hitch-hiking task. Extensive experimental results demonstrate the effectiveness of the proposed method. The autonomous hitch-hiking task toward a moving host is successfully implemented by a robotic fish for the first time. Such results may offer valuable insights into the future autonomous operation of underwater robots.

A Review of Using Visual Odometery Methods in Autonomous UAV Navigation in GPS-Denied Environment

Abstract This review paper centers on strategies employed for location determination in regions lacking GPS signals. It primarily explores a range of vision-based methods tailored for this purpose, categorizing them accordingly. The article delves into the utilization of optical flow for feature extraction-based Visual Odometry (VO) and delves into advanced optical flow estimation methods that hinge on deep learning techniques. It compares the efficacy and practical applications of frequently utilized visual localization methods while also checking the efficiency of previous researches by reapplying the algorithms to new data and comparing the results.

SRVIO: Super Robust Visual Inertial Odometry for Dynamic Environments and Challenging Loop-Closure Conditions

There has been extensive research on visual localization and odometry for autonomous robots and virtual reality during the past decades. Traditionally, this problem has been solved with the help of expensive sensors, such as light detection and ranging (LiDAR). Nowadays, the focus of the leading research in this field is on robust localization using more economic sensors, such as cameras and inertial measurement units. Consequently, geometric visual localization methods have become more accurate over time. However, these methods still suffer from significant loss and divergence in challenging environments, such as a room full of moving people. Scientists started using deep neural networks (DNNs) to mitigate this problem. The main idea behind using DNNs is to better understand challenging aspects of the data and overcome complex conditions such as the movement of a dynamic object in front of the camera that covers the full view of the camera, extreme lighting conditions, and the high speed of the camera. Prior end-to-end DNN methods did overcome some of these challenges. However, no general and robust framework is available to overcome all challenges together. In this article, we have combined geometric and DNN-based methods to have the generality and speed of geometric SLAM frameworks and overcome most of these challenging conditions with the help of DNNs and deliver the most robust framework so far. To do so, we have designed a framework based on VINS-Mono and shown that it can achieve state-of-the-art results on TUM-Dynamic, TUM-VI, ADVIO, and EuRoC datasets compared to geometric and end-to-end DNN-based simultaneous localization and mappings. Our proposed framework can also achieve outstanding results on extreme simulated cases resembling the aforementioned challenges.

Light Beacon-Aided AUV Electromagnetic Localization for Landing on a Planar Docking Station

Autonomous underwater vehicles (AUVs) play an increasingly important role in underwater missions. However, traditional visual localization methods struggle to meet the accuracy requirements for AUV terminal docking in strong-illumination or turbid water conditions. This work proposes a universal navigation system using electromagnetic (EM) and light beacons for nontorpedo AUVs to be recovered using a planar docking station. The major challenge in designing this system is to deal with the incomplete sensing of the light and EM beacons. To solve this problem, a two-step fusion method of EM localization information is given first. The kinematic model of the AUV and visual information are also utilized to improve the robustness of the navigation system. Based on the localization result, a comprehensive docking strategy is proposed to address the time-varying sensing conditions. Localization and docking experiments were conducted in a pool. The test results show that the maximum EM localization error is less than 20 cm in both static and dynamic tests. The AUV completed all six docking attempts.

Eiffel Tower: A deep-sea underwater dataset for long-term visual localization

Visual localization plays an important role in the positioning and navigation of robotics systems within previously visited environments. When visits occur over long periods of time, changes in the environment related to seasons or day-night cycles present a major challenge. Under water, the sources of variability are due to other factors such as water conditions or growth of marine organisms. Yet, it remains a major obstacle and a much less studied one, partly due to the lack of data. This paper presents a new deep-sea dataset to benchmark underwater long-term visual localization. The dataset is composed of images from four visits to the same hydrothermal vent edifice over the course of 5 years. Camera poses and a common geometry of the scene were estimated using navigation data and Structure-from-Motion. This serves as a reference when evaluating visual localization techniques. An analysis of the data provides insights about the major changes observed throughout the years. Furthermore, several well-established visual localization methods are evaluated on the dataset, showing there is still room for improvement in underwater long-term visual localization. The data is made publicly available at seanoe.org/data/00810/92226/.

CLoc: Confident Initial Estimation of Long-Term Visual Localization Using a Few Sequential Images in Large-Scale Spaces

Estimation of six degree-of-freedom camera pose with images is a fundamental task for various applications, such as autonomous driving and augmented reality. The challenges in visual localization stem primarily from the scale of spaces and the changes that can happen during the long-term intervals between when the database and query images are captured. Although most sequence-based visual localization (SVL) methods rely on the outputs of one-shot visual localization (OVL) for the initial estimation of the camera pose, OVL may yield incorrect poses because of these limitations. In this study, we aim to achieve a confident initial estimation using a few sequential images. Accordingly, we propose a pipeline called CLoc that narrows down possible solution spaces with hypotheses generated by associating pose candidates of keyframes and their relative poses to determine the most probable pose of the latest keyframe. Moreover, we create publicly available visual localization datasets including three building-scale indoor spaces. The datasets provide long-term challenges and sequential query images simultaneously, enabling the evaluation of the OVL and SVL methods in large-scale spaces. We evaluate CLoc using our datasets and a city-scale outdoor dataset and compare it with two state-of-the-art OVL methods and three SVL methods. By properly leveraging sequential data, CLoc can outperform other methods and achieves 100% accuracy in the [2m, 10°] evaluation thresholds for all datasets.

Photo-realistic 3D model based accurate visual positioning system for large-scale indoor spaces

This study presents a novel and reliable visual positioning system (VPS), KR-Net, for kidnap recovery tasks, which predicts an accurate position when a robot is first initiated. KR-Net is based on a hierarchical visual localization method and demonstrates significant robustness in large-scale indoor environments. The proposed VPS utilizes a photo-realistic 3D model to generate a dense database of any camera pose and incorporates a novel global descriptor for indoor spaces, i-GeM, that outperforms existing methods in terms of robustness. Additionally, the proposed combinatorial pooling approach overcomes the limitations of previous single image-based predictions in large-scale indoor environments, allowing for accurate discrimination between similar locations. Extensive evaluations were performed on six large-scale indoor datasets to demonstrate the contributions of each component. To the best of our knowledge, KR-Net is the first system to estimate wake-up positions with a near 100% confidence level within a 1.0m distance error threshold.

Indoor Positioning on Smartphones Using Built-In Sensors and Visual Images

With the rapid development of mobile Internet technology, localization using visual image information has become a hot problem in the field of indoor localization research, which is not affected by signal multipath and fading and can achieve high accuracy localization in indoor areas with complex electromagnetic environments. However, in practical applications, position estimation using visual images is easily influenced by the user's photo pose. In this paper, we propose a multiple-sensor-assisted visual localization method in which the method constructs a machine learning classifier using multiple smart sensors for pedestrian pose estimation, which improves the retrieval efficiency and localization accuracy. The method mainly combines the advantages of visual image location estimation and pedestrian pose estimation based on multiple smart sensors and considers the effect of pedestrian photographing poses on location estimation. The built-in sensors of smartphones are used as the source of pedestrian pose estimation data, which constitutes a feasible location estimation method based on visual information. Experimental results show that the method proposed in this paper has good localization accuracy and robustness. In addition, the experimental scene in this paper is a common indoor scene and the experimental device is a common smartphone. Therefore, we believe that the proposed method in this paper has the potential to be widely used in future indoor navigation applications in complex scenarios (e.g., mall navigation).

Learned visual localization with camera pose refinement and verification based on differentiable renderer

This manuscript presents a new CNN-based visual localization method that seeks a camera location of an input RGB image with respect to a pre-collected RGB-D images database. To determine an accurate camera pose, we employ a coarse-to-fine localization manner that firstly finds coarse location candidates via image retrieval, then refines them using local 3D structure represented by each retrieved RGB-D image. We use a CNN feature extractor and a relative pose estimator for coarse prediction that do not sufficiently require a scene-specific training. Furthermore, we propose a new pose refinement-verification module that simultaneously evaluates and refines camera poses using differentiable renderer. Experimental results on public datasets show that our proposed pipeline achieves accurate localization on both trained and unknown scenes.

Learning Visual Semantic Map-Matching for Loosely Multi-Sensor Fusion Localization of Autonomous Vehicles

Precise localization is essential but also challenging for autonomous vehicles. In this article, A novel visual localization method is proposed. Specifically, a semantic local map describing the local environment is built with images sequence and wheel-inertial ego-motion results. Then the local semantic map is matched with the online map database for camera position estimation. The key novelty of the method lies in using a supervised neural network to simplify the map-matching problem, which avoids the complex data association and optimization processes. The network encodes the maps, infers the feature similarity, and predicts the camera position. The visual localization results are then loosely integrated with other onboard sensors by an invariant Kalman filter. We evaluate the map-matching module and the overall fusion system on scenario tests. The experiment results validate the effectiveness of the learning-based map-matching method. And the accuracy of the overall system is satisfactory, with mean absolute errors of 0.039 m and 0.167 m in the lateral and longitudinal directions, respectively.

Samolokalizacja bezzałogowego statku powietrznego uwzględniająca zmienną orientację kamery

This publication proposes a visual localization method using images from a simulated camera and a georeferenced map. The UAV model and flight simulation were made in the MATLAB Simulink package, which sent UAV orientation data to the described program. The visualization of the camera image was performed in real time using the FlightGear software, the image of which was also captured by the NW program. This method is performed by two processes in two modules: Global Positioning Component and Motion Positioning Component. The first one compares the image from the simulated camera with the orthophotomap. The second determines the position based on the assessment of the displacement of characteristic points in the image in relation to the last known location. The result of the operation of both modules is illustrated in the graphic window of the NW application, which allows for a visual comparison of the obtained results. With the global method of location, additional camera orientation correction is required to determine the position in 2D space. For this purpose, data on the current camera orientation expressed in quaternions were used. This allowed for the introduction of a position correction, which significantly improved the accuracy of the result obtained in the GPC module despite significant UAV tilts during the simulated flight.

High precision visual localization method of UAV based on feature matching.

In this paper, the precision hovering problem of UAV operation is studied. Aiming at the diversity and complexity of the UAV operating environment, a high-precision visual positioning and orientation method based on image feature matching was proposed. The image feature matching based on the improved AKAZE algorithm is realized, and the optimal matching point pair screening method based on the fusion of Hamming distance and matching line angle is innovatively proposed, which greatly improves the robustness of the algorithm without affecting the performance of the algorithm. The real-time image is matched with the benchmark image for image feature matching. By reducing the deviation of image feature, the pose state correction of UAV hovering is achieved, and the precision hovering of the UAV is realized. Both simulation and real UAV tests verify the effectiveness of the proposed UAV high-precision visual positioning and orientation method.

Hierarchical Clustering-Based Image Retrieval for Indoor Visual Localization

Visual localization is employed for indoor navigation and embedded in various applications, such as augmented reality and mixed reality. Image retrieval and geometrical measurement are the primary steps in visual localization, and the key to improving localization efficiency is to reduce the time consumption of the image retrieval. Therefore, a hierarchical clustering-based image-retrieval method is proposed to hierarchically organize an off-line image database, resulting in control of the time consumption of image retrieval within a reasonable range. The image database is hierarchically organized by two stages: scene-level clustering and sub-scene-level clustering. In scene-level clustering, an improved cumulative sum algorithm is proposed to detect change points and then group images by global features. On the basis of scene-level clustering, a feature tracking-based method is introduced to further group images into sub-scene-level clusters. An image retrieval algorithm with a backtracking mechanism is designed and applied for visual localization. In addition, a weighted KNN-based visual localization method is presented, and the estimated query position is solved by the Armijo–Goldstein algorithm. Experimental results indicate that the running time of image retrieval does not linearly increase with the size of image databases, which is beneficial to improving localization efficiency.

Danish Airs and Grounds: A Dataset for Aerial-to-Street-Level Place Recognition and Localization

Place recognition and visual localization are particularly challenging in wide baseline configurations. In this letter, we contribute with the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Danish Airs and Grounds</i> (DAG) dataset, a large collection of street-level and aerial images targeting such cases. Its main challenge lies in the extreme viewing-angle difference between query and reference images with consequent changes in illumination and perspective. The dataset is larger and more diverse than current publicly available data, including more than 50 km of roads in urban, suburban and rural areas. All images are associated with accurate 6-DoF metadata that allows the benchmarking of visual localization methods. Additionally, we validate our data by presenting the results of a simple map-to-image re-localization baseline. that first estimates a dense 3D reconstruction from the aerial images and then matches query street-level images to street-level renderings of the 3D model. The dataset can be downloaded at:.