Visual Localization Research Articles

Microsaccadic Eye Movements but not Pupillary Dilation Response Characterizes the Crossmodal Freezing Effect.

In typical spatial orienting tasks, the perception of crossmodal (e.g., audiovisual) stimuli evokes greater pupil dilation and microsaccade inhibition than unisensory stimuli (e.g., visual). The characteristic pupil dilation and microsaccade inhibition has been observed in response to “salient” events/stimuli. Although the “saliency” account is appealing in the spatial domain, whether this occurs in the temporal context remains largely unknown. Here, in a brief temporal scale (within 1 s) and with the working mechanism of involuntary temporal attention, we investigated how eye metric characteristics reflect the temporal dynamics of perceptual organization, with and without multisensory integration. We adopted the crossmodal freezing paradigm using the classical Ternus apparent motion. Results showed that synchronous beeps biased the perceptual report for group motion and triggered the prolonged sound-induced oculomotor inhibition (OMI), whereas the sound-induced OMI was not obvious in a crossmodal task-free scenario (visual localization without audiovisual integration). A general pupil dilation response was observed in the presence of sounds in both visual Ternus motion categorization and visual localization tasks. This study provides the first empirical account of crossmodal integration by capturing microsaccades within a brief temporal scale; OMI but not pupillary dilation response characterizes task-specific audiovisual integration (shown by the crossmodal freezing effect).

LONG-TERM VISUAL LOCALIZATION IN LARGE SCALE URBAN ENVIRONMENTS EXPLOITING STREET LEVEL IMAGERY

Abstract. In this paper, we present our approach for robust long-term visual localization in large scale urban environments exploiting street level imagery. Our approach consists of a 2D-image based localization using image retrieval (NetVLAD) to select reference images. This is followed by a 3D-structure based localization with a robust image matcher (DenseSfM) for accurate pose estimation. This visual localization approach is evaluated by means of the ‘Sun’ subset of the RobotCar seasons dataset, which is part of the Visual Localization benchmark. As the results on the RobotCar benchmark dataset are nearly on par with the top ranked approaches, we focused our investigations on reproducibility and performance with own data. For this purpose, we created a dataset with street-level imagery. In order to have independent reference and query images, we used a road-based and a tram-based mapping campaign with a time difference of four years. The approximately 90% successfully oriented images of both datasets are a good indicator for the robustness of our approach. With about 50% success rate, every second image could be localized with a position accuracy better than 0.25 m and a rotation accuracy better than 2°.

CENTIMETRE-ACCURACY IN FORESTS AND URBAN CANYONS – COMBINING A HIGH-PERFORMANCE IMAGE-BASED MOBILE MAPPING BACKPACK WITH NEW GEOREFERENCING METHODS

Abstract. Advances in digitalization technologies lead to rapid and massive changes in infrastructure management. New collaborative processes and workflows require detailed, accurate and up-to-date 3D geodata. Image-based web services with 3D measurement functionality, for example, transfer dangerous and costly inspection and measurement tasks from the field to the office workplace. In this contribution, we introduced an image-based backpack mobile mapping system and new georeferencing methods for capture previously inaccessible outdoor locations. We carried out large-scale performance investigations at two different test sites located in a city centre and in a forest area. We compared the performance of direct, SLAM-based and image-based georeferencing under demanding real-world conditions. Both test sites include areas with restricted GNSS reception, poor illumination, and uniform or ambiguous geometry, which create major challenges for reliable and accurate georeferencing. In our comparison of georeferencing methods, image-based georeferencing improved the median precision of coordinate measurement over direct georeferencing by a factor of 10–15 to 3 mm. Image-based georeferencing also showed a superior performance in terms of absolute accuracies with results in the range from 4.3 cm to 13.2 cm. Our investigations showed a great potential for complementing 3D image-based geospatial web-services of cities as well as for creating such web services for forest applications. In addition, such accurately georeferenced 3D imagery has an enormous potential for future visual localization and augmented reality applications.

Autonomous Navigation of Multiple Robots with Sensing and Communication Constraints Based on Mixed Reality

This paper presents a robotic navigation system that uses mixed reality concepts to develop sensing and communication virtual devices, based on the visual localization of the robot in the environment. The main objective of the navigation system is to provide conditions for the use of very simple robots with severe limitations on the mentioned peripheral devices for simulation, analysis and test of multi-robot applications. In an experiment with real robots, each one receives its virtual navigation skills in an independent way from the tool that emulates the function of such peripherals. Thus, the behavior of a group of robots, independently commanded, is implemented in the virtual environment and accomplished in the real world. An experiment composed by real multiple Sphero robots executing an exploratory task within an unknown dynamic environment is carried out to validate the proposed navigation system. The use of mixed reality concepts allows an easy implementation of cooperation mechanisms based on indirect communication skill and fuzzy controllers for the robots’ movement. The results confirm the feasibility of the proposed autonomous navigation system.

Hierarchical visual localization for visually impaired people using multimodal images

Localization is one of the crucial issues in assistive technology for visually impaired people. In this paper, we propose a novel hierarchical visual localization pipeline based on the wearable assistive navigation device for visually impaired people. The proposed pipeline involves the deep descriptor network, 2D-3D geometric verification and online sequence matching. Images in different modalities (RGB, Infrared and Depth) are fed into Dual Desc network to generate robust attentive global descriptors and local features. The global descriptors are leveraged to retrieve the coarse candidates of query images. The 2D local features, as well as 3D sparse point cloud, are used in geometric verification to select the optimal results from the retrieved candidates. Finally, sequence matching robustifies the localization results by synthesizing the verified results of successive frames. The proposed unified descriptor network Dual Desc surpasses the state-of-the-art NetVLAD and its variant on the task of image description. Validated on the real-world dataset captured by the wearable assistive device, the proposed visual localization utilizes multimodal images to overcome the disadvantages of RGB images and robustifies the localization performance by deep descriptor network and hierarchical pipeline. In the challenging scenarios of the Yuquan dataset, the proposed method achieves the F1 score of 0.77 and the mean localization error of 2.75, which is satisfactory in practical use.

Error analysis based on error transfer theory and compensation strategy for LED chip visual localization systems

In the manufacturing process of LED chips, the accuracy of the LED chip visual localization system affects the quality of LED chip production directly. There are many errors that have impacts on positioning system accuracy. Therefore, the identification and compensation of the critical errors is key to efficiently improving the precision of the system. Based on this fact, an error analysis method and an error compensation strategy are proposed in this paper. The first step was to measure the relevant error sources that may affect the localization system. Then error model of localization system was established, and the validity of this error model was verified by comparing simulated and actual positioning results. In addition, the impact factors of each error source on localization system accuracy were obtained using the error transfer theory. According to the error analysis results, an efficient error compensation strategy was proposed, which could compensate the errors in order of impact factors, and judge whether the error compensation method is optimal. Finally, the experimental results proved that the proposed error analysis method was valid and the error compensation strategy could efficiently enhance the positioning accuracy to meet the industrial application requirements.

Pseudo-3D Vision-Inertia Based Underwater Self-Localization for AUVs

Autonomous underwater vehicles (AUVs) play an important role in subaqueous construction and oceanographic survey, in which AUV self-localization is a crucial component. Current localization methods mainly depend on expensive acoustic positioning systems, which limit the wide application of AUVs. In this paper, referring to the low-cost visual localization method and focusing on challenges caused by underwater environment, we propose an underwater self-localization method based on Pseudo-3D vision-inertia for AUVs. The proposed method merges depth information into 2D visual image to achieve continuous and robust localization under dramatically changing underwater environment. In order to decrease errors, we propose an online fusion method based on tightly-coupled nonlinear optimization to fuse the measurements of the pre-integrated inertial measurement unit and the observations from the down-looking camera. We also optimize four degrees-of-freedom pose graph to enhance the global consistency and design an online loop detection module to realize the underwater relocalization. In addition, we develop a low-cost, portable, and small volume sensor suite for underwater vehicle localization and test the proposed self-localization method. We test the proposed method in the underwater environment using the custom-made sensor suite, and the experimental results demonstrate the effectiveness of the proposed method under dramatically changing underwater environment.

Audio-Visual Thumble (AVT): A low-vision rehabilitation device using multisensory feedbacks.

Since the 70s sensory substitution devices have been used for blind individuals to compensate for the lack of vision and enable them to perceive environment through intact sensory modalities. In this study, we present a rehabilitation device called Audio Visual Thumble (AVT), which is a small ring-like device with LED and buzzer, that can be worn on pharynx. We focus on a unique group of low-vision individuals with a black spot or scotoma in their visual field due to a disease called Macular Degeneration. The visual localization abilities of these individuals are highly impaired due to developing scotoma. We recently showed that also their audio localization skills are impaired [9]. Rehabilitation techniques developed so far for Macular Degeneration focus on visual modality only. Since audition can also be used to improve their spatial skills, we developed the AVT device. It permits to associate the multisensory information (audio and visual feedbacks) coming from the device with the own movement (proprioceptive feedback). We propose that the AVT has the potential to help people with visual dysfunctions to improve in the identification of audio and visual targets outside or at the edge of the residual visual field. AVT could be used for a wide range of applications combined with classical rehabilitation techniques in Macular Degeneration patients.Clinical relevance- This device can be an effective addition for low-vision rehabilitation experts and can be used combined with classical rehabilitation methods.

Trade-off between resolution and frame rate of visual tracking of mini-robots on an experimental planar platform

Accurate and fast visual localization is required in many applications of mini-robotics. Obtaining the best possible result for a given platform requires a balanced combination of camera settings and efficient image processing of the acquired image. In this paper, we study the trade-off between a high-resolution and a high-speed acquisition mode of a conventional camera for an experimental platform of magnetically-propelled mini-robots. Specifically, we propose a two stage localization algorithm based on fast pre-location using block matching followed by optical flow correction for subpixel accuracy of localization. In the experimental evaluation, we show that the difference in localization using two images of the same scene in resolutions 1000 × 1000px and 200 × 200px is only 0.2px of the higher resolution. The computational cost of the lower resolution is 16times lower than that of the higher resolution. This allows greater accuracy localization at a higher frame rate, which significantly improves the dynamics of control. Experimental results demonstrate the precision and speed of the proposed algorithm in the task of tracking a magnetically propelled robot on the platform.

Visual place recognition using directed acyclic graph association measures and mutual information-based feature selection

Visual localization is a challenging problem, especially over the long run, since places can exhibit significant variation due to dynamic environmental and seasonal changes. To tackle this problem, we propose a visual place recognition method based on directed acyclic graph matching and feature maps extracted from deep convolutional neural networks (DCNN). Furthermore, in order to find the best subset of DCNN feature maps with minimal redundancy, we propose to form probability distributions on image representation features and leverage the Jensen–Shannon divergence to rank features. We evaluate the proposed approach on two challenging public datasets, namely the Bonn and the Freiburg datasets, and compare it to the state-of-the-art methods. For image representations, we evaluated the following DCNN architectures: AlexNet, OverFeat, ResNet18 and ResNet50. Due to the proposed graph structure, we are able to account for any kind of correlations in image sequences, and therefore dub our approach NOSeqSLAM. Algorithms with and without feature selection were evaluated based on precision–recall curves, area under the curve score, best recall at 100% precision score and running time, with NOSeqSLAM outperforming the counterpart approaches. Furthermore, by formulating the mutual information-based feature selection specifically for visual place recognition and by selecting the feature percentile with the best score, all the algorithms, and not just NOSeqSLAM, exhibited enhanced performance with the reduced feature set.

Image Blind Denoising Using a Generative Adversarial Network for LED Chip Visual Localization

The noise in an LED chip image directly affects its visual positioning accuracy and efficiency. The existing denoising method can improve the image visual quality after removing the noise, but the improvement of the recognition rate and the positioning accuracy of the chip is not apparent. To solve this problem, this paper proposes a convolutional neural network combined with generative adversarial networks framework, the (CNN-GAN-GAN)-based blind denoiser, comprising three subnetworks. First, a noise extraction subnetwork with the proposed adaptive residual dense block extracts noise information from noisy images. Then, the noise modeling subnetwork with a dual-discriminator learns the noise distribution and constructs a paired training dataset. Finally, the blind denoising subnetwork with a hybrid loss function is trained for denoising. Contrast experiments show the proposed method can effectively suppress image noise and artifacts and meet the requirements of LED chip visual positioning accuracy while ensuring chip recognition rate.

CEB-Map: Visual Localization Error Prediction for Safe Navigation

For safe visual navigation, areas with high localization errors should be concentrated and could be further refined by additional mapping operations. Given an environment map, we propose to predict the visual localization error and hence to either improve the navigation performance or call an additional mapping to refine the built map. Previous work adopts the uncertainty of landmarks for the error prediction. In our work, we take into account both the spatial distribution of visual landmarks and the uncertainty of landmarks. Our main idea is that standing at one place, a good spatial distribution of landmarks means a sufficient enough visible landmarks from all views at that place, i.e., enough landmarks under arbitrary view-direction. Combining the spatial distribution and the uncertainty of landmarks, we propose a new framework to predict the error of visual localization. Furthermore, we show that additional mapping in the area with high predicted error can significantly improve the visual localization precision. Experimental results show that there is a strong relationship between the predicted error and the real error, of which the absolute value of correlation coefficient is between 0.707 to 0.915. We apply our method to conduct an optimal refining policy on the built map and the experimental results show the improved localization precision. Applications on navigation test verify the superiority of our proposed method.

Visual localization for asteroid touchdown operation based on local image features

In an asteroid sample-return mission, accurate position estimation of the spacecraft relative to the asteroid is essential for landing at the target point. During the missions of Hayabusa and Hayabusa2, the main part of the visual position estimation procedure was performed by human operators on the Earth based on a sequence of asteroid images acquired and sent by the spacecraft. Although this approach is still adopted in critical space missions, there is an increasing demand for automated visual position estimation, so that the time and cost of human intervention may be reduced. In this paper, we propose a method for estimating the relative position of the spacecraft and asteroid during the descent phase for touchdown from an image sequence using state-of-the-art techniques of image processing, feature extraction, and structure from motion. We apply this method to real Ryugu images that were taken by Hayabusa2 from altitudes of 20 km-500 m. It is demonstrated that the method has practical relevance for altitudes within the range of 5-1 km. This result indicates that our method could improve the efficiency of the ground operation in the global mapping and navigation during the touchdown sequence, whereas full automation and autonomous on-board estimation are beyond the scope of this study. Furthermore, we discuss the challenges of developing a completely automatic position estimation framework.

Semantic signatures for large-scale visual localization

Visual localization is a useful alternative to standard localization techniques. It works by utilizing cameras. In a typical scenario, features are extracted from captured images and compared with geo-referenced databases. Location information is then inferred from the matching results. Conventional schemes mainly use low-level visual features. These approaches offer good accuracy but suffer from scalability issues. In order to assist localization in large urban areas, this work explores a different path by utilizing high-level semantic information. It is found that object information in a street view can facilitate localization. A novel descriptor scheme called "semantic signature" is proposed to summarize this information. A semantic signature consists of type and angle information of visible objects at a spatial location. Several metrics and protocols are proposed for signature comparison and retrieval. They illustrate different trade-offs between accuracy and complexity. Extensive simulation results confirm the potential of the proposed scheme in large-scale applications. This paper is an extended version of a conference paper in CBMI'18. A more efficient retrieval protocol is presented with additional experiment results.

Improved Deep Distance Learning for Visual Loop Closure Detection in Smart City

Visual Simultaneous Localization and Mapping (vSLAM) are expected to promote the initiatives in Smart City including driverless cars and intelligent robots. Loop closure detection (LCD) is an important module in a vSLAM system. Existing works with convolutional neural networks exhibit better performance on feature extraction, but this is far from enough. Concerning the characteristics of LCD, it is of great significance to have a customized loss function and a method to construct suitable training image sets. Based on this motivation, we propose a novel framework for LCD. Through a deep analysis of the distance relationships in the LCD problem, we propose the multi-tuplet clusters loss function together with mini-batch construction scheme. The proposed framework can map images to a low dimensional space and extract more discriminative image features, which help learn a more essential distance relationship of the LCD problem. Extensive evaluations demonstrate that our method outperforms many state-of-art approaches even in complex environments with strong appearance changes. Importantly, though the training process is computationally demanding, its online application is very efficient.

Vision and Wi-Fi fusion in probabilistic appearance-based localization

This article introduces an indoor topological localization algorithm that uses vision and Wi-Fi signals. Its main contribution is a novel way of merging data from these sensors. The designed system does not require knowledge of the building plan or the positions of the Wi-Fi access points. By making the Wi-Fi signature suited to the FABMAP algorithm, this work develops an early fusion framework that solves global localization and kidnapped robot problems. The resulting algorithm has been tested and compared with FABMAP visual localization, over data acquired by a Pepper robot in three different environments: an office building, a middle school, and a private apartment. Numerous runs of different robots have been realized over several months for a total covered distance of 6.4 km. Constraints were applied during acquisitions to make the experiments fitted to real use cases of Pepper robots. Without any tuning, our early fusion framework outperforms visual localization in all testing situations and with a significant margin in environments where vision faces problems such as moving objects or perceptual aliasing. In such conditions, 90.6% of estimated localizations are less than 5 m away from ground truth with our early fusion framework compared with 77.6% with visual localization. Furthermore, compared with other classical fusion strategies, the early fusion framework produces the best localization results because in all tested situations, it improves visual localization results without damaging them where Wi-Fi signals carry little information.

2-Entity Random Sample Consensus for Robust Visual Localization: Framework, Methods, and Verifications

Robust and efficient visual localization is essential for numerous robotic applications. However, it remains a challenging problem especially when significant environmental or perspective changes are present, as there are high percentage of outliers, i.e., incorrect feature matches between the query image and the map. In this article, we propose a novel 2-entity random sample consensus (RANSAC) framework using three-dimensional–two-dimensional point and line feature matches for visual localization with the aid of inertial measurements and derive minimal closed-form solutions using only 1 point 1 line or 2 point matches for both monocular and multi-camera system. The proposed 2-entity RANSAC can achieve higher robustness against outliers as multiple types of features are utilized and the number of matches needed to compute a pose is reduced. Furthermore, we propose a learning-based sampling strategy selection mechanism and a feature scoring network to be adaptive to different environmental characteristics such as structured and unstructured. Finally, both simulation and real-world experiments are performed to validate the robustness and effectiveness of the proposed method in scenarios with long-term and perspective changes. 1 1 https://youtu.be/Zqgxntz11hI .

Multisensory Integration Develops Prior to Crossmodal Recalibration.

It has been hypothesized that crossmodal recalibration plays a crucial role for the development of multisensory integration capabilities [1]. To test the developmental trajectory of multisensory integration and crossmodal recalibration, we used a combined ventriloquist/ventriloquist aftereffect paradigm [2] in children aged 5-9 years. The ventriloquist effect (indicating multisensory integration), that is, the shift of auditory localization toward simultaneously presented but spatially discrepant visual stimuli, was larger in children than in adults, which was attributed to a lower auditory localization precision in the children. In fact, the size of the ventriloquist effect depended on the visual stimulus reliability in both children and adults. In all groups, the ventriloquist effect was best explained by a causal inference model. In contrast to their multisensory integration capabilities, 5-year-old children did not recalibrate. The immediate ventriloquist aftereffect (indicating recalibration after a single exposure to a spatially discrepant audio-visual stimulus) emerged in 6- to 7-year-old children, whereas the cumulative ventriloquist aftereffect (reflecting recalibration to the audio-visual spatial discrepancies over the complete experiment) was not observed before the age of 8 years. First, in contrast to common beliefs, the present results provide evidence that multisensory integration precedes rather than follows crossmodal recalibration during development. Second, we report developmental evidence for a dissociation of the processes involved in multisensory integration and immediate as well as cumulative recalibration. We speculate that multisensory integration is a prerequisite for crossmodal recalibration, because the multisensory percept, rather than unimodal cues, might comprise a crucial signal for the calibration of the sensory systems.

Landing site topographic mapping and rover localization for Chang’e-4 mission

This paper presents the techniques and results of landing-site topographic mapping and rover localization using orbital, descent and rover images in the Chang’e-4 mission. High-resolution maps of the landing site are generated from orbital and descent images. Local digital elevation models and digital orthophoto maps with 0.02 m resolution are generated at each waypoint. The location of the lander is determined as (177.588° E, 45.457° S) using festure-matching techniques. The cross-site visual localization method is routinely used to localize the rover at each waypoint to reduce error accumulation from wheel slippage and IMU drift in dead reckoning. After the first five lunar days, the rover travels 186.66 m from the lander, according to the cross-site visual localization. The developed methods and results have been directly utilized to support the mission’s operations. The maps and localization information are also valuable for supporting multiple scientific explorations of the landing site.

Representational momentum in vision and touch: Visual motion information biases tactile spatial localization

After an object disappears, the vanishing point is shifted in the direction of motion, a phenomenon known as representational momentum. The present study focused on the relationship between motion information and spatial location in a crossmodal setting. In two visuotactile experiments, we studied how motion information in one sensory modality affects the perceived final location of a motion signal (congruent vs. incongruent left-right motion direction) in another modality. The results revealed a unidirectional crossmodal influence of motion information on spatial localization performance. While visual motion information influenced the perceived final location of the tactile stimulus, tactile motion information had no influence on visual localization. These results therefore extend the existing literature on crossmodal influences on spatial location and are discussed in relation to current theories of multisensory perception.