Localization Pipeline Research Articles

Landmark-in-facial-component: Towards occlusion-robust facial landmark localization

Despite great efforts in recent years to research robust facial landmark localization methods, occlusion remains a challenge. To tackle this challenge, we propose a model called the Landmark-in-Facial-Component Network (LFCNet). Unlike mainstream models that focus on boundary information, LFCNet utilizes the strong structural constraints inherent in facial anatomy to address occlusion. Specifically, two key modules are designed, a component localization module and an offset localization module. After grouping landmarks based on facial components, the component localization module accomplishes coarse localization of facial components. Offset localization module performs fine localization of landmarks based on the coarse localization results, which can also be seen as delineating the shape of facial components. These two modules form a coarse-to-fine localization pipeline and can also enable LFCNet to better learn the shape constraint of human faces, thereby enhancing LFCNet's robustness to occlusion. LFCNet achieves 4.82% normalized mean error on occlusion subset of WFLW dataset and 6.33% normalized mean error on Masked 300W dataset. The results demonstrate that LFCNet achieves excellent performance in comparison to state-of-the-art methods, especially on occlusion datasets.

Context-aware deep learning enables high-efficacy localization of high concentration microbubbles for super-resolution ultrasound localization microscopy

Ultrasound localization microscopy (ULM) enables deep tissue microvascular imaging by localizing and tracking intravenously injected microbubbles circulating in the bloodstream. However, conventional localization techniques require spatially isolated microbubbles, resulting in prolonged imaging time to obtain detailed microvascular maps. Here, we introduce LOcalization with Context Awareness (LOCA)-ULM, a deep learning-based microbubble simulation and localization pipeline designed to enhance localization performance in high microbubble concentrations. In silico, LOCA-ULM enhanced microbubble detection accuracy to 97.8% and reduced the missing rate to 23.8%, outperforming conventional and deep learning-based localization methods up to 17.4% in accuracy and 37.6% in missing rate reduction. In in vivo rat brain imaging, LOCA-ULM revealed dense cerebrovascular networks and spatially adjacent microvessels undetected by conventional ULM. We further demonstrate the superior localization performance of LOCA-ULM in functional ULM (fULM) where LOCA-ULM significantly increased the functional imaging sensitivity of fULM to hemodynamic responses invoked by whisker stimulations in the rat brain.

TreeDetector: Using Deep Learning for the Localization and Reconstruction of Urban Trees from High-Resolution Remote Sensing Images

There have been considerable efforts in generating tree crown maps from satellite images. However, tree localization in urban environments using satellite imagery remains a challenging task. One of the difficulties in complex urban tree detection tasks lies in the segmentation of dense tree crowns. Currently, methods based on semantic segmentation algorithms have made significant progress. We propose to split the tree localization problem into two parts, dense clusters and single trees, and combine the target detection method with a procedural generation method based on planting rules for the complex urban tree detection task, which improves the accuracy of single tree detection. Specifically, we propose a two-stage urban tree localization pipeline that leverages deep learning and planting strategy algorithms along with region discrimination methods. This approach ensures the precise localization of individual trees while also facilitating distribution inference within dense tree canopies. Additionally, our method estimates the radius and height of trees, which provides significant advantages for three-dimensional reconstruction tasks from remote sensing images. We compare our results with other existing methods, achieving an 82.3% accuracy in individual tree localization. This method can be seamlessly integrated with the three-dimensional reconstruction of urban trees. We visualized the three-dimensional reconstruction of urban trees generated by this method, which demonstrates the diversity of tree heights and provides a more realistic solution for tree distribution generation.

Challenging interferometric imaging: Machine learning-based source localization from uv-plane observations

Context. Rising interest in radio astronomy and upcoming projects in the field is expected to produce petabytes of data per day, questioning the applicability of traditional radio astronomy data analysis approaches under the new large-scale conditions. This requires new, intelligent, fast, and efficient methods that potentially involve less input from the domain expert. Aims. In our work, we examine, for the first time, the possibility of fast and efficient source localization directly from the uv-observations, omitting the recovering of the dirty or clean images. Methods. We propose a deep neural network-based framework that takes as its input a low-dimensional vector of sampled uv-data and outputs source positions on the sky. We investigated a representation of the complex-valued input uv-data via the real and imaginary and the magnitude and phase components. We provided a comparison of the efficiency of the proposed framework with the traditional source localization pipeline based on the state-of-the-art Python Blob Detection and Source Finder (PyBDSF) method. The investigation was performed on a data set of 9164 sky models simulated using the Common Astronomy Software Applications (CASA) tool for the Atacama Large Millimeter Array (ALMA) Cycle 5.3 antenna configuration. Results. We investigated two scenarios: (i) noise-free as an ideal case and (ii) sky simulations including noise representative of typical extra-galactic millimeter observations. In the noise-free case, the proposed localization framework demonstrates the same high performance as the state-of-the-art PyBDSF method. For noisy data, however, our new method demonstrates significantly better performance, achieving a completeness level that is three times higher for sources with uniform signal-to-noise ratios (S/N) between 1 and 10, and a high increase in completeness in the low S/N regime. Furthermore, the execution time of the proposed framework is significantly reduced (by factors ~30) as compared to traditional methods that include image reconstructions from the uv-plane and subsequent source detections. Conclusions. The proposed framework for obtaining fast and efficient source localization directly from uv-plane observations shows very encouraging results, which could open new horizons for interferometric imaging with existing and future facilities.

Towards Long-Term Retrieval-Based Visual Localization in Indoor Environments With Changes

Visual localization is a challenging task due to the presence of illumination changes, occlusion, and perception from novel viewpoints. Re-localizing the camera pose in long-term setups raises difficulties caused by changes in scene appearance and geometry introduced by human or natural deterioration. Many existing methods use static scene assumptions and fail in dynamic indoor scenes. Only a few works handle scene changes by introducing outlier awareness with pure learning methods. Other recent approaches use semantics to robustify camera localization in changing setups. However, to the best of our knowledge, no method has yet used scene graphs in feature-based approaches to introduce change awareness. In this work, we propose a novel feature-based camera re-localization method that leverages scene graphs within retrieval and feature detection and matching. Semantic scene graphs are used to estimate scene changes by matching instances and relationship triplets. The knowledge of scene changes is then used for our change-aware image retrieval and feature correspondence verification. We show the potential of integrating higher-level knowledge about the scene within a retrieval-based localization pipeline. Our method is evaluated on the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RIO10</i> benchmark with comprehensive evaluations on different levels of scene changes.

DLALoc: Deep-Learning Accelerated Visual Localization Based on Mesh Representation

Visual localization, i.e., the camera pose localization within a known three-dimensional (3D) model, is a basic component for numerous applications such as autonomous driving cars and augmented reality systems. The most widely used methods from the literature are based on local feature matching between a query image that needs to be localized and database images with known camera poses and local features. However, this method still struggles with different illumination conditions and seasonal changes. Additionally, the scene is normally presented by a sparse structure-from-motion point cloud that has corresponding local features to match. This scene representation depends heavily on different local feature types, and changing the different local feature types requires an expensive feature-matching step to generate the 3D model. Moreover, the state-of-the-art matching strategies are too resource intensive for some real-time applications. Therefore, in this paper, we introduce a novel framework called deep-learning accelerated visual localization (DLALoc) based on mesh representation. In detail, we employ a dense 3D model, i.e., mesh, to represent a scene that can provide more robust 2D-3D matches than 3D point clouds and database images. We can obtain their corresponding 3D points from the depth map rendered from the mesh. Under this scene representation, we use a pretrained multilayer perceptron combined with homotopy continuation to calculate the relative pose of the query and database images. We also use the scale consistency of 2D-3D matches to perform the efficient random sample consensus to find the best 2D inlier set for the subsequential perspective-n-point localization step. Furthermore, we evaluate the proposed visual localization pipeline experimentally on Aachen DayNight v1.1 and RobotCar Seasons datasets. The results show that the proposed approach can achieve state-of-the-art accuracy and shorten the localization time about five times.

UNav: An Infrastructure-Independent Vision-Based Navigation System for People with Blindness and Low Vision.

Vision-based localization approaches now underpin newly emerging navigation pipelines for myriad use cases, from robotics to assistive technologies. Compared to sensor-based solutions, vision-based localization does not require pre-installed sensor infrastructure, which is costly, time-consuming, and/or often infeasible at scale. Herein, we propose a novel vision-based localization pipeline for a specific use case: navigation support for end users with blindness and low vision. Given a query image taken by an end user on a mobile application, the pipeline leverages a visual place recognition (VPR) algorithm to find similar images in a reference image database of the target space. The geolocations of these similar images are utilized in a downstream task that employs a weighted-average method to estimate the end user's location. Another downstream task utilizes the perspective-n-point (PnP) algorithm to estimate the end user's direction by exploiting the 2D-3D point correspondences between the query image and the 3D environment, as extracted from matched images in the database. Additionally, this system implements Dijkstra's algorithm to calculate a shortest path based on a navigable map that includes the trip origin and destination. The topometric map used for localization and navigation is built using a customized graphical user interface that projects a 3D reconstructed sparse map, built from a sequence of images, to the corresponding a priori 2D floor plan. Sequential images used for map construction can be collected in a pre-mapping step or scavenged through public databases/citizen science. The end-to-end system can be installed on any internet-accessible device with a camera that hosts a custom mobile application. For evaluation purposes, mapping and localization were tested in a complex hospital environment. The evaluation results demonstrate that our system can achieve localization with an average error of less than 1 m without knowledge of the camera's intrinsic parameters, such as focal length.

Automatic localization of target point for subthalamic nucleus-deep brain stimulation via hierarchical attention-UNet based MRI segmentation.

Deep brain stimulation of the subthalamic nucleus (STN-DBS) is an effective treatment for patients with advanced Parkinson's disease, the outcome of this surgery is highly dependent on the accurate placement of the electrode in the optimal target of STN. In this study, we aim to develop a target localization pipeline for DBS surgery, considering that the heart of this matter is to achieve the STN and red nucleus segmentation, a deep learning-based automatic segmentation approach is proposed to tackle this issue. To address the problems of ambiguous boundaries and variable shape of the segmentation targets, the hierarchical attention mechanism with two different attention strategies is integrated into an encoder-decoder network for mining both semantics and fine-grained details for segmentation. The hierarchical attention mechanism is utilized to suppress irrelevant regions in magnetic resonance (MR) images while build long-range dependency among segmentation targets. Specifically, the attention gate (AG) is integrated into low-level features to suppress irrelevant regions in an input image while highlighting the salient features useful for segmentation. Besides, the self-attention involved in the transformer block is integrated into high-level features to model the global context. Ninety-nine brain magnetic resonance imaging (MRI) studies were collected from 99 patients with Parkinson's disease undergoing STN-DBS surgery, among which 80 samples were randomly selected as the training datasets for deep learning training, and ground truths (segmentation masks) were manually generated by radiologists. We applied five-fold cross-validation on these data to train our model, the mean results on 19 test samples are used to conduct the comparison experiments, the Dice similarity coefficient (DSC), Jaccard (JA), sensitivity (SEN), and HD95 of the segmentation for STN are 88.20%, 80.32%, 90.13%, and 1.14mm, respectively, outperforming the state-of-the-art STN segmentation method with 2.82%, 4.52%, 2.56%, and 0.02mm respectively. The source code and trained models of this work have been released in the URL below: https://github.com/liuruiqiang/HAUNet/tree/master. In this study, we demonstrate the effectiveness of the hierarchical attention mechanism for building global dependency on high-level semantic features and enhancing the fine-grained details on low-level features, the experimental results show that our method has considerable superiority for STN and red nucleus segmentation, which can provide accurate target localization for STN-DBS.

In-Hand Object Localization Using a Novel High-Resolution Visuotactile Sensor

In-hand object localization and manipulation has always been a challenging task in robotic community. In this article, we address this problem by vision-based tactile sensing with high-spatial resolution. Specifically, we design a novel tactile sensor based on stereo vision, named GelStereo, which can perceive tactile point cloud with high-spatial resolution ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$&lt;\!1$</tex-math></inline-formula> mm). A tactile-based in-hand object localization pipeline composed of saliency detection and probabilistic point-set registration algorithms of the perceived contact point cloud is presented. Furthermore, extensive qualitative and quantitative analyses of perceived tactile point cloud and in-hand localization and insertion experiments of small parts are performed on our robot platform. The experimental results verify the accuracy and robustness of the tactile point cloud sensed by the novel GelStereo tactile sensor and the proposed in-hand object localization pipeline. This novel high-resolution visuotactile sensing technology has predictable application potential in the field of dexterous robotic manipulation.

Unsupervised Semantic Segmentation of Aerial Images With Application to UAV Localization

Recent advances and applications of aerial image semantic segmentation have yielded an increase in their use in day-to-day tasks. However, state-of-art algorithms, composed mostly of deep semantic segmentation networks (DSSNs), may not be suitable for application domains in which labels (targeted output masks) are scarce. This work proposes a fully unsupervised semantic segmentation method able to find the appropriate number of semantics labels (yielded from clustering) without the need of a previously annotated dataset. Once the semantic labels are identified, a classifier trained from these labels can be used to assign semantics to new input images. In contrast to other clustering-based segmentation methods, our method does not require any input parameter to find the clustering partition that best accommodates data. Empirical results on two real datasets showed that the use of semantics in a template-matching-based unmanned aerial vehicle (UAV) localization pipeline reduced the overall position estimation error.

Processing Chain for Localization of Magnetoelectric Sensors in Real Time.

The knowledge of the exact position and orientation of a sensor with respect to a source (distribution) is essential for the correct solution of inverse problems. Especially when measuring with magnetic field sensors, the positions and orientations of the sensors are not always fixed during measurements. In this study, we present a processing chain for the localization of magnetic field sensors in real time. This includes preprocessing steps, such as equalizing and matched filtering, an iterative localization approach, and postprocessing steps for smoothing the localization outcomes over time. We show the efficiency of this localization pipeline using an exchange bias magnetoelectric sensor. For the proof of principle, the potential of the proposed algorithm performing the localization in the two-dimensional space is investigated. Nevertheless, the algorithm can be easily extended to the three-dimensional space. Using the proposed pipeline, we achieve average localization errors between 1.12 cm and 6.90 cm in a localization area of size .

The effect of density thresholding on the EEG network construction

The procedure of thresholding for graph construction is one of the common steps in calculating networks of brain connections. However, this procedure can lead to incomparable results from different studies. In the present study we aim to test the effect of thresholding or algorithmic reduction of the number of connected nodes on the construction of a set of widely used connectivity graph metrics derived from EEG data. 164 people took part in our study. Participants were recruited via social networks. EEG was recorded during resting state. At the beginning of the procedure each participant was asked to relax and not to think about anything. Source reconstruction was performed using standard source localization pipeline from MNE-package. Desikan-Killiany Atlas was used for cortical parcellation with 34 ROI per hemisphere. Synchronization was estimated with weighted phase lag index in 4–30 Hz frequency range for eyes closed and eyes open separately. We have found that All metrics except average participation coefficient vary monotonously as a function of density level (moreover, we have found, that for Cluster Coefficient, more than 95% and for the Characteristic Path Length ∼50% of the variance is related to thresholding cut-off). The different data-driven approaches to the network construction leads to significant changes in the group-level graph metrics and can eliminate the variance in the data that can be crucial for individual differences studies.

Probabilistic Visual Place Recognition for Hierarchical Localization

Visual localization techniques often comprise a hierarchical localization pipeline, with a visual place recognition module used as a coarse localizer to initialize a pose refinement stage. While improving the pose refinement step has been the focus of much recent research, most work on the coarse localization stage has focused on improvements like increased invariance to appearance change, without improving what can be loose error tolerances. In this letter, we propose two methods which adapt image retrieval techniques used for visual place recognition to the Bayesian state estimation formulation for localization. We demonstrate significant improvements to the localization accuracy of the coarse localization stage using our methods, whilst retaining state-of-the-art performance under severe appearance change. Using extensive experimentation on the Oxford RobotCar dataset, results show that our approach outperforms comparable state-of-the-art methods in terms of precision-recall performance for localizing image sequences. In addition, our proposed methods provides the flexibility to contextually scale localization latency in order to achieve these improvements. The improved initial localization estimate opens up the possibility of both improved overall localization performance and modified pose refinement techniques that leverage this improved spatial prior.

Hierarchical visual localization for visually impaired people using multimodal images

Localization is one of the crucial issues in assistive technology for visually impaired people. In this paper, we propose a novel hierarchical visual localization pipeline based on the wearable assistive navigation device for visually impaired people. The proposed pipeline involves the deep descriptor network, 2D-3D geometric verification and online sequence matching. Images in different modalities (RGB, Infrared and Depth) are fed into Dual Desc network to generate robust attentive global descriptors and local features. The global descriptors are leveraged to retrieve the coarse candidates of query images. The 2D local features, as well as 3D sparse point cloud, are used in geometric verification to select the optimal results from the retrieved candidates. Finally, sequence matching robustifies the localization results by synthesizing the verified results of successive frames. The proposed unified descriptor network Dual Desc surpasses the state-of-the-art NetVLAD and its variant on the task of image description. Validated on the real-world dataset captured by the wearable assistive device, the proposed visual localization utilizes multimodal images to overcome the disadvantages of RGB images and robustifies the localization performance by deep descriptor network and hierarchical pipeline. In the challenging scenarios of the Yuquan dataset, the proposed method achieves the F1 score of 0.77 and the mean localization error of 2.75, which is satisfactory in practical use.

Learning Matchable Image Transformations for Long-Term Metric Visual Localization

Long-term metric self-localization is an essential capability of autonomous mobile robots, but remains challenging for vision-based systems due to appearance changes caused by lighting, weather, or seasonal variations. While experience-based mapping has proven to be an effective technique for bridging the `appearance gap,' the number of experiences required for reliable metric localization over days or months can be very large, and methods for reducing the necessary number of experiences are needed for this approach to scale. Taking inspiration from color constancy theory, we learn a nonlinear RGB-to-grayscale mapping that explicitly maximizes the number of inlier feature matches for images captured under different lighting and weather conditions, and use it as a pre-processing step in a conventional single-experience localization pipeline to improve its robustness to appearance change. We train this mapping by approximating the target non-differentiable localization pipeline with a deep neural network, and find that incorporating a learned low-dimensional context feature can further improve cross-appearance feature matching. Using synthetic and real-world datasets, we demonstrate substantial improvements in localization performance across day-night cycles, enabling continuous metric localization over a 30-hour period using a single mapping experience, and allowing experience-based localization to scale to long deployments with dramatically reduced data requirements.

Unifying Visual Localization and Scene Recognition for People With Visual Impairment

With the development of computer vision and mobile computing, assistive navigation for people with visual impairment arouses the attention of research communities. As two key challenges of assistive navigation, “Where am I?” and “What are the surroundings?” are still to be resolved by taking advantage of visual information. In this paper, we leverage the prevailing compact network as the backbone to build a unified network featuring two branches that implement scene description and scene recognition separately. Based on the unified network, the proposed pipeline performs scene recognition and visual localization simultaneously in the scenario of assistive navigation. The visual localization pipeline involves image retrieval and sequence matching. In the experiments, different configurations of the proposed pipeline are tested on public datasets to search for the optimal parameters. Moreover, on the real-world datasets captured by the wearable assistive device, the proposed assistive navigation pipeline is proved to achieve satisfactory performance. On the challenging dataset, the top-5 precision of scene recognition is more than 80%, and the visual localization precision is over 60% under a recall of 60%. The related codes and datasets are open-source online at https://github.com/chengricky/ScenePlaceRecognition .

IEEGview: an open-source multifunction GUI-based Matlab toolbox for localization and visualization of human intracranial electrodes

Objective. The precise localization of intracranial electrodes is a fundamental step relevant to the analysis of intracranial electroencephalography (iEEG) recordings in various fields. With the increasing development of iEEG studies in human neuroscience, higher requirements have been posed on the localization process, resulting in urgent demand for more integrated, easy-operation and versatile tools for electrode localization and visualization. With the aim of addressing this need, we develop an easy-to-use and multifunction toolbox called iEEGview, which can be used for the localization and visualization of human intracranial electrodes. Approach. iEEGview is written in Matlab scripts and implemented with a GUI. From the GUI, by taking only pre-implant MRI and post-implant CT images as input, users can directly run the full localization pipeline including brain segmentation, image co-registration, electrode reconstruction, anatomical information identification, activation map generation and electrode projection from native brain space into common brain space for group analysis. Additionally, iEEGview implements methods for brain shift correction, visual location inspection on MRI slices and computation of certainty index in anatomical label assignment. Main results. All the introduced functions of iEEGview work reliably and successfully, and are tested by images from 28 human subjects implanted with depth and/or subdural electrodes. Significance. iEEGview is the first public Matlab GUI-based software for intracranial electrode localization and visualization that holds integrated capabilities together within one pipeline. iEEGview promotes convenience and efficiency for the localization process, provides rich localization information for further analysis and offers solutions for addressing raised technical challenges. Therefore, it can serve as a useful tool in facilitating iEEG studies.

Real‐time dense mapping for online processing and navigation

AbstractAutonomous robots require accurate localizations and dense mappings for motion planning. We consider the navigation scenario where the dense representation of the robot surrounding must be immediately available, and require that the system is capable of an instantaneous map correction if a loop closure is detected by the localization module. To satisfy the real‐time processing requirement of online robotics applications, our presented system bounds the algorithmic complexity of the localization pipeline by restricting the number of variables to be optimized at each time instant. A dense map representation along with a local dense map reconstruction strategy is also proposed. Despite the limits that are imposed by the real‐time requirement and planning safety, the mapping quality of our method is comparable to other competitive methods. For implementations, we additionally introduce a few engineering considerations, such as the system architecture, the variable initialization, the memory management, the image processing, and so forth, to improve the system performance. Extensive experimental validations of our presented system are performed on the KITTI and NewCollege datasets, and through an online experiment around the Hong Kong University of Science and Technology (HKUST) university campus. We release our implementation as open‐source robot operating system (ROS) packages for the benefit of the community.

Fusing the RGBD SLAM with Wheel Odometry

This paper deals with data fusion of existing visual SLAM algorithm and wheel odometry. The result of this connection is the possibility of suppressing measurement error of each position estimation method and creating more accurate 3D model of the examined environment. We have made a brief review of existing visual SLAM projects that are available in open-source domain and as the best option we have choose the Elastic Fusion project and we enriched the existing localization pipeline with another data source that is realized by our high-precision odometry vehicle frame. The modified technique has been validated on three different scenarios. The result is the more robust SLAM algorithm and better precision 3D model of sensed environment compared to original one.

Accurate Collaborative Globally-Referenced Digital Mapping with Standard GNSS

Exchange of location and sensor data among connected and automated vehicles will demand accurate global referencing of the digital maps currently being developed to aid positioning for automated driving. This paper explores the limit of such maps’ globally-referenced position accuracy when the mapping agents are equipped with low-cost Global Navigation Satellite System (GNSS) receivers performing standard code-phase-based navigation, and presents a globally-referenced electro-optical simultaneous localization and mapping pipeline, called GEOSLAM, designed to achieve this limit. The key accuracy-limiting factor is shown to be the asymptotic average of the error sources that impair standard GNSS positioning. Asymptotic statistics of each GNSS error source are analyzed through both simulation and empirical data to show that sub-50-cm accurate digital mapping is feasible in the horizontal plane after multiple mapping sessions with standard GNSS, but larger biases persist in the vertical direction. GEOSLAM achieves this accuracy by (i) incorporating standard GNSS position estimates in the visual SLAM framework, (ii) merging digital maps from multiple mapping sessions, and (iii) jointly optimizing structure and motion with respect to time-separated GNSS measurements.