3D Feature Points Research Articles

A Method for Tracking an Underwater Pipeline from Stereo Images Using an Autonomous Underwater Vehicle

The problem of tracking an underwater pipeline (UP) is considered in the context of an inspection mission. It is assumed that the diameter of the cylindrical pipe is known. To solve this problem, a tracking method is proposed, based on searching and calculating the center line of the UP, and determining the relative position of the autonomous underwater robot (AUV) and UP in the coordinate system of the AUV camera. Unlike well-known analogues, in which the solution is based on recognizing and constructing the boundaries of the UP in images, the proposed method searches for the true position of the center line of the UP on a set of possible options by checking their veracity. Possible options for the spatial position of the search centerline of the current UP section are generated by varying the direction of the beam in the horizontal and vertical plane. The starting point of the ray is the end point of the centerline of the previous section. The veracity criterion is to check that the 3D point features constructed in the scene belong to the cylindrical surface of the UP. The generation and matching of pointfeatures in images of a stereo pair is carried out using the SURF detector. The choice of the correct direction of the center line of the current UP section is ultimately made by voting. The end point of the center line of the current section is determined taking into account the calculated common visibility area with the previous section of the UP. To evaluate the effectiveness of the method, computational experiments were carried out on virtual scenes. The effectiveness was assessed by the accuracy of UP localization (in the AUV coordinate system) and by the speed of calculations in comparison with: a) the first version of this method, based on the use of a vectorized form of images; b) analogues using the Canny and Hough Transform detectors. The stability of the method to the accumulation of navigation accuracy errors during long-term tracking of UPs was also assessed.

Six-Degree-of-Freedom Pose Estimation Method for Multi-Source Feature Points Based on Fully Convolutional Neural Network

An object’s six-degree-of-freedom (6DoF) pose information has great importance in various fields. Existing methods of pose estimation usually detect two-dimensional (2D)-three-dimensional (3D) feature point pairs, and directly estimates the pose information through Perspective-n-Point (PnP) algorithms. However, this approach ignores the spatial association between pixels, making it difficult to obtain high-precision results. In order to apply pose estimation based on deep learning methods to real-world scenarios, we hope to design a method that is robust enough in more complex scenarios. Therefore, we introduce a method for 3D object pose estimation from color images based on farthest point sampling (FPS) and object 3D bounding box. This method detects the 2D projection of 3D feature points through a convolutional neural network, matches it with the 3D model of the object, and then uses the PnP algorithm to restore the feature point pair to the object pose. Due to the global nature of the bounding box, this approach can be considered effective even in partially occluded or complex environments. In addition, we propose a heatmap suppression method based on weighted coordinates to further improve the prediction accuracy of feature points and the accuracy of the PnP algorithm in solving the pose position. Compared with other algorithms, this method has higher accuracy and better robustness. Our method yielded 93.8% of the ADD(-s) metrics on the Linemod dataset and 47.7% of the ADD(-s) metrics on the Occlusion Linemod dataset. These results show that our method is more effective than existing methods in pose estimation of large objects.

Measuring dynamic deformation of a rotating blade by point cloud matching algorithm

In this paper, a point cloud matching algorithm is proposed for the scenario of measuring the dynamic deformation field of a rotating blade. Unlike conventional scheme based on Digital Image Correlation (DIC), the essence of this scheme is to reconstruct individual 3D feature points, instead of a group of feature points in a finite-size subset, on the measurement surface and then track their relative offset from reference baseline. The principles of 3D feature-point reconstruction and straddle-frame tracking are similar to those being used in the algorithm of Particle Tracking Velocimetry (PTV); therefore, no additional tags on feature points to facilitate the perspective matching and tracking is needed. Experiment test shows that this scheme works well in the scenario of high feature-point density (0.026 points per pixel), which cannot be achieved by existing feature-point-tagging methods. Both the rigid motion of the rotating blade and its dynamic deformation field can be reliably obtained. Comparing to DIC-based scheme, the benefit of the present proposed scheme includes the improvement of spatial resolution, the insensitivity to feature-point density, as well as the avoidance of outliers due to insufficient texture information.

PNeRFLoc: Visual Localization with Point-Based Neural Radiance Fields

Due to the ability to synthesize high-quality novel views, Neural Radiance Fields (NeRF) has been recently exploited to improve visual localization in a known environment. However, the existing methods mostly utilize NeRF for data augmentation to improve the regression model training, and their performances on novel viewpoints and appearances are still limited due to the lack of geometric constraints. In this paper, we propose a novel visual localization framework, i.e., PNeRFLoc, based on a unified point-based representation. On one hand, PNeRFLoc supports the initial pose estimation by matching 2D and 3D feature points as traditional structure-based methods; on the other hand, it also enables pose refinement with novel view synthesis using rendering-based optimization. Specifically, we propose a novel feature adaption module to close the gaps between the features for visual localization and neural rendering. To improve the efficacy and efficiency of neural rendering-based optimization, we also developed an efficient rendering-based framework with a warping loss function. Extensive experiments demonstrate that PNeRFLoc performs the best on the synthetic dataset when the 3D NeRF model can be well learned, and significantly outperforms all the NeRF-boosted localization methods with on-par SOTA performance on the real-world benchmark localization datasets. Project webpage: https://zju3dv.github.io/PNeRFLoc/.

Facial augmented reality based on hierarchical optimization of similarity aspect graph

BackgroundThe existing face matching method requires a point cloud to be drawn on the real face for registration, which results in low registration accuracy due to the irregular deformation of the patient's skin that makes the point cloud have many outlier points. MethodsThis work proposes a non-contact pose estimation method based on similarity aspect graph hierarchical optimization. The proposed method constructs a distance-weighted and triangular-constrained similarity measure to describe the similarity between views by automatically identifying the 2D and 3D feature points of the face. A mutual similarity clustering method is proposed to construct a hierarchical aspect graph with 3D pose as nodes. A Monte Carlo tree search strategy is used to search the hierarchical aspect graph for determining the optimal pose of the facial 3D model, so as to realize the accurate registration of the facial 3D model and the real face. ResultsThe proposed method was used to conduct accuracy verification experiments on the phantoms and volunteers, which were compared with four advanced pose calibration methods. The proposed method obtained average fusion errors of 1.13 ± 0.20 mm and 0.92 ± 0.08 mm in head phantom and volunteer experiments, respectively, which exhibits the best fusion performance among all comparison methods. ConclusionsOur experiments proved the effectiveness of the proposed pose estimation method in facial augmented reality.

GA-RCNN:Graph self-attention feature extraction for 3D object detection

In recent years, 3D object detection based on LiDAR point clouds is a key component of autonomous driving. In pursuit of enhancing the accuracy of 3D point cloud feature extraction and point cloud detection, this paper introduces a novel 3D object detection model, termed as Graph Self-Attention-RCNN (GA-RCNN). This model is designed to integrate voxel information and point location information, enhancing the quality of 3D object proposals while maintaining contextual accuracy. The first stage rectifies the previous approach that relied on local features for preselected boxes, overlooking crucial global contextual information. An improved method is suggested in this work, utilizing BEV to capture long-range dependencies via a cross-attention mechanism. The second stage addresses the overreliance on local neighborhood point feature extraction. The Graph Self-Attention Pooling method is proposed, characterized by its dynamic computation of contribution weights for inputs. This enhances the model’s flexibility and generalization performance. Extensive evaluations on KITTI and Waymo datasets demonstrate GA-RCNN’s superior accuracy compared to other methods, affirming its efficacy in 3D object detection.

MVS‐SLAM: Enhanced multiview geometry for improved semantic RGBD SLAM in dynamic environment

AbstractSimultaneous Localization and Mapping (SLAM) is a crucial technology for intelligent mobile robots to operate successfully in unknown environments. While many excellent SLAM systems have been developed in recent years, most assume that the environment is static, resulting in poor performance in dynamic environments. To address this limitation, we propose multiview stereo (MVS)‐SLAM (MVS‐SLAM), a real‐time semantic RGBD SLAM system with improved‐multiview geometry, built on the RGBD mode of ORB‐SLAM3. MVS‐SLAM tightly integrates semantic and geometric information to tackle the challenges posed by dynamic scenes. To meet the real‐time requirements, the semantic module leverages the latest and fastest object detection network, YOLOv7, to provide semantic prior knowledge for the geometric module. We introduce a novel geometric constraint method that capitalizes on depth images and semantic information to recover three‐dimensional (3D) feature points and initial camera pose. We use a 3D coordinate error threshold to identify dynamic points and remove them using the K‐means clustering algorithm. This approach effectively reduces the impact of dynamic points. We validate MVS‐SLAM using challenging dynamic sequences from the TUM data set, demonstrating that it significantly improves localization accuracy and system robustness in all types of dynamic environments.

SCE-SLAM: a real-time semantic RGBD SLAM system in dynamic scenes based on spatial coordinate error

Simultaneous localization and mapping (SLAM) is one of the prerequisite technologies for intelligent mobile robots to accomplish various tasks in unknown environments. In recent years, many excellent SLAM systems have emerged, but most of them have a basic assumption that the environment is static, which results in their poor performance in dynamic environments. To solve this problem, this paper presents SCE-SLAM: a novel real-time semantic RGB-D SLAM system that is built on the RGB-D mode of ORB-SLAM3. SCE-SLAM tightly combines semantic and geometric information. Considering the real-time requirements, the semantic module provides semantic prior knowledge for the geometric module using the latest and fastest object detection network YOLOv7. Then, a new geometric constraint method is proposed to filter dynamic feature points. This method takes full advantage of depth images and semantic information to recover three-dimensional (3D) feature points and the initial camera pose. A 3D coordinate error is used as a threshold, and SCE-SLAM removes dynamic points using the K-means clustering algorithm. In this way, SCE-SLAM effectively reduces the impact of dynamic points. Furthermore, we validate SCE-SLAM with challenging dynamic sequences of the TUM dataset. The results demonstrate that SCE-SLAM significantly improves the localization accuracy and system robustness in all kinds of dynamic environments.

Environment Modeling Method for Intelligent Robots Based on Binocular Vision

With the development of technology, the demand for intelligent mobile robots is growing steadily. Environment perception and modeling, are the key elements to achieve robot intelligence. Therefore, binocular stereo vision sensor is used as the core device to perform operations such as binocular camera calibration, obtaining binocular camera internal/external parameters, and distortion correction. On this basis, a multi-dimensional and all-round environment perception system is constructed, and the dynamic target detection results and 3D scene map reconstruction results are displayed dynamically in real time. The experimental results of binocular camera calibration show that the re-projection error lies in the sub-pixel interval, and the image distortion caused by the camera is basically eliminated to meet the experimental expectation of distortion correction; the experimental results of environment modeling show that the mean value of deviation between the object representation value of 3D scene map and the actual object real value is -0.041m, among which the minimum deviation value is 0.001m and the maximum deviation value is 0.532m, in The experimental results are excellent in the tasks of 3D scene map reconstruction and feature point matching. In addition, the storage space of 3D scene map is significantly reduced, and the real-time performance of the system is significantly better than the traditional method.

Cross-Modal Contrastive Learning for Domain Adaptation in 3D Semantic Segmentation

Domain adaptation for 3D point cloud has attracted a lot of interest since it can avoid the time-consuming labeling process of 3D data to some extent. A recent work named xMUDA leveraged multi-modal data to domain adaptation task of 3D semantic segmentation by mimicking the predictions between 2D and 3D modalities, and outperformed the previous single modality methods only using point clouds. Based on it, in this paper, we propose a novel cross-modal contrastive learning scheme to further improve the adaptation effects. By employing constraints from the correspondences between 2D pixel features and 3D point features, our method not only facilitates interaction between the two different modalities, but also boosts feature representations in both labeled source domain and unlabeled target domain. Meanwhile, to sufficiently utilize 2D context information for domain adaptation through cross-modal learning, we introduce a neighborhood feature aggregation module to enhance pixel features. The module employs neighborhood attention to aggregate nearby pixels in the 2D image, which relieves the mismatching between the two different modalities, arising from projecting relative sparse point cloud to dense image pixels. We evaluate our method on three unsupervised domain adaptation scenarios, including country-to-country, day-to-night, and dataset-to-dataset. Experimental results show that our approach outperforms existing methods, which demonstrates the effectiveness of the proposed method.

EXPLAINING AI: UNDERSTANDING DEEP LEARNING MODELS FOR HERITAGE POINT CLOUDS

Abstract. Deep Learning has been pivotal in many real-world applications (e.g., autonomous driving, medicine and retail). With the wide availability of consumer-grade depth sensors, acquiring 3D data has become more affordable and effective, and many 3D datasets are currently publicly available. 3D data provides a great opportunity for a better comprehension of the surrounding environment for machines. There is a growing need for innovative methods for the treatment and analysis of point clouds and for their classification. The complex hidden layers, which are at the basis of deep neural networks (DNNs), make it difficult to interpret these models, that up to a few years ago DNNs were considered and treated as black box operators. Still, with their increasing popularity, making them explainable and interpretable has become mandatory. A lot of efforts were devoted to developing an Explainable Artificial Intelligence (XAI) framework for explaining DNNs decisions with 2D data, while only a few studies have attempted to investigate the explainability of 3D DNNs and, even more, heritage scenarios. To overcome these limitations, it was proposed the BubblEX framework: a novel multimodal fusion framework to learn the 3D point features. In our work, BubblEX has been exploited to understand the decisions taken by DNNs for heritage point clouds. The approach has been applied to a Digital Cultural Heritage Dataset, which is publicly available: the ArCH (Architectural Cultural Heritage) Dataset.

A non-coplanar high-precision calibration method for cameras based on an affine coordinate correction model

Traditional non-coplanar calibration methods such as Tsai’s method have many problems, such as insufficient calibration accuracy, inconvenient operation, inaccurate models, etc. This paper proposes a new high-precision non-coplanar calibration method that aims to solve these problems. Like Tsai’s method, the proposed calibration method utilizes a one-dimensional displacement stage and a two-dimensional plane target to generate a virtual 3D feature point sequence. As an improvement, an affine coordinate correction model is applied to ensure the accuracy and orthogonality of the obtained virtual 3D coordinates. A novel and accurate camera calibration model is further established. Compared with Tsai’s model, which uses a radial alignment constraint and ignores the orthonormal constraint of the rotation matrix, the proposed calibration model fully considers the degrees of freedom of the camera’s parameters to be calibrated, as well as the lens’s nonlinear distortion parameters. More accurate analytical solutions of intrinsic and extrinsic parameters can be obtained with the proposed calibration model. Finally, a novel high-precision non-coplanar calibration method is proposed based on the proposed calibration model. The reprojection experiment proves that the calibration accuracy of this calibration method is better than that of Tsai’s and Zhang’s calibration methods under the same calibration conditions. As a supplement, a novel binocular camera system extrinsic parameter calibration method with known intrinsic parameters is proposed. With accurate intrinsic and extrinsic parameters, the binocular camera system’s relative measurement accuracy could be within 1/10 000. Overall, this method can be used in experimental and industrial applications that require high-precision calibration parameters.

Design of Unilateral Complete Presurgical Nasoalveolar Molding (PNAM) Corrector Based on Feature Points Extraction of Complex 3D Surface

Cleft lip and palate is a congenital maxillofacial deformity. Unilateral complete cleft lip and palate is one of the most common clinical types. Nasal alveolar molding (PNAM) is a recognized strategy for the treatment of cleft lip and palate. However, the current design of PNAM devices mainly relies on the subjective experience of doctors. The purpose of this paper is to describe the design and manufacture of a new computer-aided design appliance, which can be applied to the presurgical nasoalveolar molding of unilateral complete cleft lip and palate, eliminate individual differences, and improve production efficiency. In this paper, seven feature points on the healthy side and the affected side are extracted by the method of Gaussian curvature and ridge line extraction, and the healthy side rotation and built-in model are designed by using these seven feature points, which can quickly generate eight treatment stages of PNAM. The correction effects of the PNAM appliance designed in this paper were compared with the original maxillary model and the clinical PNAM appliance (hand-made by subjective experience) from the aspects of alveolar fissure width and symmetry. The PNAM appliance designed in this paper can effectively improve the symmetry of patients with unilateral complete cleft lip and palate (morphological similarity: t = 3.250, p ≤ 0.01; length similarity: t = 1.559, p = 0.150) and reduce the width of alveolar cleft (t = 8.330, p < 0.01). This can fully achieve the therapeutic effect of PNAM appliances prepared by experienced doctors and is more efficient. The method based on complex 3D surface feature point extraction can provide the basis for the design and evaluation of a unilateral complete PNAM correction model, improve the design and production efficiency of unilateral complete cleft lip and palate appliance, eliminate the design problems caused by individual differences, and reduce the burden of doctors.

Vision geometry-based UAV flocking

AbstractA distributed UAV (unmanned aerial vehicle) flocking control method based on vision geometry is proposed, in which only monocular RGB (red, green, blue) images are used to estimate the relative positions and velocities between drones. It does not rely on special visual markers and external infrastructure, nor does it require inter-UAV communication or prior knowledge of UAV size. This method combines the advantages of deep learning and classical geometry. It adopts a deep optical flow network to estimate dense matching points between two consecutive images, uses segmentation technology to classify these matching points into background and specific UAV, and then maps the classified matching points to Euclidean space based on the depth map information. In 3D matching points, also known as 3D feature point pairs, each of their classifications is used to estimate the rotation matrix, translation vector, velocity of the corresponding UAV, as well as the relative position between drones, based on RANSAC and least squares method. On this basis, a flocking control model is constructed. Experimental results in the Microsoft Airsim simulation environment show that in all evaluation metrics, our method achieves almost the same performance as the UAV flocking algorithm based on ground truth cluster state.

Image-based geometric digital twinning for stone masonry elements

We present an image-based pipeline for generating geometrical digital twins (GDTs) of stone masonry elements with detail down to the stone level. For this purpose, we acquire RGB images of the individual stones and of the wall during the construction phase. In our framework, we use structure from motion (SfM) to first generate 3D source and destination models, which are then registered to form the GDT through non-linear least squares and 2D point feature correspondences detected on the SfM images. This method contrasts with traditional techniques that register point clouds using 3D point descriptors. Because of the robustness of image feature descriptors, we found that using 2D instead of 3D point features facilitates the automation of the GDT generation. To benchmark our algorithm, we compared the results through an Euclidean–distance-based proposed metric with a known 3D textured model from which images were synthetically generated. We show the robustness and feasibility of our method for full size elements, wherein GDTs were generated for dry-stone and stone-mortar systems. This study allows researchers to produce accurate representations of the 3D geometry of walls built for experimental research, reducing therefore uncertainties related to the stone size, shape and arrangement to a minimum when comparing 3D numerical simulations of these walls to experimental results. Codes and data sets are publicly available (https://github.com/eesd-epfl/stone_masonry_GDT and https://doi.org/10.5281/zenodo.7266587).

JPV-Net: Joint Point-Voxel Representations for Accurate 3D Object Detection

Voxel and point representations are widely applied in recent 3D object detection tasks from LiDAR point clouds. Voxel representations contribute to efficiently and rapidly locating objects, whereas point representations are capable of describing intra-object spatial relationship for detection refinement. In this work, we aim to exploit the strengths of both two representations, and present a novel two-stage detector, named Joint Point-Voxel Network (JPV-Net). Specifically, our framework is equipped with a Dual Encoders-Fusion Decoder, which consists of the dual encoders to extract voxel features of sketchy 3D scenes and point features rich in geometric context, respectively, and the Feature Propagation Fusion (FP-Fusion) decoder to attentively fuse them from coarse to fine. By making use of the advantages of these features, the refinement network can effectively eliminate false detection and provide better accuracy. Besides, to further develop the perception characteristics of voxel CNN and point backbone, we design two novel intersection-over-union (IoU) estimation modules for proposal generation and refinement, both of which can alleviate the misalignment between the localization and the classification confidence. Extensive experiments on the KITTI dataset and ONCE dataset demonstrate that our proposed JPV-Net outperforms other state-of-the-art methods with remarkable margins.

DiT-SLAM: Real-Time Dense Visual-Inertial SLAM with Implicit Depth Representation and Tightly-Coupled Graph Optimization.

Recently, generating dense maps in real-time has become a hot research topic in the mobile robotics community, since dense maps can provide more informative and continuous features compared with sparse maps. Implicit depth representation (e.g., the depth code) derived from deep neural networks has been employed in the visual-only or visual-inertial simultaneous localization and mapping (SLAM) systems, which achieve promising performances on both camera motion and local dense geometry estimations from monocular images. However, the existing visual-inertial SLAM systems combined with depth codes are either built on a filter-based SLAM framework, which can only update poses and maps in a relatively small local time window, or based on a loosely-coupled framework, while the prior geometric constraints from the depth estimation network have not been employed for boosting the state estimation. To well address these drawbacks, we propose DiT-SLAM, a novel real-time Dense visual-inertial SLAM with implicit depth representation and Tightly-coupled graph optimization. Most importantly, the poses, sparse maps, and low-dimensional depth codes are optimized with the tightly-coupled graph by considering the visual, inertial, and depth residuals simultaneously. Meanwhile, we propose a light-weight monocular depth estimation and completion network, which is combined with attention mechanisms and the conditional variational auto-encoder (CVAE) to predict the uncertainty-aware dense depth maps from more low-dimensional codes. Furthermore, a robust point sampling strategy introducing the spatial distribution of 2D feature points is also proposed to provide geometric constraints in the tightly-coupled optimization, especially for textureless or featureless cases in indoor environments. We evaluate our system on open benchmarks. The proposed methods achieve better performances on both the dense depth estimation and the trajectory estimation compared to the baseline and other systems.

IMU-Assisted Online Video Background Identification.

Distinguishing between dynamic foreground objects and a mostly static background is a fundamental problem in many computer vision and computer graphics tasks. This paper presents a novel online video background identification method with the assistance of inertial measurement unit (IMU). Based on the fact that the background motion of a video essentially reflects the 3D camera motion, we leverage IMU data to realize a robust camera motion estimation for identifying background feature points by only investigating a few historical frames. We observe that the displacement of the 2D projection of a scene point caused by camera rotation is depth-invariant, and the rotation estimation by using IMU data can be quite accurate. We thus propose to analyze 2D feature points by decomposing the 2D motion into two components: rotation projection and translation projection. In our method, after establishing the 3D camera rotations, we generate the depth-relevant 2D feature point movement induced by the camera 3D translation. Then, by examining the disparity between inter-frame offset and the projection of estimated 3D camera motion, we can identify the background feature points. In the experiments, our online method is able to run at 30FPS with only 1 frame latency and outperforms state-of-the-art background identification and other relevant methods. Our method directly leads to a better camera motion estimation, which is beneficial to many applications like online video stabilization, SLAM, image stitching, etc.

BubblEX: An Explainable Deep Learning Framework for Point-Cloud Classification

Point clouds data are nowadays one of the major data sources for describing our environment. Recently, deep architectures have been proposed as a key step in understanding and retrieving semantic information. Despite the great contribution of deep learning in this field, the explainability of these models for 3D data is still fairly unexplored. Explainability, identified as a potential weakness of Deep Neural Networks (DNNs), can help researchers against skepticism, considering that these models are far from being self-explanatory. Although literature provides many examples on the exploitation of Explainable Artificial Intelligence (XAI) approaches with 2D data, only a few studies have attempted to investigate it for 3D DNNs. To overcome these limitations, it is here proposed BubblEX, a novel multimodal fusion framework to learn the 3D point features. BubblEX framework comprises two stages: “Visualization Module” for the visualisation of features learned from the network in its hidden layers and “Interpretability Module”, which aims at describing how the neighbour points are involved in the features extraction. For our experiments, it has been used Dynamic Graph CNN (DGCNN), trained on Modelnet40 dataset. The developed framework extends a method for obtaining saliency maps from image data, to deal with 3D point cloud data, allowing the analysis, comparison and contrasting of multiple features. Besides, it permits the generation of visual explanations from any DNN-based network for 3D point cloud classification without requiring architectural changes or re-training. Our findings will be extremely useful for both scientists and non-experts in understanding and improving future AI-based models.

A new head pose tracking method based on stereo visual SLAM

Real-time and reliable head pose tracking is the basis of human–computer interaction and face analysis applications. Aiming at the problems of accuracy and real time performance in current tracking method, a new head pose tracking method based on stereo visual SLAM is proposed in this paper. The sparse head map is constructed based on ORB feature points extraction and stereo matching, then the 3D-2D matching relations between 3D mappoints and 2D feature points are obtained by projection matching. Finally, the camera pose solved by the Bundle Adjustment is converted to head pose, which realizes the tracking of head pose. The experimental results show that this method can obtain high precise head pose. The mean errors of three Euler angles are all less than 1°. Therefore, the proposed head pose tracking method can track and estimate precise head pose in real time under smooth background.