Truncated Signed Distance Function Research Articles

3D Reconstruction of Ancient Buildings Using UAV Images and Neural Radiation Field with Depth Supervision

The 3D reconstruction of ancient buildings through inclined photogrammetry finds a wide range of applications in surveying, visualization and heritage conservation. Unlike indoor objects, reconstructing ancient buildings presents unique challenges, including the slow speed of 3D reconstruction using traditional methods, the complex textures of ancient structures and geometric issues caused by repeated textures. Additionally, there is a hash conflict problem when rendering outdoor scenes using neural radiation fields. To address these challenges, this paper proposes a 3D reconstruction method based on depth-supervised neural radiation fields. To enhance the representation of the geometric neural network, the addition of a truncated signed distance function (TSDF) supplements the existing signed distance function (SDF). Furthermore, the neural network’s training is supervised using depth information, leading to improved geometric accuracy in the reconstruction model through depth data obtained from sparse point clouds. This study also introduces a progressive training strategy to mitigate hash conflicts, allowing the hash table to express important details more effectively while reducing feature overlap. The experimental results demonstrate that our method, under the same number of iterations, produces images with clearer structural details, resulting in an average 15% increase in the Peak Signal-to-Noise Ratio (PSNR) value and a 10% increase in the Structural Similarity Index Measure (SSIM) value. Moreover, our reconstruction model produces higher-quality surface models, enabling the fast and highly geometrically accurate 3D reconstruction of ancient buildings.

Bio-inspired affordance learning for 6-DoF robotic grasping: A transformer-based global feature encoding approach

The 6-Degree-of-Freedom (6-DoF) robotic grasping is a fundamental task in robot manipulation, aimed at detecting graspable points and corresponding parameters in a 3D space, i.e affordance learning, and then a robot executes grasp actions with the detected affordances. Existing research works on affordance learning predominantly focus on learning local features directly for each grid in a voxel scene or each point in a point cloud scene, subsequently filtering the most promising candidate for execution. Contrarily, cognitive models of grasping highlight the significance of global descriptors, such as size, shape, and orientation, in grasping. These global descriptors indicate a grasp path closely tied to actions. Inspired by this, we propose a novel bio-inspired neural network that explicitly incorporates global feature encoding. In particular, our method utilizes a Truncated Signed Distance Function (TSDF) as input, and employs the recently proposed Transformer model to encode the global features of a scene directly. With the effective global representation, we then use deconvolution modules to decode multiple local features to generate graspable candidates. In addition, to integrate global and local features, we propose using a skip-connection module to merge lower-layer global features with higher-layer local features. Our approach, when tested on a recently proposed pile and packed grasping dataset for a decluttering task, surpassed state-of-the-art local feature learning methods by approximately 5% in terms of success and declutter rates. We also evaluated its running time and generalization ability, further demonstrating its superiority. We deployed our model on a Franka Panda robot arm, with real-world results aligning well with simulation data. This underscores our approach’s effectiveness for generalization and real-world applications.

T‐ESVO: Improved Event‐Based Stereo Visual Odometry via Adaptive Time‐Surface and Truncated Signed Distance Function

The emerging event cameras have the potential to be an excellent complement for standard cameras within various visual tasks, especially in illumination‐changing environments or situations requiring high‐temporal resolution. Herein, an event‐based stereo visual odometry (VO) system via adaptive time‐surface (TS) and truncated signed distance function (TSDF), namely, T‐ESVO, is proposed . The system consists of three carefully designed components, including the event processing unit, the mapping unit, and the tracking unit. Specifically, the event processing unit adopts a novel spatial–temporal adaptive TS that can deal with different camera motions in various environments. The mapping unit introduces the TSDF to describe the 3D representation of environments and achieves depth estimation based on the global historical depth information contained in the environmental TSDF description. The tracking unit achieves the 6‐DoF pose estimation through an 3D–2D registration method based on the left/right TS selection mechanism and the depth point selection mechanism. The effectiveness and robustness of the proposed system are evaluated on various datasets, and the experimental results show that T‐ESVO achieves good performance in both accuracy and robustness when compared with other state‐of‐the‐art event‐based stereo VO systems.

Incremental Dense Reconstruction From Monocular Video With Guided Sparse Feature Volume Fusion

Incrementally recovering 3D dense structures from monocular videos is of paramount importance since it enables various robotics and AR applications. Feature volumes have recently been shown to enable efficient and accurate incremental dense reconstruction without the need to first estimate depth, but they are not able to achieve as high of a resolution as depth-based methods due to the large memory consumption of high-resolution feature volumes. This letter proposes a real-time feature volume-based dense reconstruction method that predicts TSDF (Truncated Signed Distance Function) values from a novel sparsified deep feature volume, which is able to achieve higher resolutions than previous feature volume-based methods, and is favorable in large-scale outdoor scenarios where the majority of voxels are empty. An uncertainty-aware multi-view stereo (MVS) network is leveraged to infer initial voxel locations of the physical surface in a sparse feature volume. Then for refining the recovered 3D geometry, deep features are attentively aggregated from multiview images at potential surface locations, and temporally fused. Besides achieving higher resolutions than before, our method is shown to produce more complete reconstructions with finer detail in many cases. Extensive evaluations on both public and self-collected datasets demonstrate a very competitive real-time reconstruction result for our method compared to state-of-the-art reconstruction methods in both indoor and outdoor settings.

HandVoxNet++: 3D Hand Shape and Pose Estimation Using Voxel-Based Neural Networks.

3D hand shape and pose estimation from a single depth map is a new and challenging computer vision problem with many applications. Existing methods addressing it directly regress hand meshes via 2D convolutional neural networks, which leads to artifacts due to perspective distortions in the images. To address the limitations of the existing methods, we develop HandVoxNet++, i.e., a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner. The input to our network is a 3D voxelized-depth-map-based on the truncated signed distance function (TSDF). HandVoxNet++ relies on two hand shape representations. The first one is the 3D voxelized grid of hand shape, which does not preserve the mesh topology and which is the most accurate representation. The second representation is the hand surface that preserves the mesh topology. We combine the advantages of both representations by aligning the hand surface to the voxelized hand shape either with a new neural Graph-Convolutions-based Mesh Registration (GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach (NRGA++) which does not rely on training data. In extensive evaluations on three public benchmarks, i.e., SynHand5M, depth-based HANDS19 challenge and HO-3D, the proposed HandVoxNet++ achieves the state-of-the-art performance. In this journal extension of our previous approach presented at CVPR 2020, we gain 41.09% and 13.7% higher shape alignment accuracy on SynHand5M and HANDS19 datasets, respectively. Our method is ranked first on the HANDS19 challenge dataset (Task 1: Depth-Based 3D Hand Pose Estimation) at the moment of the submission of our results to the portal in August 2020.

Dense Incremental Metric-Semantic Mapping for Multiagent Systems via Sparse Gaussian Process Regression

In this article, we develop an online probabilistic metric-semantic mapping approach for mobile robot teams relying on streaming RGB-D observations. The generated maps contain full continuous distributional information about the geometric surfaces and semantic labels (e.g., chair, table, and wall). Our approach is based on online Gaussian process (GP) training and inference and avoids the complexity of GP classification by regressing a truncated signed distance function (TSDF) of the regions occupied by different semantic classes. Online regression is enabled through a sparse pseudo-point approximation of the GP posterior. To scale to large environments, we further consider spatial domain partitioning via a hierarchical tree structure with overlapping leaves. An extension to a multirobot setting is developed by having each robot execute its own online measurement update and then combine its posterior parameters via local weighted geometric averaging with those of its neighbors. This yields a distributed information processing architecture, in which the GP map estimates of all the robots converge to a common map of the environment while relying only on local one-hop communication. Our experiments demonstrate the effectiveness of the probabilistic metric-semantic mapping technique in 2-D and 3-D environments in both the single- and multirobot settings and in comparison to a deep TSDF neural network approach.

3D Reconstruction of Remote Sensing Mountain Areas with TSDF-Based Neural Networks

The remote sensing 3D reconstruction of mountain areas has a wide range of applications in surveying, visualization, and game modeling. Different from indoor objects, outdoor mountain reconstruction faces additional challenges, including illumination changes, diversity of textures, and highly irregular surface geometry. Traditional neural network-based methods that lack discriminative features struggle to handle the above challenges, and thus tend to generate incomplete and inaccurate reconstructions. Truncated signed distance function (TSDF) is a commonly used parameterized representation of 3D structures, which is naturally convenient for neural network computation and computer storage. In this paper, we propose a novel deep learning method with TSDF-based representations for robust 3D reconstruction from images containing mountain terrains. The proposed method takes in a set of images captured around an outdoor mountain and produces high-quality TSDF representations of the mountain areas. To address the aforementioned challenges, such as lighting variations and texture diversity, we propose a view fusion strategy based on reweighted mechanisms (VRM) to better integrate multi-view 2D features of the same voxel. A feature enhancement (FE) module is designed for providing better discriminative geometry prior in the feature decoding process. We also propose a spatial–temporal aggregation (STA) module to reduce the ambiguity between temporal features and improve the accuracy of the reconstruction surfaces. A synthetic dataset for reconstructing images containing mountain terrains is built. Our method outperforms the previous state-of-the-art TSDF-based and depth-based reconstruction methods in terms of both 2D and 3D metrics. Furthermore, we collect real-world multi-view terrain images from Google Map. Qualitative results demonstrate the good generalization ability of the proposed method.

A fully integrated system for hardware-accelerated TSDF SLAM with LiDAR sensors (HATSDF SLAM)

Simultaneous Localization and Mapping (SLAM) is one of the fundamental problems in autonomous robotics. Over the years, many approaches to solve this problem for 6D poses and 3D maps based on LiDAR sensors or depth cameras have been proposed. One of the main drawbacks of the solutions found in the literature is the required computational power and corresponding energy consumption. In this paper, we present an approach for LiDAR-based SLAM that maintains a global truncated signed distance function (TSDF) to represent the map. It is implemented on a System-On-Chip (SoC) with an integrated FPGA accelerator. The proposed system is able to track the position of state-of-the-art LiDARs in real time, while maintaining a global TSDF map that can be used to create a polygonal map of the environment. We show that our implementation delivers competitive results compared to state-of-the-art algorithms while drastically reducing the power consumption compared to classical CPU or GPU-based methods.

TOWARDS ROBUST INDOOR VISUAL SLAM AND DENSE RECONSTRUCTION FOR MOBILE ROBOTS

Abstract. Mobile robots are being increasingly employed in various indoor scenarios. The fundamental prerequisite is that the robot can reconstruct an accurate and complete map of the observed environment and estimate the track of its movements in this map. Current visual SLAM methods can perform this task reasonably well, but mostly in small spaces, such as a single room, and often tested in well-textured environments. In real-world applications of large indoor scenes, they lack robustness and fail to build a globally consistent map. To this end, we propose a novel system that can robustly solve the problem encountered by existing visual SLAM methods, such as weak texture and long-term drift. By combining information from a wheel odometer, the robot poses can be predicted smoothly in the absence of texture. The geometric cues are leveraged by aligning Truncated Signed Distance Function (TSDF) based submaps to minimize the long-term drift. To reconstruct a more complete and accurate dense map, we refine the sensor depth maps by taking advantage of color information and the optimization result of global bundle adjustment. As a result, the system can provide precise trajectory estimation and a globally consistent map for the downstream tasks. We validate the accuracy and robustness of the proposed method on both public and self-collected datasets and show the complementary nature of each module. Evaluation results based on high precision ground-truth show an improvement in the mean Absolute Trajectory Error (ATE) from 21 cm to 2 cm for the trajectory estimation, and the reconstructed map has a mean accuracy of 8 cm.

Real-Time Collision-Free Grasp Pose Detection With Geometry-Aware Refinement Using High-Resolution Volume

In this letter, we proposea novel vision-based grasp system for closed-loop 6-degrees of freedom grasping of unknown objects in cluttered environments. The key factor in our system is that we make the most of a geometry-aware scene representation based on a truncated signed distance function (TSDF) volume, which can handle multi-view observations from the vision sensor, provide comprehensive spatial information for the grasp pose detector, and allow collision checking to achieve a collision-free grasp pose. To eliminate the large computational burden caused by the volume-based data, a lightweight volume-point network (VPN) equipped with the Marching Cubes algorithm is proposed. This network predicts the point-wise grasp qualities for all the candidates in a single feed-forward operation with real-time performance, enabling the system to perform closed-loop grasping. Furthermore, a grasp pose refinement module is integrated to predict the pose residual based on the SDF observation of the gripper state in the TSDF volume. Extensive experiments show that the proposed method can achieve collision-free grasp detection with an efficiency of more than 30 Hz on a high-resolution volume. Furthermore, the model trained on only synthetic data achieves a 90.9&#x0025; grasp success rate on cluttered real-world scenes, significantly outperforming other baseline methods. The supplementary materials are available at <uri>https://sites.google.com/view/vpn-icra2022</uri>

TextureMe: High-Quality Textured Scene Reconstruction in Real Time

Three-dimensional (3D) reconstruction using an RGB-D camera has been widely adopted for realistic content creation. However, high-quality texture mapping onto the reconstructed geometry is often treated as an offline step that should run after geometric reconstruction. In this article, we propose TextureMe , a novel approach that jointly recovers 3D surface geometry and high-quality texture in real time. The key idea is to create triangular texture patches that correspond to zero-crossing triangles of truncated signed distance function (TSDF) progressively in a global texture atlas. Our approach integrates color details into the texture patches in parallel with the depth map integration to a TSDF. It also actively updates a pool of texture patches to adapt TSDF changes and minimizes misalignment artifacts that occur due to camera drift and image distortion. Our global texture atlas representation is fully compatible with conventional texture mapping. As a result, our approach produces high-quality textures without utilizing additional texture map optimization, mesh parameterization, or heavy post-processing. High-quality scenes produced by our real-time approach are even comparable to the results from state-of-the-art methods that run offline.

DFusion: Denoised TSDF Fusion of Multiple Depth Maps with Sensor Pose Noises.

The truncated signed distance function (TSDF) fusion is one of the key operations in the 3D reconstruction process. However, existing TSDF fusion methods usually suffer from the inevitable sensor noises. In this paper, we propose a new TSDF fusion network, named DFusion, to minimize the influences from the two most common sensor noises, i.e., depth noises and pose noises. To the best of our knowledge, this is the first depth fusion for resolving both depth noises and pose noises. DFusion consists of a fusion module, which fuses depth maps together and generates a TSDF volume, as well as the following denoising module, which takes the TSDF volume as the input and removes both depth noises and pose noises. To utilize the 3D structural information of the TSDF volume, 3D convolutional layers are used in the encoder and decoder parts of the denoising module. In addition, a specially-designed loss function is adopted to improve the fusion performance in object and surface regions. The experiments are conducted on a synthetic dataset as well as a real-scene dataset. The results prove that our method outperforms existing methods.

Dense point cloud map construction based on stereo VINS for mobile vehicles

Mobile vehicles require accurate localization and dense mapping for motion planning. In this paper, we propose a dense map construction algorithm based on a light-and-fast stereo visual-inertial navigation system (VINS). A tightly coupled nonlinear optimization method is used to calculate the position of adjacent keyframes. An optical flow tracking method fused with IMU information and ring matching constraints is used to improve the matching accuracy and speed of the feature points. In addition, we obtain the pose and depth values using the semi-global block matching (SGBM) method, which are used as the initial values of the depth filter to update the depth image and improve the convergence speed. Then, we further use the Truncated Signed Distance Function (TSDF) method to construct the dense map. We compare our algorithm with state-of-the-art algorithms on the EuRoc dataset and then compare the estimated depth image using the proposed algorithm and the point cloud construction with the probabilistic monocular dense reconstruction (REMODE). The experiments show that the proposed algorithm can obtain more accurate localization than VINS and OKVIS, as well as a faster tracking speed, a better depth map, a lower convergence time for the estimated image and a lower number of updated frames than REMODE.

Multi-Resolution 3D Mapping With Explicit Free Space Representation for Fast and Accurate Mobile Robot Motion Planning

With the aim of bridging the gap between high quality reconstruction and robot motion planning, we propose an efficient system that leverages the concept of adaptive-resolution volumetric mapping, which naturally integrates with the hierarchical decomposition of space in an octree data structure. Instead of a Truncated Signed Distance Function (TSDF), we adopt mapping of occupancy probabilities in log-odds representation, which allows to represent both surfaces, as well as the entire free, i.e. observed space, as opposed to unobserved space. We introduce a method for choosing resolution -on the fly- in real-time by means of a multi-scale max-min pooling of the input depth image. The notion of explicit free space mapping paired with the spatial hierarchy in the data structure, as well as map resolution, allows for collision queries, as needed for robot motion planning, at unprecedented speed. We quantitatively evaluate mapping accuracy, memory, runtime performance, and planning performance showing improvements over the state of the art, particularly in cases requiring high resolution maps.

Surface-driven Next-Best-View planning for exploration of large-scale 3D environments

Abstract In this paper, we propose a novel cluster-based informative path planning algorithm to simultaneously explore and inspect a large-scale unknown environment with an Unmanned Aerial Vehicle (UAV). Most of the existing methods address the surface inspection problem as a volume exploration problem, and consider that the surface has been scanned when the corresponding volume has been covered. Unfortunately, this approach may lead to inaccurate 3D models of the environment, and the UAV may not achieve global coverage. To overcome these critical limitations, we introduce a 3D reconstruction method based on TSDF (Truncated Signed Distance Function) mapping, which leverages the surfaces present in the environment to generate an informative exploration path for the UAV. A Probabilistic Roadmap planner, used to solve a TSP (Travelling Salesman Problem) over clusters of viewpoint configurations, ensures that the resulting 3D model is accurate and complete. Two challenging structures (a power plant and the Statue of Liberty) have been chosen to conduct realistic numerical experiments with a quadrotor UAV. Our results provide evidence that the proposed method is effective and robust.

Semantic scene completion with dense CRF from a single depth image

Abstract Scene understanding is a significant research topic in computer vision, especially for robots to understand their environment intelligently. Semantic scene segmentation can help robots to identify the objects that are present in their surroundings, while semantic scene completion can enhance the ability of the robot to infer the object shape, which is pivotal for several high-level tasks. With dense Conditional Random Field (CRF), one key issue is how to construct the long-range interactions between nodes with Gaussian pairwise potentials. Another issue is what effective and efficient inference algorithms can be adapted to resolve the optimization. In this paper, we focus on semantic scene segmentation and completion optimization technology simultaneously using dense CRF based on a single depth image only. Firstly, we convert the single depth image into different down-sampled Truncated Signed Distance Function (TSDF) or flipped TSDF voxel formats, and formulate the pairwise potentials terms with such a representation. Secondly, we use the output results of an end-to-end 3D convolutional neural network named SSCNet to obtain the unary potentials. Finally, we pursue the efficiency of different CRF inference algorithms (the mean-field inference, the negative semi-definite specific difference of convex relaxation, the proximal minimization of linear programming and its variants, etc.). The proposed dense CRF and inference algorithms are evaluated on three different datasets (SUNCG, NYU, and NYUCAD). Experimental results demonstrate that the voxel-level intersection over union (IoU) of predicted voxel’s semantic and completion can reach to state-of-the-art. Specifically, for voxel semantic segmentation, the highest IoU improvements are 2.6%, 1.3%, 3.1%, and for scene completion, the highest IoU improvements are 2.5%, 3.7%, 5.4%, respectively for SUNCG, NYU, and NYUCAD datasets.

Real-Time Large-Scale Dense Mapping with Surfels.

Real-time dense mapping systems have been developed since the birth of consumer RGB-D cameras. Currently, there are two commonly used models in dense mapping systems: truncated signed distance function (TSDF) and surfel. The state-of-the-art dense mapping systems usually work fine with small-sized regions. The generated dense surface may be unsatisfactory around the loop closures when the system tracking drift grows large. In addition, the efficiency of the system with surfel model slows down when the number of the model points in the map becomes large. In this paper, we propose to use two maps in the dense mapping system. The RGB-D images are integrated into a local surfel map. The old surfels that reconstructed in former times and far away from the camera frustum are moved from the local map to the global map. The updated surfels in the local map when every frame arrives are kept bounded. Therefore, in our system, the scene that can be reconstructed is very large, and the frame rate of our system remains high. We detect loop closures and optimize the pose graph to distribute system tracking drift. The positions and normals of the surfels in the map are also corrected using an embedded deformation graph so that they are consistent with the updated poses. In order to deal with large surface deformations, we propose a new method for constructing constraints with system trajectories and loop closure keyframes. The proposed new method stabilizes large-scale surface deformation. Experimental results show that our novel system behaves better than the prior state-of-the-art dense mapping systems.