Online 3D Reconstruction Research Articles

GP-Recon: Online Monocular Neural 3D Reconstruction with Geometric Prior.

High-fidelity online 3D scene reconstruction from monocular videos continues to be challenging, especially for coherent and fine-grained geometry reconstruction. The previous learning-based online 3D reconstruction approaches with neural implicit representations have shown a promising ability for coherent scene reconstruction, but often fail to consistently reconstruct fine-grained geometric details during online reconstruction. This paper presents a new on-the-fly monocular 3D reconstruction approach, named GP-Recon, to perform high-fidelity online neural 3D reconstruction with fine-grained geometric details. We incorporate geometric prior (GP) into a scene's neural geometry learning to better capture its geometric details and, more importantly, propose an online volume rendering optimization to reconstruct and maintain geometric details during the online reconstruction task. The extensive comparisons with state-of-the-art approaches show that our GP-Recon consistently generates more accurate and complete reconstruction results with much better fine-grained details, both quantitatively and qualitatively.

Iterative Online 3D Reconstruction from RGB Images

3D reconstruction is the computer vision task of reconstructing the 3D shape of an object from multiple 2D images. Most existing algorithms for this task are designed for offline settings, producing a single reconstruction from a batch of images taken from diverse viewpoints. Alongside reconstruction accuracy, additional considerations arise when 3D reconstructions are used in real-time processing pipelines for applications such as robot navigation or manipulation. In these cases, an accurate 3D reconstruction is already required while the data gathering is still in progress. In this paper, we demonstrate how existing batch-based reconstruction algorithms lead to suboptimal reconstruction quality when used for online, iterative 3D reconstruction and propose appropriate modifications to the existing Pix2Vox++ architecture. When additional viewpoints become available at a high rate, e.g., from a camera mounted on a drone, selecting the most informative viewpoints is important in order to mitigate long term memory loss and to reduce the computational footprint. We present qualitative and quantitative results on the optimal selection of viewpoints and show that state-of-the-art reconstruction quality is already obtained with elementary selection algorithms.

Presurgical Planning with Open-Source Horos Software for Superficial Brain Arteriovenous Malformations

Surgical planning for treating brain arteriovenous malformations (bAVMs) is challenging because it entails visualizing 3-dimensional (3D) relationships between the nidus, its feeding and en passage arteries, and draining veins. Surgical experience in developing the capacity to mentally visualize pathological bAVM angioarchitecture could be complemented by this software, and thus potentially lower the steep learning curve for understanding complex bAVM angioarchitecture. We evaluated the clinical application of freely available online 3D reconstruction software in facilitating visualization of AVM angioarchitecture for presurgical planning. Preoperative Digital Imaging and Communications in Medicine magnetic resonance imaging/magnetic resonance angiography images of 56 superficial bAVMs from 2013 to 2018 were processed using open-source software Horos. 3D rendered images were compared with the surgical view to evaluate software accuracy and determine its value as a preoperative tool. 3D reconstructed images were compared with intraoperative recordings. A useful image identifying both the main feeding artery and draining vein was achieved in 35 of 56 cases (62.5%). Reconstructions of small AVMs (nidus ≤2 cm) and those located within the temporal or cerebellar cortex were less useful due to soft tissue artifacts. Frontal and parietal lobe lesions had significantly higher rates of identifying feeding arteries and draining veins (P < 0.05). Presurgical planning for resection of superficial bAVMs using Horos software allows for a comprehensive 3D analysis of the bAVM angioarchitecture. This technique is most useful for frontal and parietal lobe lesions, and aids the surgeon in formulating an optimal surgical strategy. The 3D reconstruction of the brain surface offers a surgical map not influenced by brain shift.

Supervoxel Convolution for Online 3D Semantic Segmentation

Online 3D semantic segmentation, which aims to perform real-time 3D scene reconstruction along with semantic segmentation, is an important but challenging topic. A key challenge is to strike a balance between efficiency and segmentation accuracy. There are very few deep-learning-based solutions to this problem, since the commonly used deep representations based on volumetric-grids or points do not provide efficient 3D representation and organization structure for online segmentation. Observing that on-surface supervoxels, i.e., clusters of on-surface voxels, provide a compact representation of 3D surfaces and brings efficient connectivity structure via supervoxel clustering, we explore a supervoxel-based deep learning solution for this task. To this end, we contribute a novel convolution operation (SVConv) directly on supervoxels. SVConv can efficiently fuse the multi-view 2D features and 3D features projected on supervoxels during the online 3D reconstruction, and leads to an effective supervoxel-based convolutional neural network, termed as Supervoxel-CNN , enabling 2D-3D joint learning for 3D semantic prediction. With the Supervoxel-CNN , we propose a clustering-then-prediction online 3D semantic segmentation approach. The extensive evaluations on the public 3D indoor scene datasets show that our approach significantly outperforms the existing online semantic segmentation systems in terms of efficiency or accuracy.

Robust 3D Reconstruction of Dynamic Scenes From Single-Photon Lidar Using Beta-Divergences.

In this article, we present a new algorithm for fast, online 3D reconstruction of dynamic scenes using times of arrival of photons recorded by single-photon detector arrays. One of the main challenges in 3D imaging using single-photon lidar in practical applications is the presence of strong ambient illumination which corrupts the data and can jeopardize the detection of peaks/surface in the signals. This background noise not only complicates the observation model classically used for 3D reconstruction but also the estimation procedure which requires iterative methods. In this work, we consider a new similarity measure for robust depth estimation, which allows us to use a simple observation model and a non-iterative estimation procedure while being robust to mis-specification of the background illumination model. This choice leads to a computationally attractive depth estimation procedure without significant degradation of the reconstruction performance. This new depth estimation procedure is coupled with a spatio-temporal model to capture the natural correlation between neighboring pixels and successive frames for dynamic scene analysis. The resulting online inference process is scalable and well suited for parallel implementation. The benefits of the proposed method are demonstrated through a series of experiments conducted with simulated and real single-photon lidar videos, allowing the analysis of dynamic scenes at 325 m observed under extreme ambient illumination conditions.

A Versatile Decentralized 3D Volumetric Fusion for On-line Reconstruction

Advancement in depth-sensing technology has allowed mobile robots to visualize the surrounding environment in 3D models. Regardless of the sensing technology (i.e. active, passive, or laser-based), a complete system that integrates recent depth data in previous 3D models in real-time is done by employing Simultaneous Localization And Mapping (SLAM) algorithms followed by a 3D reconstruction engine. Unfortunately, both the SLAM algorithm and the 3D reconstruction engine are usually executed on a single computing device, making the whole system exceptionally costly and heavy and restricting the robot's mobility. This paper proposes a decentralized, modular reconstruction system capable of employing various sensors to facilitate online 3D reconstruction from a resource-limited mobile robot.

On-line Dense 3D Reconstruction Method based on ANN us-ing a Single Structured Light Pattern

Dense 3D reconstruction techniques to measuring dynamic scenes and deformable objects with little texture have been widely studied for various applications like surface deformation measurement. We assume that the actual situation of scanning is acquiring sequential shape and measuring surface deformation online at the moment when an object deforms under the action of aerodynamic forces. This paper presents such a technique for on-line acquisition of 3D surface data based on a one-shot scanning method that reconstructs 3D shape from a single image where a simple color-coded pattern using de Bruijn sequence is projected onto an object. The proposed approach has advantages that it requires no assumption of global smoothness and continuous surface. To realize 3D reconstruction from a single image, there are sever-al issues to be solved, for example, difficulty on decoding structured light pattern because of influence of chromatic aberration in both projection systems and imaging systems, and difficul-ty on establishing accurate correspondences between projector and camera pixels. This paper describes the solutions of the issues by combining two methods, that is (1) an efficient and robust pattern decoding method based on artificial neural networks (ANN), and (2) a sub-pixel matching method using phase map and multiple view geometry. Furthermore, we adopt some effective strategies to improve algorithm robustness. Practical experiments were carried out to test the accuracy and efficiency of the scanning system in typical configuration. The results show that the scanning system can reconstruct 3D shape on-line in high-resolution with the accuracy of 0.09 mm and efficiency of 10 fps.

Fast online 3D reconstruction of dynamic scenes from individual single-photon detection events.

In this paper, we present an algorithm for online 3D reconstruction of dynamic scenes using individual times of arrival (ToA) of photons recorded by single-photon detector arrays. One of the main challenges in 3D imaging using single-photon Lidar is the integration time required to build ToA histograms and reconstruct reliably 3D profiles in the presence of non-negligible ambient illumination. This long integration time also prevents the analysis of rapid dynamic scenes using existing techniques. We propose a new method which does not rely on the construction of ToA histograms but allows, for the first time, individual detection events to be processed online, in a parallel manner in different pixels, while accounting for the intrinsic spatiotemporal structure of dynamic scenes. Adopting a Bayesian approach, a Bayesian model is constructed to capture the dynamics of the 3D profile and an approximate inference scheme based on assumed density filtering is proposed, yielding a fast and robust reconstruction algorithm able to process efficiently thousands to millions of frames, as usually recorded using single-photon detectors. The performance of the proposed method, able to process hundreds of frames per second, is assessed using a series of experiments conducted with static and dynamic 3D scenes and the results obtained pave the way to a new family of real-time 3D reconstruction solutions.

Comparison of diagnostic performance in assessing the rewiring position into a jailed side branch between online 3D reconstruction systems version 1.1 and 1.2 derived from optical frequency domain imaging

The three-dimensional reconstruction of optical coherence tomography and optical frequency domain imaging (3D-OCT/OFDI) helps optimize bifurcation percutaneous coronary interventions (PCIs) with side branch (SB) dilatation by identifying the optimal rewiring position. 3D-OCT/OFDI’s diagnostic performance for assessing the rewiring position into a jailed SB is unknown. We retrospectively evaluated the diagnostic performances of a conventional (ver. 1.1) and a new (ver. 1.2) online 3D-OFDI reconstruction system based on an offline 3D reconstruction system’s performance. We analyzed 45 patients’ 52 OFDI pullbacks with main vessel stenting followed by rewiring into a jailed SB for coronary bifurcation lesions. We counted the undetected stent struts in the polygon of confluence as the stent detection performance. We assessed the diagnostic agreement regarding the rewiring position into a jailed SB by the three 3D reconstruction systems. The percentage of undetected struts and the diagnostic agreement of ver.1.2 were significantly better than those of ver.1.1 [5.1 ± 5.1% vs. 30.2 ± 14.2%; p < 0.0001, and 94.2% (49/52) vs. 76.9% (40/52); p = 0.0120]. The new online 3D-OFDI reconstruction system provides better diagnostic performance than the conventional online system for assessing the rewiring position into a jailed SB.

Online Measurement of Deposit Surface in Electron Beam Freeform Fabrication

In the process of electron beam freeform fabrication (EBF3), due to the continuous change of thermal conditions and variability in wire feeding in the deposition process, geometric deviations are generated in the deposition of each layer. In order to prevent the layer-by-layer accumulation of the deviation, it is necessary to perform online geometry measurement for each deposition layer, based on which the error compensation can be done for the previous deposition layer in the next deposition layer. However, the traditional three-dimensional reconstruction method that employs structured laser cannot meet the requirements of long-term stable operation in the manufacturing process of EBF3. Therefore, this paper proposes a method to measure the deposit surfaces based on the position information of electron beam speckle, in which an electron beam is used to bombard the surface of the deposit to generate the speckle. Based on the structured information of the electron beam in the vacuum chamber, the three-dimensional reconstruction of the surface of the deposited parts is realized without need of additional structured laser sensor. In order to improve the detection accuracy, the detection error is theoretically analyzed and compensated. The absolute error after compensation is smaller than 0.1 mm, and the precision can reach 0.1%, which satisfies the requirements of 3D reconstruction of the deposited parts. An online measurement system is built for the surface of deposited parts in the process of electron beam freeform fabrication, which realizes the online 3D reconstruction of the surface of the deposited layer. In addition, in order to improve the detection stability of the whole system, the image processing algorithm suitable for this scene is designed. The reliability and speed of the algorithm are improved by ROI extraction, threshold segmentation, and expansion corrosion. In addition, the speckle size information can also reflect the thermal conditions of the surface of the deposited parts. Hence, it can be used for online detection of defects such as infusion and voids.

Minimal vision system for linear camera self-calibration and 3D reconstruction using translation platform

Camera calibration and 3D reconstruction are two crucial steps in computer vision. With the progress of robot and autonomous rover systems, zooming cameras are widely applied, where online calibration and 3D reconstruction are becoming more and more important. This paper proposed a minimal vision system that consists of a translation platform and an un-calibrated camera mounted on. With this minimal system, we can linearly calibrate the camera online and reconstruct 3D structures from three un-calibrated images by utilizing translation motions. The two images generated by translating the camera to allow the recovery of scene depths. Depths are then analyzed by error analysis models and are utilized to determine the infinite homography between the third image and any of the two translated images. The intrinsic parameters are then calibrated linearly from the computed infinite homography. Camera motion estimation and 3D reconstruction are then readily determined from the intrinsic calibration results. We also proposed a two-step optimization method to refine both the calibration and 3D reconstruction results by minimizing the overall back-projection errors across the three images within a tiny-scale bundle adjustment framework. The proposed method has been validated with both simulation and real image data. The results demonstrate that the proposed minimal linear system can solve the online camera self-calibration problem and the reconstruction of 3D structure. The paper suggests a framework of minimal linear system by using an un-calibrated camera and a low-cost translation platform to address the problems of linear self-calibration of camera, motion estimation, 3D reconstruction, and optimization, in a practical, easy, and accurate way.

On-Line 3D reconstruction based on laser scanning for robot machining of large complex components

The digital model is the foundation of intelligent manufacturing based on robot machining. In the intelligent manufacturing of large complex components in the fields of aerospace, energy equipment, shipbuilding, etc., it is often necessary to perform online 3D reconstruction of these large-sized components. In response to this demand, this paper designs a large-scale on-line 3D reconstruction system based on laser scanning principle. A laser scanner is used to generate the contour of the component accurately. Then, a series of contour sequences are generated by the movement perpendicular to the direction of the contour plane. These contour sequences are processed by algorithms to generate the 3D point cloud model of the large complex component. The results show that the system can generate the 3D point cloud model of large-scale complex components in an on-line manner and provide the basis for the detection of the components before machining and the machining planning in some cases.

Real-time 3D scene reconstruction with dynamically moving object using a single depth camera

Online 3D reconstruction of real-world scenes has been attracting increasing interests from both the academia and industry, especially with the consumer-level depth cameras becoming widely available. Recent most online reconstruction systems take live depth data from a moving Kinect camera and incrementally fuse them to a single high-quality 3D model in real time. Although most real-world scenes have static environment, the daily objects in a scene often move dynamically, which are non-trivial to reconstruct especially when the camera is also not still. To solve this problem, we propose a single depth camera-based real-time approach for simultaneous reconstruction of dynamic object and static environment, and provide solutions for its key issues. In particular, we first introduce a robust optimization scheme which takes advantage of raycasted maps to segment moving object and background from the live depth map. The corresponding depth data are then fused to the volumes, respectively. These volumes are raycasted to extract views of the implicit surface which can be used as a consistent reference frame for the next iteration of segmentation and tracking. Particularly, in order to handle fast motion of dynamic object and handheld camera in the fusion stage, we propose a sequential 6D pose prediction method which largely increases the registration robustness and avoids registration failures occurred in conventional methods. Experimental results show that our approach can reconstruct moving object as well as static environment with rich details, and outperform conventional methods in multiple aspects.

3D scene reconstruction using a texture probabilistic grammar

In this paper, texture probabilistic grammar is defined for the first time. We have developed an algorithm to obtain the 3D information in a 2D scene by training the texture probabilistic grammar from the prebuilt model library. The well-trained texture probabilistic grammar could also be applied to 3D reconstruction. Our detailed process contains: dividing the 2D scene into texture fragments; assigning the most suitable 3D object label to the 2D texture fragments; using our texture probabilistic grammar to predict 3D information of the texture fragments in 2D scene image; constructing the 3D model of the original 2D scene image. Through experiments, it is proved that the algorithm has a better effect on reconstruction of indoor scenes and building structures, and the algorithm is superior to the traditional reconstruction method based on point clouds. Different datasets and reconstructed objects are tested, which verifies the robustness of the algorithm. As a result, our algorithm is able to deal with the large numbers of scenes with similar semantics and it is also fast enough to deal with the online 3D reconstruction.

Geometrically stable tracking for depth images based 3D reconstruction on mobile devices

With the development of hardwares such as mobile devices and portable depth cameras, on-line 3D reconstruction on the mobile devices with depth streams as input turns to be possible and promising. Most existing systems use volumetric representation methods to fuse the depth images and use ICP algorithm to estimate the poses of cameras. However, ICP tracker suffers from large drift in scenes containing insufficient geometric information. To deal with this problem, we propose a stability based sampling method which select different number of point-pairs in different windows according to their geometric stability. In addition, we fuse the ICP tracker with the IMU information through an analysis of the condition number. Then we apply the stability based sampling method to the spatially hashed volumetric representation. Qualitative and quantitative evaluations of tracking accuracy and 3D reconstruction results show that our method outperforms the current state-of-the-art systems, especially in scenes lacking sufficient geometric information. In total, our method achieves frame rates 20 Hz on an Apple iPad Air 2 and 200 Hz on a Nvidia GeForce GTX 1060 GPU.

Feature-based RGB-D camera pose optimization for real-time 3D reconstruction

In this paper we present a novel feature-based RGB-D camera pose optimization algorithm for real-time 3D reconstruction systems. During camera pose estimation, current methods in online systems suffer from fast-scanned RGB-D data, or generate inaccurate relative transformations between consecutive frames. Our approach improves current methods by utilizing matched features across all frames and is robust for RGB-D data with large shifts in consecutive frames. We directly estimate camera pose for each frame by efficiently solving a quadratic minimization problem to maximize the consistency of 3D points in global space across frames corresponding to matched feature points. We have implemented our method within two state-of-the-art online 3D reconstruction platforms. Experimental results testify that our method is efficient and reliable in estimating camera poses for RGB-D data with large shifts.

Windowed Factorization and Merging

In this work, an online 3D reconstruction algorithm is proposed which attempts to solve the structure from motion problem for occluded and degenerate data. To deal with occlusion the temporal consistency of data within a limited window is used to compute local reconstructions. These local reconstructions are transformed and merged to obtain an estimation of the 3D object shape. The algorithm is shown to accurately reconstruct a rotating and translating artificial sphere and a rotating toy dinosaur from a video. The proposed algorithm (WIFAME) provides a versatile framework to deal with missing data in the structure from motion problem.

OPTO-ACOUSTIC DATA FUSION FOR SUPPORTING THE GUIDANCE OF REMOTELY OPERATED UNDERWATER VEHICLES (ROVs)

Abstract. Remotely Operated underwater Vehicles (ROVs) play an important role in a number of operations conducted in shallow and deep water (e.g.: exploration, survey, intervention, etc.), in several application fields like marine science, offshore construction, and underwater archeology. ROVs are usually equipped with different imaging devices, both optical and acoustic. Optical sensors are able to generate better images in close range and clear water conditions, while acoustic systems are usually employed in long range acquisitions and do not suffer from the presence of turbidity, a well-known cause of coarser resolution and harder data extraction. In this work we describe the preliminary steps in the development of an opto-acoustic camera able to provide an on-line 3D reconstruction of the acquired scene. Taking full advantage of the benefits arising from the opto-acoustic data fusion techniques, the system was conceived as a support tool for ROV operators during the navigation in turbid waters, or in operations conducted by means of mechanical manipulators. The paper presents an overview of the device, an ad-hoc methodology for the extrinsic calibration of the system and a custom software developed to control the opto-acoustic camera and supply the operator with visual information.

Dynamic 3D surface reconstruction and motion modeling from a pan–tilt–zoom camera

3D surface reconstruction and motion modeling has been integrated in several industrial applications. Using a pan–tilt–zoom (PTZ) camera, we present an efficient method called dynamic 3D reconstruction (D3DR) for recovering the 3D motion and structure of a freely moving target. The proposed method estimates the PTZ measurements to keep the target in the center of the field of view (FoV) of the camera with the same size. Feature extraction and tracking approach are used in the imaging framework to estimate the target's translation, position, and distance. A selection strategy is used to select keyframes that show significant changes in target movement and directly update the recovered 3D information. The proposed D3DR method is designed to work in a real-time environment, not requiring all frames captured to be used to update the recovered 3D motion and structure of the target. Using fewer frames minimizes the time and space complexity required. Experimental results conducted on real-time video streams using different targets to prove the efficiency of the proposed method. The proposed D3DR has been compared to existing offline and online 3D reconstruction methods, showing that it uses less execution time than the offline method and uses an average of 49.6% of the total number of frames captured.

Real-time 3D reconstruction at scale using voxel hashing

Online 3D reconstruction is gaining newfound interest due to the availability of real-time consumer depth cameras. The basic problem takes live overlapping depth maps as input and incrementally fuses these into a single 3D model. This is challenging particularly when real-time performance is desired without trading quality or scale. We contribute an online system for large and fine scale volumetric reconstruction based on a memory and speed efficient data structure. Our system uses a simple spatial hashing scheme that compresses space, and allows for real-time access and updates of implicit surface data, without the need for a regular or hierarchical grid data structure. Surface data is only stored densely where measurements are observed. Additionally, data can be streamed efficiently in or out of the hash table, allowing for further scalability during sensor motion. We show interactive reconstructions of a variety of scenes, reconstructing both fine-grained details and large scale environments. We illustrate how all parts of our pipeline from depth map pre-processing, camera pose estimation, depth map fusion, and surface rendering are performed at real-time rates on commodity graphics hardware. We conclude with a comparison to current state-of-the-art online systems, illustrating improved performance and reconstruction quality.