3D Reconstruction Method Research Articles

Structure learning for 3D Point Cloud Generation from Single RGB Images

Abstract3D point clouds can represent complex 3D objects of arbitrary topologies and with fine‐grained details. They are, however, hard to regress from images using convolutional neural networks, making tasks such as 3D reconstruction from monocular RGB images challenging. In fact, unlike images and volumetric grids, point clouds are unstructured and thus lack proper parameterization, which makes them difficult to process using convolutional operations. Existing point‐based 3D reconstruction methods that tried to address this problem rely on complex end‐to‐end architectures with high computational costs. Instead, we propose in this paper a novel mechanism that decouples the 3D reconstruction problem from the structure (or parameterization) learning task, making the 3D reconstruction of objects of arbitrary topologies tractable and thus easier to learn. We achieve this using a novel Teacher‐Student network where the Teacher learns to structure the point clouds. The Student then harnesses the knowledge learned by the Teacher to efficiently regress accurate 3D point clouds. We train the Teacher network using 3D ground‐truth supervision and the Student network using the Teacher's annotations. Finally, we employ a novel refinement network to overcome the upper‐bound performance that is set by the Teacher network. Our extensive experiments on ShapeNet and Pix3D benchmarks, and on in‐the‐wild images demonstrate that the proposed approach outperforms previous methods in terms of reconstruction accuracy and visual quality.

A 3D Chromosome Structure Reconstruction method with High Resolution Hi-C Data using Nonlinear Dimensionality Reduction and Divide-and-conquer strategy.

Chromosomes are fundamental components of genetic material, and their structural characteristics play an essential role in the regulation of gene expression. The advent of high-resolution Hi-C data has enabled scientists to explore the three-dimensional structure of chromosomes. However, most of the currently available methods for reconstructing chromosome structures are unable to achieve high resolutions, such as 5 Kilobase (KB). In this study, we present NeRV-3D, an innovative method that utilizes a nonlinear dimensionality reduction visualization algorithm to reconstruct 3D chromosome structures at low resolutions. Additionally, we introduce NeRV-3D-DC, which employs a divide-and-conquer technique to reconstruct and visualize 3D chromosome structures at high resolutions. Our results demonstrate that both NeRV-3D and NeRV-3D-DC outperform existing methods in terms of 3D visualization effects and evaluation metrics on simulated and actual Hi-C datasets. The implementation of NeRV-3D-DC can be found at https://github.com/ghaiyan/NeRV-3D-DC.

A New Method for Reconstructing Tree-Level Aboveground Carbon Stocks of Eucalyptus Based on TLS Point Clouds

Eucalyptus plantation forests in southern China provide not only the economic value of producing timber, but also the ecological value service of absorbing carbon dioxide and releasing oxygen. Based on the theory of spatial colonial modeling, this paper proposes a new method for 3D reconstruction of tree terrestrial LiDAR point clouds for determining the aboveground carbon stock of eucalyptus monocotyledons, which consists of the main steps of tree branch and trunk separation, skeleton extraction and optimization, 3D reconstruction, and carbon stock calculation. The main trunk and branches of the tree point clouds are separated using a layer-by-layer judgment and clustering method, which avoids errors in judgment caused by sagging branches. By optimizing and adjusting the skeleton to remove small redundant branches, the near-parallel branches belonging to the same tree branch are fused. The missing parts of the skeleton point clouds were complemented using the cardinal curve interpolation algorithm, and finally a real 3D structural model was generated based on the complemented and smoothed tree skeleton expansion. The bidirectional Hausdoff distance, average Hausdoff distance, and F distance were used as evaluation indexes, which were reduced by 0.7453 m, 0.0028 m, and 0.0011 m, respectively, and the improved spatial colonization algorithm enhanced the accuracy of the reconstructed tree 3D structural model. To verify the accuracy of our method to determine the carbon stock and its related parameters, we cut down 41 eucalyptus trees and destructively sampled the measurement data as reference values. The R2 of the linear fit between the reconstructed single-tree aboveground carbon stock estimates and the reference values was 0.96 with a CV(RMSE) of 16.23%, the R2 of the linear fit between the trunk volume estimates and the reference values was 0.94 with a CV(RMSE) of 19.00%, and the R2 of the linear fit between the branch volume estimates and the reference values was 0.95 with a CV(RMSE) of 38.84%. In this paper, a new method for reconstructing eucalyptus carbon stocks based on TLS point clouds is proposed, which can provide decision support for forest management and administration, forest carbon sink trading, and emission reduction policy formulation.

Endoscope 3D reconstruction based on ORB-SLAM

Minimal invasive surgery (MIS) is the mainstream trend in developing surgical technology. As the endoscope is a significant tool in the surgical process, whether it can track the inner cavity and realize accurate real-time 3D reconstruction has a vital impact on the smooth progress of MIS. However, there are still problems in the endoscopic environment, such as severe image distortion, the effect of lighting conditions, and the inability to extract lumen textures. Orinted fast and rotated brief simultaneous localization and mapping (ORB-SLAM) is currently a relatively advanced simultaneous localization and mapping (SLAM) method with better performance. The ORB-SLAM-based endoscope 3D reconstruction method can improve performance and overcome the challenge of endoscope 3D reconstruction. This paper will first introduce several existing endoscope 3D reconstruction methods based on ORB-SLAM and analyze the limitations and issues of these methods through their experimental results. Then the paper will explore the solutions to the defects in this method from other methods and compare the characteristics and the result of experiments. Secondly, through the summary of the above methods and the introduction of the integration of the ORB-SLAM-based methods and other current advanced technologies, the future development trend and huge development potential of ORB-SLAM-based endoscopic 3D reconstruction are introduced. This paper will be of profound affection to further improve the optimization and application of the ORB-SLAM-based endoscope 3D reconstruction method.

Research and Implementation of Three-Dimensional Spatial Information Characterization and Visualization of Fractures in Deteriorated Sandstone

Primary fractures significantly impacted the stability of surrounding rock in underground projects. Therefore, it is vital to find a solution for the problem of performing a non-destructive detection of rocks and extracting the internal three-dimensional (3D) data field of rocks for visualization analysis. To address this problem, this paper proposed a method of 3D reconstruction for complex cracks in deteriorated sandstone and developed a program based on MATLAB. This work carried out image recognition on the CT scan images of deteriorated sandstone, then implemented a surface reconstruction technique based on object cross-section information, a contour reconstruction technique based on object contour information, a point cloud reconstruction technique for extracting point cloud data of internal cracks in deteriorated sandstone, and a Graphical User Interface (GUI) control system that combines these three reconstruction techniques. The results showed that the 3D reconstruction techniques and the GUI control system proposed in this paper were capable of precisely marking the location of the cracks on a 3D coordinate system and accurately describing their shape with a vector. With only 10 CT scan images, the point cloud reconstruction technique constructed the digital core, and the digital core can quantitatively characterize the influence of primary fractures on the stability of surrounding rock. Additionally, the calculated results of the proposed method were very close to that of Avizo. This method realized the visualization and quantitative characterization of the internal structure of rocks and offered a model for analyzing the stress-fracture-seepage field change during excavation.

Deep learning-based end-to-end 3D depth recovery from a single-frame fringe pattern with the MSUNet++ network.

Deep learning (DL)-based single-frame fringe pattern to 3D depth reconstruction methods have aroused extensive research interest. The goal is to estimate high-precision 3D shape from a single frame of fringe pattern with limited information. Therefore, the purpose of this work attempts to propose an end-to-end DL-based 3D reconstruction method from the single fringe pattern with excellent capability of achieving high accuracy depth recovery and geometry details preservation of tested objects. We construct a multi-scale feature fusion convolutional neural network (CNN) called MSUNet++, which incorporates discrete wavelet transform (DWT) in data preprocessing for extracting high-frequency signals of fringe patterns as input of the network. Additionally, a loss function that combines structural similarity with edge perception is established. Through these measures, high-frequency geometry details of the reconstruction results can be obviously enhanced, while the geometric shape can be effectively maintained. Ablation experiments are involved in validating the effectiveness of our proposed solution. 3D reconstructed results and analysis of generalization experiments on different tested samples imply that the proposed method in this research enjoys capabilities of higher accuracy, better detail preservation, and robustness in comparison with the compared methods.

3D Reconstruction of Fishes Using Coded Structured Light.

3D reconstruction of fishes provides the capability of extracting geometric measurements, which are valuable in the field of Aquaculture. In this paper, a novel method for 3D reconstruction of fishes using the Coded Structured Light technique is presented. In this framework, a binary image, called pattern, consisting of white geometric shapes, namely symbols, on a black background is projected onto the surface of a number of fishes, which belong to different species. A camera captures the resulting images, and the various symbols in these images are decoded to uniquely identify them on the pattern. For this purpose, a number of steps, such as the binarization of the images captured by the camera, symbol classification, and the correction of misclassifications, are realized. The proposed methodology for 3D reconstructions is adapted to the specific geometric and morphological characteristics of the considered fishes with fusiform body shape, something which is implemented for the first time. Using the centroids of the symbols as feature points, the symbol correspondences immediately result in point correspondences between the pattern and the images captured by the camera. These pairs of corresponding points are exploited for the final 3D reconstructions of the fishes. The extracted 3D reconstructions provide all the geometric information which is related to the real fishes. The experimentation demonstrates the high efficiency of the techniques adopted in each step of the proposed methodology. As a result, the final 3D reconstructions provide sufficiently accurate approximations of the real fishes.

3D reconstruction method based on the multi-polarization superposition coding phase pattern of LRR objects.

Conventional research in structured light measurements has utilized light intensity as a channel for information. The polarization of light can be used as an additional channel of information. In this paper, a method based on the superposition of multiple polarization states is proposed to encode structured light. By building a polarization model between the color of light and the polarization states, polarized structured light containing phase information is obtained without rotating the polarizer. It is demonstrated that the method improves the waveform quality of stripes and the accuracy of the 3D reconstruction results when measuring highly reflective objects.

A refractive three-dimensional reconstruction method for fringe projection with a planar medium

In special measurement situations, a vision measurement system needs to measure objects through a viewing observation window made of a planar transparent medium. The refraction at the interface between a medium and air leads to large measurement errors in the pin-hole imaging model. Most existing solutions require complex processes of calculating refracted light equations and/or matching stereo images. Here, we propose a refractive three-dimensional (3D) reconstruction method for fringe projection with a planar medium. We derive a coordinate conversion relationship from a pseudo-object point caused by light refraction to a real-object point based on flat refractive geometry. Then, we integrate the relationship into regular fringe projection framework for unbiased 3D reconstruction. Two experiments, including 3D shape measurements of a step and a regular sphere are performed. The results verify the effectiveness and accuracy of the proposed refractive 3D reconstruction method.

Fast 3D Reconstruction of UAV Images Based on Neural Radiance Field

Traditional methods for 3D reconstruction of unmanned aerial vehicle (UAV) images often rely on classical multi-view 3D reconstruction techniques. This classical approach involves a sequential process encompassing feature extraction, matching, depth fusion, point cloud integration, and mesh creation. However, these steps, particularly those that feature extraction and matching, are intricate and time-consuming. Furthermore, as the number of steps increases, a corresponding amplification of cumulative error occurs, leading to its continual augmentation. Additionally, these methods typically utilize explicit representation, which can result in issues such as model discontinuity and missing data during the reconstruction process. To effectively address the challenges associated with heightened temporal expenditures, the absence of key elements, and the fragmented models inherent in three-dimensional reconstruction using Unmanned Aerial Vehicle (UAV) imagery, an alternative approach is introduced—the neural radiance field. This novel method leverages neural networks to intricately fit spatial information within the scene, thereby streamlining the reconstruction steps and rectifying model deficiencies. The neural radiance field method employs a fully connected neural network to meticulously model object surfaces and directly generate the 3D object model. This methodology simplifies the intricacies found in conventional 3D reconstruction processes. Implicitly encapsulating scene characteristics, the neural radiance field allows for iterative refinement of neural network parameters via the utilization of volume rendering techniques. Experimental results substantiate the efficacy of this approach, demonstrating its ability to complete scene reconstruction within a mere 5 min timeframe, thereby reducing reconstruction time by 90% while markedly enhancing reconstruction quality.

3D Reconstruction and Measurement Analysis of a Dense Point Cloud Fused with a Depth Image

To solve the problems of holes, noise, and texture information missing in the traditional incremental reconstruction of complex surface objects, a 3D reconstruction method of depth image fusion surface dense point clouds is proposed, and texture feature creation is combined to obtain a 3D reconstruction model that takes into account the main body and details of the reconstructed object. First, the mechanism of surface dense reconstruction based on the patch-based multiview stereo (PMVS) algorithm is analyzed. Combined with the principle of view angle selection of stereo images, surface point cloud density reconstruction is performed. Then, the depth value is optimized by the region growing method, and the optimization model is established. The depth image is fused into a dense surface, and the reconstructed part is supplemented by the depth information. Finally, the Markov random field (MRF) is introduced to describe the richness of image details, and combined with the calculating method of the area coordinate, the texture coordinates are accurately calculated to reproduce the texture details of the 3D reconstruction model. 3D reconstruction experiments are performed on multiple indoor and outdoor model surfaces, and the experimental results show that the proposed method can achieve complete and accurate reconstruction of complex surface objects. Our method provides technical support for complex surface topography detection and has industrial practical significance.

Generation of a Virtual Cohort of Patients for in Silico Trials of Acute Ischemic Stroke Treatments

The development of in silico trials based on high-fidelity simulations of clinical procedures requires the availability of large cohorts of three-dimensional (3D) patient-specific anatomy models, which are often hard to collect due to limited availability and/or accessibility and imaging quality. Statistical shape modeling (SSM) allows one to identify the main modes of shape variation and to generate new samples based on the variability observed in a training dataset. In this work, a method for the automatic 3D reconstruction of vascular anatomies based on SSM is used for the generation of a virtual cohort of cerebrovascular models suitable for computational simulations, useful for in silico stroke trials. Starting from 88 cerebrovascular anatomies segmented from stroke patients’ images, an SSM algorithm was developed to generate a virtual population of 100 vascular anatomies, defined by centerlines and diameters. An acceptance criterion was defined based on geometric parameters, resulting in the acceptance of 83 generated anatomies. The 3D reconstruction method was validated by reconstructing a cerebrovascular phantom lumen and comparing the result with an STL geometry obtained from a computed tomography scan. In conclusion, the final 3D models of the generated anatomies show that the proposed methodology can produce a reliable cohort of cerebral arteries.

Stereo Reconstruction Method for 3D Surface Wave Fields around a Floating Body Using a Marker Net in a Wave Tank

Spatial wave fields around floating bodies are important for the understanding of hydrodynamics, and particularly the wave drift forces, of floating bodies in waves; however, experimental measurement of these fields is challenging. This paper presents a stereo reconstruction method for three-dimensional (3D) surface wave fields around floating bodies in a wave tank. Styrofoam markers were attached to a flexible net in a regular grid, called a marker net, and were placed on the water surface to be used as targets for stereo cameras (SCs). A thin plate spline was applied to the markers detected by the SCs to reconstruct the 3D surface wave profile around a floating body model. The proposed method was validated by measuring the wave field around a cylindrical floating body with a footing at its bottom. These experiments were conducted under regular wave incidence conditions. A wave elevation time series measured using a servo-controlled wave gauge was used as the benchmark data. The 3D surface wave field reconstruction method was applied under three different conditions: without the model, with a fixed model, and with a freely oscillating model. The results showed reliable reconstructions of the scattering and radiation waves. The marker net’s effects on the floating body’s motion and the surrounding wave fields were shown to be negligible by comparing the results acquired with and without the marker net.

3D reconstruction method based on N-step phase unwrapping

3D reconstruction method based on N-step phase unwrapping

An attention-based and deep sparse priori cascade multi-view stereo network for 3D reconstruction

Recently 3D reconstruction has received extensive attention and deep research due to the continuous development of the field of computer vision. Nonetheless, the learning-based methods for 3D reconstruction are not robust enough, resulting in the reconstruction model having many occlusions and outliers. In this study, we aim to improve feature matching correlation, aggregate global contextual information, and enhance the robustness of depth estimation to improve the quality of the reconstruction. We propose an attention-based deep sparse priori cascade multi-view stereo network, ADS-MVSNet. Firstly, we propose a feature extraction module based on the attention mechanism to obtain the regions of interest in the input scene. Secondly, we propose a depth sparse prior strategy module to estimate the depth map of the input scene more accurately. It is followed by refinement of the initial depth map using a coarse-to-fine method to improve the accuracy of point cloud reconstruction. These two modules are lightweight and effective, improving the robustness of depth map estimation. We conduct experiments on three common datasets (DTU, ETH3D, Tanks & Temples) and a dataset of a real scene created by us. The experimental results show that ADS-MVSNet performs better in reconstruction quality compared to classical methods.

3D-porous-GAN: a high-performance 3D GAN for digital core reconstruction from a single 3D image

The 3D digital rock technology is extensively utilized in analyzing rock physical properties, reservoir modeling, and other related fields. This technology enables the visualization, quantification, and analysis of microstructures in rock cores, leading to precise predictions and optimized designs of reservoir properties. Although the accuracy of 3D digital rock reconstruction algorithms based on physical experiments is high, the associated acquisition costs and reconstruction processes are expensive and complex, respectively. On the other hand, the 3D digital rock random reconstruction method based on 2D slices is advantageous in terms of its low cost and easy implementation, but its reconstruction effect still requires significant improvement. This article draws inspiration from the Concurrent single-image generative adversarial network and proposes an innovative algorithm to reconstruct 3D digital rock by improving the generator, discriminator, and noise vector in the network structure. Compared to traditional numerical reconstruction methods and generative adversarial network algorithms, the method proposed in this paper is shown to achieve good agreement with real samples in terms of Dykstra-Parson coefficient, porosity, two-point correlation function, Minkowski functionals, and visual display.

A 3D reconstruction method based on one-dimensional galvanometer laser scanning system

The line-structured light galvanometer scanning system projects the laser light strip onto the object by swinging the mirror to realize the 3D profile measurement of the object. It has the advantages of compact structure, fast measurement speed and strong adaptability to complex environments. In this paper, a three-dimensional scanning system based on a line laser, a one-dimensional galvanometer and an industrial camera is constructed, and a new mathematical model of one-dimensional galvanometer laser scanner is derived. On this basis, a high-precision and simple calibration method of the laser scanning system is proposed. Compared with other methods, this method takes into account the rotation angle error caused by the installation of the galvanometer. The whole calibration process only needs a freely moving chessboard instead of a precision electric stage for operation, which is very easy and convenient for on-site use. The experimental results show that the measurement system calibrated by the proposed method has an average relative deviation of 0.14% for the distance measurement of ceramic balls, and successfully achieved 3D reconstruction of plaster objects with complex surfaces and metal parts with high or low reflectivity.

Research on Viewpoint Planning Method for Multi-view Image 3D Reconstruction

Research on Viewpoint Planning Method for Multi-view Image 3D Reconstruction

Partial alignment of multislice spatially resolved transcriptomics data.

Spatially resolved transcriptomics (SRT) technologies measure messenger RNA (mRNA) expression at thousands of locations in a tissue slice. However, nearly all SRT technologies measure expression in two-dimensional (2D) slices extracted from a 3D tissue, thus losing information that is shared across multiple slices from the same tissue. Integrating SRT data across multiple slices can help recover this information and improve downstream expression analyses, but multislice alignment and integration remains a challenging task. Existing methods for integrating SRT data either do not use spatial information or assume that the morphology of the tissue is largely preserved across slices, an assumption that is often violated because of biological or technical reasons. We introduce PASTE2, a method for partial alignment and 3D reconstruction of multislice SRT data sets, allowing only partial overlap between aligned slices and/or slice-specific cell types. PASTE2 formulates a novel partial fused Gromov-Wasserstein optimal transport problem, which we solve using a conditional gradient algorithm. PASTE2 includes a model selection procedure to estimate the fraction of overlap between slices, and optionally uses information from histological images that accompany some SRT experiments. We show on both simulated and real data that PASTE2 obtains more accurate alignments than existing methods. We further use PASTE2 to reconstruct a 3D map of gene expression in a Drosophila embryo from a 16 slice Stereo-seq data set. PASTE2 produces accurate alignments of multislice data sets from multiple SRT technologies, enabling detailed studies of spatial gene expression across a wide range of biological applications.

TMSDNet: Transformer with multi‐scale dense network for single and multi‐view 3D reconstruction

Abstract3D reconstruction is a long‐standing problem. Recently, a number of studies have emerged that utilize transformers for 3D reconstruction, and these approaches have demonstrated strong performance. However, transformer‐based 3D reconstruction methods tend to establish the transformation relationship between the 2D image and the 3D voxel space directly using transformers or rely solely on the powerful feature extraction capabilities of transformers. They ignore the crucial role played by deep multi‐scale representation of the object in the voxel feature domain, which can provide extensive global shape and local detail information about the object in a multi‐scale manner. In this article, we propose a novel framework TMSDNet (transformer with multi‐scale dense network) for single‐view and multi‐view 3D reconstruction with transformer to solve this problem. Based on our well‐designed combined‐transformer Block, which is canonical encoder–decoder architecture, voxel features with spatial order can be extracted from the input image, which are used to further extract multi‐scale global features in parallel using a multi‐scale residual attention module. Furthermore, a residual dense attention block is introduced for deep local features extraction and adaptive fusion. Finally, the reconstructed objects are produced with the voxel reconstruction block. Experiment results on the benchmarks such as ShapeNet and Pix3D datasets demonstrate that TMSDNet outperforms the existing state‐of‐the‐art reconstruction methods substantially.