3D Reconstruction Method Research Articles

Three-Dimensional Shape Reconstruction from Digital Freehand Design Sketching Based on Deep Learning Techniques

This paper proposes a method for 3D reconstruction from Freehand Design Sketching (FDS) in architecture and industrial design. The implementation begins by extracting features from the FDS using the self-supervised learning model DINO, followed by the continuous Signed Distance Function (SDF) regression as an implicit representation through a Multi-Layer Perceptron network. Taking eyeglass frames as an example, the 2D contour and freehand sketch optimize the alignment by their geometrical similarity while exploiting symmetry to improve reconstruction accuracy. Experiments demonstrate that this method can effectively reconstruct high-quality 3D models of eyeglass frames from 2D freehand sketches, outperforming existing deep learning-based 3D reconstruction methods. This research offers practical information for understanding 3D modeling methodology for FDS, triggering multiple modes of design creativity and efficient scheme adjustments in industrial or architectural conceptual design. In conclusion, this novel approach integrates self-supervised learning and geometric optimization to achieve unprecedented fidelity in 3D reconstruction from FDS, setting a new benchmark for AI-driven design processes in industrial and architectural applications.

FA-MSVNet: multi-scale and multi-view feature aggregation methods for stereo 3D reconstruction

FA-MSVNet: multi-scale and multi-view feature aggregation methods for stereo 3D reconstruction

3D human avatar reconstruction with neural fields: A recent survey

3D human avatar reconstruction aims to reconstruct the 3D geometric shape and appearance of the human body from various data inputs, such as images, videos, and depth information, acting as a key component in human-oriented 3D vision in the metaverse. With the progress in neural fields for 3D reconstruction in recent years, significant advancements have been made in this research area for shape accuracy and appearance quality. Meanwhile, substantial efforts on dynamic avatars with the representation of neural fields have exhibited their effect. Although significant improvements have been achieved, challenges still exist in in-the-wild and complex environments, detailed shape recovery, and interactivity in real-world applications. In this survey, we present a comprehensive overview of 3D human avatar reconstruction methods using advanced neural fields. We start by introducing the background of 3D human avatar reconstruction and the mainstream paradigms with neural fields. Subsequently, representative research studies are classified based on their representation and avatar partswith detailed discussion. Moreover, we summarize the commonly used available datasets, evaluation metrics, and results in the research area. In the end, we discuss the open problems and highlight the promising future directions, hoping to inspire novel ideas and promote further research in this area.

Observation of biological and emulsion samples by newly developed three-dimensional impedance scanning electron microscopy

Imaging at nanometre-scale resolution is indispensable for many scientific fields such as biology, chemistry, material science and nanotechnology. Scanning electron microscopes (SEM) are widely used as important tools for the nanometre-scale analysis of various samples. However, because of the vacuum inside the SEM, a typical analysis requires fixation of samples, a drying process, and staining with heavy metals. Therefore, there is a need for convenient and minimally invasive methods of observing samples in solution. Recently, we have developed a new type of impedance microscopy, multi-frequency impedance SEM (IP-SEM), which allows nanoscale imaging of various specimens in water with minimal radiation damage. Here, we report a new IP-SEM system equipped with a linear-array terminal, which allows eight tilted images to be observed in a single capture by applying eight frequencies of input signals to each electrode. Furthermore, we developed a three-dimensional (3D) reconstruction method based on the Simulated Annealing (SA) algorithm, which enables us to construct a high-precision 3D model from the 8 tilted images. The method reported here can be easily used for 3D structural analysis of various biological samples, organic materials, and nanoparticles.

Scene reconstruction techniques for autonomous driving: a review of 3D Gaussian splatting

As the latest research result of the explicit radiated field technology, 3D Gaussian Splatting (3D GS) replaces the implicit expression represented by Neural Radiated Field (NeRF) and has become the hottest research direction in 3D scene reconstruction. Given the innovative work and vigorous development of 3D GS in autonomous driving, this paper comprehensively reviews and summarizes the existing related research to showcase the evolution of the 3D GS technology and possible future development directions. First, the overall research background of 3D GS is introduced based on two aspects 3D scene reconstruction methods and 3D GS research progress. Second, the relevant knowledge points of 3D GS and the core formulas to clarify the mathematical mechanism of 3D GS are presented. Third, the primary applications of the 3D scene reconstruction technology based on 3D GS in automatic driving are presented through new perspective synthesis, scene understanding, and simultaneous localization and map building (SLAM). Finally, the research frontier directions of 3D GS in autonomous driving are described, including structure optimization, 4D scene reconstruction, and cross-domain research. This paper may provide an effective and convenient pathway for researchers to understand, explore, apply this novel research method, and promote the development and application of 3D GS in automatic driving.

Non-Interleaved Shared-Aperture Full-Stokes Metalens via Prior-Knowledge-Driven Inverse Design.

Characterization of electromagnetic wave polarization states is critical in various applications of materials, biomedical, and imaging. The emergence of metasurfaces opens up the possibility of implementing highly integrated full-Stokes imagers. Despite rapid development, prevailing schemes on metasurface-based full-Stokes imagers require interleaved or cascaded designs, inevitably resulting in performance deterioration, bulky size, and complicating the imaging procedure due to misalignment. To overcome these challenges, a non-interleaved shared-aperture full-Stokes metalens enabled by the prior-knowledge-driven inverse design methodology is proposed. The metalens can be directly deployed into imagers, performing high-accuracy diffraction-limited full-Stokes imaging without other ancillary components, breaking intrinsic constraints of efficiency and resolution in interleaved design. To demonstrate this, the metalens is integrated into the W-band passive imaging system, as an alternative solution, to perform polarimetric imaging, which reduces the reliance on the high cost and high complexity of the ortho-mode transducer and broadband correlator in traditional polarimetric radiometers. Furthermore, with the assistance of a 3D reconstruction method, the feasibility of multi-polarization information for contactless surface slope measurements is explored. This work may open new paradigms for the full-Stokes imager designing and broaden the applications of metasurface polarimetric imaging.

In Situ Monitoring of Anodic Acidification Process Using 3D μ-XCT Method.

Debonding of the primary anode caused by anodic acidification is one of the major failure modes of the impressed current cathodic protection (ICCP) system in reinforced concrete structures. This study used 3D micro X-ray computed tomography (μ-XCT) to monitor the in situ evolution of the anodic acidification-affected zone. Samples were scanned after 0 to 40 days of the accelerated anodic acidification test. The anodic acidification-affected zone was identified in μ-XCT images using the gray level segmentation method. The total volume of this zone was measured using the 3D reconstruction method. It was found that detailed 3D information can be extracted using the 3D reconstruction method. The spatial heterogeneity was analyzed using this reconstructed volume information. The Faraday efficiency was calculated and found to increase after 20 days of operation. It was also found that the affected zone was proportional to the input electrical energy. The proposed model is useful for estimating the durability of an ICCP system.

3D Pixelwise damage mapping using a deep attention based modified Nerfacto

Recent advancements in structural health monitoring have highlighted the necessity for accurate three-dimensional (3D) damage mapping on digital twins, moving beyond traditional methods such as photogrammetry, which frequently struggle to capture intricate planar surfaces. To address this limitation, this paper proposes a new advanced 3D reconstruction method and its integration with 3D damage mapping techniques. As the 3D reconstruction method, an Attention-based Modified Nerfacto (ABM-Nerfacto) model is developed, and is integrated with an advanced damage segmentation method. Using a three-span continuous bridge with concrete piers as an example structure, and concrete cracks as the example damage, the state-of-the-art STRNet is utilized for crack segmentation. Through extensive parametric studies and comparative evaluations, the proposed ABM-Nerfacto model was demonstrated to produce high-quality 3D reconstructions and corresponding damage mappings for this bridge system. This integrated approach provides a promising solution for comprehensive 3D digital twin-based structural health monitoring.

3D Reconstruction with Fast Dipole Sums

We introduce a method for high-quality 3D reconstruction from multi-view images. Our method uses a new point-based representation, the regularized dipole sum, which generalizes the winding number to allow for interpolation of per-point attributes in point clouds with noisy or outlier points. Using regularized dipole sums, we represent implicit geometry and radiance fields as per-point attributes of a dense point cloud, which we initialize from structure from motion. We additionally derive Barnes-Hut fast summation schemes for accelerated forward and adjoint dipole sum queries. These queries facilitate the use of ray tracing to efficiently and differentiably render images with our point-based representations, and thus update their point attributes to optimize scene geometry and appearance. We evaluate our method in inverse rendering applications against state-of-the-art alternatives, based on ray tracing of neural representations or rasterization of Gaussian point-based representations. Our method significantly improves 3D reconstruction quality and robustness at equal runtimes, while also supporting more general rendering methods such as shadow rays for direct illumination.

LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives

Large garages are ubiquitous yet intricate scenes that present unique challenges due to their monotonous colors, repetitive patterns, reflective surfaces, and transparent vehicle glass. Conventional Structure from Motion (SfM) methods for camera pose estimation and 3D reconstruction often fail in these environments due to poor correspondence construction. To address these challenges, we introduce LetsGo, a LiDAR-assisted Gaussian splatting framework for large-scale garage modeling and rendering. We develop a handheld scanner, Polar, equipped with IMU, LiDAR, and a fisheye camera, to facilitate accurate data acquisition. Using this Polar device, we present the GarageWorld dataset, consisting of eight expansive garage scenes with diverse geometric structures, which will be made publicly available for further research. Our approach demonstrates that LiDAR point clouds collected by the Polar device significantly enhance a suite of 3D Gaussian splatting algorithms for garage scene modeling and rendering. We introduce a novel depth regularizer that effectively eliminates floating artifacts in rendered images. Additionally, we propose a multi-resolution 3D Gaussian representation designed for Level-of-Detail (LOD) rendering. This includes adapted scaling factors for individual levels and a random-resolution-level training scheme to optimize the Gaussians across different resolutions. This representation enables efficient rendering of large-scale garage scenes on lightweight devices via a web-based renderer. Experimental results on our GarageWorld dataset, as well as on ScanNet++ and KITTI-360, demonstrate the superiority of our method in terms of rendering quality and resource efficiency.

DEGAN: Detail-Enhanced Generative Adversarial Network for Monocular Depth-Based 3D Reconstruction

Although deep networks-based 3D reconstruction methods can recover the 3D geometry given few inputs, they may produce unfaithful reconstruction when predicting occluded parts of 3D objects. To address the issue, we propose Detail-Enhanced Generative Adversarial Network (DEGAN) which consists of Encoder–Decoder-Based Generator (EDGen) and Voxel-Point Embedding Network-Based Discriminator (VPDis) for 3D reconstruction from a monocular depth image of an object. Firstly, EDGen decodes the features from the 2.5D voxel grid representation of an input depth image and generates the 3D occupancy grid under GAN losses and a sampling point loss. The sampling loss can improve the accuracy of predicted points with high uncertainty. VPDis helps reconstruct the details under voxel and point adversarial losses, respectively. Experimental results show that DEGAN not only outperforms several state-of-the-art methods on both public ModelNet and ShapeNet datasets but also predicts more reliable occluded/missing parts of 3D objects.

Investigation on micro-mechanics properties of machined metamorphic layer based on crystal plasticity finite element method

Surface materials endure significant mechanical and thermal loads during machining, leading to microstructural changes and the formation of metamorphic layers. These layers exhibit altered crystallographic characteristics, such as grain size, misorientation angles, and dislocation density, resulting in mechanical properties that differ from the bulk material. This study examines the microstructural evolution of the metamorphic layer using electron back-scattered diffraction (EBSD) and X-ray diffraction (XRD). Based on microstructure characterization, a 3D reconstruction method was developed using a representative volume element (RVE). The crystal plasticity finite element method (CPFEM) was employed to establish the relationship between the microstructure and micromechanical properties, including microhardness, elastic modulus, and yield stress. The proposed method was validated by comparing simulation results with experimental data obtained from micro-pillar compression tests and nanoindentation tests. The results demonstrated a strong correlation in stress-strain curves, and the microhardness measurement error at indentation depths of 400 nm was less than 10 %.

Generative 3D Reconstruction of Martian Surfaces using Monocular Images

Abstract. High-resolution digital terrain models (DEM) are crucial for Mars scientific exploration and engineering demands, such as mission route planning, landing site selection, and topography research. However, DEMs generated based on photogrammetry or photometry suffer from limited stereo coverage and complex generative processes. Therefore, we propose a GAN-based monocular 3D reconstruction method, which utilizes high-resolution monocular orbital images to achieve pixel-level Martian terrain reconstruction. We preprocess the image data and then invert the elevation using the trained generative model. Finally, we recover the absolute scale through post-processing. In this work, we use HiRISE orthorectified images and DEMs with a resolution of 2 m and 0.25 m to validate the effectiveness of our method. We evaluated our method in four areas with different landforms and found that the predicted DEM and HiRISE DEM have height consistency with the root mean square error of about 2 m. This indicates that the proposed GAN-based method has certain effectiveness and generalization and great potential in high-resolution monocular Martian DEM reconstruction.

Transfer learning-based techniques for efficient 3D-reconstruction of functionally graded materials

The various properties and phenomena in materials significantly depend on the high-quality three-dimensional (3D) microstructure. This paper proposes a novel method for 3D reconstruction of functionally graded materials (FGMs) using transfer learning techniques to overcome the limitations of experimental 3D microscopy. The approach begins by interpolating 2D cross-sectional images of FGM microstructures through a hybrid deep learning method combining StyleGAN and Gatys techniques, generating intermediate images based on style and content. In the next step, transfer learning is applied using convolutional occupancy networks to reconstruct the 3D microstructure from the 2D image depth layers, with the model trained on point clouds obtained from 2D images. To validate the method, the reconstructed microstructures are compared to actual material images at the same depth using various metrics, including lineal path, two-point correlation, and quality of connection functions. The results show a high degree of accuracy, indicating that the reconstructed images closely match the target microstructures. Finally, to show the application of model the approach is implemented to the functionally graded lanthanum strontium manganite (LSM) and 3D microstructure of this FGM has been extracted using the approach

Georecon: a coarse-to-fine visual 3D reconstruction approach for high-resolution images with neural matching priors

ABSTRACT Visual 3D reconstruction enables rebuilding 3D scenes from captured images, serving as a fundamental data source for digital earth modeling and intelligent cities. In the foundational step, recent methods leverage learning-based descriptors for image registration and achieve tremendous advances in precision and robustness. However, these methods inevitably execute down sampling towards high-resolution images to fit the needs of neural networks, which leads to precision degradation of feature localization and matching. Thus, we propose GeoRecon: a novel coarse-to-fine visual 3D reconstruction method that optimally utilizes high-resolution images for high-quality visual 3D reconstruction. Firstly, the coarse stage conducts coarse reconstruction from downsampled images by performing neural matching with geometric priors. Secondly, we define the fine-grained stage, proposing a GPU-based algorithm for generating image-patch correspondences based on the neural matching priors to perform fine-grained image registration. Finally, based on the optimized camera poses under this coarse-to-fine paradigm, progressive dense reconstruction leveraging efficient neural radiance fields is proposed to accomplish the high-quality MVS reconstruction. Comparative experiments across various scenarios demonstrate the proposed method’s superior precision, robustness, and reconstruction quality.

Three-Dimensional Reconstruction of Retaining Structure Defects from Crosshole Ground Penetrating Radar Data Using a Generative Adversarial Network

Crosshole ground penetrating radar (GPR) is an efficient method for ensuring the quality of retaining structures without the need for excavation. However, interpreting crosshole GPR data is time-consuming and prone to inaccuracies. To address this challenge, we proposed a novel three-dimensional (3D) reconstruction method based on a generative adversarial network (GAN) to recover 3D permittivity distributions from crosshole GPR images. The established framework, named CGPR2VOX, integrates a fully connected layer, a residual network, and a specialized 3D decoder in the generator to effectively translate crosshole GPR data into 3D permittivity voxels. The discriminator was designed to enhance the generator’s performance by ensuring the physical plausibility and accuracy of the reconstructed models. This adversarial training mechanism enables the network to learn non-linear relationships between crosshole GPR data and subsurface permittivity distributions. CGPR2VOX was trained using a dataset generated through finite-difference time-domain (FDTD) simulations, achieving precision, recall and F1-score of 91.43%, 96.97% and 94.12%, respectively. Model experiments validate that the relative errors of the estimated positions of the defects were 1.67%, 1.65%, and 1.30% in the X-, Y-, and Z-direction, respectively. Meanwhile, the method exhibits noteworthy generalization capabilities under complex conditions, including condition variations, heterogeneous materials and electromagnetic noise, highlighting its reliability and effectiveness for practical quality assurance of retaining structures.

Artificial Skin Based on Visuo‐Tactile Sensing for 3D Shape Reconstruction: Material, Method, and Evaluation

AbstractArtificial skin has shown great potential in robot perception and human healthcare. It provides multifunctional tactile sensing, including 3D shape reconstruction, contact feedback, and temperature perception, where the 3D reconstruction function is indispensable for dexterous hands in tactile cognition and interaction. Vision‐based tactile sensor (VTS) is an innovative bionic tactile sensor and supports high‐resolution, high‐precision, and high‐density tactile reconstruction compared with electronic tactile sensors. Considering the unique contribution of visuo‐tactile sensing to artificial skin, this review focuses on the 3D reconstruction techniques of the VTS. 3D reconstruction methods are classified into five categories based on sensing modalities, hardware categories, and modeling approaches: 1) photometric stereo, 2) binocular depth calibration, 3) optical flow, 4) deep learning, and 5) ToF (time of flight). In addition, the association and difference of reconstruction methods are analyzed from the hardware perspective, and the development and technological details of 3D reconstruction are summarized. On this basis, the challenges and development direction are discussed. This review can be viewed as a technology guide to provide references for interested researchers. Furthermore, it is expected to promote the extensive application of the VTS in artificial skins.

Reviewing Open Data Semantic 3D City Models to Develop Novel 3D Reconstruction Methods

Abstract. Structured semantic 3D city models are pivotal in creating urban 3D digital twins. The wide adoption of such models has been primarily enabled by robust, model-based, and automatic 3D reconstruction methods. However, these methods impose requirements on the reconstruction, mainly restricting the solution space to several model types and relying on accurate 2D footprints. Recent research shows that deep-learning-based methods promise highly generic solution space and are footprint-free. Yet, the current training and test datasets are limited, hindering the methods’ development. In this work, we analyze the ubiquity of already existing, open 3D city model datasets and their potential to serve as a large-scale training and test set for 3D reconstruction, where 27 potential dataset collections have been identified. Our review shows that more than 215 million building models are readily available. We firmly believe that this review will facilitate further research on robust automatic 3D city model reconstruction and serve as a reference for benchmarking 3D city models.

3D reconstruction and depth estimation method for local anomalies of rail surface based on multi-view stereo matching

Abstract Detecting rail surface anomalies has become crucial for ensuring the safety of train operations. However, manual detection methods are time-consuming and labor-intensive. Although traditional detection models based on one-dimensional signals or two-dimensional images can effectively identify the existence of defects, they are difficult to use to obtain local depth characteristic information, leading to difficulties in accurately assessing the extent of damage in abnormal regions. Rail track maintenance strategies are typically developed based on the detected different depth ranges of defects. To address this issue, this paper proposes a local depth estimation method based on the point cloud of the target rail surface reconstructed by PatchmatchNet. Firstly, according to the acquisition method of multi-view images and the sampling environment at the site, a practical data collection protocol is established for generating a multi-view dataset of rail surface. Next, dense point cloud of the target rail surface were reconstructed by PatchmatchNet. Subsequently, a method for estimating the local anomaly depth is developed based on the point cloud. Experimental results demonstrate that the proposed method achieves a minimum error of approximately 5.5% within a maximum depth range of 0.35mm, when compared to depth measurements obtained using a structured light camera. Finally, this paper presents a more comprehensive three-dimensional representation of the local abnormal physical structure, surpassing traditional one-dimensional and two-dimensional forms. This enhanced representation offers richer information for the effective determination of whether the anomalies constitute defects and aids in distinguishing true defects from other surface irregularities. Such advancement significantly improves the precision of defect assessment and supports the development of highly precise repair strategies.

Multi-line laser 3D reconstruction method based on spatial quadric surface and geometric estimation

The traditional multi-line laser 3D reconstruction is based on binocular epipolar constraints and the equation of the laser optical plane. Firstly, matching points are identified based on the epipolar constraints. Then, the correct matching points are selected based on the optical plane equation of the multi-line laser. Finally, 3D reconstruction is performed using the selected matching points. In actual production processes involving multi-line lasers, noticeable distortion occurs due to the projection of Diffractive Optical Elements (DOE) at a large angle. This results in curved laser light planes and curved reflections on the measured objects. Under such circumstances, efficiently screening matching points using the traditional method based on the laser plane equation becomes challenging. Additionally, inherent noise in multi-line laser systems introduces errors in extracted laser center coordinates, making it impossible to directly obtain high-precision 3D data. To address these issues, this paper proposes a method that utilizes spatial quadric surfaces and geometric estimation for completing multi-line laser 3D reconstructions. By analyzing the distortion principle of DOE and the positional relationship after optical diffraction, the multi-line laser manifests as a quadric surface on the optical output plane. Consequently, by calibrating the equations of the quadric surface and applying binocular polar constraints, suitable matching points for the multi-line laser can be chosen. Once an accurate matching point is identified, a minimum geometric distance estimation can be established based on the distance constraint between the point and its corresponding epipolar line. This distance represents the separation between the left and right camera’s laser center points and their respective epipolar lines. Utilizing this estimate of the minimum geometric distance, a refined calculation can be performed to better satisfy epipolar constraints and obtain new matching points. Ultimately, utilizing these new matching points enables the completion of 3D reconstruction for multi-line lasers. In comparison with methods relying on spatial plane and epipolar constraints, our algorithm exhibits a superior degree of matchability and accuracy.According to multiple groups of test data, the average error before optimization is 0.3126 mm, and can be increased to 0.0336 mm after optimization.