3D Reconstruction Method Research Articles

Generative 3D Reconstruction of Martian Surfaces using Monocular Images

Abstract. High-resolution digital terrain models (DEM) are crucial for Mars scientific exploration and engineering demands, such as mission route planning, landing site selection, and topography research. However, DEMs generated based on photogrammetry or photometry suffer from limited stereo coverage and complex generative processes. Therefore, we propose a GAN-based monocular 3D reconstruction method, which utilizes high-resolution monocular orbital images to achieve pixel-level Martian terrain reconstruction. We preprocess the image data and then invert the elevation using the trained generative model. Finally, we recover the absolute scale through post-processing. In this work, we use HiRISE orthorectified images and DEMs with a resolution of 2 m and 0.25 m to validate the effectiveness of our method. We evaluated our method in four areas with different landforms and found that the predicted DEM and HiRISE DEM have height consistency with the root mean square error of about 2 m. This indicates that the proposed GAN-based method has certain effectiveness and generalization and great potential in high-resolution monocular Martian DEM reconstruction.

Georecon: a coarse-to-fine visual 3D reconstruction approach for high-resolution images with neural matching priors

ABSTRACT Visual 3D reconstruction enables rebuilding 3D scenes from captured images, serving as a fundamental data source for digital earth modeling and intelligent cities. In the foundational step, recent methods leverage learning-based descriptors for image registration and achieve tremendous advances in precision and robustness. However, these methods inevitably execute down sampling towards high-resolution images to fit the needs of neural networks, which leads to precision degradation of feature localization and matching. Thus, we propose GeoRecon: a novel coarse-to-fine visual 3D reconstruction method that optimally utilizes high-resolution images for high-quality visual 3D reconstruction. Firstly, the coarse stage conducts coarse reconstruction from downsampled images by performing neural matching with geometric priors. Secondly, we define the fine-grained stage, proposing a GPU-based algorithm for generating image-patch correspondences based on the neural matching priors to perform fine-grained image registration. Finally, based on the optimized camera poses under this coarse-to-fine paradigm, progressive dense reconstruction leveraging efficient neural radiance fields is proposed to accomplish the high-quality MVS reconstruction. Comparative experiments across various scenarios demonstrate the proposed method’s superior precision, robustness, and reconstruction quality.

Three-Dimensional Reconstruction of Retaining Structure Defects from Crosshole Ground Penetrating Radar Data Using a Generative Adversarial Network

Crosshole ground penetrating radar (GPR) is an efficient method for ensuring the quality of retaining structures without the need for excavation. However, interpreting crosshole GPR data is time-consuming and prone to inaccuracies. To address this challenge, we proposed a novel three-dimensional (3D) reconstruction method based on a generative adversarial network (GAN) to recover 3D permittivity distributions from crosshole GPR images. The established framework, named CGPR2VOX, integrates a fully connected layer, a residual network, and a specialized 3D decoder in the generator to effectively translate crosshole GPR data into 3D permittivity voxels. The discriminator was designed to enhance the generator’s performance by ensuring the physical plausibility and accuracy of the reconstructed models. This adversarial training mechanism enables the network to learn non-linear relationships between crosshole GPR data and subsurface permittivity distributions. CGPR2VOX was trained using a dataset generated through finite-difference time-domain (FDTD) simulations, achieving precision, recall and F1-score of 91.43%, 96.97% and 94.12%, respectively. Model experiments validate that the relative errors of the estimated positions of the defects were 1.67%, 1.65%, and 1.30% in the X-, Y-, and Z-direction, respectively. Meanwhile, the method exhibits noteworthy generalization capabilities under complex conditions, including condition variations, heterogeneous materials and electromagnetic noise, highlighting its reliability and effectiveness for practical quality assurance of retaining structures.

Artificial Skin Based on Visuo‐Tactile Sensing for 3D Shape Reconstruction: Material, Method, and Evaluation

AbstractArtificial skin has shown great potential in robot perception and human healthcare. It provides multifunctional tactile sensing, including 3D shape reconstruction, contact feedback, and temperature perception, where the 3D reconstruction function is indispensable for dexterous hands in tactile cognition and interaction. Vision‐based tactile sensor (VTS) is an innovative bionic tactile sensor and supports high‐resolution, high‐precision, and high‐density tactile reconstruction compared with electronic tactile sensors. Considering the unique contribution of visuo‐tactile sensing to artificial skin, this review focuses on the 3D reconstruction techniques of the VTS. 3D reconstruction methods are classified into five categories based on sensing modalities, hardware categories, and modeling approaches: 1) photometric stereo, 2) binocular depth calibration, 3) optical flow, 4) deep learning, and 5) ToF (time of flight). In addition, the association and difference of reconstruction methods are analyzed from the hardware perspective, and the development and technological details of 3D reconstruction are summarized. On this basis, the challenges and development direction are discussed. This review can be viewed as a technology guide to provide references for interested researchers. Furthermore, it is expected to promote the extensive application of the VTS in artificial skins.

Reviewing Open Data Semantic 3D City Models to Develop Novel 3D Reconstruction Methods

Abstract. Structured semantic 3D city models are pivotal in creating urban 3D digital twins. The wide adoption of such models has been primarily enabled by robust, model-based, and automatic 3D reconstruction methods. However, these methods impose requirements on the reconstruction, mainly restricting the solution space to several model types and relying on accurate 2D footprints. Recent research shows that deep-learning-based methods promise highly generic solution space and are footprint-free. Yet, the current training and test datasets are limited, hindering the methods’ development. In this work, we analyze the ubiquity of already existing, open 3D city model datasets and their potential to serve as a large-scale training and test set for 3D reconstruction, where 27 potential dataset collections have been identified. Our review shows that more than 215 million building models are readily available. We firmly believe that this review will facilitate further research on robust automatic 3D city model reconstruction and serve as a reference for benchmarking 3D city models.

3D reconstruction and depth estimation method for local anomalies of rail surface based on multi-view stereo matching

Abstract Detecting rail surface anomalies has become crucial for ensuring the safety of train operations. However, manual detection methods are time-consuming and labor-intensive. Although traditional detection models based on one-dimensional signals or two-dimensional images can effectively identify the existence of defects, they are difficult to use to obtain local depth characteristic information, leading to difficulties in accurately assessing the extent of damage in abnormal regions. Rail track maintenance strategies are typically developed based on the detected different depth ranges of defects. To address this issue, this paper proposes a local depth estimation method based on the point cloud of the target rail surface reconstructed by PatchmatchNet. Firstly, according to the acquisition method of multi-view images and the sampling environment at the site, a practical data collection protocol is established for generating a multi-view dataset of rail surface. Next, dense point cloud of the target rail surface were reconstructed by PatchmatchNet. Subsequently, a method for estimating the local anomaly depth is developed based on the point cloud. Experimental results demonstrate that the proposed method achieves a minimum error of approximately 5.5% within a maximum depth range of 0.35mm, when compared to depth measurements obtained using a structured light camera. Finally, this paper presents a more comprehensive three-dimensional representation of the local abnormal physical structure, surpassing traditional one-dimensional and two-dimensional forms. This enhanced representation offers richer information for the effective determination of whether the anomalies constitute defects and aids in distinguishing true defects from other surface irregularities. Such advancement significantly improves the precision of defect assessment and supports the development of highly precise repair strategies.

Multi-line laser 3D reconstruction method based on spatial quadric surface and geometric estimation

The traditional multi-line laser 3D reconstruction is based on binocular epipolar constraints and the equation of the laser optical plane. Firstly, matching points are identified based on the epipolar constraints. Then, the correct matching points are selected based on the optical plane equation of the multi-line laser. Finally, 3D reconstruction is performed using the selected matching points. In actual production processes involving multi-line lasers, noticeable distortion occurs due to the projection of Diffractive Optical Elements (DOE) at a large angle. This results in curved laser light planes and curved reflections on the measured objects. Under such circumstances, efficiently screening matching points using the traditional method based on the laser plane equation becomes challenging. Additionally, inherent noise in multi-line laser systems introduces errors in extracted laser center coordinates, making it impossible to directly obtain high-precision 3D data. To address these issues, this paper proposes a method that utilizes spatial quadric surfaces and geometric estimation for completing multi-line laser 3D reconstructions. By analyzing the distortion principle of DOE and the positional relationship after optical diffraction, the multi-line laser manifests as a quadric surface on the optical output plane. Consequently, by calibrating the equations of the quadric surface and applying binocular polar constraints, suitable matching points for the multi-line laser can be chosen. Once an accurate matching point is identified, a minimum geometric distance estimation can be established based on the distance constraint between the point and its corresponding epipolar line. This distance represents the separation between the left and right camera’s laser center points and their respective epipolar lines. Utilizing this estimate of the minimum geometric distance, a refined calculation can be performed to better satisfy epipolar constraints and obtain new matching points. Ultimately, utilizing these new matching points enables the completion of 3D reconstruction for multi-line lasers. In comparison with methods relying on spatial plane and epipolar constraints, our algorithm exhibits a superior degree of matchability and accuracy.According to multiple groups of test data, the average error before optimization is 0.3126 mm, and can be increased to 0.0336 mm after optimization.

3D X-ray microscope acts as an accurate and effective equipment of pathological diagnosis in craniofacial imaging

Craniofacial structure and dental hard tissue used to be researched on by traditional imaging tools such as light microscope, electron microscope and micro-CT. Due to the limitations of imaging principle, resolution and 3D rendering reconstruction technique, traditional imaging tools are constrained for presenting fine structure and precise measurements. Here a brand-new imaging equipment–3D X-ray microscope is introduced to realize a more efficient scanning by demonstrating the comparison of the craniofacial structures and dental hard tissue of diabetes and normal DBA mouse. To explore a higher resolution, more efficient imaging measurement and 3D reconstruction method on craniofacial structure and dental hard tissue. The study included 12 DBA mice which were divided into two groups (control group and diabetes group). The heads were separated and scanned by 3D X-ray microscope, after which regions of interest were selected, followed by measurement and 3D reconstruction based on microscope attached software Dragonfly pro©. Hemi-mandibles were collected for enamel mineral density assessment supported by QRM-MicroCT-HA phantom. Data was submitted to paired t-tests at a 95% confidence level. The automatic assessed enamel thickness of diabetes mice decreased on average, whereas the rest of manual measurements and automatic assessed density showed no statistical difference. We constructed HA phantom assisted enamel density procedure in Dragonfly software. Craniofacial structure and dental hard tissue were well-presented both in 2D slide and 3D reconstruction viewport by 3D X-ray microscope which can be routinely used as craniofacial structure and dental hard tissue imaging tool.

Three-Dimensional Reconstruction of Zebra Crossings in Vehicle-Mounted LiDAR Point Clouds

In the production of high-definition maps, it is necessary to achieve the three-dimensional instantiation of road furniture that is difficult to depict on traditional maps. The development of mobile laser measurement technology provides a new means for acquiring road furniture data. To address the issue of traffic marking extraction accuracy in practical production, which is affected by degradation, occlusion, and non-standard variations, this paper proposes a 3D reconstruction method based on energy functions and template matching, using zebra crossings in vehicle-mounted LiDAR point clouds as an example. First, regions of interest (RoIs) containing zebra crossings are obtained through manual selection. Candidate point sets are then obtained at fixed distances, and their neighborhood intensity features are calculated to determine the number of zebra stripes using non-maximum suppression. Next, the slice intensity feature of each zebra stripe is calculated, followed by outlier filtering to determine the optimized length. Finally, a matching template is selected, and an energy function composed of the average intensity of the point cloud within the template, the intensity information entropy, and the intensity gradient at the template boundary is constructed. The 3D reconstruction result is obtained by solving the energy function, performing mode statistics, and normalization. This method enables the complete 3D reconstruction of zebra stripes within the RoI, maintaining an average planar corner accuracy within 0.05 m and an elevation accuracy within 0.02 m. The matching and reconstruction time does not exceed 1 s, and it has been applied in practical production.

Improved Real-time 3D Reconstruction Method for Mixed Reality Telepresence

Improved Real-time 3D Reconstruction Method for Mixed Reality Telepresence

A 3D Reconstruction Method for Heterogeneous Data of Power Equipment Based on Improved Neural Radiation Fields

A 3D Reconstruction Method for Heterogeneous Data of Power Equipment Based on Improved Neural Radiation Fields

Parametric indoor 3D reconstruction based on binary image feature point extraction

Abstract In recent years, with the rapid advancement of laser scanning technology, indoor 3D modeling technology has significantly developed, becoming a research hotspot in the field of 3D reconstruction. Addressing the issues of high complexity in decoding point cloud surfaces in traditional indoor reconstruction, this paper proposes an indoor 3D reconstruction method based on binary image feature point extraction. The method utilizes point cloud data for semantic segmentation of the indoor environment to accurately extract the ceiling area, which is then projected onto the XOY plane and converted into a binary image, thereby transforming the 3D problem into a 2D problem. Subsequently, key feature points are identified through feature point detection technology, and systematic model construction is carried out based on the sequence of these feature points and their associated parameters. This method effectively simplifies the complexity of traditional 3D modeling, achieving accurate and parametric indoor 3D reconstruction.

3D reconstruction method of welding area by fusion of coding raster and semantic segmentation network

3D reconstruction method of welding area by fusion of coding raster and semantic segmentation network

A New 3D reconstruction method for pressure-sensitive paint measurements under limited optical access

Reconstruction of 3D pressure field is crucial for pressure-sensitive paint measurements in turbomachinery experiments. Existing methods impose high requirements on feature points in terms of quantity, spatial distribution, and positional accuracy, and is difficult to apply under limited optical access, such as in planar cascade wind tunnels. We propose a novel direct linear transformation method based on contour matching (CM-DLT). This method leverages model contours captured from a tilted camera perspective, which provides the basis for identifying the projection matrix linking image pixels to 3D spatial points. The subsequent optimization of the projection matrix significantly enhances the accuracy and robustness of 3D pressure field reconstruction. In a benchmark experiment, the efficiency and reprojection accuracy of 3D reconstruction are investigated. Subsequently, the CM-DLT method is successfully applied in the cascade wind tunnel experiment. The obtained 3D pressure fields of two surfaces of adjacent blades are analyzed.

Bridge Surface Defect Localization Based on Panoramic Image Generation and Deep Learning-Assisted Detection Method

Applying unmanned aerial vehicles (UAVs) and vision-based analysis methods to detect bridge surface damage significantly improves inspection efficiency, but the existing techniques have difficulty in accurately locating damage, making it difficult to use the results to assess a bridge’s degree of deterioration. Therefore, this study proposes a method to generate panoramic bridge surface images using multi-view images captured by UAVs, in order to automatically identify and locate damage. The main contributions are as follows: (1) We propose a UAV-based image-capturing method for various bridge sections to collect close-range, multi-angle, and overlapping images of the surface; (2) we propose a 3D reconstruction method based on multi-view images to reconstruct a textured bridge model, through which an ultra-high resolution panoramic unfolded image of the bridge surface can be obtained by projecting from multiple angles; (3) we applied the Swin Transformer to optimize the YOLOv8 network and improve the detection accuracy of small-scale damages based on the established bridge damage dataset and employed sliding window segmentation to detect damage in the ultra-high resolution panoramic image. The proposed method was applied to detect surface damage on a three-span concrete bridge. The results indicate that this method automatically generates panoramic images of the bridge bottom, deck, and sides with hundreds of millions of pixels and recognizes damage in the panoramas. In addition, the damage detection accuracy reached 98.7%, which is improved by 13.6% when compared with the original network.

High accuracy patient-specific 3D coronary reconstruction from angiography and intravascular optical coherence tomography.

The integration of digital subtraction angiography (DSA) with intravascular optical coherence tomography (IVOCT) offers a comprehensive 3D arterial model, which is invaluable for the analysis of vascular anatomy and biomechanics. However, the process of image fusion is often hindered by the challenge of accurately orienting IVOCT images. This paper introduces a novel, to our knowledge, dual-path 3D reconstruction method that leverages the guidewire and the vessel's centerline to establish cross sectional direction vectors within the IVOCT images and spatial direction vectors along the guidewire's trajectory. This approach minimizes the accumulation of reconstruction errors by ensuring the precise orientation of each vascular cross section. The efficacy of the proposed method is validated through vascular phantom experiments and the reconstruction of patient-specific 3D coronary models.

Three-Dimensional Reconstruction of Indoor Scenes Based on Implicit Neural Representation.

Reconstructing 3D indoor scenes from 2D images has always been an important task in computer vision and graphics applications. For indoor scenes, traditional 3D reconstruction methods have problems such as missing surface details, poor reconstruction of large plane textures and uneven illumination areas, and many wrongly reconstructed floating debris noises in the reconstructed models. This paper proposes a 3D reconstruction method for indoor scenes that combines neural radiation field (NeRFs) and signed distance function (SDF) implicit expressions. The volume density of the NeRF is used to provide geometric information for the SDF field, and the learning of geometric shapes and surfaces is strengthened by adding an adaptive normal prior optimization learning process. It not only preserves the high-quality geometric information of the NeRF, but also uses the SDF to generate an explicit mesh with a smooth surface, significantly improving the reconstruction quality of large plane textures and uneven illumination areas in indoor scenes. At the same time, a new regularization term is designed to constrain the weight distribution, making it an ideal unimodal compact distribution, thereby alleviating the problem of uneven density distribution and achieving the effect of floating debris removal in the final model. Experiments show that the 3D reconstruction effect of this paper on ScanNet, Hypersim, and Replica datasets outperforms the state-of-the-art methods.

Three-Dimensional Reconstruction of Forest Scenes with Tree–Shrub–Grass Structure Using Airborne LiDAR Point Cloud

Fine three-dimensional (3D) reconstruction of real forest scenes can provide a reference for forestry digitization and forestry resource management applications. Airborne LiDAR technology can provide valuable data for large-area forest scene reconstruction. This paper proposes a 3D reconstruction method for complex forest scenes with trees, shrubs, and grass, based on airborne LiDAR point clouds. First, forest vertical distribution characteristics are used to segment tree, shrub, and ground–grass points from an airborne LiDAR point cloud. For ground–grass points, a ground–grass grid model is constructed. For tree points, a method based on hierarchical canopy point fitting is proposed to construct a trunk model, and a crown model is constructed with the 3D α-shape algorithm. For shrub points, a shrub model is directly constructed based on the 3D α-shape algorithm. Finally, tree, shrub, and ground–grass models are spatially combined to achieve the reconstruction of real forest scenes. Taking six forest plots located in Hebei, Yunnan, and Guangxi provinces in China and Baden-Württemberg in Germany as study areas, experimental results show that the accuracy of individual tree segmentation reaches 87.32%, the accuracy of shrub segmentation reaches 60.00%, the height accuracy of the grass model is evaluated with an RMSE < 0.15 m, the volume accuracy of shrub and tree models is assessed with R2 > 0.848 and R2 > 0.904, respectively. Furthermore, we compared the model constructed in this study with simplified point cloud and voxel models. The results demonstrate that the proposed modeling approach can meet the demand for the high-accuracy and lightweight modeling of large-area forest scenes.

NeRF-OR: neural radiance fields for operating room scene reconstruction from sparse-view RGB-D videos.

RGB-D cameras in the operating room (OR) provide synchronized views of complex surgical scenes. Assimilation of this multi-view data into a unified representation allows for downstream tasks such as object detection and tracking, pose estimation, and action recognition. Neural radiance fields (NeRFs) can provide continuous representations of complex scenes with limited memory footprint. However, existing NeRF methods perform poorly in real-world OR settings, where a small set of cameras capture the room from entirely different vantage points. In this work, we propose NeRF-OR, a method for 3D reconstruction of dynamic surgical scenes in the OR. Where other methods for sparse-view datasets use either time-of-flight sensor depth or dense depth estimated from color images, NeRF-OR uses a combination of both. The depth estimations mitigate the missing values that occur in sensor depth images due to reflective materials and object boundaries. We propose to supervise with surface normals calculated from the estimated depths, because these are largely scale invariant. We fit NeRF-OR to static surgical scenes in the 4D-OR dataset and show that its representations are geometrically accurate, where state of the art collapses to sub-optimal solutions. Compared to earlier work, NeRF-OR grasps fine scene details while training 30 faster. Additionally, NeRF-OR can capture whole-surgery videos while synthesizing views at intermediate time values with an average PSNR of 24.86dB. Last, we find that our approach has merit in sparse-view settings beyond those in the OR, by benchmarking on the NVS-RGBD dataset that contains as few as three training views. NeRF-OR synthesizes images with a PSNR of 26.72dB, a 1.7% improvement over state of the art. Our results show that NeRF-OR allows for novel view synthesis with videos captured by a small number of cameras with entirely different vantage points, which is the typical camera setting in the OR. Code is available via: github.com/Beerend/NeRF-OR .

Wear particles image enhancement using long short-term memory 3D recurrent reconstruction neural network (LSTM 3D-R2N2)

3D modeling of wear particles has proven to be a useful tool for monitoring mechanical failure conditions. In this work, a new method for 3D reconstruction of wear particles in uncontaminated oil (healthy oil) and contaminated oil (used oil) was proposed. The image acquisition device can capture multi-view images of moving wear particles in both healthy and used oil by using the reflected light. The images were pretreated first, and the image color inversion was conducted using the Pillow library. The pretreated wear particle images were used for 3D reconstruction using long short-term memory 3D recurrent reconstruction neural network. The current results were verified against existing results, and good agreement can be found. It can be concluded that we can reconstruct the similar 3D wear particle results with fewer images by comparison with other methods. Specifically, only 4–6 image samples were used for the 3D reconstruction of wear particles, and at least 8 image samples were needed for other existing reports.