High-quality 3D Reconstruction Research Articles

Georecon: a coarse-to-fine visual 3D reconstruction approach for high-resolution images with neural matching priors

ABSTRACT Visual 3D reconstruction enables rebuilding 3D scenes from captured images, serving as a fundamental data source for digital earth modeling and intelligent cities. In the foundational step, recent methods leverage learning-based descriptors for image registration and achieve tremendous advances in precision and robustness. However, these methods inevitably execute down sampling towards high-resolution images to fit the needs of neural networks, which leads to precision degradation of feature localization and matching. Thus, we propose GeoRecon: a novel coarse-to-fine visual 3D reconstruction method that optimally utilizes high-resolution images for high-quality visual 3D reconstruction. Firstly, the coarse stage conducts coarse reconstruction from downsampled images by performing neural matching with geometric priors. Secondly, we define the fine-grained stage, proposing a GPU-based algorithm for generating image-patch correspondences based on the neural matching priors to perform fine-grained image registration. Finally, based on the optimized camera poses under this coarse-to-fine paradigm, progressive dense reconstruction leveraging efficient neural radiance fields is proposed to accomplish the high-quality MVS reconstruction. Comparative experiments across various scenarios demonstrate the proposed method’s superior precision, robustness, and reconstruction quality.

Fringe photometric stereo

In fringe projection profilometry (FPP), a high spatial frequency of fringe typically indicates powerful error suppression capability; however, it complicates phase unwrapping, necessitating a greater number of patterns and thus compromising the real-time performance of scanning. In this letter, we report a fringe photometric stereo (FPS) to completely bypass phase unwrapping for recovering the continuous 3D surface. We reveal the linear mapping relationship between the phase gradient and depth gradient, thereby facilitating the computation of depth gradients from phase gradients. Thus, we can employ depth gradients to reconstruct high-quality 3D profiles. Experimental results demonstrate that our FPS surpasses traditional phase-shifting profilometry in mitigating Gaussian noise. Our approach enables high-quality and unambiguous 3D reconstruction using single-frequency fringe patterns, relying solely on a camera and a projector without the need for dual, multiple, or light field cameras, thereby paving a path to high-speed and precision 3D imaging.

Explicit 3D reconstruction from images with dynamic graph learning and rendering-guided diffusion

High-quality 3D reconstruction is becoming increasingly important in a variety of fields. Recently, implicit representation methods have made significant progress in image-based 3D reconstruction. However, these methods tend to yield entangled neural representations which lack support for standard 3D pipelines, and their reconstruction results usually experience a sharp drop in quality when input views are reduced. To obtain high-quality 3D content, we propose an explicit 3D reconstruction method that directly extracts textured meshes from images and remains robust using reduced input views. Our central components include a dynamic graph convolutional network (GCN) and a rendering-guided diffusion model. The dynamic GCN aims to improve mesh reconstruction quality by effectively aggregating features from vertex neighborhoods. The aggregation is accelerated through sampling geometric-related neighbors with different SDF signs, which gradually converges in quantity during training. The rendering-guided diffusion model learns prior distributions for unseen regions to improve reconstruction performance using sparse-view inputs. It uses the rendered image under an interpolated camera pose as conditioned input and its diffusion strength can be controlled with the rendering loss of explicit reconstruction. In addition, the rendering-guided diffusion model can be jointly trained to generate plausible novel views with 3D consistency. Experiments demonstrate that our method can produce high-quality explicit reconstruction results and maintain realistic reconstruction using sparse-view inputs.

IW-NeRF: Using Implicit Watermarks to Protect the Copyright of Neural Radiation Fields

The neural radiance field (NeRF) has demonstrated significant advancements in computer vision. However, the training process for NeRF models necessitates extensive computational resources and ample training data. In the event of unauthorized usage or theft of the model, substantial losses can be incurred by the copyright holder. To address this concern, we present a novel algorithm that leverages the implicit neural representation (INR) watermarking technique to safeguard NeRF model copyrights. By encoding the watermark information implicitly, we integrate its parameters into the NeRF model’s network using a unique key. Through this key, the copyright owner can extract the embedded watermarks from the NeRF model for ownership verification. To the best of our knowledge, this is the pioneering implementation of INR watermarking for the protection of NeRF model copyrights. Our experimental results substantiate that our approach not only offers robustness and preserves high-quality 3D reconstructions but also ensures the flawless (100%) extraction of watermark content, thereby effectively securing the copyright of the NeRF model.

Polarimetric Helmholtz Stereopsis.

Helmholtz stereopsis (HS) exploits the reciprocity principle of light propagation (i.e., the Helmholtz reciprocity) for 3D reconstruction of surfaces with arbitrary reflectance. In this paper, we present the polarimetric Helmholtz stereopsis (polar-HS), which extends the classical HS by considering the polarization state of light in the reciprocal paths. With the additional phase information from polarization, polar-HS requires only one reciprocal image pair. We derive the reciprocity relationship of Mueller matrix and formulate new reciprocity constraint that takes polarization state into account. We also utilize polarimetric constraints and extend them to the case of perspective projection. For the recovery of surface depths and normals, we incorporate reciprocity constraint with diffuse/specular polarimetric constraints in a unified optimization framework. For depth estimation, we further propose to utilize the consistency of diffuse angle of polarization. For normal estimation, we develop a normal refinement strategy based on degree of linear polarization. Using a hardware prototype, we show that our approach produces high-quality 3D reconstruction for different types of surfaces, ranging from diffuse to highly specular.

Iterative intelligent algorithm-based on CT three-dimensional reconstruction in endoscopic thyroidectomy and perioperative comprehensive nursing

This research was aimed to explore the application of computed tomography (CT) 3D reconstruction technology based on iterative intelligent algorithm in endoscopic thyroidectomy (ET) and perioperative comprehensive nursing. A total of 140 patients with thyroid mass receiving ET were randomly divided into controls (conventional ultrasound combined with perioperative routine nursing) and experimental group (CT 3D reconstruction based on iterative intelligent algorithm combined with perioperative comprehensive nursing), 70 cases in each group. The iterative intelligent algorithm used in this article is an image processing algorithm based on deep learning. Its goal is to generate more accurate and clear 3D reconstructed images by learning and optimizing the original CT images for multiple iterations. The detailed process of the algorithm is as follows: Data preprocessing, Feature extraction and representation, Iterative optimization, 3D reconstruction image generation. Finally, after multiple iterations of optimization, the algorithm will generate high-quality 3D reconstruction images, which can clearly show the anatomical structure of the thyroid mass and the surrounding tissue of the lesion. In order to evaluate the effectiveness of the algorithm, this article recorded the maximum vertical diameter, maximum left and right diameter, and maximum anteroposterior diameter of thyroid masses measured by the two imaging methods and solid specimens. In addition, the postoperative pain of patients was evaluated by visual analogue scale (VAS), and the incidence of complications and nursing satisfaction were counted.

Radiance-field holography for high-quality 3D reconstruction

Conventional light field-based computer-generated holograms, such as holographic stereograms, can provide nature parallax and accurate occlusion relation but are restricted by the inherent tradeoff concerning the angular and spatial resolution of the reconstructed optical field. In contrast, Wigner distribution function based light field generated holograms, also called non-hogel-based holograms, release this limitation by using basic ray-to-wave transformation frameworks. However, at present, this transformation framework is accurate only for generating holograms from orthogonal light field images of 3D scenes with a limited depth range and size. Thus, the radiance field holography method is proposed in this paper to generate high-quality 3D holograms for arbitrary scenes that can achieve high-resolution reconstruction within a deep depth range. In our method, a proposed radiance field is constructed from the perspective light field, which consists of multiple reference planes to resample and redistribute the light rays to represent the diffracted optical field of the 3D objects according to our developed sampling criterion. The complex optical field of the 3D object can be synthesized from these reference planes based on inverse Wigner transform and Fresnel propagation. As a result, both the numerical simulations and the optical experiments are implemented, and it is demonstrated that the high-quality 3D reconstruction of the synthesized hologram is achieved, which reflects the highest reconstruction resolution among the existing methods, the large depth range with accurate refocus and defocus, and the natural view-dependent effects.

A Preconditioned Landweber iteration-based Bundle adjustment for large-scale 3D reconstruction

Bundle Adjustment (BA) plays a crucial role in 3D reconstruction methods and Structure from Motion (SFM) algorithms, as it involves nonlinear minimization of reprojection error. The efficiency and accuracy of BA solutions are of paramount importance for achieving high-quality 3D reconstructions. The solution of the normal equations derived from nonlinear minimization is typically the most time-consuming step in BA and directly impacts the reconstruction accuracy. In this paper, we propose a Preconditioned Landweber Iteration solver for the normal equations. This solver utilizes a preconditioner and a corresponding relaxation strategy to improve the conditioning of the normal equations, leading to improved convergence speed and accuracy of the iteration. Compared to traditional direct and iterative algorithms, our approach significantly enhances the accuracy and solution time of BA. Experimental results demonstrate that our proposed solver outperforms the Schur complement solver in terms of both speed and accuracy when solving the normal equations. This improvement contributes to further acceleration and accuracy in large-scale 3D reconstruction, particularly for challenging large-scale intensive scenarios.

Hybrid-MVS: Robust Multi-View Reconstruction With Hybrid Optimization of Visual and Depth Cues

Consumer-level RGB-D cameras have been widely used for dense 3D reconstruction of scenes. Especially for textureless or non-lambertian surfaces, consumer RGB-D cameras can ensure completeness of the reconstructed models at a low cost. However, the reconstruction quality relies heavily on the accuracy of the depth sensors. Digital cameras are also used popularly for capturing high-resolution pictures to achieve high-quality dense reconstruction of the scenes, but cannot handle textureless or non-lambertian regions well due to the visual ambiguity problem. To ensure both completeness and accuracy of the reconstructed 3D models, we propose a hybrid multi-view reconstruction pipeline named Hybrid-MVS, which combines the high-resolution images taken by a digital camera and the low-resolution RGB-D frames captured by a consumer RGB-D camera for robust reconstruction of complicated scenes with challenging textureless and non-lambertian surfaces. Unlike most existing multi-sensor systems which require explicit hardware calibration and synchronization of various sensors, the calibration and synchronization problems between the digital camera and RGB-D camera are implicitly solved for compositing reliable depth prior of the digital images in our pipeline. Especially, we propose a hybrid MVS framework for robust PatchMatch stereo and Delaunay meshing, which tightly couples both visual cues given by the digital images and depth cues from the RGB-D frames to maximize the complementary advantages. The experiments with quantitative and qualitative evaluations demonstrate the effectiveness of the proposed Hybrid-MVS framework, which can successfully achieve high-quality 3D reconstruction of complicated natural scenes with robustness to weakly textured and non-lambertian areas.

Non-iterative reconstruction of interferenceless coded aperture correlation holography enabled high quality three-dimensional imaging

Interferenceless coded aperture correlation holography (I-COACH) is a rapidly developing indirect three-dimensional (3D) imaging technique. However the ideal point spread function of imaging system is not obtained in most cases, the object hologram is usually correlated with the system response of a limited-size pinhole during the reconstruction process. As a result, the quality of reconstructed images is contaminated by noise. In this paper, the system response of a limited-size pinhole is regarded as the convolution of the pinhole (as an object) intensity and the ideal point spread function of I-COACH, a mathematical analysis of the background noise is conducted and a general high-quality reconstruction mathematic model of I-COACH system is derived when a limited-size pinhole is used to record point spread hologram, in which the reconstructions and noise is separated. Then a non-iterative cross-correlation reconstruction algorithm has been developed here to achieve high-quality 3D reconstruction. The proposed method has been implemented on a regular I-COACH system in both transmission as well as reflection illumination modes to demonstrate the imaging ability of our method. The experimental results show that high quality reconstructions can be achieved by our proposal with a single-exposure object hologram.

Increasing visitors attention with introductory portal technology to complex cultural sites

We investigate the effectiveness of visually impacting interfaces which were located next to the entrance of the San Martino Charterhouse in Naples (Italy), by using high quality 3D reconstructions annotated with semantic information. Semantic data were used to develop a gesture based orienting interface, for the generation of camera movements and pointing gestures for a 3D avatar. Using the Visitor Employed Photography protocol, we collected data about visitors noticing details and discovered that those exposed to the informative systems were able to detect more details than visitors who did not experience it.

Multi-UAV Cooperative and Continuous Path Planning for High-Resolution 3D Scene Reconstruction

Unmanned aerial vehicles (UAVs) are extensively employed for urban image captures and the reconstruction of large-scale 3D models due to their affordability and versatility. However, most commercial flight software lack support for the adaptive capture of multi-view images. Furthermore, the limited performance and battery capacity of a single UAV hinder efficient image capturing of large-scale scenes. To address these challenges, this paper presents a novel method for multi-UAV continuous trajectory planning aimed at the image captures and reconstructions of a scene. Our primary contribution lies in the development of a path planning framework rooted in task and search principles. Within this framework, we initially ascertain optimal task locations for capturing images by assessing scene reconstructability, thereby enhancing the overall quality of reconstructions. Furthermore, we curtail energy costs of trajectories by allocating task sequences, characterized by minimal corners and lengths, among multiple UAVs. Ultimately, we integrate considerations of energy costs, safety, and reconstructability into a unified optimization process, facilitating the search for optimal paths for multiple UAVs. Empirical evaluations demonstrate the efficacy of our approach in facilitating collaborative full-scene image captures by multiple UAVs, achieving low energy costs while attaining high-quality 3D reconstructions.

CIGNet: Category-and-Intrinsic-Geometry Guided Network for 3D coarse-to-fine reconstruction

3D object reconstruction from arbitrary view intensity images is a challenging but meaningful research topic in computer vision. The main limitations of existing approaches are that they lack complete and efficient prior information and might not be able to deal with serious occlusion or partial observation of 3D objects, which may produce incomplete and unreliable reconstructions. To reconstruct structure and recover missing or unseen parts of objects, category prior and intrinsic geometry relation are particularly useful and necessary during the 3D reconstruction process. In this paper, we propose Category-and-Intrinsic-Geometry Guided Network (CIGNet) for 3D coarse-to-fine reconstruction from arbitrary view intensity images by leveraging category prior and intrinsic geometry relation. CIGNet combines a category prior guided reconstruction module with an intrinsic geometry relation guided refinement module. In the first reconstruction module, we leverage semantic class context by adding a supervision term over object categories to output coarse reconstructed results. In the second refinement module, we model the coarse 3D volumetric data as 2D slices and consider intrinsic geometry relations between them to design graph structures of coarse 3D volumes to finish the graph-based refinement. CIGNet can accomplish high-quality 3D reconstruction tasks by exploring the intra-category characteristics of objects as well as the intrinsic geometry relations of each object, both of which serve as useful complements to the visual information of images, in a coarse-to-fine fashion. Extensive quantitative and qualitative experiments on a synthetic dataset ShapeNet and real-world datasets Pix3D, Statue Model Repository, and BlendedMVS indicate that CIGNet outperforms several state-of-the-art methods in terms of accuracy and detail recovery.

Back to life: Techniques for developing high-quality 3D reconstructions of plants and animals from digitized specimens.

Expanded use of 3D imaging in organismal biology and paleontology has substantially enhanced the ability to visualize and analyze specimens. These techniques have improved our understanding of the anatomy of many taxa, and the integration of downstream computational tools applied to 3D datasets have broadened the range of analyses that can be performed (e.g., finite element analyses, geometric morphometrics, biomechanical modeling, physical modeling using 3D printing). However, morphological analyses inevitably present challenges, particularly in fossil taxa where taphonomic or preservational artifacts distort and reduce the fidelity of the original morphology through shearing, compression, and disarticulation, for example. Here, we present a compilation of techniques to build high-quality 3D digital models of extant and fossil taxa from 3D imaging data using freely available software for students and educators. Our case studies and associated step-by-step supplementary tutorials present instructions for working with reconstructions of plants and animals to directly address and resolve common issues with 3D imaging data. The strategies demonstrated here optimize scientific accuracy and computational efficiency and can be applied to a broad range of taxa.

Laser ablation tomography (LATscan) as a new tool for anatomical studies of woody plants.

Traditionally, botanists study plant anatomy by carefully sectioning samples, histological staining to highlight tissues of interests, then imaging slides under light microscopy. This approach generates significant details; however, this workflow is laborious, particularly in woody vines (lianas) with heterogeneous anatomies, and ultimately yields two-dimensional (2D) images. Laser ablation tomography (LATscan) is a high-throughput imaging system that yields hundreds of images per minute. This method has proven useful for studying the structure of delicate plant tissues; however, its utility in understanding the structure of woody tissues is underexplored. We report LATscan-derived anatomical data from several stems of lianas (c. 20 mm) of seven species and compare these results with those obtained through traditional anatomical techniques. LATscan successfully allows the description of tissue composition by differentiating cell type, size, and shape, but also permits the recognition of distinct cell wall composition (e.g. lignin, suberin, cellulose) based on differential fluorescent signals on unstained samples. LATscan generate high-quality 2D images and 3D reconstructions of woody plant samples; therefore, this new technology is useful for both qualitative and quantitative analyses. This high-throughput imaging technology has the potential to bolster phenotyping of vegetative and reproductive anatomy, wood anatomy, and other biological systems.

An Efficient Visual SfM Framework Using Planar Markers

Structure-from-Motion (SfM) is a prevalent solution for vision measurement. Traditional SfM methods rely on sparse feature matching to establish visual associations between different views. However, both the incremental-based and global-based SfM show sensitiveness to outliers generated by existing hand-crafted features under challenging environments. Planar marker-assisted SfM is an alternative solution that employs artificial landmarks to help robust pose estimation. Nevertheless, using planar markers also exists the planar pose ambiguity problem, which means there may be more than one reasonable solution for pose estimation; it is a great challenge for camera pose estimation. This paper proposes an efficient planar marker-assisted global SfM method and resolves the planar pose ambiguity problem by examining the rotation consistency across multiple views. In particular, we construct two complementary graphs and present a graph filter method to filter out the views with pose ambiguity according to our designed rotation consistency checking. Afterward, we perform sparse feature matching following the proposed graph structure and make a pose-fixed bundle adjustment for high-quality 3D reconstruction. Extensive experiments have been conducted on synthetic and real datasets, and the results verify the effectiveness of the proposed method. Our method outperforms several state-of-the-art methods with encouraging efficiency, especially in challenging scenes.

Three-dimensional reconstruction of moving HDR object based on PSP

High quality 3D reconstruction of HDR object can be achieved by projecting extra fringe patterns with different intensity. However, it requires the object to be kept static. The object motion not only introduces phase shift among the fringe patterns, but also changes the position of the overexposure pixels. This paper proposes a new method to reconstruct the moving HDR object by utilizing the features of motion. For the specific point on the object, overexposure will not happen among all the fringe patterns as the overexposure position is changed with the motion. The unsaturation fringe patterns are selected from the captured fringe patterns and the correct phase information is retrieved based on unequal phase shift algorithm. The effectiveness of the proposed algorithm is verified by the experiments.

Fusion-based high-quality polarization 3D reconstruction

The fusion of shape from polarization (SFP) and depth sensors is an effective method for high-quality 3D reconstruction. However, accurate registration of images from different sensors is a difficult problem. In this paper, we propose a 3D reconstruction method based on the fusion of SFP and polarization-modulated ranging (PMR) for the first time, where only a single image sensor is utilized to acquire both polarized images and depth data. Therefore, the challenging image registration problem can be avoided. This method is based on the following observation: SFP can retrieve inaccurate objects’ 3D shapes with fine textures, whereas PMR can provide coarse but accurate absolute depths, resulting in a good necessity of fusing these two modalities. To this end, we propose two fusion models: a joint azimuth estimation model to obtain a fused azimuth angle with π−ambiguity corrected, and a joint zenith estimation model to estimate an accurate fused zenith angle. Finally, the reconstructed depth is integrated from the fused azimuth and zenith angles, which can provide accurate absolute depth with fine textures. Extensive experiments have been conducted to verify the efficiency of the proposed method over other state-of-the-art methods besides its robustness to the targets with different kinds of features, demonstrating its broad application prospects in 3D reconstruction.

X23D—Intraoperative 3D Lumbar Spine Shape Reconstruction Based on Sparse Multi-View X-ray Data

Visual assessment based on intraoperative 2D X-rays remains the predominant aid for intraoperative decision-making, surgical guidance, and error prevention. However, correctly assessing the 3D shape of complex anatomies, such as the spine, based on planar fluoroscopic images remains a challenge even for experienced surgeons. This work proposes a novel deep learning-based method to intraoperatively estimate the 3D shape of patients’ lumbar vertebrae directly from sparse, multi-view X-ray data. High-quality and accurate 3D reconstructions were achieved with a learned multi-view stereo machine approach capable of incorporating the X-ray calibration parameters in the neural network. This strategy allowed a priori knowledge of the spinal shape to be acquired while preserving patient specificity and achieving a higher accuracy compared to the state of the art. Our method was trained and evaluated on 17,420 fluoroscopy images that were digitally reconstructed from the public CTSpine1K dataset. As evaluated by unseen data, we achieved an 88% average F1 score and a 71% surface score. Furthermore, by utilizing the calibration parameters of the input X-rays, our method outperformed a counterpart method in the state of the art by 22% in terms of surface score. This increase in accuracy opens new possibilities for surgical navigation and intraoperative decision-making solely based on intraoperative data, especially in surgical applications where the acquisition of 3D image data is not part of the standard clinical workflow.

LANet: Stereo matching network based on linear-attention mechanism for depth estimation optimization in 3D reconstruction of inter-forest scene.

The 3D reconstruction of forests provides a strong basis for scientific regulation of tree growth and fine survey of forest resources. Depth estimation is the key to the 3D reconstruction of inter-forest scene, which directly determines the effect of digital stereo reproduction. In order to solve the problem that the existing stereo matching methods lack the ability to use environmental information to find the consistency of ill-posed regions, resulting in poor matching effect in regions with weak texture, occlusion and other inconspicuous features, LANet, a stereo matching network based on Linear-Attention mechanism is proposed, which improves the stereo matching accuracy by effectively utilizing the global and local information of the environment, thereby optimizing the depth estimation effect. An AM attention module including a spatial attention module (SAM) and a channel attention module (CAM) is designed to model the semantic relevance of inter-forest scenes from the spatial and channel dimensions. The linear-attention mechanism proposed in SAM reduces the overall complexity of Self-Attention from O(n2) to O(n), and selectively aggregates the features of each position by weighted summation of all positions, so as to learn rich contextual relations to capture long-range dependencies. The Self-Attention mechanism used in CAM selectively emphasizes interdependent channel maps by learning the associated features between different channels. A 3D CNN module is optimized to adjust the matching cost volume by combining multiple stacked hourglass networks with intermediate supervision, which further improves the speed of the model while reducing the cost of inferential calculation. The proposed LANet is tested on the SceneFlow dataset with EPE of 0.82 and three-pixel-error of 2.31%, and tested on the Forest dataset with EPE of 0.68 and D1-all of 2.15% both of which outperform some state-of-the-art methods, and the comprehensive performance is very competitive. LANet can obtain high-precision disparity values of the inter-forest scene, which can be converted to obtain depth information, thus providing key data for high-quality 3D reconstruction of the forest.