Volumetric Reconstruction Research Articles

Unravelling technological behaviors through core reduction intensity. The case of the early Protoaurignacian assemblage from Fumane Cave

This paper investigates core reduction intensity in the early Protoaurignacian lithic assemblage from Fumane Cave in northeastern Italy. Reduction intensity serves as a key tool to characterize blank selection strategies, raw material management, and the variability of knapping strategies throughout the reduction sequence by reconstructing the operatory field of core assemblages. Finally, it also aids in addressing the relationship between blades and bladelets, providing valuable insights into the behavioral and chrono-cultural significance of laminar productions within the Aurignacian technocomplex. To achieve these research goals, experimental work employing 3D scanning technology was conducted. This facilitated the comparison of different methods and variables for measuring reduction intensity, including the percentage of non-cortical surface, the Scar Density Index (SDI), and a novel adaptation of the Volumetric Reconstruction Method (VRM). Results demonstrate the effectiveness and potential of adapting the VRM for the study of reduction intensity in Upper Paleolithic laminar cores, and the provided R scripts and datasets will enable this method to be applied to other contexts with minimal need for modification to the workflow. Analysis of reduction intensity measures applied to the Protoaurignacian assemblage from Fumane Cave reveals slight variations based on factors such as the abundance and proximity of selected raw materials for blank production. Notably, the most prevalent raw material variety, the Maiolica, yields a higher number of less reduced cores, while reduction levels across all cores discarded at the site remain relatively high. The observed variability in the operatory field and the interrelation between blade and bladelet productions underscore the complexity and flexibility of Protoaurignacian behavior. This inherent complexity challenges any definitive separation between the operatory fields of blade and bladelet productions. These findings are particularly important to emphasize the importance of considering reduction intensity when examining technological variability and human behavior in Aurignacian studies. The proposed adaptation of the VRM and the effective combination with other measures of reduction, promises to allow future research to incorporate reduction intensity as a vital temporal component within studies on stone tool production. This integration offers a pathway to enhancing our understanding of the adaptive behaviors exhibited by Homo sapiens across diverse ecological settings and provides a clearer framework for better framing the development of the Upper Paleolithic.

Robust Fusion of Multi-Source Images for Accurate 3D Reconstruction of Complex Urban Scenes

Integrated reconstruction is crucial for 3D modeling urban scenes using multi-source images. However, large viewpoint and illumination variations pose challenges to existing solutions. A novel approach for accurate 3D reconstruction of complex urban scenes based on robust fusion of multi-source images is proposed. Firstly, georeferenced sparse models are reconstructed from the terrestrial and aerial images using GNSS-aided incremental SfM, respectively. Then, cross-platform match pairs are selected based on point-on-image observability. The terrestrial and aerial images are robustly matched based on the selected match pairs to generate cross-platform tie points. Thirdly, the tie points are triangulated to derive cross-platform 3D correspondences. The 3D correspondences are refined using a novel outlier detection method. Finally, the terrestrial and aerial sparse models are merged based on the refined correspondences, and the integrated model is globally optimized to obtain an accurate reconstruction of the scene. The proposed methodology is evaluated on five benchmark datasets, and extensive experiments are performed. The proposed pipeline is compared with a state-of-the-art methodology and three widely used software packages. Experimental results demonstrate that the proposed methodology outperforms the other pipelines in terms of robustness and accuracy.

Deep learning based coherence holography reconstruction of 3D objects.

We propose a reconstruction method for coherence holography using deep neural networks. cGAN and U-NET models were developed to reconstruct 3D complex objects from recorded interferograms. Our proposed methods, dubbed deep coherence holography (DCH), predict the non-diffracted fields or the sub-objects included in the 3D object from the captured interferograms, yielding better reconstructed objects than the traditional analytical imaging methods in terms of accuracy, resolution, and time. The DCH needs one image per sub-object as opposed to N images for the traditional sin-fit algorithm, and hence the total reconstruction time is reduced by N×. Furthermore, with noisy interferograms the DCH amplitude mean square reconstruction error (MSE) is 5×104× and 104× and phase MSE is 102× and 3×103× better than Fourier fringe and sin-fit algorithms, respectively. The amplitude peak signal to noise ratio (PSNR) is 3× and 2× and phase PSNR is 5× and 3× better than Fourier fringe and sin-fit algorithms, respectively. The reconstruction resolution is the same as sin-fit but 2× better than the Fourier fringe analysis technique.

Single Image 3D Reconstruction Algorithm Based on Parallel Double-Branch Pyramid Stacking Network

Limited visual information contained in single images and complex motion models of objects may lead to severe fragmentation and confusion of model backbone in the 3D reconstruction of objects. In order to fully extract feature information in a single image and reduce noise interference caused by environmental factors in 3D reconstruction, a Parallel Dual-Branch Pyramid Stacking Network (PDB-PSN) model is proposed. A parallel dual-branch network model is used to construct an encoder-decoder framework based on cascaded feature extraction, encode/decode high and low-resolution features in a single image, and then convert from 2D to 3D through implicit functions to achieve the 3D reconstruction of target objects in a single image. A cascaded feature extraction network is used as a low-resolution feature extraction network to extract global features of objects. In the high-resolution feature extraction branch, three concatenated hourglass networks and dilated convolutions are used in the hourglass network to increase the receptive field and obtain more global information in order to maintain the integrity of the reconstructed object limbs. A threshold processing module is set to remove irrelevant information and ensure the integrity of global information, and meanwhile, to reduce the interference of irrelevant noise information. Simulation experiments on a self-built terracotta dataset show that the PDB-PSN model can completely reconstruct the 3D model in a single image and effectively eliminate model fragmentation in the reconstruction results.

Multi-Scale Label-Free Human Brain Imaging with Integrated Serial Sectioning Polarization Sensitive Optical Coherence Tomography and Two-Photon Microscopy.

The study of aging and neurodegenerative processes in the human brain requires a comprehensive understanding of cytoarchitectonic, myeloarchitectonic, and vascular structures. Recent computational advances have enabled volumetric reconstruction of the human brain using thousands of stained slices, however, tissue distortions and loss resulting from standard histological processing have hindered deformation-free reconstruction. Here, the authorsdescribe an integrated serial sectioning polarization-sensitive optical coherence tomography (PSOCT) and two photon microscopy (2PM) system to provide label-free multi-contrast imaging of intact brain structures, including scattering, birefringence, and autofluorescence of human brain tissue. The authors demonstrate high-throughput reconstruction of 4 × 4 × 2cm3 sample blocks and simple registration between PSOCT and 2PM images that enablecomprehensive analysis of myelin content, vascular structure, and cellular information. The high-resolution 2PM images provide microscopic validation and enrichment of the cellular information provided by the PSOCT optical properties on the same sample, revealing the densely packed fibers, capillaries, and lipofuscin-filled cell bodies in the cortex and white matter. It is shown that theimaging system enables quantitative characterization of various pathological features in aging process, including myelin degradation, lipofuscin accumulation, and microvascular changes, which opens up numerous opportunities in the study of neurodegenerative diseases in the future.

Digital twin models of replicative ground stones: insight into simulating usage of Upper Paleolithic tools

This work presents the first attempt to create a physics-based digital twin model for predictive analysis of damage evolution during the use of ground stone tools (GSTs) in transformative tasks, encompassing the processing of raw resources for nutritional and non-alimentary purposes. The proposed methodology introduces a digital twin of the GSTs developed from 3D models generated using a photogrammetric technique based on Structure-from-Motion and Multi-View Stereo reconstruction. These models serve as the foundation for the development of the finite element (FE)-based digital twin model of the GSTs that exploits a contact formulation and the phase-field approach to simulate tool damage during pounding and grinding tasks. Defining the initial relative positions of the stones, their mechanical behaviour, and controlling the movement of the active stone in a way as close as possible to the real one, the digital twin model has been devised to evaluate how the surface damage is affected by perturbations in the loading conditions. The simulated damage is compared with the surface traces observed from experiments. The developed digital twin model aims at demonstrating its potentials for the GSTs investigations, as a supporting tool for experiments and for simulated tests on the archaeological records.

Single-stripe-enhanced spacetime stereo reconstruction for concrete defect identification

Few studies explored the 3D reconstruction of concrete defects, and existing methods often suffer from noise interference or impracticality in real-world applications. This study proposes a novel single-stripe-enhanced spacetime stereo reconstruction method that combines a laser liner for structured light generation and a stereo vision camera. This method analyses the morphology and intensity variation of laser stripes from images captured from single scans, achieving accurate defect reconstructions. It is designed for handheld scenarios with image re-alignment eliminating the vibration caused by handheld operation. The morphology of laser stripes at crack sections is analysed for robust fine crack reconstruction. Active frame searching is introduced to improve the calculation efficiency, enabling spacetime stereo to operate over hundreds of frames. A comparative study of the proposed method and two commonly-used stereo matching methods demonstrates its ability to mitigate the impact of handheld operation and generate 3D reconstructions that highlight potential crack regions.

Optimal Pre-processing of Laser Scanning Data for Indoor Scene Analysis and 3D Reconstruction of Building Models

The large-scale point cloud data representing buildings should be prudently pre-processed and down-sampled prior to post-data processing activities, such as data segmentation or classification and indoor scene analysis, and visualization of the processed results as three-dimensional (3D) digital models. However, current pre-processing tasks and down-sampling procedures encountered the challenge of preserving the geometric and semantic features of the scanned object and contributing to accurate 3D reconstruction due to the loss of significant features on buildings. Therefore, this paper proposed a framework that pre-processes unstructured laser scanning data to optimize the efficiency of data processing activities and the accuracy of the produced output, i.e., reconstructed 3D building models. The pre-processing framework includes building the topology of the unstructured dataset, down-sampling the raw data through an improved voxel-based approach, and performing indoor scene analysis such as normal estimation, point alignment, and data classification. The experimental results verified that the proposed framework efficiently pre-processes the input data and preserves the geometric characteristics of the digital models with high accuracy, thereby supporting reliable 3D reconstruction for various applications.

Surface defects 3D localization for fluorescent magnetic particle inspection via regional reconstruction and partial-in-complete point clouds registration

Accurate three-dimensional localization of defects plays a crucial role in the automated defect marking and grinding process. However, this task becomes increasingly intricate when precisely recovering the three-dimensional information of small defects due to the presence of noise and damage in point clouds generated through three-dimensional reconstruction. Simultaneously, separately reconstructing defects at different locations on the component would significantly impact efficiency. This study focuses on fluorescent magnetic particle inspection, with the aim of enhancing the accuracy of fine crack localization and multi-defects reconstruction efficiency. Its basic idea is to align the precise complete point cloud with the regionally reconstructed noisy point cloud of a component, achieving its spatial positioning within the visual system while eliminating noise and imperfections. We commence by discussing various methods for acquiring offline complete point clouds and their respective advantages and disadvantages. To enhance flexibility and minimize regional reconstruction noise, we have devised a multi-camera vision system and optimized a multi-view stereo reconstruction algorithm. We introduce a filtering function for culling back-facing points during the sparse point cloud projection process. The feasibility of our approach is empirically validated, with a particular focus on its precision and efficiency. Experimental results indicate that our method can execute within 5 s, with a relative error of less than 2 millimeters for 95% of the points and a root mean square error less than 1 millimeter. Our approach exhibits significant advantages in addressing noise issues associated with point cloud data generated through image-based 3D reconstruction processes.

Jrender: An efficient differentiable rendering library based on Jittor

Differentiable rendering has been proven as a powerful tool to bridge 2D images and 3D models. With the aid of differentiable rendering, tasks in computer vision and computer graphics could be solved more elegantly and accurately. To address challenges in the implementations of differentiable rendering methods, we present an efficient and modular differentiable rendering library named Jrender based on Jittor. Jrender supports surface rendering for 3D meshes and volume rendering for 3D volumes. Compared with previous differentiable renderers, Jrender exhibits a significant improvement in both performance and rendering quality. Due to the modular design, various rendering effects such as PBR materials shading, ambient occlusions, soft shadows, global illumination, and subsurface scattering could be easily supported in Jrender, which are not available in other differentiable rendering libraries. To validate our library, we integrate Jrender into applications such as 3D object reconstruction and NeRF, which show that our implementations could achieve the same quality with higher performance.

Retracted: Improved Multiview Decomposition for Single-Image High-Resolution 3D Object Reconstruction

Retracted: Improved Multiview Decomposition for Single-Image High-Resolution 3D Object Reconstruction

GAMNet: Global attention via multi‐scale context for depth estimation algorithm and application

AbstractDeep neural networks significantly enhance the accuracy of the stereo‐based disparity estimation. Some current methods suffer from inefficient use of the global context information, which will lead to the loss of structural details in ill‐posed areas. To this end, a novel stereo network GAMNet is designed, composed of three core components (GDA, MPF, DCA) for estimating the depth prediction in challenging real‐world environments. First, a lightweight attention module is presented, integrating the global semantic cues for every feature position across the channel and spatial dimensions. Next, the MPF module is constructed to fuse the diverse semantic and contextual information from different levels of the feature pyramid. Finally, cost volume is aggregated with a stacked encoder‐decoder composed of the DCA module and 3D convolutions, filtering the transmission of matching clues and capturing the rich global contexts. Substantial experiments conducted on KITTI 2012, KITTI 2015, SceneFlow, and Middlebury‐v3 datasets manifest that GAMNet surpasses preceding methods with contour‐preserving disparity predictions. In addition, the first 3D scene reconstruction linear evaluation strategy on spatial grasping points for the end‐to‐end stereo networks in an unsupervised mode is proposed, and it is deployed on the designed robot vision‐guided system. In application experiments, the method can produce densely high‐precision 3D reconstructions to implement the grasping task in complex real‐world scenes, and achieve excellent robust performance with competitive inference efficiency.

Joint analysis of macroscopic response and microfabric evolution of coral sand during earthquake-induced liquefaction

This study addresses the microfabric evolution of coral sand during earthquake-induced liquefaction using the joint tests of macroscopic shaking table and microscopic X-ray computed tomography. Based on the non-destructive scanning and image reconstruction in three-dimensional, the change in the grain–pore structure of sands under earthquakes is discussed. Furthermore, the special microstructural evolution of coral sand is studied by the contrast tests with Fujian sand. The results show that the coordination number of coral sand is larger than that of Fujian sand after liquefaction, which is consistent with that the reduction in acceleration of coral sand is smaller than that of the Fujian sand, and the coral sand foundation still has a larger shear strength due to the smaller excess pore pressure accumulation. Moreover, the contact index of Fujian sand and Reigate sand fluctuates in a similar range with the change of void ratio and coordination number compared with coral sand. This is due to the fact that the Fujian sand and Reigate sand are both general terrigenous sands (quartz sand), while coral sand has irregular particle shapes and larger surface friction caused by marine biogenesis. The test results deepen the understanding of the liquefaction mechanism of coral sand.

Fast-SNARF: A Fast Deformer for Articulated Neural Fields.

Neural fields have revolutionized the area of 3D reconstruction and novel view synthesis of rigid scenes. A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space. We propose a new articulation module for neural fields, Fast-SNARF, which finds accurate correspondences between canonical space and posed space via iterative root finding. Fast-SNARF is a drop-in replacement in functionality to our previous work, SNARF, while significantly improving its computational efficiency. We contribute several algorithmic and implementation improvements over SNARF, yielding a speed-up of 150×. These improvements include voxel-based correspondence search, pre-computing the linear blend skinning function, and an efficient software implementation with CUDA kernels. Fast-SNARF enables efficient and simultaneous optimization of shape and skinning weights given deformed observations without correspondences (e.g. 3D meshes). Because learning of deformation maps is a crucial component in many 3D human avatar methods and since Fast-SNARF provides a computationally efficient solution, we believe that this work represents a significant step towards the practical creation of 3D virtual humans.

Reconstruction of 3D Objects Based on Data from a Single View

Reconstruction of 3D Objects Based on Data from a Single View

Structure learning for 3D Point Cloud Generation from Single RGB Images

Abstract3D point clouds can represent complex 3D objects of arbitrary topologies and with fine‐grained details. They are, however, hard to regress from images using convolutional neural networks, making tasks such as 3D reconstruction from monocular RGB images challenging. In fact, unlike images and volumetric grids, point clouds are unstructured and thus lack proper parameterization, which makes them difficult to process using convolutional operations. Existing point‐based 3D reconstruction methods that tried to address this problem rely on complex end‐to‐end architectures with high computational costs. Instead, we propose in this paper a novel mechanism that decouples the 3D reconstruction problem from the structure (or parameterization) learning task, making the 3D reconstruction of objects of arbitrary topologies tractable and thus easier to learn. We achieve this using a novel Teacher‐Student network where the Teacher learns to structure the point clouds. The Student then harnesses the knowledge learned by the Teacher to efficiently regress accurate 3D point clouds. We train the Teacher network using 3D ground‐truth supervision and the Student network using the Teacher's annotations. Finally, we employ a novel refinement network to overcome the upper‐bound performance that is set by the Teacher network. Our extensive experiments on ShapeNet and Pix3D benchmarks, and on in‐the‐wild images demonstrate that the proposed approach outperforms previous methods in terms of reconstruction accuracy and visual quality.

Insights into solid dosage forms with nonlinear optical imaging

Microscopic chemical and solid-state structures and their changes in solid drugs and dosage forms can profoundly affect pharmaceutical performance and patient safety. Despite this, their detailed spatially-resolved analysis can be difficult or impossible with established analytical technologies. Multimodal non-linear optical imaging represents opportunities for sensitive and specific chemical and solid-state pharmaceutical imaging. Non-linear optical imaging encompasses several nonlinear optical phenomena, including coherent anti-Stokes Raman scattering (CARS), stimulated Raman scattering (SRS), and sum frequency/second harmonic generation (SFG/SHG). Imaging in 3D with (sub)micron resolution is rapid, non-destructive, possible in situ in aqueous media, and generally does not require prior sample preparation. This mini-review explores several applications of non-linear optical imaging for solid drug and dosage form analysis.

Real Time Volumetric MRI for MR-Guided 3D Motion Tracking via Sparse Prior-Augmented Neural Representation Learning

To reconstruct volumetric MRI from orthogonal cine acquisition aided by sparse priors of 2 static 3D MRI through implicit neural representation (NeRP) learning, with the goal of eliminating large-scale training datasets for data-driven sparse MRI reconstruction and supporting clinical workflow of real time 3D motion tracking during MR-guided radiotherapy. A multi-layer perceptron network was trained to learn the NeRP of a patient-specific MRI dataset, where the network takes 4D data coordinates of voxel locations and motion states as inputs and outputs corresponding voxel intensities. By first learning the NeRP of 2 static 3D MRI with different breathing motion states, prior knowledge of patient breathing motion was embedded into network weights through optimization. The prior knowledge was then augmented from 2 to 31 motion states by querying the optimized network at interpolated/extrapolated motion state coordinates. Starting from the prior-augmented network as an initialization point, the network was further trained using sparse samples of 2 orthogonal cine slices. The final volumetric reconstruction was obtained by querying the trained network at desired 3D spatial locations. We evaluated the proposed method using 5-minute volumetric MRI time series with 340 ms temporal resolution collected from 7 liver carcinoma patients. The time series was acquired using golden-angle radial MRI sequence and reconstructed through retrospective sorting. Two MRI with inhale and exhale states respectively were selected from the first 30 sec of the time series for prior embedding and augmentation. The remaining 4.5-min time series was used for volumetric reconstruction evaluation, where we retrospectively subsampled each MRI to 2 orthogonal slices and compared network-reconstructed images to ground truth images in terms of image quality and the capability of supporting 3D target motion tracking. Across the 7 patients evaluated, the peak signal to noise ratio between model reconstruction and ground truth was 54.66 ± 6.16 dB and the structural similarity index measure was 0.99 ± 0.01. Gross tumor volume (GTV) contours estimated by deforming a reference state MRI to model-reconstructed and ground truth MRI showed good consistency. The 95-percentile Hausdorff distance between GTV contours was 1.89 ± 1.13 mm, which is less than the voxel dimension. The mean GTV centroid position difference between ground truth and model estimation was less than 1 mm in all 3 orthogonal directions. Volumetric MRI from orthogonal cine acquisition with sparse priors is feasible by modeling prior knowledge through implicit neural representation learning. The model-reconstructed images showed sufficient accuracy in supporting 3D motion tracking of abdominal targets. By eliminating the need for large scale training datasets, the method promises to enable clinical implementation of 3D motion tracking for precision radiation therapy.

Hybrid-Supervised Deep Learning for Proton-Acoustic Reconstruction for 3D In Vivo Proton Dose Verification

Proton-acoustic (PA) image has shown great potential to provide real-time 3D dose verification of proton therapy. However, the PA image quality suffers from severe limited view artifacts, which significantly impairs its accuracy for dose verification. In this study, we developed a hybrid-supervised deep learning method for PA reconstruction to address the limited-view issues. Our method consists of two stages. In the first stage, a transformer-based network was proposed to reconstruct initial pressure maps from protoacoustic signals. The network was first trained using supervision by the iteratively reconstructed pressure map and then fine-tuned using transfer learning and self-supervision based on the data fidelity constraint. In the second stage, the PA image was further enhanced by a 3D U-net. The final PA images were converted to dose maps using conversion coefficients derived from CT images. Data from 126 prostate cancer patients treated by proton therapy were collected under an IRB protocol and were split into 86 and 40 patients for model training and testing, respectively. Data of each patient contains the planning CT scan, the corresponding clinical treatment plan, and the dose map calculated by commercial software. The radiofrequency signals were generated by performing proton acoustic simulation based on CT images and the ground truth pressure map derived from the treatment plan. An ultrasound detector matrix with 64 × 64 size and 500kHz central frequency was simulated under the perineum to acquire the signals in the prostate area. In the testing results, the method's accuracy was evaluated using Root-mean-squared-error (RMSE) and structural-similarity-index-measure (SSIM) between the reconstructed and ground truth pressure map and dose distribution. Testing results showed that the reconstructed pressure map achieved an average RMSE/SSIM of 0.0292/0.96, demonstrating excellent 3D information with details. Dose maps derived from the pressure map achieved an average RMSE/SSIM of 0.018/0.99 with a gamma index of 94.7% and 95.7% for 1%/3 mm and 1%/5 mm criteria compared to the ground truth dose maps. The reconstruction time was 6s, which can be further reduced using GPU. Our study achieves start-of-the-art performance in the challenging task of direct reconstruction from limited-view radiofrequency signals, demonstrating the great promise of PA imaging as a highly efficient and accurate tool for in-vivo 3D proton dose verification. Such high-precision 3D online dose verification can substantially reduce the range uncertainties of proton therapy to significantly improve its precision and outcomes.

Camera and LiDAR Fusion for Urban Scene Reconstruction and Novel View Synthesis via Voxel-Based Neural Radiance Fields

3D reconstruction of urban scenes is an important research topic in remote sensing. Neural Radiance Fields (NeRFs) offer an efficient solution for both structure recovery and novel view synthesis. The realistic 3D urban models generated by NeRFs have potential future applications in simulation for autonomous driving, as well as in Augmented and Virtual Reality (AR/VR) experiences. Previous NeRF methods struggle with large-scale, urban environments. Due to the limited model capability of NeRF, directly applying them to urban environments may result in noticeable artifacts in synthesized images and inferior visual fidelity. To address this challenge, we propose a sparse voxel-based NeRF. First, our approach leverages LiDAR odometry to refine frame-by-frame LiDAR point cloud alignment and derive accurate initial camera pose through joint LiDAR-camera calibration. Second, we partition the space into sparse voxels and perform voxel interpolation based on 3D LiDAR point clouds, and then construct a voxel octree structure to disregard empty voxels during subsequent ray sampling in the NeRF, which can increase the rendering speed. Finally, the depth information provided by the 3D point cloud on each viewpoint image supervises our NeRF model, which is further optimized using a depth consistency loss function and a plane constraint loss function. In the real-world urban scenes, our method significantly reduces the training time to around an hour and enhances reconstruction quality with a PSNR improvement of 1–2 dB, outperforming other state-of-the-art NeRF models.