Volumetric Reconstruction Research Articles

Real-time 3D shape measurement of dynamic scenes using fringe projection profilometry: lightweight NAS-optimized dual frequency deep learning approach.

Achieving real-time and high-accuracy 3D reconstruction of dynamic scenes is a fundamental challenge in many fields, including online monitoring, augmented reality, and so on. On one hand, traditional methods, such as Fourier transform profilometry (FTP) and phase-shifting profilometry (PSP), are struggling to balance measuring efficiency and accuracy. On the other hand, deep learning-based approaches, which offer the potential for improved accuracy, are hindered by large parameter amounts and complex structures less amenable to real-time requirements. To solve this problem, we proposed a network architecture search (NAS)-based method for real-time processing and 3D measurement of dynamic scenes with rate equivalent to single-shot. A NAS-optimized lightweight neural network was designed for efficient phase demodulation, while an improved dual-frequency strategy was employed coordinately for flexible absolute phase unwrapping. The experiment results demonstrate that our method can effectively perform 3D reconstruction with a reconstruction speed of 58fps, and realize high-accuracy measurement of dynamic scenes based on deep learning for what we believe to be the first time with the average RMS error of about 0.08 mm.

Establishing the Manner of Death: A 3D Reconstruction of a Case of Hanging

Establishing the manner of death is one of the most challenging tasks for forensic pathologists. We present the case of a 24-year-old woman found dead in the early morning on a flyover. The body was sitting on the ground with the back leaning against a wall. The neck was encircled by a white phone charger cable knotted to the staircase’s handrail. The victim had argued with her boyfriend and tried to jump out of his car while coming home from a wedding party the night before. After that, she left home alone with her phone charger in her hand. Due to self-harm behaviors, the first hypothesis was suicide by hanging. However, the ligature crossed immediately beneath the thyroid cartilage and encircled the neck twice horizontally; the two ends of the cable overlapped, forming a cross-over point in the front-right of the neck. Then, the ligature passed obliquely through the nape, gradually disappearing, forming a gap in the mark. The mark was sharply defined, stiff, yellow, and parchment-like. The investigators performed a three-dimensional scene reconstruction using the Trimble X7 Laser Scanner and the PC-Crash Multibody System. Even though the geometry of the ligature mark in the present case raised doubts about the manner of death, the three-dimensional reconstruction confirmed that the hanging was feasible without any external intervention.

High-quality 3D shape recovery from scattering scenario via deep polarization neural networks

High-quality three-dimension(3D) shape recovery from scattering scenario is of great importance with various potential applications. However, in turbid water scenarios, the absorption and scattering by water and suspending particles have a great disturbance on the generation of image clarity, texture details, and pixel-wise polarized properties such as degree of polarization (DoP) and angle of polarization (AoP), which significantly affect the 3D reconstructed performance. In this study, we present a data-driven polarimetric method for 3D reconstruction of underwater objects in turbid scenes, leveraging deep learning image priors. An underwater descattering network is designed to fit the underwater transport function of target signal light and generate clear underwater images, which are utilized to enhance the texture and remove the backscattering light. Subsequently, accurate pixel-wise DoP and AoP of the object could be calculated using the clear image prior and underwater imaging model. Finally, utilizing the obtained clear image and polarimetric information, polarimetric 3D reconstruction of object in turbid underwater scenarios is achieved. Experimental results demonstrate that accurate 3D shape of objects in turbid water can be well-reconstructed. The proposed technique has great potential for advancing marine exploration research.

Unravelling technological behaviors through core reduction intensity. The case of the early Protoaurignacian assemblage from Fumane Cave

This paper investigates core reduction intensity in the early Protoaurignacian lithic assemblage from Fumane Cave in northeastern Italy. Reduction intensity serves as a key tool to characterize blank selection strategies, raw material management, and the variability of knapping strategies throughout the reduction sequence by reconstructing the operatory field of core assemblages. Finally, it also aids in addressing the relationship between blades and bladelets, providing valuable insights into the behavioral and chrono-cultural significance of laminar productions within the Aurignacian technocomplex. To achieve these research goals, experimental work employing 3D scanning technology was conducted. This facilitated the comparison of different methods and variables for measuring reduction intensity, including the percentage of non-cortical surface, the Scar Density Index (SDI), and a novel adaptation of the Volumetric Reconstruction Method (VRM). Results demonstrate the effectiveness and potential of adapting the VRM for the study of reduction intensity in Upper Paleolithic laminar cores, and the provided R scripts and datasets will enable this method to be applied to other contexts with minimal need for modification to the workflow. Analysis of reduction intensity measures applied to the Protoaurignacian assemblage from Fumane Cave reveals slight variations based on factors such as the abundance and proximity of selected raw materials for blank production. Notably, the most prevalent raw material variety, the Maiolica, yields a higher number of less reduced cores, while reduction levels across all cores discarded at the site remain relatively high. The observed variability in the operatory field and the interrelation between blade and bladelet productions underscore the complexity and flexibility of Protoaurignacian behavior. This inherent complexity challenges any definitive separation between the operatory fields of blade and bladelet productions. These findings are particularly important to emphasize the importance of considering reduction intensity when examining technological variability and human behavior in Aurignacian studies. The proposed adaptation of the VRM and the effective combination with other measures of reduction, promises to allow future research to incorporate reduction intensity as a vital temporal component within studies on stone tool production. This integration offers a pathway to enhancing our understanding of the adaptive behaviors exhibited by Homo sapiens across diverse ecological settings and provides a clearer framework for better framing the development of the Upper Paleolithic.

Robust Fusion of Multi-Source Images for Accurate 3D Reconstruction of Complex Urban Scenes

Integrated reconstruction is crucial for 3D modeling urban scenes using multi-source images. However, large viewpoint and illumination variations pose challenges to existing solutions. A novel approach for accurate 3D reconstruction of complex urban scenes based on robust fusion of multi-source images is proposed. Firstly, georeferenced sparse models are reconstructed from the terrestrial and aerial images using GNSS-aided incremental SfM, respectively. Then, cross-platform match pairs are selected based on point-on-image observability. The terrestrial and aerial images are robustly matched based on the selected match pairs to generate cross-platform tie points. Thirdly, the tie points are triangulated to derive cross-platform 3D correspondences. The 3D correspondences are refined using a novel outlier detection method. Finally, the terrestrial and aerial sparse models are merged based on the refined correspondences, and the integrated model is globally optimized to obtain an accurate reconstruction of the scene. The proposed methodology is evaluated on five benchmark datasets, and extensive experiments are performed. The proposed pipeline is compared with a state-of-the-art methodology and three widely used software packages. Experimental results demonstrate that the proposed methodology outperforms the other pipelines in terms of robustness and accuracy.

Deep learning based coherence holography reconstruction of 3D objects.

We propose a reconstruction method for coherence holography using deep neural networks. cGAN and U-NET models were developed to reconstruct 3D complex objects from recorded interferograms. Our proposed methods, dubbed deep coherence holography (DCH), predict the non-diffracted fields or the sub-objects included in the 3D object from the captured interferograms, yielding better reconstructed objects than the traditional analytical imaging methods in terms of accuracy, resolution, and time. The DCH needs one image per sub-object as opposed to N images for the traditional sin-fit algorithm, and hence the total reconstruction time is reduced by N×. Furthermore, with noisy interferograms the DCH amplitude mean square reconstruction error (MSE) is 5×104× and 104× and phase MSE is 102× and 3×103× better than Fourier fringe and sin-fit algorithms, respectively. The amplitude peak signal to noise ratio (PSNR) is 3× and 2× and phase PSNR is 5× and 3× better than Fourier fringe and sin-fit algorithms, respectively. The reconstruction resolution is the same as sin-fit but 2× better than the Fourier fringe analysis technique.

Single Image 3D Reconstruction Algorithm Based on Parallel Double-Branch Pyramid Stacking Network

Limited visual information contained in single images and complex motion models of objects may lead to severe fragmentation and confusion of model backbone in the 3D reconstruction of objects. In order to fully extract feature information in a single image and reduce noise interference caused by environmental factors in 3D reconstruction, a Parallel Dual-Branch Pyramid Stacking Network (PDB-PSN) model is proposed. A parallel dual-branch network model is used to construct an encoder-decoder framework based on cascaded feature extraction, encode/decode high and low-resolution features in a single image, and then convert from 2D to 3D through implicit functions to achieve the 3D reconstruction of target objects in a single image. A cascaded feature extraction network is used as a low-resolution feature extraction network to extract global features of objects. In the high-resolution feature extraction branch, three concatenated hourglass networks and dilated convolutions are used in the hourglass network to increase the receptive field and obtain more global information in order to maintain the integrity of the reconstructed object limbs. A threshold processing module is set to remove irrelevant information and ensure the integrity of global information, and meanwhile, to reduce the interference of irrelevant noise information. Simulation experiments on a self-built terracotta dataset show that the PDB-PSN model can completely reconstruct the 3D model in a single image and effectively eliminate model fragmentation in the reconstruction results.

Multi-Scale Label-Free Human Brain Imaging with Integrated Serial Sectioning Polarization Sensitive Optical Coherence Tomography and Two-Photon Microscopy.

The study of aging and neurodegenerative processes in the human brain requires a comprehensive understanding of cytoarchitectonic, myeloarchitectonic, and vascular structures. Recent computational advances have enabled volumetric reconstruction of the human brain using thousands of stained slices, however, tissue distortions and loss resulting from standard histological processing have hindered deformation-free reconstruction. Here, the authorsdescribe an integrated serial sectioning polarization-sensitive optical coherence tomography (PSOCT) and two photon microscopy (2PM) system to provide label-free multi-contrast imaging of intact brain structures, including scattering, birefringence, and autofluorescence of human brain tissue. The authors demonstrate high-throughput reconstruction of 4 × 4 × 2cm3 sample blocks and simple registration between PSOCT and 2PM images that enablecomprehensive analysis of myelin content, vascular structure, and cellular information. The high-resolution 2PM images provide microscopic validation and enrichment of the cellular information provided by the PSOCT optical properties on the same sample, revealing the densely packed fibers, capillaries, and lipofuscin-filled cell bodies in the cortex and white matter. It is shown that theimaging system enables quantitative characterization of various pathological features in aging process, including myelin degradation, lipofuscin accumulation, and microvascular changes, which opens up numerous opportunities in the study of neurodegenerative diseases in the future.

Digital twin models of replicative ground stones: insight into simulating usage of Upper Paleolithic tools

This work presents the first attempt to create a physics-based digital twin model for predictive analysis of damage evolution during the use of ground stone tools (GSTs) in transformative tasks, encompassing the processing of raw resources for nutritional and non-alimentary purposes. The proposed methodology introduces a digital twin of the GSTs developed from 3D models generated using a photogrammetric technique based on Structure-from-Motion and Multi-View Stereo reconstruction. These models serve as the foundation for the development of the finite element (FE)-based digital twin model of the GSTs that exploits a contact formulation and the phase-field approach to simulate tool damage during pounding and grinding tasks. Defining the initial relative positions of the stones, their mechanical behaviour, and controlling the movement of the active stone in a way as close as possible to the real one, the digital twin model has been devised to evaluate how the surface damage is affected by perturbations in the loading conditions. The simulated damage is compared with the surface traces observed from experiments. The developed digital twin model aims at demonstrating its potentials for the GSTs investigations, as a supporting tool for experiments and for simulated tests on the archaeological records.

Single-stripe-enhanced spacetime stereo reconstruction for concrete defect identification

Few studies explored the 3D reconstruction of concrete defects, and existing methods often suffer from noise interference or impracticality in real-world applications. This study proposes a novel single-stripe-enhanced spacetime stereo reconstruction method that combines a laser liner for structured light generation and a stereo vision camera. This method analyses the morphology and intensity variation of laser stripes from images captured from single scans, achieving accurate defect reconstructions. It is designed for handheld scenarios with image re-alignment eliminating the vibration caused by handheld operation. The morphology of laser stripes at crack sections is analysed for robust fine crack reconstruction. Active frame searching is introduced to improve the calculation efficiency, enabling spacetime stereo to operate over hundreds of frames. A comparative study of the proposed method and two commonly-used stereo matching methods demonstrates its ability to mitigate the impact of handheld operation and generate 3D reconstructions that highlight potential crack regions.

Optimal Pre-processing of Laser Scanning Data for Indoor Scene Analysis and 3D Reconstruction of Building Models

The large-scale point cloud data representing buildings should be prudently pre-processed and down-sampled prior to post-data processing activities, such as data segmentation or classification and indoor scene analysis, and visualization of the processed results as three-dimensional (3D) digital models. However, current pre-processing tasks and down-sampling procedures encountered the challenge of preserving the geometric and semantic features of the scanned object and contributing to accurate 3D reconstruction due to the loss of significant features on buildings. Therefore, this paper proposed a framework that pre-processes unstructured laser scanning data to optimize the efficiency of data processing activities and the accuracy of the produced output, i.e., reconstructed 3D building models. The pre-processing framework includes building the topology of the unstructured dataset, down-sampling the raw data through an improved voxel-based approach, and performing indoor scene analysis such as normal estimation, point alignment, and data classification. The experimental results verified that the proposed framework efficiently pre-processes the input data and preserves the geometric characteristics of the digital models with high accuracy, thereby supporting reliable 3D reconstruction for various applications.

Surface defects 3D localization for fluorescent magnetic particle inspection via regional reconstruction and partial-in-complete point clouds registration

Accurate three-dimensional localization of defects plays a crucial role in the automated defect marking and grinding process. However, this task becomes increasingly intricate when precisely recovering the three-dimensional information of small defects due to the presence of noise and damage in point clouds generated through three-dimensional reconstruction. Simultaneously, separately reconstructing defects at different locations on the component would significantly impact efficiency. This study focuses on fluorescent magnetic particle inspection, with the aim of enhancing the accuracy of fine crack localization and multi-defects reconstruction efficiency. Its basic idea is to align the precise complete point cloud with the regionally reconstructed noisy point cloud of a component, achieving its spatial positioning within the visual system while eliminating noise and imperfections. We commence by discussing various methods for acquiring offline complete point clouds and their respective advantages and disadvantages. To enhance flexibility and minimize regional reconstruction noise, we have devised a multi-camera vision system and optimized a multi-view stereo reconstruction algorithm. We introduce a filtering function for culling back-facing points during the sparse point cloud projection process. The feasibility of our approach is empirically validated, with a particular focus on its precision and efficiency. Experimental results indicate that our method can execute within 5 s, with a relative error of less than 2 millimeters for 95% of the points and a root mean square error less than 1 millimeter. Our approach exhibits significant advantages in addressing noise issues associated with point cloud data generated through image-based 3D reconstruction processes.

Jrender: An efficient differentiable rendering library based on Jittor

Differentiable rendering has been proven as a powerful tool to bridge 2D images and 3D models. With the aid of differentiable rendering, tasks in computer vision and computer graphics could be solved more elegantly and accurately. To address challenges in the implementations of differentiable rendering methods, we present an efficient and modular differentiable rendering library named Jrender based on Jittor. Jrender supports surface rendering for 3D meshes and volume rendering for 3D volumes. Compared with previous differentiable renderers, Jrender exhibits a significant improvement in both performance and rendering quality. Due to the modular design, various rendering effects such as PBR materials shading, ambient occlusions, soft shadows, global illumination, and subsurface scattering could be easily supported in Jrender, which are not available in other differentiable rendering libraries. To validate our library, we integrate Jrender into applications such as 3D object reconstruction and NeRF, which show that our implementations could achieve the same quality with higher performance.

Retracted: Improved Multiview Decomposition for Single-Image High-Resolution 3D Object Reconstruction

Retracted: Improved Multiview Decomposition for Single-Image High-Resolution 3D Object Reconstruction

GAMNet: Global attention via multi‐scale context for depth estimation algorithm and application

AbstractDeep neural networks significantly enhance the accuracy of the stereo‐based disparity estimation. Some current methods suffer from inefficient use of the global context information, which will lead to the loss of structural details in ill‐posed areas. To this end, a novel stereo network GAMNet is designed, composed of three core components (GDA, MPF, DCA) for estimating the depth prediction in challenging real‐world environments. First, a lightweight attention module is presented, integrating the global semantic cues for every feature position across the channel and spatial dimensions. Next, the MPF module is constructed to fuse the diverse semantic and contextual information from different levels of the feature pyramid. Finally, cost volume is aggregated with a stacked encoder‐decoder composed of the DCA module and 3D convolutions, filtering the transmission of matching clues and capturing the rich global contexts. Substantial experiments conducted on KITTI 2012, KITTI 2015, SceneFlow, and Middlebury‐v3 datasets manifest that GAMNet surpasses preceding methods with contour‐preserving disparity predictions. In addition, the first 3D scene reconstruction linear evaluation strategy on spatial grasping points for the end‐to‐end stereo networks in an unsupervised mode is proposed, and it is deployed on the designed robot vision‐guided system. In application experiments, the method can produce densely high‐precision 3D reconstructions to implement the grasping task in complex real‐world scenes, and achieve excellent robust performance with competitive inference efficiency.

Joint analysis of macroscopic response and microfabric evolution of coral sand during earthquake-induced liquefaction

This study addresses the microfabric evolution of coral sand during earthquake-induced liquefaction using the joint tests of macroscopic shaking table and microscopic X-ray computed tomography. Based on the non-destructive scanning and image reconstruction in three-dimensional, the change in the grain–pore structure of sands under earthquakes is discussed. Furthermore, the special microstructural evolution of coral sand is studied by the contrast tests with Fujian sand. The results show that the coordination number of coral sand is larger than that of Fujian sand after liquefaction, which is consistent with that the reduction in acceleration of coral sand is smaller than that of the Fujian sand, and the coral sand foundation still has a larger shear strength due to the smaller excess pore pressure accumulation. Moreover, the contact index of Fujian sand and Reigate sand fluctuates in a similar range with the change of void ratio and coordination number compared with coral sand. This is due to the fact that the Fujian sand and Reigate sand are both general terrigenous sands (quartz sand), while coral sand has irregular particle shapes and larger surface friction caused by marine biogenesis. The test results deepen the understanding of the liquefaction mechanism of coral sand.

Fast-SNARF: A Fast Deformer for Articulated Neural Fields.

Neural fields have revolutionized the area of 3D reconstruction and novel view synthesis of rigid scenes. A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space. We propose a new articulation module for neural fields, Fast-SNARF, which finds accurate correspondences between canonical space and posed space via iterative root finding. Fast-SNARF is a drop-in replacement in functionality to our previous work, SNARF, while significantly improving its computational efficiency. We contribute several algorithmic and implementation improvements over SNARF, yielding a speed-up of 150×. These improvements include voxel-based correspondence search, pre-computing the linear blend skinning function, and an efficient software implementation with CUDA kernels. Fast-SNARF enables efficient and simultaneous optimization of shape and skinning weights given deformed observations without correspondences (e.g. 3D meshes). Because learning of deformation maps is a crucial component in many 3D human avatar methods and since Fast-SNARF provides a computationally efficient solution, we believe that this work represents a significant step towards the practical creation of 3D virtual humans.

Reconstruction of 3D Objects Based on Data from a Single View

Reconstruction of 3D Objects Based on Data from a Single View

Structure learning for 3D Point Cloud Generation from Single RGB Images

Abstract3D point clouds can represent complex 3D objects of arbitrary topologies and with fine‐grained details. They are, however, hard to regress from images using convolutional neural networks, making tasks such as 3D reconstruction from monocular RGB images challenging. In fact, unlike images and volumetric grids, point clouds are unstructured and thus lack proper parameterization, which makes them difficult to process using convolutional operations. Existing point‐based 3D reconstruction methods that tried to address this problem rely on complex end‐to‐end architectures with high computational costs. Instead, we propose in this paper a novel mechanism that decouples the 3D reconstruction problem from the structure (or parameterization) learning task, making the 3D reconstruction of objects of arbitrary topologies tractable and thus easier to learn. We achieve this using a novel Teacher‐Student network where the Teacher learns to structure the point clouds. The Student then harnesses the knowledge learned by the Teacher to efficiently regress accurate 3D point clouds. We train the Teacher network using 3D ground‐truth supervision and the Student network using the Teacher's annotations. Finally, we employ a novel refinement network to overcome the upper‐bound performance that is set by the Teacher network. Our extensive experiments on ShapeNet and Pix3D benchmarks, and on in‐the‐wild images demonstrate that the proposed approach outperforms previous methods in terms of reconstruction accuracy and visual quality.

Insights into solid dosage forms with nonlinear optical imaging

Microscopic chemical and solid-state structures and their changes in solid drugs and dosage forms can profoundly affect pharmaceutical performance and patient safety. Despite this, their detailed spatially-resolved analysis can be difficult or impossible with established analytical technologies. Multimodal non-linear optical imaging represents opportunities for sensitive and specific chemical and solid-state pharmaceutical imaging. Non-linear optical imaging encompasses several nonlinear optical phenomena, including coherent anti-Stokes Raman scattering (CARS), stimulated Raman scattering (SRS), and sum frequency/second harmonic generation (SFG/SHG). Imaging in 3D with (sub)micron resolution is rapid, non-destructive, possible in situ in aqueous media, and generally does not require prior sample preparation. This mini-review explores several applications of non-linear optical imaging for solid drug and dosage form analysis.