Volumetric Reconstruction Research Articles

Whole-cell multi-target single-molecule super-resolution imaging in 3D with microfluidics and a single-objective tilted light sheet.

Multi-target single-molecule super-resolution fluorescence microscopy offers a powerful means of understanding the distributions and interplay between multiple subcellular structures at the nanoscale. However, single-molecule super-resolution imaging of whole mammalian cells is often hampered by high fluorescence background and slow acquisition speeds, especially when imaging multiple targets in 3D. In this work, we have mitigated these issues by developing a steerable, dithered, single-objective tilted light sheet for optical sectioning to reduce fluorescence background and a pipeline for 3D nanoprinting microfluidic systems for reflection of the light sheet into the sample. This easily adaptable novel microfluidic fabrication pipeline allows for the incorporation of reflective optics into microfluidic channels without disrupting efficient and automated solution exchange. By combining these innovations with point spread function engineering for nanoscale localization of individual molecules in 3D, deep learning for analysis of overlapping emitters, active 3D stabilization for drift correction and long-term imaging, and Exchange-PAINT for sequential multi-target imaging without chromatic offsets, we demonstrate whole-cell multi-target 3D single-molecule super-resolution imaging with improved precision and imaging speed.

Three-dimensional reconstruction of a shooting crime scene

The crime scene is technically the place where an illegal act took place. The investigation first starts at the crime scene. Documentation of the crime scene is of great importance in order to make correct decisions in the investigation and prosecution stages, With the development of technology; the use of different devices and different technologies in crime scene investigation and crime scene documentation has created a new working area for many forensic scientists and law enforcement officers. Taking a general view of the crime scene without contamination, low margin of error in measurements can be shown as the main reasons for the use of laser scanning devices at the crime scene. A firearm crime scene was staged using different evidence. In the staged crime scene, a laser scanning device was also used along with the use of photography, sketching and note-taking methods from traditional documentation methods. In this study, the adequacy of the traditional methods alone, the adequacy of the laser scanning device alone, and the adequacy of the two methods together for reconstruction were discussed. The aim of this study is to determine the contributions and limitations of documenting the crime scene with a laser scanning device in combination with traditional documentation methods to the three-dimensional reconstruction of the crime scene.

Three-dimensional object reconstruction based on spin–orbit interaction of light

Three-dimensional (3D) reconstruction has received extensive attention in recent years and is of great significance to various application fields. We propose a simple method of 3D object reconstruction based on spin–orbit interactions occurring from light reflection at the air–glass interface. By rotating the object, edge images of the object in various orientations are obtained; we overlap these edge images to ultimately reconstruct the 3D object. This 3D image detection based on spin–orbit interaction of light provides a low energy consumption, low cost, and more algorithmic method compared to traditional 3D image processing. This can be used in fields such as medicine, autonomous vehicles, planetary observation, and other areas.

Three-Dimensional Reconstruction of Forest Scenes with Tree–Shrub–Grass Structure Using Airborne LiDAR Point Cloud

Fine three-dimensional (3D) reconstruction of real forest scenes can provide a reference for forestry digitization and forestry resource management applications. Airborne LiDAR technology can provide valuable data for large-area forest scene reconstruction. This paper proposes a 3D reconstruction method for complex forest scenes with trees, shrubs, and grass, based on airborne LiDAR point clouds. First, forest vertical distribution characteristics are used to segment tree, shrub, and ground–grass points from an airborne LiDAR point cloud. For ground–grass points, a ground–grass grid model is constructed. For tree points, a method based on hierarchical canopy point fitting is proposed to construct a trunk model, and a crown model is constructed with the 3D α-shape algorithm. For shrub points, a shrub model is directly constructed based on the 3D α-shape algorithm. Finally, tree, shrub, and ground–grass models are spatially combined to achieve the reconstruction of real forest scenes. Taking six forest plots located in Hebei, Yunnan, and Guangxi provinces in China and Baden-Württemberg in Germany as study areas, experimental results show that the accuracy of individual tree segmentation reaches 87.32%, the accuracy of shrub segmentation reaches 60.00%, the height accuracy of the grass model is evaluated with an RMSE < 0.15 m, the volume accuracy of shrub and tree models is assessed with R2 > 0.848 and R2 > 0.904, respectively. Furthermore, we compared the model constructed in this study with simplified point cloud and voxel models. The results demonstrate that the proposed modeling approach can meet the demand for the high-accuracy and lightweight modeling of large-area forest scenes.

NeRF-OR: neural radiance fields for operating room scene reconstruction from sparse-view RGB-D videos.

RGB-D cameras in the operating room (OR) provide synchronized views of complex surgical scenes. Assimilation of this multi-view data into a unified representation allows for downstream tasks such as object detection and tracking, pose estimation, and action recognition. Neural radiance fields (NeRFs) can provide continuous representations of complex scenes with limited memory footprint. However, existing NeRF methods perform poorly in real-world OR settings, where a small set of cameras capture the room from entirely different vantage points. In this work, we propose NeRF-OR, a method for 3D reconstruction of dynamic surgical scenes in the OR. Where other methods for sparse-view datasets use either time-of-flight sensor depth or dense depth estimated from color images, NeRF-OR uses a combination of both. The depth estimations mitigate the missing values that occur in sensor depth images due to reflective materials and object boundaries. We propose to supervise with surface normals calculated from the estimated depths, because these are largely scale invariant. We fit NeRF-OR to static surgical scenes in the 4D-OR dataset and show that its representations are geometrically accurate, where state of the art collapses to sub-optimal solutions. Compared to earlier work, NeRF-OR grasps fine scene details while training 30 faster. Additionally, NeRF-OR can capture whole-surgery videos while synthesizing views at intermediate time values with an average PSNR of 24.86dB. Last, we find that our approach has merit in sparse-view settings beyond those in the OR, by benchmarking on the NVS-RGBD dataset that contains as few as three training views. NeRF-OR synthesizes images with a PSNR of 26.72dB, a 1.7% improvement over state of the art. Our results show that NeRF-OR allows for novel view synthesis with videos captured by a small number of cameras with entirely different vantage points, which is the typical camera setting in the OR. Code is available via: github.com/Beerend/NeRF-OR .

Multi-view neural 3D reconstruction of micro- and nanostructures with atomic force microscopy

Atomic Force Microscopy (AFM) is a widely employed tool for micro- and nanoscale topographic imaging. However, conventional AFM scanning struggles to reconstruct complex 3D micro- and nanostructures precisely due to limitations such as incomplete sample topography capturing and tip-sample convolution artifacts. Here, we propose a multi-view neural-network-based framework with AFM, named MVN-AFM, which accurately reconstructs surface models of intricate micro- and nanostructures. Unlike previous 3D-AFM approaches, MVN-AFM does not depend on any specially shaped probes or costly modifications to the AFM system. To achieve this, MVN-AFM employs an iterative method to align multi-view data and eliminate AFM artifacts simultaneously. Furthermore, we apply the neural implicit surface reconstruction technique in nanotechnology and achieve improved results. Additional extensive experiments show that MVN-AFM effectively eliminates artifacts present in raw AFM images and reconstructs various micro- and nanostructures, including complex geometrical microstructures printed via two-photon lithography and nanoparticles such as poly(methyl methacrylate) (PMMA) nanospheres and zeolitic imidazolate framework-67 (ZIF-67) nanocrystals. This work presents a cost-effective tool for micro- and nanoscale 3D analysis.

3D reconstruction and volume measurement of irregular objects based on RGB-D camera

Abstract To address the challenge of measuring volumes of irregular objects, this paper proposes a volume measurement method based on 3D point cloud reconstruction. The point clouds of the object with multiple angles are obtained from an RGB-D camera mounted on a robotic arm, and then are reconstructed to form a whole complete point cloud to calculate the volume of the object. Firstly, the robotic arm is controlled to move to four angles for capturing the original point clouds of the target. Then, by using the rotation and translation matrices obtained from the calibration block pre-registration, the point clouds data from the four angles are fused and reconstructed. Subsequently, the issue of missing bottom point cloud data is addressed using a bottom-filling algorithm. Following this, the efficiency of the point cloud volume calculation algorithm is enhanced through the application of AABB filtering. Finally, the reconstructed point cloud volume is calculated using a slicing algorithm that integrates 2D point cloud segmentation and point cloud sorting. Experimental results show that this method achieves a volume measurement accuracy of over 95% for irregular objects and exhibits good robustness.

Reconstruction of three-dimensional scenes based on video flow data

This work is dedicated to the application of modern algorithms for reconstructing spatial scenes from images to restore spatial information from video. The work is looking at a variety of modern methods, approaches, algorithms and trends in the field. The attention was paid to the sequence of development of approaches to the completion of the task. While researching the field and results related to three-dimensional reconstruction based on images and video streams, an algorithm was invented that allows constructing dense depth maps using information from all video frames. The idea is to use ready-made, commonly accepted, and tested solutions to solve two problems: COLMAP for visual odometry, and RAFT for computing optical flow. The algorithm shows quite accurate results and reconstructs the depth map in detail on arbitrary static scenes.

Long-term reprojection loss for self-supervised monocular depth estimation in endoscopic surgery

Aim: Depth information plays a key role in enhanced perception and interaction in image-guided surgery. However, it is difficult to obtain depth information with monocular endoscopic surgery due to a lack of reliable cues for perceiving depth. Although there are reprojection loss-based self-supervised learning techniques to estimate depth and pose, the temporal information from the adjacent frames is not efficiently utilized to handle occlusion in surgery. Methods: We design long-term reprojection loss (LT-RL) self-supervised monocular depth estimation techniques by integrating longer temporal sequences into reprojection to learn better perception and to address occlusion artifacts in image-guided laparoscopic and robotic surgery. For this purpose, we exploit four temporally adjacent source frames before and after the target frame, where conventional reprojection loss uses two adjacent frames. The pixels that are visible in the target frame but occluded in the immediate two adjacent frames will produce the inaccurate depth but a higher chance to appear in the four adjacent frames during the calculation of minimum reprojection loss. Results: We validate LT-RL on the benchmark surgical datasets of Stereo correspondence and reconstruction of endoscopic data (SCARED) and Hamlyn to compare the performance with other state-of-the-art depth estimation methods. The experimental results show that our proposed technique yields 2%-4% better root-mean-squared error (RMSE) over the baselines of vanilla reprojection loss. Conclusion: Our LT-RL self-supervised depth and pose estimation technique is a simple yet effective method to tackle occlusion artifacts in monocular surgical video. It does not add any training parameters, making it flexible for integration with any network architecture and improving the performance significantly.

Physics-informed neural network for volumetric sound field reconstruction of speech signals

Recent developments in acoustic signal processing have seen the integration of deep learning methodologies, alongside the continued prominence of classical wave expansion-based approaches, particularly in sound field reconstruction. Physics-informed neural networks (PINNs) have emerged as a novel framework, bridging the gap between data-driven and model-based techniques for addressing physical phenomena governed by partial differential equations. This paper introduces a PINN-based approach for the recovery of arbitrary volumetric acoustic fields. The network incorporates the wave equation to impose a regularization on signal reconstruction in the time domain. This methodology enables the network to learn the physical law of sound propagation and allows for the complete characterization of the sound field based on a limited set of observations. The proposed method’s efficacy is validated through experiments involving speech signals in a real-world environment, considering varying numbers of available measurements. Moreover, a comparative analysis is undertaken against state-of-the-art frequency domain and time domain reconstruction methods from existing literature, highlighting the increased accuracy across the various measurement configurations.

Hyoid Position and Aging: A Comprehensive Analysis Using AI-assisted Segmentation of 282 Computed Tomography Scans.

With neck, aging the cervicomental angle becomes obtuse and may be influenced by hyoid bone aging. An understanding of hyoid position changes with aging will further our understanding of its role in neck contour changes. A 3D volumetric reconstruction of 282 neck computed tomography scans was performed. The cohort was categorized into three groups based on age: 20 years or older and younger than 40 years, 40 years or older and younger than 60 years, and 60 years or older and younger than 80 years. The vertical and horizontal hyoid distances in relation to the mandible were calculated for each patient. A total of 282 patients (153 women, 129 men) were included in the cohort. The age groups were evenly distributed in men and women. Mean hyoid vertical and horizontal distances differed between women and men in all age groups. There was a significant difference in the hyoid vertical distance between 20-39 years old to 40-59 years old in men (P < 0.01), and 20-39 years old to 60-79 years old in both genders (women P = 0.005, men P < 0.01). Hyoid horizontal distance was not affected by age and sex (age and sex: P > 0.05), but rather by body mass index (BMI). Every 5 BMI points corresponded to a forward movement of 2 mm. As individuals age, the hyoid bone descends in both sexes, and an increase in BMI is associated with forward movement. Additional studies are needed to assess the correlation of the hyoid position between upright and supine positions.

Neural 3D Scene Reconstruction With Indoor Planar Priors.

This paper addresses the challenge of reconstructing 3D indoor scenes from multi-view images. Many previous works have shown impressive reconstruction results on textured objects, but they still have difficulty in handling low-textured planar regions, which are common in indoor scenes. An approach to solving this issue is to incorporate planar constraints into the depth map estimation in multi-view stereo-based methods, but the per-view plane estimation and depth optimization lack both efficiency and multi-view consistency. In this work, we show that the planar constraints can be conveniently integrated into the recent implicit neural representation-based reconstruction methods. Specifically, we use an MLP network to represent the signed distance function as the scene geometry. Based on the Manhattan-world assumption and the Atlanta-world assumption, planar constraints are employed to regularize the geometry in floor and wall regions predicted by a 2D semantic segmentation network. To resolve the inaccurate segmentation, we encode the semantics of 3D points with another MLP and design a novel loss that jointly optimizes the scene geometry and semantics in 3D space. Experiments on ScanNet and 7-Scenes datasets show that the proposed method outperforms previous methods by a large margin on 3D reconstruction quality.

Development of multi-view 3D reconstruction system for bubble flow measurement

This work introduces the development of bubble measurement method utilizing a three-dimensional (3D) reconstruction technique from multi-view images. Multiple synchronized cameras were positioned around a water container, and calibration was performed to obtain external and internal camera parameters. Images of bubbles emerging from a nozzle were captured, and a machine learning technique was used to extract bubble silhouettes as foreground probability distributions, enabling the extraction of bubbles from images obtained with a simple lighting setup. These distributions were then projected onto a 3D voxel space using the visual hull method. It was confirmed that the method can successfully capture bubble generation, detachment, and rise behaviors, offering insights into understanding bubble dynamics and potential applications in 3D computational fluid dynamics validation.

High-resolution in vivo 4D-OCT fish-eye imaging using 3D-UNet with multi-level residue decoder.

Optical coherence tomography (OCT) allows high-resolution volumetric imaging of biological tissues in vivo. However, 3D-image acquisition often suffers from motion artifacts due to slow frame rates and involuntary and physiological movements of living tissue. To solve these issues, we implement a real-time 4D-OCT system capable of reconstructing near-distortion-free volumetric images based on a deep learning-based reconstruction algorithm. The system initially collects undersampled volumetric images at a high speed and then upsamples the images in real-time by a convolutional neural network (CNN) that generates high-frequency features using a deep learning algorithm. We compare and analyze both dual-2D- and 3D-UNet-based networks for the OCT 3D high-resolution image reconstruction. We refine the network architecture by incorporating multi-level information to accelerate convergence and improve accuracy. The network is optimized by utilizing the 16-bit floating-point precision for network parameters to conserve GPU memory and enhance efficiency. The result shows that the refined and optimized 3D-network is capable of retrieving the tissue structure more precisely and enable real-time 4D-OCT imaging at a rate greater than 10 Hz with a root mean square error (RMSE) of ∼0.03.

Method of 3D reconstruction of underwater concrete by laser line scanning

High-precision 3D (three-dimensional) reconstruction of underwater concrete is helpful for the detection and analysis of underwater concrete diseases. A prevalent method involves mounting optical cameras in waterproof housings to enable the 3D reconstruction of underwater concrete. Achieving millimetric accuracy in underwater 3D reconstruction poses a significant challenge in current research. The refraction of light on surfaces of different media and its reflection underwater complicate the process. In this paper, a 3D reconstruction method of underwater concrete based on laser line scanning is proposed. Firstly, the underwater imaging model is established by the ray tracing method, and the refraction conversion matrix is derived. Then, based on this matrix, the conversion relationship between the measured 3D size and the real 3D size of the underwater object is obtained. Finally, image restoration technology is introduced in the process of laser stripe extraction to eliminate the influence of reflected light on the extraction of laser stripe center, to improve the reconstruction accuracy. We used the independently developed laser line 3D scanning equipment to carry out experimental analysis. The results showed that the error rate caused by direct reconstruction of underwater objects was 9.75 %. Additionally, the error rate obtained from 3D reconstruction of underwater objects using the conversion model was 2.89 %. This study provides technical support for the 3D detection of underwater concrete.

EnhancedNet, an End-to-End Network for Dense Disparity Estimation and its Application to Aerial Images

Recent developments in deep learning technology have boosted the performance of dense stereo reconstruction. However, the state-of-the-art deep learning-based stereo matching methods are mainly trained using close-range synthetic images. Consequently, the application of these methods in aerial photogrammetry and remote sensing is currently far from straightforward. In this paper, we propose a new disparity estimation network for stereo matching and investigate its generalization abilities in regard to aerial images. First, we propose an end-to-end deep learning network for stereo matching, regularized by disparity gradients, which includes a residual cost volume and a reconstruction error volume in a refinement module, and multiple losses. In order to investigate the influence of the multiple losses, a comprehensive analysis is presented. Second, based on this network trained with synthetic close-range data, we propose a new pipeline for matching high-resolution aerial imagery. The experimental results show that the proposed network improves the disparity accuracy by up to 40% in terms of errors larger than 1 px compared to results when not including the refinement network, especially in areas containing detailed small objects. In addition, in qualitative and quantitative experiments, we are able to show that our model, pre-trained on a synthetic stereo dataset, achieves very competitive sub-pixel geometric accuracy on aerial images. These results confirm that the domain gap between synthetic close-range and real aerial images can be satisfactorily bridged using the proposed new deep learning method for dense image matching.

Study on the impact of camera ISP algorithms on stripe structured light 3D reconstruction and the improvement of 3D reconstruction accuracy by photon input

This paper investigates the impact of camera image signal processing (ISP) algorithms on stripe structured light 3D reconstruction and explores the improvement of 3D reconstruction accuracy by photon input. Existing 3D reconstruction technologies have broad applications in fields such as intelligent manufacturing, healthcare, and consumer electronics, but their high precision requirements often cannot be met by current ISP algorithms. By using handheld devices to capture images and combining key techniques such as the multi-step phase shifting method, multi-frequency phase unwrapping method, and triangulation method, this paper conducts an in-depth study on the application of photon input in 3D reconstruction. The study demonstrates that using non-visual information (RAW images) as input can significantly improve reconstruction accuracy, producing more accurate results compared to images processed by ISP. The paper also quantifies the impact of ISP processing on 3D reconstruction results by comparing the depth information and point cloud data of two sets of images. Experimental results show that disabling certain ISP algorithms, such as bilateral noise filtering (BNF), edge enhancement (EEH), and non-local means denoising (NLM), can further improve reconstruction accuracy and reduce errors. In conclusion, this paper proposes a photon image-based 3D reconstruction method that, combined with artificial intelligence technology and differentiable point cloud rendering techniques, holds promise for achieving higher precision and faster 3D reconstruction. This technology is of great significance in practical applications, particularly in the field of industrial close-range 3D reconstruction.

OpenECAD: An efficient visual language model for editable 3D-CAD design

Computer-aided design (CAD) tools are utilized in the manufacturing industry for modeling everything from cups to spacecraft. These programs are complex to use and typically require years of training and experience to master. Structured and well-constrained 2D sketches and 3D constructions are crucial components of CAD modeling. A well-executed CAD model can be seamlessly integrated into the manufacturing process, thereby enhancing production efficiency. Deep generative models of 3D shapes and 3D object reconstruction models have garnered significant research interest. However, most of these models produce discrete forms of 3D objects that are not editable. Moreover, the few models based on CAD operations often have substantial input restrictions. In this work, we fine-tuned pre-trained models to create OpenECAD models (0.55B, 0.89B, 2.4B and 3.1B), leveraging the visual, logical, coding, and general capabilities of visual language models. OpenECAD models can process images of 3D designs as input and generate highly structured 2D sketches and 3D construction commands, ensuring that the designs are editable. These outputs can be directly used with existing CAD tools’ APIs to generate project files. To train our network, we created a series of OpenECAD datasets. These datasets are derived from existing public CAD datasets, adjusted and augmented to meet the specific requirements of vision language model (VLM) training. Additionally, we have introduced an approach that utilizes dependency relationships to define and generate sketches, further enriching the content and functionality of the datasets.

ClearScope: a fully integrated light sheet theta microscope for sub-cellular resolution imaging without lateral size constraints.

Three-dimensional (3D) ex vivo imaging of cleared intact brains of animal models and large human and non-human primate postmortem brain specimens is important for understanding the physiological neural network connectivity patterns and the pathological alterations underlying neuropsychiatric and neurological disorders. Light-sheet microscopy has emerged as a highly effective imaging modality for rapid high-resolution imaging of large cleared samples. However, the orthogonal arrangements of illumination and detection optics in light sheet microscopy limits the size of specimen that can be imaged. Recently developed light sheet theta microscopy (LSTM) technology addressed this by utilizing a unique arrangement of two illumination light paths oblique to the detection light path, while allowing perpendicular arrangement of the detection light path relative to the specimen surface. Here, we report development of a next-generation, fully integrated, and user-friendly LSTM system for rapid sub-cellular resolution imaging uniformly throughout a large specimen without constraining the lateral (XY) size. In addition, we provide a seamlessly integrated workflow for image acquisition, data storage, pre- and post-processing, enhancement, and quantitative analysis. We demonstrate the system performance by high-resolution 3D imaging of intact mouse brains and human brain samples, and complete data analysis including digital neuron tracing, vessel reconstruction and design-based stereological analysis in 3D. This technically enhanced and user-friendly LSTM implementation will enable rapid quantitative mapping of molecular and cellular features of interests in diverse types of very large samples.

Assessing the 3D resolution of refocused correlation plenoptic images using a general-purpose image quality estimator

Correlation plenoptic imaging (CPI) is emerging as a promising approach to light-field imaging (LFI), a technique enabling simultaneous measurement of light intensity distribution and propagation direction from a scene. LFI allows single-shot 3D sampling, offering fast 3D reconstruction for a wide range of applications. However, the array of micro-lenses typically used in LFI to obtain 3D information limits image resolution, which rapidly declines with enhanced volumetric reconstruction capabilities. CPI addresses this limitation by decoupling the measurement of the light field on two photodetectors with spatial resolution, eliminating the need for micro-lenses. 3D information is encoded in a four-dimensional correlation function, which is decoded in post-processing to reconstruct images without the resolution loss seen in conventional LFI. This paper evaluates the tomographic performance of CPI, demonstrating that the refocusing reconstruction method provides axial sectioning capabilities comparable to conventional imaging systems. A general-purpose analytical approach based on image fidelity is proposed to quantitatively study axial and lateral resolution. The analysis fully characterizes the volumetric resolution of any CPI architecture, offering a comprehensive evaluation of its imaging performance.