Volumetric Reconstruction Research Articles

A solid-state lidar-inertial 3D scene reconstruction method based on wearable laser helmet

A solid-state lidar-inertial 3D scene reconstruction method based on wearable laser helmet

Self-Supervised Monocular Depth Estimation for Endoscopic Imaging.

Endoscopy holds a pivotal role in the early detection and treatment of diverse diseases, with artificial intelligence (AI)-assisted methods increasingly gaining prominence in disease screening. Among them, the depth estimation from endoscopic sequences is crucial for a spectrum of AI-assisted surgical techniques. However, the development of endoscopic depth estimation algorithms presents a formidable challenge due to the unique environmental intricacies and constraints within the dataset. This paper proposes a self-supervised depth estimation network to comprehensively explore the brightness changes in endoscopic images, and fuse different features at multiple levels to achieve an accurate prediction of endoscopic depth. First, a FlowNet is designed to evaluate the brightness changes of adjacent frames by calculating the multi-scale structural similarity. Second, a feature fusion module is presented to capture multi-scale contextual information. Experiments show that the average accuracy of the algorithm is 97.03% in the Stereo Correspondence and Reconstruction of Endoscopic Data (SCARED dataset). Based on the training parameters of the SCARED dataset, the algorithm achieves superior performance on the other two datasets (EndoSLAM and KVASIR dataset), indicating that the algorithm has good generalization performance.

Image-Based 3D Reconstruction in Laparoscopy: A Review Focusing on the Quantitative Evaluation by Applying the Reconstruction Error.

Image-based 3D reconstruction enables laparoscopic applications as image-guided navigation and (autonomous) robot-assisted interventions, which require a high accuracy. The review's purpose is to present the accuracy of different techniques to label the most promising. A systematic literature search with PubMed and google scholar from 2015 to 2023 was applied by following the framework of "Review articles: purpose, process, and structure". Articles were considered when presenting a quantitative evaluation (root mean squared error and mean absolute error) of the reconstruction error (Euclidean distance between real and reconstructed surface). The search provides 995 articles, which were reduced to 48 articles after applying exclusion criteria. From these, a reconstruction error data set could be generated for the techniques of stereo vision, Shape-from-Motion, Simultaneous Localization and Mapping, deep-learning, and structured light. The reconstruction error varies from below one millimeter to higher than ten millimeters-with deep-learning and Simultaneous Localization and Mapping delivering the best results under intraoperative conditions. The high variance emerges from different experimental conditions. In conclusion, submillimeter accuracy is challenging, but promising image-based 3D reconstruction techniques could be identified. For future research, we recommend computing the reconstruction error for comparison purposes and use ex/in vivo organs as reference objects for realistic experiments.

The role of a digital twin in supporting criminal investigations - a case report about a possible abuse.

As part of a comprehensive analysis, this case report presents a possible case of child maltreatment that can serve as a basis for forensic and medical examiner investigations. This case concerns the death of an infant who was approximately two months old. During a routine examination by the pediatrician at the end of May 2021, the child was found to have a normal head circumference of 31 cm. No other abnormalities were noted. On June 19, 2021, the child died, and an autopsy revealed a head circumference of 44 cm and a subdural hematoma as the cause of death. Questions arose as to who might have abused the child and when. The only evidence was a low-quality cell phone video taken by the child's parents on June 13, 2021, six days before the child's death, in which the child could be seen lying on a pillow. It was necessary to determine whether the child in this video already had an unnatural head circumference. This study presents a novel workflow that demonstrates how to analyze and deal with low quality video to answer questions like the above. The workflow demonstrates the creation of 3D scenes from digital image and video material. These 3D scenes can be used for object measurement and to support forensic and medical investigations. In the present case, where only low quality smartphone images were available, the presented workflow was used to create a 3D scene of the child lying on the pillow. In this 3D scene, it was possible to determine the child's head circumference. These measurements support the findings of the medical examiner (dated June 24, 2021) and confirm the suspicion that possible child abuse had already taken place on June 13, 2021. The innovative approach makes it possible to identify evidence of possible abuse based on a specific point in time, in this case the child's private footage. It also demonstrates the potential of 3D scene reconstruction in complex forensic and medical scenarios.

GAN-based quantitative oblique back-illumination microscopy enables computationally efficient epi-mode refractive index tomography.

Quantitative oblique back-illumination microscopy (qOBM) is a novel imaging technology that enables epi-mode 3D quantitative phase imaging and refractive index (RI) tomography of thick scattering samples. The technology uses four oblique back illumination images captured at the same focal plane and a fast 2D deconvolution reconstruction algorithm to reconstruct 2D phase cross-sections of thick samples. Alternatively, a through-focus z-stack of oblique back illumination images can be used to recover 3D RI tomograms with improved RI quantitative fidelity at the cost of a more computationally expensive reconstruction algorithm. Here, we report on a generative adversarial network (GAN) assisted approach to reconstruct 3D RI tomograms with qOBM that achieves high fidelity and greatly reduces processing time. The proposed approach achieves high-fidelity 3D RI tomography using differential phase contrast images from three adjacent z-planes. A ∼9-fold improvement in volumetric reconstruction time is achieved. We further show that this technique provides high SNR RI tomograms with high quantitative fidelity, reduces motion artifacts, and generalizes to different tissue types. This work can lead to real-time, high-fidelity RI tomographic imaging for in-vivo pre-clinical and clinical applications.

Explicit 3D reconstruction from images with dynamic graph learning and rendering-guided diffusion

High-quality 3D reconstruction is becoming increasingly important in a variety of fields. Recently, implicit representation methods have made significant progress in image-based 3D reconstruction. However, these methods tend to yield entangled neural representations which lack support for standard 3D pipelines, and their reconstruction results usually experience a sharp drop in quality when input views are reduced. To obtain high-quality 3D content, we propose an explicit 3D reconstruction method that directly extracts textured meshes from images and remains robust using reduced input views. Our central components include a dynamic graph convolutional network (GCN) and a rendering-guided diffusion model. The dynamic GCN aims to improve mesh reconstruction quality by effectively aggregating features from vertex neighborhoods. The aggregation is accelerated through sampling geometric-related neighbors with different SDF signs, which gradually converges in quantity during training. The rendering-guided diffusion model learns prior distributions for unseen regions to improve reconstruction performance using sparse-view inputs. It uses the rendered image under an interpolated camera pose as conditioned input and its diffusion strength can be controlled with the rendering loss of explicit reconstruction. In addition, the rendering-guided diffusion model can be jointly trained to generate plausible novel views with 3D consistency. Experiments demonstrate that our method can produce high-quality explicit reconstruction results and maintain realistic reconstruction using sparse-view inputs.

Three-dimensional artistic design method of ceramic products based on recurrent neural network technology

Three-dimensional digital technology has made breakthroughs and shown unique advantages in all walks of life. On the basis of practicality, the three-dimensional artistic design of ceramic products gradually adds some aesthetic, artistic design elements, which brings beautiful enjoyment to people’s lives and makes people’s lives colorful. This paper presents a three-dimensional artistic design method for ceramic products based on RNN (recurrent neural network) technology. With the establishment of the 3D YOLOv3 framework, the new model training is faster and more stable, the convergence speed of the loss function is faster, and the reconstructed 3D model is more accurate. After training for a certain number of times, the network gradually becomes stable, the accuracy rate is kept at 95%, and the loss function value is reduced below 0.2. The accuracy of the network model and the precision of semantic segmentation are improved. The semantic segmentation and object recognition under 3D scene reconstruction studied in this paper have certain theoretical value and high feasibility.

Volumetric Reconstruction Of Stagnated Wake Structures In Adverse Pressure Gradient

To gather insights about turbulent structures of a wake in adverse pressure gradient, a wind tunnel model is investigated with PIV methods. A flat plate of L = 1.058 m at zero angle of attack is placed in the MUB wind tunnel at TU Braunschweig. The wake convects downstream into a carefully designed diffuser, where it undergoes a pressure increase of ∆c p = +0.6. Wakes, with the common example of a cylinder shedding van-Karman vortices, are generally prone to instability. The wake instability of the sharp-edged plate is attenuated by the adverse pressure gradient, manifesting in large coherent structures that carry up to 37 % of the turbulent kinetic energy. While the primary shedding motion is effectively a quasi-2D motion, a secondary mode of is also observed: Over the span S = 1.3 m of the wake, it organizes in three distinct cells. Their motion is similar to "owl-eyes", that are known to appear on separated flow over smooth 2D bodies. The presence of these modes reduces the average flow velocity in the wake centre up to stagnation in the mean sense. Because stagnation and shedding does occur far away from any solid surfaces, this test case offers excellent observability of the phenomenon with PIV methods. RANS results of this setup cannot predict the flow stagnation accurately. Because the shedding frequency is low compared to freestream velocity, scale and time resolving IDDES results need to run many convective time units until the observed instability develops. While the original goal of observing small scale turbulent wake structures is compromised by the large shedding modes, the observation gives an insight on wake-bursting phenomena that occur on multi-element high-lift devices. The mode shedding and consequent wake stagnation is sensitive to the Reynolds-number (Re = [1.6 ... 3.2] · 10 6 ) as well as to the initial boundary layer thickness of the flat plate.

3D Virtual Histopathology by Phase-Contrast X-Ray Micro-CT for Follicular Thyroid Neoplasms.

Histological analysis is the core of follicular thyroid carcinoma (FTC) classification. The histopathological criteria of capsular and vascular invasion define malignancy and aggressiveness of FTC. Analysis of multiple sections is cumbersome and as only a minute tissue fraction is analyzed during histopathology, under-sampling remains a problem. Application of an efficient tool for complete tissue imaging in 3D would speed-up diagnosis and increase accuracy. We show that X-ray propagation-based imaging (XPBI) of paraffin-embedded tissue blocks is a valuable complementary method for follicular thyroid carcinoma diagnosis and assessment. It enables a fast, non-destructive and accurate 3D virtual histology of the FTC resection specimen. We demonstrate that XPBI virtual slices can reliably evaluate capsular invasions. Then we discuss the accessible morphological information from XPBI and their significance for vascular invasion diagnosis. We show 3D morphological information that allow to discern vascular invasions. The results are validated by comparing XPBI images with clinically accepted histology slides revised by and under supervision of two experienced endocrine pathologists.

Volumetric reconstruction of flow particles through light field particle image velocimetry and deep neural network

Volumetric reconstruction of flow particles through light field particle image velocimetry and deep neural network

Clinical development of ileofemoral thrombosis caused by malposition of the ileosacral screw

Introduction Treatment of unstable injuries of the pelvic bones is one of the unsolved problems in modern traumatology. The instability of the injuries is determined by the nature of the destruction of the posterior semi-ring of the pelvis. The most used technique for osteosynthesis of sacral fractures for unstable pelvic injuries is ileosacral screw fixation.The purpose of the work was to demonstrate a clinical case of an iatrogenic complication: ileofemoral thrombosis caused by compression of the internal iliac vein due to malposition of the ileosacral screw.Materials and methods Medical records of a 34-year-old patient injured in a traffic accident (front seat passenger) were studied, who was referred from a district hospital on the fourth day after the injury diagnosed with a fracture of the transverse process of L2 vertebra on the left, closed fracture of the pubic and ischial bones on the left, fracture of the lateral mass of the sacrum on the right. Alcohol intoxication. Traumatic shock stage 1. Treatment results were monitored throughout the inpatient and outpatient treatment periods. Long-term follow-up was 6 months.Results Timely suspicion of the complication based on physical examination data with CT angiography that assisted to exclude internal bleeding and urgently carry out repeated surgical intervention to correct the implant malposition, reosteosynthesis of S1 body without loss of reduction, to achieve regression of the clinical picture of ileofemoral thrombosis, and also to mobilize the patient in the shortest possible time. During inpatient treatment, regression of the symptoms of the complication was achieved, the patient was fully activated, and was discharged for outpatient follow-ups.Discussion Variability in the anatomy of the sacrum and the characteristics of fractures hinder the navigation while inserting iliosacral screws, which in some cases result in malposition. Compression of the common iliac vein without damaging it causes difficulty in blood outflow and the development of a clinical picture of ileofemoral thrombosis in the early postoperative period.Conclusion This clinical case shows that X-ray images in standard views are not a reliable method for diagnosing sacral injuries and are not sufficient for preoperative planning. The use of standard C-arm did not provide sufficient intraoperative visualization to correctly assess the position of the iliosacral screw, especially in cases with sacral dysmorphia. The 3D volumetric image reconstruction significantly influenced the correct perception of the anatomical structure of the bony structures of the pelvis. Malposition of the screw and its exit to the anterior surface of the sacrum caused disruption of the iliac vein outflow and development of thrombosis in the early postoperative period.

Enhanced multi view 3D reconstruction with improved MVSNet

Although 3D reconstruction has been widely used in many fields as a key component of environment perception, existing technologies still have the potential for further improvement in 3D scene reconstruction. We propose an improved reconstruction algorithm based on the MVSNet network architecture. To glean richer pixel details from images, we suggest deploying a DE module integrated with a residual framework, which supplants the prevailing feature extraction mechanism. The DE module uses ECA-Net and dilated convolution to expand the receptive field range, performing feature splicing and fusion through the residual structure to retain the global information of the original image. Moreover, harnessing attention mechanisms refines the 3D cost volume's regularization process, bolstering the integration of information across multi-scale feature volumes, consequently enhancing depth estimation precision. When assessed our model using the DTU dataset, findings highlight the network's 3D reconstruction scoring a completeness (comp) of 0.411 mm and an overall quality of 0.418 mm. This performance is higher than that of traditional methods and other deep learning-based methods. Additionally, the visual representation of the point cloud model exhibits marked advancements. Trials on the Blended MVS dataset signify that our network exhibits commendable generalization prowess.

A Performance Comparison of 3D Survey Instruments for Their Application in the Cultural Heritage Field.

Thanks to the recent development of innovative instruments and software with high accuracy and resolution, 3D modelling provides useful insights in several sectors (from industrial metrology to cultural heritage). Moreover, the 3D reconstruction of objects of artistic interest is becoming mandatory, not only because of the risks to which works of art are increasingly exposed (e.g., wars and climatic disasters) but also because of the leading role that the virtual fruition of art is taking. In this work, we compared the performance of four 3D instruments based on different working principles and techniques (laser micro-profilometry, structured-light topography and the phase-shifting method) by measuring four samples of different sizes, dimensions and surface characteristics. We aimed to assess the capabilities and limitations of these instruments to verify their accuracy and the technical specifications given in the suppliers' data sheets. To this end, we calculated the point densities and extracted several profiles from the models to evaluate both their lateral (XY) and axial (Z) resolution. A comparison between the nominal resolution values and those calculated on samples representative of cultural artefacts was used to predict the performance of the instruments in real case studies. Overall, the purpose of this comparison is to provide a quantitative assessment of the performance of the instruments that allows for their correct application to works of art according to their specific characteristics.

Volumetric Ultrasound Imaging for the Whole Soft Tissue: Toward Enhanced Thyroid Disease Examination

Ultrasound imaging (USI) is the gold standard in the clinical diagnosis of thyroid diseases. Compared with two-dimensional (2D) USI, three-dimensional (3D) USI could provide more structural information. However, the unstable pressure generated by the hand-hold ultrasound probe scanning can cause tissue deformation, especially in soft tissues such as the thyroid. The deformation is manifested as tissue structure being compressed in 2D USI, which results in structural discontinuity in 3D USI. Furthermore, multiple scans apply pressure in different directions to the tissue, which will cause relative displacement between the 3D images obtained from multiple thyroid scans. In this work, we proposed a framework to minimize the influence of the variation of pressure in thyroid 3D USI. To correct pressure artifacts in a single scanning sequence, an adaptive method to smooth the position of the 2D ultrasound (US) image sequence is adopted before performing volumetric reconstruction. To build a whole 3D US image including both sides of the thyroid gland, an iterative closest point (ICP) based registration pipeline is adopted to eliminate the relative displacement caused by different pressure directions. Our proposed method was validated by in vivo experiments, including healthy volunteers and volunteers with thyroid nodules at different grading levels. The thyroid gland and nodule are rendered intelligently in the whole scanning region to facilitate the observation of 3D USI results by the doctor. This work might make a positive contribution to the clinical diagnosis of diseases of the thyroid or other soft tissues.

Automatic segmentation and implicit surface representation of dynamic cardiac data

Segmentation of anatomical structures on 2D images of cardiac exams is necessary for performing 3D volumetric analysis, enabling the computation of parameters for diagnosing cardiovascular disease. In this work, we present robust algorithms to automatically segment cardiac imaging data and generate a volumetric anatomical reconstruction of a patient-specific heart model by propagating active contour output within a patient stack through a self-supervised learning model. Contour initializations are automatically generated, then output segmentations on sparse image slices are transferred and merged across a stack of images within the same heart data set during the segmentation process. We demonstrate whole-heart segmentation and compare the results with ground truth manual annotations. Additionally, we provide a framework to represent segmented heart data in the form of implicit surfaces, allowing interpolation operations to generate intermediary models of heart sections and volumes throughout the cardiac cycle and to estimate ejection fraction.

Comparative Evaluation of NeRF Algorithms on Single Image Dataset for 3D Reconstruction

Abstract. The reconstruction of three-dimensional scenes from a single image represents a significant challenge in computer vision, particularly in the context of cultural heritage digitisation, where datasets may be limited or of poor quality. This paper addresses this challenge by conducting a study of the latest and most advanced algorithms for single-image 3D reconstruction, with a focus on applications in cultural heritage conservation. Exploiting different single-image datasets, the research evaluates the strengths and limitations of various artificial intelligence-based algorithms, in particular Neural Radiance Fields (NeRF), in reconstructing detailed 3D models from limited visual data. The study includes experiments on scenarios such as inaccessible or non-existent heritage sites, where traditional photogrammetric methods fail. The results demonstrate the effectiveness of NeRF-based approaches in producing accurate, high-resolution reconstructions suitable for visualisation and metric analysis. The results contribute to advancing the understanding of NeRF-based approaches in handling single-image inputs and offer insights for real-world applications such as object location and immersive content generation.

HoloGS: Instant Depth-based 3D Gaussian Splatting with Microsoft HoloLens 2

Abstract. In the fields of photogrammetry, computer vision and computer graphics, the task of neural 3D scene reconstruction has led to the exploration of various techniques. Among these, 3D Gaussian Splatting stands out for its explicit representation of scenes using 3D Gaussians, making it appealing for tasks like 3D point cloud extraction and surface reconstruction. Motivated by its potential, we address the domain of 3D scene reconstruction, aiming to leverage the capabilities of the Microsoft HoloLens 2 for instant 3D Gaussian Splatting. We present HoloGS, a novel workflow utilizing HoloLens sensor data, which bypasses the need for pre-processing steps like Structure from Motion by instantly accessing the required input data i.e. the images, camera poses and the point cloud from depth sensing. We provide comprehensive investigations, including the training process and the rendering quality, assessed through the Peak Signal-to-Noise Ratio, and the geometric 3D accuracy of the densified point cloud from Gaussian centers, measured by Chamfer Distance. We evaluate our approach on two self-captured scenes: An outdoor scene of a cultural heritage statue and an indoor scene of a fine-structured plant. Our results show that the HoloLens data, including RGB images, corresponding camera poses, and depth sensing based point clouds to initialize the Gaussians, are suitable as input for 3D Gaussian Splatting.

Novel View Synthesis with Neural Radiance Fields for Industrial Robot Applications

Abstract. Neural Radiance Fields (NeRFs) have become a rapidly growing research field with the potential to revolutionize typical photogrammetric workflows, such as those used for 3D scene reconstruction. As input, NeRFs require multi-view images with corresponding camera poses as well as the interior orientation. In the typical NeRF workflow, the camera poses and the interior orientation are estimated in advance with Structure from Motion (SfM). But the quality of the resulting novel views, which depends on different parameters such as the number and distribution of available images, the accuracy of the related camera poses and interior orientation, but also the reflection characteristics of the depicted scene, is difficult to predict. In addition, SfM is a time-consuming pre-processing step, and its robustness and quality strongly depend on the image content. Furthermore, the undefined scaling factor of SfM hinders subsequent steps in which metric information is required. In this paper, we evaluate the potential of NeRFs for industrial robot applications. To start with, we propose an alternative to SfM pre-processing: we capture the input images with a calibrated camera that is attached to the end effector of an industrial robot and determine accurate camera poses with metric scale based on the robot kinematics. We then investigate the quality of the novel views by comparing them to ground truth, and by computing an internal quality measure based on ensemble methods. For evaluation purposes, we acquire multiple datasets that pose challenges for reconstruction typical of industrial applications, like reflective objects, poor texture, and fine structures. We show that the robot-based pose determination reaches similar accuracy as SfM in non-demanding cases, while having clear advantages in more challenging scenarios. We also report results of the novel view quality for different NeRF approaches, showing that an additional online pose refinement may be disadvantageous. Finally, we present first results of applying the ensemble method to estimate the quality of the synthetic novel view in the absence of a ground truth.

Unveiling universal aspects of the cellular anatomy of the brain

Recent cellular-level volumetric brain reconstructions have revealed high levels of anatomic complexity. Determining which structural aspects of the brain to focus on, especially when comparing with computational models and other organisms, remains a major challenge. Here we quantify aspects of this complexity and show evidence that brain anatomy satisfies universal scaling laws, establishing the notion of structural criticality in the cellular structure of the brain. Our framework builds upon understanding of critical systems to provide clear guidance in selecting informative structural properties of cellular brain anatomy. As an illustration, we obtain estimates for critical exponents in the human, mouse and fruit fly brains and show that they are consistent between organisms, to the extent that data limitations allow. Such universal quantities are robust to many of the microscopic details of the cellular structures of individual brains, providing a key step towards generative computational models of the cellular structure of the brain, and also clarifying in which sense one animal may be a suitable anatomic model for another.

The Potential of Neural Radiance Fields and 3D Gaussian Splatting for 3D Reconstruction from Aerial Imagery

Abstract. In this paper, we focus on investigating the potential of advanced Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting for 3D scene reconstruction from aerial imagery obtained via sensor platforms with an almost nadir-looking camera. Such a setting for image acquisition is convenient for capturing large-scale urban scenes, yet it poses particular challenges arising from imagery with large overlap, very short baselines, similar viewing direction and almost the same but large distance to the scene, and it therefore differs from the usual object-centric scene capture. We apply a traditional approach for image-based 3D reconstruction (COLMAP), a modern NeRF-based approach (Nerfacto) and a representative for the recently introduced 3D Gaussian Splatting approaches (Splatfacto), where the latter two are provided in the Nerfstudio framework. We analyze results achieved on the recently released UseGeo dataset both quantitatively and qualitatively. The achieved results reveal that the traditional COLMAP approach still outperforms Nerfacto and Splatfacto approaches for various scene characteristics, such as less-textured areas, areas with high vegetation, shadowed areas and areas observed from only very few views.