Volumetric Reconstruction Research Articles

Quasi/Periodic Noise Reduction in Images Using Modified Multiresolution-Convolutional Neural Networks for 3D Object Reconstructions and Comparison with Other Convolutional Neural Network Models

Quasi/Periodic Noise Reduction in Images Using Modified Multiresolution-Convolutional Neural Networks for 3D Object Reconstructions and Comparison with Other Convolutional Neural Network Models

Self-supervised rigid object 3-D motion estimation from monocular video

Independent object 3D motion estimation is a fundamental problem in 3D computer vision. Directly segmenting and estimating rigid object 3D motion from a consistent video frame is an ill-posed problem. We present a self-supervised framework for segmenting moving independent rigid objects and estimating their motion information including location, driving direction, and speed from a monocular video. Specifically, we first estimate depth, optical flow, and camera pose between a pair of video frames, and then synthesize a new 3D viewpoint from this pair. Subsequently, the Motion Recurrent All-Pairs Field Transforms (MRAFT) module is introduced to extract 3D scene flow and a motion area binary mask from a pair of images and depth. After that, a Rigid Object Motion Estimation Module (ROMEM) with a slot attention mechanism is proposed to extract rigid object motion masks from a multi-layer motion field, including optical flow, depth changes, refined scene flow, and motion masks. Finally, 2D images and 3D scene reconstruction errors are used to facilitate self-supervised training for rigid object motion. Experiments on the FlyingThings3D and KITTI datasets demonstrate that our method outperforms other advanced algorithms in estimating depth, optical flow, scene flow, and rigid moving object masks, demonstrating the benefits of our approach.

Stereo reconstruction from microscopic images for computer-assisted ophthalmic surgery.

This work presents a novel platform for stereo reconstruction in anterior segment ophthalmic surgery to enable enhanced scene understanding, especially depth perception, for advanced computer-assisted eye surgery by effectively addressing the lack of texture and corneal distortions artifacts in the surgical scene. The proposed platform for stereo reconstruction uses a two-step approach: generating a sparse 3D point cloud from microscopic images, deriving a dense 3D representation by fitting surfaces onto the point cloud, and considering geometrical priors of the eye anatomy. We incorporate a pre-processing step to rectify distortion artifacts induced by the cornea's high refractive power, achieved by aligning a 3D phenotypical cornea geometry model to the images and computing a distortion map using ray tracing. The accuracy of 3D reconstruction is evaluated on stereo microscopic images of ex vivo porcine eyes, rigid phantom eyes, and synthetic photo-realistic images. The results demonstrate the potential of the proposed platform to enhance scene understanding via an accurate 3D representation of the eye and enable the estimation of instrument to layer distances in porcine eyes with a mean average error of 190 , comparable to the scale of surgeons' hand tremor. This work marks a significant advancement in stereo reconstruction for ophthalmic surgery by addressing corneal distortions, a previously often overlooked aspect in such surgical scenarios. This could improve surgical outcomes by allowing for intra-operative computer assistance, e.g., in the form of virtual distance sensors.

Super-resolution of biomedical volumes with 2D supervision.

Volumetric biomedical microscopy has the potential to increase the diagnostic information extracted from clinical tissue specimens and improve the diagnostic accuracy of both human pathologists and computational pathology models. Unfortunately, barriers to integrating 3-dimensional (3D) volumetric microscopy into clinical medicine include long imaging times, poor depth/z-axis resolution, and an insufficient amount of high-quality volumetric data. Leveraging the abundance of high-resolution 2D microscopy data, we introduce masked slice diffusion for super-resolution (MSDSR), which exploits the inherent equivalence in the data-generating distribution across all spatial dimensions of biological specimens. This intrinsic characteristic allows for super-resolution models trained on high-resolution images from one plane (e.g., XY) to effectively generalize to others (XZ, YZ), overcoming the traditional dependency on orientation. We focus on the application of MSDSR to stimulated Raman histology (SRH), an optical imaging modality for biological specimen analysis and intraoperative diagnosis, characterized by its rapid acquisition of high-resolution 2D images but slow and costly optical z-sectioning. To evaluate MSDSR's efficacy, we introduce a new performance metric, SliceFID, and demonstrate MSDSR's superior performance over baseline models through extensive evaluations. Our findings reveal that MSDSR not only significantly enhances the quality and resolution of 3D volumetric data, but also addresses major obstacles hindering the broader application of 3D volumetric microscopy in clinical diagnostics and biomedical research.

Multi-granularity relationship reasoning network for high-fidelity 3D shape reconstruction

Monocular image-based 3D reconstruction is widely used in virtual reality, augmented reality, and autonomous driving, which benefits from the rapid development of deep learning approaches. Most of the available methods focused on reconstructing the overall shape of the object while ignoring some fine-grained details. Moreover, these methods make it hard to exactly reconstruct complex topological structures. In this paper, we propose a multi-granularity relationship reasoning network (MGRRNet), which aims to recover 3D shapes with high fidelity and rich details via the relationship reasoning between different granularity information. Specifically, our model captures the discriminative and detailed features at different granularities for extracting attentional regions. Then we perform the relationship reasoning between different granularities to reinforce the multi-granularity consistency and inter-granularity correlation. By doing this, our network is able to achieve robust feature representation and fine reconstruction. During the learning process, we jointly optimize procedures of different granularity feature representations via a sequence of inter-granularity cycle loss iterations. Extensive experimental results on two publicly available datasets justify that our approach achieves competitive performance compared to the state-of-the-art methods. Codes and all resources will be publicly available at https://github.com/Ray-tju/MGRRNet.

Multi-view 3D reconstruction of seedling using 2D image contour

3D reconstruction of seedling can provide comprehensive and quantitative spatial structure information, offering an effective digital tool for breeding research. However, accurate and efficient reconstruction of seedling is still a challenging work due to limited performance of depth sensor for seedling with small-size stem and unavoidable error for multi-view point cloud registration. Therefore, in this paper, we propose an accurate multi-view 3D reconstruction method for seedling using 2D image contour to constrain 3D point cloud. The rotation axis is calibrated and optimised by minimising point-to-contour distance between 2D image contour and projected exterior points from 3D point cloud. Then, to remove outliers and noise, we introduce the seedling mask of 2D image to constrained and delete projected outlier points of 3D model from corresponding view. Furthermore, we propose a residual-guided method to recognise missing region for 3D model and complete 3D model of small-size stem. Finally, we can obtain an accurate 3D model of seedling. The reconstruction accuracy is evaluated by average distance between projected contour of 3D model and 2D image contour of all views (0.3185 mm). Then, the phenotypic parameters were calculated from 3D model and the results are close to manual measurements (Plant height: R2 = 0.98, RMSE = 2.3 mm, rRMSE = 1.52%; Petioles inclination angle: R2 = 0.99, RMSE = 0.73°, rRMSE = 1.41%; Leaf area: R2 = 0.66, RMSE = 1.05 cm2, rRMSE = 7.63%; Leaf inclination angle: R2 = 0.99, RMSE = 1.01°, rRMSE = 1.72%; Stem diameter: R2 = 0.95, RMSE = 0.12 mm, rRMSE = 5.43%). Breeders can improve the selection of more resilient varieties and cultivars to different growing conditions starting from the dynamic analysis of their phenotype.

Planar and volumetric imaging dosimetry for longitudinal projection extended cone beam CT based adaptive radiotherapy

BackgroundThe objective of this study is to provide a detailed insight into the broad beam dosimetric characteristics of two commercial cone beam computed tomography (CBCT) imaging systems to complement the longitudinal projection extended CBCT based image guided adaptive radiotherapy (IGART). Methods and materialsThe absolute dose calibration, planar and volumetric dosimetric characteristics of two CBCT imaging systems viz., the Clinac 2100 C/D On-Board Imager (OBI) and the TrueBeam X-ray Imaging system (XI) (Varian Medical Systems, Palo Alto, CA) were investigated. Reference dosimetry specific to volumetric CBCT imaging mode was performed as per American Association of Physicists in Medicine (AAPM) task group (TG) 61 protocol using calibrated 0.6 cc Farmer ionization chamber (30010, PTW Dosimetry Company, Germany). The rotational planar and volumetric dosimetry was performed after calibrating the OCTAVIUS 1500 detector array as a mere photon detector, with and without the detachable hemisphere of 4D Modular phantom. ResultsThe absolute dose of half-fan XI CBCT was found to be 1.2 times lower when compared to OBI CBCT. The abutting kV cone beam dosimetry specific to dual-scan CBCT fusion technique showed a wider in-plane profile for the OBI system when compared to the XI system, due to an additional field opening of 3.4 cm used for half-fan mode in the former. The reconstructed volumetric imaging dose distribution specific to half-fan geometry and dynamic movement of gantry, brought to light valuable details about the kV dose distribution. ConclusionThe planar and 3D imaging dosimetry reported in this study paves the way for the integration of volumetric imaging dose reconstruction capabilities during high precision radiotherapy to demonstrate a complete picture of the total dose delivered to the patient during the planned course of IGART.

GraM: Geometric Structure Embedding into Attention Mechanisms for 3D Point Cloud Registration

3D point cloud registration is a crucial technology for 3D scene reconstruction and has been successfully applied in various domains, such as smart healthcare and intelligent transportation. With theoretical analysis, we find that geometric structural relationships are essential for 3D point cloud registration. The 3D point cloud registration method achieves excellent performance only when fusing local and global features with geometric structure information. Based on these discoveries, we propose a 3D point cloud registration method based on geometric structure embedding into the attention mechanism (GraM), which can extract the local features of the non-critical point and global features of the corresponding point containing geometric structure information. According to the local and global features, the simple regression operation can obtain the transformation matrix of point cloud pairs, thereby eliminating the semantics that ignores the geometric structure relationship. GraM surpasses the state-of-the-art results by 0.548° and 0.915° regarding the relative rotation error on ModelNet40 and LowModelNet40, respectively.

Improved 3D human face reconstruction from 2D images using blended hard edges

This study reports an effective and robust edge-based scheme for the reconstruction of 3D human faces from input of single images, addressing drawbacks of existing methods in case of large face pose angles or noisy input images. Accurate 3D face reconstruction from 2D images is important, as it can enable a wide range of applications, such as face recognition, animations, games and AR/VR systems. Edge features extracted from 2D images contain wealthy and robust 3D geometric information, which were used together with landmarks for face reconstruction purpose. However, the accurate reconstruction of 3D faces from contour features is a challenging task, since traditional edge or contour detection algorithms introduce a great deal of noise, which would adversely affect the reconstruction. This paper reports on the use of a hard-blended face contour feature from a neural network and a Canny edge extractor for face reconstruction. The quantitative results indicate that our method achieves a notable improvement in face reconstruction with a Euclidean distance error of 1.64 mm and a normal vector distance error of 1.27 mm when compared to the ground truth, outperforming both traditional and other deep learning-based methods. These metrics show particularly significant advancements, especially in face shape reconstruction under large pose angles. The method also achieved higher accuracy and robustness on in-the-wild images under conditions of blurring, makeup, occlusion and poor illumination.

Multiscale Attention and Dilated Convolutional Neural Network-based 3D Scene Reconstruction for Moving Objects

Multiscale Attention and Dilated Convolutional Neural Network-based 3D Scene Reconstruction for Moving Objects

Photographic Parameters in Three-Dimensional Facial Image Acquisition. A Scoping Review.

Orthognathic surgery is a viable and reproducible treatment for facial deformities. Despite the precision of the skeletal planning of surgical procedures, there is little information about the relations between hard and soft tissues in three-dimensional (3D) analysis, resulting in unpredictable soft tissue outcomes. Three-dimensional photography is a viable tool for soft tissue analysis because it is easy to use, has wide availability, low cost, and is harmless. This review aims to establish parameters for acquiring consistent and reproducible 3D facial images. A scoping review was conducted across PubMed, SCOPUS, Scientific Electronic Library Online (SciELO), and Web of Science databases, adhering to "Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews" guidelines. Articles presenting 3D facial photographs in the diagnostic phase were considered. A total of 79 articles were identified, of which 29 were selected for analysis. The predominant use of automated systems like 3dMD and VECTRA M3 was noted. User positioning has highest agreement among authors. Noteworthy aspects include the importance of proper lighting, facial expression, and dental positioning, with observed discrepancies and inconsistencies among authors. Finally, the authors proposed a 3D image acquisition protocol based on this research findings.

Consumer-grade imaging system for NDVI measurement at plant scale by a farmer robot

The Normalized Difference Vegetation Index (NDVI) is a valuable indicator of plant vigor that is frequently used in agronomic practices to make timely and targeted decisions with the aim of increasing the productivity of the system. NDVI measurements of large-scale fields are typically performed using remote sensing from satellite and aerial imaging devices. However, due to their low spatial and temporal resolution, these technologies may have limitations in precision viticulture. This paper investigates the potential of a proximal sensing system to characterize the vine foliage that makes use of data collected by a farmer robot equipped with an Intel RealSense D435 camera. The camera includes two infrared (IR) sensors in stereoscopic configuration and one RGB sensor, which provide, for each observation, both infrared and visible red channel information, thus making possible pixel-per-pixel NDVI calculation. Solutions to IR filtering and radiometric calibration issues are proposed that significantly improve measurement accuracy and reliability. Since the camera also provides stereo-based 3D scene reconstruction, depth information can be used to separate the plant canopy from the background before NDVI measurement. At the same time, range data can be employed to extract geometric properties of the crop, such as plant height and/or volume. The system is validated in the field in a commercial vineyard at different phenological stages, from the formation of the berries to leaf discoloration and fall. Experimental results show good agreement compared with ground truth provided by a GreenSeeker, with a mean absolute percentage error in the NDVI estimation of 4.6% and a R2 of 0.87 tested over the whole grapevine cycle. Therefore, the proposed sensing system could be a feasible solution to automated NDVI estimation at plant-scale.

Depth estimation and 3D reconstruction from UAV-borne imagery: Evaluation on the UseGeo dataset

Depth estimation and 3D model reconstruction from aerial imagery is an important task in photogrammetry, remote sensing, and computer vision. To compare the performance of different image-based approaches, this study presents a benchmark for UAV-based aerial imagery using the UseGeo dataset. The contributions include the release of various evaluation routines on GitHub, as well as a comprehensive comparison of baseline approaches, such as methods for offline multi-view 3D reconstruction resulting in point clouds and triangle meshes, online multi-view depth estimation, as well as single-image depth estimation using self-supervised deep learning. With the release of our evaluation routines, we aim to provide a universal protocol for the evaluation of depth estimation and 3D reconstruction methods on the UseGeo dataset. The conducted experiments and analyses show that each method excels in a different category: the depth estimation from COLMAP outperforms that of the other approaches, ACMMP achieves the lowest error and highest completeness for point clouds, while OpenMVS produces triangle meshes with the lowest error. Among the online methods for depth estimation, the approach from the Plane-Sweep Library outperforms the FaSS-MVS approach, while the latter achieves the lowest processing time. And even though the particularly challenging nature of the dataset and the small amount of training data leads to a significantly higher error in the results of the self-supervised single-image depth estimation approach, it outperforms all other approaches in terms of processing time and frame rate. In our evaluation, we have also considered modern learning-based approaches that can be used for image-based 3D reconstruction, such as NeRFs. However, due to the significantly lower quality of the resulting 3D models, we have only included a qualitative comparison between NeRF-based and conventional approaches in the scope of this work.

Elucidating the Benefit of Perforated vs Non-Perforated Membranes in Guided Bone Regeneration: An in Vivo Histologic Evaluation and Histomorphometric Analysis.

Non-perforated Polytetrafluoroethylene (PTFE) membranes are effectively utilized in guided bone regeneration (GBR) but may hinder cell migration due to limited interaction with the periosteum. This study compared bone regeneration using occlusive or perforated membranes combined with acellular collagen sponge (ACS) and recombinant human bone morphogenic protein-2 (rhBMP-2) in a canine mandibular model. Male beagle dogs (n=3) received two mandibular defects each to compare ACS/rhBMP-2 with experimental (perforated group) and control (non-perforated group) membranes (n=3 defects/group). Tissue healing was assessed histomorphologically, histomorphometrically and through volumetric reconstruction using microcomputed tomography. The perforated group showed increased bone formation and reduced soft tissue formation compared to the non-perforated group. For the primary outcome, histomorphometric analysis revealed significantly greater total regenerated bone in the perforated group (67.08 ± 6.86%) relative to the nonperforated group (25.18 ± 22.44%) (p = 0.036). Perforated membranes had less soft tissue infiltration (32.91 ± 6.86%) compared to non-perforated membranes (74.82 ± 22.44%) (p = 0.036). The increased permeability of membranes in the perforated group potentially enabled periosteal precursor cells greater accessibility to rhBMP-2. The availability may have accelerated their differentiation into mature bone-forming cells, contributing to the stimulation of new bone production, relative to the non-perforated group.

A method of three-dimensional muon imaging using cosmic ray ground array

Cosmic ray muon imaging technology is an effective non-destructive imaging technique. It is currently used to survey the internal structure of large-scale objects such as active volcanoes, pyramids, and some buildings. Additionally, it is used to detect high-Z materials in scenarios such as border security, nuclear reactor monitoring, and container inspection. All applications of cosmic ray muons require a detector to reveal and measure the flux or angular variations of muons. However, detectors may have specific characteristics for each application depending on the detection requirements. Unlike single-point track detectors, we propose using ground array detectors to investigate how ground array detection technology can be used for 3D reconstruction of objects. We use ground array detection technology for simulation studies and apply the Iterative Correction Algorithm (ICA) to reconstruct two-dimensional projections and obtain three-dimensional images of density distribution. We have also addressed the convergence problem and analyzed and optimized some parameters that may affect the detection results. The advantages of using ground arrays include their ability to simultaneously measure the transmittance flux data of muons in different directions, achieve multi-angle detection, and have a large field of view, enabling the collection of sufficient data for more accurate detection results. To verify the feasibility of our new method, we conducted a simple experimental validation on a cylindrical cement building located in the Large High Altitude Air Shower Observatory (LHAASO). By comparing the experimental results with the cement building in the array, we found that their shape structure and density information are basically consistent, demonstrating the effectiveness of our method.

State of the Art on Diffusion Models for Visual Computing

AbstractThe field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes. In these domains, diffusion models are the generative AI architecture of choice. Within the last year alone, the literature on diffusion‐based tools and applications has seen exponential growth and relevant papers are published across the computer graphics, computer vision, and AI communities with new works appearing daily on arXiv. This rapid growth of the field makes it difficult to keep up with all recent developments. The goal of this state‐of‐the‐art report (STAR) is to introduce the basic mathematical concepts of diffusion models, implementation details and design choices of the popular Stable Diffusion model, as well as overview important aspects of these generative AI tools, including personalization, conditioning, inversion, among others. Moreover, we give a comprehensive overview of the rapidly growing literature on diffusion‐based generation and editing, categorized by the type of generated medium, including 2D images, videos, 3D objects, locomotion, and 4D scenes. Finally, we discuss available datasets, metrics, open challenges, and social implications. This STAR provides an intuitive starting point to explore this exciting topic for researchers, artists, and practitioners alike.

M-NeuS: Volume rendering based surface reconstruction and material estimation

Although significant advances have been made in the field of multi-view 3D reconstruction using implicit neural field-based methods, existing reconstruction methods overlook the estimation of the material information (e.g. the base color, albedo, roughness, and metallic) during the learning process. In this paper, we propose a novel differentiable rendering framework, named as material NueS (M-NeuS), for simultaneously achieving precise surface reconstruction and competitive material estimation. For surface reconstruction, we perform multi-view geometry optimization by proposing an enhanced-low-to-high frequency encoding registration strategy (EFERS) and a second-order interpolated signed distance function (SI-SDF) for precise details and outline reconstruction. For material estimation, inspired by the NeuS, we first propose a volume-rendering-based material estimation strategy (VMES) to estimate the base color, albedo, roughness, and metallic accurately. And then, different from most material estimation methods that need ground-truth geometric priors, we use the geometry information reconstructed in the surface reconstruction stage and the directions of incidence from different viewpoints to model a neural light field, which can extract the lighting information from image observations. Next, the extracted lighting and the estimated base color, albedo, roughness, and metallic are optimized by the physics-based rendering equation. Extensive experiments demonstrate that our M-NeuS can not only reconstruct more precise geometry surface than existing state-of-the-art (SOTA) reconstruction methods but also can estimate competitive material information: the base color, albedo, roughness, and metallic.

FusionVision: A Comprehensive Approach of 3D Object Reconstruction and Segmentation from RGB-D Cameras Using YOLO and Fast Segment Anything.

In the realm of computer vision, the integration of advanced techniques into the pre-processing of RGB-D camera inputs poses a significant challenge, given the inherent complexities arising from diverse environmental conditions and varying object appearances. Therefore, this paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. Traditional computer vision systems face limitations in simultaneously capturing precise object boundaries and achieving high-precision object detection on depth maps, as they are mainly proposed for RGB cameras. To address this challenge, FusionVision adopts an integrated approach by merging state-of-the-art object detection techniques, with advanced instance segmentation methods. The integration of these components enables a holistic (unified analysis of information obtained from both color RGB and depth D channels) interpretation of RGB-D data, facilitating the extraction of comprehensive and accurate object information in order to improve post-processes such as object 6D pose estimation, Simultanious Localization and Mapping (SLAM) operations, accurate 3D dataset extraction, etc. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. Subsequently, FastSAM, an innovative semantic segmentation model, is applied to delineate object boundaries, yielding refined segmentation masks. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation, enhancing overall precision in 3D object segmentation.

Sequential selection and calibration of video frames for 3D outdoor scene reconstruction

Abstract3D scene understanding and reconstruction aims to obtain a concise scene representation from images and reconstruct the complete scene, including the scene layout, objects bounding boxes and shapes. Existing holistic scene understanding methods primarily recover scenes from single images, with a focus on indoor scenes. Due to the complexity of real‐world, the information provided by a single image is limited, resulting in issues such as object occlusion and omission. Furthermore, captured data from outdoor scenes exhibits characteristics of sparsity, strong temporal dependencies and a lack of annotations. Consequently, the task of understanding and reconstructing outdoor scenes is highly challenging. The authors propose a sparse multi‐view images‐based 3D scene reconstruction framework (SMSR). It divides the scene reconstruction task into three stages: initial prediction, refinement, and fusion stage. The first two stages extract 3D scene representations from each viewpoint, while the final stage involves selection, calibration and fusion of object positions and orientations across different viewpoints. SMSR effectively address the issue of object omission by utilizing small‐scale sequential scene information. Experimental results on the general outdoor scene dataset UrbanScene3D‐Art Sci and our proprietary dataset Software College Aerial Time‐series Images, demonstrate that SMSR achieves superior performance in the scene understanding and reconstruction.

Hierarchical mussel farm reconstruction from video with object tracking

ABSTRACT As the mussel farming industry expands, particularly in regions such as New Zealand, there is a growing need for advanced monitoring and management solutions to ensure sustainability and operational efficiency. The current reliance on manual infrequent observation of aquaculture structures limits farmers' ability to monitor them in real time. Addressing these challenges, large-scale 3D reconstruction provides a practical solution by facilitating the creation of replicable depictions of mussel farm scenes and floatation buoys derived from recorded video, supporting off-site evaluation and enabling precise decision making. This paper introduces a novel approach to enhance the visualisation and monitoring of mussel farm floatation through a hierarchical reconstruction process. In contrast to earlier studies, we focus on recovering not only the overall environment (background) but also the finer details of key elements such as buoys to create a comprehensive representation of mussel farm geometry and appearance. We propose to segment the scene into the background and granular object instances and reconstruct them separately in a multi-stage process. The initial 3D scene reconstruction is performed using the Structure-from-Motion (SfM) technique, leveraging video footage captured by a vessel-mounted camera. This coarse reconstruction serves as the foundation for subsequent fine-grained enhancements. To recover finer details, object tracking is applied and the trajectories obtained are then conjunct with geometry triangulation to determine the real-world positions of individual buoys. A multiple-scale denoising method, grounded in dominant direction correlation, is implemented to eliminate non-reliable tracking objects and reduce reconstruction artifacts, ensuring the accuracy of the final results. This hierarchical reconstruction approach contributes to the advancement of mussel farm management by offering a powerful technology for comprehensive visualisation, enabling farmers to make informed decisions based on a detailed understanding of the mussel farm's geometry and dynamics.