Accurate 3D Reconstruction Research Articles

Combined imaging and chromatic confocal microscopy technique to characterize size and shape of ensembles of cuboidal particles

The presence of needle- and plate-like particles has detrimental consequences on their downstream processing in the fine chemicals sector. Therefore, the ability to accurately characterize the particle size and shape is essential to quantify and predict their impact on the product processability. Nonetheless, tools to characterize both the size and shape of ensembles of cuboidal particles are seldom available. The overarching goal of this work is to provide a fast and accurate offline size and shape characterization tool. To this aim, we have designed and experimentally validated a combined imaging and chromatic confocal microscopy technique. We propose two modes of operation: one that facilitates the accurate 3D reconstruction of particles; and the other that facilitates their rapid characterization. We validate the performance of our technique using a commercial technique. We show that our technique can accurately characterize thousands of particles, making it a valuable addition to existing process analytical technology.

Deep learning-based correction of defocused fringe patterns for high-speed 3D measurement

Digital fringe projection profilometry often faces a trade-off between measurement accuracy and efficiency. Defocus technology is commonly employed to address this challenge and improve the efficiency of high-speed three-dimensional (3D) measurement. This technology uses 1-bit binary fringe patterns instead of traditional 8-bit sinusoidal patterns, but accurately measuring 3D shapes with both high speed and accuracy remains a challenge due to defocus errors. These errors are introduced by the manual adjustment of lens focal length and reduce both fringe pattern quality and measurement accuracy. To overcome this limitation, we propose a multi-stage generative adversarial network with a self-attention mechanism to correct inaccurate fringe patterns and transform them into more ideal sinusoidal fringe patterns. Our generation network comprises a multi-stage feature extraction network with a self-attention mechanism and an encoder-decoder network. A multi-stage network integrating residual and transformer modules is constructed to mine global feature information. The self-attention mechanism accurately detects key areas for correction, and the encoder-decoder network generates rectified sinusoidal fringe patterns by combining the feature information with the attention area. We use a discriminative network to evaluate whether the output of the generative network is good enough to be true. In our experiments, we considered different fringe widths and measured objects of various types and colors. The results show that our proposed method improves the quality of defocus fringe patterns and the accuracy of subsequent 3D reconstruction compared to existing direct defocus methods.

Robust structured light 3D imaging with two fringe patterns using recurrent classification neural network

Robust and accurate 3D reconstruction using a limited number of fringe patterns has posed a challenge in the field of structured light 3D imaging. Unlike traditional approaches that rely on multiple fringe patterns, using only one or two patterns makes phase recovery and unwrapping difficult. To address this issue, a recurrent classification neural network (RCNN) has been developed, transforming the phase recovery and unwrapping tasks into a unified phase classification task. First, a training dataset consisting of 1200 groups of data was collected to generate a total of 38 400 training samples, enabling the RCNN to learn the mapping between the input fringe patterns and the corresponding label maps. Then, based on the well-trained network, a label map is generated based on the input two fringe patterns using the output classification results. Finally, 3D reconstruction data could be obtained by combining the inferred label map with the vision system’s parameters. A series of comprehensive experiments have been conducted to validate the performance of the proposed method.

Endoscope 3D reconstruction based on ORB-SLAM

Minimal invasive surgery (MIS) is the mainstream trend in developing surgical technology. As the endoscope is a significant tool in the surgical process, whether it can track the inner cavity and realize accurate real-time 3D reconstruction has a vital impact on the smooth progress of MIS. However, there are still problems in the endoscopic environment, such as severe image distortion, the effect of lighting conditions, and the inability to extract lumen textures. Orinted fast and rotated brief simultaneous localization and mapping (ORB-SLAM) is currently a relatively advanced simultaneous localization and mapping (SLAM) method with better performance. The ORB-SLAM-based endoscope 3D reconstruction method can improve performance and overcome the challenge of endoscope 3D reconstruction. This paper will first introduce several existing endoscope 3D reconstruction methods based on ORB-SLAM and analyze the limitations and issues of these methods through their experimental results. Then the paper will explore the solutions to the defects in this method from other methods and compare the characteristics and the result of experiments. Secondly, through the summary of the above methods and the introduction of the integration of the ORB-SLAM-based methods and other current advanced technologies, the future development trend and huge development potential of ORB-SLAM-based endoscopic 3D reconstruction are introduced. This paper will be of profound affection to further improve the optimization and application of the ORB-SLAM-based endoscope 3D reconstruction method.

UniRender: Reconstructing 3D Surfaces from Aerial Images with a Unified Rendering Scheme

While recent advances in the field of neural rendering have shown impressive 3D reconstruction performance, it is still a challenge to accurately capture the appearance and geometry of a scene by using neural rendering, especially for remote sensing scenes. This is because both rendering methods, i.e., surface rendering and volume rendering, have their own limitations. Furthermore, when neural rendering is applied to remote sensing scenes, the view sparsity and content complexity that characterize these scenes will severely hinder its performance. In this work, we aim to address these challenges and to make neural rendering techniques available for 3D reconstruction in remote sensing environments. To achieve this, we propose a novel 3D surface reconstruction method called UniRender. UniRender offers three improvements in locating an accurate 3D surface by using neural rendering: (1) unifying surface and volume rendering by employing their strengths while discarding their weaknesses, which enables accurate 3D surface position localization in a coarse-to-fine manner; (2) incorporating photometric consistency constraints during rendering, and utilizing the points reconstructed by structure from motion (SFM) or multi-view stereo (MVS), to constrain reconstructed surfaces, which significantly improves the accuracy of 3D reconstruction; (3) improving the sampling strategy by locating sampling points in the foreground regions where the surface needs to be reconstructed, thus obtaining better detail in the reconstruction results. Extensive experiments demonstrate that UniRender can reconstruct high-quality 3D surfaces in various remote sensing scenes.

Completing point clouds using structural constraints for large-scale points absence in 3D building reconstruction

The completion of point cloud directly affects the accuracy of primitive parameter extraction and 3D reconstruction. Due to the limitations of sensor scanning, point cloud are often incomplete due to occlusions during the data collection. To solve the problem of incomplete building point clouds and to support geometric detail LoD2 reconstruction, a point cloud completion method with structural constraint is proposed in this paper. First, based on the idea of cloth simulation, regularity and planarity of the building were used to impose structural constraints on the cloth nodes. Combined with the rigidity of the building, the force rules of nodes under the distribution of the building structure are formulated. Then, the sinking and rebounding stages of the node were simulated to recover the position of point cloud in the area of interest. Finally, the missing surface in each façade direction was completed by attitude adjustment. The solution of a multi-layer structural façade was realized by the selection of parameters during the sinking and rebounding stage. The feasibility and accuracy of the proposed point cloud completion method have been verified through experiments. It is demonstrated that the proposed method can improve the integrity of a building point cloud by achieving more than 90% coverage. Compared with existing methods, the proposed method can deal with the problem of large-scale structure missing from buildings with fewer limitations. The patched points do not rely on the geometric primitives’ extraction and are based on the distribution of existing points without redundant observations, so the patched points adapt better to the existing building points. The proposed method has enhanced the utilization of point clouds and improved the detail of LoD2 reconstruction. It provides a new feasible method for point cloud completion. The code of the method is available at https://github.com/bufanzhao/point-cloud-completion.git.

Automatic reconstruction and modeling of dormant jujube trees using three-view image constraints for intelligent pruning applications

Automatic reconstruction and modeling of dormant jujube trees provide a basis and prerequisite for intelligent pruning applications in large-scale, dwarf, and high-density planting orchards. For accurate 3D reconstruction of the trunk and pruning branches, this study presents a method based on a multi-view image sequence of dormant jujube trees. It involves four main steps. First, we determine the way of image acquisition, equipment parameters and posture, and reconstruction accuracy by investigating and analyzing their planting mode to meet the requirements of robot intelligent pruning operation. Second, performing image matching in images of dormant jujube trees is challenging due to few local feature regions, weak continuity, and difference in individual morphological structures. We provide a bilateral image-matching method based on the three-view geometry constraint. Third, this study optimizes a camera self-calibration algorithm based on image EXIF tags by broadening its applicability to improve reconstruction efficiency. It does not rely on the particular geometric constraints in the specific scenes. Finally, a dense 3D point cloud of a tree is built automatically with this improved SfM reconstruction strategy and combined with the PMVS algorithm. Both qualitative and quantitative evaluations are performed on generating 3D models of eight dormant jujube trees and their number of branches, average plant height, and crown are used as indicators. The results show that our method identifies the number of pruning branches with an accuracy of 92% and provides a performance of 95% for measuring the diameter of the plant height and canopy. The results provide visual research basis and technical support for the intelligent pruning of jujube trees.

A novel method for obtaining gamma precoding values

AbstractThe grating projection‐based three‐dimensional (3D) measurement technique has been widely used in various fields, such as industrial inspection, cultural relic protection, and reverse engineering. However, due to the nonlinear response between the input and output of the measurement system, phase measurement errors occur, which reduces the accuracy of 3D reconstruction, which is known as the gamma effect. To address this issue, a novel method has been proposed to reduce the nonlinear response of the system. Ultimately, this method reduces the impact of the gamma value on the phase accuracy in subsequent measurements. Experimental results have shown that this method can quickly calibrate the correct precoding gamma value, effectively reduce the gamma nonlinear response during the measurement process, and improve measurement accuracy.

3D reconstruction of coronary artery bifurcations from intravascular ultrasound and angiography

Coronary bifurcation lesions represent a challenging anatomical subset, and the understanding of their 3D anatomy and plaque composition appears to play a key role in devising the optimal stenting strategy. This study proposes a new approach for the 3D reconstruction of coronary bifurcations and plaque materials by combining intravascular ultrasound (IVUS) and angiography. Three patient-specific silicone bifurcation models were 3D reconstructed and compared to micro-computed tomography (µCT) as the gold standard to test the accuracy and reproducibility of the proposed methodology. The clinical feasibility of the method was investigated in three diseased patient-specific bifurcations of varying anatomical complexity. The IVUS-based 3D reconstructed bifurcation models showed high agreement with the µCT reference models, with r2 values ranging from 0.88 to 0.99. The methodology successfully 3D reconstructed all the patient bifurcations, including plaque materials, in less than 60 min. Our proposed method is a simple, time-efficient, and user-friendly tool for accurate 3D reconstruction of coronary artery bifurcations. It can provide valuable information about bifurcation anatomy and plaque burden in the clinical setting, assisting in bifurcation stent planning and education.

Improving Monocular Camera Localization for Video-Based Three-Dimensional Outer Ear Reconstruction Tasks

This work addresses challenges related to camera 3D localization while reconstructing a 3D model of an ear. This work explores the potential solution of using a cap, specifically designed not to obstruct the ear, and its efficiency in enhancing the camera localization for structure-from-motion (SfM)-based object reconstruction. The proposed solution is described, and an elaboration of the experimental scenarios used to investigate the background textures is provided; data collection and software tools used in the research are reported. The results show that the proposed method is effective, and using the cap with texture leads to a reduction in the camera localization error. Errors in the 3D location reconstruction of the camera were calculated by comparing cameras localized within typical ear reconstruction situations to those of higher-accuracy reconstructions. The findings also show that caps with sparse dot patterns and a regular knitted patterned winter hat are the preferred patterns. The study provides a contribution to the field of 3D modeling, particularly in the context of creating 3D models of the human ear, and offers a step towards more accurate, reliable, and feasible 3D ear modeling and reconstruction.

Adjustable method based on body parts for improving the accuracy of 3D reconstruction in visually important body parts from silhouettes

Adjustable method based on body parts for improving the accuracy of 3D reconstruction in visually important body parts from silhouettes

Virtual alignment of pathology image series for multi-gigapixel whole slide images

Interest in spatial omics is on the rise, but generation of highly multiplexed images remains challenging, due to cost, expertise, methodical constraints, and access to technology. An alternative approach is to register collections of whole slide images (WSI), generating spatially aligned datasets. WSI registration is a two-part problem, the first being the alignment itself and the second the application of transformations to huge multi-gigapixel images. To address both challenges, we developed Virtual Alignment of pathoLogy Image Series (VALIS), software which enables generation of highly multiplexed images by aligning any number of brightfield and/or immunofluorescent WSI, the results of which can be saved in the ome.tiff format. Benchmarking using publicly available datasets indicates VALIS provides state-of-the-art accuracy in WSI registration and 3D reconstruction. Leveraging existing open-source software tools, VALIS is written in Python, providing a free, fast, scalable, robust, and easy-to-use pipeline for registering multi-gigapixel WSI, facilitating downstream spatial analyses.

UPU-SNet: Siamese network for unsupervised point cloud upsampling based on spatial-aware transformers

In the majority of cases, the paired dataset is not easy to obtain, which makes it difficult to train a supervised point cloud upsampling network directly. Therefore, we propose a novel siamese network architecture for the unsupervised upsampling task, called UPU-SNet. This architecture contains two branches based on hierarchical spatial-aware transformers. Specifically, one branch includes a global-local transformer (GLT) module and a standard reconstruction module for coarse point generation. The other contains a shared GLT module, a transformer shuffle (TranShuffle) module and a shared reconstruction module for dense point generation. For multi-level feature extraction, the GLT module we propose aims to explore feature fusions of global-to-local and local-to-global, respectively, with the aid of the multi-head cross-attention mechanism of the transformers. Then, the TranShuffle module is constructed closely after the GLT module to further optimize and expand the extracted features. Moreover, taking full account of sparse points, coarse points and dense points, we design a joint loss function, so as to support the network to generate denser and more uniform point clouds without using ground-truth. Through extensive experiments, we show how our proposal not only outperforms existing unsupervised methods but also achieves competitive results against previous supervised and self-supervised methods. Also, we present various surface reconstruction results on both synthetic and real point clouds, and show our network can enable accurate 3D reconstructions.

Towards an Accurate 3D Reconstruction of Nano-porous Structures using FIB Tomography and Monte Carlo Simulations with Machine Learning.

Towards an Accurate 3D Reconstruction of Nano-porous Structures using FIB Tomography and Monte Carlo Simulations with Machine Learning.

Application Value of Artificial Intelligence-assisted Three-dimensional Reconstruction in Planning Thoracoscopic Segmentectomy

The three-dimensional (3D) can assist in planning lung segmentectomy. 3D reconstruction software based on artificial intelligence algorithm is gradually applied in clinic. The aim of this study was to evaluate the accuracy and safety of 3D reconstruction assisted planning of thoracoscopic segmentectomy. A total of 90 patients admitted to Department of Thoracic Surgery of Lanzhou University Second Hospital were evaluated for thoracoscopic segmentectomy. Before operation, artificial intelligence 3D reconstruction software was used to make 3D lung images and conduct preoperative planning. Surgical videos were saved during the operation and perioperative data were recorded. Video recordings of 38 patients were selected to explore the effectiveness of artificial intelligence 3D reconstruction for surgical planning. The results of artificial intelligence 3D reconstruction and Mimics 21 software reconstruction were compared with the actual results in the operation, and the detection and classification ability of bronchus and blood vessels of the two reconstruction methods were compared. All the 90 patients underwent artificial intelligence 3D reconstruction planning, including 57 patients (63.3%) with single lung segmentectomy and 33 patients (36.7%) with combined sub-segmentectomy. The accuracy of artificial intelligence 3D reconstruction for lesion localization was 100.0%, and the accuracy of computed tomography (CT) was 94.4% (85/90). The detection accuracy of artificial intelligence 3D reconstruction and Mimics 21 software was 92.1% (35/38) and 89.5% (34/38), and the anatomic typing accuracy was 89.5% (34/38) and 84.2% (32/38), and the total accuracy was 76.3% (29/38) and 71.1% (27/38). In the comparative observation of 38 surgical videos and reconstructed images, the consistent rates of target segment planning, surgical approach, artery dissection, vein dissection and bronchial dissection for preoperative planning using artificial intelligence 3D reconstruction were 92.1% (35/38), 92.1% (35/38), 89.5% (34/38), 86.8% (33/38) and 94.7% (36/38). The overall planning operational consistency rate was 68.4% (26/38). It is accurate and safe to use artificial intelligence 3D reconstruction to assist planning thoracoscopic segmentectomy.

UHRNet: a deep learning-based method for accurate 3D reconstruction from a single fringe-pattern

ABSTRACT The quick and accurate retrieval of an object’s height from a single fringe-pattern in Fringe Projection Profilometry has been a topic of ongoing research. While existing single-shot fringe-to-depth CNN methods can directly generate height map from one pattern, their accuracy lags behind traditional phase-shifting techniques. To improve accuracy, we propose a U-shaped High-resolution Network (UHRNet). The network utilizes U-Net’s encoding-decoding structure as the backbone, employing Multi-Level Conv Blocks and High-resolution Fusion Blocks to extract features. Additionally, a compound loss function, combining Structural Similarity Index Measure Loss (SSIMLoss) and chunked L2 loss function, is devised to enhance the details of 3D reconstruction. Our method has been experimentally proven to be efficient with an average RMSE of 0.443 mm, which is 64.67% lower than hNet and 33.31% lower than ResUNet. These results indicate that the proposed approach significantly enhances the accuracy of 3D reconstruction from a single fringe-pattern.

The Calibration Method of Multi-Channel Spatially Varying Amplitude-Phase Inconsistency Errors in Airborne Array TomoSAR

Airborne array tomographic synthetic aperture radar (TomoSAR) can acquire three-dimensional (3D) information of the observed scene in a single pass. In the process of airborne array TomoSAR data imaging, due to the disturbance of factors such as inconsistent antenna patterns and baseline errors, there are spatially varying amplitude-phase inconsistency errors in the multi-channel Single-Look-Complex (SLC) images. The existence of the errors degrades the quality of the 3D imaging results, which suffer from positioning errors, stray points, and spurious targets. In this paper, a new calibration method based on multiple prominent points is proposed to calibrate the errors of amplitude-phase inconsistency. Firstly, the prominent points are selected from the multi-channel SLC data. Then, the subspace decomposition method and maximum interference spectrum method are used to extract the multi-channel amplitude-phase inconsistency information at each point. The last step is to fit the varying curve and to compensate for the errors. The performance of the method is verified using actual data. The experimental results show that compared with the traditional fixed amplitude-phase inconsistency calibration method, the proposed method can effectively calibrate spatially varying amplitude-phase inconsistency errors, thus improving on the accuracy of 3D reconstruction results for large-scale scenes.

Fast-Track Virtual Reality to Facilitate 3D Reconstruction in Congenital Heart Disease.

A limitation of the clinical use of 3D reconstruction and virtual reality (VR) systems is the relatively high cost and experience required to use hardware and software to effectively explore medical images. We have tried to simplify the process and validate a new tool developed for this purpose with a novel software. Five patients with right partial anomalous pulmonary venous return with adequate preoperative imaging acquired by magnetic resonance were enrolled. Five volunteers with no previous experience in the field of 3D reconstruction were instructed to use the software after a short video tutorial. Users were then asked to create a 3D model of each patient's heart with DIVA and their results were compared quantitatively and qualitatively with a benchmark reconstruction performed by an experienced user. All our participants recreated 3D models in a relatively short time, maintaining a good overall quality (average quality score ≥ 3 on a scale of 1-5). The overall trend of all the parameters analysed showed a statistical improvement between Case 1 and Case 5, as users become more and more experienced. DIVA is a very simple software that allows accurate 3D reconstruction in a relatively short time ("fast-track" VR). In this study, we demonstrated the potential use of DIVA by inexperienced users, with a significant improvement in quality and time after a few cases were performed. Further studies are needed to confirm the potential application of this technology on a larger scale.

Deep learning method of stochastic reconstruction of three-dimensional digital cores from a two-dimensional image.

Digital cores can characterize the true internal structure of rocks at the pore scale. This method has become one of the most effective ways to quantitatively analyze the pore structure and other properties of digital cores in rock physics and petroleum science. Deep learning can precisely extract features from training images for a rapid reconstruction of digital cores. Usually, the reconstruction of three-dimensional (3D) digital cores is performed by optimization using generative adversarial networks. The training data required for the 3D reconstruction are 3D training images. In practice, two-dimensional (2D) imaging devices are widely used because they can achieve faster imaging, higher resolution, and easier identification of different rock phases, so replacing 3D images with 2D ones avoids the difficulty of acquiring 3D images. In this paper, we propose a method, named EWGAN-GP, for the reconstruction of 3D structures from a 2D image. Our proposed method includes an encoder, a generator, and three discriminators. The main purpose of the encoder is to extract statistical features of a 2D image. The generator extends the extracted features into 3D data structures. Meanwhile, the three discriminators have been designed to gauge the similarity of morphological characteristics between cross sectionsof the reconstructed 3D structure and the real image. The porosity loss function is used to control the distribution of each phase in general. In the entire optimization process, a strategy using Wasserstein distance with gradient penalty makes the convergence of the training process faster and the reconstruction result more stable; it also avoids the problems of gradient disappearance and mode collapse. Finally, the reconstructed 3D structure and the target 3D structure are visualized to ascertain their similar morphologies. The morphological parameter indicators of the reconstructed 3D structure were consistent with those of the target 3D structure. The microstructure parameters of the 3D structure were also compared and analyzed. The proposed method can achieve accurate and stable 3D reconstruction compared with classical stochastic methods of image reconstruction.

Analysis of Polarization Detector Performance Parameters on Polarization 3D Imaging Accuracy.

Three-dimensional (3D) reconstruction of objects using the polarization properties of diffuse light on the object surface has become a crucial technique. Due to the unique mapping relation between the degree of polarization of diffuse light and the zenith angle of the surface normal vector, polarization 3D reconstruction based on diffuse reflection theoretically has high accuracy. However, in practice, the accuracy of polarization 3D reconstruction is limited by the performance parameters of the polarization detector. Improper selection of performance parameters can result in large errors in the normal vector. In this paper, the mathematical models that relate the polarization 3D reconstruction errors to the detector performance parameters including polarizer extinction ratio, polarizer installation error, full well capacity and analog-to-digital (A2D) bit depth are established. At the same time, polarization detector parameters suitable for polarization 3D reconstruction are provided by the simulation. The performance parameters we recommend include an extinction ratio ≥ 200, an installation error ∈ [-1°, 1°], a full-well capacity ≥ 100 Ke-, and an A2D bit depth ≥ 12 bits. The models provided in this paper are of great significance for improving the accuracy of polarization 3D reconstruction.