3D Reconstruction Method Research Articles

3D Reconstruction and Semantic Modeling of Eyelashes

AbstractHigh‐fidelity digital human modeling has become crucial in various applications, including gaming, visual effects and virtual reality. Despite the significant impact of eyelashes on facial aesthetics, their reconstruction and modeling have been largely unexplored. In this paper, we introduce the first data‐driven generative model of eyelashes based on semantic features. This model is derived from real data by introducing a new 3D eyelash reconstruction method based on multi‐view images. The reconstructed data is made available which constitutes the first dataset of 3D eyelashes ever published. Through an innovative extraction process, we determine the features of any set of eyelashes, and present detailed descriptive statistics of human eyelashes shapes. The proposed eyelashes model, which exclusively relies on semantic parameters, effectively captures the appearance of a set of eyelashes. Results show that the proposed model enables interactive, intuitive and realistic eyelashes modeling for non‐experts, enriching avatar creation and synthetic data generation pipelines.

A color 3D reconstruction method for waste mechanical parts based on improved SGM

In order to improve the remanufacturing efficiency of scrap mechanical parts and comprehensively detect their surface fault status, this paper proposes a color three-dimensional reconstruction method of scrap mechanical parts based on an improved semi-global matching (SGM) algorithm. In experiments, this method demonstrated significant performance advantages in dealing with complex mechanical component structures and large illumination interference environments. Experimental results show that the three-dimensional color model reconstructed by this method has clear texture and small dimensional error, and is suitable for online analysis of surface fault information of scrap mechanical parts in actual production lines. Through quantitative analysis, compared with the traditional SGM method, the method in this paper improves the structural similarity index (SSIM) by an average of 19.8% and reduces the mean square error (MSE) by an average of 33.1%.

Efficient EndoNeRF reconstruction and its application for data-driven surgical simulation

PurposeThe healthcare industry has a growing need for realistic modeling and efficient simulation of surgical scenes. With effective models of deformable surgical scenes, clinicians are able to conduct surgical planning and surgery training on scenarios close to real-world cases. However, a significant challenge in achieving such a goal is the scarcity of high-quality soft tissue models with accurate shapes and textures. To address this gap, we present a data-driven framework that leverages emerging neural radiance field technology to enable high-quality surgical reconstruction and explore its application for surgical simulations.MethodWe first focus on developing a fast NeRF-based surgical scene 3D reconstruction approach that achieves state-of-the-art performance. This method can significantly outperform traditional 3D reconstruction methods, which have failed to capture large deformations and produce fine-grained shapes and textures. We then propose an automated creation pipeline of interactive surgical simulation environments through a closed mesh extraction algorithm.ResultsOur experiments have validated the superior performance and efficiency of our proposed approach in surgical scene 3D reconstruction. We further utilize our reconstructed soft tissues to conduct FEM and MPM simulations, showcasing the practical application of our method in data-driven surgical simulations.ConclusionWe have proposed a novel NeRF-based reconstruction framework with an emphasis on simulation purposes. Our reconstruction framework facilitates the efficient creation of high-quality surgical soft tissue 3D models. With multiple soft tissue simulations demonstrated, we show that our work has the potential to benefit downstream clinical tasks, such as surgical education.

3D hand pose estimation and reconstruction based on multi-feature fusion

3D hand pose estimation and shape reconstruction is to recover the hand joint points and hand mesh vertices coordinates from the image. However, existing methods usually only use the high-level semantic features extracted by the backbone network to represent the hand mesh vertex features, which leads to a single representation of the hand vertices features and cannot fully utilize the feature information extracted by the network. In this paper, we propose a method for real-time 3D reconstruction of hands from a single RGB image, which enriches the 3D semantic information of the mesh vertices through multi-feature fusion. Firstly, we regress the 2D features of mesh vertices through Integral Pose Regression (IPR) and regard them as prior information to 3D features. Then we design a Multi-Scale Sampling(MSS) module to extract multi-scale information. Finally we fuse 2D prior features, multi-scale features, and high-level semantic features extracted by backbone to represent 3D initial feature. Additionally, we propose a Multi-Root(MR) loss function to address the imbalance problem caused by a single root joint. The experimental results indicate that our network achieves competitive performance on the FreiHAND and HO-3D public datasets, achieving fast inference speed with fewer parameters.

Three-dimensional information measurement and reconstruction for micro-hole array based on the modified shape from focus

The geometrical characteristics of the micro-holes and the consistency of spatial distribution among the micro-holes in the micro-hole array fabricated by laser beam drilling (LBD) directly influence the cooling performance, reliability, and longevity of the workpiece. It is of great significance to establish the relationship between the LBD processing parameters and the characteristics of the micro-hole array through the three-dimensional (3D) information measurement and reconstruction of the micro-hole array, as well as to guide and improve the LBD processing technology. However, due to the small size and high aspect ratio of the micro-holes in the micro-hole array, the 3D information measurement and reconstruction of the micro-hole array face challenges. The conventional micro-hole measurement methods suffer from low efficiency, high measurement costs, and potentially destructive measurement processes. Furthermore, the conventional micro-hole measurement methods face challenges in simultaneously obtaining the geometrical characteristics of the micro-holes and the spatial distribution among the micro-holes in the micro-hole array. Therefore, a non-contact and non-destructive 3D information measurement and reconstruction method for the micro-hole array based on the modified shape from focus is proposed. A novel focus measure operator based on grayscale difference and adaptive neighborhood size is proposed to accurately evaluate the focus value and obtain the initial focus volume matrix. A novel image processing operator based on the gradient information of the image sequence is proposed to enhance the initial focus volume matrix. The experimental results indicate that the proposed focus measure operator, combined with the novel image processing operator, is more suitable and reliable for detecting the micro-holes due to its better unimodality, higher contrast, and narrower peak width than the conventional focus measure operator. The optimal focus position of each image point is obtained using the modified Gaussian curve fitting method. On this basis, the 3D information measurement and reconstruction of the micro-hole array are realized. By comparing the results with the conventional micro-hole measurement method, the feasibility and accuracy of the proposed method have been confirmed to be an effective measurement method for the micro-hole array.

PlaneAC: Line-guided planar 3D reconstruction based on self-attention and convolution hybrid model

Planar 3D reconstruction aims to simultaneously extract plane instances and reconstruct the local 3D model through the estimated plane parameters. Existing methods achieve promising results either through self-attention or convolution neural network (CNNs), but usually ignore the complementary properties of them. In this paper, we propose a line-guided planar 3D reconstruction method PlaneAC, which leverages the advantage of self-attention and CNNs to capture long-range dependencies and alleviate the computational burden. In addition, explicit connection between two adjacent attention layers is built for better leveraging the transferable knowledge and facilitating the information flow between tokenized feature from different layers. Therefore, the subsequent attention layer can directly interact with previous results. Finally, a line segment filtering method is presented to remove irrelevant guiding information from indistinctive line segments extracted from the image. Extensive experiments on ScanNet and NYUv2 public datasets demonstrate the preferable performance of our proposed method, and the results show that PlaneAC achieves a better trade-off between accuracy and computation cost compared with other state-of-the-art methods.

Use of Phase-Angle Model for Full-Field 3D Reconstruction under Efficient Local Calibration.

Currently, 3D reconstruction methods in structured light are generally implemented in a pre-calibrated area. To realize a full-field reconstruction, the calibration plate can be moved to multiple positions in a time-consuming manner, or the whole field can be calibrated with the help of a large calibration plate, which is more costly. In this paper, we address this problem by proposing a method for obtaining a global phase-angle model under a locally calibrated region, and based on this relationship, we investigate and analyze the reconstruction inside and outside of the calibrated zone. The results show that the method can reconstruct the object outside of the calibration zone completely, and can keep the planarity error around 0.1 mm and the sphericity error below 0.06 mm. The method only requires local calibration of the projected fringes at the two calibration positions to realize the 3D reconstruction of the full-field, which makes the method more advantageous.

Research on the Application of Deep Learning in Medical Image Segmentation and 3D Reconstruction

Medical image segmentation (MIS) and 3D reconstruction are crucial research directions in the field of medical imaging, which is of great significance for disease diagnosis, treatment planning and surgical navigation. In recent years, with the rapid development of Deep Learning (DL) technology, DL has made remarkable progress in the field of medical image processing and has become one of the important methods of MIS and 3D reconstruction. In this paper, the application of DL technologies in MIS and 3D reconstruction is systematically studied and discussed. Firstly, the paper introduces the basic concepts and technical challenges of MIS and 3D reconstruction, including image quality, noise interference and edge detection. Secondly, the paper introduces the data acquisition process in detail, including the medical image data set and data preprocessing method. Then, the paper puts forward the DL model framework based on self-attention mechanism, as well as the loss function and optimizer used in the training process. Then, the model is verified by experiments, and the performance of different models in MIS and 3D reconstruction is analyzed. Finally, the experimental results are comprehensively analyzed, and the application prospect and future development direction of DL in MIS and 3D reconstruction are discussed. The research results of this paper provide important theoretical and practical guidance for improving medical image processing technology and promoting the development and clinical application of medical imaging.

A Reflective Object Reconstruction Method Based on Neural Radiance Field

We propose a neural rendering-based 3D reconstruction method for reconstructing the geometry and BRDF of reflective objects from multi-view images captured in unknown environments. Multi-view reconstruction of reflective objects is extremely challenging because specular reflections are view-dependent and therefore violate multi-view consistency, which is the cornerstone of most multi-view reconstruction methods. Recent neural rendering techniques can model the interaction between ambient light and object surfaces to adapt to view-dependent reflections, making it possible to reconstruct reflective objects from multi-view images. However, accurately modeling ambient light in neural rendering is difficult. We propose a two-step approach to solve this problem. First, by introducing view-dependent photometric losses, our method accurately reconstructs the geometry of reflective objects. Then, with the object geometry fixed, we use more accurate sampling to recover the BRDF of the ambient light and object. Experiments show that our method is able to accurately reconstruct the geometry and BRDF of reflective objects from only RGB images without knowing the ambient light and object masks.

A Comprehensive Review of Vision-Based 3D Reconstruction Methods.

With the rapid development of 3D reconstruction, especially the emergence of algorithms such as NeRF and 3DGS, 3D reconstruction has become a popular research topic in recent years. 3D reconstruction technology provides crucial support for training extensive computer vision models and advancing the development of general artificial intelligence. With the development of deep learning and GPU technology, the demand for high-precision and high-efficiency 3D reconstruction information is increasing, especially in the fields of unmanned systems, human-computer interaction, virtual reality, and medicine. The rapid development of 3D reconstruction is becoming inevitable. This survey categorizes the various methods and technologies used in 3D reconstruction. It explores and classifies them based on three aspects: traditional static, dynamic, and machine learning. Furthermore, it compares and discusses these methods. At the end of the survey, which includes a detailed analysis of the trends and challenges in 3D reconstruction development, we aim to provide a comprehensive introduction for individuals who are currently engaged in or planning to conduct research on 3D reconstruction. Our goal is to help them gain a comprehensive understanding of the relevant knowledge related to 3D reconstruction.

High-Precision 3D reconstruction and quantitative structure description: Linking microstructure to macroscopic heat transfer of aerogels

The intricate interplay between microstructure and macroscopic transfer processes in porous materials has long been recognized, yet achieving a definitive link between them remains a formidable challenge. Herein, we propose an effective approach to address this challenge by introducing a high-precision 3D reconstruction and structure descriptor database analysis method of nanoporous aerogels, which enables deeper understanding and accurate predictions of structure-directed properties. By employing focused ion beam-scanning electron microscope (FIB-SEM) technique supplemented with image preprocessing and image segmentation, we successfully built a 3D reconstruction model that exhibits high consistency with experimental results. Then, a structure descriptor database from the view of original porous structure, structure connectivity, and topological information is established, and 10 descriptors are preliminarily filtered relevance with thermal conductivity. Consequently, the quantitative relationship between descriptor and thermal conductivity is evaluated via a back propagation (BP) neural network, and three categories structure descriptors which have the most significant impact are determined as (i) the description of gaseous heat transfer pathways, (ii) the description of three-dimensional connectivity, and (iii) the description of interface complexity. With the profound understanding of structure descriptor, the present work offers a novel, structure-focused strategy that closely link the microstructure to macroscopic properties of porous materials.

Three-dimensional reconstruction of polarized ambient light separation in complex illumination

In current research, it is still a hot topic for 3D reconstruction under complex illumination. This paper uses a polarization camera combined with a coding technique to propose a new 3D reconstruction method for polarized ambient light separation. Based on the polarization camera, a specific separation model is established to analyze the relationship between the polarization characteristics of polarized and natural light. Specular reflections were filtered first and then analyzed based on the stocks vector and muller matrix. A specific calculation process was used to calculate different polarization azimuths according to the polarization characteristics, and finally, the polarized light and ambient light were separated. The experimental results show that the use of this polarization camera approach reduces the number of steps required to rotate the polarizer multiple times. This not only reduces the shooting time but also improves the efficiency. Moreover, after separating the ambient light, polarization imaging suppresses the interference of the ambient light, which helps to highlight the complete point cloud image more clearly in the 3D reconstruction. The standard deviation of 3D reconstruction was improved to 0.1675 mm by using this method in indoor and outdoor experiments.

An Integrated Approach for Accurate 3D Point Cloud Recognition of Substation Equipment Using PSO and kNN Classification

Low efficiency and poor accuracy are caused by missing data in traditional 3D reconstruction methods. This study suggests a new 3D point cloud recognition technique for substation equipment based on 3D laser scanning point clouds, which combines the k-nearest neighbour (KNN) classification algorithm and particle swarm optimisation (PSO) algorithm, to address these issues. The particle swarm optimisation algorithm optimises the coefficient weights of each subspace feature. The k-nearest neighbour classification algorithm is then used to finish the classification. To confirm the superiority and accuracy of the suggested approach, the impact of the point cloud subspace’s size and loss rate on the recognition effect is examined experimentally and contrasted with the enhanced iterative nearest point algorithm. With an average recognition time of 0.19 seconds and a recognition accuracy of over 95\%, the experimental results demonstrate the method’s good performance in terms of efficiency and accuracy, opening up a wide range of potential applications.

Shape-aware speckle matching network for cross-domain 3D reconstruction

The learning-based binocular 3D reconstruction method outperforms traditional vision algorithms in terms of precision and efficiency. However, in some cross-domain scenes, the precision of the well-trained network drastically decreases when handling samples in diverse contexts. This paper proposes an end-to-end shape-aware speckle matching network (SSMNet) that combines shape-mask information to achieve improved precision and completeness of disparity calculation in cross-domain applications. The cascade attention mechanism is inserted in the feature extraction stage to concentrate on valuable regions. The shape-aware module is designed to learn additional shape contour information, and multiscale features are integrated simultaneously to construct the cost-volume for the subsequent lightweight 3D aggregation. In addition, instance normalization is adopted to guarantee style migration, and a hybrid loss function is used to supervise the learning process. Furthermore, a high-precision binocular speckle dataset is built, including training and testing sets in different distributions. Extensive quantitative and qualitative experiments demonstrate that SSMNet enhances cross-domain capability and achieves state-of-the-art performance. Measurement precision evaluation illustrates that the proposed method can realize the desired highly precise 3D shape measurement in a real industrial scenario.

Single-Image-Based 3D Reconstruction of Endoscopic Images.

A wireless capsule endoscope (WCE) is a medical device designed for the examination of the human gastrointestinal (GI) tract. Three-dimensional models based on WCE images can assist in diagnostics by effectively detecting pathology. These 3D models provide gastroenterologists with improved visualization, particularly in areas of specific interest. However, the constraints of WCE, such as lack of controllability, and requiring expensive equipment for operation, which is often unavailable, pose significant challenges when it comes to conducting comprehensive experiments aimed at evaluating the quality of 3D reconstruction from WCE images. In this paper, we employ a single-image-based 3D reconstruction method on an artificial colon captured with an endoscope that behaves like WCE. The shape from shading (SFS) algorithm can reconstruct the 3D shape using a single image. Therefore, it has been employed to reconstruct the 3D shapes of the colon images. The camera of the endoscope has also been subjected to comprehensive geometric and radiometric calibration. Experiments are conducted on well-defined primitive objects to assess the method's robustness and accuracy. This evaluation involves comparing the reconstructed 3D shapes of primitives with ground truth data, quantified through measurements of root-mean-square error and maximum error. Afterward, the same methodology is applied to recover the geometry of the colon. The results demonstrate that our approach is capable of reconstructing the geometry of the colon captured with a camera with an unknown imaging pipeline and significant noise in the images. The same procedure is applied on WCE images for the purpose of 3D reconstruction. Preliminary results are subsequently generated to illustrate the applicability of our method for reconstructing 3D models from WCE images.

Digital 3D Reconstruction of Ancient Chinese Great Wild Goose Pagoda by TLS Point Cloud Hierarchical Registration

Digital reconstruction of ancient buildings is very challenging due to its architectural complexity and structural delicacy. Therefore, how to apply new Earth Observation (EO) technology to digitally reconstruct the complete and real model of ancient buildings without damage has become an urgent issue. This article proposes a multi-scan point cloud hierarchical registration method for 3D reconstruction of the ancient Chinese Great Wild Goose Pagoda based on Terrestrial Laser Scanner (TLS) data. First, a multi-scan point cloud hierarchical registration method is elaborately designed for 3D reconstruction. Then, the outline feature of the ancient Chinese Great Wild Goose Pagoda is automatically extracted by point cloud slicing. Finally, fine modeling is carried out according to the outline feature extracted from the reconstructed point cloud, and after that block combination and texture mapping are conducted for 3D digitalization and documentation. The experiments indicate that the Root Mean Square Error (RMSE) of multi-scan hierarchical registration of the translation and rotation are less than 3 cm and 0.862 ∘ , respectively, and the complete and real model of the Great Wild Goose Pagoda is obtained by fine modeling and texture mapping, which provides an important reference for 3D digital protection of ancient buildings.

Establishment and preliminary application of personalized three-dimensional reconstruction of thyroid gland with automatic detection of thyroid nodules based on ultrasound videos.

A well display of the spatial location of thyroid nodules in the thyroid is important for surgical path planning and surgeon-patient communication. The aim of this study was to establish a three-dimensional (3D) reconstruction method of the thyroid gland, thyroid nodule, and carotid artery with automatic detection based on two-dimensional (2D) ultrasound videos, and to evaluate its clinical value. Ultrasound videos, including the thyroid gland with nodule, isthmus of thyroid gland, and ipsilateral carotid artery, were recorded. BC-UNet, MTN-Net, and RDPA-U-Net network models were innovatively employed for segmentation of the thyroid glands, the thyroid nodules, and the carotid artery respectively. Marching Cubes algorithm was used for reconstruction, while Laplacian smoothing algorithm was employed to smooth the 3D model surface. Using this model, 20 patients and 15 surgeons completed surveys on the effectiveness of this model for the pre-surgery demonstration of nodule location as well as surgeon-patient communication. The thyroid gland with nodule, isthmus of gland, and carotid artery were reconstructed and displayed. With the 3D model, the understanding of the spatial location of thyroid nodules improved in all three surgeon groups, eliminating the influence of professional levels. In the patient survey, the patients' understanding of the thyroid nodule location and procedure for surgery were significantly improved. In addition, with the 3D model, the time for doctors to explain to patients was significantly reduced (16.75 vs. 8.85min, p=0.001). To our knowledge, this is the first report of constructing a 3D thyroid model using a deep learning technique for personalized thyroid segmentation based on 2D ultrasound videos. The preliminary clinical application showed that it was conducive to the comprehension of the location of thyroid nodules for surgeons and patients, with significant improvement on the efficiency of surgeon-patient communication.

Self-Supervised 3D Human Mesh Recovery from a Single Image with Uncertainty-Aware Learning

Despite achieving impressive improvement in accuracy, most existing monocular 3D human mesh reconstruction methods require large-scale 2D/3D ground-truths for supervision, which limits their applications on unlabeled in-the-wild data that is ubiquitous. To alleviate the reliance on 2D/3D ground-truths, we present a self-supervised 3D human pose and shape reconstruction framework that relies only on self-consistency between intermediate representations of images and projected 2D predictions. Specifically, we extract 2D joints and depth maps from monocular images as proxy inputs, which provides complementary clues to infer accurate 3D human meshes. Furthermore, to reduce the impacts from noisy and ambiguous inputs while better concentrate on the high-quality information, we design an uncertainty-aware module to automatically learn the reliability of the inputs at body-joint level based on the consistency between 2D joints and depth map. Experiments on benchmark datasets show that our approach outperforms other state-of-the-art methods at similar supervision levels.

Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views

Reconstructing 3D objects from extremely sparse views is a long-standing and challenging problem. While recent techniques employ image diffusion models for generating plausible images at novel viewpoints or for distilling pre-trained diffusion priors into 3D representations using score distillation sampling (SDS), these methods often struggle to simultaneously achieve high-quality, consistent, and detailed results for both novel-view synthesis (NVS) and geometry. In this work, we present Sparse3D, a novel 3D reconstruction method tailored for sparse view inputs. Our approach distills robust priors from a multiview-consistent diffusion model to refine a neural radiance field. Specifically, we employ a controller that harnesses epipolar features from input views, guiding a pre-trained diffusion model, such as Stable Diffusion, to produce novel-view images that maintain 3D consistency with the input. By tapping into 2D priors from powerful image diffusion models, our integrated model consistently delivers high-quality results, even when faced with open-world objects. To address the blurriness introduced by conventional SDS, we introduce the category-score distillation sampling (C-SDS) to enhance detail. We conduct experiments on CO3DV2 which is a multi-view dataset of real-world objects. Both quantitative and qualitative evaluations demonstrate that our approach outperforms previous state-of-the-art works on the metrics regarding NVS and geometry reconstruction.

Vibration response characteristics of Jujube trees based on finite element method and structure-from-motion

It is an indispensable step to choose optimal vibration parameters in jujube fruits vibration harvesting. Excessive vibration frequency will cause jujube tree damage, otherwise, the separation of fruit and fruit stalk cannot be achieved, resulting in harvesting failure. In our work, finite element method and vibration tests are applied to find optimal vibration parameters. To improve the accuracy of finite element method, a low-cost 3D reconstruction method is proposed to build jujube tree model. Experimental result showed that the phenotypic information of the reconstructed jujube tree and the real jujube tree have a high similarity, and their determination coefficient R2 reach to 0.984. Moreover, the natural frequencies of jujube tree are analyzed by modal analysis, transient analysis is applied to analyze vibration transfer characteristics. The simulation and experimental results show that the optimal vibration frequency range of jujube tree is 25∼30 Hz, the optimal vibration excitation location is on the main trunk and as high as possible close to the secondary branches, which can obtain maximum vibration response for optimum harvesting efficiency. The average determination coefficient is 0.66 between test and simulation results at different frequencies. This study can provide references for accurate 3D modeling of jujube trees, selection of appropriate vibration parameter range and the development of jujube harvesting machinery.