Volumetric Reconstruction Research Articles

Image-guided programming deep brain stimulation improves clinical outcomes in patients with Parkinson’s disease

Deep brain stimulation (DBS) is an effective treatment for patients with Parkinson’s disease (PD). However, some patients may not respond optimally to clinical programming adjustments. Advances in DBS technology have led to more complex and time-consuming programming. Image-guided programming (IGP) could optimize and improve programming leading to better clinical outcomes in patients for whom DBS programming is not ideal due to sub-optimal response. We conducted a prospective single-center study including 31 PD patients with subthalamic nucleus (STN) DBS and suboptimal responses refractory to clinical programming. Programming settings were adjusted according to the volumetric reconstruction of the stimulation field using commercial postoperative imaging software. Clinical outcomes were assessed at baseline and at 3-month follow-up after IGP, using motor and quality of life (QoL) scales. Additionally, between these two assessment points, follow-up visits for fine-tuning amplitude intensity and medication were conducted at weeks 2, 4, 6, and 9. After IGP, twenty-six patients (83.9%) experienced motor and QoL improvements, with 25.8% feeling much better and 38.7% feeling moderately better according to the patient global impression scale. Five patients (16.1%) had no clinical or QoL changes after IGP. The MDS-UPDRS III motor scale showed a 21.9% improvement and the DBS-IS global score improved by 41.5%. IGP optimizes STN-DBS therapy for PD patients who are experiencing suboptimal clinical outcomes. These findings support using IGP as a standard tool in clinical practice, which could save programming time and improve patients’ QoL.

BIMBot for Autonomous Laser Scanning in Built Environments

Accurate and complete 3D point clouds are essential in creating as-built building information modeling (BIM) models, although there are challenges in automating the process for 3D point cloud creation in complex environments. In this paper, an autonomous scanning system named BIMBot is introduced, which integrates advanced light detection and ranging (LiDAR) technology with robotics to create 3D point clouds. Using our specially developed algorithmic pipeline for point cloud processing, iterative registration refinement, and next best view (NBV) calculation, this system facilitates an efficient, accurate, and fully autonomous scanning process. The BIMBot’s performance was validated using a case study in a campus laboratory, featuring complex structural and mechanical, electrical, and plumbing (MEP) elements. The experimental results showed that the autonomous scanning system produced 3D point cloud mappings in fewer scans than the manual method while maintaining comparable detail and accuracy, demonstrating its potential for wider application in complex built environments.

SA-Pmnet: Utilizing Close-Range Photogrammetry Combined with Image Enhancement and Self-Attention Mechanisms for 3D Reconstruction of Forests

Efficient and precise forest surveys are crucial for in-depth understanding of the present state of forest resources and conducting scientific forest management. Close-range photogrammetry (CRP) technology enables the convenient and fast collection of highly overlapping sequential images, facilitating the reconstruction of 3D models of forest scenes, which significantly improves the efficiency of forest surveys and holds great potential for forestry visualization management. However, in practical forestry applications, CRP technology still presents challenges, such as low image quality and low reconstruction rates when dealing with complex undergrowth vegetation or forest terrain scenes. In this study, we utilized an iPad Pro device equipped with high-resolution cameras to collect sequential images of four plots in Gaofeng Forest Farm in Guangxi and Genhe Nature Reserve in Inner Mongolia, China. First, we compared the image enhancement effects of two algorithms: histogram equalization (HE) and median–Gaussian filtering (MG). Then, we proposed a deep learning network model called SA-Pmnet based on self-attention mechanisms for 3D reconstruction of forest scenes. The performance of the SA-Pmnet model was compared with that of the traditional SfM+MVS algorithm and the Patchmatchnet network model. The results show that histogram equalization significantly increases the number of matched feature points in the images and improves the uneven distribution of lighting. The deep learning networks demonstrate better performance in complex environmental forest scenes. The SA-Pmnet network, which employs self-attention mechanisms, improves the 3D reconstruction rate in the four plots to 94%, 92%, 94%, and 96% by capturing more details and achieves higher extraction accuracy of diameter at breast height (DBH) with values of 91.8%, 94.1%, 94.7%, and 91.2% respectively. These findings demonstrate the potential of combining of the image enhancement algorithm with deep learning models based on self-attention mechanisms for 3D reconstruction of forests, providing effective support for forest resource surveys and visualization management.

CGAN-Based Forest Scene 3D Reconstruction from a Single Image

Forest scene 3D reconstruction serves as the fundamental basis for crucial applications such as forest resource inventory, forestry 3D visualization, and the perceptual capabilities of intelligent forestry robots in operational environments. However, traditional 3D reconstruction methods like LiDAR present challenges primarily because of their lack of portability. Additionally, they encounter complexities related to feature point extraction and matching within multi-view stereo vision sensors. In this research, we propose a new method that not only reconstructs the forest environment but also performs a more detailed tree reconstruction in the scene using conditional generative adversarial networks (CGANs) based on a single RGB image. Firstly, we introduced a depth estimation network based on a CGAN. This network aims to reconstruct forest scenes from images and has demonstrated remarkable performance in accurately reconstructing intricate outdoor environments. Subsequently, we designed a new tree silhouette depth map to represent the tree’s shape as derived from the tree prediction network. This network aims to accomplish a detailed 3D reconstruction of individual trees masked by instance segmentation. Our approach underwent validation using the Cityscapes and Make3D outdoor datasets and exhibited exceptional performance compared with state-of-the-art methods, such as GCNDepth. It achieved a relative error as low as 8% (with an absolute error of 1.76 cm) in estimating diameter at breast height (DBH). Remarkably, our method outperforms existing approaches for single-image reconstruction. It stands as a cost-effective and user-friendly alternative to conventional forest survey methods like LiDAR and SFM techniques. The significance of our method lies in its contribution to technical support, enabling the efficient and detailed utilization of 3D forest scene reconstruction for various applications.

Flow diverter stents for endovascular treatment of aneurysms: a comparative study of efficacy and safety between FREDX and FRED.

The FRED X flow diverter (FREDX), as the second generation in the FRED series, aims to improve the treatment of cerebral aneurysms. This study compares the efficacy and safety of FREDX with its predecessor, FRED. This prospective registry included patients treated with FRED and FREDX devices. Efficacy was assessed using digital subtraction angiography with 3D volumetric reconstruction at immediate and 1 year follow-ups. Safety was evaluated by recording complications, analyzed through univariate contrasts, generalized mixed models, and Bayesian network analyses. We treated 287 patients with 385 aneurysms, with 77.9% receiving FRED and 22.1% FREDX. The median age was 55 years (IQR 47-65) and 78.4% were women. The FREDX group showed a higher prevalence of saccular-like aneurysms (70.6% vs 52.7%, P=0.012) and a higher rate of complete occlusion compared with FRED interventions (79.4% vs 59.3%, P=0.022). After adjusting for confounders, these differences represented a 3.04-fold increased likelihood (95% CI 1.44 to 6.41, P=0.003) of achieving complete occlusion at 1 year with FREDX interventions. Regarding safety, two (3.5%) complications (both non-symptomatic) were observed in the FREDX group and 23 (10.4%) in the FRED group (P=0.166). Bayesian network analysis suggested a trend towards fewer complications for FREDX, with a median reduction of 5.5% in the posterior distribution of the prevalence of complications compared with FRED interventions. The FREDX device shows improved complete occlusion rates at 1 year compared with the FRED device while maintaining a favourable safety profile, indicating its potential advantage in the treatment of cerebral aneurysms.

Neural radiance fields for multi-scale constraint-free 3D reconstruction and rendering in orchard scenes

Efficiently, accurately, and realistically reconstructing large-scale 3D orchard scenes in a virtual world is an immensely challenging task. This complexity stems from the intricate and expansive of real orchard scenes. Traditional 3D reconstruction and rendering methods have encountered limitations in terms of modeling efficiency and computational costs, hindering the ability to provide users with immersive experiences. In response to these challenges, this study introduces a strategy for 3D scene reconstruction and rendering grounded in implicit neural representation: the NeRF-Ag model. Building upon the baseline NeRF, this model integrates a multi-resolution latent feature encoding technique, notably heightening training efficiency and elevating modeling precision. Furthermore, by means of environmental factor embedding, the model's robustness and practical applicability are further enhanced. The experimental outcomes illustrate that NeRF-Ag attains photo-realistic rendering outcomes across small, medium, and large scales. Moreover, it surpasses NeRF concerning the evaluation metrics of PSNR, SSIM, and LPIPS. Notably, the training speed of NeRF-Ag is roughly 39 times faster than NeRF. In 3D reconstruction tasks, NeRF-Ag showcases enhanced texture detail representation and higher modeling accuracy compared to the COLMAP-based 3D reconstruction method. Additionally, this study accomplishes free-viewpoint rendering of 3D scenes employing NeRF-Ag and provides evidence substantiating the connection between the quantity of training images and the precision of 3D rendering. The conclusions of this study will contribute to supporting and referencing the implementation of immersive visual interactive features within agricultural digital twin systems.

Image-based 3D reconstruction and permeability modelling of rock using enhanced interpretable deep residual learning

The study presents a novel deep residual learning framework with interpretability for the prediction of rock permeability. The proposed framework, termed ResNet-NiN integrates a 3D deep residual network (ResNet) with network in network (NiN) connections. A total of 1331 datasets were generated using voxel segmentation of DRP samples obtained from actual Berea Sandstone and subsequently, a percolation simulation was performed. The learning rate decay strategy of Cosine Annealing with warmup restarts is used to train the predictive model and prevent the predicted permeability from being trapped in the local optima. To improve the trust and comprehensibility of the model and visualize the ResNet-NiN architecture, we established a compound approach that leverages the internal gradient change of the neural network into the deep learning framework. This integrated approach encompasses the analysis of Integrated Gradients (IG) from the input perspective and Gradient-weighted Class Activation Mapping (Grad-CAM) from the output perspective. The experimental findings demonstrate that the developed model exhibits superiority over the typically employed 3D convolutional neural network architecture in terms of both accuracy and stability. The unified interpretability technique offers a coherent explanation for the provided sample, validating the model's capability and elucidating the factors and relationships underlying the model's prediction.

Comparison of TLS and SLAM technologies for 3D reconstruction of objects with different geometries

Technological advances have made the 3D mapping process easily available and simpler. However, there are still aspects that need to be improved and enhanced. The efficient acquisition of 3D data and reconstruction of objects with high accuracy continues to be a challenge for the scientific community. One of the most frequently used 3D mapping methods is Terrestrial Laser Scanning (TLS), which allows the collection of high-resolution and precise data. Another method gaining popularity among researchers and professionals is mobile scanning technology, which enables real-time data capture. Its mobility and speed make it an effective alternative to traditional scanning technologies. This article compares two mapping technologies: SLAM (Simultaneous Localization and Mapping) and TLS taking into account the technical aspects of the instruments, processing methods, time and cost, and concluding with an assessment of the final accuracy. The geometry of several selected objects was analyzed. The resulting root mean square error (RMSE) for the compared distances on the two point clouds was 5 cm, which proves that the SLAM technology can be successfully applied for scenarios requiring centimeter-level accuracy.

SEM-micro-photogrammetry and stereo reconstruction on the Tescan Amber X electron microscope

SEM-micro-photogrammetry and stereo reconstruction on the Tescan Amber X electron microscope

Three-dimensional Transesophageal Echocardiography in Infective Endocarditis: What Does It Add?

Infective endocarditis (IE) diagnosis is based on a clinical suspicion supported by consistent microbiological and instrumental data. Evidence of involvement of cardiac valves (native or prosthetic) or prosthetic intracardiac material is a major diagnostic criterion of IE. Transthoracic echocardiography (TTE) is the initial technique of choice for the diagnosis while transesophageal echocardiography (TEE) is recommended in patients with an inconclusive or negative TTE, in patients with high suspicion of IE, as well as in patients with a positive TTE, in order to document local complications. Repeating TTE and/or TEE should be considered during follow-up of uncomplicated IE, in order to detect new silent complications and monitor vegetation size. In the setting of IE, the role of three-dimensional (3D) TEE is increasing; in fact, this technique has also been shown to be useful for the diagnosis of IE and its complications as it allows to obtain infinite planes and volumetric reconstructions. In this review, we will describe the usefulness of 3D-TEE and its added value in the management of IE.

Exploration of New Digital Media Art Forms under the Combination of Virtual and Augmented Reality Technology

Abstract VR and AR not only expand the art field, but also enrich the form and content of digital media art and promote the development of digital media art. This paper proposes a 3D reconstruction of media art scenes based on the improved Neural Radiation Field (NeRF) technology and then completes the style migration, through which a unique scene art effect is formed, with a view to exploring new ways for the development of new digital media art forms. In the virtual orchard construction, the PSNR value of the optimized NeRF method is improved by 0.205, the SSIM value is improved by 0.015, and the LPIPS value is reduced by 0.018 compared with the state-of-the-art method, which can be concluded that the optimized NeRF method has a good image synthesis effect.

Everted U-Net for 3D scene reconstruction and segmentation

Abstract. The field of data science related to the processing of three-dimensional objects is becoming more and more relevant. After the successes in image processing, the apotheosis of which was the development of generative neural net-works, the intensification of efforts in the direction of three-dimensional data processing looks logical. Although there are now numerous systems for the reconstruction of three-dimensional objects and other processing, almost all of the existing solutions are aimed at working with a single object. The advances in image processing with neural networks have largely been made possible by huge datasets. There are also large datasets available for model training in this area. Freely avail-able datasets such as ShapeNet and ModelNet contain many thousands of different models, allowing for a high diversity of data. However, most of them provide single individual objects, which allows them to be used in tasks involving the processing of a single three-dimensional object, but when working with scenes containing many objects, there is often a problem of finding an appropriate dataset. This work is aimed at solving the problem of reconstruction and segmentation of three-dimensional scenes, as well as generation of datasets for the task of processing scenes from the real world.

Construction of an Interactive Platform for Piano Teaching in Virtual Reality Environment

Abstract Under the impact of the wave of new technology, the form of piano teaching is no longer limited to the classroom and paper textbooks, and modern information technology, such as the Internet and virtual reality, is constantly integrated into traditional piano teaching. This paper utilizes the Unity3D engine to create a virtual reality environment for piano teaching interaction, based on VR technology, and to construct an interactive platform for piano teaching. Firstly, based on binocular stereo vision for the three-dimensional reconstruction of the teaching interactive scene, based on the improvement of the LeNet5 network for the user’s head gesture tracking, the constructed virtual scene realizes the real-time movement with the movement of the human head. The test set images using CPU-i76700HQ and GTX960 graphics cards consume 78ms and 25ms per frame, respectively, which meets the real-time system requirements. In this paper, the improved LeNet5 algorithm for head pose estimation without face detection is applicable and has a high detection accuracy of 99.31%, which is better than the algorithms such as Adaboost+Hough, LBF+BP, CNN+Heatmaps, DCNN, and FSA-Net, and is able to satisfy the real-time detection requirements of the virtual scene. The virtual piano teaching interactive platform has an average rating of more than 4 in attractiveness, subjectivity, interactivity, and convenience, but the running speed is only 3.46, so it is necessary to strengthen the technical support and improve the software and hardware level to break through the development bottleneck. Traditional piano teaching is given new vitality and kinetic energy by the interactive platform constructed in this paper for piano teaching in a virtual reality environment.

Contribution of 3D reconstruction techniques to the animated restoration of historical scenes

Abstract Amid the escalating demand for heightened realism across diverse sectors such as film, television, gaming, tourism, virtual display, and the preservation of cultural heritage, there has been a notable advancement in reverse modeling technology that facilitates the extraction of three-dimensional models from image sequences. This study focuses on the classification of 3D reconstruction for the animation restoration of historical scenes. It delves into the technical pathways for the 3D reconstruction of such scenes, employing a clustering-based Scale Invariant Feature Transform (SIFT) feature-matching acceleration algorithm for the extraction and matching of image features. Subsequently, it integrates the Bundler method of camera self-calibration with the Patch-based Multi-view Stereo (PMVS) algorithm for dense reconstruction, culminating in the assembly and testing of the reconstructed historical scene. The acceleration time of the SIFT algorithm is higher than that of the standard algorithm when the number of images is 10. The standard algorithm’s acceleration time is the longest when there are more than 10 images. When the number of images is greater than 50, the acceleration ratio of sigma=60 is greater than that of sigma=120. This indicates that the clustering-based SIFT algorithm is suitable for accelerated matching of large-scale image sets. The acceleration effect can be improved by appropriately decreasing the value of sigma. The maximum difference between the five independent vectors in model 1 and model 2 is 3.18. After scaling model 2, the difference is narrowed to[ 0.01,0.03]. The small error of the model in scenes 3~9 and the clear graphic texture indicate that the 3D reconstruction model of the historical scene designed in this paper has high accuracy and provides a model reference for the animation restoration of the historical scene.

SA-MVSNet: Self-attention-based multi-view stereo network for 3D reconstruction of images with weak texture

Multi-view stereo (MVS) reconstruction is a key task of image-based 3D reconstruction, and deep learning-based methods can achieve better results than traditional algorithms. However, most of the current deep learning-based MVS methods use convolutional neural networks (CNNs) to extract image features, which cannot achieve the aggregation of long-distance context information and capture robust global information. In addition, in the process of fusing depth maps into point clouds, the confidence filters will filter out the depth values with low confidence in weak texture areas. These problems will lead to the low completeness of 3D reconstruction of weak texture and texture-less areas. To address the above problems, this paper proposes SA-MVSNet based on the PatchmatchNet with a self-attentive mechanism. First, we design a coarse-to-fine network framework to advance depth map estimation. In the feature extraction network, a module with a pyramid structure based on Swin Transformer Block is used to replace the original Feature Pyramid Network (FPN), and the self-correlation between weak texture areas is enhanced by applying a global self-attention mechanism. Then, we also propose a self-attention-based adaptive propagation module (SA-AP), which applies a self-attention calculation within depth value propagation window to obtain the relative weight values of current pixel and others, and then adaptively samples the depth values of neighbors on the same surface for propagation. Experiments show that SA-MVSNet has significantly improved the completeness of 3D reconstruction for the images with weak texture on DTU (provided by Danish Technical University), BlendedMVS, and Tanks and Temple datasets. Our code is available at https://github.com/miaowang525/SA-MVSNet.

Automatic Skull Shape Completion of Defective Skulls Using Transformers for Cranial Implant Design

Cranioplasty is a surgical method that restores the aesthetic and protecting function of a damaged skull by implanting material into the damaged area.Fast and accurate design of patient specific cranial implants is very much required in the process of cranioplasty.The time consumption for designing and manufacturing of patient specific cranial implant has become an obstruction for cranioplasty procedures. Hence, a fully automatic and fast design of cranial implant becomes very important. The cranial implant design processmainly comprises of two steps. The former step concentrates on the automatic skull shape completion of defective skulls to fill the gaps and the cracks created in the skull. While the second step computes the difference between defective input and the completed skull for generating the implant. Currently computer aided design is used for the skull shape completion task which is a time consuming process. The application of deep learning techniques may result to faster and accurate skull shape completionwhich can be used for the design of patient specific cranial implants. This work proposes a novel approach combining 3D U-Net with Transformers for the automatic skull shape completion task.We are using a vision transformer in the encoder section of the 3D U-Net architecture to consider the volumetric skull reconstruction as sequence- to-sequence prediction problem and to efficaciously grasp the global contextual information. The work also compares its performance with the famous variants of 3D U-Net deep network model namely, 3D U-Net and 3D U-Net with attention. From the values resulted for the dice coefficient metric it is clear that the proposed 3D U-Net with transformer approach performs better than the other two models on test images.

Overview of image-based 3D reconstruction technology

Three-dimensional (3D) reconstruction technology is the key technology to establish and express the objective world by using computer, and it is widely used in real 3D, automatic driving, aerospace, navigation and industrial robot applications. According to different principles, it is mainly divided into methods based on traditional multi-view geometry and methods based on deep learning. This paper introduces the above methods from the perspective of three-dimensional space representation. The feature extraction and stereo matching theory of traditional 3D reconstruction methods are the theoretical basis of 3D reconstruction methods based on deep learning, so the paper focuses on them. With the development of traditional 3D reconstruction methods and the development of deep learning related theories, the explicit deep learning 3D reconstruction method represented by MVSNet and the implicit 3D reconstruction method represented by NeRF have been gradually developed. At the same time, the dataset and evaluation indicators for 3D reconstruction were introduced. Finally, a summary of image based 3D reconstruction was provided.

3D Reconstruction with Spherical Cameras

The goal of image-based 3D reconstruction is to establish a high-quality 3D expression from images. In order to achieve a high-resolution and real-time 3D model, inspired by the open source COLMAP, we propose a novel framework (3DMAP) to reconstruct 3D scenes based on spherical cameras. Unlike traditional methods which focus on building a 3D plane by Poisson distribution function, our method illustrates the key processes of 3D reconstruction: locating the camera based on global feature, estimating the scene’s relative depth from monocular panoramic images, and obtaining a high-quality 3D surface reconstruction. In the camera locating part, we use a global descriptor augmentation model to build a labeled panorama dataset GDAP, in which the images are captured by our designed spherical cameras; In the depth estimation part, we propose a new network UMDE that can estimate the depth of both indoor and outdoor scenes; Finally, in the 3D surface reconstruction section, we turn the reconstruction problem to a graph optimization problem, called GraphFit, in which, we optimize the point clouds with s-t graph and smoothing method successively. We conduct experiments on our own dataset to demonstrate the effectiveness of our proposed 3DMAP framework. Experimental results show that our 3DMAP has achieved good evaluation scores and visual effects.

GP-Recon: Online Monocular Neural 3D Reconstruction with Geometric Prior.

High-fidelity online 3D scene reconstruction from monocular videos continues to be challenging, especially for coherent and fine-grained geometry reconstruction. The previous learning-based online 3D reconstruction approaches with neural implicit representations have shown a promising ability for coherent scene reconstruction, but often fail to consistently reconstruct fine-grained geometric details during online reconstruction. This paper presents a new on-the-fly monocular 3D reconstruction approach, named GP-Recon, to perform high-fidelity online neural 3D reconstruction with fine-grained geometric details. We incorporate geometric prior (GP) into a scene's neural geometry learning to better capture its geometric details and, more importantly, propose an online volume rendering optimization to reconstruct and maintain geometric details during the online reconstruction task. The extensive comparisons with state-of-the-art approaches show that our GP-Recon consistently generates more accurate and complete reconstruction results with much better fine-grained details, both quantitatively and qualitatively.

Decouple Ego-View Motions for Predicting Pedestrian Trajectory and Intention.

Pedestrian trajectory prediction is a critical component of autonomous driving in urban environments, allowing vehicles to anticipate pedestrian movements and facilitate safer interactions. While egocentric-view-based algorithms can reduce the sensing and computation burdens of 3D scene reconstruction, accurately predicting pedestrian trajectories and interpreting their intentions from this perspective requires a better understanding of the coupled vehicle (camera) and pedestrian motions, which has not been adequately addressed by existing models. In this paper, we present a novel egocentric pedestrian trajectory prediction approach that uses a two-tower structure and multi-modal inputs. One tower, the vehicle module, receives only the initial pedestrian position and ego-vehicle actions and speed, while the other, the pedestrian module, receives additional prior pedestrian trajectory and visual features. Our proposed action-aware loss function allows the two-tower model to decompose pedestrian trajectory predictions into two parts, caused by ego-vehicle movement and pedestrian movement, respectively, even when only trained on combined ego-view motions. This decomposition increases model flexibility and provides a better estimation of pedestrian actions and intentions, enhancing overall performance. Experiments on three publicly available benchmark datasets show that our proposed model outperforms all existing algorithms in ego-view pedestrian trajectory prediction accuracy.