3D Image Reconstruction Research Articles

MP-NeRF: More refined deblurred neural radiance field for 3D reconstruction of blurred images

Neural Radiance Fields (NeRF) has gained prominence in the domain of 3D reconstruction. Despite its popularity, NeRF algorithm typically require clear, static images to function effectively, leading to reduced performance when dealing with real-world scenarios that present non-ideal conditions such as complex reflections, low dynamic range, dark scenes and blurriness resulting from camera motion or defocus photography. The resilience of NeRF to such blurred inputs has not been sufficiently examined, presenting a gap in current research, which also tends to neglect the role of 3D scene context in image deblurring. Addressing these challenges, we introduce Multi-branch Fusion Network and Prior-based Learnable Weights NeRF (MP-NeRF), a novel approach engineered for the precise reconstruction of scenes from blurred images. MP-NeRF innovatively applies distinct priors for various blur types, thus better interpreting the blur formation mechanism. It integrates a Multi-branch Fusion Network (MBFNet) with a Prior-based Learnable Weights (PLW), which collectively enhance the capture of complex scene details, including textural and pattern information. Our experimental findings validate that MP-NeRF considerably enhances the visual accuracy of reconstructions achieved through NeRF, yielding substantial improvements in Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) compared to state-of-the-art models. The source code for MP-NeRF is made available at the following URL: https://github.com/luckhui0505/MP-NeRF.

Optimal image 2D/3D by Krawtchouk moments and ABC algorithm

In our research, we provide an enhanced and effective Krawtchouk moments parameter p optimization approach using the Artificial Bee Colony (ABC) algorithm optimization technique for the purpose of completing tasks including the reconstruction and classification of 3D and 2D images with high quality. The proposed method, which uses the ABC algorithm to generate a suitable parameter based on Krawtchouck moments, aims to accurately characterize the image as a vector known as the descriptor vector of moments, which is then used to reconstruct the image for each order. The simulation and analysis of the findings demonstrate the high level of quality and effectiveness of the suggested KM-ABC approach for image reconstruction, as well as its excellent accuracy in images with and without noise.

Reconstructing 3D De-Blurred Structures from Limited Angles of View through Turbid Media Using Deep Learning

Recent studies in transillumination imaging for developing an optical computed tomography device for small animal and human body parts have used deep learning networks to suppress the scattering effect, estimate depth information of light-absorbing structures, and reconstruct three-dimensional images of de-blurred structures. However, they still have limitations, such as knowing the information of the structure in advance, only processing simple structures, limited effectiveness for structures with a depth of about 15 mm, and the need to use separated deep learning networks for de-blurring and estimating information. Furthermore, the current technique cannot handle multiple structures distributed at different depths next to each other in the same image. To overcome the mentioned limitations in transillumination imaging, this study proposed a pixel-by-pixel scanning technique in combination with deep learning networks (Attention Res-UNet for scattering suppression and DenseNet-169 for depth estimation) to estimate the existence of each pixel and the relative structural depth information. The efficacy of the proposed method was evaluated through experiments that involved a complex model within a tissue-equivalent phantom and a mouse, achieving a reconstruction error of 2.18% compared to the dimensions of the ground truth when using the fully convolutional network. Furthermore, we could use the depth matrix obtained from the convolutional neural network (DenseNet-169) to reconstruct the absorbing structures using a binary thresholding method, which produced a reconstruction error of 6.82%. Therefore, only one convolutional neural network (DenseNet-169) must be used for depth estimation and explicit image reconstruction. Therefore, it reduces time and computational resources. With depth information at each pixel, reconstruction of 3D image of the de-blurred structures could be performed even from a single blurred image. These results confirm the feasibility and robustness of the proposed pixel-by-pixel scanning technique to restore the internal structure of the body, including intricate networks such as blood vessels or abnormal tissues.

Orthoscopic elemental image synthesis for 3D light field display using lens design software and real-world captured neural radiance field.

The elemental images (EIs) generation of complex real-world scenes can be challenging for conventional integral imaging (InIm) capture techniques since the pseudoscopic effect, characterized by a depth inversion of the reconstructed 3D scene, occurs in this process. To address this problem, we present in this paper a new approach using a custom neural radiance field (NeRF) model to form real and/or virtual 3D image reconstruction from a complex real-world scene while avoiding distortion and depth inversion. One of the advantages of using a NeRF is that the 3D information of a complex scene (including transparency and reflection) is not stored by meshes or voxel grid but by a neural network that can be queried to extract desired data. The Nerfstudio API was used to generate a custom NeRF-related model while avoiding the need for a bulky acquisition system. A general workflow that includes the use of ray-tracing-based lens design software is proposed to facilitate the different processing steps involved in managing NeRF data. Through this workflow, we introduced a new mapping method for extracting desired data from the custom-trained NeRF-related model, enabling the generation of undistorted orthoscopic EIs. An experimental 3D reconstruction was conducted using an InIm-based 3D light field display (LFD) prototype to validate the effectiveness of the proposed method. A qualitative comparison with the actual real-world scene showed that the 3D reconstructed scene is accurately rendered. The proposed work can be used to manage and render undistorted orthoscopic 3D images from custom-trained NeRF-related models for various InIm applications.

Enhancing 3D human pose estimation with NIR single-pixel imaging and time-of-flight technology: a deep learning approach.

The extraction of 3D human pose and body shape details from a single monocular image is a significant challenge in computer vision. Traditional methods use RGB images, but these are constrained by varying lighting and occlusions. However, cutting-edge developments in imaging technologies have introduced new techniques such as single-pixel imaging (SPI) that can surmount these hurdles. In the near-infrared spectrum, SPI demonstrates impressive capabilities in capturing a 3D human pose. This wavelength can penetrate clothing and is less influenced by lighting variations than visible light, thus providing a reliable means to accurately capture body shape and pose data, even in difficult settings. In this work, we explore the use of an SPI camera operating in the NIR with time-of-flight (TOF) at bands 850-1550nm as a solution to detect humans in nighttime environments. The proposed system uses the vision transformers (ViT) model to detect and extract the characteristic features of humans for integration over a 3D body model SMPL-X through 3D body shape regression using deep learning. To evaluate the efficacy of NIR-SPI 3D image reconstruction, we constructed a laboratory scenario that simulates nighttime conditions, enabling us to test the feasibility of employing NIR-SPI as a vision sensor in outdoor environments. By assessing the results obtained from this setup, we aim to demonstrate the potential of NIR-SPI as an effective tool to detect humans in nighttime scenarios and capture their accurate 3D body pose and shape.

Three-dimentional reconstruction of underwater side-scan sonar images based on shape-from-shading and monocular depth fusion

Modern marine research requires high-precision three-dimensional (3D) underwater data. Underwater environments experience severe visible light attenuation, which leads to inferior imaging compared with air. In contrast, sound waves are less affected underwater; hence side-scan sonar is used for underwater 3D reconstruction. Typically, the shape-from-shading algorithm (SfS) is widely used to reconstruct surface normal or heights from side-scan sonar images. However, this approach has challenges because of global information loss and noise. To address these issues, this study introduces a surface-normal fusion method. Specifically, we propose a frequency separation SfS algorithm using a discrete cosine transform, which provides a surface-normal map with less noise. We then fuse the surface-normal map with a novel depth estimation network to achieve high-precision 3D reconstruction of underwater side-scan sonar images. We conducted experiments on synthetic, NYU-depth-v2, and real side-scan sonar datasets to demonstrate the effectiveness of the proposed method.

The features of dome-shape Macula and related complications in high myopia on optical coherence tomography.

This study used optical coherence tomography scanning and 3D reconstruction of the macular region in high myopia to examine more thoroughly and carefully the differences between high myopia-related macular complications with and without dome-shape macula (DSM) and to determine whether the DSM's fine structure has an effect on them. Retrospective analysis of the medical records of 345 eyes with high myopia who underwent an optical coherence tomography (OCT) examination. They were divided into the DSM group (69 eyes) and the group without DSM (276 eyes). Macular complications between the two groups were compared. The height of the DSM and the diameter of the dome base were measured. And then the association between DSM type, protrusion height and macular problems were analyzed. Epiretinal membrane (ERM) and extrafoveal schisis occurred more frequently in the DSM group, but the was no statistically significant difference in the frequency of foveal schisis between the two groups. The majority of eyes in the DSM categorization had a horizontal oval-shaped domain. In the DSM group, there was no evident difference in the percentage of eyes with macular complications in the groups below 150 um and above 150 um. OCT examination-based fine macular structure analysis reveals that DSM affects various macular problems in distinct ways. DSM could increase the risk of extrafoveal schisis and ERM while decreasing the risk of foveal schisis. The height of the DSM had no obvious impact on the prevalence of macular complications.

3D imaging reconstruction and laparoscopic robotic surgery: A winning combination for a complex case of multiple myomectomy

3D imaging reconstruction and laparoscopic robotic surgery: A winning combination for a complex case of multiple myomectomy

Large viewing angle integral imaging 3D display system based on a symmetrical compound lens array.

We propose a large viewing angle integral imaging 3D display system based on a symmetrical compound lens array (SCLA). The display system comprises a high-resolution 2D display panel, an SCLA, and a light shaping diffuser. The high-resolution 2D display panel presents an elemental image array, the SCLA modulates the light rays emitted from the 2D display panel to form 3D images in space, and the light shaping diffuser eliminates the gaps between 3D pixels of the 3D images. We find that the lateral aberration is a crucial factor that affects the resolution of the reconstructed 3D image. The symmetrical structure of the SCLA enables a reduced focal length and the elimination of lateral aberration, improving the viewing angle and the 3D image resolution simultaneously. The experimental results confirm that the proposed display system increases the viewing angle to 68.6°, achieving a comparable resolution of the full field of view while maintaining a simple structure.

P037 Modelling gut complexity and disease: development of a next-generation adult stem cell based tubular gut-on-a-chip model

Abstract Background Developing effective therapies for Inflammatory Bowel Disease (IBD) necessitates advanced in vitro models that faithfully replicate the intricacies of the gut. The current challenge is to create models that encompass the 3D morphology, heterogeneity, and boundary characteristics of intestinal tissue while also being amenable to high-throughput screening. Microfluidic techniques are emerging as vital tools to introduce physiologically relevant cues into conventional cell cultures. Methods We present an innovative in vitro model employing human adult stem cell-derived colon organoids cultivated as perfused epithelial tubules within the MIMETAS OrganoPlate®. This microfluidic platform facilitates the simultaneous culture of up to 64 independent gut tubules on an extracellular matrix, offering continuous media perfusion without artificial membrane interference. The barrier function of the epithelial tubules is assessed in real-time through transepithelial electrical resistance (TEER) measurements. Immunostaining and 3D reconstruction of confocal images evaluate the expression of relevant epithelial markers. Results The tubular epithelial model of the intestinal tract demonstrated rapid cell polarization, expression of cell-specific markers, tight junction formation, and the presence of functional transporters. TEER measurements indicated functional and reproducible barrier integrity across tubules generated from different organoid donors for at least 10 days. Incorporation of additional cell types, such as patient-derived immune cells and endothelium, marks a groundbreaking advancement in understanding disease pathogenesis. The plug-and-play modularity of these models allows for flexibility in incorporating specific cell types based on the biological pathway of interest. Conclusion Our next-generation gut-on-a-chip model faithfully mimics the IBD phenotype and facilitates screening for potential drug targets. Combining adult human stem cell-derived intestinal organoids with microfluidic technology provides a robust platform to study physiology and disease mechanisms in patient-specific gut models. The inclusion of IBD patient-derived cells promises further refinement, enabling a more profound exploration of disease phenotypes and treatment responsiveness. Moreover, organoids, as one of the most physiologically relevant cell sources, authentically represent patient biology, enhancing the translational potential of our findings.

Airway Management with Tracheal Stents -Ventrain® System: Report of Two Cases

In our case experience with the use of VENTRAIN system and the reconstruction of 3D images and videos, they would constitute an alternative in the management of oxygenation, ventilation and isolation of the air route when performing this procedure

Single depth image 3D face reconstruction via domain adaptive learning

Single depth image 3D face reconstruction via domain adaptive learning

Unveiling the Inner Structure of Micrometric Hollow Polymeric Fibers Using Synchrotron X-Ray Nanotomography.

In this study, a novel application of synchrotron X-ray nanotomography based on high-resolution full-field transmission X-ray microscopy for characterizing the structure and morphology of micrometric hollow polymeric fibers is presented. By employing postimage analysis using an open-source software such as Tomviz and ImageJ, various key parameters in fiber morphology, including diameter, wall thickness, wall thickness distribution, pore size, porosity, and surface roughness, were assessed. Electrospun polycaprolactone fibers with micrometric diameters and submicrometric features with induced porosity via gas dissolution foaming were used to this aim. The acquired synchrotron X-ray nanotomography data were analyzed using two approaches: 3D tomographic reconstruction and 2D radiographic projection-based analysis. The results of the combination of both approaches demonstrate unique capabilities of this technique, not achievable by other available techniques, allowing for a full characterization of the internal and external morphology and structure of the fibers as well as to obtain valuable qualitative insights into the overall fiber structure.

Design of on-chip coded high-resolution 2D imaging via 3D compressed sensing

This manuscript introduces a computational imaging approach where a static coded aperture is integrated with the image sensor to replace the extra Fourier optical elements or dynamic modulation in the previous computational high/super-resolution tasks. A two-dimensional (2D) high-resolution image with N×N pixels is formulated as a three-dimensional (3D) low-resolution cube with Nk×Nk×k2 voxels in the forward model where N is the spatial resolution and k is a factor, respectively. Thus, the 2D image reconstruction can be performed with a 3D compressed sensing model in the snapshot-compressive-imaging format which has been mathematically proved to be convergent. Our proposed method successfully performs on the scaling factors such as 4× enlargement and the PSNR gains for natural images, remote sensing images and infrared image are 1.8 dB, 3.5 dB and 1.3 dB, respectively with only 1000 paired training images.

3D reconstruction of light microscopic images and its significance for better clinical decisions: A systematic review emphasizing gastric histopathology

3D reconstruction of light microscopic images and its significance for better clinical decisions: A systematic review emphasizing gastric histopathology

Экспериментальная оценка рекомендаций метода по сопровождению оператора во время съемки фотографий для последующей 3D-реконструкции

The availability of 3D reconstruction technologies expands the possibilities of their application for analyzing the state of complex technical objects. However, solutions for 3D reconstruction still heavily depend on the quality of the input data and require significant time investments to build high-quality 3D models. The goal of this study is to expedite the process of constructing detailed and defect-free 3D models by creating an automated method to assist the operator during the capture of RGB images for subsequent 3D reconstruction. The method is used for preliminary analysis of the collected data and allows for the identification of problematic frames in advance, as well as determining the areas of the object to be captured that will be reconstructed with defects or low level of detail. The study describes algorithms for the two final stages of the method - global analysis and recommendations, based on the real-time construction of sparse point clouds and 3D models, as well as their properties assessment in conjunction with the operator's trajectory, reconstructed from accelerometer and gyroscope data. The effectiveness of the final stages of the method, based on the proposed algorithms, was evaluated through experiments. It was demonstrated that the use of global analysis and recommendation stages not only allows for predicting defective areas and low-detail zones in dense 3D models on mobile devices, but also accelerates the process of obtaining a high-quality 3D model by at least five times through automated analysis of the collected data.

Image fusion for 3D reconstruction of SEM images

Image fusion for 3D reconstruction of SEM images

Image Reconstruction for Accelerated MR Scan With Faster Fourier Convolutional Neural Networks.

High quality image reconstruction from undersampled k -space data is key to accelerating MR scanning. Current deep learning methods are limited by the small receptive fields in reconstruction networks, which restrict the exploitation of long-range information, and impede the mitigation of full-image artifacts, particularly in 3D reconstruction tasks. Additionally, the substantial computational demands of 3D reconstruction considerably hinder advancements in related fields. To tackle these challenges, we propose the following: 1) A novel convolution operator named Faster Fourier Convolution (FasterFC), aims at providing an adaptable broad receptive field for spatial domain reconstruction networks with fast computational speed. 2) A split-slice strategy that substantially reduces the computational load of 3D reconstruction, enabling high-resolution, multi-coil, 3D MR image reconstruction while fully utilizing inter-layer and intra-layer information. 3) A single-to-group algorithm that efficiently utilizes scan-specific and data-driven priors to enhance k -space interpolation effects. 4) A multi-stage, multi-coil, 3D fast MRI method, called the faster Fourier convolution based single-to-group network (FAS-Net), comprising a single-to-group k -space interpolation algorithm and a FasterFC-based image domain reconstruction module, significantly minimizes the computational demands of 3D reconstruction through split-slice strategy. Experimental evaluations conducted on the NYU fastMRI and Stanford MRI Data datasets reveal that the FasterFC significantly enhances the quality of both 2D and 3D reconstruction results. Moreover, FAS-Net, characterized as a method that can achieve high-resolution (320, 320, 256), multi-coil, (8 coils), 3D fast MRI, exhibits superior reconstruction performance compared to other state-of-the-art 2D and 3D methods.

PLMP - Point-Line Minimal Problems in Complete Multi-View Visibility.

We present a complete classification of all minimal problems for generic arrangements of points and lines completely observed by calibrated perspective cameras. We show that there are only 30 minimal problems in total, no problems exist for more than 6 cameras, for more than 5 points, and for more than 6 lines. We present a sequence of tests for detecting minimality starting with counting degrees of freedom and ending with full symbolic and numeric verification of representative examples. For all minimal problems discovered, we present their algebraic degrees, i.e.the number of solutions, which measure their intrinsic difficulty. It shows how exactly the difficulty of problems grows with the number of views. Importantly, several new minimal problems have small degrees that might be practical in image matching and 3D reconstruction.

Transformer-Based High-Fidelity Facial Displacement Completion for Detailed 3D Face Reconstruction

In this paper, we tackle a special face completion task, facial displacement completion, which can offer a key component for many single image 3D face reconstruction systems. To produce a detailed 3D face with ear-to-ear complete displacement UV map, we propose a novel Displacement Completion method based on Transformer (DCT). Current transformer based image inpainting methods usually follow a two-stage scheme, which firstly recovers the masked pixels in low resolution with transformer, and then replenishes the inpainting result in high resolution with GAN. Although these methods have achieved great success, they suffer from information loss from two aspects when applied in face completion: 1) The downsampling operation makes transformer only produce a coarse appearance prior for GAN, incurring middle and low level information loss. 2) Some meaningful facial semantics can be well captured by transformer and further benefit the completion, but it's has not yet been explored. Motivated by the above considerations, we come up with three key designs in the proposed DCT: PCA tokenization, BERT-style learning, and style modulation. Firstly, we use PCA tokenization to replace the downsampling in transformer to preserve more meaningful structures. Secondly, we make transformer simulate the two tasks in BERT, Masked Language Model (MLM) and Next Sentence Prediction (NSP), for both masked pixels and facial attributes recovery. Thirdly, we encode the outcome of transformer as the latent code to guide an image translation network in the StyleGAN2 modulation way. Experments on both FaceScape dataset and in-the-wild data demonstrate DCT's better performance compared with other transformer based or GAN based completion methods.