3D Shape Reconstruction Research Articles

Artificial Skin Based on Visuo‐Tactile Sensing for 3D Shape Reconstruction: Material, Method, and Evaluation

AbstractArtificial skin has shown great potential in robot perception and human healthcare. It provides multifunctional tactile sensing, including 3D shape reconstruction, contact feedback, and temperature perception, where the 3D reconstruction function is indispensable for dexterous hands in tactile cognition and interaction. Vision‐based tactile sensor (VTS) is an innovative bionic tactile sensor and supports high‐resolution, high‐precision, and high‐density tactile reconstruction compared with electronic tactile sensors. Considering the unique contribution of visuo‐tactile sensing to artificial skin, this review focuses on the 3D reconstruction techniques of the VTS. 3D reconstruction methods are classified into five categories based on sensing modalities, hardware categories, and modeling approaches: 1) photometric stereo, 2) binocular depth calibration, 3) optical flow, 4) deep learning, and 5) ToF (time of flight). In addition, the association and difference of reconstruction methods are analyzed from the hardware perspective, and the development and technological details of 3D reconstruction are summarized. On this basis, the challenges and development direction are discussed. This review can be viewed as a technology guide to provide references for interested researchers. Furthermore, it is expected to promote the extensive application of the VTS in artificial skins.

FSH3D: 3D Representation via Fibonacci Spherical Harmonics

AbstractSpherical harmonics are a favorable technique for 3D representation, employing a frequency‐based approach through the spherical harmonic transform (SHT). Typically, SHT is performed using equiangular sampling grids. However, these grids are non‐uniform on spherical surfaces and exhibit local anisotropy, a common limitation in existing spherical harmonic decomposition methods. This paper proposes a 3D representation method using Fibonacci Spherical Harmonics (FSH3D). We introduce a spherical Fibonacci grid (SFG), which is more uniform than equiangular grids for SHT in the frequency domain. Our method employs analytical weights for SHT on SFG, effectively assigning sampling errors to spherical harmonic degrees higher than the recovered band‐limited function. This provides a novel solution for spherical harmonic transformation on non‐equiangular grids. The key advantages of our FSH3D method include: 1) With the same number of sampling points, SFG captures more features without bias compared to equiangular grids; 2) The root mean square error of 32‐degree spherical harmonic coefficients is reduced by approximately 34.6% for SFG compared to equiangular grids; and 3) FSH3D offers more stable frequency domain representations, especially for rotating functions. FSH3D enhances the stability of frequency domain representations under rotational transformations. Its application in 3D shape reconstruction and 3D shape classification results in more accurate and robust representations. Our code is publicly available at https://github.com/Miraclelzk/Fibonacci-Spherical-Harmonics.

Interpolation-Filtering Method for Image Improvement in Digital Holography

Digital holography is actively used for the characterization of objects and 3D-scenes, tracking changes in medium parameters, 3D shape reconstruction, detection of micro-object positions, etc. To obtain high-quality images of objects, it is often necessary to register a set of holograms or to select a noise suppression method for specific experimental conditions. In this paper, we propose a method to improve filtering in digital holography. The method requires a single hologram only. It utilizes interpolation upscaling of the reconstructed image size, filtering (e.g., median, BM3D, or NLM), and interpolation to the original image size. The method is validated on computer-generated and experimentally registered digital holograms. Interpolation methods coefficients and filter parameters were analyzed. The quality is improved in comparison with digital image filtering up to 1.4 times in speckle contrast on the registered holograms and up to 17% and 29% in SSIM and NSTD values on the computer-generated holograms. The proposed method is convenient in practice since its realization requires small changes of standard filters, improving the quality of the reconstructed image.

Deep diffusion learning of mutual-reflective structured light patterns for multi-body three-dimensional imaging

Simultaneous structured light imaging of multiple objects has become more demanding and widely in many scenarios involving robot operations in intelligent manufacturing. However, it is challenged by pattern aliasing caused by mutual reflection between high-reflective objects. To this end, we propose to learn clear fringe patterns from aliased mutual-reflective observations by diffusion models for achieving high-fidelity multi-body reconstruction in line with typical phase-shift algorithms. Regarding mutual reflection imaging as a formation of adding significant noise, we build a supervised generative learning framework based on diffusion models and then train a self-attention-based deep network with a U-Net-like skip-connected encoder-decoder architecture. We demonstrate the generalization capability of the trained model in fringe pattern recovery and its performance in phase and three-dimensional (3D) shape reconstruction. Both experimental results show that the proposed method has the expected feasibility and accuracy, heralding a promising solution for addressing the current challenge in various multi-body mutual-reflective 3D reconstruction tasks.

Dynamic three-dimensional reconstruction with phase shift coding division multiplexing

Fringe projection profilometry with temporal phase unwrapping (FPP-TPU) has been gotten more and more attention in reconstructing the object with complex surfaces. However, a general contradiction in the existing FPP-TPUs is that more auxiliary coded patterns may limit sampling rate in dynamic three-dimensional (3D) measurement. In this paper, inheriting the single-frame auxiliary pattern of the phase-shifting temporal phase unwrapping (PS-TPU), an improved phase shift coding temporal phase unwrapping with higher robustness is proposed. And new projection stratagem with phase shift coding division multiplexing (PSC-DM) is also proposed to enhance the reconstructing rate in dynamic 3D scene. Firstly, six codewords instead of the traditional eight codewords are embedded into the single-frame auxiliary coding pattern, which guarantees the decoding robustness and provides a faster decoding process with single-step direction unifying. Then, the new projection sequence stratagem is designed with every two sets of 3-step phase-shifting patterns projection followed by the single-frame phase-shifting coding pattern projection. During phase unwrapping, every adjacent three frames of phase-shifting pattern sequentially resolving a wrapped phase and a highly robust 3D reconstruction can be achieved by using the closest phase-shifting coding pattern so as to achieve phase-shifting coding division multiplexing. It can obtain higher reconstructing rate. In addition, a phase shift segmentation is used to correct mismatched errors of fringe order. The experimental results demonstrate that the proposed method can increase the reconstructing rate from the rate of 30 fps in traditional PS-TPU method to 68.57 fps at the digital light projector (DLP) projection rate of 120 fps, which shows its potential prospective in dynamic 3D shape reconstruction.

3D Shape Reconstruction of Ge Nanowires during Vapor-Liquid-Solid Growth under Modulating Electric Field.

Bottom-up growth offers precise control over the structure and geometry of semiconductor nanowires (NWs), enabling a wide range of possible shapes and seamless heterostructures for applications in nanophotonics and electronics. The most common vapor-liquid-solid (VLS) growth method features a complex interaction between the liquid metal catalyst droplet and the anisotropic structure of the crystalline NW, and the growth is mainly orchestrated by the triple-phase line (TPL). Despite the intrinsic mismatch between the droplet and the NW symmetries, its discussion has been largely avoided because of its complexity, which has led to the situation when multiple observed phenomena such as NW axial asymmetry or the oscillating truncation at the TPL still lack detailed explanation. The introduction of an electric field control of the droplet has opened even more questions, which cannot be answered without properly addressing three-dimensional (3D) structure and morphology of the NW and the droplet. This work describes the details of electric-field-controlled VLS growth of germanium (Ge) NWs using environmental transmission electron microscopy (ETEM). We perform TEM tomography of the droplet-NW system during an unperturbed growth, then track its evolution while modulating the bias potential. Using 3D finite element method (FEM) modeling and crystallographic considerations, we provide a detailed and consistent mechanism for VLS growth, which naturally explains the observed asymmetries and features of a growing NW based on its crystal structure. Our findings provide a solid framework for the fabrication of complex 3D semiconductor nanostructures with ultimate control over their morphology.

Multi-factor quality assessment of digital speckle pattern for speckle projection profilometry

Abstract The digital speckle pattern (DSP) is an essential component in the speckle projection profilometry (SPP) task, its quality directly affects the results of three-dimensional (3D) shape reconstruction. However, the SPP field lacks specialized numerical metrics for evaluating speckle quality. To address this issue, this study introduces a multi-factor metric (MFM) for comprehensive DSP assessment. Through comparing the metric, optimal parameter ranges for DSP design and the advisable matching subset size can be determined for SPP algorithm. A global indicator named valid feature distribution (VFD) based on scale-invariant feature transform (SIFT) and Delaunay triangulation, is defined to analyze the overall information distribution in DSPs. In addition, MFM incorporates a local metric called mean subset intensity gradient (MSIG), which aids in selecting the suitable radius for different DSPs to balance the accuracy and efficiency. The quality assessment targets the speckle scene images, allowing for the reverse adjustment of the most suitable DSP according to different scenes. The performance of DSPs can be evaluated based on the accuracy and completeness of 3D reconstruction results. By conducting simulation experiments on the 3ds Max platform, the recommended parameter range for DSP can be inferred, including speckle density ratio, speckle diameter, and random variation rate. Appropriate subset sizes for different scenes are also investigated. Furthermore, the MFM is verified on a real binocular speckle device, demonstrating that the measurement standard deviation of a complex workpiece can be reduced to 0.078 mm using the recommended DSP.

FUSION3D: Multimodal data fusion for 3D shape reconstruction: A soft-sensing approach

Traditional high-fidelity imaging techniques, such as X-ray computer tomography (CT), excel in capturing intricate shape details through high-resolution two-dimensional (2D) images. However, the extensive cost or impracticality of obtaining X-ray images poses a significant challenge in various applications (e.g., dense metal components in manufacturing). To overcome this limitation, we propose a soft sensing approach, named FUSION3D, aimed at reconstructing three-dimensional (3D) shapes from 2D sliced images and heterogenous process inputs using a novel model based on multi-modal process knowledge. The developed FUSION3D method is built upon a deep learning framework that utilizes continuous normalizing flow to model the complex shape distribution of 3D objects. We propose a principled probabilistic framework to regularize 3D point clouds by modeling them as a distribution of distributions. Specifically, we learn a two-level hierarchy of distributions where the first level is the distribution of shapes, and the second level is the distribution of points given a shape. Our generative regularization model learns each level of the distribution with a continuous normalizing flow. The invertibility of normalizing flows enables the computation of the likelihood during training and allows us to train our model in the variational inference framework. Our methodology is validated through a case study in additive manufacturing, demonstrating its effectiveness as a soft sensing approach for accurate 3D shape reconstruction without relying on high-fidelity imaging techniques like CT during model deployment.

Learning intrinsic shape representations via spectral mesh convolutions

We introduce spectral-based convolutional operators embedded within Generalized Graph Neural Networks (G-GNNs). These operators enable deep learning on graphs through a learnable, energy-driven evolution process. This approach empowers us to impose specific properties on the graph convolutional kernel directly derived from the corresponding variational formulations. Our model incorporates both parameterized and non-parameterized graph Laplacian-based energies within the generalized graph convolutional layer to address features like smoothness, sharpness, and compact support. By making appropriate choices within our G-GNN framework, we pave the way for designing novel paradigms for 3D shape representation, reconstruction, and processing, while also enabling effective feature embeddings for intrinsic neural fields.

Active shape reconstruction using a novel visuotactile palm sensor

Tactile sensing enables high-precision 3D shape perception when vision is limited. However, tactile-based shape reconstruction remains a challenging problem. In this paper, a novel visuotactile sensor, GelStereo Palm 2.0, is proposed to better capture 3D contact geometry. Leveraging the dense tactile point cloud captured by GelStereo Palm 2.0, an active shape reconstruction pipeline is presented to achieve accurate and efficient 3D shape reconstruction on irregular surfaces. GelStereo Palm 2.0 achieves a spatial resolution of 1.5 mm and a reconstruction accuracy of 0.3 mm. The accuracy of the proposed active shape reconstruction pipeline reaches 2.3 mm within 18 explorations. The proposed method has potential applications in the shape reconstruction of transparent or underwater objects.

Multi-granularity relationship reasoning network for high-fidelity 3D shape reconstruction

Monocular image-based 3D reconstruction is widely used in virtual reality, augmented reality, and autonomous driving, which benefits from the rapid development of deep learning approaches. Most of the available methods focused on reconstructing the overall shape of the object while ignoring some fine-grained details. Moreover, these methods make it hard to exactly reconstruct complex topological structures. In this paper, we propose a multi-granularity relationship reasoning network (MGRRNet), which aims to recover 3D shapes with high fidelity and rich details via the relationship reasoning between different granularity information. Specifically, our model captures the discriminative and detailed features at different granularities for extracting attentional regions. Then we perform the relationship reasoning between different granularities to reinforce the multi-granularity consistency and inter-granularity correlation. By doing this, our network is able to achieve robust feature representation and fine reconstruction. During the learning process, we jointly optimize procedures of different granularity feature representations via a sequence of inter-granularity cycle loss iterations. Extensive experimental results on two publicly available datasets justify that our approach achieves competitive performance compared to the state-of-the-art methods. Codes and all resources will be publicly available at https://github.com/Ray-tju/MGRRNet.

Single-Shot 3D Reconstruction via Nonlinear Fringe Transformation: Supervised and Unsupervised Learning Approaches.

The field of computer vision has been focusing on achieving accurate three-dimensional (3D) object representations from a single two-dimensional (2D) image through deep artificial neural networks. Recent advancements in 3D shape reconstruction techniques that combine structured light and deep learning show promise in acquiring high-quality geometric information about object surfaces. This paper introduces a new single-shot 3D shape reconstruction method that uses a nonlinear fringe transformation approach through both supervised and unsupervised learning networks. In this method, a deep learning network learns to convert a grayscale fringe input into multiple phase-shifted fringe outputs with different frequencies, which act as an intermediate result for the subsequent 3D reconstruction process using the structured-light fringe projection profilometry technique. Experiments have been conducted to validate the practicality and robustness of the proposed technique. The experimental results demonstrate that the unsupervised learning approach using a deep convolutional generative adversarial network (DCGAN) is superior to the supervised learning approach using UNet in image-to-image generation. The proposed technique's ability to accurately reconstruct 3D shapes of objects using only a single fringe image opens up vast opportunities for its application across diverse real-world scenarios.

3D colored object reconstruction from a single view image through diffusion

In this paper we propose a novel 3D colored object reconstruction method from a single view image. Given a reference image, a conditional diffusion probabilistic model is built to reconstruct both a 3D point cloud shape and the corresponding color features at each point, and then images from arbitrary views can be synthesized using a volume rendering technique. The approach involves several sequential steps. First, the reference RGB image is encoded into separate shape and color latent variables. Then, a shape prediction module predicts reverse geometric noise based on the shape latent variable within the diffusion model. Next, a color prediction module predicts color features for each 3D point using information from the color latent variable. Finally, a volume rendering module transforms the generated colored point cloud into 2D image space, facilitating training based solely on a reference image. Experimental results demonstrate that the proposed method achieves competitive performance on colored 3D shape reconstruction and novel view image synthesis.

Reconstruction of the real 3D shape of the SARS-CoV-2 virus

The photographs of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus taken by electron transmission microscopy and cryoelectron microscopy provide only a 2D silhouette. The viruses appear to look like distorted circles. The present paper questions the real shape of the SARS-CoV-2 virus and makes an attempt to give an answer. Is this a general ellipsoid, a spheroid with rotational symmetry, a sphere, or something else? The answer requires the application of tools from three different disciplines: structural mechanics, microbiology, and statistics. A total of 590 virus photographs taken from 22 recently published papers were examined. From this experimental data pool, the histogram of diameter ratios was built from the 283 measurements where the virus images could be approximated as ellipses. The curve peaks at the diameter ratio of 1.22. The transformation equation for the spatial shape to the planar shade was derived for a fixed light source of the microscope. This equation involves an unknown orientation of the viruses with respect to the microscope. Two sets of models were developed, one with a uniform distribution of the virus orientation and the other with the orientation defined by the normalized beta distribution. In both sets of models, the unknown diameter ratio of the spheroidal virus was regarded as a random realization from translated gamma distributions. The parameters of the distribution of the kernel functions were determined by minimizing the mean square difference between the predicted and measured 2D histograms. The information included in the measured histograms was found to be insufficient to find an unknown distribution of the virus’s orientation. Simply too many unknown parameters render the solution physically unrealistic. The minimization procedure with a uniform probability of virus orientation predicted the peak of the aspect ratio of the 3D spheroid at 1.32. Based on this result, models of the virus will be developed in the continuation of this research for a full dynamic analysis.

3D hand pose estimation and reconstruction based on multi-feature fusion

3D hand pose estimation and shape reconstruction is to recover the hand joint points and hand mesh vertices coordinates from the image. However, existing methods usually only use the high-level semantic features extracted by the backbone network to represent the hand mesh vertex features, which leads to a single representation of the hand vertices features and cannot fully utilize the feature information extracted by the network. In this paper, we propose a method for real-time 3D reconstruction of hands from a single RGB image, which enriches the 3D semantic information of the mesh vertices through multi-feature fusion. Firstly, we regress the 2D features of mesh vertices through Integral Pose Regression (IPR) and regard them as prior information to 3D features. Then we design a Multi-Scale Sampling(MSS) module to extract multi-scale information. Finally we fuse 2D prior features, multi-scale features, and high-level semantic features extracted by backbone to represent 3D initial feature. Additionally, we propose a Multi-Root(MR) loss function to address the imbalance problem caused by a single root joint. The experimental results indicate that our network achieves competitive performance on the FreiHAND and HO-3D public datasets, achieving fast inference speed with fewer parameters.

Monocular reconstruction of shapes of natural objects from orthographic and perspective images.

Human subjects were tested in perception of shapes of 3D objects. The subjects reconstructed 3D shapes by viewing orthographic and perspective images. Perception of natural shapes was very close to veridical and was clearly better than perception of random symmetrical polyhedra. Viewing perspective images led to only slightly better performance than viewing orthographic images. In order to account for subjects' performance, we elaborated the previous computational models of 3D shape reconstruction. The previous models used as constraints mirror-symmetry and 3D compactness. The critical additional constraint was the use of a secondary mirror-symmetry that exists in most natural shapes. It is known that two planes of mirror symmetry are sufficient for a unique and veridical shape reconstruction. We also generalized the model so that it applies to both orthographic and perspective images. The results of our experiment suggest that the human visual system uses two planes of symmetry in addition to two forms of 3D compactness. Performance of the new model was highly correlated with subjects' performance with both orthographic and perspective images, which supports the claim that the most important 3D shape constraints that are used by the human visual system have been identified.

High dynamic range 3D shape reconstruction: Combination of adaptive fringe intensity, iterative exposure time adjustment and least quadratic curve fitting

For three-dimensional (3D) reconstruction of specular surfaces based on fringe projection profilometry (FPP), a hybrid approach combining adaptive fringe projection, iterative exposure time adjustment and least quadratic curve fitting is proposed. First, we propose an adaptive projection fringe intensity generation method in the camera image. Second, we build a new pixel matching between the camera and projector, and correct the pixel matching on the specular surfaces based on linear interpolation. Third, we propose an iterative exposure time adjustment method based on saturated pixel statistics. Fourth, we fuse the multiple results to obtain the final 3D reconstructed surface. Finally, we adopt a least quadratic curve fitting algorithm to fit the residual phase errors. Experiments confirmed the validity of the proposed method.

Advancements in the 3D shape reconstruction of Phobos: An analysis of shape models and future exploration directions

Aims. Our research focuses on developing a high-precision and relatively high-resolution shape model of Phobos. Methods. We employed advanced photogrammetric techniques combined with novel computer vision methods to reconstruct the 3D shape of Phobos from nearly 900 Mars Express/SRC and Viking Orbiter images. This research also involved a comparison of the newly developed shape model with previous models to identify differences for future missions. Results. This shape model was used to generate new measurements of the volume (5740 ± 30) km3, the surface area (1629 ± 8) km2, and the bulk density (1847 ± 11) kg m−3 of Phobos. By comparing our reconstructed shape model with prior models, we have identified key differences, especially in areas such as the Opik crater and near the Shklovsky crater. These findings highlight critical areas that warrant further investigation in future missions dedicated to exploring Phobos.

Single-Image-Based 3D Reconstruction of Endoscopic Images.

A wireless capsule endoscope (WCE) is a medical device designed for the examination of the human gastrointestinal (GI) tract. Three-dimensional models based on WCE images can assist in diagnostics by effectively detecting pathology. These 3D models provide gastroenterologists with improved visualization, particularly in areas of specific interest. However, the constraints of WCE, such as lack of controllability, and requiring expensive equipment for operation, which is often unavailable, pose significant challenges when it comes to conducting comprehensive experiments aimed at evaluating the quality of 3D reconstruction from WCE images. In this paper, we employ a single-image-based 3D reconstruction method on an artificial colon captured with an endoscope that behaves like WCE. The shape from shading (SFS) algorithm can reconstruct the 3D shape using a single image. Therefore, it has been employed to reconstruct the 3D shapes of the colon images. The camera of the endoscope has also been subjected to comprehensive geometric and radiometric calibration. Experiments are conducted on well-defined primitive objects to assess the method's robustness and accuracy. This evaluation involves comparing the reconstructed 3D shapes of primitives with ground truth data, quantified through measurements of root-mean-square error and maximum error. Afterward, the same methodology is applied to recover the geometry of the colon. The results demonstrate that our approach is capable of reconstructing the geometry of the colon captured with a camera with an unknown imaging pipeline and significant noise in the images. The same procedure is applied on WCE images for the purpose of 3D reconstruction. Preliminary results are subsequently generated to illustrate the applicability of our method for reconstructing 3D models from WCE images.

Uncertainty elimination in 2D shape reconstruction of irregular particles in Interferometric particle imaging

Interferometric Particle Imaging (IPI) technique is presently utilized in measuring irregular particle whose shape and size are indispensable parameters in various scientific fields, while orientation and size uncertainty are tough problems in IPI measurement. Herein, an effective method is proposed to eliminate aforementioned uncertainties. Based on Square Gradient function and multi-plane phase retrieval algorithm, our method eliminates conjugate solution in single-plane shape reconstruction thereby making the orientation of the reconstructed particle shape unique, while position search of focused image has significantly reduced the size error of reconstructed particle shape. Dual-detector IPI system is constructed to measure actual irregular particles whose shapes are successfully recovered from detected defocused speckles with an average shape similarity of 0.960 in Jaccard Index, and theoretical size error greatly decreases from 44.7% to 0.8% under the same system configuration. The proposed method is potential in further work such as real-time particle measurement and 3D shape reconstruction.