Shape Reconstruction Research Articles

Shape Sensing of Flexible Robots Based on Deep Learning

In this article, a deep learning method for the shape sensing of continuum robots based on multicore fiber bragg grating (FBG) fiber is introduced. The proposed method, based on an artificial neural network (ANN), differs from traditional approaches, where accurate shape reconstruction requires a tedious characterization of many characteristic parameters. A further limitation of traditional approaches is that they require either multiple fibers, whose location relative to the centerline must be precisely known (calibrated), or a single multicore fiber whose position typically coincides with the neutral line. The proposed method addresses this limitation and, thus, allows shape sensing based on a single multicore fiber placed off-center. This helps in miniaturizing and leaves the central channel available for other purposes. The proposed approach was compared to a recent state-of-the-art model-based shape sensing approach. A two-degree-of-freedom benchtop fluidics-driven catheter system was built to validate the proposed ANN. The proposed ANN-based shape sensing approach was evaluated on a 40-mm-long steerable continuum robot in both 3-D free-space and 2-D constrained environments, yielding an average shape sensing error of 0.24 and 0.49 mm, respectively. With these results, the superiority of the proposed approach compared to the recent model-based shape sensing method was demonstrated.

Spatial Shape Sensing of a Multisection Continuum Robot With Integrated DTG Sensor for Maxillary Sinus Surgery

Owing to the curved and narrow pathway of the maxillary sinus, reaching blind regions using rigid instruments in the traditional method of treatment is difficult. In this article, a novel multisection continuum robot (CR) integrated with essential instruments and sensors is developed for the maxillary sinus surgery. Moreover, shape sensing is important for the control of the CR, but this is still a challenge due to the confined operating space. To track the spatial shape of the developed CR, a draw tower grating (DTG) sensor is integrated. A shape sensing algorithm is proposed to reconstruct the shape of the CR based on the relation between the DTG sensor and the CR centerline. Simulations and experiments on a maxillary sinus phantom are conducted to test the ability of the CR to reach the blind regions. Different groups of experiments are performed to evaluate the performance of the shape reconstructions, including DTG sensor shape tests and CR shape tests in planar and spatial space. Experiments show a mean planar error of 0.23 mm (0.51%) and a mean spatial error of 0.61 mm (1.36%). The results demonstrate the effectiveness of the multisection CR shape sensing method suggested in this study.

Efficient structural model updating with spatially sparse modal data: A Bayesian perspective

Structural model updating was born from the need to adjust advanced finite element models to match with the observations of the real asset. An important branch of the discipline is dedicated to the development of Bayesian approaches to solving the updating problem. In the literature, Gibbs samplers were developed to assimilate modal data. This imposes both a forward model linearity in the parameters and a spatial reconstruction of the complete mode shapes from the partial observations. In this paper, a new forward model is constructed, based on a novel two-stage condensation technique. The latter allows one to recover the unobserved degrees of freedom from the observed ones while remaining linear in the parameters. To solve some ill-conditioning issues, a perturbed state of the problem is used that gives more flexibility in the inference. The perturbed two-stage condensation provides a more robust identifiability even in presence of spatially sparse observations. An illustrative numerical example from hydroelectric industry is proposed. Sensitivity analyses are performed to explore performances and limitations.

FOSS-Based Method for Thin-Walled Structure Deformation Perception and Shape Reconstruction

To improve the accuracy of deformation perception and shape reconstruction of flexible thin-walled structures, this paper proposes a method based on the combination of FOSS (fiber optic sensor system) and machine learning. In this method, the sample collection of strain measurement and deformation change at each measuring point of the flexible thin-walled structure was completed by ANSYS finite element analysis. The outliers were removed by the OCSVM (one-class support vector machine) model, and the unique mapping relationship between the strain value and the deformation variables (three directions of x-, y-, and z-axis) at each point was completed by a neural-network model. The test results show that the maximum error of the measuring point in the direction of the three coordinate axes: the x-axis is 2.01%, the y-axis is 29.49%, and the z-axis is 15.52%. The error of the coordinates in the y and z directions was large, and the deformation variables were small, the reconstructed shape had good consistency with the deformation state of the specimen under the existing test environment. This method provides a new idea with high accuracy for real-time monitoring and shape reconstruction of flexible thin-walled structures such as wings, helicopter blades, and solar panels.

HyM3D: A hybrid method for the automatic 3D reconstruction of a defective cranial vault

Background and objectiveThe ability to accomplish a consistent restoration of a missing or deformed anatomical area is a fundamental step for defining a custom implant, especially in the maxillofacial and cranial reconstruction where the aesthetical aspect is crucial for a successful surgical outcome. At the same time, this task is also the most difficult, time-consuming, and complicated across the whole reconstruction process. This is mostly due to the high geometric complexity of the anatomical structures, insufficient references, and significant interindividual anatomical heterogeneity. Numerous solutions, specifically for the neurocranium, have been put forward in the scientific literature to address the reconstruction issue, but none of them has yet been persuasive enough to guarantee an easily automatable approach with a consistent shape reconstruction. MethodsThis work aims to present a novel reconstruction method (named HyM3D) for the automatic restoration of the exocranial surface by ensuring both the symmetry of the resulting skull and the continuity between the reconstructive patch and the surrounding bone. To achieve this goal, the strengths of the Template-based methods are exploited to provide knowledge of the missing or deformed region and to guide a subsequent Surface Interpolation-based algorithm. HyM3D is an improved version of a methodology presented by the authors in a previous publication for the restoration of unilateral defects. Differently from the first version, the novel procedure applies to all kinds of cranial defects, whether they are unilateral or not. ResultsThe presented method has been tested on several test cases, both synthetic and real, and the results show that it is reliable and trustworthy, providing a consistent outcome with no user intervention even when dealing with complex defects. ConclusionsHyM3D method proved to be a valid alternative to the existing approaches for the digital reconstruction of a defective cranial vault; furthermore, with respect to the current alternatives, it demands less user interaction since the method is landmarks-independent and does not require any patch adaptation.

3D shape reconstruction of Japanese traditional puppet head from CT images by graph cut and machine learning methods

In this study, we discuss the digital archiving of Japanese traditional puppets. We propose two methods for extracting the puppet head shape from computed tomography (CT) images. The first is the graph cut method, and the second is a machine learning method based on U-Net. According to the experimental results of the extraction of puppet heads from CT images, the U-Net-based method can extract puppet heads more accurately than the graph cut method. Moreover, the U-Net-based method can extract puppet heads with multiple materials. However, the extraction of metal parts is inaccurate because of metal artefacts in the X-ray CT images and insufficient learning data.

3D shape reconstruction of the femur from planar X-ray images using statistical shape and appearance models

Major trauma is a condition that can result in severe bone damage. Customised orthopaedic reconstruction allows for limb salvage surgery and helps to restore joint alignment. For the best possible outcome three dimensional (3D) medical imaging is necessary, but its availability and access, especially in developing countries, can be challenging. In this study, 3D bone shapes of the femur reconstructed from planar radiographs representing bone defects were evaluated for use in orthopaedic surgery. Statistical shape and appearance models generated from 40 cadaveric X-ray computed tomography (CT) images were used to reconstruct 3D bone shapes. The reconstruction simulated bone defects of between 0% and 50% of the whole bone, and the prediction accuracy using anterior–posterior (AP) and anterior–posterior/medial–lateral (AP/ML) X-rays were compared. As error metrics for the comparison, measures evaluating the distance between contour lines of the projections as well as a measure comparing similarities in image intensities were used. The results were evaluated using the root-mean-square distance for surface error as well as differences in commonly used anatomical measures, including bow, femoral neck, diaphyseal–condylar and version angles between reconstructed surfaces from the shape model and the intact shape reconstructed from the CT image. The reconstructions had average surface errors between 1.59 and 3.59 mm with reconstructions using the contour error metric from the AP/ML directions being the most accurate. Predictions of bow and femoral neck angles were well below the clinical threshold accuracy of 3°, diaphyseal–condylar angles were around the threshold of 3° and only version angle predictions of between 5.3° and 9.3° were above the clinical threshold, but below the range reported in clinical practice using computer navigation (i.e., 17° internal to 15° external rotation). This study shows that the reconstructions from partly available planar images using statistical shape and appearance models had an accuracy which would support their potential use in orthopaedic reconstruction.

A data-driven sensor placement strategy for reconstruction of mode shapes by using recurrent Gaussian process regression

Current Optimal Sensor Placement (OSP) strategies for bridges mostly rely on data from a finite element model rather than from the real structure due to high cost in placing massive sensors for data collection. For large-scale bridges, however, it is difficult to formulate a precise model and thus the OSP strategies building upon a finite element model inevitably suffer from modelling errors. Besides, the finite element model cannot account for real measurement noise. Premised on the fact that it is not expensive to make in-situ trial measurements with a few sensors on a target bridge before deploying a structural health monitoring (SHM) system on it, a data-driven OSP strategy is proposed in this study which aims at accurately reconstructing mode shapes (to facilitate vibration-based structural damage detection) by using only a few vibration sensors to be included in the SHM system. The proposed OSP strategy is also applicable for the upgrade of a long-term SHM system currently deployed on a bridge, by using historical data collected from the current SHM system. To precisely reconstruct mode shapes, a two-stage OSP strategy in terms of Recurrent Gaussian Process Regression (RGPR) is developed, and its performance is validated on a simulation model and a real bridge. In the first stage, the greedy algorithm is leveraged to temporarily deploy sensors on the structure and train accurate RGPR models using the collected data, which are used to afford spatially complete mode shape data for optimization later. Starting from a few sensors temporarily deployed on the bridge, a one-by-one sensor adding procedure is performed to configure increasing sensors until the target is achieved. In the second stage, Cuckoo Search (CS) algorithm is pursued to obtain the globally optimal sensor placement solution, from which the temporarily deployed sensors can be re-configured to the optimum positions. Both the best sensor quantity and positions are obtained by the proposed OSP strategy.

Simultaneous Measurement of Bending and Torsion in Optical Fiber Shape Sensor

Optical fiber shape sensing technology provides an excellent solution for the tracking and guiding of minimally invasive surgical instruments. However, when some large-deformed instrument operation tasks are involved, restricted by the operating environment, a significant torsion will be produced by the curved structure of the instrument, posing a threat to the accuracy of its shape reconstruction notably. In this paper we propose and experimentally illustrate a new distributed curvature and torsion measurement method based on optical frequency domain reflectometer using helical multi-core fibers. A theoretical model of total strain induced by the bending and torsion is established. Two diverse means of inverse algorithms to decouple bending and torsion are explored. The experimental results of two methods demonstrated that the torsion measurement errors are around 6.6% and 7.3%.

Implicit Solutions of the Electrical Impedance Tomography Inverse Problem in the Continuous Domain with Deep Neural Networks.

Electrical impedance tomography (EIT) is a non-invasive imaging modality used for estimating the conductivity of an object Ω from boundary electrode measurements. In recent years, researchers achieved substantial progress in analytical and numerical methods for the EIT inverse problem. Despite the success, numerical instability is still a major hurdle due to many factors, including the discretization error of the problem. Furthermore, most algorithms with good performance are relatively time consuming and do not allow real-time applications. In our approach, the goal is to separate the unknown conductivity into two regions, namely the region of homogeneous background conductivity and the region of non-homogeneous conductivity. Therefore, we pose and solve the problem of shape reconstruction using machine learning. We propose a novel and simple jet intriguing neural network architecture capable of solving the EIT inverse problem. It addresses previous difficulties, including instability, and is easily adaptable to other ill-posed coefficient inverse problems. That is, the proposed model estimates the probability for a point of whether the conductivity belongs to the background region or to the non-homogeneous region on the continuous space Rd∩Ω with d∈{2,3}. The proposed model does not make assumptions about the forward model and allows for solving the inverse problem in real time. The proposed machine learning approach for shape reconstruction is also used to improve gradient-based methods for estimating the unknown conductivity. In this paper, we propose a piece-wise constant reconstruction method that is novel in the inverse problem setting but inspired by recent approaches from the 3D vision community. We also extend this method into a novel constrained reconstruction method. We present extensive numerical experiments to show the performance of the architecture and compare the proposed method with previous analytic algorithms, mainly the monotonicity-based shape reconstruction algorithm and iteratively regularized Gauss-Newton method.

Reconstructing dynamic human shapes from sparse silhouettes via latent space optimization of Parametric shape models

The problem of dynamic 3D reconstruction has gained popularity over the last few years with most approaches relying on data driven learning and optimization methods. However this is quite a challenging task because of the need for tracking different features in both space and time?that too of deformable objects-where such robust tracking may not always be possible. A common way to better ground the problem is by using some forms of regularizations primarily on the shape representations. Over the years, mesh-based linear blend skinning models have been the standard for fitting templates of humans to the observed time series data of human deformation. However, this approach suffers from optimization difficulties arising from maintaining a consistent mesh topology. In this paper, a novel algorithm for reconstructing dynamic human shapes has been proposed, which uses only sparse silhouette information. This is achieved by first creating shape models based on the signed distance neural fields which are subsequently optimized via volumetric differentiable rendering to best match the observed data. Several experiments have been carried out in this work to test the robustness of this method and the results show it to be quite robust, outperforming prior state of the art on dynamic human shape reconstruction by 45%.

Laser ultrasonic imaging of subsurface defects with the linear sampling method.

Laser ultrasonics is a remote nondestructive evaluation technique suitable for real-time monitoring of fabrication processes in semiconductor metrology, advanced manufacturing, and other applications where non-contact, high fidelity measurements are required. Here we investigate laser ultrasonic data processing approaches to reconstruct images of subsurface side drilled holes in aluminum alloy specimens. We demonstrate through simulation that the model-based linear sampling method (LSM) can perform accurate shape reconstruction of single and multiple holes and produce images with well-defined boundaries. We experimentally confirm that LSM produces images that represent the internal geometric features of an object, some of which may be missed by conventional imaging.

Analysis of Strain Transfer in FBG-Based Embedded Optical Fiber 3-D Shape Sensor

The FBG-based embedded optical fiber 3-D shape sensor (FBG-EOFSS) composed of several different materials finds potential applications in various industrial and medical applications. The strain loss during the strain transfer affects the change of the fiber Bragg grating (FBG) sensor’s wavelength, which in turn deteriorates the position accuracy. In this article, first, we analyzed the strain transfer in FBG-EOFSS and derived the optical–mechanical transformation equation. Then, a strain transfer model combining mechanics, materials, and optics is developed based on the finite-element analysis (FEA) method. The FEA simulation results show that the strain transfer rate decreases by 27.7%/mm with the increase of the adhesive thickness under FBG. The optimal average strain transfer rate is about 0.97 when the elastic modulus of the adhesive is in the range of 109–1014 Pa. Moreover, the effect of the strain transfer on the position accuracy is analyzed. The relative position error of 3-D shape reconstruction increases by 8.4% with the strain transfer rate decreasing by 10%. Finally, an experimental system composed of the clamp, the displacement stage, and the FBG demodulator is developed. The strain sensitivity coefficients of bare FBGs and FBG sensors were tested quantitatively. The corresponding average strain transfer rate was calculated by the strain sensitivity coefficients. The results can potentially conduce to the design of 3-D shape sensors and improve the positioning accuracy.

Capture My Head: A Convenient and Accessible Approach Combining 3D Shape Reconstruction and Size Measurement from 2D Images for Headwear Design

3D physical head capture has many applications for headwear and facewear design. However, the existing methods, usually using commercial scanners, are not cheap or convenient, thereby limiting their popularity and accessibility. Therefore, we propose a convenient and cheap approach to 3D head capture by combining shape reconstruction and size measurement from images, using only two printed templates and a mobile phone camera. For the shape reconstruction process, we presented a 3D morphable model (3DMM)-based method to reconstruct the 3D shape from an arbitrary number of face images. Using this method, two powerful 3DMMs of head shapes and textures are constructed, and a novel facial landmark-to-edge loss based on corrected landmarks is developed. For size measurement, we proposed a coarse-to-fine circular object diameter measurement method, used to measure a circular label or object (e.g., a coin) placed on the forehead. The experimental results confirm a cost-effective number and size of input face images for our method. Furthermore, the method comparisons demonstrate that our head reconstruction methods have superior accuracy to state-of-the-art methods. Our convenient approach can be applied to the ergonomic design field, as well as individual customizations, virtual try-ons, and size selection for head-related products.

Shadow-Robust Silhouette Reconstruction for Small-Body Applications

Small bodies’ exploration highlighted the need to develop new algorithms for deep space probes navigation. The estimation of the small-body shape is a crucial step for relative navigation, mission planning, and gravity investigation. This paper develops a shape from silhouette algorithm that takes as input a series of images to construct a polyhedral shape for small bodies of the solar system. First, the silhouette associated with the visible part of the body is extracted and modified to account for shadowing effects. Second, the shape is computed by intersecting the viewing cones, i.e., the cone whose apex is the camera center and whose vertices are the silhouette points. The algorithm output is a polyhedral shape whose reconstruction is robust to shadowing. The algorithm is validated by reconstructing the shape of Itokawa and Bennu from synthetic images simulated with Airbus Defence & Space’s rendering engine SurRender™. Moreover, the algorithm is tested on real images from Hayabusa’s approach. Results prove the capability to provide an efficient shape reconstruction for small-body applications.

Static and Dynamic Multiparameter Assessment of Structural Elements Using Chirped Fiber Bragg Gratings.

This paper presents the development, analysis, and application of chirped fiber Bragg gratings (CFBGs) for dynamic and static measurements of beams of different materials in the single-cantilever configuration. In this case, the beams were numerically analyzed using the finite-element method (FEM) for the assessment of the natural frequencies and vibration modes of the beam for the dynamic analysis of the structural element. Furthermore, the static numerical analysis was performed using a load at the free end of the beam, where the maximum strain and its distribution along the beam were analyzed, especially in the region at which the FBG was positioned. The experimental evaluation of the proposed CFBG sensor was performed in static conditions for forces from 0 to 50 N (in 10 N steps) applied at the free end of the beam, whereas the dynamic evaluation was performed by means of positioning an unbalanced motor at the end of the beam, which was excited at 16 Hz, 65 Hz, 100 Hz, and 131 Hz. The results showed the feasibility of the proposed device for the simultaneous assessment of the force and strain distribution along the CFBG region using the wavelength shift and the full-width at half-maximum (FWHM), respectively. In these cases, the determination coefficients of the spectral features as a function of the force and strain distribution were higher than 0.99 in all analyzed cases, where a potential resolution of 0.25 N was obtained on the force assessment. In the dynamic tests, the frequency spectrum of the sensor responses indicated a frequency peak at the excited frequency in all analyzed cases. Therefore, the proposed sensor device is a suitable option to extend the performance of sensors for structural health assessment, since it is possible to simultaneously measure different parameters in dynamic and static conditions using only one sensor device, which, due to its multiplexing capabilities, can be integrated with additional optical fiber sensors for the complete shape reconstruction with millimeter-range spatial resolution.

Clustering by fast detection of main density peaks within a peak digraph

The well-known Density Peak Clustering algorithm (DPC) proposed a heuristic center detection idea, i.e., to find density peaks as cluster centers. Nevertheless, such a center detection idea cannot work well on multi-peak clusters of complex shapes. Besides, DPC needs the distances between data, making it prohibitively time-consuming. To overcome these problems, a Main Density Peak Clustering algorithm (MDPC+)—clustering by fast detection of main density peaks within a peak digraph—is proposed, where a main density peak is the highest density peak in a cluster. MDPC+ can easily detect the real centers of multi-peak clusters based on its new center assumption. In MDPC+, the clustering problem is viewed as a graph cut problem and a specific graph structure is designed for non-peak and density peak allocation, respectively, so it can reasonably reconstruct clusters of complex shapes. Meanwhile, a satellite peak attenuation technique is embedded into MDPC+ to give it a high resistance to the interference of satellite peaks (i.e., non-center density peaks). Besides, MDPC+ only needs kNN distances of data as its input, so it is suitable for large datasets. Experimental results on both synthetic and real-world datasets demonstrate the superiority of MDPC+ in center detection, complex shape reconstruction, and running speed.

Probabilistic instance shape reconstruction with sparse LiDAR for monocular 3D object detection

Monocular 3D object detection aims to localize objects in 3D space from a single image. This is a difficult problem due to the lack of accurate depth measurements. Many methods predict depths upfront using a pre-trained vision-based depth estimator to assist in 3D object detection. However, the methods show limited improved performances because of the depth inaccuracy and the neglect of depth confidence. In this paper, we propose a new end-to-end 3D object detection framework by combining a monocular camera with a cheap 4-beam LiDAR. The minimal LiDAR signal is leveraged as an additional input to predict high-quality and dense depth maps from monocular images. Meanwhile, several 3D proposals are generated by a keypoint-based detector. The key challenge is encoding the depth confidence to capture the depth estimation uncertainty. Therefore, we propose probabilistic instance shape reconstruction to exploit instance shape information for box refinement. Our method employs a fully differentiable end-to-end framework, making it simple and efficient. The experimental results on the KITTI dataset demonstrate that the proposed method achieves state-of-the-art performance, thus validating the effectiveness of the sparse LiDAR data and the probabilistic instance shape reconstruction. Code is available at https://github.com/xjtuwh/SparseLiDAR_fusion.

Deep learning estimation of three-dimensional left atrial shape from two-chamber and four-chamber cardiac long axis views.

Left atrial volume is commonly estimated using the bi-plane area-length method from two-chamber (2CH) and four-chamber (4CH) long axes views. However, this can be inaccurate due to a violation of geometric assumptions. We aimed to develop a deep learning neural network to infer 3D left atrial shape, volume and surface area from 2CH and 4CH views. A 3D UNet was trained and tested using 2CH and 4CH segmentations generated from 3D coronary computed tomography angiography (CCTA) segmentations (n = 1700, with 1400/100/200 cases for training/validating/testing). An independent test dataset from another institution was also evaluated, using cardiac magnetic resonance (CMR) 2CH and 4CH segmentations as input and 3D CCTA segmentations as the ground truth (n = 20). For the 200 test cases generated from CCTA, the network achieved a mean Dice score value of 93.7%, showing excellent 3D shape reconstruction from two views compared with the 3D segmentation Dice of 97.4%. The network also showed significantly lower mean absolute error values of 3.5 mL/4.9 cm2 for LA volume/surface area respectively compared to the area-length method errors of 13.0 mL/34.1 cm2 respectively (P < 0.05 for both). For the independent CMR test set, the network achieved accurate 3D shape estimation (mean Dice score value of 87.4%), and a mean absolute error values of 6.0 mL/5.7 cm2 for left atrial volume/surface area respectively, significantly less than the area-length method errors of 14.2 mL/19.3 cm2 respectively (P < 0.05 for both). Compared to the bi-plane area-length method, the network showed higher accuracy and robustness for both volume and surface area.

Intelligent microscopic 3D shape reconstruction method based on 3D time-frequency transformation

Three-dimensional (3D) shape reconstruction methods based on image focus information generally use a unified focus evaluation metric for the depth-of-field image sequences of microscopic objects. With the absence of the correlation among these sequences, this kind of reconstruction method fails to correct the depth errors of consecutive frame images that result from textural sparsity or low contrast in images. In this paper, different image sequences of microscopic objects are interpreted as 3D image data, such that the correlation among all the sequences can be introduced to the reconstruction. Then, a reconstruction framework encompassing multiview analysis, locally stable clustering, and selective fusion as its key components is established from the perspective of 3D time-frequency transformation. To be specific, we analyze the advantage of 3D image data in shape reconstruction over general 2D image data and construct a 3D time-frequency transform that maps a 3D image to depth images, which contain information of varying scales, regions, and directions. Next, to enhance depth image features, a locally stable clustering algorithm is constructed based on multimodal textural features, which improves the adaptive selection ability of depth images with good homogeneity. Finally, we propose the selective fusion of depth images, which outputs accurate 3D morphology results by fitting multiple layers of noise-free depth images through a layer-by-layer sifting balance tree designed in this study. The effectiveness of the proposed method is verified on simulation data and real scenes. The reconstruction method of this perspective may bring fresh air to the field of submicron industrial measurement and sophisticated manufacturing.