Accurate Shape Reconstruction Research Articles

3D shape sensor based on discrete-point Rayleigh reflectors inscribed by femtosecond pulses in multicore fibers

Fiber optic sensors which use reflectometry methods to process Rayleigh backscattering signal, are susceptible to any optical losses due to the inherently low level of Rayleigh backscattering in standard telecom optical fibers. In this work, we fabricate and study the shape sensor based on a multicore optical fiber with randomly spaced discrete-point reflectors inscribed in its cores by femtosecond laser pulses. By testing the sensor on different 3D shape samples, we demonstrate the robustness of the shape reconstruction accuracy to introduced optical losses of the signal up to 20 dB with the relative reconstruction error being less than 4 %. In contrast, such level of optical losses dramatically increases the reconstruction error when the unmodified fiber cores are used. A higher signal-to-noise ratio together with low birefringence of the inscribed point reflectors enable accurate shape sensing including the forms with low curvature.

Accuracy improvement of two-dimensional shape reconstruction based on OFDR using first-order differential local filtering

The accuracy of two-dimensional (2D) shape reconstruction is highly susceptible to fake peaks in the strain distribution measured by optical frequency domain reflectometry (OFDR). In this paper, a post-processing method using first-order differential local filtering is proposed to suppress fake peaks and further improve the accuracy of shape reconstruction. By analyzing the principles of 2D shape reconstruction, an explanation of how fake peaks lead to shape reconstruction errors is provided, along with the introduction of an error evaluation standard. The principle of first-order differential local filtering is presented, and its feasibility is verified by simulation. An OFDR 2D shape reconstruction system is built, with three groups of 2D shape reconstruction experiments carried out, including up bending, down bending and arch bending. The experimental results show that the end errors of the three groups of shape reconstruction are respectively reduced from 2.33%, 2.97%, and 1.07% to 0.25%, 0.78%, and 0.20%, at the shape reconstruction length of 0.5 m. The research demonstrates that the accuracy of OFDR 2D shape reconstruction can be improved by using first-order differential local filtering.

Lie Group Formulation and Sensitivity Analysis for Shape Sensing of Variable Curvature Continuum Robots With General String Encoder Routing

This article considers a combination of actuation tendons and measurement strings to achieve accurate shape sensing and direct kinematics of continuum robots. Assuming general string routing, a methodical Lie group formulation for the shape sensing of these robots is presented. The shape kinematics is expressed using arc-length-dependent curvature distributions parameterized by modal functions, and the Magnus expansion for Lie group integration is used to express the shape as a product of exponentials. The tendon and string length kinematic constraints are solved for the modal coefficients and the configuration space and body Jacobian are derived. The noise amplification index for the shape reconstruction problem is defined and used for optimizing the string/tendon routing paths, and a planar simulation study shows the minimal number of strings/tendons needed for accurate shape reconstruction. A torsionally stiff continuum segment is used for experimental evaluation, demonstrating mean (maximal) end-effector absolute position error of less than 2% (5%) of total length. Finally, a simulation study of a torsionally compliant segment demonstrates the approach for general deflections and string routings. We believe that the methods of this article can benefit the design process, sensing, and control of continuum and soft robots.

Shape Sensing of Flexible Robots Based on Deep Learning

In this article, a deep learning method for the shape sensing of continuum robots based on multicore fiber bragg grating (FBG) fiber is introduced. The proposed method, based on an artificial neural network (ANN), differs from traditional approaches, where accurate shape reconstruction requires a tedious characterization of many characteristic parameters. A further limitation of traditional approaches is that they require either multiple fibers, whose location relative to the centerline must be precisely known (calibrated), or a single multicore fiber whose position typically coincides with the neutral line. The proposed method addresses this limitation and, thus, allows shape sensing based on a single multicore fiber placed off-center. This helps in miniaturizing and leaves the central channel available for other purposes. The proposed approach was compared to a recent state-of-the-art model-based shape sensing approach. A two-degree-of-freedom benchtop fluidics-driven catheter system was built to validate the proposed ANN. The proposed ANN-based shape sensing approach was evaluated on a 40-mm-long steerable continuum robot in both 3-D free-space and 2-D constrained environments, yielding an average shape sensing error of 0.24 and 0.49 mm, respectively. With these results, the superiority of the proposed approach compared to the recent model-based shape sensing method was demonstrated.

Laser ultrasonic imaging of subsurface defects with the linear sampling method.

Laser ultrasonics is a remote nondestructive evaluation technique suitable for real-time monitoring of fabrication processes in semiconductor metrology, advanced manufacturing, and other applications where non-contact, high fidelity measurements are required. Here we investigate laser ultrasonic data processing approaches to reconstruct images of subsurface side drilled holes in aluminum alloy specimens. We demonstrate through simulation that the model-based linear sampling method (LSM) can perform accurate shape reconstruction of single and multiple holes and produce images with well-defined boundaries. We experimentally confirm that LSM produces images that represent the internal geometric features of an object, some of which may be missed by conventional imaging.

Optical fiber shape sensing of flexible medical instruments with temperature compensation

Needle shape monitoring techniques ensure the safety of modern interventional surgery. Optical fiber sensors such as fiber Bragg grating (FBG) are being increasingly employed for needle shape sensing. However, existing studies have overlooked the simultaneous strain and temperature sensitivities of the sensors in the fiber grating sensing array, directly affecting the shape reconstruction accuracy under variable temperature environments. Thus, this study proposed a method to optimize shape reconstruction-based fiber optic sensing by introducing a strain and temperature sensitivity matrix. First, the relationship between the wavelength shift of the fiber grating implanted in the needle and its strain and temperature was analyzed based on the fiber grating sensing theory. Thereafter, the strain sensitivity matrix was introduced to improve the accuracy of the shape reconstruction algorithm. Moreover, temperature compensation eliminated temperature influence on the sensor. Next, the local unit geometric parameter relationship and coordinate transformation equation of the needle were derived, and the shape sensing model of the FBG array-based needle was established. Finally, the effectiveness of the method was verified by conducting variable temperature shape reconstruction experiments. The proposed method had increased accuracy over the conventional method in shape reconstruction under variable temperature environments, thus indicating its superiority and application feasibility.

Photogrammetry-Based Volume Measurement Framework for the Particle Density Estimation of LECA.

This paper presents a photogrammetry-based volume measurement framework for the particle density estimation of Lightweight expanded clay aggregate (LECA). The results are compared with computed tomography (CT) and Archimedes’ method measurements. All of the steps required in order to apply the proposed approach are explained. Next, we discuss how the interpretation of open pores affects the results of volume measurements. We propose to process the shapes obtained from different methods by applying an Ambient Occlusion algorithm with the same threshold, t = 0.175. The difference between the CT and SfM methods is less than 0.006 g/cm3, proving that the photogrammetry-based approach is accurate enough. The Archimedes’ method significantly overestimates the density of the particles. Nevertheless, its accuracy is acceptable for most engineering purposes. Additionally, we evaluate the accuracy of shape reconstruction (in terms of the Hausdorff distance). For 95% of the grain’s surface, the maximum error is between 0.073 mm and 0.129 mm (depending on the grain shape). The presented approach is helpful for measuring the particle density of porous aggregates. The proposed methodology can be utilized in order to estimate intergranular porosity, which is valuable information for the calibration of DEM models.

Design and Measurement of a Dual FBG High-Precision Shape Sensor for Wing Shape Reconstruction.

FBG shape sensors based on soft substrates are currently one of the research focuses of wing shape reconstruction, where soft substrates and torque are two important factors affecting the performance of shape sensors, but the related analysis is not common. A high-precision soft substrates shape sensor based on dual FBGs is designed. First, the FBG soft substrate shape sensor model is established to optimize the sensor size parameters and get the optimal solution. The two FBG cross-laying method is adopted to effectively reduce the influence of torque, the crossover angle between the FBGs is 2α, and α = 30° is selected as the most sensitive angle to the torquer response. Second, the calibration test platform of this shape sensor is built to obtain the linear relationship among the FBG wavelength drift and curvature, rotation radian loaded vertical force and torque. Finally, by using the test specimen shape reconstruction test, it is verified that this shape sensor can improve the shape reconstruction accuracy, and that its reconstruction error is 6.13%, which greatly improves the fit of shape reconstruction. The research results show that the dual FBG high-precision shape sensor successfully achieves high accuracy and reliability in shape reconstruction.

Robust Catheter Tracking by Fusing Electromagnetic Tracking, Fiber Bragg Grating and Sparse Fluoroscopic Images

Endovascular catheterization is an intervention which offers a low risk alternative to open surgery in many patients. Today’s interventions rely heavily on fluoroscopic imaging to guide interventionalists. Fluoroscopy only produces 2D visualization of the catheter and also exposes both the patient and interventionalists to harmful radiation. Different approaches have been proposed to overcome the limitations of fluoroscopy. Fiber Bragg Grating (FBG)-based shape sensing is becoming popular to reconstruct the catheter shape. Multi-core fibers with parallel optical cores are interesting as they allow 3D shape reconstruction with a single fiber. A common issue with FBG-based shape sensing is its sensitivity to variations in twist. Even small amounts of twist can significantly impact the overall shape reconstruction accuracy. This work proposes a novel approach which combines electromagnetic tracking (EMT), FBG-based shape sensing, and sparse fluoroscopic images. The method provides real-time 3D visualization of the catheter without the need of continuous fluoroscopy. A unique feature of the proposed method is the selective use of imaging for dynamic twist-compensation of the FBG sensor. The proposed sensor-fusion method improved 3D reconstruction accuracy. Real-world in-vitro experiments promising results. For a catheter with an embedded fiber length of 170 mm, the proposed approach the 3D shape with a median root-mean-square (rms) error of 0.39 mm and an interquartile range of 0.10 mm in the 2D experiment in which the catheter was bent in a plane. A median rms error of 0.54 mm and an interquartile range of 0.07 mm were achieved in the 3D experiments.

Learning an animatable detailed 3D face model from in-the-wild images

While current monocular 3D face reconstruction methods can recover fine geometric details, they suffer several limitations. Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression. Other methods are trained on high-quality face scans and do not generalize well to in-the-wild images. We present the first approach that regresses 3D face shape and animatable details that are specific to an individual but change with expression. Our model, DECA (Detailed Expression Capture and Animation), is trained to robustly produce a UV displacement map from a low-dimensional latent representation that consists of person-specific detail parameters and generic expression parameters, while a regressor is trained to predict detail, shape, albedo, expression, pose and illumination parameters from a single image. To enable this, we introduce a novel detail-consistency loss that disentangles person-specific details from expression-dependent wrinkles. This disentanglement allows us to synthesize realistic person-specific wrinkles by controlling expression parameters while keeping person-specific details unchanged. DECA is learned from in-the-wild images with no paired 3D supervision and achieves state-of-the-art shape reconstruction accuracy on two benchmarks. Qualitative results on in-the-wild data demonstrate DECA's robustness and its ability to disentangle identity- and expression-dependent details enabling animation of reconstructed faces. The model and code are publicly available at https://deca.is.tue.mpg.de.

Learning an animatable detailed 3D face model from in-the-wild images

While current monocular 3D face reconstruction methods can recover fine geometric details, they suffer several limitations. Some methods produce faces that cannot be realistically animated because they do not model how wrinkles vary with expression. Other methods are trained on high-quality face scans and do not generalize well to in-the-wild images. We present the first approach that regresses 3D face shape and animatable details that are specific to an individual but change with expression. Our model, DECA (Detailed Expression Capture and Animation), is trained to robustly produce a UV displacement map from a low-dimensional latent representation that consists of person-specific detail parameters and generic expression parameters, while a regressor is trained to predict detail, shape, albedo, expression, pose and illumination parameters from a single image. To enable this, we introduce a novel detail-consistency loss that disentangles person-specific details from expression-dependent wrinkles. This disentanglement allows us to synthesize realistic person-specific wrinkles by controlling expression parameters while keeping person-specific details unchanged. DECA is learned from in-the-wild images with no paired 3D supervision and achieves state-of-the-art shape reconstruction accuracy on two benchmarks. Qualitative results on in-the-wild data demonstrate DECA's robustness and its ability to disentangle identity- and expression-dependent details enabling animation of reconstructed faces. The model and code are publicly available at https://deca.is.tue.mpg.de.

Piecewise constant level-set enhanced active shape reconstruction for electrical impedance tomography

To accurately access the quantitative information of industrial and biomedical processes, a piecewise constant level-set enhanced inclusion boundary reconstruction (PLBR) method is proposed for EIT. The target inclusion boundaries are parameterized by a parametric active contour joint piecewise constant level set model. An augmented least square approach is presented to simultaneously estimate the unknown boundary and conductivity parameters from the EIT voltage data. Compared with the other active shape reconstruction methods of EIT, the PLBR does not need to know the number of target inclusion and the number of conductivity phases, thus owns wider applications. The numerical and experiential results from a tank model with both single- and multi-phase conductivity further proved the topological robustness and shape reconstruction accuracy of the proposed method.

Accuracy of position tracking and fabrication of thin diameter sensor

The shape reconstruction of medical surgical instruments in robotic laryngeal surgery, positioning a needle to puncture a breast tumour, shape display during intestinal endoscopy, and positioning a cardiac vascular catheter, is affected by not only the end position and the shape reconstruction accuracy but also the diameter and curvature of the sensor owing to the limited application space. A sensor with a substrate is manufactured by pasting a fibre Bragg grating (FBG) on the surface of nitinol wire. The measuring range of the substrate is improved by applying a certain preload before glue injection. The sensor has the advantages of being less prone to breakage and greater durability but has a limited bending curvature and diameter. In minimally invasive surgery, the curvature of the body cavity is complex, and the space is narrow, and thus the FBG sensor must have a smaller diameter and good bending curvature. The present paper proposes a method of filling and encapsulating a small-diameter substrate sensor. A substrate-less sensor with a diameter of 1.5 mm and sensing length of 200 mm is encapsulated by improving the packaging positioning piece, current glue injection method, and packaging process. Four FBGs with two grating points for each fibre are distributed at 90° to each other, and four measuring points are obtained by wavelength division multiplexing and space division multiplexing. A calibration and shape reconstruction experiment is carried out for the shape sensor, while a packaging error compensation calibration method is introduced to improve the precision of shape reconstruction. Results of a two-dimensional experiment show that the shape reconstruction accuracy of the sensor on the plane bending is 2.71%. In a three-dimensional spiral experiment, the measurement accuracy of FBG shape sensor out-of-plane bending is 3.42%.

Three‐dimensional shape reconstruction of objects from a single depth view using deep U‐Net convolutional neural network with bottle‐neck skip connections

Three-dimensional (3D) shape reconstruction of objects requires multiple scans and complex reconstruction algorithms. An alternative approach is to infer the 3D shape of an object from a single depth image (i.e. single depth view). This study presents such a 3D shape reconstructor based on U-Net 3D-convolutional neural network (3D-CNN) with bottle-neck skipped connection blocks (U-Net BNSC 3D-CNN) to infer the 3D shapes of objects from only a single depth view. The BNSC block is a fully convolutional block that uses skip connections to improve the performance of the sequential 3D-convolutional layers of U-Net. The primary advantage of U-Net BNSC 3D-CNN is improving the accuracy of shape reconstruction while reducing the computational load. The evaluation of the proposed U-Net BNSC 3D-CNN uses unseen views from trained and untrained objects with two public databases, ShapeNet and Grasp database. Our reconstructor achieves 72.17% and 69.97% accuracy in terms of the Jaccard similarity index for trained and untrained objects, respectively, with the ShapeNet database, whereas previous reconstructor based on 3D-CNN achieves 66.43% and 58.35%. With Grasp database, our reconstructor achieves 87.03% and 85.35%, whereas 3D-CNN 76.52% and 76.02%. Also, our U-Net BNSC 3D-CNN reduces the computational load of the standard 3D-CNN reconstructor by 6.67% in the computation time and by 98.69% in the number of trainable parameters.

Fast reconstruction of single-shot wide-angle diffraction images through deep learning

Single-shot x-ray imaging of short-lived nanostructures such as clusters and nanoparticles near a phase transition or non-crystalizing objects such as large proteins and viruses is currently the most elegant method for characterizing their structure. Using hard x-ray radiation provides scattering images that encode two-dimensional projections, which can be combined to identify the full three-dimensional object structure from multiple identical samples. Wide-angle scattering using XUV or soft x-rays, despite yielding lower resolution, provides three-dimensional structural information in a single shot and has opened routes towards the characterization of non-reproducible objects in the gas phase. The retrieval of the structural information contained in wide-angle scattering images is highly non-trivial, and currently no efficient rigorous algorithm is known. Here we show that deep learning networks, trained with simulated scattering data, allow for fast and accurate reconstruction of shape and orientation of nanoparticles from experimental images. The gain in speed compared to conventional retrieval techniques opens the route for automated structure reconstruction algorithms capable of real-time discrimination and pre-identification of nanostructures in scattering experiments with high repetition rate—thus representing the enabling technology for fast femtosecond nanocrystallography.

Fully Auto-calibrated Active-stereo-based 3D Endoscopic System using Correspondence Estimation with Graph Convolutional Network.

We have developed a series of 3D endoscopic systems where a micro-sized pattern projector is inserted through the instrument channel of the endoscope and shapes are reconstructed by a structured light technique using captured images of the endoscopic camera. One problem of the previous works is that the accuracy of shape reconstruction is low, because the projector cannot be fixed to the endoscope, and thus, the pose of the pattern projector w.r.t. the camera cannot be pre-calibrated. In this paper, we propose a method to auto-calibrate the pose of the projector without using any special devices nor manual process. Since the technique is one-shot, multiple shapes can be reconstructed from an image sequence and a large 3D scene can be recovered by merging them. Experiments are conducted using the real system.

Updated shape sensing algorithm for space curves with FBG sensors

When the terminal position and shape detection properties of a fiber Bragg grating (FBG) curve sensor are used in medicine, the positioning accuracy of the FBG curve sensor is critical in the prediagnosis and treatment of diseases. To improve the shape reconstruction accuracy, two updated reconstruction algorithms are proposed based on the error accumulation of a single-point recursive reconstruction algorithm according to the Frenet coordinate system. First, the motion coordinate system depends on the curve and tangent vector. Next, the curvature vector is synthesized in the motion coordinate system on the close plane, the motion coordinate transformation matrix of the discrete point is calculated in the close plane, and the relevant formula is deduced subsequently. Finally, the improved algorithms are verified out. Experimental results indicate that the proposed algorithms improved the terminal position accuracy. Compared with the single point recursive algorithm, in a single bending experiment, the maximum shape errors of the terminal position are 2.65% and 2.53% using the fitting method of multipoint curvature and nonfixed point fitting method based on the Frenet frame, respectively. In double bending experiment, the maximum shape errors of the applied improved algorithms are 2.74% and 2.60%, respectively, which exhibit good planar shape reconstruction accuracy. In an out-of-plane three-dimensional shape reconstruction experiment, the maximum shape errors of these two improved methods are 3.40% and 3.32%, respectively. The improved algorithms provide an algorithm basis for the improvement of space curve reconstruction accuracy, and they can be applied in the detection of wing deformation structure and real-time detection of soil deformation in shield engineering.

Second Iteration of Photogrammetric Processing to Refine Image Orientation with Improved Tie-Points †.

Photogrammetric processing is available in various software solutions and can easily deliver 3D pointclouds as accurate as 1 pixel. Certain applications, e.g., very accurate shape reconstruction in industrial metrology or change detection for deformation studies in geosciences, require results of enhanced accuracy. The tie-point extraction step is the opening in the photogrammetric processing chain and therefore plays a key role in the quality of the subsequent image orientation, camera calibration and 3D reconstruction. Improving its precision will have an impact on the obtained 3D. In this research work we describe a method which aims at enhancing the accuracy of image orientation by adding a second iteration photogrammetric processing. The result from the classical processing is used as a priori information to guide the extraction of refined tie-points of better photogrammetric quality. Evaluated on indoor and UAV acquisitions, the proposed methodology shows a significant improvement on the obtained 3D point accuracy.

Using a single-cube beam-splitter as a fringe pattern generator within a structured-light projection system for surface metrology

We show how a sinusoidal fringe pattern can be obtained by using a single cube beam splitter based on the Gates’ interferometer configuration. When an expanded and collimated laser beam hits the binding edge of a nonpolarizing cube beam splitter parallel to the splitter coating, it generates interference fringes at the exit due to the internal reflections and refractions of the laser beam passing through the cube. Unlike common digital projection systems, the proposed optical arrangement generates a perfectly sinusoidal and continuous fringe pattern, minimizing the problems associated with the discretization of a synthetic digital signal. The fast Fourier transform and phase-shifting techniques are used to demodulate the captured fringe patterns. Experimental results are presented for the three-dimensional shape reconstruction of the relief of a coin and of a spherical indentation on a piece of aluminum with a maximum height of about 150 μm. In addition, we evaluate the accuracy and resolution of the proposed measuring device: shape reconstruction accuracy is about 1.4% and axial resolution is 0.15 μm. Due to its simple and compact setup, the proposed system is particularly suited to be miniaturized.

Electromagnetic Positioning for Tip Tracking and Shape Sensing of Flexible Robots

Wire-driven flexible robots are efficient devices for minimally invasive surgery, since they can work well in complex and confined environments. However, the real-time positional and shape information of the robot cannot be well estimated, especially when there is an unknown payload or force working on the end effector. In this paper, we propose a novel tip tracking and shape sensing method for wire-driven flexible robots. The proposed method is based on the length of each section of the robot as well as the positional and directional information of the distal end of each section of the robot. For each section, an electromagnetic sensor will be mounted at the distal end to estimate the positional and directional information. A reconstruction algorithm, which is based on a three-order Bezier curve, is carried out utilizing the positional and directional information along with the length information of the section. This method provides the advantage of good tracking results and high shape reconstruction accuracy with limited modification to the robot. Compared with other reconstruction methods, no kinematic model is needed for reconstruction. Therefore, this method works well with an unknown payload that applied at the tip of the robot. The feasibility of the proposed method is verified by simulation and experimental results.