Three-dimensional Shape Reconstruction Research Articles

Deep Learning-Based Ensemble Approach for Autonomous Object Manipulation with an Anthropomorphic Soft Robot Hand

Autonomous object manipulation is a challenging task in robotics because it requires an essential understanding of the object’s parameters such as position, 3D shape, grasping (i.e., touching) areas, and orientation. This work presents an autonomous object manipulation system using an anthropomorphic soft robot hand with deep learning (DL) vision intelligence for object detection, 3D shape reconstruction, and object grasping area generation. Object detection is performed using Faster-RCNN and an RGB-D sensor to produce a partial depth view of the objects randomly located in the working space. Three-dimensional object shape reconstruction is performed using U-Net based on 3D convolutions with bottle-neck layers and skip connections generating a complete 3D shape of the object from the sensed single-depth view. Then, the grasping position and orientation are computed based on the reconstructed 3D object information (e.g., object shape and size) using U-Net based on 3D convolutions and Principal Component Analysis (PCA), respectively. The proposed autonomous object manipulation system is evaluated by grasping and relocating twelve objects not included in the training database, achieving an average of 95% successful object grasping and 93% object relocations.

Three-dimensional surface shape reconstruction of fiber bragg gratings in a ring arrangement

Three-dimensional surface shape reconstruction of fiber bragg gratings in a ring arrangement

A novel algorithm for three-dimensional shape reconstruction for microscopic objects based on shape from focus

A novel algorithm for three-dimensional shape reconstruction for microscopic objects based on shape from focus

A three-dimensional vision measurement method based on double-line combined structured light

In this paper, a structured light vision measurement method using a scanning laser line and a positioning laser line is proposed. The novel method enables the scanning laser plane to slide along a slide rail while maintaining intersection with the positioning laser plane, eliminating the need to determine the scanning direction and moving step. During the measurement process, the laser plane equations need to be recalibrated for each new position, so a real-time calibration method is given. Initially, the geometric barycenter method is employed to detect the subpixel coordinates of the light stripe intersection point. Subsequently, these coordinates are projected into the camera coordinate system using the initial equations of the positioning laser plane. Finally, leveraging the normal information of the initial equation of the scanning laser plane and the three-dimensional coordinates of the light stripe intersection point, the real-time calibration of the scanning laser plane equations can be accomplished. The proposed method enables the three-dimensional reconstruction of objects, and its accuracy is verified through measurements on gauge blocks. Experimental results demonstrate that this method achieves precise and stable three-dimensional reconstruction of object surface shape.

Integration of orthogonal Fourier profilometry with blood flow mapping for 3D fingertip biometric authentication

We present herein the combination of Fourier transform profilometry based orthogonal projection with a laser speckle imaging method to provide three-dimensional (3D) fingertip blood flow imaging and shape reconstruction for biometric authentication purposes. In the proposed approach, orthogonal sinusoidal grating patterns with a fixed spatial frequency emanating from a digital projector first illuminate the surface of a person's finger. Then, the projector is blocked and a laser source illuminates the finger. The projector and laser work alternately such that only one source illuminates the finger at a given time allowing no illumination crosstalk between the two devices. The deformed grating patterns and the diffuse backscattered laser light, respectively, are recorded on the same CCD camera. In the computer, the 3D shape of the finger is reconstructed simultaneously with a blood flow map of the finger. Finally, the images are merged yielding a dynamic flow topography visualization of the person's finger. This dual-modality strategy utilizes the strengths of each imaging method in a complementary way, thus enabling simple 3D shape recovery and characterization of a specific user, and increasing the specificity of authentication security. The method is experimentally validated on both human and fake fingers through VanderLugt optical correlation framework. Overall, the results demonstrate the utility of our proposed approach for biometric authentication. We envision that our optoelectronic setup could be potentially integrated into a wide variety security platform.

Time-Distributed Framework for 3D Reconstruction Integrating Fringe Projection with Deep Learning.

In recent years, integrating structured light with deep learning has gained considerable attention in three-dimensional (3D) shape reconstruction due to its high precision and suitability for dynamic applications. While previous techniques primarily focus on processing in the spatial domain, this paper proposes a novel time-distributed approach for temporal structured-light 3D shape reconstruction using deep learning. The proposed approach utilizes an autoencoder network and time-distributed wrapper to convert multiple temporal fringe patterns into their corresponding numerators and denominators of the arctangent functions. Fringe projection profilometry (FPP), a well-known temporal structured-light technique, is employed to prepare high-quality ground truth and depict the 3D reconstruction process. Our experimental findings show that the time-distributed 3D reconstruction technique achieves comparable outcomes with the dual-frequency dataset (p = 0.014) and higher accuracy than the triple-frequency dataset (p = 1.029 × 10-9), according to non-parametric statistical tests. Moreover, the proposed approach's straightforward implementation of a single training network for multiple converters makes it more practical for scientific research and industrial applications.

Three-dimensional reconstruction of the furrow shape in orchards using a low-cost lidar

Three-dimensional reconstruction of the furrow shape in orchards using a low-cost lidar

Scan-free time-of-flight-based three-dimensional imaging through a scattering layer.

Reconstructing an object's three-dimensional shape behind a scattering layer with a single exposure is of great significance in real-life applications. However, due to the little information captured by a single exposure while strongly perturbed by the scattering layer and encoded by free-space propagation, existing methods cannot achieve scan-free three-dimensional reconstruction through the scattering layer in macroscopic scenarios using a short acquisition time of seconds. In this paper, we proposed a scan-free time-of-flight-based three-dimensional reconstruction method based on explicitly modeling and inverting the time-of-flight-based scattering light propagation in a non-confocal imaging system. The non-confocal time-of-flight-based scattering imaging model is developed to map the three-dimensional object shape information to the time-resolved measurements, by encoding the three-dimensional object shape into the free-space propagation result and then convolving with the scattering blur kernel derived from the diffusion equation. To solve the inverse problem, a three-dimensional shape reconstruction algorithm consisting of the deconvolution and diffractive wave propagation is developed to invert the effects caused by the scattering diffusion and the free-space propagation, which reshapes the temporal and spatial distribution of scattered signal photons and recovers the object shape information. Experiments on a real scattering imaging system are conducted to demonstrate the effectiveness of the proposed method. The single exposure used in the experiment only takes 3.5s, which is more than 200 times faster than confocal scanning methods. Experimental results show that the proposed method outperforms existing methods in terms of three-dimensional reconstruction accuracy and imaging limit subjectively and objectively. Even though the signal photons captured by a single exposure are too highly scattered and attenuated to present any valid information in time gating, the proposed method can reconstruct three-dimensional objects located behind the scattering layer of 9.6 transport mean free paths (TMFPs), corresponding to the round-trip scattering length of 19.2 TMFPs.

Deep learning framework-based 3D shape reconstruction of tanks from a single RGB image

In recent times, complicated three-dimensional shape reconstruction from a single RGB image has become a crucial technology in many industries such as Automotive, Healthcare, and Military. It is particularly challenging to reconstruct the complex shape of a military tank. Former methods infer 3D information from 2D images via shape deformation from ellipsoids, which has the problems of local adhesion, uneven surfaces, and distortion of the structure. This study introduces a new single-view 3D shape reconstruction (SVSR) framework with multi-scale feature extraction that splits shape reconstruction into three tasks—camera parameter prediction, initial shape construction, and deformation prediction. The shape-initialization module provides a variable initial shape by predicting the stretch and displacement parameters of each geometric component based on their topological relationships. The shape-deformation module predicts the directional deformation of each vertex. These two modules are dedicated separately to avoid shape adhesion and improve the local detail performance. Silhouette images of the tank’s overall shape and local geometric components are employed to eliminate the impact of component shielding and avoid structural distortion via both pixel loss and perceptual loss. Experiments on the main battle tank dataset demonstrate that our method can predict complicated 3D shapes with a low Chamfer distance value (0.0017). Our approach outperforms the other state-of-the-art methods in terms of Chamfer distance and F-score with at least a 10% improvement, with more realistic overall contours and part details. It has huge application prospects in dealing with other complex shape predictions by modifying the shape-initialization module.

Research on surface shape measurement of highly reflective surfaces based on stripe projection

Metal workpieces with highly reflective surfaces are particularly common in modern industrial manufacturing systems, and their three-dimensional dimensional measurement has great applications and needs in various fields. Among the various three-dimensional dimensional measurement technologies, projection three-dimensional measurement technology is adopted for its non-contact, low-cost, high-precision, and fast data processing characteristics. However, because of the high reflectance of the highly reflective surface, the structured light photos taken by the camera will be overexposed and underexposed, which will lead to the lack of three-dimensional information obtained. Due to the large range of changes in surface reflectance and the limited grayscale range of the camera, the traditional stripe projection method cannot correctly measure the three-dimensional morphology of highly reflective surfaces [1]. This paper introduces an adaptive sinusoidal stripe grating based on the reflectance of the surface of an object, which is used to compensate for the overexposure of the camera when shooting on a highly reflective surface. Finally, the generated adaptive stripes are projected onto the measured object, and the phase solution and three-dimensional surface shape reconstruction are carried out by the heterodyne multi-frequency phase shift method.

Intelligent microscopic 3D shape reconstruction method based on 3D time-frequency transformation

Three-dimensional (3D) shape reconstruction methods based on image focus information generally use a unified focus evaluation metric for the depth-of-field image sequences of microscopic objects. With the absence of the correlation among these sequences, this kind of reconstruction method fails to correct the depth errors of consecutive frame images that result from textural sparsity or low contrast in images. In this paper, different image sequences of microscopic objects are interpreted as 3D image data, such that the correlation among all the sequences can be introduced to the reconstruction. Then, a reconstruction framework encompassing multiview analysis, locally stable clustering, and selective fusion as its key components is established from the perspective of 3D time-frequency transformation. To be specific, we analyze the advantage of 3D image data in shape reconstruction over general 2D image data and construct a 3D time-frequency transform that maps a 3D image to depth images, which contain information of varying scales, regions, and directions. Next, to enhance depth image features, a locally stable clustering algorithm is constructed based on multimodal textural features, which improves the adaptive selection ability of depth images with good homogeneity. Finally, we propose the selective fusion of depth images, which outputs accurate 3D morphology results by fitting multiple layers of noise-free depth images through a layer-by-layer sifting balance tree designed in this study. The effectiveness of the proposed method is verified on simulation data and real scenes. The reconstruction method of this perspective may bring fresh air to the field of submicron industrial measurement and sophisticated manufacturing.

Three-dimensional shape reconstruction of aggregate in concrete using two-dimensional periodic wavelet series

Three-dimensional shape reconstruction of aggregate in concrete using two-dimensional periodic wavelet series

Development of a multispectral stereo-camera system comparable to Hayabusa2 optical navigation camera (ONC-T) for observing samples returned from asteroid (162173) Ryugu

Hayabusa2 collected 5.4 g of samples from the asteroid (162173) Ryugu and brought them back to Earth. Obtaining multiband images of these samples with the spectral bands comparable to those used for remote-sensing observations is important for characterizing the collected samples and examining how representative the sample is compared with spacecraft observations of Ryugu as a whole. In this study, we constructed a multiband microscopic camera system that enables both visual multispectral imaging at 390 (ul), 475 (b), 550 (v), 590 (Na), 700 (w), and 850 nm (x), and three-dimensional (3D) shape reconstruction of individual grain samples based on stereo imaging. The imaging system yields the images of 4096 x 2160 pixels with the pixel resolution of 1.93 {\mu}m/pix and field of view of 7.9 x 4.2 mm. Our validation measurements demonstrate that our multispectral imaging system, which observes the samples with the spectral bands comparable to those on the telescopic optical navigation camera (ONC-T) on Hayabusa2, yields reflectance spectra with a relative error of 3% and a 3D model with an error of 5%. These results indicate that the multiband imaging system with a 3D shape reconstruction capability yields accurate spectral and shape data on the returned samples. Using this instrument, we conducted multispectral measurements of two Ryugu samples (grains in the dishes A3 and C1) acquired from two locations on the asteroid. The average spectra of the measured Ryugu samples were flat and consistent with the global averaged spectrum of Ryugu. The 550-nm band (v-band) reflectance of the returned grains in the dishes was 2.4% on average, higher than that of the global averaged spectrum of Ryugu observed with ONC-T. This apparent difference could be because the returned grains have greater specular reflectance.

Three-dimensional large deformation reconstruction for slender wing-like structures from strain measurements

Shape-sensing technology has significant applications in real-time monitoring and controlling flexible aerospace structures. Aiming at large deformation of flexible wing-like structures, an algorithm that incorporates geometric nonlinearity is developed for three-dimensional shape reconstruction based on strain measurements. The full-field three-dimensional deformation is described in terms of the deflection of the spatial reference line and the motions of the cross-sectional plane. A certain number of strain measurements distributed on the structural surface are used to estimate the generalized strains and the warping deformation, which can further determine the position of an arbitrary material point in the structure by a series of recursion relations. Instead of the experimental research, the desired strains and the deformation as references are simulated by high-fidelity finite element models. A simple tube model and a more realistic composite wing model are investigated separately to demonstrate the present method in reconstructing deformed shapes under various applied loads. Besides, the modal method was also used to calculate the deformation as a comparison. The result has demonstrated the high accuracy and capability of the present method in three-dimensional shape reconstruction, especially for large deformation problems. The warping effect is demonstrated to have a significant impact on predicting the twist angle. More attention should be paid to accurately describing the warping deformation of complicated wing models, especially when the bending deformation is large.

Single-shot 3D shape reconstruction for complex surface objects with colour texture based on deep learning

Three-dimensional (3D) shape reconstruction based on structured light technique is one of the most crucial techniques in the field of optical measurement due to the nature of non-contact and high-precision. In recent years, researchers found that the deep-learning method can significantly improve the quality and efficiency of 3D reconstruction. So far, the research object of structured light 3D reconstruction technology related to deep-learning methods usually adopts plaster models. However, the surface of the reconstructed objects always contains rich colour texture information in a real-world scenario, which affects the accuracy of the reconstruction, especially for some single-shot applications. To solve the above problem, we propose a deep-learning-based method single-shot 3D reconstruction method (SSR) for colour textured objects, and it can reconstruct colour textured objects with high precision. The experimental results show that the proposed method provides good guidance for the practical application and scientific research of fringe projection profilometry based on colour textured objects.

Comparison of Bone Segmentation Software over Different Anatomical Parts

Three-dimensional bone shape reconstruction is a fundamental step for any subject-specific musculo-skeletal model. Typically, medical images are processed to reconstruct bone surfaces via slice-by-slice contour identification. Freeware software packages are available, but commercial ones must be used for the necessary certification in clinics. The commercial software packages also imply expensive hardware and demanding training, but offer valuable tools. The aim of the present work is to report the performance of five commercial software packages (Mimics®, AmiraTM, D2PTM, SimplewareTM, and Segment 3D PrintTM), particularly the time to import and to create the model, the number of triangles of the mesh, and the STL file size. DICOM files of three different computed tomography scans from five different human anatomical areas were utilized for bone shape reconstruction by using each of these packages. The same operator and the same hosting hardware were used for these analyses. The computational time was found to be different between the packages analyzed, probably because of the pre-processing implied in this operation. The longer “time-to-import” observed in one software is likely due to the volume rendering during uploading. A similar number of triangles per megabyte (approximately 20 thousand) was observed for the five commercial packages. The present work showed the good performance of these software packages, with the main features being better than those analyzed previously in freeware packages.

Computer Aided Orthognathic Surgery: A General Method for Designing and Manufacturing Personalized Cutting/Repositioning Templates

Orthognathic surgery allows broad-spectrum deformity correction involving both aesthetic and functional aspects on the TMJ (temporo-mandibular joint) and on the facial skull district. The combination of Reverse Engineering (RE), Virtual Surgery Planning (VSP), Computer Aided Design (CAD), Additive Manufacturing (AM), and 3D visualization allows surgeons to plan, virtually, manipulations and the translation of the human parts in the operating room. This work’s aim was to define a methodology, in the form of a workflow, for surgery planning and for designing and manufacturing templates for orthognathic surgery. Along the workflow, the error chain was checked and the maximum error in virtual planning was evaluated. The three-dimensional reconstruction of the mandibular shape and bone fragment movements after segmentation allow complete planning of the surgery and, following the proposed method, the introduction of both the innovative evaluation of the transversal intercondylar distance variation after mandibular arch advancement/set and the possibility of use of standard plates to plan and realize a customized surgery. The procedure was adopted in one clinical case on a patient affected by a class III malocclusion with an associated open bite and right deviation of the mandible with expected good results. Compared with the methods from most recent literature, the presented method introduces two elements of novelty and improves surgery results by optimizing costs and operating time. A new era of collaboration among surgeons and engineer has begun and is now bringing several benefits in personalized surgery.

Radar Interferometric Experimental Reconstruction of Three-Dimensional Displacement Vectors and Mode Shapes for Masonry Constructions

Abstract Radar interferometry is an innovative measurement technique capable of remotely measuring vibrations in terms of displacement history. It apparently allows overcoming some major drawbacks of the plodding and time-consuming conventional accelerometric experimental setup for vibration testing. Indeed, radar interferometry does not require any access to the structure; moreover, its use is simpler and faster. Therefore, this technique appears very appealing for ambient vibration testing on structures, and in particular on architectural heritage masonry constructions. However, radar interferometry is still affected by some relevant limitations, that currently outweigh the advantages. One of the most important limitations is that displacements are measured only along the line-of-sight; therefore, it is not possible to determine the whole three-dimensional displacement vector for moving structural points and, consequently, to reconstruct three-dimensional mode shapes of structures, whose knowledge is crucial for characterizing the dynamic behavior of structures and for detecting eventual damages. In this paper, a theoretical and experimental approach for reconstructing both three-dimensional displacement vectors and mode shapes of masonry constructions is proposed. This approach is based on the simultaneous use of two synchronized radar interferometers, and on the application of a theoretical model assuming a specific kinematical constraint, which is generally plausible in the case of masonry constructions. The proposed approach has been validated through in-situ experimental tests on a masonry bell tower.

Design of thin-structured-light projection system for small-height measurement.

A thin-structured light projection system with simple setup is proposed and applied to small-height measurement. In this system, the design of structured light pattern is carried out on the computer. The structure, color, geometric dimension, motion mode, projection brightness, contrast, and other attributes of the pattern can be edited easily. The optical system with zoom function is designed, which can output thin-structured-light fringe with a minimum width of 10μm. The camera and microscope are used to construct a vision system, which is used to capture the pattern of structured light. The three-dimensional (3D) shape reconstruction is realized by analyzing the pattern of structured light. The results show that this system can project a small width of thin-structured light pattern, which is very suitable for 3D shape reconstruction of microscopic objects with the small-height of tens of microns to hundreds of microns.

Accurate 3D Shape Reconstruction from Single Structured-Light Image via Fringe-to-Fringe Network

Accurate three-dimensional (3D) shape reconstruction of objects from a single image is a challenging task, yet it is highly demanded by numerous applications. This paper presents a novel 3D shape reconstruction technique integrating a high-accuracy structured-light method with a deep neural network learning scheme. The proposed approach employs a convolutional neural network (CNN) to transform a color structured-light fringe image into multiple triple-frequency phase-shifted grayscale fringe images, from which the 3D shape can be accurately reconstructed. The robustness of the proposed technique is verified, and it can be a promising 3D imaging tool in future scientific and industrial applications.