Three-dimensional Object Reconstruction Research Articles

Laboratory X-ray microphotography: a method of inner three-dimensional structure reconstruction of different nature objects

Laboratory X-ray microphotography: a method of inner three-dimensional structure reconstruction of different nature objects

Everted U-Net for 3D scene reconstruction and segmentation

Abstract. The field of data science related to the processing of three-dimensional objects is becoming more and more relevant. After the successes in image processing, the apotheosis of which was the development of generative neural net-works, the intensification of efforts in the direction of three-dimensional data processing looks logical. Although there are now numerous systems for the reconstruction of three-dimensional objects and other processing, almost all of the existing solutions are aimed at working with a single object. The advances in image processing with neural networks have largely been made possible by huge datasets. There are also large datasets available for model training in this area. Freely avail-able datasets such as ShapeNet and ModelNet contain many thousands of different models, allowing for a high diversity of data. However, most of them provide single individual objects, which allows them to be used in tasks involving the processing of a single three-dimensional object, but when working with scenes containing many objects, there is often a problem of finding an appropriate dataset. This work is aimed at solving the problem of reconstruction and segmentation of three-dimensional scenes, as well as generation of datasets for the task of processing scenes from the real world.

Non-Local Means Hole Repair Algorithm Based on Adaptive Block

RGB-D cameras provide depth and color information and are widely used in 3D reconstruction and computer vision. In the majority of existing RGB-D cameras, a considerable portion of depth values is often lost due to severe occlusion or limited camera coverage, thereby adversely impacting the precise localization and three-dimensional reconstruction of objects. In this paper, to address the issue of poor-quality in-depth images captured by RGB-D cameras, a depth image hole repair algorithm based on non-local means is proposed first, leveraging the structural similarities between grayscale and depth images. Second, while considering the cumbersome parameter tuning associated with the non-local means hole repair method for determining the size of structural blocks for depth image hole repair, an intelligent block factor is introduced, which automatically determines the optimal search and repair block sizes for various hole sizes, resulting in the development of an adaptive block-based non-local means algorithm for repairing depth image holes. Furthermore, the proposed algorithm’s performance are evaluated using both the Middlebury stereo matching dataset and a self-constructed RGB-D dataset, with performance assessment being carried out by comparing the algorithm against other methods using five metrics: RMSE, SSIM, PSNR, DE, and ALME. Finally, experimental results unequivocally demonstrate the innovative resolution of the parameter tuning complexity inherent in-depth image hole repair, effectively filling the holes, suppressing noise within depth images, enhancing image quality, and achieving elevated precision and accuracy, as affirmed by the attained results.

A three-dimensional vision measurement method based on double-line combined structured light

In this paper, a structured light vision measurement method using a scanning laser line and a positioning laser line is proposed. The novel method enables the scanning laser plane to slide along a slide rail while maintaining intersection with the positioning laser plane, eliminating the need to determine the scanning direction and moving step. During the measurement process, the laser plane equations need to be recalibrated for each new position, so a real-time calibration method is given. Initially, the geometric barycenter method is employed to detect the subpixel coordinates of the light stripe intersection point. Subsequently, these coordinates are projected into the camera coordinate system using the initial equations of the positioning laser plane. Finally, leveraging the normal information of the initial equation of the scanning laser plane and the three-dimensional coordinates of the light stripe intersection point, the real-time calibration of the scanning laser plane equations can be accomplished. The proposed method enables the three-dimensional reconstruction of objects, and its accuracy is verified through measurements on gauge blocks. Experimental results demonstrate that this method achieves precise and stable three-dimensional reconstruction of object surface shape.

Analysis of Polarization Detector Performance Parameters on Polarization 3D Imaging Accuracy.

Three-dimensional (3D) reconstruction of objects using the polarization properties of diffuse light on the object surface has become a crucial technique. Due to the unique mapping relation between the degree of polarization of diffuse light and the zenith angle of the surface normal vector, polarization 3D reconstruction based on diffuse reflection theoretically has high accuracy. However, in practice, the accuracy of polarization 3D reconstruction is limited by the performance parameters of the polarization detector. Improper selection of performance parameters can result in large errors in the normal vector. In this paper, the mathematical models that relate the polarization 3D reconstruction errors to the detector performance parameters including polarizer extinction ratio, polarizer installation error, full well capacity and analog-to-digital (A2D) bit depth are established. At the same time, polarization detector parameters suitable for polarization 3D reconstruction are provided by the simulation. The performance parameters we recommend include an extinction ratio ≥ 200, an installation error ∈ [-1°, 1°], a full-well capacity ≥ 100 Ke-, and an A2D bit depth ≥ 12 bits. The models provided in this paper are of great significance for improving the accuracy of polarization 3D reconstruction.

Multi-scale Edge-guided Learning for 3D Reconstruction

Single-view three-dimensional (3D) object reconstruction has always been a long-term challenging task. Objects with complex topologies are hard to accurately reconstruct, which makes existing methods suffer from blurring of shape boundaries between multiple components in the object. Moreover, most of them cannot balance learning between global geometric structure information and local detail information. In this article, we propose a multi-scale edge-guided learning network (MEGLN) to utilize the global edge information guiding the network to better capture and recover local details. The goal is to exploit the multi-scale learning strategy to learn global edge information and local details, thus achieving robust 3D object reconstruction. We first design a multi-scale Gaussian difference block (MGDB) to extract global edge geometry features for input images of different scales and adopt the attention mechanism to aggregate the extracted global edge geometry features of different scales. Second, we design a multi-scale feature interaction block (MFIB) to learn local details, which utilizes the multi-scale feature interaction to capture the features of multiple objects or components at multiple scales. The MFIB can learn and capture better as much local detail information as possible under the guidance of global edge information. Finally, we dynamically fuse the predicted probabilities of the MGDB and MFIB to obtain the final predicted result, which makes our MEGLN able to recover 3D shapes with global complex topological structures and rich local details via the multi-scale learning strategy. Extensive qualitative and quantitative experimental results on the ShapeNet dataset demonstrate that our approach achieves competitive performance compared with state-of-the-art methods. Code is available at https://github.com/Ray-tju/MEGLN .

Research on camera calibration optimization method based on improved sparrow search algorithm

Accurate calibration of camera internal parameters is an important step in realizing three-dimensional object reconstruction and target location. Large reprojection error is a frontier problem in the field of camera calibration. A real calibration image dataset and a reprojection error model are established. In combination with the fast iteration of elite opposition-based learning strategy and the population diversity of sparrow search algorithm, the calibration parameters were optimized to improve the accuracy of calibration parameters and reduce the reprojection error. Experimental results demonstrate the effectiveness of the proposed algorithm. It can effectively reduce the reprojection error.

General framework of a two-dimensional complex wavelet for fringe projection profilometry.

Single-shot high-accuracy reconstruction of three-dimensional (3D) objects plays an essential role in many fields. Wavelet-based fringe projection profilometry (WFPP) is a popular single-shot imaging method due to its multiple-resolution analysis and non-contact property. However, the construction of mother wavelets for high-accuracy WFPP is still challenging. In this paper, a general framework of two-dimensional (2D) compact support complex wavelet employing the dual-tree concept combined with the one-dimensional (1D) asymmetric wavelet function is proposed. The constructed wavelet has an asymmetric frequency envelope in the radial direction and concentrated energy in a single quadrant of the frequency domain. A method that the constructed wavelet applied in dual-angle rotation-based 2D WFPP is presented for higher accuracy. The theoretical analysis of the proposed method is presented, and a comparison of frequency characteristics between the constructed wavelet and other popular wavelets is conducted. Computer simulations and experiments have demonstrated that the proposed method provides a better reconstruction of 3D shapes in single-shot WFPP.

SURFACE NORMAL RECONSTRUCTION USING POLARIZATION-UNET

Abstract. Today, three-dimensional reconstruction of objects has many applications in various fields, and therefore, choosing a suitable method for high resolution three-dimensional reconstruction is an important issue and displaying high-level details in three-dimensional models is a serious challenge in this field. Until now, active methods have been used for high-resolution three-dimensional reconstruction. But the problem of active three-dimensional reconstruction methods is that they require a light source close to the object. Shape from polarization (SfP) is one of the best solutions for high-resolution three-dimensional reconstruction of objects, which is a passive method and does not have the drawbacks of active methods. The changes in polarization of the reflected light from an object can be analyzed by using a polarization camera or locating polarizing filter in front of the digital camera and rotating the filter. Using this information, the surface normal can be reconstructed with high accuracy, which will lead to local reconstruction of the surface details. In this paper, an end-to-end deep learning approach has been presented to produce the surface normal of objects. In this method a benchmark dataset has been used to train the neural network and evaluate the results. The results have been evaluated quantitatively and qualitatively by other methods and under different lighting conditions. The MAE value (Mean-Angular-Error) has been used for results evaluation. The evaluations showed that the proposed method could accurately reconstruct the surface normal of objects with the lowest MAE value which is equal to 18.06 degree on the whole dataset, in comparison to previous physics-based methods which are between 41.44 and 49.03 degree.

A Method for High-Resolution Three-Dimensional Reconstruction with Ewald Sphere Curvature Correction from Transmission Electron Images.

A method for three-dimensional reconstruction of objects from defocused images collected at multiple illumination directions in high-resolution transmission electron microscopy is presented. The method effectively corrects for the Ewald sphere curvature by taking into account the in-particle propagation of the electron beam. Numerical simulations demonstrate that the proposed method is capable of accurately reconstructing biological molecules or nanoparticles from high-resolution defocused images under conditions achievable in single-particle electron cryo-microscopy or electron tomography with realistic radiation doses, non-trivial aberrations, multiple scattering, and other experimentally relevant factors. The physics of the method is based on the well-known Diffraction Tomography formalism, but with the phase-retrieval step modified to include a conjugation of the phase (i.e., multiplication of the phase by a negative constant). At each illumination direction, numerically backpropagating the beam with the conjugated phase produces maximum contrast at the location of individual atoms in the molecule or nanoparticle. The resultant algorithm, Conjugated Holographic Reconstruction, can potentially be incorporated into established software tools for single-particle analysis, such as, for example, RELION or FREALIGN, in place of the conventional contrast transfer function correction procedure, in order to account for the Ewald sphere curvature and improve the spatial resolution of the three-dimensional reconstruction.

PROBABILISTIC SILHOUETTE-BASED CLOSE-RANGE PHOTOGRAMMETRY USING A NOVEL 3D OCCUPANCY-BASED RECONSTRUCTION

Abstract. Digital three-dimensional (3D) reconstruction of objects has many applications in computer vision, archaeology, and the entertainment industry. Digital 3D reconstruction can be used to preserve the appearance of valuable historical artifacts; it can be used to track the pose of an object in the images, and it can facilitate object modelling. 3D reconstruction of objects in the past has been achieved using many sensors such as cameras and laser-strip scanners. Monocular camera-based object 3D modelling can be categorized into sparse feature detector/descriptor-based and dense silhouette-based approaches. Feature-based methods identify distinctive features on the objects (captured from many images). In contrast, silhouette-based methods only require a distinguishable boundary between the object and the background. Silhouette-based methods have the advantage that in the controlled setups, a special background can be designed to be distinguishable from the object of interest; therefore, uniquely identifiable textures on the object’s surface are not required. Despite their advantages, silhouette-based probabilistic reconstruction remains a challenge. This article proposes a new probabilistic approach using 3D occupancy grids for the silhouette-based digital reconstruction of an object. The proposed method is designed to be usable with monocular cameras and achieves an accurate reconstruction using only sixteen images. Compared to similar silhouette-based volumetric approaches, the voxels are not discarded immediately during the reconstruction, and the occupancy grid mapping continuously changes the occupancy probability of the voxels with each new image included.

Asymmetric Encryption of Invisible Structured Light 3D Imaging

The research proposes a novel invisible structured light 3D object encryption method. The system projects invisible light on the target plane to achieve three-dimensional object reconstruction. The encryption is conducted using keys from eight stripe patterns and two fingerprint patterns using an elliptic curve encryption algorithm to generate eight corresponding ciphertexts. The three-dimensional object is reconstructed using eight fringe patterns with the elliptic curve decryption algorithm. The proposed method greatly reduces the interference of background light in the system to achieve a better 3D imaging accuracy. The elliptic curve cryptosystem is able to ensure 3D object information transmission security. The simulation results validated the robustness and effectiveness of the proposed scheme. The proposed method has practical security-sensitive applications.

Reconstruction of three-dimensional objects in layered composite structures from multimode orbital angular momentum.

Because of helical phase wavefront distribution, vortex electromagnetic waves are considered to carry more information and additional degrees of freedom than traditional spherical waves. Therefore, a vortex wave carrying orbital angular momentum (OAM) can improve inversion and imaging accuracy. In this work, we revisit the reconstruction of three-dimensional objects in layered composite structures extended with OAM. In forward modeling, the concentric uniform circle array is used to generate electromagnetic vortex beams. To analyze the difference of vortex beams, the electric field radiation pattern and phase pattern distribution of OAM waves with different modes are calculated. Then, the scattered field of layered media illuminated by different OAM beams is determined by the dyadic Green's function and the stabilized biconjugate gradient technique with a fast Fourier transform algorithm. In the inversion, the variational Born iterative method is used to reconstruct targets in layered composite structures, and multiple OAM modes are used to improve the reconstruction results. The numerical results prove that the permittivity of the target can be better reconstructed by using the multiple OAM modes rather than the traditional spherical wave. With the increase of OAM mode number, the reconstructed target parameters are closer to the true value. We expect that our results will provide a better understanding of the OAM and pave the way for the improvement of inversion and optical imaging technology using vortex waves.

Accurate 3D Reconstruction of Dynamic Objects by Spatial-Temporal Multiplexing and Motion-Induced Error Elimination.

Three-dimensional (3D) reconstruction of dynamic objects has broad applications, including object recognition and robotic manipulation. However, achieving high-accuracy reconstruction and robustness to motion simultaneously is a challenging task. In this paper, we present a novel method for 3D reconstruction of dynamic objectS, whose main features are as follows. Firstly, a structured-light multiplexing method is developed that only requires 3 patterns to achieve high-accuracy encoding. Fewer projected patterns require shorter image acquisition time, thus, the object motion is reduced in each reconstruction cycle. The three patterns, i.e. spatial-temporally encoded patterns, are generated by embedding a specifically designed spatial-coded texture map into the temporal-encoded three-step phase-shifting fringes. A temporal codeword and three spatial codewords are extracted from the composite patterns using a proposed extraction algorithm. The two types of codewords are utilized separately in stereo matching: the temporal codeword ensures the high accuracy, while the spatial codewords are responsible for removing phase ambiguity. Secondly, we aim to eliminate the reconstruction error induced by motion between frames abbreviated as motion induced error (MiE). Instead of assuming the object to be static when acquiring the 3 images, we derive the motion of projection pixels among frames. Using the extracted spatial codewords, correspondences between different frames are found, i.e. pixels with the same codewords are traceable in the image sequences. Therefore, we can obtain the phase map at each image-acquisition moment without being affected by the object motion. Then the object surfaces corresponding to all the images can be recovered. Experimental results validate the high reconstruction accuracy and precision of the proposed method for dynamic objects with different motion speeds. Comparative experiments show that the presented method demonstrates superior performance with various types of motion, including translation in different directions and deformation.

Untrained deep learning-based fringe projection profilometry

Deep learning-based fringe projection profilometry (FPP) shows potential for challenging three-dimensional (3D) reconstruction of objects with dynamic motion, complex surface, and extreme environment. However, the previous deep learning-based methods are all supervised ones, which are difficult to be applied for scenes that are different from the training, thus requiring a large number of training datasets. In this paper, we propose a new geometric constraint-based phase unwrapping (GCPU) method that enables an untrained deep learning-based FPP for the first time. An untrained convolutional neural network is designed to achieve correct phase unwrapping through a network parameter space optimization. The loss function of the optimization is constructed by following the 3D, structural, and phase consistency. The designed untrained network directly outputs the desired fringe order with the inputted phase and fringe background. The experiments verify that the proposed GCPU method provides higher robustness compared with the traditional GCPU methods, thus resulting in accurate 3D reconstruction for objects with a complex surface. Unlike the commonly used temporal phase unwrapping, the proposed GCPU method does not require additional fringe patterns, which can also be used for the dynamic 3D measurement.

Accurate 3D Shape Reconstruction from Single Structured-Light Image via Fringe-to-Fringe Network

Accurate three-dimensional (3D) shape reconstruction of objects from a single image is a challenging task, yet it is highly demanded by numerous applications. This paper presents a novel 3D shape reconstruction technique integrating a high-accuracy structured-light method with a deep neural network learning scheme. The proposed approach employs a convolutional neural network (CNN) to transform a color structured-light fringe image into multiple triple-frequency phase-shifted grayscale fringe images, from which the 3D shape can be accurately reconstructed. The robustness of the proposed technique is verified, and it can be a promising 3D imaging tool in future scientific and industrial applications.

ПОРІВНЯЛЬНИЙ АНАЛІЗ МЕТОДІВ ПОШУКУ НАЙБЛИЖЧИХ ТОЧОК НА ЗОБРАЖЕННЯХ ОБ’ЄКТІВ ТЕХНІЧНИХ СИСТЕМ

The subject of the study is the process of mapping the construction of point clouds of technical systems using the algorithm of the nearest points. The goal is to minimize the alignment criterion by converting a set of cloud points Y into a set of cloud points X in a software product that uses an iterative closest point (ICP) algorithm. Objectives: to analyze the properties of input images that contain point clouds; to review the algorithms for identifying and comparing key points; implement a cloud comparison algorithm using the ISR algorithm; consider an example of the algorithm for estimating the approximate values of the elements of mutual orientation; implement software that allows you to compare files that contain point clouds and draw conclusions about the possibility of comparing them. The methods used are: search for points using the algorithm of iterative nearest points, the algorithm for estimating the approximate values of the elements of mutual orientation, the method of algorithm theory for the analysis of file structures STL (standard template library - format template library) format. The following results were obtained. The choice of the ICP algorithm for the task of reconstruction of the object of technical systems is substantiated; the main features of the ISR algorithm are considered; the algorithm of comparison of key points, and also optimization that allows reducing criterion of combination at the reconstruction of three-dimensional objects of technical systems results. Conclusions. The study found that the iterative near-point algorithm is more detailed and accurate when modeling objects. At the same time, this method requires very accurate values and when calculating the degree of proximity, the complexity of calculation by this algorithm increases many times. Whereas the algorithm for estimating the approximate values of the elements of mutual orientation does not require information about the approximate orientation of the point clouds, which simplifies the work and reduces the simulation time. It was found that not all files are comparable. Therefore, the software is implemented, which gives an opinion on the possibility of comparing points in the proposed two files, which contain clouds of points in the structure of the STL format.

One-Shot M-Array Pattern Based on Coded Structured Light for Three-Dimensional Object Reconstruction

Pattern encoding and decoding are two challenging problems in a three-dimensional (3D) reconstruction system using coded structured light (CSL). In this paper, a one-shot pattern is designed as an M-array with eight embedded geometric shapes, in which each 2 × 2 subwindow appears only once. A robust pattern decoding method for reconstructing objects from a one-shot pattern is then proposed. The decoding approach relies on the robust pattern element tracking algorithm (PETA) and generic features of pattern elements to segment and cluster the projected structured light pattern from a single captured image. A deep convolution neural network (DCNN) and chain sequence features are used to accurately classify pattern elements and key points (KPs), respectively. Meanwhile, a training dataset is established, which contains many pattern elements with various blur levels and distortions. Experimental results show that the proposed approach can be used to reconstruct 3D objects.

Three-Dimensional Reconstruction of Welding Pool Surface by Binocular Vision

Current research of binocular vision systems mainly need to resolve the camera’s intrinsic parameters before the reconstruction of three-dimensional (3D) objects. The classical Zhang’ calibration is hardly to calculate all errors caused by perspective distortion and lens distortion. Also, the image-matching algorithm of the binocular vision system still needs to be improved to accelerate the reconstruction speed of welding pool surfaces. In this paper, a preset coordinate system was utilized for camera calibration instead of Zhang’ calibration. The binocular vision system was modified to capture images of welding pool surfaces by suppressing the strong arc interference during gas metal arc welding. Combining and improving the algorithms of speeded up robust features, binary robust invariant scalable keypoints, and KAZE, the feature information of points (i.e., RGB values, pixel coordinates) was extracted as the feature vector of the welding pool surface. Based on the characteristics of the welding images, a mismatch-elimination algorithm was developed to increase the accuracy of image-matching algorithms. The world coordinates of matching feature points were calculated to reconstruct the 3D shape of the welding pool surface. The effectiveness and accuracy of the reconstruction of welding pool surfaces were verified by experimental results. This research proposes the development of binocular vision algorithms that can reconstruct the surface of welding pools accurately to realize intelligent welding control systems in the future.

Next-best-view regression using a 3D convolutional neural network

Automated three-dimensional (3D) object reconstruction is the task of building a geometric representation of a physical object by means of sensing its surface. Even though new single-view reconstruction techniques can predict the surface, they lead to incomplete models, specially, for non-commons objects such as antique objects or art sculptures. Therefore, to achieve the task’s goals, it is essential to automatically determine the locations where the sensor will be placed so that the surface will be completely observed. This problem is known as the next-best-view problem. In this paper, we propose a data-driven approach to address the problem. The proposed approach trains a 3D convolutional neural network (3D CNN) with previous reconstructions in order to regress the position of the next-best-view. To the best of our knowledge, this is one of the first works that directly infers the next-best-view in a continuous space using a data-driven approach for the 3D object reconstruction task. We have validated the proposed approach making use of two groups of experiments. In the first group, several variants of the proposed architecture are analyzed. Predicted next-best-views were observed to be closely positioned to the ground truth. In the second group of experiments, the proposed approach is requested to reconstruct several unseen objects, namely, objects not considered by the 3D CNN during training nor validation. Coverage percentages of up to 90 % were observed. With respect to current state-of-the-art methods, the proposed approach improves the performance of previous next-best-view classification approaches and it is quite fast in running time (3 frames per second), given that it does not compute the expensive ray tracing required by previous information metrics.