3D Reconstruction System Research Articles

Comparison of diagnostic performance in assessing the rewiring position into a jailed side branch between online 3D reconstruction systems version 1.1 and 1.2 derived from optical frequency domain imaging

The three-dimensional reconstruction of optical coherence tomography and optical frequency domain imaging (3D-OCT/OFDI) helps optimize bifurcation percutaneous coronary interventions (PCIs) with side branch (SB) dilatation by identifying the optimal rewiring position. 3D-OCT/OFDI’s diagnostic performance for assessing the rewiring position into a jailed SB is unknown. We retrospectively evaluated the diagnostic performances of a conventional (ver. 1.1) and a new (ver. 1.2) online 3D-OFDI reconstruction system based on an offline 3D reconstruction system’s performance. We analyzed 45 patients’ 52 OFDI pullbacks with main vessel stenting followed by rewiring into a jailed SB for coronary bifurcation lesions. We counted the undetected stent struts in the polygon of confluence as the stent detection performance. We assessed the diagnostic agreement regarding the rewiring position into a jailed SB by the three 3D reconstruction systems. The percentage of undetected struts and the diagnostic agreement of ver.1.2 were significantly better than those of ver.1.1 [5.1 ± 5.1% vs. 30.2 ± 14.2%; p < 0.0001, and 94.2% (49/52) vs. 76.9% (40/52); p = 0.0120]. The new online 3D-OFDI reconstruction system provides better diagnostic performance than the conventional online system for assessing the rewiring position into a jailed SB.

3D Reconstruction system for collaborative scanning based on multiple RGB-D cameras

Current 3D reconstruction systems can easily fall into the problems of insufficient memory and low efficiency when reconstructing complex large scenes, and the reconstructed model drifts a lot at the same time. To solve the above problems, a 3D reconstruction system for collaborative scanning based on multiple RGB-D Cameras (Xtion sensors) is presented. In the case of no pose estimation, this paper can obtain the camera pose initial values of the sequence images through the image acquisition platform that has been calibrated in advance, which can well cope with the problems existing in the current 3D reconstruction system. The main innovations of this paper are as follows: (1) In view of the large amount of human-computer interaction required for the current 3D reconstruction system and the high requirements for data acquisition, a collaborative scanning 3D reconstruction system based on multiple Xiton sensors (RGB-D camera) is developed. The proposed system does not require human-computer interaction and can capture high quality image sequences fully automatically; (2) In view of the problems of memory shortage and low efficiency in current 3D reconstruction systems applied to complex large-scale scenes, this paper proposes a system which can obtain camera poses in advance calibration without the need for camera pose estimation; (3) In terms of camera pose optimization, a segmented bundle adjustment method has been presented to obtain the high-precision camera poses. A large number of experiments demonstrate that the proposed system can effectively solve the existing problems. Meanwhile, the proposed reconstruction system can obtain high-precision 3D models suitable for various complex large scenes, which can be widely used in human-computer interaction, virtual reality and other fields.

Three-dimensional rapid flame chemiluminescence tomography via deep learning.

Flame chemiluminescence tomography (FCT) plays an important role in combustion monitoring and diagnostics due to the easy implementation and non-intrusion. However, on account of the high data throughput and the inefficiency of the conventional iteration methods, the 3D reconstructions in FCT are typically conducted off-line and time-consuming. In this work, we present a 3D rapid FCT reconstruction system based on convolutional neural networks (CNN) model for practical combustion measurement, which has the ability to reconstruct 3D flame distribution rapidly after training process. First, the numerical simulation has been performed by creating three cases of phantoms which are designed to mimic the 3D conical flame. Next, after the evaluation of loss function and training time, the optimal CNN architecture has been determined and certificated quantitatively. Finally, a real time FCT system consisting of 12 color CCD cameras is realized and multispectral separation algorithm is adopted to extract CH* and C2* components. Certificated by practical measurements testing, the proposed CNN model is able to reconstruct 3D flame structure from real time captured projections with credible accuracy and structure similarity. Furthermore, compared with conventional iteration reconstruction method, the proposed CNN model shows better performance on obviously improving reconstruction speed and it is expected to achieve 3D rapid monitoring of flames.

An Analytical Tension Model for Continuum Robots with n Generally Positioned Tendons

The estimation of tension loads in multi-tendon continuum robots or catheters plays an important role not only in the design process but also in the control algorithm to avoid slack. An analytical tension loading model is developed that, for any given beam configuration within the workspace, calculates tendon tensions in [Formula: see text]-tendon continuum robots with general tendon positioning. The model accounts for the bending and axial compliance of the manipulator as well as tendon compliance. A 6-tendon continuum robot integrated with a stereo vision-based 3D reconstruction system is utilized to experimentally validate the proposed analytical model in open-loop control architecture. The proposed model demonstrates around 95% accuracy in estimating tendon tensions in a continuum robot with general tendon positioning and axial stretch in its tendons for all of the trials and experiments.

Real-Time 3D Motion Tracking and Reconstruction System Using Camera and IMU Sensors

Jointly using the data of visual and inertial sensors to achieve higher accuracy and robustness is a problem of high computational complexity. In this work, this paper presents a 3D motion tracking and reconstruction system on a mobile device using camera and inertial measurement unit (IMU) sensors. The mobile device has very small active infrared projection depth sensors with high-performance IMU and wide field of view cameras. We utilize visual-inertial odometry (VIO) to integrate visual and IMU data. However, traditional VIO method is too complex to apply to a mobile device with limited computing resources in real time. We employ sliding window filter (SWF) for the proposed system to achieve accurate 3D motion tracking and high-quality reconstruction models in real time based on delayed state marginalization. Moreover, we also extensively evaluate in the dataset and real world. Many experiments show the accurate trajectories and high quality 3D reconstruction models by the proposed system. To the end, the qualitative and quantitative experimental results indicate the proposed system based on SWF has much better performance than existing methods in motion tracking and 3D reconstruction models.

Minimal vision system for linear camera self-calibration and 3D reconstruction using translation platform

Camera calibration and 3D reconstruction are two crucial steps in computer vision. With the progress of robot and autonomous rover systems, zooming cameras are widely applied, where online calibration and 3D reconstruction are becoming more and more important. This paper proposed a minimal vision system that consists of a translation platform and an un-calibrated camera mounted on. With this minimal system, we can linearly calibrate the camera online and reconstruct 3D structures from three un-calibrated images by utilizing translation motions. The two images generated by translating the camera to allow the recovery of scene depths. Depths are then analyzed by error analysis models and are utilized to determine the infinite homography between the third image and any of the two translated images. The intrinsic parameters are then calibrated linearly from the computed infinite homography. Camera motion estimation and 3D reconstruction are then readily determined from the intrinsic calibration results. We also proposed a two-step optimization method to refine both the calibration and 3D reconstruction results by minimizing the overall back-projection errors across the three images within a tiny-scale bundle adjustment framework. The proposed method has been validated with both simulation and real image data. The results demonstrate that the proposed minimal linear system can solve the online camera self-calibration problem and the reconstruction of 3D structure. The paper suggests a framework of minimal linear system by using an un-calibrated camera and a low-cost translation platform to address the problems of linear self-calibration of camera, motion estimation, 3D reconstruction, and optimization, in a practical, easy, and accurate way.

VISION BASED AUTOMATED ANTHROPOLOGICAL MEASUREMENTS AND ANALYSIS

Abstract. Modern techniques of optical 3D measurements such as photogrammetry, computer tomography, laser 3D scanning provide new possibilities for acquiring accurate 2D and 3D data of high resolution, thus creating new conditions for anthropological data analysis. Traditional anthropological manual point measurements can be substituted by analysis of accurate textured 3D models, which allow to retrieve more information about studied object and easily to share data for independent analysis. The paper presents the vision-based techniques for anthropological measurement and investigation needed in various applications, such as morphometric analysis, craniofacial identification, face approximation and other. Photogrammetric methods and their practical implementation in the automatic system for accurate digital 3D reconstruction of anthropological objects are described.

Calibration of a panoramic 3D reconstruction system

Owing to their large field of view, a catadioptric imaging system is widely used in several research areas, including robot navigation, surveillance, and three-dimensional (3D) reconstruction. The modelling and calibration of these systems allow the accurate measurement of objects. This paper presents a calibration method for a panoramic 3D reconstruction system based on pattern projection composed of two modules. The first module, which contains a charged coupled device camera and a parabolic mirror allows the acquisition of catadioptric images. The second module is a catadioptric projection system made of a light projector and a parabolic mirror that generates a projected pattern over the target for reconstruction purposes. Both the modules are calibrated independently. The method achieves a reprojection error of 0.6 pixels in the camera module and 1.2 pixels in the catadioptric projection system. To evaluate the performance of the proposed method, the results show 3D reconstructions of multiple objects with tolerances around 1.3 mm.

Real-time 3D reconstruction method using massive multi-sensor data analysis and fusion

This paper proposes a method to reconstruct three-dimensional (3D) objects using real-time fusion and analysis of multiple sensor data. This paper attempts to create a realistic 3D visualization with which a remote pilot can intuitively control a remote unmanned robot by utilizing the characteristics of massive sensor data. The 3D reconstruction system proposed in this paper is comprised of 3D and two-dimensional (2D) data segmentation method, a 3D reconstruction method applied to each object, and a projective texture mapping method. Specifically, we propose applying both a 2D region extraction method and a 3D mesh modeling method to each object. The proposed schemes are implemented as a real-time application to verify real-time performance. This paper proves that 3D meshes can be modeled in real time by using the proposed method. The proposed method allows the remote control of a robot for real-time 3D rendering of remote scenes, which is essential for various tasks in areas that cannot be easily accessed by humans.

3D image reconstruction from different image formats using marching cubes technique

Structure from motion (SFM) is the problem of reconstructing the 3D image from 2D images. The main problem of 3D reconstruction is the quality of the 3D image that depends on the number of 2D slices input to the system. A large number of 2D slices may lead to high processing time. This paper introduces a new model to reconstruct the 3D image from any 2D image by using marching cubes algorithm. We use the LabVIEW program to build the system and use the Biomedical Toolkit to read and registered any 2D images. Our main goal is to implement the 3D reconstruction system to produce a high-quality 3D image with a minimum number of 2D slices and to decrease the execution time as possible. We apply our system on two datasets; all the experimental results have proved the efficiency and effectiveness of this system in 3D image reconstruction from any 2D image type. As shown in results, changing iso_value, image type and a number of images, affects the quality of 3D image reconstruction, and the processing time.

High-Quality 3D Reconstruction With Depth Super-Resolution and Completion

The 3D reconstruction is an important topic in computer vision with many applications, such as robotics and augmented reality. Since the raw depth images captured by consumer RGB-D cameras are often low resolution (LR), noisy, and incomplete. How to obtain high-quality 3D models with a consumer RGB-D camera is still a challenge for the existing systems. In this paper, we propose a new depth super-resolution and completion method implemented in a deep learning framework and build a high-quality 3D reconstruction system. We first improve the resolution of LR depth image with a depth super-resolution network and remove the outliers in high-resolution (HR) depth image based on gradient saliency. To further enhance the quality of HR depth image with the guide of HR color image, we learn surface normal and occlusion boundary images from the corresponding HR color image through two deep fully convolutional networks. In particular, the blurriness of HR color image is also detected and pixel-wise quantized. Finally, we obtain a completed HR depth image by optimizing the HR depth image with the surface normal, occlusion boundary, and color image blurriness. We have carried out qualitative and quantitative evaluations with baseline methods on public datasets. The experimental results demonstrate that our method has better performance both on single depth image enhancement and 3D reconstruction.

Compact omnidirectional multi-stereo vision system for 3D reconstruction.

Omnidirectional stereovision is beneficial in wide applications due to its large field of view. Traditional implementation of such a system employs two separate cameras looking at separate mirrors to achieve 3D stereo reconstruction. In this paper, a compact omnidirectional multi-stereo system that consists of only one camera and multiple mirrors is proposed. The novelty lies in the designing of a principal mirror above the array of sub-mirrors to achieve multiple omnidirectional stereo pairs that have appropriate baselines and multiple viewpoints in a compact manner. The system's imaging projection and error propagation model are presented. Experiments based on simulation and real data verify the effectiveness of our system.

Using Laser Measuring and SFM Algorithm for Fast 3D Reconstruction of Objects

Effective measurement of the reflective or transparent surface of an object has always been a disadvantage in laser scanning modeling. We propose a fast and complete three-dimensional (3D) reconstruction method for small static objects using laser scanning and the structure from motion (SFM) algorithm. Meanwhile, a complete reconstruction workflow is designed and a multi-angle 3d reconstruction system is set up. To generate the complete point cloud model of the object, the SFM algorithm is used to reconstruct the surface part of the object, the data for which cannot be obtained by the laser measuring instrument. The experimental results show that this method not only improves the speed, accuracy, integrity, and visual effect of 3D reconstruction of small objects, but also extends the scope of 3D reconstruction of laser measurement.

Underwater Target Detection and 3D Reconstruction System Based on Binocular Vision.

To better solve the problem of target detection in marine environment and to deal with the difficulty of 3D reconstruction of underwater target, a binocular vision-based underwater target detection and 3D reconstruction system is proposed in this paper. Two optical sensors are used as the vision of the system. Firstly, denoising and color restoration are performed on the image sequence acquired by the vision of the system and the underwater target is segmented and extracted according to the image saliency using the super-pixel segmentation method. Secondly, aiming to reduce mismatch, we improve the semi-global stereo matching method by strictly constraining the matching in the valid target area and then optimizing the basic disparity map within each super-pixel area using the least squares fitting interpolation method. Finally, based on the optimized disparity map, triangulation principle is used to calculate the three-dimensional data of the target and the 3D structure and color information of the target can be given by MeshLab. The experimental results show that for a specific size underwater target, the system can achieve higher measurement accuracy and better 3D reconstruction effect within a suitable distance.

Automatic reconstruction method for large scene based on multi-site point cloud stitching

At present, the three-dimensional (3D) reconstruction system is based on hand-held camera which causes certain problems, such as needing for a large number of human-computer interactions, demanding on high quality image data and insufficient accuracy of 3D reconstruction. Aiming at these problems, a fully automatic reconstruction of large scene based on multi-site point cloud stitching is presented in this paper. The proposed method uses the Kinect sensor for image acquisition. Since, there are multiple sites in the room, each site of image data is processed using a separate model in order to get good 3D point cloud data. Then, these sites are used to constitute a local area network, and the method of bundle block adjustment is employed to stitch each site point cloud data. The proposed method achieves a high-degree automation and provides a high-precision 3D reconstruction which has two main advantages: (i) the reconstruction process is fully automatic, without any human-computer interaction; (ii) the automatic reconstruction is robust. Experimental results show that proposed automatic reconstruction method is convenient and practical, and can provide better 3D reconstruction model than commonly used methods. Moreover, it can be applied to virtual reality shopping malls, Virtual Reality (VR) and other fields.

Structured-Light Based 3D Reconstruction System for Cultural Relic Packaging.

The non-contact three-dimensional measurement and reconstruction techniques have played a significant role in the packaging and transportation of precious cultural relics. This paper develops a structured light based three-dimensional measurement system, with a low-cost for cultural relics packaging. The structured light based system performs rapid measurements and generates 3D point cloud data, which is then denoised, registered and merged to achieve accurate 3D reconstruction for cultural relics. The multi-frequency heterodyne method and the method in this paper are compared. It is shown that the relative accuracy of the proposed low-cost system can reach a level of 1/1000. The high efficiency of the system is demonstrated through experimental results.

On-Line 3D reconstruction based on laser scanning for robot machining of large complex components

The digital model is the foundation of intelligent manufacturing based on robot machining. In the intelligent manufacturing of large complex components in the fields of aerospace, energy equipment, shipbuilding, etc., it is often necessary to perform online 3D reconstruction of these large-sized components. In response to this demand, this paper designs a large-scale on-line 3D reconstruction system based on laser scanning principle. A laser scanner is used to generate the contour of the component accurately. Then, a series of contour sequences are generated by the movement perpendicular to the direction of the contour plane. These contour sequences are processed by algorithms to generate the 3D point cloud model of the large complex component. The results show that the system can generate the 3D point cloud model of large-scale complex components in an on-line manner and provide the basis for the detection of the components before machining and the machining planning in some cases.

Simulation of Multi-Camera Indoor and Outdoor 3D Scanner

Photogrammetry allows a three-dimensional reconstruction of the object based on its multiple photography’s. The quality of the reconstruction result depends mostly on the gloss, the diversity of the texture, the lighting conditions, the quality of the camera calibration and the shape of the object. The article presents the results of a simulation of a multi-camera reconstruction system, for the needs of designing indoor and outdoor 3D scanner. The 3D reconstruction system works by simultaneously taking photographs of cameras located around the object. The simulation was created to investigate the optimal distribution of cameras and projectors casting a pattern that increases the number of characteristic points on the surface of the object. The impact of background removal in images on the reconstruction result as well as the texture quality of the object depending on the resolution and distance of the cameras from the object were also investigated. The graphic engine used to create the simulation also allows testing of impact of various conditions. The presented results prove that the parameters of the system structure, such as the placement of cameras, projectors, the selection of patterns projected by the projectors are important and their values can be determined at the stage of system simulation. The acquired results are promising and will be further investigated.

Dense tracking, mapping and scene labeling using a depth camera

We present a system for dense tracking, 3D reconstruction, and object detection of desktop-like environments, using a depth camera; the Kinect sensor. The camera is moved by hand meanwhile its pose is estimated, and a dense model, with evolving color information of the scene, is constructed. Alternatively, the user can couple the object detection module (YOLO: you only look once [1]) for detecting and propagating to the model information of categories of objects commonly found over desktops, like monitors, keyboards, books, cups, and laptops, getting a model with color associated to object categories. The camera pose is estimated using a model-to-frame technique with a coarse-to-fine iterative closest point algorithm (ICP), achieving a drift-free trajectory, robustness to fast camera motion and to variable lighting conditions. Simultaneously, the depth maps are fused into the volumetric structure from the estimated camera poses. For visualizing an explicit representation of the scene, the marching cubes algorithm is employed. The tracking, fusion, marching cubes, and object detection processes were implemented using commodity graphics hardware for improving the performance of the system. We achieve outstanding results in camera pose, high quality of the model’s color and geometry, and stability in color from the detection module (robustness to wrong detections) and successful management of multiple instances of the same category.

A Real-Time 3D Perception and Reconstruction System Based on a 2D Laser Scanner

This paper presents a real-time and low-cost 3D perception and reconstruction system which is suitable for autonomous navigation and large-scale environment reconstruction. The 3D mapping system is based on a rotating 2D planar laser scanner driven by a step motor, which is suitable for continuous mapping. However, for such a continuous mapping system, the challenge is that the range measurements are received at different times when the 3D LiDAR is moving, which will result in big distortion of the local 3D point cloud. As a result, the errors in motion estimation can cause misregistration of the resulting point cloud. In order to continuously estimate the trajectory of the sensor, we first extract feature points from the local point cloud and then estimate the transformation between current frame to local map to get the LiDAR odometry. After that, we use the estimated motion to remove the distortion of the local point cloud and then register the undistorted local point cloud to the global point cloud to get accurate global map. Finally, we propose a coarse-to-fine graph optimization method to minimize the global drift. The proposed 3D sensor system is advantageous due to its mechanical simplicity, mobility, low weight, low cost, and real-time estimation. To validate the performance of the proposed system, we carried out several experiments to verify its accuracy, robustness, and efficiency. The experimental results show that our system can accurately estimate the trajectory of the sensor and build a quality 3D point cloud map simultaneously.