Stereo Technique Research Articles

Simultaneous shape and camera-projector parameter estimation for 3D endoscopic system using CNN-based grid-oneshot scan.

For effective in situ endoscopic diagnosis and treatment, measurement of polyp sizes is important. For this purpose, 3D endoscopic systems have been researched. Among such systems, an active stereo technique, which projects a special pattern wherein each feature is coded, is a promising approach because of simplicity and high precision. However, previous works of this approach have problems. First, the quality of 3D reconstruction depended on the stabilities of feature extraction from the images captured by the endoscope camera. Second, due to the limited pattern projection area, the reconstructed region was relatively small. In this Letter, the authors propose a learning-based technique using convolutional neural networks to solve the first problem and an extended bundle adjustment technique, which integrates multiple shapes into a consistent single shape, to address the second. The effectiveness of the proposed techniques compared to previous techniques was evaluated experimentally.

Improved Visual Inspection Through 3D Image Reconstruction of Defects Based on the Photometric Stereo Technique.

Visual inspections of nuclear power plant (NPP) reactors are important for understanding current NPP conditions. Unfortunately, the existing visual inspection methods only provide limited two-dimensional (2D) information due to a loss of depth information, which can lead to errors identifying defects. However, the high cost of developing new equipment can be avoided by using advanced data processing technology with existing equipment. In this study, a three-dimensional (3D) photometric stereo (PS) reconstruction technique is introduced to recover the lost depth information in NPP images. The system uses conventional inspection equipment, equipped with a camera and four light-emitting diodes (LEDs). The 3D data of the object surface are obtained by capturing images under multiple light sources oriented in different directions. The proposed method estimates the light directions and intensities for various image pixels in order to reduce the limitation of light calibration, which results in improved performance. This novel technique is employed to test specimens with various defects under laboratory conditions, revealing promising results. This study provides a new visual inspection method for NPP reactors.

Practical BRDF reconstruction using reliable geometric regions from multi-view stereo

In this paper, we present a practical method for reconstructing the bidirectional reflectance distribution function (BRDF) from multiple images of a real object composed of a homogeneous material. The key idea is that the BRDF can be sampled after geometry estimation using multi-view stereo (MVS) techniques. Our contribution is selection of reliable samples of lighting, surface normal, and viewing directions for robustness against estimation errors of MVS. Our method is quantitatively evaluated using synthesized images and its effectiveness is shown via real-world experiments.

Three-dimensional image-based approach for imperfect structures surface modeling

Great attention has been given, in the scientific literature, to the effect of initial imperfections on the structural behavior of experimental samples under compression loads. Geometrical reconstruction of the as-built surface is therefore required to allow accurate numerical modeling. High-precision systems for inverse engineering are expensive, and most of the times geometrical imperfections are rather complex to be described without high computational efforts. In this paper, an image-based approach to model the surface of imperfect structures using open-source software and a common digital camera is presented. The proposed approach aims to generate high-quality surfaces from real imperfect structures, by employing the surface-from-motion and multi-view stereo techniques. A controlled frame capture is introduced to decrease both the computational effort and number of repeated correspondences. The surface fit is then computed by meshing the dense cloud of points and adjusting several surface regions describing the samples’ profiles. The procedure is illustrated by using two damaged tubular member samples reconstructed by the proposed approach. Then, resulted geometries are verified by comparing measures from a three-dimensional high-precision laser scanner and a common mechanical procedure. Finally, a comparison of three-dimensional mappings of a ship panel with proposed approach and commercial photogrammetry software is performed. Verification results refer to the possibility of prototyping geometry surfaces from real imperfect samples by using inexpensive hardware and public domain software tools with acceptable level of accuracy.

MARESye: A hybrid imaging system for underwater robotic applications

This article presents an innovative hybrid imaging system that provides dense and accurate 3D information from harsh underwater environments. The proposed system is called MARESye and captures the advantages of both active and passive imaging methods: multiple light stripe range (LSR) and a photometric stereo (PS) technique, respectively. This hybrid approach fuses information from these techniques through a data-driven formulation to extend the measurement range and to produce high density 3D estimations in dynamic underwater environments. This hybrid system is driven by a gating timing approach to reduce the impact of several photometric issues related to the underwater environments such as, diffuse reflection, water turbidity and non-uniform illumination. Moreover, MARESye synchronizes and matches the acquisition of images with sub-sea phenomena which leads to clear pictures (with a high signal-to-noise ratio). Results conducted in realistic environments showed that MARESye is able to provide reliable, high density and accurate 3D data. Moreover, the experiments demonstrated that the performance of MARESye is less affected by sub-sea conditions since the SSIM index was 0.655 in high turbidity waters. Conventional imaging techniques obtained 0.328 in similar testing conditions. Therefore, the proposed system represents a valuable contribution for the inspection of maritime structures as well as for the navigation procedures of autonomous underwater vehicles during close range operations.

Evaluating earthquake-related ground failure mapping by combined traditional and modern methods

Abstract. A wide range of ground failure such as earthquake faulting (surface rupture), landslides, and liquefaction occur after a large earthquake. In this study, in order to rapidly determine the distribution of failure over a wide area after an earthquake, we combined traditional methods such as aerial photo interpretation and modern methods such as unmanned aerial vehicle (UAV) or interferometric synthetic aperture radar (InSAR) techniques. Moreover, elevation variations obtained using DEMs and the structure from motion and multi-view stereo (SfM-MVS) technique were employed to understand local ground deformation factors, such as reclaimed valley deformation. Using ortho-mosaic images, surface fissures caused by the 2016 Kumamoto Earthquake in Japan were rapidly interpreted and mapped, which enabled early interpretation of the ground failure situation. Furthermore, surface displacement properties extracted from SAR interferograms allowed for more advanced earthquake fault detection; surface displacement associated with liquefaction was also identified from SAR interferograms. In addition, InSAR was used to detect reclaimed valley deformation. Comparing this with the reclaimed valley distribution map created by the SfM-MVS technique improved our understanding of this phenomenon. However, many of these techniques require large amounts of manpower and time and can be influenced by differences in analyst skill level. In future, the development of mechanically automated ground failure identification will improve earthquake disaster responses.

LAVA DOME CHANGES DETECTION AT AGUNG MOUNTAIN DURING HIGH LEVEL OF VOLCANIC ACTIVITY USING UAV PHOTOGRAMMETRY

Abstract. Lava dome changes detection during increasingly high volcanic activity are essential for hazard assessment purposes. However, it is challenging to conduct direct field measurement due to safety reason. Here, we investigate the lava dome changes of Mount Agung in Indonesia during the highest level of volcanic activity. On 22 September 2017, the rumbling and seismic activity in this volcano started increasing to the highest level for a period of time. We afterwards collected image data at lava dome area by using UAV over this time period. To accomplish the goal of change detection, we assembled and developed a fixed-wing UAV platform, i.e. Buffalo FX-79 to acquire images of Mount Agung whose elevation is roughly 3,142 m above sea level. We acquired the UAV images on two dates, i.e. Oct 19 and Oct 21 of 2017. Due to an exclusion zone surround the volcano, we could only operate the UAV at 20 km distance from the crater. With these data set, we produced three-dimensional point clouds, high-resolution Digital Elevation Model and orthophoto by using Structure from Motion (SfM) and Multi View Stereo (SfM-MVS) technique with Photoscan Pro software. From orthophoto data, we found two fluid areas at the crater's surface in NE direction (4,375.9 sq-m) and SE direction (3,749.8 sq-m). We also detected a fumarole which emitted steam and gases in the eastern part that continued for several days. In order to reveal the changes in lava dome surface, we used DEM to create cross-section profile. After that, we applied cloud to cloud comparison (C2C) algorithm to calculate the difference of lava dome based on two data set of point clouds and compared it with interferometric result from Sentinel-1A data. The data from the Sentinel-1A satellite (15 Oct – 27 Oct 2017) were processed to obtain the interferogram image of Mount Agung. This research therefore demonstrates a potential method to detect lava dome changes during high level of volcanic activity with photogrammetric methods by using UAV images. Within only two days the data were successfully acquired. From the DEM data and cross-section profile between two data set, we noticed that no significant surface change was found around the lava dome surface. Moreover, we also found that there was no significant lava dome changes and vertical displacement during these two time periods as the point cloud comparison and distance result. The average of difference distance is 2.27 cm with a maximal and minimal displacement of 255 cm and 0.37 cm respectively. This result was then validated by using InSAR Sentinel that showed small displacement, i.e 6.88 cm. It indicated that UAV photogrammetry showed a good performance to detect surface changes in centimeter fraction.

SHALLOW WATER BATHYMETRY MAPPING FROM UAV IMAGERY BASED ON MACHINE LEARNING

Abstract. The determination of accurate bathymetric information is a key element for near offshore activities, hydrological studies such as coastal engineering applications, sedimentary processes, hydrographic surveying as well as archaeological mapping and biological research. UAV imagery processed with Structure from Motion (SfM) and Multi View Stereo (MVS) techniques can provide a low-cost alternative to established shallow seabed mapping techniques offering as well the important visual information. Nevertheless, water refraction poses significant challenges on depth determination. Till now, this problem has been addressed through customized image-based refraction correction algorithms or by modifying the collinearity equation. In this paper, in order to overcome the water refraction errors, we employ machine learning tools that are able to learn the systematic underestimation of the estimated depths. In the proposed approach, based on known depth observations from bathymetric LiDAR surveys, an SVR model was developed able to estimate more accurately the real depths of point clouds derived from SfM-MVS procedures. Experimental results over two test sites along with the performed quantitative validation indicated the high potential of the developed approach.

A spatiotemporal data acquisition toolkit for volume estimation tools in precision agriculture

SummaryA data acquisition system for precision agriculture is presented. The system can collects various time‐series data on target plants and their environmental facts. The designed system consists of a server and multiple intra‐networked stations and includes a software toolkit that has two independent but synchronized data acquisition processes. The one is for measurements on environmental facts around a target plant and the other is for volume estimation of the target plant adopting multi‐view stereo techniques. An analysis on sampling rate and data acquisition experiments for testing purposes were performed. A data filtering method based on the fast Fourier transform is applied to analyze the noise immunity of the collected data. For the volume estimation, three different methods, ie, method‐I using 2‐D projection on slices, method‐II using polyhedrons on voxels, and method‐III using enclosed surfaces, were devised to handle various plant types and examined for three different type of target plants. The volume estimation errors were measured in range of 6 ∼ 30% in the experiments.

Simple evaluation of the degradation state of cultural heritage based on multi-view stereo

In this study, degradation of cultural heritage is discussed based on the comparison between the present shape of the Wareishi Rock Cliff Sculpture and its replica made 30 years ago through simple three-dimensional measurement using a multi-view stereo. The Wareishi Rock Cliff Sculpture, an important cultural property of Hiroshima Prefecture, is located near a shoreline. Local people often highlight that the sculpture seems to have recently been suffering from rapid degradation because of the direct impact of tidal ebb and flow. The present three-dimensional shape of this sculpture was investigated through multi-view stereo technique to accurately estimate the current degradation state. The same investigation was performed on the replica of the sculpture made in 1986 using a mold and exhibited in the Hiroshima Prefectural History Museum. The difference between the present shape of the sculpture and that of the replica revealed through the above investigation can be regarded as degradation of the sculpture that has occurred over the previous 30 years. The result is as follows. The difference was detected mainly on the middle part of the sculpture, which is a belt-shaped portion parallel to the sea surface that is ~ 20 cm high. The total area where more than 3 mm of displacement was detected constitutes ~ 0.56% of the whole measured area. Accordingly, the degradation of the Wareishi Rock Cliff Sculpture does not seem to be recently accelerated but seems to be progressing at a regular pace since it was carved, especially in relation to the sea level rise. If the degradation extends up to the face of the sculpture in the future, some countermeasures may be required. The method used in this study will contribute to the conservation of cultural heritage through evaluating the degradation state of the heritage precisely and to the study of weathering by detecting long-term weathering.

Target Three-Dimensional Reconstruction From the Multi-View Radar Image Sequence

Target three-dimensional (3D) reconstruction is a hot topic and also a challenge in remote sensing applications. In this paper, a new reconstruction algorithm is proposed to reconstruct the 3D surface of the stable attitude target from its multi-view radar image sequence. Uniform explicit expression of the radar and optical imaging geometries is derived to bridge the 3D target structure and these two sorts of observation images. In this way, the visual hull of the target is reconstructed by exploiting the multi-view stereo techniques to the silhouette information extracted from the radar image sequence. Meanwhile, the target absolute attitude is also determined. Furthermore, we analyze the primary difficulty of the method induced from the limited radar observation view in a typical application, the 3D reconstruction of an in-orbit satellite. Then, an extended algorithm is proposed with the feature fusion of the radar and optical images to achieve dramatic performance enhancement of the reconstruction in this condition. The feasibility of the proposed algorithm is confirmed in the experiment part, and some conclusions are drawn to guide the future work about extended applications of the proposed algorithm as well.

Practical SVBRDF acquisition of 3D objects with unstructured flash photography

Capturing spatially-varying bidirectional reflectance distribution functions (SVBRDFs) of 3D objects with just a single, hand-held camera (such as an off-the-shelf smartphone or a DSLR camera) is a difficult, open problem. Previous works are either limited to planar geometry, or rely on previously scanned 3D geometry, thus limiting their practicality. There are several technical challenges that need to be overcome: First, the built-in flash of a camera is almost colocated with the lens, and at a fixed position; this severely hampers sampling procedures in the light-view space. Moreover, the near-field flash lights the object partially and unevenly. In terms of geometry, existing multiview stereo techniques assume diffuse reflectance only, which leads to overly smoothed 3D reconstructions, as we show in this paper. We present a simple yet powerful framework that removes the need for expensive, dedicated hardware, enabling practical acquisition of SVBRDF information from real-world, 3D objects with a single, off-the-shelf camera with a built-in flash. In addition, by removing the diffuse reflection assumption and leveraging instead such SVBRDF information, our method outputs high-quality 3D geometry reconstructions, including more accurate high-frequency details than state-of-the-art multiview stereo techniques. We formulate the joint reconstruction of SVBRDFs, shading normals, and 3D geometry as a multi-stage, iterative inverse-rendering reconstruction pipeline. Our method is also directly applicable to any existing multiview 3D reconstruction technique. We present results of captured objects with complex geometry and reflectance; we also validate our method numerically against other existing approaches that rely on dedicated hardware, additional sources of information, or both.

Structural analysis of clastic dikes using Structure from Motion - Multi-View Stereo: a case-study in the Paraná Basin, southeastern Brazil

This work presents the development of a three-dimensional (3D) model of an outcrop of the Corumbataí Formation (Permian, Paraná Basin, Brazil) using Structure from Motion - Multi-View Stereo (SfM-MVS) technique in order to provide a structural analysis of clastic dikes cutting through siltstone layers. While traditional photogrammetry requires the user to input a series of parameters related to the camera orientation and its characteristics (such as focal distance), in SfM-MVS the scene geometry, camera position and orientations are automatically determined by a bundle adjustment, an iterative procedure based on a set of overlapping images. It is considered a low-cost technique in terms of hardware and software, also being able to provide point density and accuracy on par to the ones obtained withTerrestrial Laser Scanning. The results acquired on this research have good agreement with previous works, yielding a NNW main orientation for the dikes measured in the field and on the 3D model. The development of this work showed that SfM-MVS use and practice on geosciences still needs more studies on the optimization of the involved parameters (such as camera orientation, image overlap and angle of illumination), which, when accomplished, will result in less processing time and more accurate models.

Noncontact speckle contrast diffuse correlation tomography of blood flow distributions in tissues with arbitrary geometries.

A noncontact electron multiplying charge-coupled-device (EMCCD)-based speckle contrast diffuse correlation tomography (scDCT) technology has been recently developed in our laboratory, allowing for noninvasive three-dimensional measurement of tissue blood flow distributions. One major remaining constraint in the scDCT is the assumption of a semi-infinite tissue volume with a flat surface, which affects the image reconstruction accuracy for tissues with irregular geometries. An advanced photometric stereo technique (PST) was integrated into the scDCT system to obtain the surface geometry in real time for image reconstruction. Computer simulations demonstrated that a priori knowledge of tissue surface geometry is crucial for precisely reconstructing the anomaly with blood flow contrast. Importantly, the innovative integration design with one single-EMCCD camera for both PST and scDCT data collection obviates the need for offline alignment of sources and detectors on the tissue boundary. The in vivo imaging capability of the updated scDCT is demonstrated by imaging dynamic changes in forearm blood flow distribution during a cuff-occlusion procedure. The feasibility and safety in clinical use are evidenced by intraoperative imaging of mastectomy skin flaps and comparison with fluorescence angiography.

A Novel Iterative Water Refraction Correction Algorithm for Use in Structure from Motion Photogrammetric Pipeline

Photogrammetry using structure from motion (SfM) techniques has evolved into a powerful tool for a variety of applications. Nevertheless, limits are imposed when two-media photogrammetry is needed, in cases such as submerged archaeological site documentation. Water refraction poses a clear limit on photogrammetric applications, especially when traditional methods and standardized pipelines are followed. This work tries to estimate the error introduced to depth measurements when no refraction correction model is used and proposes an easy to implement methodology in a modern photogrammetric workflow dominated by SfM and multi-view stereo (MVS) techniques. To be easily implemented within current software and workflow, this refraction correction approach is applied at the photo level. Results over two test sites in Cyprus against reference data suggest that despite the assumptions and approximations made the proposed algorithm can reduce the effect of refraction to two times the ground pixel size, regardless of the depth.

Low-cost single-pixel 3D imaging by using an LED array.

We propose a method to perform color imaging with a single photodiode by using light structured illumination generated with a low-cost color LED array. The LED array is used to generate a sequence of color Hadamard patterns which are projected onto the object by a simple optical system while the photodiode records the light intensity. A field programmable gate array (FPGA) controls the LED panel allowing us to obtain high refresh rates up to 10 kHz. The system is extended to 3D imaging by simply adding a low number of photodiodes at different locations. The 3D shape of the object is obtained by using a non-calibrated photometric stereo technique. Experimental results are provided for an LED array with 32 × 32 elements.

Optimization of the photometric stereo method for measuring pavement texture properties

The texture of the pavement surface, in addition to the traffic safety, plays a major role in the driving comfort and performance of the vehicle. A dozen of methods have been introduced for determining the pavement texture and each of them has advantages and disadvantages. The usage of the image processing as one of the newest methods is expanding because of its significant advantages. In this study, by constructing a device for taking a picture with different light angles from the pavement surface and using a four lights photometric stereo method, the pavements with different texture have been investigated. Finally, for pavements with different texture type, the optimal angles of different light have been introduced for use in photometric stereo technique. The results show that the adverse relationship between the optimum zenith angle and the depth of texture of the pavement surface, such that by increasing the mean texture depth, using the less zenith angle will give better results.

A Review of Depth and Normal Fusion Algorithms.

Geometric surface information such as depth maps and surface normals can be acquired by various methods such as stereo light fields, shape from shading and photometric stereo techniques. We compare several algorithms which deal with the combination of depth with surface normal information in order to reconstruct a refined depth map. The reasons for performance differences are examined from the perspective of alternative formulations of surface normals for depth reconstruction. We review and analyze methods in a systematic way. Based on our findings, we introduce a new generalized fusion method, which is formulated as a least squares problem and outperforms previous methods in the depth error domain by introducing a novel normal weighting that performs closer to the geodesic distance measure. Furthermore, a novel method is introduced based on Total Generalized Variation (TGV) which further outperforms previous approaches in terms of the geodesic normal distance error and maintains comparable quality in the depth error domain.

Visibility-consistent thin surface reconstruction using multi-scale kernels

One of the key properties of many surface reconstruction techniques is that they represent the volume in front of and behind the surface, e.g., using a variant of signed distance functions. This creates significant problems when reconstructing thin areas of an object since the backside interferes with the reconstruction of the front. We present a two-step technique that avoids this interference and thus imposes no constraints on object thickness. Our method first extracts an approximate surface crust and then iteratively refines the crust to yield the final surface mesh. To extract the crust, we use a novel observation-dependent kernel density estimation to robustly estimate the approximate surface location from the samples. Free space is similarly estimated from the samples' visibility information. In the following refinement, we determine the remaining error using a surface-based kernel interpolation that limits the samples' influence to nearby surface regions with similar orientation and iteratively move the surface towards its true location. We demonstrate our results on synthetic as well as real datasets reconstructed using multi-view stereo techniques or consumer depth sensors.

Applied algorithm in the liner inspection of solid rocket motors

In rocket motors, the bonding between the solid propellant and thermal insulation is accomplished by a thin adhesive layer, known as liner. The liner application method involves a complex sequence of tasks, which includes in its final stage, the surface integrity inspection. Nowadays in Brazil, an expert carries out a thorough visual inspection to detect defects on the liner surface that may compromise the propellant interface bonding. Therefore, this paper proposes an algorithm that uses the photometric stereo technique and the K-nearest neighbor (KNN) classifier to assist the expert in the surface inspection. Photometric stereo allows the surface information recovery of the test images, while the KNN method enables image pixels classification into two classes: non-defect and defect. Tests performed on a computer vision based prototype validate the algorithm. The positive results suggest that the algorithm is feasible and when implemented in a real scenario, will be able to help the expert in detecting defective areas on the liner surface.