Dense Reconstruction Research Articles

Casual 3D photography

We present an algorithm that enables casual 3D photography. Given a set of input photos captured with a hand-held cell phone or DSLR camera, our algorithm reconstructs a 3D photo , a central panoramic, textured, normal mapped, multi-layered geometric mesh representation. 3D photos can be stored compactly and are optimized for being rendered from viewpoints that are near the capture viewpoints. They can be rendered using a standard rasterization pipeline to produce perspective views with motion parallax. When viewed in VR, 3D photos provide geometrically consistent views for both eyes. Our geometric representation also allows interacting with the scene using 3D geometry-aware effects, such as adding new objects to the scene and artistic lighting effects. Our 3D photo reconstruction algorithm starts with a standard structure from motion and multi-view stereo reconstruction of the scene. The dense stereo reconstruction is made robust to the imperfect capture conditions using a novel near envelope cost volume prior that discards erroneous near depth hypotheses. We propose a novel parallax-tolerant stitching algorithm that warps the depth maps into the central panorama and stitches two color-and-depth panoramas for the front and back scene surfaces. The two panoramas are fused into a single non-redundant, well-connected geometric mesh. We provide videos demonstrating users interactively viewing and manipulating our 3D photos.

TOWARDS ROBUST SELF-CALIBRATION FOR HANDHELD 3D LINE LASER SCANNING

Abstract. This paper studies self-calibration of a structured light system, which reconstructs 3D information using video from a static consumer camera and a handheld cross line laser projector. Intersections between the individual laser curves and geometric constraints on the relative position of the laser planes are exploited to achieve dense 3D reconstruction. This is possible without any prior knowledge of the movement of the projector. However, inaccurrately extracted laser lines introduce noise in the detected intersection positions and therefore distort the reconstruction result. Furthermore, when scanning objects with specular reflections, such as glossy painted or metalic surfaces, the reflections are often extracted from the camera image as erroneous laser curves. In this paper we investiagte how robust estimates of the parameters of the laser planes can be obtained despite of noisy detections.

Event-Based, 6-DOF Camera Tracking from Photometric Depth Maps

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high-speed motions or in scenes characterized by high dynamic range. These features, along with a very low power consumption, make event cameras an ideal complement to standard cameras for VR/AR and video game applications. With these applications in mind, this paper tackles the problem of accurate, low-latency tracking of an event camera from an existing photometric depth map (i.e., intensity plus depth information) built via classic dense reconstruction pipelines. Our approach tracks the 6-DOF pose of the event camera upon the arrival of each event, thus virtually eliminating latency. We successfully evaluate the method in both indoor and outdoor scenes and show that-because of the technological advantages of the event camera-our pipeline works in scenes characterized by high-speed motion, which are still inaccessible to standard cameras.

Proximal robust factorization for piecewise planar reconstruction

In this paper, we aim to obtain a dense piecewise planar reconstruction of the scene from multiple image frames based on a factorization framework. Integrating all the relevant constraints in a global objective function, we are able to effectively leverage on the scene smoothness prior afforded by the dense formulation, as well as imposing the necessary algebraic constraints required by the shape matrix. These constraints also help to robustly decompose the measurement matrix into the underlying low-rank subspace and the sparse outlier part. Numerically, we achieve the constrained factorization and decomposition via modifying a recently proposed proximal alternating robust subspace minimization algorithm. The results show that our algorithm is effective in handling real life sequences, and outperforms other algorithms in recovering motions and dense scene estimate.

Feasibility of 3D Reconstruction of Neural Morphology Using Expansion Microscopy and Barcode-Guided Agglomeration.

We here introduce and study the properties, via computer simulation, of a candidate automated approach to algorithmic reconstruction of dense neural morphology, based on simulated data of the kind that would be obtained via two emerging molecular technologies—expansion microscopy (ExM) and in-situ molecular barcoding. We utilize a convolutional neural network to detect neuronal boundaries from protein-tagged plasma membrane images obtained via ExM, as well as a subsequent supervoxel-merging pipeline guided by optical readout of information-rich, cell-specific nucleic acid barcodes. We attempt to use conservative imaging and labeling parameters, with the goal of establishing a baseline case that points to the potential feasibility of optical circuit reconstruction, leaving open the possibility of higher-performance labeling technologies and algorithms. We find that, even with these conservative assumptions, an all-optical approach to dense neural morphology reconstruction may be possible via the proposed algorithmic framework. Future work should explore both the design-space of chemical labels and barcodes, as well as algorithms, to ultimately enable routine, high-performance optical circuit reconstruction.

Dense Reconstruction of Transparent Objects by Altering Incident Light Paths Through Refraction

This paper addresses the problem of reconstructing the surface shape of transparent objects. The difficulty of this problem originates from the viewpoint dependent appearance of a transparent object, which quickly makes reconstruction methods tailored for diffuse surfaces fail disgracefully. In this paper, we introduce a fixed viewpoint approach to dense surface reconstruction of transparent objects based on refraction of light. We present a simple setup that allows us to alter the incident light paths before light rays enter the object by immersing the object partially in a liquid, and develop a method for recovering the object surface through reconstructing and triangulating such incident light paths. Our proposed approach does not need to model the complex interactions of light as it travels through the object, neither does it assume any parametric form for the object shape nor the exact number of refractions and reflections taken place along the light paths. It can therefore handle transparent objects with a relatively complex shape and structure, with unknown and inhomogeneous refractive index. We also show that for thin transparent objects, our proposed acquisition setup can be further simplified by adopting a single refraction approximation. Experimental results on both synthetic and real data demonstrate the feasibility and accuracy of our proposed approach.

Robust Small Target Co-Detection from Airborne Infrared Image Sequences

In this paper, a novel infrared target co-detection model combining the self-correlation features of backgrounds and the commonality features of targets in the spatio-temporal domain is proposed to detect small targets in a sequence of infrared images with complex backgrounds. Firstly, a dense target extraction model based on nonlinear weights is proposed, which can better suppress background of images and enhance small targets than weights of singular values. Secondly, a sparse target extraction model based on entry-wise weighted robust principal component analysis is proposed. The entry-wise weight adaptively incorporates structural prior in terms of local weighted entropy, thus, it can extract real targets accurately and suppress background clutters efficiently. Finally, the commonality of targets in the spatio-temporal domain are used to construct target refinement model for false alarms suppression and target confirmation. Since real targets could appear in both of the dense and sparse reconstruction maps of a single frame, and form trajectories after tracklet association of consecutive frames, the location correlation of the dense and sparse reconstruction maps for a single frame and tracklet association of the location correlation maps for successive frames have strong ability to discriminate between small targets and background clutters. Experimental results demonstrate that the proposed small target co-detection method can not only suppress background clutters effectively, but also detect targets accurately even if with target-like interference.

Automatic graph based spatiotemporal extrinsic calibration of multiple Kinect V2 ToF cameras

Abstract This paper presents a novel technique for the automatic calibration of motion capture systems composed of multiple Time of Flight cameras (ToF) or any equivalent device able to provide 3D point clouds together with chromatic information. The calibration procedure is designed to be simple, fast and unsupervised: a colored sphere moved freely by hand is used as calibration tool. Three elements in particular distinguish the method we propose from other state of the art solutions: it takes into account and propagates measurement uncertainties, it allows the identification and recovery of systematic time delays in camera synchronization, it optimize the extrinsic parameters by means of a graph based approach. A widely used ToF, Microsoft Kinect V2, was used for the comparison with state of the art multi-ToF calibration techniques. Experimental results demonstrated that the proposed method is the most accurate both in terms of shape reconstruction and spatial consistency. Furthermore, the experimental results underlined that the application of the global graph optimization over a more standard couple-based unweighed matching achieves a higher accuracy in the assessment of the extrinsic parameters. Thus enabling the reduction of both systematic errors and data dispersion in 3D point clouds of reference shapes. The comparison among acquisition systems composed of 3 and 6 devices underlined that graph optimization becomes relevant when the number of devices grows. Taking into account several 3D configurations calibrated with the proposed method, it was assessed that the uncertainty of the extrinsic parameters is generally lower than 2 mm and 1 0 − 2 radiants, 0.6°. The resulting dense 3D reconstruction of objects, both static and in motion, achieved experimental errors lower than to 10 mm, approximately half of other state of the art methods, and, in first approximation, homogeneous for the entire monitored area. The software developed can be found at the site of the MIRo lab (Measurement Instrumentation and Robotics lab) at link 1 under the Creative Commons Attribution conditions.

A NEW APPROACH FOR PROGRESSIVE DENSE RECONSTRUCTION FROM CONSECUTIVE IMAGES BASED ON PRIOR LOW-DENSITY 3D POINT CLOUDS

Abstract. In recent years, the increasing incidence of climate-related disasters has tremendously affected our environment. In order to effectively manage and reduce dramatic impacts of such events, the development of timely disaster management plans is essential. Since these disasters are spatial phenomena, timely provision of geospatial information is crucial for effective development of response and management plans. Due to inaccessibility of the affected areas and limited budget of first-responders, timely acquisition of the required geospatial data for these applications is usually possible only using low-cost imaging and georefencing sensors mounted on unmanned platforms. Despite rapid collection of the required data using these systems, available processing techniques are not yet capable of delivering geospatial information to responders and decision makers in a timely manner. To address this issue, this paper introduces a new technique for dense 3D reconstruction of the affected scenes which can deliver and improve the needed geospatial information incrementally. This approach is implemented based on prior 3D knowledge of the scene and employs computationally-efficient 2D triangulation, feature descriptor, feature matching and point verification techniques to optimize and speed up 3D dense scene reconstruction procedure. To verify the feasibility and computational efficiency of the proposed approach, an experiment using a set of consecutive images collected onboard a UAV platform and prior low-density airborne laser scanning over the same area is conducted and step by step results are provided. A comparative analysis of the proposed approach and an available image-based dense reconstruction technique is also conducted to prove the computational efficiency and competency of this technique for delivering geospatial information with pre-specified accuracy.

Axonal synapse sorting in medial entorhinal cortex.

Research on neuronal connectivity in the cerebral cortex has focused on the existence and strength of synapses between neurons, and their location on the cell bodies and dendrites of postsynaptic neurons. The synaptic architecture of individual presynaptic axonal trees, however, remains largely unknown. Here we used dense reconstructions from three-dimensional electron microscopy in rats to study the synaptic organization of local presynaptic axons in layer 2 of the medial entorhinal cortex, the site of grid-like spatial representations. We observe path-length-dependent axonal synapse sorting, such that axons of excitatory neurons sequentially target inhibitory neurons followed by excitatory neurons. Connectivity analysis revealed a cellular feedforward inhibition circuit involving wide, myelinated inhibitory axons and dendritic synapse clustering. Simulations show that this high-precision circuit can control the propagation of synchronized activity in the medial entorhinal cortex, which is known for temporally precise discharges.

EMF- MHCRF: Enhanced Median Filter (EMF) Based Noise Removal and Multilayer Hidden Conditional Random Field (MHCRF) Model for Dense Depth Map Reconstruction

EMF- MHCRF: Enhanced Median Filter (EMF) Based Noise Removal and Multilayer Hidden Conditional Random Field (MHCRF) Model for Dense Depth Map Reconstruction

Live User-Guided Intrinsic Video for Static Scenes

We present a novel real-time approach for user-guided intrinsic decomposition of static scenes captured by an RGB-D sensor. In the first step, we acquire a three-dimensional representation of the scene using a dense volumetric reconstruction framework. The obtained reconstruction serves as a proxy to densely fuse reflectance estimates and to store user-provided constraints in three-dimensional space. User constraints, in the form of constant shading and reflectance strokes, can be placed directly on the real-world geometry using an intuitive touch-based interaction metaphor, or using interactive mouse strokes. Fusing the decomposition results and constraints in three-dimensional space allows for robust propagation of this information to novel views by re-projection. We leverage this information to improve on the decomposition quality of existing intrinsic video decomposition techniques by further constraining the ill-posed decomposition problem. In addition to improved decomposition quality, we show a variety of live augmented reality applications such as recoloring of objects, relighting of scenes and editing of material appearance.

Low-dimension local descriptor for dense stereo matching and scene reconstruction

The DAISY descriptor has been widely used in dense stereo matching and scene reconstruction. However, DAISY is vulnerable to similar feature regions because the construction method of DAISY sequentially arranges the description of center and neighbor sample points and does not consider their relationships. To enhance the discriminative power of the local descriptor and accelerate the speed of dense matching and scene reconstruction, we propose a low-dimensional local descriptor. The proposed descriptor is inspired from the local binary pattern (LBP). In image space, LBP describes local detail texture by computing the difference between center and neighbor sample points. We introduce this advantage in scale space to extend the DAISY descriptor and make it more efficient for dense matching similar features in the different regions. On this basis, a two-dimensional discrete cosine transform (2D-DCT) is utilized to reduce the dimensions of the descriptor as well as reduce the computation cost of dense matching and scene reconstruction. Through a variety of experiments on the benchmark laser-scanned ground truth scenes as well as indoor and outdoor scenes, we show the proposed descriptor can get more accurate depth maps and more complete reconstruction results than that of using other common descriptors, and the computational speed is much faster than that of using DAISY.

THREE DIMENSIONAL RECONSTRUCTION WORKFLOWS FOR LOST CULTURAL HERITAGE MONUMENTS EXPLOITING PUBLIC DOMAIN AND PROFESSIONAL PHOTOGRAMMETRIC IMAGERY

Abstract. In our paper, we document experiments and results of image-based 3d reconstructions of famous heritage monuments which were recently damaged or completely destroyed by the so-called Islamic state in Syria and Iraq. The specific focus of our research is on the combined use of professional photogrammetric imagery and of publicly available imagery from the web for optimally 3d reconstructing those monuments. The investigated photogrammetric reconstruction techniques include automated bundle adjustment and dense multi-view 3d reconstruction using public domain and professional imagery on the one hand and an interactive polygonal modelling based on projected panoramas on the other. Our investigations show that the combination of these two image-based modelling techniques delivers better results in terms of model completeness, level of detail and appearance.

Stereo vision during adverse weather — Using priors to increase robustness in real-time stereo vision

Stereo vision can deliver a dense 3D reconstruction of the environment in real-time for driver assistance as well as autonomous driving. Semi-Global Matching (SGM) is a popular method of choice for solving this task which is already in use for production vehicles. Despite the enormous progress in the field and the high level of performance of modern stereo methods, one key challenge remains: robust stereo vision in automotive scenarios during rain, snow and darkness. Under these circumstances, current methods generate strong temporal noise, many disparity outliers and false positives on object level. These problems are addressed in this work by regularizing stereo vision via prior information. We formulate a temporal prior and a scene prior, which we apply to SGM in order to overcome the deficiencies. The temporal prior integrates knowledge from the previous disparity map to exploit the high temporal correlation, the scene prior exploits knowledge of a representative traffic scene. Using these priors, the object detection rate improves significantly on a driver assistance dataset of 3000 frames including bad weather while reducing the rate of erroneous object detections. We also outperform the ECCV Robust Vision Challenge 2012 winner, iSGM, on this dataset. In addition, results are presented for the KITTI dataset, even showing improvements under good weather conditions when exploiting the temporal prior.We also show that the temporal and scene priors are easy and efficient to implement on a hybrid CPU/reconfigurable hardware platform. The use of these priors can be extended to other application areas such as mobile robotics.

WASS: An open-source pipeline for 3D stereo reconstruction of ocean waves

Stereo 3D reconstruction of ocean waves is gaining more and more popularity in the oceanographic community and industry. Indeed, recent advances of both computer vision algorithms and computer processing power now allow the study of the spatio-temporal wave field with unprecedented accuracy, especially at small scales. Even if simple in theory, multiple details are difficult to be mastered for a practitioner, so that the implementation of a sea-waves 3D reconstruction pipeline is in general considered a complex task. For instance, camera calibration, reliable stereo feature matching and mean sea-plane estimation are all factors for which a well designed implementation can make the difference to obtain valuable results. For this reason, we believe that the open availability of a well tested software package that automates the reconstruction process from stereo images to a 3D point cloud would be a valuable addition for future researches in this area.We present WASS (http://www.dais.unive.it/wass), an Open-Source stereo processing pipeline for sea waves 3D reconstruction. Our tool completely automates all the steps required to estimate dense point clouds from stereo images. Namely, it computes the extrinsic parameters of the stereo rig so that no delicate calibration has to be performed on the field. It implements a fast 3D dense stereo reconstruction procedure based on the consolidated OpenCV library and, lastly, it includes set of filtering techniques both on the disparity map and the produced point cloud to remove the vast majority of erroneous points that can naturally arise while analyzing the optically complex nature of the water surface.In this paper, we describe the architecture of WASS and the internal algorithms involved. The pipeline workflow is shown step-by-step and demonstrated on real datasets acquired at sea.

COMPARATIVE ANALYSIS OF 3D POINT CLOUDS GENERATED FROM A FREEWARE AND TERRESTRIAL LASER SCANNER

Abstract. In the recent past, several heritage structures have faced destruction due to both human-made incidents and natural calamities that have caused a great loss to the human race regarding its cultural achievements. In this context, the importance of documenting such structures to create a substantial database cannot be emphasised enough. The Clock Tower of Dehradun, India is one such structure. There is a lack of sufficient information in the digital domain, which justified the need to carry out this study. Thus, an attempt has been made to gauge the possibilities of using open source 3D tools such as VSfM to quickly and easily obtain point clouds of an object and assess its quality. The photographs were collected using consumer grade cameras with reasonable effort to ensure overlap. The sparse reconstruction and dense reconstruction were carried out to generate a 3D point cloud model of the tower. A terrestrial laser scanner (TLS) was also used to obtain a point cloud of the tower. The point clouds obtained from the two methods were analyzed to understand the quality of the information present; TLS acquired point cloud being a benchmark to assess the VSfM point cloud. They were compared to analyze the point density and subjected to a plane-fitting test for sample flat portions on the structure. The plane-fitting test revealed the planarity of the point clouds. A Gauss distribution fit yielded a standard deviation of 0.002 and 0.01 for TLS and VSfM, respectively. For more insight, comparisons with Agisoft Photoscan results were also made.

A High-Throughput, Field-Based Phenotyping Technology for Tall Biomass Crops.

Recent advances in omics technologies have not been accompanied by equally efficient, cost-effective, and accurate phenotyping methods required to dissect the genetic architecture of complex traits. Even though high-throughput phenotyping platforms have been developed for controlled environments, field-based aerial and ground technologies have only been designed and deployed for short-stature crops. Therefore, we developed and tested Phenobot 1.0, an auto-steered and self-propelled field-based high-throughput phenotyping platform for tall dense canopy crops, such as sorghum (Sorghum bicolor). Phenobot 1.0 was equipped with laterally positioned and vertically stacked stereo RGB cameras. Images collected from 307 diverse sorghum lines were reconstructed in 3D for feature extraction. User interfaces were developed, and multiple algorithms were evaluated for their accuracy in estimating plant height and stem diameter. Tested feature extraction methods included the following: (1) User-interactive Individual Plant Height Extraction (UsIn-PHe) based on dense stereo three-dimensional reconstruction; (2) Automatic Hedge-based Plant Height Extraction (Auto-PHe) based on dense stereo 3D reconstruction; (3) User-interactive Dense Stereo Matching Stem Diameter Extraction; and (4) User-interactive Image Patch Stereo Matching Stem Diameter Extraction (IPaS-Di). Comparative genome-wide association analysis and ground-truth validation demonstrated that both UsIn-PHe and Auto-PHe were accurate methods to estimate plant height, while Auto-PHe had the additional advantage of being a completely automated process. For stem diameter, IPaS-Di generated the most accurate estimates of this biomass-related architectural trait. In summary, our technology was proven robust to obtain ground-based high-throughput plant architecture parameters of sorghum, a tall and densely planted crop species.

Four-dimensional dense reconstruction of retinogenesis in zebrafish

Four-dimensional dense reconstruction of retinogenesis in zebrafish

Light-field camera-based 3D volumetric particle image velocimetry with dense ray tracing reconstruction technique

This paper presents a dense ray tracing reconstruction technique for a single light-field camera-based particle image velocimetry. The new approach pre-determines the location of a particle through inverse dense ray tracing and reconstructs the voxel value using multiplicative algebraic reconstruction technique (MART). Simulation studies were undertaken to identify the effects of iteration number, relaxation factor, particle density, voxel–pixel ratio and the effect of the velocity gradient on the performance of the proposed dense ray tracing-based MART method (DRT-MART). The results demonstrate that the DRT-MART method achieves higher reconstruction resolution at significantly better computational efficiency than the MART method (4–50 times faster). Both DRT-MART and MART approaches were applied to measure the velocity field of a low speed jet flow which revealed that for the same computational cost, the DRT-MART method accurately resolves the jet velocity field with improved precision, especially for the velocity component along the depth direction.