Visual Hull Research Articles

Development of multi-view 3D reconstruction system for bubble flow measurement

This work introduces the development of bubble measurement method utilizing a three-dimensional (3D) reconstruction technique from multi-view images. Multiple synchronized cameras were positioned around a water container, and calibration was performed to obtain external and internal camera parameters. Images of bubbles emerging from a nozzle were captured, and a machine learning technique was used to extract bubble silhouettes as foreground probability distributions, enabling the extraction of bubbles from images obtained with a simple lighting setup. These distributions were then projected onto a 3D voxel space using the visual hull method. It was confirmed that the method can successfully capture bubble generation, detachment, and rise behaviors, offering insights into understanding bubble dynamics and potential applications in 3D computational fluid dynamics validation.

Accelerating NeRF with the Visual Hull

Accelerating NeRF with the Visual Hull

Quality-preserving multilevel mesh generation for building models

ABSTRACT Three-dimensional (3D) high-fidelity building models are crucial for digital twin cities. Multi-scale modeling is crucial for accurately capturing the complexity of urban environments; however, existing methods often lack the flexibility required to satisfy the varied demands of diverse applications. In this paper, we propose a framework to continuously simplify 3D building models, it dynamically adjusts mesh accuracy to meet diverse needs, enabling seamless transitions from high-poly to medium and low-poly meshes. Our method contains three main stages: First, in the mesh repair stage, we introduce a novel mesh repair framework based on an advanced Marching Cubes 33 algorithm to ensure a watertight and clean topology mesh. Then, for generating medium-poly meshes, we employ a feature-preserving mesh simplification algorithm that combines curvature and area-weighted quadratic error metrics to achieve a high-fidelity simplified mesh. For generating low-poly meshes, we use the concept of carving visual hull to generate low-poly mesh with high visual similarity. Our method has been evaluated on building subsets of the Free3D and Thingi10 K datasets, as well as on various building models generated from different observation scales, and its superiority has been validated through extensive comparisons with state-of-the-art techniques.

Three-dimensional diagnosis of lean premixed turbulent swirl flames using tomographic background oriented Schlieren

The necessity of minimizing NOx emissions drives the pursuit of ultra-lean premixed combustion in aeroengines and gas turbines, characterized by susceptibility to combustion instabilities. To tackle this issue, swirling flow design is widely incorporated into lean premixed combustor design, enhancing flame stability, and shortening flame length. This study utilizes the tomographic background-oriented Schlieren (TBOS) to reconstruct the spatial distribution of the refractive index gradient of lean premixed turbulent swirl flames with an aeroengine combustor configuration. A parametric study of the TBOS reconstruction quality is conducted, and the results reveal that view sparseness primarily degrades the reconstruction quality compared to the specific iterative algorithm used. The classic visual hull approach is explored to address this challenge, highlighting the significance of visual hull size. Furthermore, to improve the reconstruction quality, a posterior support constraint method is proposed, involving the removal of voxels of nearly constant refractive index within the central volume surrounded by flames. Results demonstrate that implementing this posterior support constraint further improves the reconstruction quality of lean premixed turbulent swirl flames. Finally, the robustness of this posterior support constraint method is validated by introducing high-level noise to the light deflection data, showcasing the potential of combining the dedicated designed visual hull and proposed posterior support constraint in addressing the view sparseness challenge for TBOS measurements.

From prediction to measurement, an efficient method for digital human model obtainment

Digital human has been increasingly used in industry, for example in Metaverse which has been a popular topic in recent years. The existing method of obtaining digital human models are either expensive or lack of accuracy. In this paper, we discuss a novel method to reconstruct a 3D human model from 2D images captured by a monocular camera. The input of our method only requires a set of rotated human body images that can accept slight movement. First, we apply a deep learning method to predict an initial 3D human body model from multi-view human body images. Then the total detailed digital human model will be computed and optimized. The typical method requires the human body and cameras fixed to obtain a visual hull from a significant number of camera images. This could be extremely expensive and inconvenient when such an application is developed for online users. Compared to the structural lighting measurement system, our predict-optimized framework only requires several input images captured by personal equipment to provide enough accuracy and online use resolution results.

Simple Non‐Destructive and 3D Multi‐Layer Visual Hull Reconstruction with an Ultrabroadband Carbon Nanotubes Photo‐Imager

AbstractWhile millimeter‐wave (MMW)–infrared (IR) broad photo‐imaging exhibits significant potential as an inspection technique, photo‐thermoelectric (PTE) detector operations employing carbon nanotubes (CNTs) are highly suitable because of their efficient absorption characteristics in the above bands. Such an imaging device design and the associated material coordination allow the aggregation of multiple wavelength‐specific optical information for the constituent identification of target objects in a non‐destructive manner with high sensitivity. On the other side, computer vision techniques facilitate 3D structure reconstruction as approaches from inspection methodologies, in addition to the aforementioned imaging device design. Although non‐destructive inspections require collectively satisfying both constituent identification and structural reconstruction, investigations combining MMW–IR broad photo imaging and computer vision monitoring are still insufficient. This work demonstrates computer vision‐driven simple 3D composite multi‐layer reconstructions via non‐destructive broadband multiple‐wavelength monitoring with the CNT film PTE imager. Visual hull evaluation, which is one of the representative computer vision techniques, converts 2D transmissive PTE images with different viewpoints into 3D reconstruction models. The subsequential graphical superposition of these wavelength‐specific models results in non‐destructive entire reconstructions of 3D composite multi‐layer target objects. Therefore, the present computer vision‐driven PTE broadband 3D reconstruction bridges the gap between methodologies and device design strategies toward non‐destructive inspection applications.

Seedscreener: A novel integrated wheat germplasm phenotyping platform based on NIR-feature detection and 3D-reconstruction

Crop seed phenomics provides a new operational basis for breeders. However, a scarcity of seed phenotypic characterization, especially of the single seed kernels, severely restricts their use in plant breeding. In this study, we focus on designing and deploying a prototype for automatic and high throughput seed phenotyping. The novel setup, Seedscreener, allows simultaneous RGB imaging and NIR spectrum analysis pipeline to evaluate single kernel 3D morphologies and internal biochemicals. With the assistance of a data acquisition device centered on a near-infrared spectrometer and an industrial camera, as well as an automated acquisition software under the QT development framework, Seedscreener has the ability to acquire both endophenotype and exophenotype traits of single wheat kernels simultaneously. The developed biochemical endophenotype traits prediction model can forecast the protein content (PC), starch content (SC) and gluten content (GC) of wheat kernel by the full-band absorbance spectra. The developed morphological exophenotype traits extraction model based on Marching Cubes (MC) algorithm can calculate the wheat grain length (GL), grain width (GW), grain thickness (GT), grain surface area (GSA) and convex hull volume (CHV) using the three-dimensional visual hull constructed by the lateral profile information of the wheat grain. The results show that the platform could reach 94% success rate of single wheat phenotypic data collection, the average accuracy of endophenotype traits prediction is 0.85, the average accuracy of exophenotype traits extraction is 0.93, and the working flux of the platform is about 90 grains/h. Compared with existing individual wheat kernels phenotyping equipment, Seedscreener could obtain both endophenotype and exophenotype characteristics of wheat seeds with high throughput and precision.

High-Resolution Volumetric Reconstruction for Clothed Humans

We present a novel method for reconstructing clothed humans from a sparse set of, e.g., 1–6 RGB images. Despite impressive results from recent works employing deep implicit representation, we revisit the volumetric approach and demonstrate that better performance can be achieved with proper system design. The volumetric representation offers significant advantages in leveraging 3D spatial context through 3D convolutions, and the notorious quantization error is largely negligible with a reasonably large yet affordable volume resolution, e.g., 512. To handle memory and computation costs, we propose a sophisticated coarse-to-fine strategy with voxel culling and subspace sparse convolution. Our method starts with a discretized visual hull to compute a coarse shape and then focuses on a narrow band nearby the coarse shape for refinement. Once the shape is reconstructed, we adopt an image-based rendering approach, which computes the colors of surface points by blending input images with learned weights. Extensive experimental results show that our method significantly reduces the mean point-to-surface (P2S) precision of state-of-the-art methods by more than 50% to achieve approximately 2mm accuracy with a 512 volume resolution. Additionally, images rendered from our textured model achieve a higher peak signal-to-noise ratio (PSNR) compared to state-of-the-art methods.

Unified shape and appearance reconstruction with joint camera parameter refinement

In this paper, we present an inverse rendering method for the simple reconstruction of shape and appearance of real-world objects from only roughly calibrated RGB images captured under collocated point light illumination. To this end, we gradually reconstruct the lower-frequency geometry information using automatically generated occupancy mask images based on a visual hull initialization of the mesh, to infer the object topology, and a smoothness-preconditioned optimization. By combining this geometry estimation with learning-based SVBRDF parameter inference as well as intrinsic and extrinsic camera parameter refinement in a joint and unified formulation, our novel method is able to reconstruct shape and an isotropic SVBRDF from fewer input images than previous methods. Unlike in other works, we also estimate normal maps as part of the SVBRDF to capture and represent higher-frequency geometric details in a compact way. Furthermore, by regularizing the appearance estimation with a GAN-based SVBRDF generator, we are able to meaningfully limit the solution space. In summary, this leads to a robust automatic reconstruction algorithm for shape and appearance. We evaluated our algorithm on synthetic as well as on real-world data and demonstrate that our method is able to reconstruct complex objects with high-fidelity reflection properties in a robust way, also in the presence of imperfect camera parameter data.

NHK Meta Studio: A Compact Volumetric TV Studio for 3-D Reconstruction

Many studies have been conducted on the 3D reconstruction of subjects using multiple fixed cameras. Accepting the trade-off between the number of cameras and reconstruction quality, our studio is designed to capture high-quality models of one or two subjects for TV program use. Several cameras are mounted on a hemispherical dome with the stage in the center and a cloth cover on the frame for chroma-keying. The optimal camera numbers and placements for reconstruction were determined by simulation, and the 3D reconstruction was performed as a point cloud by a combination of visual hull and stereo matching. The quality was still not high enough, however, so we also added a surface light field to the point cloud to obtain the weighted average of rays from camera images close to the viewpoint. In the final stage, the images were then combined to the video, and errors generated during the reconstruction were compensated for by use of a deep neural network (DNN) for video translation. An offline processing studio has been built as a preliminary step towards real-time processing, and the reconstructed 3D images have been evaluated subjectively for a number of subjects. These studies confirm the effectiveness of this studio design.

Real-time light-field generation based on the visual hull for the 3D light-field display with free-viewpoint texture mapping.

Real-time dense view synthesis based on three-dimensional (3D) reconstruction of real scenes is still a challenge for 3D light-field display. It's time-consuming to reconstruct an entire model, and then the target views are synthesized afterward based on volume rendering. To address this issue, Light-field Visual Hull (LVH) is presented with free-viewpoint texture mapping for 3D light-field display, which can directly produce synthetic images with the 3D reconstruction of real scenes in real-time based on forty free-viewpoint RGB cameras. An end-to-end subpixel calculation procedure of the synthetic image is demonstrated, which defines a rendering ray for each subpixel based on light-field image coding. In the ray propagation process, only the essential spatial point of the target model is located for the corresponding subpixel by projecting the frontmost point of the ray to all the free-viewpoints, and the color of each subpixel is identified in one pass. A dynamic free-viewpoint texture mapping method is proposed to solve the correct graphic texture considering the free-viewpoint cameras. To improve the efficiency, only the visible 3D position and texture that contributes to the synthetic image are calculated based on backward ray tracing rather than computing the entire 3D model and generating all elemental images. In addition, an incremental calibration method by dividing camera groups is proposed to satisfy the accuracy. Experimental results show the validity of our method. All the rendered views are analyzed for justifying the texture mapping method, and the PSNR is improved by an average of 11.88dB. Finally, LVH can achieve a natural and smooth viewing effect at 4K resolution and the frame rate of 25 ∼ 30fps with a large viewing angle.

Geometry characterization and thermal radiation prediction of buoyant square and line diffusion flames based on 3D flame reconstruction

This paper presents an experimental study on the instantaneous and mean geometrical features and radiation properties of buoyant square and line diffusion flames with heat release rates (Q˙) of 5.0–30.0 kW using the visual hull reconstruction method. Equal fuel exit areas were selected for the square and linear burners to reveal the effect of fire source shape on the combustion process. Results show that the reconstructed three-dimensional (3D) instantaneous flames could well reflect the temporal evolutions of the square and line flames. The instantaneous flame surface area and volume exhibit excellent synchronization with time. Consistent with the predictions based on the turbulent mixing concepts, the mean flame surface area and volume of the square flame are lower than the line flame under equal Q˙. The heat release rates per unit surface area remain unchanged, and the heat release rates per unit volume decrease slightly with Q˙ for the square and line flames. The mean beam lengths, calculated from the mean flame surface area and volume of the 3D flame shape, increase with Q˙ steadily for the two types of flames. A 3D flame radiation model is developed based on the reconstructed 3D instantaneous flame shape. It could accurately predict the instantaneous and time-averaged radiative heat fluxes in the near-field and far-field for axisymmetric and non-axisymmetric flames.

Cross-view SLAM solver: Global pose estimation of monocular ground-level video frames for 3D reconstruction using a reference 3D model from satellite images

Accurate pose estimation of monocular ground-level images with respect to a satellite/aerial photogrammetric dataset is an extremely challenging task. Existing solutions often perform an offline post-registration on 3D results from both sources, which however, suffers from non-rigid geometric distortions of the 3D monocular reconstruction and the lack of overlaps between the air to ground content. This paper provides an online solution that performs accurate pose estimation of the ground images with respect to a 3D model derived from satellite images, followed by a dense 3D reconstruction. Our solution takes advantage of the simultaneous localization and mapping (SLAM) paradigm to dynamically incorporate reference observations from the satellite 3D model during the incremental pose estimation, called a cross-view SLAM solver, which leverages both ground-to-satellite error and image-level reprojection errors at the frame level to yield image poses that are well-registered to the satellite 3D model for facade point cloud reconstruction. This process also has the advantage of correcting non-rigid distortions and trajectory drifts that are often presented in monocular SLAM systems. In addition, our solution leverages both the geometric and semantic information from the satellite model and ground images to perform a per-frame correction for frame-level pose initialization, in which a novel scheme called pose buffer is introduced to initialize the pose of each keyframe through robust visual hull alignment of ground objects. The proposed approach has been experimented using four trajectories of monocular videos collections (around 7,000 frames per trajectory on average) and a 3D semantic model from multi-view satellite images to estimate the poses of the video frames and yield point clouds consistent with the satellite 3D models, evaluated by using LiDAR ground-truth. Both qualitative and quantitative experiments demonstrate that our solution yields accurate, drift-free poses and point clouds consistent with the satellite data and visually much more pleasing 3D models with facade information. Compared to the LiDAR ground-truth, the derived 3D models with ground-level images have achieved a mean absolute error of 1.78 m (improved from 3.15 m achieved using SLAM without utilizing satellite 3D models) (A testing program will be made available through https://github.com/GDAOSU/Cross-View-SLAM).

Autonomous Exploration of Small Bodies Toward Greater Autonomy for Deep Space Missions.

Autonomy is becoming increasingly important for the robotic exploration of unpredictable environments. One such example is the approach, proximity operation, and surface exploration of small bodies. In this article, we present an overview of an estimation framework to approach and land on small bodies as a key functional capability for an autonomous small-body explorer. We use a multi-phase perception/estimation pipeline with interconnected and overlapping measurements and algorithms to characterize and reach the body, from millions of kilometers down to its surface. We consider a notional spacecraft design that operates across all phases from approach to landing and to maneuvering on the surface of the microgravity body. This SmallSat design makes accommodations to simplify autonomous surface operations. The estimation pipeline combines state-of-the-art techniques with new approaches to estimating the target’s unknown properties across all phases. Centroid and light-curve algorithms estimate the body–spacecraft relative trajectory and rotation, respectively, using a priori knowledge of the initial relative orbit. A new shape-from-silhouette algorithm estimates the pole (i.e., rotation axis) and the initial visual hull that seeds subsequent feature tracking as the body gets more resolved in the narrow field-of-view imager. Feature tracking refines the pole orientation and shape of the body for estimating initial gravity to enable safe close approach. A coarse-shape reconstruction algorithm is used to identify initial landable regions whose hazardous nature would subsequently be assessed by dense 3D reconstruction. Slope stability, thermal, occlusion, and terra-mechanical hazards would be assessed on densely reconstructed regions and continually refined prior to landing. We simulated a mission scenario for approaching a hypothetical small body whose motion and shape were unknown a priori, starting from thousands of kilometers down to 20 km. Results indicate the feasibility of recovering the relative body motion and shape solely relying on onboard measurements and estimates with their associated uncertainties and without human input. Current work continues to mature and characterize the algorithms for the last phases of the estimation framework to land on the surface.

EFFICIENT MULTI-VIEW 3D TRACKING OF ARBITRARY ROCK FRAGMENTS UPON IMPACT

Abstract. This paper presents a new methodology to accurately obtain 3D rotational velocities of blocks and fragments. Four high speed cameras are used to capture the scene. An additional two tilted mirrors are used to multiply the number of views. Hence, a total of six different viewing perspectives can be used to track translational and rotational velocities in 3D. The focus in the current work is on the rotational velocities, as tracking of the translation is generally straightforward. A common outline tracking algorithm based on the visual hull is adapted. The visual hull is further meshed using triangular elements to approximate the shape of the object. This 3D reconstruction is then used to track the 3D motion of the object. However, the accuracy of the results strongly depends on the accuracy of the 3D reconstruction which is mainly influenced by the number and position of the available views. In any case, the 3D reconstruction from the visual hull is only an approximation and significant errors can be introduced which influence the tracking accuracy. Hence, an in-house post-processing algorithm based on the knowledge of the real geometry of the object, which can generally be accurately determined after a test, was developed. The improved performance of this new post-processing method is shown by controlled spinning tests. Finally, results of a real example of an impact fragmentation test are discussed.

Poisson Surface Reconstruction with Envelope Constraints

AbstractReconstructing surfaces from scanned 3D points has been an important research area for several decades. One common approach that has proven efficient and robust to noise is implicit surface reconstruction, i.e. fitting to the points a 3D scalar function (such as an indicator function or signed‐distance field) and then extracting an isosurface. Though many techniques fall within this category, existing methods either impose no boundary constraints or impose Dirichlet/Neumann conditions on the surface of a bounding box containing the scanned data.In this work, we demonstrate the benefit of supporting Dirichlet constraints on a general boundary. To this end, we adapt the Screened Poisson Reconstruction algorithm to input a constraint envelope in addition to the oriented point cloud. We impose Dirichlet boundary conditions, forcing the reconstructed implicit function to be zero outside this constraint surface. Using a visual hull and/or depth hull derived from RGB‐D scans to define the constraint envelope, we obtain substantially improved surface reconstructions in regions of missing data.

A review of silhouette extraction algorithms for use within visual hull pipelines

ABSTRACT Markerless motion capture would permit the study of human biomechanics in environments where marker-based systems are impractical, e.g. outdoors or underwater. The visual hull tool may enable such data to be recorded, but it requires the accurate detection of the silhouette of the object in multiple camera views. This paper reviews the top-performing algorithms available to date for silhouette extraction, with the visual hull in mind as the downstream application; the rationale is that higher-quality silhouettes would lead to higher-quality visual hulls, and consequently better measurement of movement. This paper is the first attempt in the literature to compare silhouette extraction algorithms that belong to different fields of Computer Vision, namely background subtraction, semantic segmentation, and multi-view segmentation. It was found that several algorithms exist that would be substantial improvements over the silhouette extraction algorithms traditionally used in visual hull pipelines. In particular, FgSegNet v2 (a background subtraction algorithm), DeepLabv3+ JFT (a semantic segmentation algorithm), and Djelouah 2013 (a multi-view segmentation algorithm) are the most accurate and promising methods for the extraction of silhouettes from 2D images to date, and could seamlessly be integrated within a visual hull pipeline for studies of human movement or biomechanics.

Three-dimensional Visual Mapping of Underwater Ship Hull Surface Using Piecewise-planar SLAM

In-water visual ship hull inspection using unmanned underwater vehicles needs to be performed at very close range to the target surface because of the visibility limitations in underwater environments mainly due to light attenuation, scattering, and water turbidity. These environmental challenges result in ineffective photometric and geometric information in hull surface images and, therefore, the performance of conventional three-dimensional (3D) reconstruction techniques is often unsatisfactory. This paper addresses a visual mapping method for 3D reconstruction of underwater ship hull surface using a monocular camera as a primary mapping sensor. The main idea of the proposed approach is to model the moderately curved hull surface as a combination of piecewise-planar panels, and to generate a global map by aligning the local images in a two-dimensional reference frame and correcting them appropriately to reflect the information of perspective projections of the 3D panels. The estimated 3D panels associated with the local images are used to extract the loop-closure relative measurements in the framework of simultaneous localization and mapping (SLAM) for precise camera trajectory estimation and 3D reconstruction results. The validity and practical feasibility of the proposed method are demonstrated using a dataset obtained in a field experiment with a full-scale ship in a real sea environment.

Diamond Inclusion Reconstruction

In this paper, we make an improvement on the conventional visual hull reconstruction method that runs on a single consumer graphics card. The target application is a high-precision diamond inclusion reconstruction system. One major contribution of this paper is an evaluation system of voxels for high-precision reconstruction. In contrast to existing approaches, it allows us to reconstruct the thin structure of the diamond inclusion. Based on this, we obtain a significant improvement in reconstruction precision, especially for thin inclusion.

Visual Hull Tree: A New Progressive Method to Represent Voxel Data

A visual hull is an approximation of a three-dimensional (3-D) object generated by the shape-from-silhouettes (SfS) technique. Because a visual hull is calculated from silhouettes, a visual hull can be represented by silhouette images, encoded by a small number of bits. However, it is challenging to represent the concave regions of a visual hull. In this paper, we model voxel data with a set of visual hulls, thereby handling the concave regions of voxel data. To accomplish this, silhouettes are generated from input voxel data using virtual cameras, and a visual hull is computed by SfS using the silhouettes. To handle the concave regions, we calculate the residuals of visual hulls, and the residuals are represented by visual hulls again. This process is repeated until all concave regions are processed, and a hierarchical data structure, i.e., a visual hull tree, is generated. Because the visual hull tree is constituted from a set of visual hulls, it can represent the details of the voxel data even in the root node. Also, because a set of visual hulls can be represented by silhouettes, a visual hull tree has a small number of bits. From the experiments, we compare our method to the octree-based representation, and our method demonstrates good encoding performance.