Visual Hull Research Articles

Toward cinematizing our daily lives

We introduce a cinematographic video production system to create movie-like attractive footage from our indoor daily life. Since the system is designed for ordinary users in non-studio environments, it is composed of standard hardware components, provides a simple interface, and works in near real-time of 5 ~ 6 frames/sec. The proposed system reconstructs a visual hull from acquired multiple videos and then generates final videos from the model by referring to the camera shots used in film-making. The proposed method utilizes “Reliability” to compensate for errors that may have occurred in non-studio environments and to produce the most natural scene from the reconstructed model. By using a virtual camera control system, even non-experts can easily convert the 3D model to movies that look as if they were created by experienced filmmakers.

Accurate Camera Calibration from Multi-View Stereo and Bundle Adjustment

The advent of high-resolution digital cameras and sophisticated multi-view stereo algorithms offers the promise of unprecedented geometric fidelity in image-based modeling tasks, but it also puts unprecedented demands on camera calibration to fulfill these promises. This paper presents a novel approach to camera calibration where top-down information from rough camera parameter estimates and the output of a multi-view-stereo system on scaled-down input images is used to effectively guide the search for additional image correspondences and significantly improve camera calibration parameters using a standard bundle adjustment algorithm (Lourakis and Argyros 2008). The proposed method has been tested on six real datasets including objects without salient features for which image correspondences cannot be found in a purely bottom-up fashion, and objects with high curvature and thin structures that are lost in visual hull construction even with small errors in camera parameters. Three different methods have been used to qualitatively assess the improvements of the camera parameters. The implementation of the proposed algorithm is publicly available at Furukawa and Ponce (2008b).

Automated hull reconstruction motion tracking (HRMT) applied to sideways maneuvers of free-flying insects

Flying insects perform aerial maneuvers through slight manipulations of their wing motions. Because such manipulations in wing kinematics are subtle, a reliable method is needed to properly discern consistent kinematic strategies used by the insect from inconsistent variations and measurement error. Here, we introduce a novel automated method that accurately extracts full, 3D body and wing kinematics from high-resolution films of free-flying insects. This method combines visual hull reconstruction, principal components analysis, and geometric information about the insect to recover time series data of positions and orientations. The technique has small, well-characterized errors of under 3 pixels for positions and 5 deg. for orientations. To show its utility, we apply this motion tracking to the flight of fruit flies, Drosophila melanogaster. We find that fruit flies generate sideways forces during some maneuvers and that strong lateral acceleration is associated with differences between the left and right wing angles of attack. Remarkably, this asymmetry can be induced by simply altering the relative timing of flips between the right and left wings, and we observe that fruit flies employ timing differences as high as 10% of a wing beat period while accelerating sideways at 40% g.

Incorporating object-centered sampling and Delaunay tetrahedrization for visual hull reconstruction

In this paper we present a novel shape from silhouette algorithm. For an object to be modeled, the algorithm first computes a cloud of points located on a pencil of rays and distributed evenly on the visual hull surface, inside and outside the visual hull. Then Delaunay tetrahedrization is applied to the point cloud to partition its convex hull into a set of tetrahedrons. Finally, outlier tetrahedrons are removed by tetrahedron peeling, and a mesh model of the visual hull is extracted. The algorithm is robust, free from discretization artifacts, and produces a mesh model composed of well-shaped triangles.

Real-Time Body Pose Recognition Using 2D or 3D Haarlets

This article presents a novel approach to markerless real-time pose recognition in a multicamera setup. Body pose is retrieved using example-based classification based on Haar wavelet-like features to allow for real-time pose recognition. Average Neighborhood Margin Maximization (ANMM) is introduced as a powerful new technique to train Haar-like features. The rotation invariant approach is implemented for both 2D classification based on silhouettes, and 3D classification based on visual hulls.

Posturographic analysis through markerless motion capture without ground reaction forces measurement

The ability of the central nervous system to control posture and balance has been used with increasing frequency for the diagnosis and/or treatment evaluation of various neuromuscular diseases. Typically this analysis (Posturographic Analysis) is based on tracking the motion of the center of mass (COM) during quiet standing, however direct measurement of the COM has been commonly approximated using the movement of the center of pressure (COP). The purpose of this study was to apply and validate a new method to track the COM (center of mass) and COP (center of pressure) from a visual hull measured using a markerless motion capture (MMC) method. The method was tested by comparing the calculation of the COP from direct measurements of the COP. The deviations between the methods, below 2 mm, were small relative to the average range of movement guaranteeing a satisfactory signal to noise ratio. This new method requires only kinematic data through MMC method and without the need of a force plate can identify the influence of individual body segments to motion of the COM.

Collective Body Interactive art using 3D Scanner Technology

3次元形状測定により観客の身体形状、動作を実時間で測定しインタラクティブアートのコンテンツとするCollective Bodyを提案する。本作品は、観客の身体形状から生成した3次元モデルに観客の身体動作によって実時間で特殊視覚効果を与える作品である。本作品では、グラフィックスと特殊視覚効果をそれぞれ2種類ずつ実装した。3次元モデルは、パーティクルの集合体で表現し、特殊視覚効果は生命活動をモチーフとしパーティクルそれぞれに運動方程式を与え独自に挙動させることで表現した。ユーザーは、自分自身の身体形状、動作から生成された3次元モデルを自由視点で鑑賞することにより、本作品を通し新しいインタラクティブ体験が出来ると共に新しい視点で自身の身体を捉えることが可能となる。

Fitting solid meshes to animated surfaces using linear elasticity

Computing correspondence between time frames of a time-dependent 3D surface is essential for the understanding of its motion and deformation. In particular, it can be a useful tool in compression, editing, texturing, or analysis of the physical or structural properties of deforming objects. However, correspondence information is not trivial to obtain for experimentally acquired 3D animations, such as time-dependent visual hulls (typically represented as either a binary occupancy grid or as a sequence of meshes of varying connectivity). In this article we present a new nonrigid fitting method that can compute such correspondence information for objects that do not undergo large volume or topological changes, such as living creatures. Experimental results show that it is robust enough to handle visual hull data, allowing to convert it into a constant connectivity mesh with vertices moving in time. Our procedure first creates a rest-state mesh from one of the input frames. This rest-state mesh is then fitted to the consecutive frames. We do this by iteratively displacing its vertices so that a combination of surface distance and elastic potential energy is minimized. A novel rotation compensation method enables us to obtain high-quality results with linear elasticity, even in presence of significant bending.

Bottom and Concave Surface Rendering in Image-based Visual Hull

Bottom and concave shapes on object surface are difficult to reconstruct in image-based visual hull method. In this paper, we propose a simple but efficient method to solve these problems in regular image-based visual hull framework. With the help of a simple image acquiring platform which involves a glass and a mirror, we can capture images of the object from both upper and lower side at the same time. Using these images, silhouette cones necessary for reconstructing the bottom and the concave surfaces could be generated. Therefore the final rendering result of the object can be significantly improved in accuracy and reality, especially in the parts of bottom and concaves.

Efficient Polyhedral Modeling from Silhouettes

Modeling from silhouettes is a popular and useful topic in computer vision. Many methods exist to compute the surface of the visual hull from silhouettes, but few address the problem of ensuring sane topological properties of the surface, such as manifoldness. This article provides an efficient algorithm to compute such a surface in the form of a polyhedral mesh. It relies on a small number of geometric operations to compute a visual hull polyhedron in a single pass. Such simplicity enables the algorithm to combine the advantages of being fast, producing pixel-exact surfaces, and repeatably yield manifold and watertight polyhedra in general experimental conditions with real data, as verified with all datasets tested. The algorithm is fully described, its complexity analyzed and modeling results given.

Synthesis of Silhouettes and Visual Hull Reconstruction for Articulated Humans

In this paper, we propose a complete framework for improved synthesis and understanding of the human pose from a limited number of silhouette images. It combines the active image-based visual hull (IBVH) algorithm and a contour-based body part segmentation technique. We derive a simple, approximate algorithm to decide the extrinsic parameters of a virtual camera, and synthesize the turntable image collection of the person using the IBVH algorithm by actively moving the virtual camera on a properly computed circular trajectory around the person. Using the turning function distance as the silhouette similarity measurement, this approach can be used to generate the desired pose-normalized images for recognition applications. In order to overcome the inability of the visual hull (VH) method to reconstruct concave regions, we propose a contour-based human body part localization algorithm to segment the silhouette images into convex body parts. The body parts observed from the virtual view are generated separately from the corresponding body parts observed from the input views and then assembled together for a more accurate VH reconstruction. Furthermore, the obtained turntable image collection helps to improve the body part segmentation and identification process. By using the inner distance shape context (IDSC) measurement, we are able to estimate the body part locations more accurately from a synthesized view where we can localize the body part more precisely. Experiments show that the proposed algorithm can greatly improve body part segmentation and hence shape reconstruction results.

Shield fields

We describe a unified representation of occluders in light transport and photography using shield fields: the 4D attenuation function which acts on any light field incident on an occluder. Our key theoretical result is that shield fields can be used to decouple the effects of occluders and incident illumination. We first describe the properties of shield fields in the frequency-domain and briefly analyze the "forward" problem of efficiently computing cast shadows. Afterwards, we apply the shield field signal-processing framework to make several new observations regarding the "inverse" problem of reconstructing 3D occluders from cast shadows -- extending previous work on shape-from-silhouette and visual hull methods. From this analysis we develop the first single-camera, single-shot approach to capture visual hulls without requiring moving or programmable illumination. We analyze several competing camera designs, ultimately leading to the development of a new large-format, mask-based light field camera that exploits optimal tiled-broadband codes for light-efficient shield field capture. We conclude by presenting a detailed experimental analysis of shield field capture and 3D occluder reconstruction.

Automatic Generation of a Subject-Specific Model for Accurate Markerless Motion Capture and Biomechanical Applications

A novel approach for the automatic generation of a subject-specific model consisting of morphological and joint location information is described. The aim is to address the need for efficient and accurate model generation for markerless motion capture (MMC) and biomechanical studies. The algorithm applied and expanded on previous work on human shapes space by embedding location information for ten joint centers in a subject-specific free-form surface. The optimal locations of joint centers in the 3-D mesh were learned through linear regression over a set of nine subjects whose joint centers were known. The model was shown to be sufficiently accurate for both kinematic (joint centers) and morphological (shape of the body) information to allow accurate tracking with MMC systems. The automatic model generation algorithm was applied to 3-D meshes of different quality and resolution such as laser scans and visual hulls. The complete method was tested using nine subjects of different gender, body mass index (BMI), age, and ethnicity. Experimental training error and cross-validation errors were 19 and 25 mm, respectively, on average over the joints of the ten subjects analyzed in the study.

3DTV view generation using uncalibrated pure rotating and zooming cameras

This paper proposes a novel method for synthesizing free viewpoint video captured by uncalibrated pure rotating and zooming cameras. Neither intrinsic nor extrinsic parameters of our cameras are known. Projective grid space (PGS), which is the 3D space defined by the epipolar geometry of two basis cameras, is employed for weak camera calibration. Trifocal tensors are used to relate non-basis cameras to PGS. Given trifocal tensors in the initial frame, our method automatically computes trifocal tensors in the other frames. Scale invariant feature transform (SIFT) is used for finding corresponding points in a natural scene between the initial frame and the other frames. Finally, free viewpoint video is synthesized based on the reconstructed visual hull. In the experimental results, free viewpoint video captured by uncalibrated hand-held cameras is successfully synthesized using the proposed method.

The Theory and Practice of Coplanar Shadowgram Imaging for Acquiring Visual Hulls of Intricate Objects

Acquiring 3D models of intricate objects (like tree branches, bicycles and insects) is a challenging task due to severe self-occlusions, repeated thin structures, and surface discontinuities. In theory, a shape-from-silhouettes (SFS) approach can overcome these difficulties and reconstruct visual hulls that are close to the actual shapes, regardless of the complexity of the object. In practice, however, SFS is highly sensitive to errors in silhouette contours and the calibration of the imaging system, and has therefore not been used for obtaining accurate shapes with a large number of views. In this work, we present a practical approach to SFS using a novel technique called coplanar shadowgram imaging that allows us to use dozens to even hundreds of views for visual hull reconstruction. A point light source is moved around an object and the shadows (silhouettes) cast onto a single background plane are imaged. We characterize this imaging system in terms of image projection, reconstruction ambiguity, epipolar geometry, and shape and source recovery. The coplanarity of the shadowgrams yields unique geometric properties that are not possible in traditional multi-view camera-based imaging systems. These properties allow us to derive a robust and automatic algorithm to recover the visual hull of an object and the 3D positions of the light source simultaneously, regardless of the complexity of the object. We demonstrate the acquisition of several intricate shapes with severe occlusions and thin structures, using 50 to 120 views.

Image-based three-dimensional model reconstruction for Chinese treasure—Jadeite Cabbage with Insects

This paper presents a novel 3D reconstruction system for the famous Chinese treasure, Jadeite Cabbage with Insects, from uncalibrated image sequences. There are two major challenges for this 3D model reconstruction problem. The first is the difficult image registration problem due to the semi-diaphaneity and the highly specular property of jadeite materials. Secondly, the unknown camera information, including the intrinsic (calibration) and extrinsic (position and orientation) parameters, to be recovered from the uncalibrated image sequences makes the 3D reconstruction problem very challenging. The proposed 3D modeling process first recovers the camera information as well as sparse 3D structure by using a robust structure from motion algorithm. Then an approximate 3D object model is recovered from the silhouettes at the corresponding multiple views by using the visual hull technique. The final process refines the 3D model by further integrating the 3D information extracted from dense correspondences between image patches of different views. In the proposed 3D reconstruction system, we successfully combine the structure from motion and visual hull techniques to accomplish this challenging task of reconstructing an accurate 3D model for jadeite object from uncalibrated multi-view images. Finally, we assess the 3D reconstruction results for the Chinese jadeite treasure and simulated data by using the proposed 3D reconstruction system.

Reduced Depth and Visual Hulls of Complex 3D Scenes

AbstractDepth and visual hulls are useful for quick reconstruction and rendering of a 3D object based on a number of reference views. However, for many scenes, especially multi‐object, these hulls may contain significant artifacts known as phantom geometry. In depth hulls the phantom geometry appears behind the scene objects in regions occluded from all the reference views. In visual hulls the phantom geometry may also appear in front of the objects because there is not enough information to unambiguously imply the object positions.In this work we identify which parts of the depth and visual hull might constitute phantom geometry. We define the notion of reduced depth hull and reduced visual hull as the parts of the corresponding hull that are phantom‐free. We analyze the role of the depth information in identification of the phantom geometry. Based on this, we provide an algorithm for rendering the reduced depth hull at interactive frame‐rates and suggest an approach for rendering the reduced visual hull. The rendering algorithms take advantage of modern GPU programming techniques.Our techniques bypass explicit reconstruction of the hulls, rendering the reduced depth or visual hull directly from the reference views.

Floating Textures

AbstractWe present a novel multi‐view, projective texture mapping technique. While previous multi‐view texturing approaches lead to blurring and ghosting artefacts if 3D geometry and/or camera calibration are imprecise, we propose a texturing algorithm that warps (“floats”) projected textures during run‐time to preserve crisp, detailed texture appearance. Our GPU implementation achieves interactive to real‐time frame rates. The method is very generally applicable and can be used in combination with many image‐based rendering methods or projective texturing applications. By using Floating Textures in conjunction with, e.g., visual hull rendering, light field rendering, or free‐viewpoint video, improved rendering results are obtained from fewer input images, less accurately calibrated cameras, and coarser 3D geometry proxies.

Carved Visual Hulls for Image-Based Modeling

This article presents a novel method for acquiring high-quality solid models of complex 3D shapes from multiple calibrated photographs. After the purely geometric constraints associated with the silhouettes found in each image have been used to construct a coarse surface approximation in the form of a visual hull, photoconsistency constraints are enforced in three consecutive steps: (1) the rims where the surface grazes the visual hull are first identified through dynamic programming; (2) with the rims now fixed, the visual hull is carved using graph cuts to globally optimize the photoconsistency of the surface and recover its main features; (3) an iterative (local) refinement step is finally used to recover fine surface details. The proposed approach has been implemented, and experiments with seven real data sets are presented, along with qualitative and quantitative comparisons with several state-of-the-art image-based-modeling algorithms.

O fly, where art thou?

In this paper, the design of a real-time image acquisition system for tracking the movement of Drosophila in three-dimensional space is presented. The system uses three calibrated and synchronized cameras to detect multiple flies and integrates the detected fly silhouettes to construct the three-dimensional visual hull models of each fly. We used an extended Kalman filter to estimate the state of each fly, given past positions from the reconstructed fly visual hulls. The results show that our approach constructs the three-dimensional visual hull of each fly from the detected image silhouettes and robustly tracks them at real-time rates. The system is suitable for a more detailed analysis of fly behaviour.