Pose Recovery Research Articles

Photography-based façade recovery & 3-d modeling: A CAD application in Cultural Heritage

Abstract This paper deals with the problem of monument's facade pose recovery from a single image acquired with a completely uncalibrated camera (e.g. historical photography). The five camera intrinsic parameters are directly representing from the Image of the Absolute Conic (IAC) and the recovering procedure is based on some prior knowledge on camera's aspect ratio ( r ) value domain (one independent quadratic constraint on IAC) and monument's geometric properties (four independent linearly constraints on IAC). So, for each r discrete value, the five camera intrinsic parameters are estimated using these five independent constraints on IAC, and then monument's facade pose is recovered and 3-d projective and metric reconstruction is achieved relative to the selected aspect ratio. Following, by back-projecting the reconstructed 3-d model onto the single image on screen (CAD design session) and applying a minimization function on the discrepancy vectors between the 3-d model and the single photography image, the final r value is estimated. This new presence-of-skew stratified self-calibration method, despite the fact that it is an iterative one, is experimentally shown to be advantageous over traditional zero-skew calibration methods, when historical, or on-site, completely uncalibrated photography, of rich in geometric constraints monuments, is concerned. The method is of interest for cultural heritage documentation, digital architecture, archaeology, reverse engineering and virtual reality. Also, the proposed approach can be used in any vision problem where camera self-calibration is involved, either for enhancing robustness or removing ambiguities.

3D human pose recovery from image by efficient visual feature selection

In this paper we propose a new examplar-based approach to recover 3D human poses from monocular images. Given the visual feature of each frame, pose retrieval is first conducted in the examplar database to find relevant pose candidates. Then, dynamic programming is applied on the pose candidates to recover a continuous pose sequence. We make two contributions within this framework. First, we propose to use an efficient feature selection algorithm to select effective visual feature components. The task is formulated as a trace-ratio criterion which measures the score of the selected feature component subset, and the criterion is efficiently optimized to achieve the global optimum. The selected components are used instead of the original full feature set to improve the accuracy and efficiency of pose recovery. As second contribution, we propose to use sparse representation to retrieve the pose candidates, where the measured visual feature is expressed as a sparse linear combination of the examplars in the database. Sparse representation ensures that semantically similar poses have larger probability to be retrieved. The effectiveness of our approach is validated quantitatively through extensive evaluations on both synthetic and real data, and qualitatively by inspecting the results of the real time system we have implemented.

Recovery of surface pose from texture orientation statistics under perspective projection

In a seminal paper, Witkin (1981) derived a model of surface slant and tilt recovery based on the statistics of the orientations of texture elements (texels) on a planar surface. This model made use of basic mathematical properties of probability distributions to formulate a posterior distribution on slant and tilt given a set of image orientations under orthographic projection. One problem with the Witkin model was that it produced a posterior distribution with multiple maxima, reflecting the inherent ambiguity in scene reconstruction under orthographic projection. In the present article, we extend Witkin's method to incorporate the effects of perspective projection. An identical approach is used to that of Witkin; however, the model now reflects the effects of perspective projection on texel orientation. Performance of the new model is compared against that of Witkin's model in a basic surface pose recovery task using both a maximum a posteriori (MAP) decision rule and a rule based on the expected value of the posterior distribution. The resultant posterior of the new model is shown to have only one maximum and thereby the ambiguity in scene interpretation is resolved. Furthermore, the model performs better than Witkin's model using both MAP and expected value decision rules. The results are discussed in the context of human slant estimation.

Action and Gait Recognition From Recovered 3-D Human Joints

A common viewpoint-free framework that fuses pose recovery and classification for action and gait recognition is presented in this paper. First, a markerless pose recovery method is adopted to automatically capture the 3-D human joint and pose parameter sequences from volume data. Second, multiple configuration features (combination of joints) and movement features (position, orientation, and height of the body) are extracted from the recovered 3-D human joint and pose parameter sequences. A hidden Markov model (HMM) and an exemplar-based HMM are then used to model the movement features and configuration features, respectively. Finally, actions are classified by a hierarchical classifier that fuses the movement features and the configuration features, and persons are recognized from their gait sequences with the configuration features. The effectiveness of the proposed approach is demonstrated with experiments on the Institut National de Recherche en Informatique et Automatique Xmas Motion Acquisition Sequences data set.

Visual Navigation for Mobile Devices

This article presents the integration of an improved camera pose recovery method into a landmark-based visual navigation system for mobile devices.

3D curve structure reconstruction from a sparse set of unordered images

A flexible and portable approach to reconstructing 3D curve structures of free-form objects from a sparse set of unordered images taken by a hand-held CCD camera is presented. To make the whole pipeline robust and accurate, we mark the interested curves on the object to make them visually outstand from the background. Some specifically designed markers are distributed around the object as well. Followed by an automatic camera pose recovery and camera lens distortion correction step, a user-friendly semi-automatic method is put forward to extract the curves in the images. The curve correspondences in stereo image pairs are then optimally established and the 3D curve structures are recovered by using the stereo triangulation method. Experiments show that the method is applicable for reverse modeling of freeform objects with moderate accuracy requirements.

The Efficient E-3D Visual Servoing

A vision-based control technique is proposed to automatically drive a robot to a given desired pose which has never been reached beforehand. Hence, the corresponding desired image is not available. Furthermore, since we deal with unknown scenes, standard pose reconstruction algorithms cannot be applied. To efficiently solve this problem, we represent the scene as a collection of planes. A robust detector is employed to explicitly identify planes, since they may leave the image during extended navigation tasks. These planes are then exploited by an efficient direct method for pose recovery, leading to fast and accurate estimates. The framework is validated with synthetic and real imagery.

A spatio-temporal 2D-models framework for human pose recovery in monocular sequences

This paper addresses the pose recovery problem of a particular articulated object: the human body. In this model-based approach, the 2D-shape is associated to the corresponding stick figure allowing the joint segmentation and pose recovery of the subject observed in the scene. The main disadvantage of 2D-models is their restriction to the viewpoint. To cope with this limitation, local spatio-temporal 2D-models corresponding to many views of the same sequences are trained, concatenated and sorted in a global framework. Temporal and spatial constraints are then considered to build the probabilistic transition matrix (PTM) that gives a frame to frame estimation of the most probable local models to use during the fitting procedure, thus limiting the feature space. This approach takes advantage of 3D information avoiding the use of a complex 3D human model. The experiments carried out on both indoor and outdoor sequences have demonstrated the ability of this approach to adequately segment pedestrians and estimate their poses independently of the direction of motion during the sequence.

A Robust and Accurate Two-Stage Approach for Automatic Recovery of Distal Locking Holes in Computer-Assisted Intramedullary Nailing of Femoral Shaft Fractures

It has been recognized that one of the most difficult steps in intramedullary nailing of femoral shaft fractures is the distal locking - the insertion of distal transverse interlocking screws, for which it is necessary to know the positions and orientations of the distal locking holes (DLHs) of the intramedullary nail (IMN). This paper presents a robust and accurate approach for solving this problem based on two calibrated and registered fluoroscopic images. The problem is formulated as a two-stage model-based optimal fitting process. The first stage, nail detection, automatically estimates the axis of the distal part of the IMN (DP-IMN) by iteratively fitting a cylindrical model to the images. The second stage, pose recovery, resolves the translations and the rotations of the DLHs around the estimated axis by iteratively fitting the geometrical models of the DLHs to the images. An iterative best matched projection point (IBMPP) algorithm is combined with random sample strategies to effectively and robustly solve the fitting problems in both stages. We designed and conducted comprehensive experiments to validate the robustness and the accuracy of the present approach. Our in vitro experiments show on average less than 14 s execution time on a Linux machine, a mean angular error of 0.48 degrees (std = 0.21 degrees ), and a mean translational error of 0.09 mm (std = 0.041 mm). We conclude that the present approach is fast, robust, and accurate for distal locking applications.

Quasi-Facial Communication for Online Learning Using 3D Modeling Techniques

Online interaction with 3D facial animation is an alternative way of face-to-face communication for distance education. 3D facial modeling is essential for virtual educational environments establishment. This article presents a novel 3D facial modeling solution that facilitates quasi-facial communication for online learning. Our algorithm builds 3D facial models from a single image, with support of a 3D face database. First from the image, we extract a set of feature points, which are then used to automatically estimate the head pose parameters using the 3D mean face in our database as a reference model. After the pose recovery, a similarity measurement function is proposed to locate the neighborhood for the given image in the 3D face database. The scope of neighborhood can be determined adaptively using our cross-validation algorithm. Furthermore, the individual 3D shape is synthesized by neighborhood interpolation. Texture mapping is achieved based on feature points. The experimental results show that our algorithm can robustly produce 3D facial models from images captured in various scenarios to enhance the lifelikeness in distant learning.

A Full-Body Layered Deformable Model for Automatic Model-Based Gait Recognition

This paper proposes a full-body layered deformable model (LDM) inspired by manually labeled silhouettes for automatic model-based gait recognition from part-level gait dynamics in monocular video sequences. The LDM is defined for the fronto-parallel gait with 22 parameters describing the human body part shapes (widths and lengths) and dynamics (positions and orientations). There are four layers in the LDM and the limbs are deformable. Algorithms for LDM-based human body pose recovery are then developed to estimate the LDM parameters from both manually labeled and automatically extracted silhouettes, where the automatic silhouette extraction is through a coarse-to-fine localization and extraction procedure. The estimated LDM parameters are used for model-based gait recognition by employing the dynamic time warping for matching and adopting the combination scheme in AdaBoost.M2. While the existing model-based gait recognition approaches focus primarily on the lower limbs, the estimated LDM parameters enable us to study full-body model-based gait recognition by utilizing the dynamics of the upper limbs, the shoulders and the head as well. In the experiments, the LDM-based gait recognition is tested on gait sequences with differences in shoe-type, surface, carrying condition and time. The results demonstrate that the recognition performance benefits from not only the lower limb dynamics, but also the dynamics of the upper limbs, the shoulders and the head. In addition, the LDM can serve as an analysis tool for studying factors affecting the gait under various conditions.

Accurate and precise 2D–3D registration based on X-ray intensity

This paper addresses the problem of estimating the 3D rigid pose of an object from its digitized X-ray projection. We considered the cases of homogeneous (CAD models) and inhomogeneous (attenuation map obtained from computed tomography) X-ray attenuation in an optimization framework based on a mutual information similarity measure. Convergence of object pose recovery is highly precise and obtained with sub-millimeter accuracy for both screen-film and digital radiographs by three major enhancements: (i) special care is given to the model of Parzen distribution used in the mutual information estimator (data pre-sphering in the bivariate case and bandwidth estimation in the univariate case); (ii) a quasi-global optimization scheme based on a modified version of stochastic clustering is used in conjunction with an object mesh resampling stage to reduce variance of the final pose estimator; (iii) nonlinear response to the radiograph is also estimated for screen-film radiographs.

Accurate and reliable pose recovery of distal locking holes in computer-assisted intra-medullary nailing of femoral shaft fractures: A preliminary study

Objective: One of the difficult steps in intra-medullary nailing of femoral shaft fractures is distal locking – the insertion of distal interlocking screws. Conventionally, this is performed using repeated image acquisitions, which leads to considerable irradiation of the patient and surgical team. Virtual fluoroscopy has been used to reduce radiation exposure, but can only provide multi-planar two-dimensional projection views. In this study, two calibrated fluoroscopic images were used to automatically recover the positions and orientations of the distal locking holes (DLHs). The ultimate goal is to provide precise three-dimensional guidance during distal locking.Methods: A model-based optimal fitting process was used to reconstruct the positions and orientations of the DLHs from two calibrated fluoroscopic images. No human intervention is required. A preliminary in vitro validation study was conducted to analyze the accuracy and reliability of the technique using images acquired from different viewpoints. The ground truths of the DLH were obtained by inserting a custom-made steel rod through the hole and then digitizing both the top and bottom centers of the rod using a sharp pointer. The recovery errors were computed by comparing the computed results to the ground truths.Results: In all experiments, the poses of the DLHs could be recovered fully automatically. When the recovered positions and orientations of the DLHs were compared to their associated ground truths, a mean angular error of 0.5° (STD = 0.2°), and a mean translational error of 0.1 mm (STD = 0.0 mm) were found.Conclusions: Accurate and reliable pose recovery of distal locking holes from two calibrated fluoroscopic images is achievable. Our preliminary in vitro experimental results demonstrate that the recovered poses of the distal locking holes are sufficiently accurate for intra-operative use.

Nonlinear manifold learning for dynamic shape and dynamic appearance

Our objective is to learn representations for the shape and the appearance of moving (dynamic) objects that support tasks such as synthesis, pose recovery, reconstruction, and tracking. In this paper, we introduce a framework that aims to learn landmark-free correspondence-free global representations of dynamic appearance manifolds. We use nonlinear dimensionality reduction to achieve an embedding of the global deformation manifold which preserves the geometric structure of the manifold. Given such embedding, a nonlinear mapping is learned from the embedding space into the visual input space. Therefore, any visual input is represented by a linear combination of nonlinear bases functions centered along the manifold in the embedding space. We also show how approximate solution for the inverse mapping can be obtained in a closed form which facilitate recovery of the intrinsic body configuration. We use the framework to learn the gait manifold as an example of a dynamic shape manifold, as well as to learn the manifolds for some simple gestures and facial expressions as examples of dynamic appearance manifolds.

Accurate and reliable pose recovery of distal locking holes in computer-assisted intra-medullary nailing of femoral shaft fractures: A preliminary study

Objective: One of the difficult steps in intra-medullary nailing of femoral shaft fractures is distal locking – the insertion of distal interlocking screws. Conventionally, this is performed using repeated image acquisitions, which leads to considerable irradiation of the patient and surgical team. Virtual fluoroscopy has been used to reduce radiation exposure, but can only provide multi-planar two-dimensional projection views. In this study, two calibrated fluoroscopic images were used to automatically recover the positions and orientations of the distal locking holes (DLHs). The ultimate goal is to provide precise three-dimensional guidance during distal locking.Methods: A model-based optimal fitting process was used to reconstruct the positions and orientations of the DLHs from two calibrated fluoroscopic images. No human intervention is required. A preliminary in vitro validation study was conducted to analyze the accuracy and reliability of the technique using images acquired from different viewpoints. The ground truths of the DLH were obtained by inserting a custom-made steel rod through the hole and then digitizing both the top and bottom centers of the rod using a sharp pointer. The recovery errors were computed by comparing the computed results to the ground truths.Results: In all experiments, the poses of the DLHs could be recovered fully automatically. When the recovered positions and orientations of the DLHs were compared to their associated ground truths, a mean angular error of 0.5° (STD = 0.2°), and a mean translational error of 0.1 mm (STD = 0.0 mm) were found.Conclusions: Accurate and reliable pose recovery of distal locking holes from two calibrated fluoroscopic images is achievable. Our preliminary in vitro experimental results demonstrate that the recovered poses of the distal locking holes are sufficiently accurate for intra-operative use.

An augmented reality human–computer interface for object localization in a cognitive vision system

The European Cognitive Vision project VAMPIRE uses mobile AR-kits to interact with a visual active memory for teaching and retrieval purposes. This paper describes concept and technical realization of the used mobile AR-kits and discusses interactive learning and retrieval in office environments, and the active memory infrastructure. The focus is on 3D interaction for pointing in a scene coordinate system. This is achieved by 3D augmented pointing, which combines inside-out tracking for head pose recovery and 3D stereo human–computer interaction. Experimental evaluation shows that the accuracy of this 3D cursor is within a few centimeters, which is sufficient to point at an object in an office. Finally, an application of the cursor in VAMPIRE is presented, where in addition to the mobile system, at least one stationary active camera is used to obtain different views of an object. There are many potential applications, for example an improved view-based object recognition.

TU-EE-A3-03: Automated Segmentation of Radiographic Fiducials for C-Arm Tracking

Purpose: Intraoperative quantitative C-arm fluoroscopy guidance depends on discerning the relative pose of images (pose recovery). A possible method is to use radiographic fiducials visible in fluoro images [1,2]. We propose a robust and fast method for segmenting fiducials designed for brachytherapy applications. Methods and materials: The fiducial contains points, lines and ellipses made from BBs and wires[1]. The algorithm integrates the a-priori knowledge of fiducial's mechanical construction in a cleverly devised workflow. The BB segmentation is achieved using morphological top-hat transform. This information serves as a heuristic input to line segmentation realized by a curve tracing algorithm which operates on edge image, followed by augmenting information from intensity image. Once the lines are segmented, this information feeds to the ellipse extraction step. For ellipse segmentation, intensity image is morphologically processed to eliminate background noise, followed by elimination of BB-s and lines from the information obtained in prior steps. The resulting image consists of only ellipse segments. A fast variation of Hough transform is used to rectify the full ellipse from the segments. Results: The fiducial algorithm identified all the features (BBs, lines and ellipses) visible to human eye in all ten clinical images. Next the accuracy of fiducial segmentation was assessed numerically by feeding the results to the pose recovery algorithm of [1]. The fiducial was moved on an accurate mechanical platform (as ground truth) while the C-arm was stationary. We reconstructed the relative poses with an accuracy of 1.2 mm in translation and 0.3 degrees in rotational based on the segmented fiducials. Conclusions: The algorithm makes effective use of a-priori knowledge and combines the techniques of morphological segmentation, curve tracing, and Hough transform, resulting in a novel curve segmentation strategy.

Simultaneous pose recovery and camera registration from multiple views of a walking person

We present an algorithm to estimate the body pose of a walking person given synchronized video input from multiple uncalibrated cameras. We construct an appearance model of human walking motion by generating examples from the space of body poses and camera locations, and clustering them using expectation–maximization. Given a segmented input video sequence, we find the closest matching appearance cluster for each silhouette and use the sequence of matched clusters to extrapolate the position of the camera with respect to the person's direction of motion. For each frame, the matching cluster also provides an estimate of the walking phase. We combine these estimates from all views and find the most likely sequence of walking poses using a cyclical, feed-forward hidden Markov model. Our algorithm requires no manual initialization and no prior knowledge about the locations of the cameras.

Recovering 3D human pose from monocular images

We describe a learning-based method for recovering 3D human body pose from single images and monocular image sequences. Our approach requires neither an explicit body model nor prior labeling of body parts in the image. Instead, it recovers pose by direct nonlinear regression against shape descriptor vectors extracted automatically from image silhouettes. For robustness against local silhouette segmentation errors, silhouette shape is encoded by histogram-of-shape-contexts descriptors. We evaluate several different regression methods: ridge regression, Relevance Vector Machine (RVM) regression, and Support Vector Machine (SVM) regression over both linear and kernel bases. The RVMs provide much sparser regressors without compromising performance, and kernel bases give a small but worthwhile improvement in performance. The loss of depth and limb labeling information often makes the recovery of 3D pose from single silhouettes ambiguous. To handle this, the method is embedded in a novel regressive tracking framework, using dynamics from the previous state estimate together with a learned regression value to disambiguate the pose. We show that the resulting system tracks long sequences stably. For realism and good generalization over a wide range of viewpoints, we train the regressors on images resynthesized from real human motion capture data. The method is demonstrated for several representations of full body pose, both quantitatively on independent but similar test data and qualitatively on real image sequences. Mean angular errors of 4-6 degrees are obtained for a variety of walking motions.

FTRAC—A robust fluoroscope tracking fiducial

C-arm fluoroscopy is ubiquitous in contemporary surgery, but it lacks the ability to accurately reconstruct three-dimensional (3D) information. A major obstacle in fluoroscopic reconstruction is discerning the pose of the x-ray image, in 3D space. Optical/magnetic trackers tend to be prohibitively expensive, intrusive and cumbersome in many applications. We present single-image-based fluoroscope tracking (FTRAC) with the use of an external radiographic fiducial consisting of a mathematically optimized set of ellipses, lines, and points. This is an improvement over contemporary fiducials, which use only points. The fiducial encodes six degrees of freedom in a single image by creating a unique view from any direction. A nonlinear optimizer can rapidly compute the pose of the fiducial using this image. The current embodiment has salient attributes: small dimensions (3 x 3 x 5 cm); need not be close to the anatomy of interest; and accurately segmentable. We tested the fiducial and the pose recovery method on synthetic data and also experimentally on a precisely machined mechanical phantom. Pose recovery in phantom experiments had an accuracy of 0.56 mm in translation and 0.33 degrees in orientation. Object reconstruction had a mean error of 0.53 mm with 0.16 mm STD. The method offers accuracies similar to commercial tracking systems, and appears to be sufficiently robust for intraoperative quantitative C-arm fluoroscopy. Simulation experiments indicate that the size can be further reduced to 1 x 1 X 2 cm, with only a marginal drop in accuracy.