Non-rigid Shape Research Articles

A Face-Detection Postprocessing Scheme Using a Geometric Analysis for Multimedia Applications

Abstract—Human faces have been broadly studied in digital image and video processing fields. An appearance-based method, the adaptive boosting learning algorithm using integral image representations has been successfully employed for face detection, taking advantage of the feature extraction’s low computational complexity. In this paper, we propose a face-detection postprocessing method that equalizes instantaneous facial regions in an efficient hardware architecture for use in real-time multimedia applications. The proposed system requires low hardware resources and exhibits robust performance in terms of the movements, zooming, and classification of faces. A series of experimental results obtained using video sequences collected under dynamic conditions are discussed. Index Terms—Face detection, adaptive boosting algorithm, face-region stabilization, face recognition, single-port line memory I. I NTRODUCTION Human faces have been broadly studied in digital image and video processing fields. As one of the most detailed attributes in the human-computer interface, the face conveys a great deal of the information including behavioral cues, emotional state, identity, human race, age, and so on. Face-based algorithms thus have been used in a wide range of multimedia applications [1-3]. In particular, most digital imaging systems have integrated face-based auto-focusing functions to provide users with a convenient interface from the perspective of visual attention. In addition, the face-recognition systems require a correct face-detection scheme to extract facial features fed into pre-trained classifiers [4]. Detection and feature extraction are concurrently performed while searching human faces in digital images under dynamic visual deformation conditions such as position, scale, in-plane rotation, orientation, pose, and illumination. Depending upon the specific applications, robust face segmentation has been employed to solve several problems including the non-rigid shape of faces, clever alignment, and occlusion. In order to successfully achieve face detection in tiny mobile multimedia platforms, the hardware architecture must be computationally optimized in terms of memory usage and operation time. An appearance-based method, the adaptive boosting (AdaBoost) learning algorithm has been widely accepted in face detection using integral image representations. It is ideal for real-time hardware systems such as digital cameras and mobile phones [5] and requires low computational complexity for feature extraction. The AdaBoost employs local binary pattern (LBP) images to avoid the effects of illumination and to express detailed textures. The pyramid representation is assigned for multifarious sizes of human faces. To segment face regions according to the in-plane rotations of the human face, pre-defined feature factors are used. Although AdaBoost exhibits acceptable performance, instantaneous face detection in the spatial domain caused by cascaded structures limits its performance. For example, the pyramid representation is used to compare

Stochastic Exploration of Ambiguities for Nonrigid Shape Recovery

Recovering the 3D shape of deformable surfaces from single images is known to be a highly ambiguous problem because many different shapes may have very similar projections. This is commonly addressed by restricting the set of possible shapes to linear combinations of deformation modes and by imposing additional geometric constraints. Unfortunately, because image measurements are noisy, such constraints do not always guarantee that the correct shape will be recovered. To overcome this limitation, we introduce a stochastic sampling approach to efficiently explore the set of solutions of an objective function based on point correspondences. This allows us to propose a small set of ambiguous candidate 3D shapes and then use additional image information to choose the best one. As a proof of concept, we use either motion or shading cues to this end and show that we can handle a complex objective function without having to solve a difficult nonlinear minimization problem. The advantages of our method are demonstrated on a variety of problems including both real and synthetic data.

Relaxations for Minimizing Metric Distortion and Elastic Energies for 3D Shape Matching

We present two methods for non-rigid shape matching. Both methods formulate shape matching as an energy minimization problem, where the energy measures distortion of the metric defined on the shapes in one case, or directly describes the physical deformation relating the two shapes in the other case. The first method considers a parametrized relaxation of the widely adopted quadratic assignment problem (QAP) formulation for minimum distortion correspondence between deformable shapes. In order to control the accuracy/sparsity trade-o a weighting parameter is introduced to combine two existing relaxations, namely spectral and game-theoretic. This leads to an approach for deformable shape matching with controllable sparsity. The second method focuses on computing a geometrically consistent and spatially dense matching between two 3D shapes. Rather than mapping points to points it matches infinitesimal surface patches while preserving the geometric structures. In this spirit, matchings are considered as dieomorphisms between the objects’ surfaces which are by definition geometrically consistent. Based on the observation that such dieomorphisms can be represented as closed and continuous surfaces in the product space of the two shapes, this leads to a minimal surface problem in this product space. The proposed discrete formulation describes the search space with linear constraints. Computationally, the approach results in a binary linear program whose relaxed version can be solved eciently in a globally optimal manner.

Scale Invariant Geometry for Nonrigid Shapes

In nature, different animals of the same species frequently exhibit local variations in scale. New developments in shape matching research thus increasingly provide us with the tools to answer such fascinating questions as the following: How should we measure the discrepancy between a small dog with large ears and a large one with small ears? Are there geometric structures common to both an elephant and a giraffe? What is the morphometric similarity between a blue whale and a dolphin? Currently, there are only two methods that allow us to quantify similarities between surfaces which are insensitive to deformations in size: scale invariant local descriptors and global normalization methods. Here, we propose a new tool for shape exploration. We introduce a scale invariant metric for surfaces that allows us to analyze nonrigid shapes, generate locally invariant features, produce scale invariant geodesics, embed one surface into another despite changes in local and global size, and assist in the computational...

Non-rigid 3D shape tracking from multiview video

We present a fast and efficient non-rigid shape tracking method for modeling dynamic 3D objects from multiview video. Starting from an initial mesh representation, the shape of a dynamic object is tracked over time, both in geometry and topology, based on multiview silhouette and 3D scene flow information. The mesh representation of each frame is obtained by deforming the mesh representation of the previous frame towards the optimal surface defined by the time-varying multiview silhouette information with the aid of 3D scene flow vectors. The whole time-varying shape is then represented as a mesh sequence which can efficiently be encoded in terms of restructuring and topological operations, and small-scale vertex displacements along with the initial model. The proposed method has the ability to deal with dynamic objects that may undergo non-rigid transformations and topological changes. The time-varying mesh representations of such non-rigid shapes, which are not necessarily of fixed connectivity, can successfully be tracked thanks to restructuring and topological operations employed in our deformation scheme. We demonstrate the performance of the proposed method both on real and synthetic sequences.

Robust Dense Registration of Partial Nonrigid Shapes

This paper presents a complete and robust solution for dense registration of partial nonrigid shapes. Its novel contributions are founded upon the newly proposed heat kernel coordinates (HKCs) that can accurately position points on the shape, and the priority-vicinity search that ensures geometric compatibility during the registration. HKCs index points by computing heat kernels from multiple sources, and their magnitudes serve as priorities of queuing points in registration. We start with shape features as the sources of heat kernels via feature detection and matching. Following the priority order of HKCs, the dense registration is progressively propagated from feature sources to all points. Our method has a superior indexing ability that can produce dense correspondences with fewer flips. The diffusion nature of HKCs, which can be interpreted as a random walk on a manifold, makes our method robust to noise and small holes avoiding surface surgery and repair. Our method searches correspondence only in a small vicinity of registered points, which significantly improves the time performance. Through comprehensive experiments, our new method has demonstrated its technical soundness and robustness by generating highly compatible dense correspondences.

Gaussian-curvature-derived invariants for isometry

Surface deformations without tearing or stretching, preserving the intrinsic properties, are called isometries. This paper presents a new definition of Gaussian curvature moments (GCMs) by the integral of n power of Gaussian curvature. Then a series of moment invariants, called Gaussian curvature moment invariants (GCMIs), are derived via GCMs. These moment invariants share many good properties under rigid transformations and isometric non-rigid transformations, and Gaussian-Bonnet theorem is a special case of GCMI. GCMIs are invariant under isometry and scaling transformations. We construct an invariant vector as a descriptor for a surface via GCMIs, and a modified χ2 distance is defined as a measure of similarity. Finally, experiments show that it is a reliable descriptor for isometric non-rigid shape.

Facial Performance Transfer via Deformable Models and Parametric Correspondence

The issue of transferring facial performance from one person's face to another's has been an area of interest for the movie industry and the computer graphics community for quite some time. In recent years, deformable face models, such as the Active Appearance Model (AAM), have made it possible to track and synthesize faces in real time. Not surprisingly, deformable face model-based approaches for facial performance transfer have gained tremendous interest in the computer vision and graphics community. In this paper, we focus on the problem of real-time facial performance transfer using the AAM framework. We propose a novel approach of learning the mapping between the parameters of two completely independent AAMs, using them to facilitate the facial performance transfer in a more realistic manner than previous approaches. The main advantage of modeling this parametric correspondence is that it allows a "meaningful" transfer of both the nonrigid shape and texture across faces irrespective of the speakers' gender, shape, and size of the faces, and illumination conditions. We explore linear and nonlinear methods for modeling the parametric correspondence between the AAMs and show that the sparse linear regression method performs the best. Moreover, we show the utility of the proposed framework for a cross-language facial performance transfer that is an area of interest for the movie dubbing industry.

Shape Recognition with Spectral Distances

Recent works have shown the use of diffusion geometry for various pattern recognition applications, including nonrigid shape analysis. In this paper, we introduce spectral shape distance as a general framework for distribution-based shape similarity and show that two recent methods for shape similarity due to Rustamov and Mahmoudi and Sapiro are particular cases thereof.

Affine-invariant geodesic geometry of deformable 3D shapes

Natural objects can be subject to various transformations yet still preserve properties that we refer to as invariants. Here, we use definitions of affine-invariant arclength for surfaces in R 3 in order to extend the set of existing non-rigid shape analysis tools. We show that by re-defining the surface metric as its equi-affine version, the surface with its modified metric tensor can be treated as a canonical Euclidean object on which most classical Euclidean processing and analysis tools can be applied. The new definition of a metric is used to extend the fast marching method technique for computing geodesic distances on surfaces, where now, the distances are defined with respect to an affine-invariant arclength. Applications of the proposed framework demonstrate its invariance, efficiency, and accuracy in shape analysis.

Template-synthesized nanostructure morphology influenced by building block structure

We report the synthesis and characterization of a series of nanostructures and determine whether several distinct types of building blocks can be fashioned into linear nanostructures using hard-templates and electrochemical methods. We determine the aspects of molecular structure that influence morphology and propose a mechanism whereby morphology changes as a function of building block. We show how hydrophobic side-chains, such as 3-hexyl and 3-(2-ethyl)hexyl, can be used to prepare nanostructures with a geometry that is different from the shape of the template from which they derive. These nanostructures exhibit collapsed, nonlinear, and nonrigid shapes as observed by SEM and TEM. Hydrophilic side-chains such as 3-(2-(2-(2-ethoxy)ethoxy)ethoxy)methyl and 3-(hydroxy)methyl lead to linear nanostructures. We are also able to prepare linear nanostructures composed of the narrow HOMO–LUMO gap heterocycles selenophene and thieno[3,2-b]thiophene. The results open up many new opportunities for preparing nanostructures with greater complexity, functionality, and distinct optical properties that are imparted by their building block.

Combined shape and feature-based video analysis and its application to non-rigid object tracking

Many video object tracking systems use block matching algorithm (BMA) because of its simple computational structure and robust performance. The BMA, however, exhibits fundamental limitations resulting from non-rigid shapes and similar patterns to the background. The authors propose a combined shape and feature-based non-rigid object tracking algorithm, which is tightly coupled with an adaptive background generation to overcome the limit of block matching. The proposed algorithm is robust to the object's sudden movement or the change of features. This becomes possible by tracking both feature points and their neighbouring regions. Combination of background and shape boundary information significantly improves the tracking performance because the target object and the corresponding feature points on the boundary can be easily found. The shape control points (SCPs) are regularly distributed on the contour of the object, and the authors compare and update the centroid during the tracking process, where straying SCPs are removed, and the tracking continues with only qualified SCPs. As a result, the proposed method becomes free from potential failing factors such as spatio-temporal similarity between object and background, object deformation and occlusion, to name a few. Experiments have been performed using several in-house video sequences including various objects such as a moving robot, swimming fish and walking people. In order to demonstrate the performance of the proposed tracking algorithm, a number of experiments have been performed under noisy and low-contrast environment. For more objective comparison, performance evaluation of tracking surveillance 2002 data sets were also used.

Shape google

The computer vision and pattern recognition communities have recently witnessed a surge of feature-based methods in object recognition and image retrieval applications. These methods allow representing images as collections of “visual words” and treat them using text search approaches following the “bag of features” paradigm. In this article, we explore analogous approaches in the 3D world applied to the problem of nonrigid shape retrieval in large databases. Using multiscale diffusion heat kernels as “geometric words,” we construct compact and informative shape descriptors by means of the “bag of features” approach. We also show that considering pairs of “geometric words” (“geometric expressions”) allows creating spatially sensitive bags of features with better discriminative power. Finally, adopting metric learning approaches, we show that shapes can be efficiently represented as binary codes. Our approach achieves state-of-the-art results on the SHREC 2010 large-scale shape retrieval benchmark.

Motion-based unusual event detection in human crowds

Analyzing human crowds is an important issue in video surveillance and is a challenging task due to their nature of non-rigid shapes. In this paper, optical flows are first estimated and then used for a clue to cluster human crowds into groups in unsupervised manner using our proposed method of adjacency-matrix based clustering (AMC). While the clusters of human crowds are obtained, their behaviors with attributes, orientation, position and crowd size, are characterized by a model of force field. Finally, we can predict the behaviors of human crowds based on the model and then detect if any anomalies of human crowd(s) present in the scene. Experimental results obtained by using extensive dataset show that our system is effective in detecting anomalous events for uncontrolled environment of surveillance videos.

An Optimization Based Framework for Human Pose Estimation

In computer vision community, human pose estimation and nonrigid shape recovery have evolved into different subfields. The state-of-the-art optimization techniques have been applied to the problem of deformable surface reconstruction successfully and recent methods in this area have focused on designing formulations that are easier to solve. In general, these techniques lay their success on the assumption that sufficient 2-D-3-D correspondences can be detected. By contrast, confronted with the similar ambiguity problem, many techniques for human pose estimation adopt stochastic searching or discriminative predictions, which allow for more generative image cues. However, the global optimization cannot be guaranteed via the stochastic methods; and discriminative techniques usually suffer from inaccuracy. In this letter, we absorb ideas from both domains and propose a unified approach for articulated human pose estimation. Specifically, we optimize the human pose to account for the discriminative pose prediction, bone length preservation in parallel with the point-topoint image observation. Moreover, the L2 norm minimization is solved iteratively as a linear system with high computational efficiency.

Shape regression machine and efficient segmentation of left ventricle endocardium from 2D B-mode echocardiogram

We present a machine learning approach called shape regression machine (SRM) for efficient segmentation of an anatomic structure that exhibits a deformable shape in a medical image, e.g., left ventricle endocardial wall in an echocardiogram. The SRM achieves efficient segmentation via statistical learning of the interrelations among shape, appearance, and anatomy, which are exemplified by an annotated database. The SRM is a two-stage approach. In the first stage that estimates a rigid shape to solve an automatic initialization problem, it derives a regression solution to object detection that needs just one scan in principle and a sparse set of scans in practice, avoiding the exhaustive scanning required by the state-of-the-art classification-based detection approach while yielding comparable detection accuracy. In the second stage that estimates the nonrigid shape, it again learns a nonlinear regressor to directly associate nonrigid shape with image appearance. The underpinning of both stages is a novel image-based boosting ridge regression (IBRR) method that enables multivariate, nonlinear modeling and accommodates fast evaluation. We demonstrate the efficiency and effectiveness of the SRM using experiments on segmenting the left ventricle endocardium from a B-mode echocardiogram of apical four chamber view. The proposed algorithm is able to automatically detect and accurately segment the LV endocardial border in about 120ms.

Full and Partial Symmetries of Non-rigid Shapes

Symmetry and self-similarity are the cornerstone of Nature, exhibiting themselves through the shapes of natural creations and ubiquitous laws of physics. Since many natural objects are symmetric, the absence of symmetry can often be an indication of some anomaly or abnormal behavior. Therefore, detection of asymmetries is important in numerous practical applications, including crystallography, medical imaging, and face recognition, to mention a few. Conversely, the assumption of underlying shape symmetry can facilitate solutions to many problems in shape reconstruction and analysis. Traditionally, symmetries are described as extrinsic geometric properties of the shape. While being adequate for rigid shapes, such a description is inappropriate for non-rigid ones: extrinsic symmetry can be broken as a result of shape deformations, while its intrinsic symmetry is preserved. In this paper, we present a generalization of symmetries for non-rigid shapes and a numerical framework for their analysis, addressing the problems of full and partial exact and approximate symmetry detection and classification.

Non-rigid metric reconstruction from perspective cameras

The metric reconstruction of a non-rigid object viewed by a generic camera poses new challenges since current approaches for Structure from Motion assume the rigidity constraint of a shape as an essential condition. In this work, we focus on the estimation of the 3-D Euclidean shape and motion of a non-rigid shape observed by a perspective camera. In such case deformation and perspective effects are difficult to decouple – the parametrization of the 3-D non-rigid body may mistakenly account for the perspective distortion. Our method relies on the fact that it is often a reasonable assumption that some of the points on the object’s surface are deforming throughout the sequence while others remain rigid. Thus, relying on the rigidity constraints of a subset of rigid points, we estimate the perspective to metric upgrade transformation. First, we use an automatic segmentation algorithm to identify the set of rigid points. These are then used to estimate the internal camera calibration parameters and the overall rigid motion. Finally, we formulate the problem of non-rigid shape and motion estimation as a non-linear optimization where the objective function to be minimized is the image reprojection error. The prior information that some of the points in the object are rigid can also be added as a constraint to the non-linear minimization scheme in order to avoid ambiguous configurations. We perform experiments on different synthetic and real data sets which show that even when using a minimal set of rigid points and when varying the intrinsic camera parameters it is possible to obtain reliable metric information.

A Probabilistic Contour Observer for Online Visual Tracking

This paper presents an online, recursive filtering strategy for contour-based tracking. Approaching the tracking problem from an estimation perspective leads to an observer design for the visual track signal associated with an individual target in an image sequence. The track state of the observer is decomposed into group and shape components that describe the gross location and the nonrigid shape, respectively, of the object. A probabilistic representation describes the shape nonparametrically. The constitutive components of the observer are detailed, which include a dynamical prediction model and a correction mechanism. Incorporating the probabilistic observer into the tracking process leads to improved performance and segmentations. The improvements are validated through application of the observer to recorded imagery with evaluation via objective measures of quality.

Signature Detection and Matching for Document Image Retrieval

As one of the most pervasive methods of individual identification and document authentication, signatures present convincing evidence and provide an important form of indexing for effective document image processing and retrieval in a broad range of applications. However, detection and segmentation of free-form objects such as signatures from clustered background is currently an open document analysis problem. In this paper, we focus on two fundamental problems in signature-based document image retrieval. First, we propose a novel multiscale approach to jointly detecting and segmenting signatures from document images. Rather than focusing on local features that typically have large variations, our approach captures the structural saliency using a signature production model and computes the dynamic curvature of 2D contour fragments over multiple scales. This detection framework is general and computationally tractable. Second, we treat the problem of signature retrieval in the unconstrained setting of translation, scale, and rotation invariant nonrigid shape matching. We propose two novel measures of shape dissimilarity based on anisotropic scaling and registration residual error and present a supervised learning framework for combining complementary shape information from different dissimilarity metrics using LDA. We quantitatively study state-of-the-art shape representations, shape matching algorithms, measures of dissimilarity, and the use of multiple instances as query in document image retrieval. We further demonstrate our matching techniques in offline signature verification. Extensive experiments using large real-world collections of English and Arabic machine-printed and handwritten documents demonstrate the excellent performance of our approaches.