3D Deformable Model Research Articles

Deformation Modeling for Robust 3D Face Matching

Face recognition based on 3D surface matching is promising for overcoming some of the limitations of current 2D image-based face recognition systems. The 3D shape is generally invariant to the pose and lighting changes, but not invariant to the non-rigid facial movement, such as expressions. Collecting and storing multiple templates to account for various expressions for each subject in a large database is not practical. We propose a facial surface modeling and matching scheme to match 2.5D facial scans in the presence of both non-rigid deformations and pose changes (multiview) to a 3D face template. A hierarchical geodesic-based resampling approach is applied to extract landmarks for modeling facial surface deformations. We are able to synthesize the deformation learned from a small group of subjects (control group) onto a 3D neutral model (not in the control group), resulting in a deformed template. A user-specific (3D) deformable model is built by combining the templates with synthesized deformations. The matching distance is computed by fitting this generative deformable model to a test scan. A fully automatic and prototypic 3D face matching system has been developed. Experimental results demonstrate that the proposed deformation modeling scheme increases the 3D face matching accuracy.

TU‐B‐AUD C‐01: Deformable Image Registration: Methods and Clinical Endpoints

Acquisition of anatomic and functional data from magnetic resonance imaging and nuclear medicine studies is becoming increasingly common for patient management in radiation therapy. These data can help improve tumor localization and normal tissue delineation for treatment planning and may provide information about treatment efficacy during or after a course of radiotherapy. Time series data from serial and 4D CT before and during the treatment course, including CT data acquired in the treatment room at the time of treatment, is also helping to estimate motion and shape changes of relevant anatomy. In order to fully realize the benefits of these data, the different imaging studies must be registered to each other or to a common coordinate system. The geometric transformation required to register the different image data can range from simple rotate‐translate to account for differences in patient orientation to 3D or 4D deformation models to account for changes in internal anatomy during and over the course of therapy. Once registered, data derived from the various studies such as anatomic outlines and computed dose can be integrated or fused to help construct a more complete and accurate representation of the patient.This lecture will focus on the mechanics of registering and displaying data from different imaging studies using distinct modalities or a single modality over time. A taxonomy of the different methods will be described. Methods for display and interaction with multimodality data will also be presented. The overall goal is to provide the basic knowledge required to understand what is happening “under‐the‐hood” of the different registration systems one might encounter in the clinic, the different ways these systems are being used for patient management and their limitations.Educational Objectives:1. Understand the basic mechanics of deformable image registration and data fusion techniques.2. Understand the tools used to combine, display & interact with multimodality/4D image and dose data.3. Understand the clinical use and limitations of these techniques for Tx planning, Tx delivery and plan adaptation.

Inversion handling for stable deformable modeling

In 3D deformable modeling approaches based on FEM, inverted tetrahedral elements can cause undesired visual artifacts and the breakdown of the simulation. As inversion can never be avoided and sometimes is even the correct behavior of elements, there is a strong need for stable inversion handling. In this paper, we propose a novel method to resolve inverted elements which is motivated by previous work of Irving et al. [6]. In combination with an efficient handling of degenerated elements, our approach yields a stable simulation of arbitrary deformations. Although we focus on the corotational formulation of linear FEM, the method can be implemented within arbitrary constitutive models.

3D numerical deformation model of the intrusive event forerunning the 2001 Etna eruption

3D finite elements models were carried out in order to evaluate the ground deformation produced by the dike intrusion occurred at Etna volcano in the 2001. The finite element method (FEM) allows for considering the medium heterogeneity and the real topography of the volcano. Firstly, we validated the method in a homogeneous elastic half-space and compared the results with those obtained from analytical dislocation models. We performed a convergence analysis to quantify the discretization errors, which are sensitive to the size and quality of the mesh elements and can cause inaccurate numerical solutions. Secondly, several numerical models were conducted to appreciate how the complex distribution of elastic medium parameters and the topography make the numerical results differ from the simple analytical solutions. The numerical model, which account for the real topography of Etna volcano and medium heterogeneity, well match the ground deformation observed at the GPS stations between 11 and 16 July 2001. It appears that topography and heterogeneities provide more details about the ground deformation near the summit crater area that are missed when homogeneous half-space models are used. However, only few discrepancies between the analytical and the numerical models are observed at the GPS stations, which coarsely sample the volcanic edifice far away from the summit craters.

Activity Representation Using 3D Shape Models

We present a method for characterizing human activities using 3D deformable shape models. The motion trajectories of points extracted from objects involved in the activity are used to build models for each activity, and these models are used for classification and detection of unusual activities. The deformable models are learnt using the factorization theorem for nonrigid 3D models. We present a theory for characterizing the degree of deformation in the 3D models from a sequence of tracked observations. This degree, termed as deformation index (DI), is used as an input to the 3D model estimation process. We study the special case of ground plane activities in detail because of its importance in video surveillance applications. We present results of our activity modeling approach using videos of both high-resolution single individual activities and ground plane surveillance activities.

Facial movement analysis in ASL

In the age of speech and voice recognition technologies, sign language recognition is an essential part of ensuring equal access for deaf people. To date, sign language recognition research has mostly ignored facial expressions that arise as part of a natural sign language discourse, even though they carry important grammatical and prosodic information. One reason is that tracking the motion and dynamics of expressions in human faces from video is a hard task, especially with the high number of occlusions from the signers’ hands. This paper presents a 3D deformable model tracking system to address this problem, and applies it to sequences of native signers, taken from the National Center of Sign Language and Gesture Resources (NCSLGR), with a special emphasis on outlier rejection methods to handle occlusions. The experiments conducted in this paper validate the output of the face tracker against expert human annotations of the NCSLGR corpus, demonstrate the promise of the proposed face tracking framework for sign language data, and reveal that the tracking framework picks up properties that ideally complement human annotations for linguistic research.

Estimate of the interchamber pillar stability based on the damage accumulation criterion

The 3D viscoelastic deformation model has been developed for estimating stability of pillars based on the criterion of damage accumulation. It is proposed to determine parameters of the state-of-damage function by the data of the triaxial creep test of samples. The dependence of pillar longevity on the value of the tectonic stresses is proved.

Surface Deformation Models for Nonrigid 3D Shape Recovery

Three-dimensional detection and shape recovery of a nonrigid surface from video sequences require deformation models to effectively take advantage of potentially noisy image data. Here, we introduce an approach to creating such models for deformable 3D surfaces. We exploit the fact that the shape of an inextensible triangulated mesh can be parameterized in terms of a small subset of the angles between its facets. We use this set of angles to create a representative set of potential shapes, which we feed to a simple dimensionality reduction technique to produce low-dimensional 3D deformation models. We show that these models can be used to accurately model a wide range of deforming 3D surfaces from video sequences acquired under realistic conditions.

Towards a measure of deformability of shape sequences

In this paper we develop a theory for characterizing how deformable a shape is given a sequence of its observations. We define a term called “deformability index” (DI) for shapes. The novelty of the proposed method lies in its ability to separate the effects of observation noise from the underlying non-rigid deformation process. The DI is computed from the tracked positions of a sequence of deformable shapes, using an affine camera model. Our method assumes that a deformable shape sequence can be represented by a linear combination of rigid basis shapes, where the weights assigned to each basis shape change with time. The tracked points obtained from the 2D shape sequence are transformed to a 3D shape space, whose dimension is then estimated using spectral analysis methods. The dimension of this shape space determines the number of basis shapes needed to represent the shape sequence, which, in turn, determines the deformability index. Our method is different from existing techniques since it is non-iterative, does not require setting an arbitrary threshold and is able to precisely model the effects of noise in the feature positions. Rigid 3D transformations of the shape are taken into account in estimating the DI; however, the method does not require estimation of 3D structure or motion. Experimental results show that the DI is in accordance with our intuitive judgment. Applications and comparative analysis on 3D deformable object modeling are also presented.

SU‐FF‐I‐58: A Software Toolkit for Multi‐Image Registration and Segmentation in IGRT and ART

Purpose: Extracting information from images for image‐guided and adaptive radiation therapy usually involves segmentation and/or registration with a reference image. The purpose of this research was to develop and make freely available a software toolkit for research use that combines several registration and segmentation methods and is applicable to problems involving multiple sequential images. Method and Materials: Two automatic approaches have been integrated with standard interactive methods. One automatic approach solves PDEs describing visco‐elastic flow to register a set of target images with a reference image. Segmentations can then be transferred from the reference image to target images via corresponding deformation fields. The second method segments images via Bayesian posterior optimization of a 3D deformable shape model called an m‐rep. User efficiency has been improved by the ability to queue, view, work with, and compare multiple images and segmentations during a single session. The software was developed in C++ and runs on Windows and Linux. Results: The GUI, including a window with panes for standard orthogonal 2D views linked to a second 3D window, preserves a common look and feel for all three approaches. A set of treatment images of the same patient initially are rigidly registered to a reference image. The images are then segmented by the methods of choice. Unsatisfactory automatic results are edited with conventional methods. Registered images and segmentations are output in DICOM RT format. Conclusion: The toolkit combines multiple approaches for image registration and segmentation in a system specifically designed to support research on IGRT and ART. A cost‐free license to download and use the software is available at http://planunc.radonc.unc.edu.Research supported by NCI/NCRR R01 RR 01861501.

TH-B-M100A-01: Multimodality and 4D Image Registration: Methods and Clinical Use

Acquisition of anatomic and functional data from magnetic resonance imaging and nuclear medicine studies is becoming increasingly common for patient management in radiation therapy. These data can help improve tumor localization and normal tissue delineation for treatment planning and may provide information about treatment efficacy during or after a course of radiotherapy. Time series data from serial and 4D CT before and during the treatment course, including CT data acquired in the treatment room at the time of treatment, is also helping to estimate motion and shape changes of relevant anatomy which can be used to accumulate dose and possibly adapt treatments. In order to fully realize the benefits of the multitude of data we can now acquire, the different imaging studies must be registered to each other or to a common coordinate system. The geometric transformation required to register the image data can range from simple rotate-translate to account for differences in patient orientation to 3D or 4D deformation models to account for changes in internal anatomy during and over the course of therapy. Once registered, data derived from the various studies such as anatomic outlines and computed dose can be integrated or fused to help construct a more complete and accurate representation of the patient. This lecture will focus on the mechanics of registering and displaying data from different imaging studies using distinct modalities or a single modality over time. A description and taxonomy of the different methods will be described. Methods for display and interaction with multimodality data will also be presented. The overall goal is to provide the basic knowledge required to understand what is happening “under-the-hood” of the different registration systems one might encounter in the clinic, the different ways these systems are being used and their limitations. Educational Objectives: 1. Understand the basic mechanics of multi-modality and 4D image registration techniques. 2. Understand the different techniques used to combine, display & interact with multimodality/4D image and dose data. 3. Understand the clinical use and limitations of these techniques for Tx planning, Tx delivery and plan adaptation.

Deformable model retrieval based on topological and geometric signatures.

With the increasing popularity of 3D applications such as computer games, a lot of 3D geometry models are being created. To encourage sharing and reuse, techniques that support matching and retrieval of these models are emerging. However, only a few of them can handle deformable models, that is, models of different poses, and these methods are generally very slow. In this paper, we present a novel method for efficient matching and retrieval of 3D deformable models. Our research idea stresses using both topological and geometric features at the same time. First, we propose Topological Point Ring (TPR) analysis to locate reliable topological points and rings. Second, we capture both local and global geometric information to characterize each of these topological features. To compare the similarity of two models, we adapt the Earth Mover Distance (EMD) as the distance function and construct an indexing tree to accelerate the retrieval process. We demonstrate the performance of the new method, both in terms of accuracy and speed, through a large number of experiments.

A nonlinear elastic deformable template for soft structure segmentation: application to the heart segmentation in MRI

This paper proposes a nonlinear 3D deformable model for the image segmentation of soft structures. The template is modelled as an elastic body which is deformed by forces derived from the image. It relies on a template, which is a topological, geometrical and material model of the structure to segment. This model is based on the nonlinear three-dimensional elasticity problem with a boundary condition of pure traction. In addition, the applied forces are successive, as they depend on the displacements. For computations, an incremental algorithm is proposed to minimize the global energy of template deformation. Sufficient conditions of the convergence for the incremental algorithm are given. Finally, a discrete algorithm using the finite element method is presented and evaluated on synthetic images and actual MR images of mouse hearts.

The discrete modal transform and its application to lossy image compression

This paper introduces the discrete modal transform (DMT), a 1D and 2D discrete, non-separable transform for signal processing, which, in the mathematical sense, is a generalization of the well-known discrete cosine transform (DCT). A 3D deformable surface model is used to represent the image intensity and the introduced discrete transform is a by-product of the explicit surface deformation governing equations. The properties of the proposed transform are similar to those of the DCT. To illustrate these properties, the proposed transform is applied to lossy image compression and the obtained results are compared to those of a DCT-based compression scheme. Experimental results show that DMT, which includes an embedded compression ratio selection mechanism, has excellent energy compaction properties and achieves comparable compression results to DCT at low compression ratios, while being in general better than DCT at high compression ratios.

2D vs. 3D Deformable Face Models: Representational Power, Construction, and Real-Time Fitting

Model-based face analysis is a general paradigm with applications that include face recognition, expression recognition, lip-reading, head pose estimation, and gaze estimation. A face model is first constructed from a collection of training data, either 2D images or 3D range scans. The face model is then fit to the input image(s) and the model parameters used in whatever the application is. Most existing face models can be classified as either 2D (e.g. Active Appearance Models) or 3D (e.g. Morphable Models). In this paper we compare 2D and 3D face models along three axes: (1) representational power, (2) construction, and (3) real-time fitting. For each axis in turn, we outline the differences that result from using a 2D or a 3D face model.

Interactive deformable geometry maps

Haptics on 3D deformable models is a challenge because of the inevitable and expensive 3D deformation computation. In this paper, we propose a new technique that extends the conventional rigid geometry images approach proposed by Gu et al. [9]. Our approach not only flattens the geometry, but also helps to accomplish deformation in an effective and efficient manner. Our approach is suitable for haptics computing, as it performs the deformation on the geometry map itself thereby avoiding the expensive 3D deformation computation. We demonstrate construction of the deformable geometry map representation and its application utilizing practical methods for interactive surgery simulation and interactive textile simulation.

SU‐FF‐J‐111: Quantification of Patient Specific Target Motion in the Presence of Limited Data Through Population Models

Purpose: Integrate 2D cine imaging into 3D patient population models of physiological motion for patient specific quantification of deformation. Method and Materials: Inhale and exhale CT images of the liver were obtained for 5 patients under an REB approved protocol. A finite element model was constructed from the exhale contours of one patients liver and deformed into the exhale then inhale position of each patient, providing a common model of the 3D deformation of each patient's respiration, using Morfeus, a finite element‐based deformable registration algorithm. Average deformation maps were computed for all combinations of 4 patients. Simulated 2D coronal cine images were generated from the exhale and inhale data of each patient. Narrow 2D channels were used to obtain image intensity data at the dome of the liver and the inferior tip of the liver on the exhale image and at the same location on the inhale image. A least squares fit was performed to align the image intensities in the superior‐inferior direction, providing an estimate of the coronal motion of the liver. The motion from the channel was applied to the population deformation model, providing patient specific refinement by scaling the motion of the population model by the ratio of the motion at the navigator for the specific patient to the motion at the navigator for the population model. Results: The average superior‐inferior displacement difference over the liver volume between the 3D deformation model via Morfeus and the patient‐refined population model was −0.12 cm (SD 0.15 cm). The average standard deviation across the five patients was 0.38 cm. Conclusion: Population models of volumetric organ deformation due to physiological motion can be used to predict patient specific motion when limited data is available, such as with 2D cine data. Conflict of Interest: Research supported in part by Varian Medical Systems.

Fitting 3D face models for tracking and active appearance model training

In this paper, we consider fitting a 3D deformable face model to continuous video sequences for the tasks of tracking and training. We propose two appearance-based methods that only require a simple statistical facial texture model and do not require any information about an empirical or analytical gradient matrix, since the best search directions are estimated on the fly. The first method computes the fitting using a locally exhaustive and directed search where the 3D head pose and the facial actions are simultaneously estimated. The second method decouples the estimation of these parameters. It computes the 3D head pose using a robust feature-based pose estimator incorporating a facial texture consistency measure. Then, it estimates the facial actions with an exhaustive and directed search. Fitting and tracking experiments demonstrate the feasibility and usefulness of the developed methods. A performance evaluation also shows that the proposed methods can outperform the fitting based on an active appearance model search adopting a pre-computed gradient matrix. Although the proposed schemes are not as fast as the schemes adopting a directed continuous search, they can tackle many disadvantages associated with such approaches.

Implicit meshes for surface reconstruction

Deformable 3D models can be represented either as traditional explicit surfaces, such as triangulated meshes, or as implicit surfaces. Explicit surfaces are widely accepted because they are simple to deform and render, but fitting them involves minimizing a nondifferentiable distance function. By contrast, implicit surfaces allow fitting by minimizing a differentiable algebraic distance, but are harder to meaningfully deform and render. Here, we propose a method that combines the strength of both approaches. It relies on a technique that can turn a completely arbitrary triangulated mesh, such as one taken from the Web, into an implicit surface that closely approximates it and can deform in tandem with it. This allows both automated algorithms to take advantage of the attractive properties of implicit surfaces for fitting purposes and people to use standard deformation tools they feel comfortable for interaction and animation purposes. We demonstrate the applicability of our technique to modeling the human upper-body, including face, neck, shoulders, and ears, from noisy stereo and silhouette data.

Pose and Illumination Invariant Face Recognition Based on 3D Face Reconstruction

Pose and illumination changings from picture to picture are two main barriers toward full automatic face recognition. In this paper, a novel method to handle both pose and lighting conditions simultaneously is proposed, which calibrates the pose and lighting to a predefined reference condition through an illumination invariant 3D face reconstruction. First, some located facial landmarks and a priori statistical deformable 3D model are used to recover an elaborate 3D shape. Based on the recovered 3D shape, the texture calibrated to a standard illumination is generated by spherical harmonics ratio image and finally the illumination independent 3D face is reconstructed completely. The proposed method combines the strength of statistical deformable model to describe the shape information and the compact representations of the illumination in spherical frequency space, and handles both the pose and illumination variation simultaneously. This algorithm can be used to synthesize virtual views of a given face image and enhance the performance of face recognition. Experimental results on CMU PIE database show that this method can significantly improve the accuracy of the existing face recognition method when pose and illumination are inconsistent between gallery and probe sets.