Deformable Shape Model Research Articles

BOSS: Bones, organs and skin shape model

A virtual anatomical model of a patient can be a valuable tool for enhancing clinical tasks such as workflow automation, patient-specific X-ray dose optimization, markerless tracking, positioning, and navigation assistance in image-guided interventions. For these tasks, it is highly desirable that the patient’s surface and internal organs are of high quality for any pose and shape estimate. At present, the majority of statistical shape models (SSMs) are restricted to a small number of organs or bones or do not adequately represent the general population. To address this, we propose a deformable human shape and pose model that combines skin, internal organs, and bones, learned from CT images. By modeling the statistical variations in a pose-normalized space using probabilistic PCA while also preserving joint kinematics, our approach offers a holistic representation of the body that can be beneficial for automation in various medical applications. In an interventional setup, our model could, for example, facilitate automatic system/patient positioning, organ-specific iso-centering, automated collimation or collision prediction. We assessed our model’s performance on a registered dataset, utilizing the unified shape space, and noted an average error of 3.6 mm for bones and 8.8 mm for organs. By utilizing solely skin surface data or patient metadata like height and weight, we find that the overall combined error for bone-organ measurement is 8.68 mm and 8.11 mm, respectively. To further verify our findings, we conducted additional tests on publicly available datasets with multi-part segmentations, which confirmed the effectiveness of our model. In the diverse TotalSegmentator dataset, the errors for bones and organs are observed to be 5.10mm and 8.72mm, respectively. Our work shows that anatomically parameterized statistical shape models can be created accurately and in a computationally efficient manner. The proposed approach enables the construction of shape models that can be directly integrated into to various medical applications.

Superadditivity and Convex Optimization for Globally Optimal Cell Segmentation Using Deformable Shape Models.

Cell nuclei segmentation is challenging due to shape variation and closely clustered or partially overlapping objects. Most previous methods are not globally optimal, limited to elliptical models, or are computationally expensive. In this work, we introduce a globally optimal approach based on deformable shape models and global energy minimization for cell nuclei segmentation and cluster splitting. We propose an implicit parameterization of deformable shape models and show that it leads to a convex energy. Convex energy minimization yields the global solution independently of the initialization, is fast, and robust. To jointly perform cell nuclei segmentation and cluster splitting, we developed a novel iterative global energy minimization method, which leverages the inherent property of superadditivity of the convex energy. This property exploits the lower bound of the energy of the union of the models and improves the computational efficiency. Our method provably determines a solution close to global optimality. In addition, we derive a closed-form solution of the proposed global minimization based on the superadditivity property for non-clustered cell nuclei. We evaluated our method using fluorescence microscopy images of five different cell types comprising various challenges, and performed a quantitative comparison with previous methods. Our method achieved state-of-the-art or improved performance.

Snake Game Using Hand Recognition System

Abstract: This paper presents engineering and execution nuances of a hand's movement furthermore motion acknowledgment framework which has been created at Meerut Institute of Engineering and Technology. The framework utilizes live 15-piece shading video from an arranged camera, runs in genuine time (On a DEC Alpha, 25 edges per second), and adapts well to foundation mess. Following is accomplished utilizing the 2D deformable Active Shape Models (savvy snakes), what's more a genetic calculation is utilized to play out an underlying worldwide picture search. The Point Distribution Model, which was used for both the snakes and the hereditary calculations, is a nonexclusive and extensible model that can be used to follow any 2D deformable object. Keywords: OpenCv, Feature tracking, Saavy Snakes, Genetic Algorithm, Lightning issues.

Semantic keypoint-based pose estimation from single RGB frames

This paper presents an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. Unlike prior investigators, we are agnostic to whether the object is textured or textureless, as the convnet learns the optimal representation from the available training-image data. Furthermore, the approach can be applied to instanceand class-based pose recovery. Additionally, we accompany our main pipeline with a technique for semi-automatic data generation from unlabeled videos. This procedure allows us to train the learnable components of our method with minimal manual intervention in the labeling process. Empirically, we show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios even against a cluttered background. We apply our approach both to several, existing, large-scale datasets - including PASCAL3D+, LineMOD-Occluded, YCB-Video, and TUD-Light - and, using our labeling pipeline, to a new dataset with novel object classes that we introduce here. Extensive empirical evaluations show that our approach is able to provide pose estimation results comparable to the state of the art.

Finsler geometry modeling and Monte Carlo study of skyrmion shape deformation by uniaxial stress

Skyrmions in chiral magnetic materials are topologically stable and energetically balanced spin configurations appearing under the presence of ferromagnetic interaction (FMI) and Dzyaloshinskii-Moriya interaction (DMI). Much of the current interest has focused on the effects of magneto-elastic coupling on these interactions under mechanical stimuli, such as uniaxial stresses for future applications in spintronics devices. Recent studies suggest that skyrmion shape deformations in thin films are attributed to an anisotropy in the coefficient of DMI, such that $D_{x}\!\not=\!D_{y}$, which makes the ratio $\lambda/D$ anistropic, where the coefficient of FMI $\lambda$ is isotropic. It is also possible that $\lambda_{x}\!\not=\!\lambda_{y}$ while $D$ is isotropic for $\lambda/D$ to be anisotropic. In this paper, we study this problem using a new modeling technique constructed based on Finsler geometry (FG). Two possible FG models are examined: In the first (second) model, the FG modeling prescription is applied to the FMI (DMI) Hamiltonian. We find that these two different FG models' results are consistent with the reported experimental data for skyrmion deformation. We also study responses of helical spin orders under lattice deformations corresponding to uniaxial extension/compression and find a clear difference between these two models in the stripe phase, elucidating which interaction of FMI and DMI is deformed to be anisotropic by uniaxial stresses.

Estimating Reference Shape Model for Personalized Surgical Reconstruction of Craniomaxillofacial Defects.

To estimate a patient-specific reference bone shape model for a patient with craniomaxillofacial (CMF) defects due to facial trauma. We proposed an automatic facial bone shape estimation framework using pre-traumatic conventional portrait photos and post-traumatic head computed tomography (CT) scans via a 3D face reconstruction and a deformable shape model. Specifically, a three-dimensional (3D) face was first reconstructed from the patient's pre-traumatic portrait photos. Second, a correlation model between the skin and bone surfaces was constructed using a sparse representation based on the CT images of training normal subjects. Third, by feeding the reconstructed 3D face into the correlation model, an initial reference shape model was generated. In addition, we refined the initial estimation by applying non-rigid surface matching between the initially estimated shape and the patient's post-traumatic bone based on the adaptive-focus deformable shape model (AFDSM). Furthermore, a statistical shape model, built from the training normal subjects, was utilized to constrain the deformation process to avoid overfitting. The proposed method was evaluated using both synthetic and real patient data. Experimental results show that the patient's abnormal facial bony structure can be recovered using our method, and the estimated reference shape model is considered clinically acceptable by an experienced CMF surgeon. The proposed method is more suitable to the complex CMF defects for CMF reconstructive surgical planning.

Epithelial Layer Estimation Using Curvatures and Textural Features for Dysplastic Tissue Detection

Boundary effect in digital pathology is a phenomenon where the tissue shapes of biopsy samples get distorted during the sampling process. The morphological pattern of an epithelial layer is greatly affected. Theoretically, the shape deformation model can normalise the distortions, but it needs a 2D image. Curvatures theory, on the other hand, is not yet tested on digital pathology images. Therefore, this work proposed a curvature detection to reduce the boundary effects and estimates the epithelial layer. The boundary effect on the tissue surfaces is normalised using the frequency of a curve deviates from being a straight line. The epithelial layer’s depth is estimated from the tissue edges and the connected nucleolus only. Then, the textural and spatial features along the estimated layer are used for dysplastic tissue detection. The proposed method achieved better performance compared to the whole tissue regions in terms of detecting dysplastic tissue. The result shows a leap of kappa points from fair to a substantial agreement with the expert’s ground truth classification. The improved results demonstrate that curvatures have been effective in reducing the boundary effects on the epithelial layer of tissue. Thus, quantifying and classifying the morphological patterns for dysplasia can be automated. The textural and spatial features on the detected epithelial layer can capture the changes in tissue.

Suppression of soft spots and excited modes in the shape deformation model with spatio-temporal growth noise

The shapes of plant organs such as sepals are precise and reproducible, although the cells composing these organs are highly heterogeneous. We investigate the collective behavior of a shape deformation model with spatio-temporal fluctuations in elasticity (Hong et al., Dev. Cell, 38, 15, 2016). It is shown that the spatio-temporal growth noise has two opposing effect: the temporal noise promotes the reproducibility of the organ shape, whereas the spatial noise disturbs it, leading to the organ undergoing an abnormal shape deformation. To understand how such abnormality can emerge, we study the eigenvectors of the correlation matrix of the growth velocity. It is revealed that the anomalous shape deformations can be created by two independent factors: (1) excited modes where the orientations of the eigenvectors with m ≥ 2 are spatially correlated, and/or (2) soft spots where spot structures with high fluctuations in the magnitudes of the eigenvectors appear non-uniformly.

Mathematical modeling of skyrmion shape deformation under uni-axial stresses

Skyrmion is a topologically stable spin texture and expected to be applied to the future computer memory. On the semiconductors such as FeGe and MnSi, the skyrmion configuration is stable in the sense that it is not strongly affected by a small variation of external stimuli such as temperature, magnetic field etc. In recent experiments, it was reported that the skymion shape is deformed from isotropic (or circular shape) to anisotropic (or elliptic shape) by an external mechanical stress. This shape deformation is caused by the so-called “strain-induced anisotropy (SIA)” of Dzyaloshinskii-Moriya interaction (DMI). In this presentation, we study the reason why this SIA appears in the DMI coefficient on the basis of Finsler geometry modeling technique by introducing a microscopic strain field τ, which is caused by or interacts with the applied external mechanical force.

Eyelid and Pupil Landmark Detection and Blink Estimation Based on Deformable Shape Models for Near-Field Infrared Video

The eyelid contour, pupil contour and blink event are important features of eye activity, and their estimation is a crucial research area for emerging wearable camera-based eyewear in a wide range of applications e.g. mental state estimation. Current approaches often estimate a single eye activity, such as blink or pupil center, from far-field and non-infrared (IR) eye images, and often depend on the knowledge of other eye components. This paper presents a unified approach to simultaneously estimate the landmarks for the eyelids, the iris and the pupil, and detect blink from near-field IR eye images based on a statistically learned deformable shape model and local appearance. Unlike the facial landmark estimation problem, by comparison, different shape models are applied to all eye states – closed eye, open eye with iris visible, and open eye with iris and pupil visible – to deal with the self-occluding interactions among the eye components. The most likely eye state is determined based on the learned local appearance. Evaluation on three different realistic datasets demonstrates that the proposed three-state deformable shape model achieves state-of-the-art performance for the open eye with iris and pupil state, where the normalized error was lower than 0.04. Blink detection can be as high as 90% in recall performance, without direct use of pupil detection. Cross-corpus evaluation results show that the proposed method improves on the state-of-the-art eyelid detection algorithm. This unified approach greatly facilitates eye activity analysis for research and practice when different types of eye activity are required rather than employ different techniques for each type. Our work is the first study proposing a unified approach for eye activity estimation from near-field IR eye images and achieved the state-of-the-art eyelid estimation and blink detection performance.

Normalization of face illumination using basic knowledge and information extracted from a single image

This paper presents a method for face image normalization that can be applied to the extraction of illumination invariant facial features or used to remove bad lighting effects and produce high-quality, photorealistic results. Most of the existing approaches concentrate on separating the constant albedo from the variable light intensity; that concept, however, is based on the Lambertian model, which fails in the presence of specularities and cast shadows. Therefore, to tackle this problem, various methods use bootstrap sets to learn the reflectance model of a given person or a face in general. Unfortunately, algorithms of this type are usually not practical for real applications. The proposed approach does not require any training procedure, as the normalization is performed only on the basis of information that is contained in the image that is being processed. Still, external knowledge that is represented by the deformable shape model is employed to execute the localization of the important points, such as eye centers. Additionally, an assumption regarding the symmetry of the face is utilized. The rest of the normalization algorithm relies mostly on simple image processing techniques. Experiments on the CMU-PIE and Extended Yale B databases show that the new method not only provides satisfactory recognition results but is also able to generate natural-looking images.

Robust brain ROI segmentation by deformation regression and deformable shape model.

We propose a robust and efficient learning-based deformable model for segmenting regions of interest (ROIs) from structural MR brain images. Different from the conventional deformable-model-based methods that deform a shape model locally around the initialization location, we learn an image-based regressor to guide the deformable model to fit for the target ROI. Specifically, given any voxel in a new image, the image-based regressor can predict the displacement vector from this voxel towards the boundary of target ROI, which can be used to guide the deformable segmentation. By predicting the displacement vector maps for the whole image, our deformable model is able to use multiple non-boundary predictions to jointly determine and iteratively converge the initial shape model to the target ROI boundary, which is more robust to the local prediction error and initialization. In addition, by introducing the prior shape model, our segmentation avoids the isolated segmentations as often occurred in the previous multi-atlas-based methods. In order to learn an image-based regressor for displacement vector prediction, we adopt the following novel strategies in the learning procedure: (1) a joint classification and regression random forest is proposed to learn an image-based regressor together with an ROI classifier in a multi-task manner; (2) high-level context features are extracted from intermediate (estimated) displacement vector and classification maps to enforce the relationship between predicted displacement vectors at neighboring voxels. To validate our method, we compare it with the state-of-the-art multi-atlas-based methods and other learning-based methods on three public brain MR datasets. The results consistently show that our method is better in terms of both segmentation accuracy and computational efficiency.

Compactly-supported smooth interpolators for shape modeling with varying resolution

In applications that involve interactive curve and surface modeling, the intuitive manipulation of shapes is crucial. For instance, user interaction is facilitated if a geometrical object can be manipulated through control points that interpolate the shape itself. Additionally, models for shape representation often need to provide local shape control and they need to be able to reproduce common shape primitives such as ellipsoids, spheres, cylinders, or tori. We present a general framework to construct families of compactly-supported interpolators that are piecewise-exponential polynomial. They can be designed to satisfy regularity constraints of any order and they enable one to build parametric deformable shape models by suitable linear combinations of interpolators. They allow to change the resolution of shapes based on the refinability of B-splines. We illustrate their use on examples to construct shape models that involve curves and surfaces with applications to interactive modeling and character design.

DETECTION AND 3D MODELLING OF VEHICLES FROM TERRESTRIAL STEREO IMAGE PAIRS

Abstract. The detection and pose estimation of vehicles plays an important role for automated and autonomous moving objects e.g. in autonomous driving environments. We tackle that problem on the basis of street level stereo images, obtained from a moving vehicle. Processing every stereo pair individually, our approach is divided into two subsequent steps: the vehicle detection and the modelling step. For the detection, we make use of the 3D stereo information and incorporate geometric assumptions on vehicle inherent properties in a firstly applied generic 3D object detection. By combining our generic detection approach with a state of the art vehicle detector, we are able to achieve satisfying detection results with values for completeness and correctness up to more than 86%. By fitting an object specific vehicle model into the vehicle detections, we are able to reconstruct the vehicles in 3D and to derive pose estimations as well as shape parameters for each vehicle. To deal with the intra-class variability of vehicles, we make use of a deformable 3D active shape model learned from 3D CAD vehicle data in our model fitting approach. While we achieve encouraging values up to 67.2% for correct position estimations, we are facing larger problems concerning the orientation estimation. The evaluation is done by using the object detection and orientation estimation benchmark of the KITTI dataset (Geiger et al., 2012).

In-Plane Shape-Deviation Modeling and Compensation for Fused Deposition Modeling Processes

Additive manufacturing (AM) or 3-D printing refers to a new class of technologies that actively construct products directly from any 3-D digital model. In the future, the broader applications of AM will require a cost reduction of AM machines. Currently, the products fabricated by low-end machines, such as those fabricated using fused deposition modeling (FDM) processes, suffer from the issue of low dimensional accuracy due to multiple error sources. To properly manage error sources for improved prevision, this paper proposes a novel strategy for error compensation in the FDM processes. First, we consecutively attribute the dimensional inaccuracy to two major error sources that affect the geometric shape of the product: 1) positioning error of the extruder and 2) shape deformation induced by processing error, including material phase change and other variations that occur. The extruder positioning error is characterized by a Kriging model, while the modeling of shape deformation due to processing error follows the method developed by Huang et al. Second, using error equivalence concept, we transform the positioning error into the equivalent amount of design input error. Finally, we adjust the design to compensate for the overall shape deviation. To validate this strategy, we conduct a designed experiment for the shape deviation prediction and the compensation. The experimental results successfully demonstrate the effectiveness of the proposed three-step strategy to manage multiple error sources in the FDM processes.

Multisensor X-ray inspection of internal defects in horticultural products

A combination of 3-D vision and X-ray radiography is proposed to enable low-cost, generally applicable online inspection of internal quality of horticultural and potentially other products. The underlying concept assumes that the shape of the product is known beforehand through a deformable shape model. A 3-D vision system is used in combination with the shape model to accurately determine the complete outer surface shape of the sample. This shape is voxelized to generate a reference product from which a X-ray radiograph is simulated to be compared with a measured radiograph, hence revealing the presence of any defects or disorders. Advantages of this method are that small deviations in internal density are detected easily since the cumulative information of the bulk object shape is removed. Furthermore, no specific detection algorithms have to be developed for different types of defects, since the method will directly identify deviations from the ideal. Validation on two datasets and comparison with two reference detections methods (classical image processing and a human operator) shows that the proposed method reliably (accuracy >99%) detect defects larger than 3.5mm radius with densities differences between sample and defects as small as 10%. Voids are reliably (accuracy >99%) detected down to a radius of 1.5mm, corresponding to a volume of less than 0.02cm3.

Prescriptive Modeling and Compensation of In-Plane Shape Deformation for 3-D Printed Freeform Products

Although 3-D printing or additive manufacturing (AM) holds great promise as a direct manufacturing technology, the geometric accuracy of AM built products remains a critical issue, especially for freeform products with complex geometric shapes. Efforts have long been attempted to improve the accuracy of AM built freeform products. But there is a lack of generic and prescriptive methodology transparent to specific designs and AM processes. This paper fills the gap by establishing the methodology to predict and compensate the in-plane ( $x-y$ plane) geometric deformation of AM built freeform products based on a limited number of simple trial shapes. Built upon our previous predictive model and optimal compensation study for the cylinder and polyhedron shapes, this paper makes a breakthrough by directly controlling arbitrary freeform shape deformation from computer-aided design. Experimental investigation using stereolithography process successfully validates the proposed prescriptive modeling and compensation methodology. This paper provides the prospect of proactively improving printing accuracy of arbitrary products built by a variety of AM processes.

Sparse Representation for 3D Shape Estimation: A Convex Relaxation Approach.

We investigate the problem of estimating the 3D shape of an object defined by a set of 3D landmarks, given their 2D correspondences in a single image. A successful approach to alleviating the reconstruction ambiguity is the 3D deformable shape model and a sparse representation is often used to capture complex shape variability. But the model inference is still challenging due to the nonconvexity in the joint optimization of shape and viewpoint. In contrast to prior work that relies on an alternating scheme whose solution depends on initialization, we propose a convex approach to addressing this challenge and develop an efficient algorithm to solve the proposed convex program. We further propose a robust model to handle gross errors in the 2D correspondences. We demonstrate the exact recovery property of the proposed method, the advantage compared to several nonconvex baselines and the applicability to recover 3D human poses and car models from single images.

Sequentially adaptive active appearance model with regression-based online reference appearance template

Statistically motivated approaches, such as the active appearance model (AAM), have been widely used for non-rigid objects registration and tracking. As an extension of AAM, sequential AAM (SAAM) was proposed, in which both an incremental updated component and a reference component were employed simultaneously in the fitting scheme. To make SAAM more adaptive to facial context variations during tracking, a regression-based online reference appearance model (ORAM) is presented to update the subject-specific appearance of the SAAM. The spatial map between scattered local feature correspondences and structured landmark correspondences is learned via Kernel Ridge Regression (KRR). Additionally, a shape deformation and appearance model evaluation strategies help to improve the accuracy and efficiency of the algorithm. The approach is experimentally validated by tracking face videos with improved fitting accuracy.

Face Landmark Fitting via Optimized Part Mixtures and Cascaded Deformable Model.

This paper addresses the problem of facial landmark localization and tracking from a single camera. We present a two-stage cascaded deformable shape model to effectively and efficiently localize facial landmarks with large head pose variations. In initialization stage, we propose a group sparse optimized mixture model to automatically select the most salient facial landmarks. By introducing 3D face shape model, we apply procrustes analysis to provide pose-aware landmark initialization. In landmark localization stage, the first step uses mean-shift local search with constrained local model to rapidly approach the global optimum. The second step uses component-wise active contours to discriminatively refine the subtle shape variation. Our framework simultaneously handles face detection, pose-robust landmark localization and tracking in real time. Extensive experiments are conducted on both laboratory environmental databases and face-in-the-wild databases. The results reveal that our approach consistently outperforms state-of-the-art methods for face alignment and tracking.