Generic Shape Model Research Articles

Towards multi-view consistency in neural ray fields using parametric medial surfaces

Deep learning methods are revolutionizing the solutions to visual computing problems, such as shape retrieval and generative shape modeling, but require novel shape representations that are both fast and differentiable. Neural ray fields and their improved rendering performance are promising in this regard, but struggle with a reduced fidelity and multi-view consistency when compared to the more studied coordinate-based methods which, however, are slower in training and evaluation. We propose PMARF, an improved ray field which explicitly models the skeleton of the target shape as a set of (0-thickness) parametric medial surfaces. This formulation reduces by construction the degrees-of-freedom available in the reconstruction domain, improving multi-view consistency even from sparse training views. This in turn improves fidelity while facilitating a reduction in the network size.

A Generative Shape Compositional Framework to Synthesize Populations of Virtual Chimeras.

Generating virtual organ populations that capture sufficient variability while remaining plausible is essential to conduct in silico trials (ISTs) of medical devices. However, not all anatomical shapes of interest are always available for each individual in a population. The imaging examinations and modalities used can vary between subjects depending on their individualized clinical pathways. Different imaging modalities may have various fields of view and are sensitive to signals from other tissues/organs, or both. Hence, missing/partially overlapping anatomical information is often available across individuals. We introduce a generative shape model for multipart anatomical structures, learnable from sets of unpaired datasets, i.e., where each substructure in the shape assembly comes from datasets with missing or partially overlapping substructures from disjoint subjects of the same population. The proposed generative model can synthesize complete multipart shape assemblies coined virtual chimeras (VCs). We applied this framework to build VCs from databases of whole-heart shape assemblies that each contribute samples for heart substructures. Specifically, we propose a graph neural network-based generative shape compositional framework, which comprises two components, a part-aware generative shape model that captures the variability in shape observed for each structure of interest in the training population and a spatial composition network that assembles/composes the structures synthesized by the former into multipart shape assemblies (i.e., VCs). We also propose a novel self-supervised learning scheme that enables the spatial composition network to be trained with partially overlapping data and weak labels. We trained and validated our approach using shapes of cardiac structures derived from cardiac magnetic resonance (MR) images in the UK Biobank (UKBB). When trained with complete and partially overlapping data, our approach significantly outperforms a principal component analysis (PCA)-based shape model (trained with complete data) in terms of generalizability and specificity. This demonstrates the superiority of the proposed method, as the synthesized cardiac virtual populations are more plausible and capture a greater degree of shape variability than those generated by the PCA-based shape model.

SDFEst: Categorical Pose and Shape Estimation of Objects From RGB-D Using Signed Distance Fields

Rich geometric understanding of the world is an important component of many robotic applications such as planning and manipulation. In this paper, we present a modular pipeline for pose and shape estimation of objects from RGB-D images given their category. The core of our method is a generative shape model, which we integrate with a novel initialization network and a differentiable renderer to enable 6D pose and shape estimation from a single or multiple views. We investigate the use of discretized signed distance fields as an efficient shape representation for fast analysis-by-synthesis optimization. Our modular framework enables multi-view optimization and extensibility. We demonstrate the benefits of our approach over state-of-the-art methods in several experiments on both synthetic and real data. We open-source our approach at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/roym899/sdfest</uri> .

The 3D skull 0–4 years: A validated, generative, statistical shape model

BackgroundThis study aims to capture the 3D shape of the human skull in a healthy paediatric population (0–4 years old) and construct a generative statistical shape model. MethodsThe skull bones of 178 healthy children (55% male, 20.8 ± 12.9 months) were reconstructed from computed tomography (CT) images. 29 anatomical landmarks were placed on the 3D skull reconstructions. Rotation, translation and size were removed, and all skull meshes were placed in dense correspondence using a dimensionless skull mesh template and a non-rigid iterative closest point algorithm. A 3D morphable model (3DMM) was created using principal component analysis, and intrinsically and geometrically validated with anthropometric measurements. Synthetic skull instances were generated exploiting the 3DMM and validated by comparison of the anthropometric measurements with the selected input population. ResultsThe 3DMM of the paediatric skull 0–4 years was successfully constructed. The model was reasonably compact - 90% of the model shape variance was captured within the first 10 principal components. The generalisation error, quantifying the ability of the 3DMM to represent shape instances not encountered during training, was 0.47 mm when all model components were used. The specificity value was <0.7 mm demonstrating that novel skull instances generated by the model are realistic. The 3DMM mean shape was representative of the selected population (differences <2%). Overall, good agreement was observed in the anthropometric measures extracted from the selected population, and compared to normative literature data (max difference in the intertemporal distance) and to the synthetic generated cases. ConclusionThis study presents a reliable statistical shape model of the paediatric skull 0–4 years that adheres to known skull morphometric measures, can accurately represent unseen skull samples not used during model construction and can generate novel realistic skull instances, thus presenting a solution to limited availability of normative data in this field.

Six degree-of-freedom body-fixed hovering over unmapped asteroids via LIDAR altimetry and reinforcement meta-learning

We optimize a six degrees of freedom hovering policy using reinforcement meta-learning. The policy maps flash LIDAR measurements directly to on/off spacecraft body-frame thrust commands, allowing hovering at a fixed position and attitude in the asteroid body-fixed reference frame. Importantly, the policy does not require position and velocity estimates, and can operate in environments with unknown dynamics, and without an asteroid shape model or navigation aids. Indeed, during optimization the agent is confronted with a new randomly generated asteroid for each episode, insuring that it does not learn an asteroid's shape, texture, or environmental dynamics. This allows the deployed policy to generalize well to novel asteroid characteristics, which we demonstrate in our experiments. Moreover, our experiments show that the optimized policy adapts to actuator failure and sensor noise. Although the policy is optimized using randomly generated synthetic asteroids, it is tested on two shape models from actual asteroids: Bennu and Itokawa. We find that the policy generalizes well to these shape models. The hovering controller has the potential to simplify mission planning by allowing asteroid body-fixed hovering immediately upon the spacecraft's arrival to an asteroid. This in turn simplifies shape model generation and allows resource mapping via remote sensing immediately upon arrival at the target asteroid.

Three-dimensional computer modeling of standard orbital mean shape in Asians

Although some data for western norms in orbit shape were reported, the standard norms for Asian orbits were not established yet. The data would be very valuable for the various surgical procedures as well as the production of the appropriate instruments and implants. Therefore, we suggest a Korean orbit mean shape model based on the three-dimensional computer modeling, which includes the analysis of the various parameters with the calculated average value, thereby providing a standard mean shape orbital model that could be used for the Asian patients' orbital surgeries. This paper would be the first literature that provides the standard orbit model for Asians. We developed orbit-specific computer software (AMC-SWUⓇ) for the production of an orbit mean shape model. The production steps included semi-automatic segmentation, shape reconstruction, statistical shape model generation, and mean shape and variance model production. The study included records of 48 male and 48 female patients who met the inclusion criteria. Three-dimensional facial bone computed tomography (CT) images of 96 patients were obtained, and these images were used to produce a representative mean shape model. The mean models had vertical dimensions of 36.93 and 35.11 mm, horizontal dimensions of 38.49 and 36.79 mm, and rim dimensions of 45.76 and 42.90 mm for males and females, respectively. We developed a realistic, visualized three-dimensional Korean orbit mean shape model and compared its parameters with calculated values. There is a variance in orbital dimensions between the sexes and the orbital changes with age. We also demonstrated orbital anatomic differences between ethnic groups.

The deformable most-likely-point paradigm.

In this paper, we present three deformable registration algorithms designed within a paradigm that uses 3D statistical shape models to accomplish two tasks simultaneously: 1) register point features from previously unseen data to a statistically derived shape (e.g., mean shape), and 2) deform the statistically derived shape to estimate the shape represented by the point features. This paradigm, called the deformable most-likely-point paradigm, is motivated by the idea that generative shape models built from available data can be used to estimate previously unseen data. We developed three deformable registration algorithms within this paradigm using statistical shape models built from reliably segmented objects with correspondences. Results from several experiments show that our algorithms produce accurate registrations and reconstructions in a variety of applications with errors up to CT resolution on medical datasets. Our code is available at https://github.com/AyushiSinha/cisstICP.

Robust detection and recognition of traffic signs on road panels

Traffic panels provide vital information on roads with the aid of iconic symbols and text strings. Recognising these signs accurately at the right time is crucial for car drivers to ensure safe journey. The automatic visual recognition and classification of the information contained in the panel could be very useful for driver assistance application. In this paper, a method is proposed to identify and to recognise the information contained in the traffic panels, as an application utilising colour space conversion for image acquisition then the colour decomposition and shape model generation using active appearance model (AAM). The traffic panel is detected using character descriptor-based adaptive fuzzy clustering (AFC). Multiframe detection strategy is a simple way of using precision and recall. Finally the prediction accuracy is computed by using massive training artificial neural network (MTANN). The efficiency of the system is evaluated with the help of the MATLAB-based experimental results which are compared with the bag of visual words for text geolocation and recognising symbols and texts (BTG & ST) method and shape and colour (SC)-based traffic sign detection methods in terms of the sensitivity, specificity and recognition rate.

Robust detection and recognition of traffic signs on road panels

Traffic panels provide vital information on roads with the aid of iconic symbols and text strings. Recognising these signs accurately at the right time is crucial for car drivers to ensure safe journey. The automatic visual recognition and classification of the information contained in the panel could be very useful for driver assistance application. In this paper, a method is proposed to identify and to recognise the information contained in the traffic panels, as an application utilising colour space conversion for image acquisition then the colour decomposition and shape model generation using active appearance model (AAM). The traffic panel is detected using character descriptor-based adaptive fuzzy clustering (AFC). Multiframe detection strategy is a simple way of using precision and recall. Finally the prediction accuracy is computed by using massive training artificial neural network (MTANN). The efficiency of the system is evaluated with the help of the MATLAB-based experimental results which are compared with the bag of visual words for text geolocation and recognising symbols and texts (BTG & ST) method and shape and colour (SC)-based traffic sign detection methods in terms of the sensitivity, specificity and recognition rate.

Statistical model for simulation of deformable elastic endometrial tissue shapes

Statistical shape analysis plays a key role in various medical imaging applications. Such methods provide tools for registering, deforming, comparing, averaging, and modeling anatomical shapes. In this work, we focus on the application of a recent method for statistical shape analysis of parameterized surfaces to simulation of endometrial tissue shapes. The clinical data contains magnetic resonance imaging (MRI) endometrial tissue surfaces, which are used to learn a generative shape model. We generate random tissue shapes from this model, and apply elastic semi-synthetic deformations to them. This provides two types of simulated data: (1) MRI-type (without deformation) and (2) transvaginal ultrasound (TVUS)-type, which undergo an additional deformation due to the transducer׳s pressure. The proposed models can be used for validation of automatic, multimodal image registration, which is a crucial step in diagnosing endometriosis.

A comprehensive statistical framework for elastic shape analysis of 3D faces

We develop a comprehensive statistical framework for analyzing shapes of 3D faces. In particular, we adapt a recent elastic shape analysis framework to the case of hemispherical surfaces, and explore its use in a number of processing applications. This framework provides a parameterization-invariant, elastic Riemannian metric, which allows the development of mathematically rigorous tools for statistical analysis. Specifically, this paper describes methods for registration, comparison and deformation, averaging, computation of covariance and summarization of variability using principal component analysis, random sampling from generative shape models, symmetry analysis, and expression and identity classification. An important aspect of this work is that all tasks are preformed under a unified metric, which has a natural interpretation in terms of bending and stretching of one 3D face to align it with another. We use a subset of the BU-3DFE face dataset, which contains varying magnitudes of expression.

Deformable segmentation via sparse representation and dictionary learning

"Shape" and "appearance", the two pillars of a deformable model, complement each other in object segmentation. In many medical imaging applications, while the low-level appearance information is weak or mis-leading, shape priors play a more important role to guide a correct segmentation, thanks to the strong shape characteristics of biological structures. Recently a novel shape prior modeling method has been proposed based on sparse learning theory. Instead of learning a generative shape model, shape priors are incorporated on-the-fly through the sparse shape composition (SSC). SSC is robust to non-Gaussian errors and still preserves individual shape characteristics even when such characteristics is not statistically significant. Although it seems straightforward to incorporate SSC into a deformable segmentation framework as shape priors, the large-scale sparse optimization of SSC has low runtime efficiency, which cannot satisfy clinical requirements. In this paper, we design two strategies to decrease the computational complexity of SSC, making a robust, accurate and efficient deformable segmentation system. (1) When the shape repository contains a large number of instances, which is often the case in 2D problems, K-SVD is used to learn a more compact but still informative shape dictionary. (2) If the derived shape instance has a large number of vertices, which often appears in 3D problems, an affinity propagation method is used to partition the surface into small sub-regions, on which the sparse shape composition is performed locally. Both strategies dramatically decrease the scale of the sparse optimization problem and hence speed up the algorithm. Our method is applied on a diverse set of biomedical image analysis problems. Compared to the original SSC, these two newly-proposed modules not only significant reduce the computational complexity, but also improve the overall accuracy.

Delving deeper into the whorl of flower segmentation

We describe an algorithm for automatically segmenting flowers in colour photographs. This is a challenging problem because of the sheer variety of flower classes, the variability within a class and within a particular flower, and the variability of the imaging conditions – lighting, pose, foreshortening, etc. The method couples two models – a colour model for foreground and background, and a light generic shape model for the petal structure. This shape model is tolerant to viewpoint changes and petal deformations, and applicable across many different flower classes. The segmentations are produced using a MRF cost function optimized using graph cuts. We show how the components of the algorithm can be tuned to overcome common segmentation errors, and how performance can be optimized by learning parameters on a training set. The algorithm is evaluated on 13 flower classes and more than 750 examples. Performance is assessed against ground truth trimap segmentations. The algorithms is also compared to several previous approaches for flower segmentation.

Segmentation of kidneys using a new active shape model generation technique based on non-rigid image registration

Active shape models (ASMs) are widely used for applications in the field of image segmentation. Building an ASM requires to determine point correspondences for input training data, which usually results in a set of landmarks distributed according to the statistical variations. State-of-the-art methods solve this problem by minimizing the description length of all landmarks using a parametric mapping of the target shape (e.g. a sphere). In case of models composed of multiple sub-parts or highly non-convex shapes, these techniques feature substantial drawbacks. This article proposes a novel technique for solving the crucial correspondence problem using non-rigid image registration. Unlike existing approaches the new method yields more detailed ASMs and does not require explicit or parametric formulations of the problem. Compared to other methods, the already built ASM can be updated with additional prior knowledge in a very efficient manner. For this work, a training set of 3-D kidney pairs has been manually segmented from 41 CT images of different patients and forms the basis for a clinical evaluation. The novel registration based approach is compared to an already established algorithm that uses a minimum description length (MDL) formulation. The presented results indicate that the use of non-rigid image registration to solve the point correspondence problem leads to improved ASMs and more accurate segmentation results. The sensitivity could be increased by approximately 10%. Experiments to analyze the dependency on the user initialization also show a higher sensitivity of 5–15%. The mean squared error of the segmentation results and the ground truth manually classified data could also be reduced by 20–34% with respect to varying numbers of training samples.

Pedestrian Detection and Tracking Using a Mixture of View-Based Shape–Texture Models

This paper presents a robust multicue approach to the integrated detection and tracking of pedestrians in a cluttered urban environment. A novel spatiotemporal object representation is proposed, which combines a generative shape model and a discriminative texture classifier, both of which are composed of a mixture of pose-specific submodels. Shape is represented by a set of linear subspace models, which is an extension of point distribution models, with shape transitions being modeled by a first-order Markov process. Texture, i.e., the shape-normalized intensity pattern, is represented by a manifold that is implicitly delimited by a set of pattern classifiers, whereas texture transition is modeled by a random walk. Direct 3-D measurements that are provided by a stereo system are further incorporated into the observation density function. We employ a Bayesian framework based on particle filtering to achieve integrated object detection and tracking. Large-scale experiments that involve pedestrian detection and tracking from a moving vehicle demonstrate the benefit of the proposed approach.

Brain Anatomical Structure Segmentation by Hybrid Discriminative/Generative Models

In this paper, a hybrid discriminative/generative model for brain anatomical structure segmentation is proposed. The learning aspect of the approach is emphasized. In the discriminative appearance models, various cues such as intensity and curvatures are combined to locally capture the complex appearances of different anatomical structures. A probabilistic boosting tree (PBT) framework is adopted to learn multiclass discriminative models that combine hundreds of features across different scales. On the generative model side, both global and local shape models are used to capture the shape information about each anatomical structure. The parameters to combine the discriminative appearance and generative shape models are also automatically learned. Thus, low-level and high-level information is learned and integrated in a hybrid model. Segmentations are obtained by minimizing an energy function associated with the proposed hybrid model. Finally, a grid-face structure is designed to explicitly represent the 3-D region topology. This representation handles an arbitrary number of regions and facilitates fast surface evolution. Our system was trained and tested on a set of 3-D magnetic resonance imaging (MRI) volumes and the results obtained are encouraging.

Automatic shape model building based on principal geodesic analysis bootstrapping

We present a novel method for automatic shape model building from a collection of training shapes. The result is a shape model consisting of the mean model and the major modes of variation with a dense correspondence map between individual shapes. The framework consists of iterations where a medial shape representation is deformed into the training shapes followed by computation of the shape mean and modes of shape variation. In the first iteration, a generic shape model is used as starting point - in the following iterations in the bootstrap method, the resulting mean and modes from the previous iteration are used. Thereby, we gradually capture the shape variation in the training collection better and better. Convergence of the method is explicitly enforced. The method is evaluated on collections of artificial training shapes where the expected shape mean and modes of variation are known by design. Furthermore, collections of real prostates and cartilage sheets are used in the evaluation. The evaluation shows that the method is able to capture the training shapes close to the attainable accuracy already in the first iteration. Furthermore, the correspondence properties measured by generality, specificity, and compactness are improved during the shape model building iterations.

Generic vs. person specific active appearance models

Active Appearance Models (AAMs) are generative parametric models that have been successfully used in the past to model faces. Anecdotal evidence, however, suggests that the performance of an AAM built to model the variation in appearance of a single person across pose, illumination, and expression (a Person Specific AAM) is substantially better than the performance of an AAM built to model the variation in appearance of many faces, including unseen subjects not in the training set (a Generic AAM). In this paper, we present an empirical evaluation that shows that Person Specific AAMs are, as expected, both easier to build and more robust to fit than Generic AAMs. Moreover, we show that: (1) building a generic shape model is far easier than building a generic appearance model, and (2) the shape component is the main cause of the reduced fitting robustness of Generic AAMs. We then proceed to describe two refinements to Generic AAMs to improve their performance: (1) a refitting procedure to improve the quality of the ground-truth data used to build the AAM and (2) a new fitting algorithm. For both refinements we demonstrate dramatically improved fitting performance. Finally, we evaluate the effect of these improvements on a combined model construction and fitting task.

Acquisition of 2D Shape Models from Scenes with Overlapping Objects using String Matching

In this paper we describe a system that is able to acquire models of 2D shapes from cluttered scenes. The input of the system is a sequence of images, each of which shows an unknown number of overlapping unknown 2D objects. The system identifies matching partial shapes across different images and combines them into complete 2D shape models, thus giving a complete interpretation of the input scenes. The identification of partial shapes is based on string matching, whereas a graph search procedure is used for shape model generation. The system has been fully implemented and tested on images containing parts of a jigsaw puzzle.