Latent Shape Research Articles

ICDDPM: Image-conditioned denoising diffusion probabilistic model for real-world complex point cloud single view reconstruction

Point cloud single-view reconstruction (PC-SVR) generates high-quality point clouds from low-cost 2D images, offering a cost-effective solution to the expensive and inefficient acquisition of point cloud for real-world scene typically achieved through expensive LiDAR systems. However, current models that generate point clouds from real-world 2D images (e.g. maritime vessels) still have shortcomings in terms of quality and model generalization. In this paper, we proposed a image-conditioned denoising diffusion probabilistic model (ICDDPM) for real-world complex point cloud single view reconstruction to address these issues. We re-designed the structure of classic diffusion model, use latent shape vectors to seamlessly integrate 2D image encoder, point cloud encoder, and conditional diffusion model, to cater PC-SVR task. By guiding the diffusion process with the 2D images, which serve as crucial conditional information, ICDDPM achieves end-to-end point cloud generation with superior quality. 2D images are also employed as input in the reverse diffusion process to further achieve point cloud generation. We conducted qualitative and quantitative experiments on synthetic dataset ShapeNet and real-world dataset PASCAL3D+ (focused on experiments of vessel point cloud data specifically). The results indicate that the ICDDPM model demonstrates superior performance compared to state-of-the-art models. It is capable of generating point clouds with a greater level of global and local details from various 2D image data. Additionally, the model exhibits strong generalization abilities and requires fewer computational resources.

Non-Gaussian Liability Distribution for Depression in the General Population.

Unlike depression sum scores, the underlying risk for depression is typically assumed to be normally distributed across the general population. To assess the true empirical shape of depression risk, we created a continuous-valued estimate of the latent depression density, using the Davidian-Curve Item Response Theory (DC-IRT) and the National Health and Nutrition Examination Survey (NHANES) cohorts from 2005 to 2018 (n = 36,244 on the Nine-item Patient Health Questionnaire; PHQ-9). We conducted simulations to investigate the performance of DC-IRT for large samples and realistic items. The method can recover complex latent-risk distributions even when they are not evident from sum scores. However, estimation accuracy for different sample sizes depends on the method of model selection. In addition to full-data analysis, random samples of a few thousand observations were drawn for analysis. The latent shape of depression was left-skewed and bimodal in both investigations, indicating that the latent-normality assumption does not hold for depression.

Collaborative Tooth Motion Diffusion Model in Digital Orthodontics

Tooth motion generation is an essential task in digital orthodontic treatment for precise and quick dental healthcare, which aims to generate the whole intermediate tooth motion process given the initial pathological and target ideal tooth alignments. Most prior works for multi-agent motion planning problems usually result in complex solutions. Moreover, the occlusal relationship between upper and lower teeth is often overlooked. In this paper, we propose a collaborative tooth motion diffusion model. The critical insight is to remodel the problem as a diffusion process. In this sense, we model the whole tooth motion distribution with a diffusion model and transform the planning problem into a sampling process from this distribution. We design a tooth latent representation to provide accurate conditional guides consisting of two key components: the tooth frame represents the position and posture, and the tooth latent shape code represents the geometric morphology. Subsequently, we present a collaborative diffusion model to learn the multi-tooth motion distribution based on inter-tooth and occlusal constraints, which are implemented by graph structure and new loss functions, respectively. Extensive qualitative and quantitative experiments demonstrate the superiority of our framework in the application of orthodontics compared with state-of-the-art methods.

TopoGCL: Topological Graph Contrastive Learning

Graph contrastive learning (GCL) has recently emerged as a new concept which allows for capitalizing on the strengths of graph neural networks (GNNs) to learn rich representations in a wide variety of applications which involve abundant unlabeled information. However, existing GCL approaches largely tend to overlook the important latent information on higher-order graph substructures. We address this limitation by introducing the concepts of topological invariance and extended persistence on graphs to GCL. In particular, we propose a new contrastive mode which targets topological representations of the two augmented views from the same graph, yielded by extracting latent shape properties of the graph at multiple resolutions. Along with the extended topological layer, we introduce a new extended persistence summary, namely, extended persistence landscapes (EPL) and derive its theoretical stability guarantees. Our extensive numerical results on biological, chemical, and social interaction graphs show that the new Topological Graph Contrastive Learning (TopoGCL) model delivers significant performance gains in unsupervised graph classification for 8 out of 12 considered datasets and also exhibits robustness under noisy scenarios.

Topological data analysis reveals core heteroblastic and ontogenetic programs embedded in leaves of grapevine (Vitaceae) and maracuyá (Passifloraceae).

Leaves are often described in language that evokes a single shape. However, embedded in that descriptor is a multitude of latent shapes arising from evolutionary, developmental, environmental, and other effects. These confounded effects manifest at distinct developmental time points and evolve at different tempos. Here, revisiting datasets comprised of thousands of leaves of vining grapevine (Vitaceae) and maracuyá (Passifloraceae) species, we apply a technique from the mathematical field of topological data analysis to comparatively visualize the structure of heteroblastic and ontogenetic effects on leaf shape in each group. Consistent with a morphologically closer relationship, members of the grapevine dataset possess strong core heteroblasty and ontogenetic programs with little deviation between species. Remarkably, we found that most members of the maracuyá family also share core heteroblasty and ontogenetic programs despite dramatic species-to-species leaf shape differences. This conservation was not initially detected using traditional analyses such as principal component analysis or linear discriminant analysis. We also identify two morphotypes of maracuyá that deviate from the core structure, suggesting the evolution of new developmental properties in this phylogenetically distinct sub-group. Our findings illustrate how topological data analysis can be used to disentangle previously confounded developmental and evolutionary effects to visualize latent shapes and hidden relationships, even ones embedded in complex, high-dimensional datasets.

Point Cloud Completion Via Skeleton-Detail Transformer.

Point cloud shape completion plays a central role in diverse 3D vision and robotics applications. Early methods used to generate global shapes without local detail refinement. Current methods tend to leverage local features to preserve the observed geometric details. However, they usually adopt the convolutional architecture over the incomplete point cloud to extract local features to restore the diverse information of both latent shape skeleton and geometric details, where long-distance correlation among the skeleton and details is ignored. In this work, we present a coarse-to-fine completion framework, which makes full use of both neighboring and long-distance region cues for point cloud completion. Our network leverages a Skeleton-Detail Transformer, which contains cross-attention and self-attention layers, to fully explore the correlation from local patterns to global shape and utilize it to enhance the overall skeleton. Also, we propose a selective attention mechanism to save memory usage in the attention process without significantly affecting performance. We conduct extensive experiments on the ShapeNet dataset and real-scanned datasets. Qualitative and quantitative evaluations demonstrate that our proposed network outperforms current state-of-the-art methods.

Rethinking Rotation Invariance with Point Cloud Registration

Recent investigations on rotation invariance for 3D point clouds have been devoted to devising rotation-invariant feature descriptors or learning canonical spaces where objects are semantically aligned. Examinations of learning frameworks for invariance have seldom been looked into. In this work, we review rotation invariance (RI) in terms of point cloud registration (PCR) and propose an effective framework for rotation invariance learning via three sequential stages, namely rotation-invariant shape encoding, aligned feature integration, and deep feature registration. We first encode shape descriptors constructed with respect to reference frames defined over different scales, e.g., local patches and global topology, to generate rotation-invariant latent shape codes. Within the integration stage, we propose an Aligned Integration Transformer (AIT) to produce a discriminative feature representation by integrating point-wise self- and cross-relations established within the shape codes. Meanwhile, we adopt rigid transformations between reference frames to align the shape codes for feature consistency across different scales. Finally, the deep integrated feature is registered to both rotation-invariant shape codes to maximize their feature similarities, such that rotation invariance of the integrated feature is preserved and shared semantic information is implicitly extracted from shape codes. Experimental results on 3D shape classification, part segmentation, and retrieval tasks prove the feasibility of our framework. Our project page is released at: https://rotation3d.github.io/.

Automated morphological phenotyping using learned shape descriptors and functional maps: A novel approach to geometric morphometrics.

The methods of geometric morphometrics are commonly used to quantify morphology in a broad range of biological sciences. The application of these methods to large datasets is constrained by manual landmark placement limiting the number of landmarks and introducing observer bias. To move the field forward, we need to automate morphological phenotyping in ways that capture comprehensive representations of morphological variation with minimal observer bias. Here, we present Morphological Variation Quantifier (morphVQ), a shape analysis pipeline for quantifying, analyzing, and exploring shape variation in the functional domain. morphVQ uses descriptor learning to estimate the functional correspondence between whole triangular meshes in lieu of landmark configurations. With functional maps between pairs of specimens in a dataset we can analyze and explore shape variation. morphVQ uses Consistent ZoomOut refinement to improve these functional maps and produce a new representation of shape variation, area-based and conformal (angular) latent shape space differences (LSSDs). We compare this new representation of shape variation to shape variables obtained via manual digitization and auto3DGM, an existing approach to automated morphological phenotyping. We find that LSSDs compare favorably to modern 3DGM and auto3DGM while being more computationally efficient. By characterizing whole surfaces, our method incorporates more morphological detail in shape analysis. We can classify known biological groupings, such as Genus affiliation with comparable accuracy. The shape spaces produced by our method are similar to those produced by modern 3DGM and to auto3DGM, and distinctiveness functions derived from LSSDs show us how shape variation differs between groups. morphVQ can capture shape in an automated fashion while avoiding the limitations of manually digitized landmarks, and thus represents a novel and computationally efficient addition to the geometric morphometrics toolkit.

Latent shape image learning via disentangled representation for cross-sequence image registration and segmentation.

Cross-sequence magnetic resonance image (MRI) registration and segmentation are two essential steps in a variety of medical image analysis tasks. And have attracted considerable research interest. However, they remain challenging due to domain shifts between different sequences. This study is aiming at proposing a novel method via disentangled representations, latent shape image learning (LSIL), for cross-sequence image registration and segmentation. Images from different sequences were firstly decomposed into a shared domain-invariant shape space and a domain-specific appearance space via an unsupervised image-to-image translation approach. A latent shape image learning model is then built on the disentangled shape representations to generate latent shape images. A series of experiments including cross-sequence image registration and segmentation were performed to qualitatively and quantitatively verify the validity of our method. Dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (HD95) were adopted as our evaluation metrics. The performance of our method was evaluated based on 2 datasets total of 50 MRIs. The experimental results showed the superiority of the proposed framework over the state-of-the-art cross-sequence registration and segmentation approaches. The proposed method shows the mean DSCs of 0.711 and 0.867, respectively, in cross-sequence registration and segmentation. We proposed a novel method based on representation disentangling to solve the cross-sequence registration and segmentation problem. Experimental results prove the feasibility and generalization of the generated latent shape images. The proposed method demonstrates significant potential for use in clinical environments of missing sequences. The source code is available at https://github.com/wujiong-hub/LSIL .

A novel method for point cloud completion: Adaptive region shape fusion network

Point cloud data obtained by 3D scanning equipment is often incomplete. In recent years, to complete the missing point clouds has become an important task in computer visualization research. This paper adopts a coarse-to-fine completion strategy and build a novel adaptive region shape fused network (ARSF-Net), which contains three core modules, namely region shape encoding module (RSE), adaptive feature selection-aggregation module (ASA), and encoding-attention transformer module (EAT). The RSE module adaptively aggregates the latent shape information contained in local features according to the feature strength. In ASA module, we first treat point coordinates and shape features as parent nodes and design a hybrid correlation method to adaptively group parent nodes. Then, each set of parent nodes generates a child node. Finally, we splice the features and points in the parent node and child node separately to double the number. For EAT module, we learn features from the encoding stage and use a coordinate-based embedding transformer to generate uniform high-resolution point clouds. Compared with previous methods, we pay special attention to the difference among the latent shape information contained in the local point clouds, thus making the local feature extraction more interpretable. At the same time, to generate valid detail features from the original ones, we abundantly consider the correlation among the original ones, and directly combine the original features with the generated ones. Our experiments on different datasets verify the good performance of ARSF-Net in the point cloud completion task.

Aortic Valve Leaflet Shape Synthesis With Geometric Prior From Surrounding Tissue.

Even though the field of medical imaging advances, there are structures in the human body that are barely assessible with classical image acquisition modalities. One example are the three leaflets of the aortic valve due to their thin structure and high movement. However, with an increasing accuracy of biomechanical simulation, for example of the heart function, and extense computing capabilities available, concise knowledge of the individual morphology of these structures could have a high impact on personalized therapy and intervention planning as well as on clinical research. Thus, there is a high demand to estimate the individual shape of inassessible structures given only information on the geometry of the surrounding tissue. This leads to a domain adaptation problem, where the domain gap could be very large while typically only small datasets are available. Hence, classical approaches for domain adaptation are not capable of providing sufficient predictions. In this work, we present a new framework for bridging this domain gap in the scope of estimating anatomical shapes based on the surrounding tissue's morphology. Thus, we propose deep representation learning to not map from one image to another but to predict a latent shape representation. We formalize this framework and present two different approaches to solve the given problem. Furthermore, we perform a proof-of-concept study for estimating the individual shape of the aortic valve leaflets based on a volumetric ultrasound image of the aortic root. Therefore, we collect an ex-vivo porcine data set consisting of both, ultrasound volume images as well as high-resolution leaflet images, evaluate both approaches on it and perform an analysis of the model's hyperparameters. Our results show that using deep representation learning and domain mapping between the identified latent spaces, a robust prediction of the unknown leaflet shape only based on surrounding tissue information is possible, even in limited data scenarios. The concept can be applied to a wide range of modeling tasks, not only in the scope of heart modeling but also for all kinds of inassessible structures within the human body.

Application of Topological Data Analysis to Multi-Resolution Matching of Aerosol Optical Depth Maps

Topological data analysis (TDA) combines concepts from algebraic topology, machine learning, statistics, and data science which allow us to study data in terms of their latent shape properties. Despite the use of TDA in a broad range of applications, from neuroscience to power systems to finance, the utility of TDA in Earth science applications is yet untapped. The current study aims to offer a new approach for analyzing multi-resolution Earth science datasets using the concept of data shape and associated intrinsic topological data characteristics. In particular, we develop a new topological approach to quantitatively compare two maps of geophysical variables at different spatial resolutions. We illustrate the proposed methodology by applying TDA to aerosol optical depth (AOD) datasets from the Goddard Earth Observing System, Version 5 (GEOS-5) model over the Middle East. Our results show that, contrary to the existing approaches, TDA allows for systematic and reliable comparison of spatial patterns from different observational and model datasets without regridding the datasets into common grids.

Semantics‐Guided Latent Space Exploration for Shape Generation

AbstractWe introduce an approach to incorporate user guidance into shape generation approaches based on deep networks. Generative networks such as autoencoders and generative adversarial networks are trained to encode shapes into latent vectors, effectively learning a latent shape space that can be sampled for generating new shapes. Our main idea is to enable users to explore the shape space with the use of high‐level semantic keywords. Specifically, the user inputs a set of keywords that describe the general attributes of the shape to be generated, e.g., “four legs” for a chair. Then, our method maps the keywords to a subspace of the latent space, where the subspace captures the shapes possessing the specified attributes. The user then explores only this subspace to search for shapes that satisfy the design goal, in a process similar to using a parametric shape model. Our exploratory approach allows users to model shapes at a high level without the need for advanced artistic skills, in contrast to existing methods that allow to guide the generation with sketching or partial modeling of a shape. Our technical contribution to enable this exploration‐based approach is the introduction of a label regression neural network coupled with shape encoder/decoder networks. The label regression network takes the user‐provided keywords and maps them to distributions in the latent space. We show that our method allows users to explore the shape space and generate a variety of shapes with selected high‐level attributes.

In-Vehicle Object-Level 3D Reconstruction of Traffic Scenes

Emerging automotive applications such as in-vehicle Augmented Reality (AR) and fully automated parking require a comprehensive understanding of the vehicle’s three-dimensional surrounding represented as an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">object-level</i> environmental model. In this model, not only 3D poses (positions and orientations) and 3D sizes of detected objects are registered, but 3D shapes (geometries) need to be reconstructed precisely. A combination of 3D object detection and 3D surface reconstruction techniques, referred to as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">object-level 3D reconstruction</i> , is fundamental to building such environmental models. However, the possibilities to incorporate object-level 3D reconstruction in a car have not been sufficiently explored either in academic research or in the industry. This primarily stems from the cost and resource constraints associated with the automotive domain. In this paper, we address these constraints by proposing implementations of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-vehicle object-level 3D reconstruction</i> in two specific use cases: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(i)</i> augmented reality and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">(ii)</i> automated parking. For augmented reality, we propose a cost-efficient solution called monocular <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3D Shaping</i> that requires only a single frame from a monocular camera as input. For automated parking, we propose a resource-efficient alternative that generates more precise 3D reconstruction results by taking advantage of additional 3D sensors (such as Lidars). The crux of our proposed approaches lies in the use of a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Latent Shape Space</i> , where various 3D shapes are represented using only two parameters. As a result, highly complex 3D shapes can now be transmitted using a low- to medium-bandwidth in-vehicle communication infrastructure in a cost-effective manner.

Learning Discriminative and Generative Shape Embeddings for Three-Dimensional Shape Retrieval

As an important solution for 3D shape retrieval, a multi-view shape descriptor has achieved impressive performance. One crucial part of view-based shape descriptors is to interpret 3D structures through various 2D observations. Most existing methods like MVCNN believe that a strong classification model trained with deep learning, can often provide an efficient shape embedding for 3D shape retrieval. However, these methods pay much attention to discriminative models and none of them necessarily incorporate the underlying 3D properties of the objects from 2D images. In this paper, we present a novel encoder-decoder recurrent feature aggregation network (ERFA-Net) to address this problem. Aiming at emphasizing the 3D properties of 3D shapes in the fusion of multiple view features, 3D properties prediction tasks are introduced into the 3D shape retrieval. Specifically, an image sequence of the shape is recurrently aggregated into a discriminative shape embedding based on LSTM network, and then this latent shape embedding is trained to predict the original voxel grids and estimate images of unseen viewpoints. This generation task gives an effective supervision which makes the network exploit 3D properties of shapes through various 2D images. Our method achieves the state-of-the-art performance for 3D shape retrieval, on two large-scale 3D shape datasets, ModelNet and ShapeNetCore55. Extensive experiments show that the proposed 3D representation performs robust discrimination against view occlusion, and strong generation ability for various 3D shape tasks.

Shape Constrained Fully Convolutional DenseNet with Adversarial Training for Multi-organ Segmentation on Head and Neck Low Field MR Images

MR guided radiotherapy provides images not only for patient positioning but also for online adaptive radiotherapy. Accurate delineation of organs-at-risk (OARs) on Head and Neck (H&N) MR images is valuable to both initial treatment planning and adaptive planning, but manual contouring is laborious and inconsistent. A novel method based on the generative adversarial network (GAN) with shape constraint (SC-GAN) is developed for fully automated H&N OARs segmentation on CT and low field MRI. A deep supervised fully convolutional DenseNet is employed as the segmentation network for voxel-wise prediction. A CNN based discriminator network is then utilized to correct predicted errors and image-level inconsistency between the prediction and ground truth. An additional shape representation loss between the prediction and ground truth in the latent shape space is integrated into the segmentation and adversarial loss functions to reduce false positivity and constrain the predicted shapes. The proposed segmentation method was first benchmarked on a public H&N database including 32 patients, and then on 25 0.35T MR images obtained from an MR guided radiotherapy system. The OARs include brainstem, optical chiasm, larynx (MR only), mandible, pharynx (MR only), parotid glands (both left and right), optical nerves (both left and right), and submandibular glands (both left and right, CT only). The performance of the proposed SC-GAN was compared with GAN alone and GAN with SRM but without the Densenet (GAN-SRM) to quantify the contributions of shape constraint and DenseNet in the deep neural network segmentation. The proposed SC-GAN slightly but consistently improves the segmentation accuracy on the benchmark H&N CT images compared with our previous deep segmentation network, which outperformed other published methods on the same or similar CT H&N dataset. On the low field MR dataset, the following average Dice’s indices were obtained using improved SC-GAN: 0.916 (Brainstem), 0.589 (Optical chiasm), 0.816 (Mandible), 0.703 (Optical nerves), 0.799 (Larynx), 0.706 (Pharynx), and 0.845 (Parotid glands). The average surface distances ranged from 0.68mm (Brainstem) to 1.70mm (Larynx). The 95% surface distance ranged from 1.48mm (Left optical nerve) to 3.92mm (Larynx). Compared with CT, using 95% surface distance evaluation, the automated segmentation accuracy is higher on MR for the brainstem, optical chiasm, optical nerves and parotids, and lower for the mandible. The SC-GAN performance is superior to GAN-SRM, which is more accurate than GAN alone on both the CT and MR datasets. The segmentation time for one patient is 14 seconds using a single GPU. The shape constrained FCNN is further improved by incorporating GAN and DenseNet. With the novel method, we showed that the low field MR images acquired on a MR guided radiation radiotherapy system can support accurate and fully automated segmentation of both bony and soft tissue OARs for adaptive radiotherapy.

Shape constrained fully convolutional DenseNet with adversarial training for multiorgan segmentation on head and neck CT and low-field MR images.

Image-guided radiotherapy provides images not only for patient positioning but also for online adaptive radiotherapy. Accurate delineation of organs-at-risk (OARs) on Head and Neck (H&N) CT and MR images is valuable to both initial treatment planning and adaptive planning, but manual contouring is laborious and inconsistent. A novel method based on the generative adversarial network (GAN) with shape constraint (SC-GAN) is developed for fully automated H&N OARs segmentation on CT and low-field MRI. A deep supervised fully convolutional DenseNet is employed as the segmentation network for voxel-wise prediction. A convolutional neural network (CNN)-based discriminator network is then utilized to correct predicted errors and image-level inconsistency between the prediction and ground truth. An additional shape representation loss between the prediction and ground truth in the latent shape space is integrated into the segmentation and adversarial loss functions to reduce false positivity and constrain the predicted shapes. The proposed segmentation method was first benchmarked on a public H&N CT database including 32 patients, and then on 25 0.35T MR images obtained from an MR-guided radiotherapy system. The OARs include brainstem, optical chiasm, larynx (MR only), mandible, pharynx (MR only), parotid glands (both left and right), optical nerves (both left and right), and submandibular glands (both left and right, CT only). The performance of the proposed SC-GAN was compared with GAN alone and GAN with the shape constraint (SC) but without the DenseNet (SC-GAN-ResNet) to quantify the contributions of shape constraint and DenseNet in the deep neural network segmentation. The proposed SC-GAN slightly but consistently improve the segmentation accuracy on the benchmark H&N CT images compared with our previous deep segmentation network, which outperformed other published methods on the same or similar CT H&N dataset. On the low-field MR dataset, the following average Dice's indices were obtained using improved SC-GAN: 0.916 (brainstem), 0.589 (optical chiasm), 0.816 (mandible), 0.703 (optical nerves), 0.799 (larynx), 0.706 (pharynx), and 0.845 (parotid glands). The average surface distances ranged from 0.68mm (brainstem) to 1.70mm (larynx). The 95% surface distance ranged from 1.48mm (left optical nerve) to 3.92mm (larynx). Compared with CT, using 95% surface distance evaluation, the automated segmentation accuracy is higher on MR for the brainstem, optical chiasm, optical nerves and parotids, and lower for the mandible. The SC-GAN performance is superior to SC-GAN-ResNet, which is more accurate than GAN alone on both the CT and MR datasets. The segmentation time for one patient is 14seconds using a single GPU. The performance of our previous shape constrained fully CNNs for H&N segmentation is further improved by incorporating GAN and DenseNet. With the novel segmentation method, we showed that the low-field MR images acquired on a MR-guided radiation radiotherapy system can support accurate and fully automated segmentation of both bony and soft tissue OARs for adaptive radiotherapy.

Effects of Distance and Shape on the Estimation of the Piecewise Growth Mixture Model

The piecewise growth mixture model is used in longitudinal studies to tackle non-continuous trajectories and unobserved heterogeneity in a compound way. This study investigated how factors such as latent distance and shape influence the model. Two simulation studies were used exploring the 2- and 3-class situation with sample size, latent distance (Mahalanobis distance), and shape being considered as the influencing factor. The results of two simulations showed that a non-parallel shape led to a slightly better overall model fit. Parameter estimation is affected by the shape, mainly through the parameter differences between latent classes.

Learning to Predict 3D Surfaces of Sculptures from Single and Multiple Views

The objective of this work is to reconstruct the 3D surfaces of sculptures from one or more images using a view-dependent representation. To this end, we train a network, SiDeNet, to predict the Silhouette and Depth of the surface given a variable number of images; the silhouette is predicted at a different viewpoint from the inputs (e.g. from the side), while the depth is predicted at the viewpoint of the input images. This has three benefits. First, the network learns a representation of shape beyond that of a single viewpoint, as the silhouette forces it to respect the visual hull, and the depth image forces it to predict concavities (which don’t appear on the visual hull). Second, as the network learns about 3D using the proxy tasks of predicting depth and silhouette images, it is not limited by the resolution of the 3D representation. Finally, using a view-dependent representation (e.g. additionally encoding the viewpoint with the input image) improves the network’s generalisability to unseen objects. Additionally, the network is able to handle the input views in a flexible manner. First, it can ingest a different number of views during training and testing, and it is shown that the reconstruction performance improves as additional views are added at test-time. Second, the additional views do not need to be photometrically consistent. The network is trained and evaluated on two synthetic datasets—a realistic sculpture dataset (SketchFab), and ShapeNet. The design of the network is validated by comparing to state of the art methods for a set of tasks. It is shown that (i) passing the input viewpoint (i.e. using a view-dependent representation) improves the network’s generalisability at test time. (ii) Predicting depth/silhouette images allows for higher quality predictions in 2D, as the network is not limited by the chosen latent 3D representation. (iii) On both datasets the method of combining views in a global manner performs better than a local method. Finally, we show that the trained network generalizes to real images, and probe how the network has encoded the latent 3D shape.

Fully automatic multi-organ segmentation for head and neck cancer radiotherapy using shape representation model constrained fully convolutional neural networks.

Intensity modulated radiation therapy (IMRT) is commonly employed for treating head and neck (H&N) cancer with uniform tumor dose and conformal critical organ sparing. Accurate delineation of organs-at-risk (OARs) on H&N CT images is thus essential to treatment quality. Manual contouring used in current clinical practice is tedious, time-consuming, and can produce inconsistent results. Existing automated segmentation methods are challenged by the substantial inter-patient anatomical variation and low CT soft tissue contrast. To overcome the challenges, we developed a novel automated H&N OARs segmentation method that combines a fully convolutional neural network (FCNN) with a shape representation model (SRM). Based on manually segmented H&N CT, the SRM and FCNN were trained in two steps: (a) SRM learned the latent shape representation of H&N OARs from the training dataset; (b) the pre-trained SRM with fixed parameters were used to constrain the FCNN training. The combined segmentation network was then used to delineate nine OARs including the brainstem, optic chiasm, mandible, optical nerves, parotids, and submandibular glands on unseen H&N CT images. Twenty-two and 10 H&N CT scans provided by the Public Domain Database for Computational Anatomy (PDDCA) were utilized for training and validation, respectively. Dice similarity coefficient (DSC), positive predictive value (PPV), sensitivity (SEN), average surface distance (ASD), and 95% maximum surface distance (95%SD) were calculated to quantitatively evaluate the segmentation accuracy of the proposed method. The proposed method was compared with an active appearance model that won the 2015 MICCAI H&N Segmentation Grand Challenge based on the same dataset, an atlas method and a deep learning method based on different patient datasets. An average DSC = 0.870 (brainstem), DSC = 0.583 (optic chiasm), DSC = 0.937 (mandible), DSC = 0.653 (left optic nerve), DSC = 0.689 (right optic nerve), DSC = 0.835 (left parotid), DSC = 0.832 (right parotid), DSC = 0.755 (left submandibular), and DSC = 0.813 (right submandibular) were achieved. The segmentation results are consistently superior to the results of atlas and statistical shape based methods as well as a patch-wise convolutional neural network method. Once the networks are trained off-line, the average time to segment all 9 OARs for an unseen CT scan is 9.5 s. Experiments on clinical datasets of H&N patients demonstrated the effectiveness of the proposed deep neural network segmentation method for multi-organ segmentation on volumetric CT scans. The accuracy and robustness of the segmentation were further increased by incorporating shape priors using SMR. The proposed method showed competitive performance and took shorter time to segment multiple organs in comparison to state of the art methods.