3D Shape Model Research Articles

Measuring 3D face deformations from RGB images of expression rehabilitation exercises

BackgroundThe accurate (quantitative) analysis of 3D face deformation is a problem of increasing interest in many applications. In particular, defining a 3D model of the face deformation into a 2D target image to capture local and asymmetric deformations remains a challenge in existing literature. A measure of such local deformations may be a relevant index for monitoring the rehabilitation exercises of patients suffering from Parkinson’s or Alzheimer’s disease or those recovering from a stroke. MethodsIn this paper, a complete framework that allows the construction of a 3D morphable shape model (3DMM) of the face is presented for fitting to a target RGB image. The model has the specific characteristic of being based on localized components of deformation. The fitting transformation is performed from 3D to 2D and guided by the correspondence between landmarks detected in the target image and those manually annotated on the average 3DMM. The fitting also has the distinction of being performed in two steps to disentangle face deformations related to the identity of the target subject from those induced by facial actions. ResultsThe method was experimentally validated using the MICC-3D dataset, which includes 11 subjects. Each subject was imaged in one neutral pose and while performing 18 facial actions that deform the face in localized and asymmetric ways. For each acquisition, 3DMM was fit to an RGB frame whereby, from the apex facial action and the neutral frame, the extent of the deformation was computed. The results indicate that the proposed approach can accurately capture face deformation, even localized and asymmetric deformations. ConclusionThe proposed framework demonstrated that it is possible to measure deformations of a reconstructed 3D face model to monitor facial actions performed in response to a set of targets. Interestingly, these results were obtained using only RGB targets, without the need for 3D scans captured with costly devices. This paves the way for the use of the proposed tool in remote medical rehabilitation monitoring.

SPAGHETTI

Neural implicit fields are quickly emerging as an attractive representation for learning based techniques. However, adopting them for 3D shape modeling and editing is challenging. We introduce a method for E diting I mplicit S hapes T hrough P art A ware G enera T ion, permuted in short as SPAGHETTI. Our architecture allows for manipulation of implicit shapes by means of transforming, interpolating and combining shape segments together, without requiring explicit part supervision. SPAGHETTI disentangles shape part representation into extrinsic and intrinsic geometric information. This characteristic enables a generative framework with part-level control. The modeling capabilities of SPAGHETTI are demonstrated using an interactive graphical interface, where users can directly edit neural implicit shapes. Our code, editing user interface demo and pre-trained models are available at github.com/amirhertz/spaghetti.

Shape Transformers: Topology‐Independent 3D Shape Models Using Transformers

AbstractParametric 3D shape models are heavily utilized in computer graphics and vision applications to provide priors on the observed variability of an object's geometry (e.g., for faces). Original models were linear and operated on the entire shape at once. They were later enhanced to provide localized control on different shape parts separately. In deep shape models, nonlinearity was introduced via a sequence of fully‐connected layers and activation functions, and locality was introduced in recent models that use mesh convolution networks. As common limitations, these models often dictate, in one way or another, the allowed extent of spatial correlations and also require that a fixed mesh topology be specified ahead of time. To overcome these limitations, we present Shape Transformers, a new nonlinear parametric 3D shape model based on transformer architectures. A key benefit of this new model comes from using the transformer's self‐attention mechanism to automatically learn nonlinear spatial correlations for a class of 3D shapes. This is in contrast to global models that correlate everything and local models that dictate the correlation extent. Our transformer 3D shape autoencoder is a better alternative to mesh convolution models, which require specially‐crafted convolution, and down/up‐sampling operators that can be difficult to design. Our model is also topologically independent: it can be trained once and then evaluated on any mesh topology, unlike most previous methods. We demonstrate the application of our model to different datasets, including 3D faces, 3D hand shapes and full human bodies. Our experiments demonstrate the strong potential of our Shape Transformer model in several applications in computer graphics and vision.

Learning the shape of female breasts: an open-access 3D statistical shape model of the female breast built from 110 breast scans

We present the Regensburg Breast Shape Model (RBSM)—a 3D statistical shape model of the female breast built from 110 breast scans acquired in a standing position, and the first publicly available. Together with the model, a fully automated, pairwise surface registration pipeline used to establish dense correspondence among 3D breast scans is introduced. Our method is computationally efficient and requires only four landmarks to guide the registration process. A major challenge when modeling female breasts from surface-only 3D breast scans is the non-separability of breast and thorax. In order to weaken the strong coupling between breast and surrounding areas, we propose to minimize the variance outside the breast region as much as possible. To achieve this goal, a novel concept called breast probability masks (BPMs) is introduced. A BPM assigns probabilities to each point of a 3D breast scan, telling how likely it is that a particular point belongs to the breast area. During registration, we use BPMs to align the template to the target as accurately as possible inside the breast region and only roughly outside. This simple yet effective strategy significantly reduces the unwanted variance outside the breast region, leading to better statistical shape models in which breast shapes are quite well decoupled from the thorax. The RBSM is thus able to produce a variety of different breast shapes as independently as possible from the shape of the thorax. Our systematic experimental evaluation reveals a generalization ability of 0.17 mm and a specificity of 2.8 mm. To underline the expressiveness of the proposed model, we finally demonstrate in two showcase applications how the RBSM can be used for surgical outcome simulation and the prediction of a missing breast from the remaining one. Our model is available at https://www.rbsm.re-mic.de/.

Numerical approach to synthesizing realistic asteroid surfaces from morphological parameters

Context. The complex shape of asteroids and comets is a critical parameter in many scientific and operational studies. From the global irregular shape down to the local surface details, these topographies reflect the formation and evolutionary processes that remould the celestial body. Furthermore, these processes control how the surface will continue to evolve: from mass wasting on high slopes to spin-up due to anisotropic re-emission of thermal radiation. In addition, for space missions, the irregular coarse shape and complex landscape are a hazard to navigation, which must be accounted for in the planning phase. Aims. In this paper, we propose a novel method to synthesize physically correct 3D shape models of small celestial bodies, such as asteroids, to support the testing of a wide range of parameters in scientific and operational studies. Methods. We modeled virtual asteroid shapes using non-uniform sphere packings to represent the coarse shape, define an implicit surface, and then synthesize high-resolution topography with user-defined, locally controlled spot noise models. This effectively replaces the random noise model (e.g., Perlin noise) used in traditional approaches and allows us to construct a morphology based on actual physical shapes of the most common features observed on asteroids and comets. As an example of such a feature, we propose several kernel functions to add virtual craters to the coarse shape of the asteroid, of which the spatial distribution is controlled by typical crater production functions (e.g., a power law). Results. We demonstrate how this technique can be used to generate a variety of asteroid shapes and topographies using different cratering parameters and distributions. We apply our technique to artificially increase the resolution of existing models of the Didymos-Dimorphos system, the target of the Double Asteroid Redirection Test, and Hera missions. We show that our approach generates models that are suitable for typical analysis relying on detailed asteroid shapes, as well as operational scenarios for space missions. The meshes created with our algorithm can be directly used with existing visualization software and operations or science pipelines to generate data suitable for mission planning and to validate data analysis techniques.

A systematic comparison of generative models for medical images

PurposeThis work aims for a systematic comparison of popular shape and appearance models. Here, two statistical and four deep-learning-based shape and appearance models are compared and evaluated in terms of their expressiveness described by their generalization ability and specificity as well as further properties like input data format, interpretability and latent space distribution and dimension.MethodsClassical shape models and their locality-based extension are considered next to autoencoders, variational autoencoders, diffeomorphic autoencoders and generative adversarial networks. The approaches are evaluated in terms of generalization ability, specificity and likeness depending on the amount of training data. Furthermore, various latent space metrics are presented in order to capture further major characteristics of the models.ResultsThe experimental setup showed that locality statistical shape models yield best results in terms of generalization ability for 2D and 3D shape modeling. However, the deep learning approaches show strongly improved specificity. In the case of simultaneous shape and appearance modeling, the neural networks are able to generate more realistic and diverse appearances. A major drawback of the deep-learning models is, however, their impaired interpretability and ambiguity of the latent space.ConclusionsIt can be concluded that for applications not requiring particularly good specificity, shape modeling can be reliably established with locality-based statistical shape models, especially when it comes to 3D shapes. However, deep learning approaches are more worthwhile in terms of appearance modeling.

3D shape modeling of the optic nerve head with central retinal vessels

3D shape modeling of the optic nerve head with central retinal vessels

Left ventricular shape predicts arrhythmic risk in fibrotic dilated cardiomyopathy.

AimsRemodelling of the left ventricular (LV) shape is one of the hallmarks of non-ischaemic dilated cardiomyopathy (DCM) and may contribute to ventricular arrhythmias and sudden cardiac death. We sought to investigate a novel three dimensional (3D) shape analysis approach to quantify LV remodelling for arrhythmia prediction in DCM.Methods and resultsWe created 3D LV shape models from end-diastolic cardiac magnetic resonance images of 156 patients with DCM and late gadolinium enhancement (LGE). Using the shape models, principle component analysis, and Cox-Lasso regression, we derived a prognostic LV arrhythmic shape (LVAS) score which identified patients who reached a composite arrhythmic endpoint of sudden cardiac death, aborted sudden cardiac death, and sustained ventricular tachycardia. We also extracted geometrical metrics to look for potential prognostic markers. During a follow-up period of up to 16 years (median 7.7, interquartile range: 3.9), 25 patients met the arrhythmic endpoint. The optimally prognostic LV shape for predicting the time-to arrhythmic event was a paraboloidal longitudinal profile, with a relatively wide base. The corresponding LVAS was associated with arrhythmic events in univariate Cox regression (hazard ratio = 2.0 per quartile; 95% confidence interval: 1.3–2.9), in univariate Cox regression with propensity score adjustment, and in three multivariate models; with LV ejection fraction, New York Heart Association Class III/IV (Model 1), implantable cardioverter-defibrillator receipt (Model 2), and cardiac resynchronization therapy (Model 3).ConclusionBiomarkers of LV shape remodelling in DCM can help to identify the patients at greatest risk of lethal ventricular arrhythmias.

Reconstruction of a 3D Human Foot Shape Model Based on a Video Stream Using Photogrammetry and Deep Neural Networks

Reconstructed 3D foot models can be used for 3D printing and further manufacturing of individual orthopedic shoes, as well as in medical research and for online shoe shopping. This study presents a technique based on the approach and algorithms of photogrammetry. The presented technique was used to reconstruct a 3D model of the foot shape, including the lower arch, using smartphone images. The technique is based on modern computer vision and artificial intelligence algorithms designed for image processing, obtaining sparse and dense point clouds, depth maps, and a final 3D model. For the segmentation of foot images, the Mask R-CNN neural network was used, which was trained on foot data from a set of 40 people. The obtained accuracy was 97.88%. The result of the study was a high-quality reconstructed 3D model. The standard deviation of linear indicators in length and width was 0.95 mm, with an average creation time of 1 min 35 s recorded. Integration of this technique into the business models of orthopedic enterprises, Internet stores, and medical organizations will allow basic manufacturing and shoe-fitting services to be carried out and will help medical research to be performed via the Internet.

OP09.03: From three‐dimensional (3D) ultrasound to 3D shape model of the fetal face: determining a normal reference standard

OP09.03: From three‐dimensional (3D) ultrasound to 3D shape model of the fetal face: determining a normal reference standard

Multi-Task Joint Learning of 3D Keypoint Saliency and Correspondence Estimation

3D keypoint detection is an essential problem in computer graphics and computer vision, especially for 3D shape analysis and model matching. In this paper, we propose a novel multi-task joint learning network architecture for 3D keypoint saliency estimation and correspondence estimation. To better capture the local and global features of the 3D model, we design a spatial multi-scale perception module that concatenates feature maps at different scales during the extraction of point cloud features. In the multi-task joint learning process, we obtain the offset vector of each point to the keypoint in the 3D model through a voting mechanism. This mechanism predicts the confidence value of each point in the 3D model and then filters out low-confidence points to generate a reliable voting result. Afterwards, keypoint saliency estimation is achieved through clustering. In parallel, keypoint correspondence estimation is learned by predicting the semantic label of the selected high-confidence points. In extensive evaluations, ablation studies and comparisons, we demonstrate that the proposed architecture can both efficiently and accurately detect the position and semantic labels of the 3D keypoints, which enables it to outperform state-of-the-art approaches for 3D keypoint detection.

Learning Cuboid Abstraction of 3D Shapes via Iterative Error Feedback

Abstracting human-made 3D models by a set of primitives, such as cuboid abstraction, is a fundamental task in 3D shape modelling and analysis. Traditionally, different forms of representations, such as edges, volumes, or curves, were used as primitives. Although methods that apply local operations to compute such primitives can produce satisfactory results with their own merits, the computations can be very slow with complex models. Learning-based abstraction methods are much faster but cannot guarantee the fitting precision between the primitives and the original shape. To solve this problem, we propose an unsupervised learning approach for shape abstraction. Our method’s key idea is to use an iterative error feedback (IEF)-based network to improve primitive precision. Our method contains two main steps. First, we use a regression network to predict the initial primitives. Second, we increase the accuracy of the initial primitives by using an IEF-based network, which iteratively outputs the primitive updates. We demonstrate the advantages of our method by comparing it to existing state-of-the-art methods. We also thoroughly evaluate our method by ablation studies.

Automation of 3D average human body shape modeling using Rhino and Grasshopper Algorithm

The aim of this study is to develop an automated process for modeling average 3D human body according to body types using both NUBRS-based modeling software Rhinoceros 3D® (Rhino) and Grasshopper as an algorithm editor. First, we categorized men aged 36 to 55 years included in SizeUSA 3D data into the three body types (normal, overweight, and obese), and selected seven samples in each body type. To execute the automated process of generating an average 3D model of their lower bodies in a step-by-step manner, the following procedures were performed: (1) Determine the main reference lines on the 3D-scanned lower bodies, including six horizontal reference lines and six vertical reference lines; (2) Create horizontal and vertical line grids and intersection points (3) Generate an average 3D model in a position that corresponds to the average coordinates of the intersection points (vertex coordinates) of seven samples for each body type. A Grasshopper algorithm was formulated to automatically execute all procedures that had to be repeatedly performed. As a way to verify the average model’s size and shape, the girth measurements of the samples for each body type were averaged, and the results were compared with those of the 3D average body shape. It was found that the deviation was less than 1 cm, which indicates the validity of the 3D modeling approach applied in the present study. Each process was incorporated into commands available in the Rhino interface, and this automation allowed a number of 3D body shape modeling operations to be implemented in a significantly reduced time period.

Influence of medial open wedge high tibial osteotomy on tibial tuberosity-trochlear groove distance.

Medial open wedge high tibial osteotomy (MOWHTO) is an effective treatment option for realignment of a varus knee. However, a simple supra-tuberositary osteotomy can lead to patella baja and potentially increases the tibial tuberosity-trochlear groove distance (TTTG). The purpose of this study was to quantify the influence of MOWHTO on TTTG. Three-dimensional (3D) surface models of five lower extremities with a varus hip-knee-ankle angle (HKA) and a borderline TTTG (≥ 15mm), five lower extremities with a varus HKA and a normal TTTG (< 15mm) and a 3D statistical shape model (SSM) of a neutrally aligned healthy knee were analysed by simulating MOWHTO with a stepwise increment of one degree of valgisation from the preoperative coronal deformity (0°-15°) for each patient, resulting in a total of 165 simulations. Postoperative 3D TTTG and tibial torsion (TT) were measured for each simulation. A mathematical formula was developed to calculate the increase of TTTG after MOWHTO. Mean differences between simulated and calculated TTTG were analysed. Mean preoperative HKA was 6.5 ± 3.0° varus (range 0.8°-11.5°). Mean TTTG increased from 14.2 ± 3.2mm (range 9.6-19.1) preoperatively to 18.8 ± 3.8mm (range 14.5-25.0) postoperatively (p = 0.001). TTTG increased approximately linear by + 0.5 ± 0.2° (range 0.3-0.8) per 1° of valgisation with a high positive correlation (0.99, p = 0.001) from 0° to 15°. Mean difference between simulated and calculated TTTG was 0.03 ± 0.02mm (range 0.01-0.07) per 1° of valgisation (p < 0.001). MOWHTO results in an approximately linear increase in TTTG of + 0.5mm per 1° of valgisation in the range from 0° to 15° and the lateralisation of the tibial tuberosity can be calculated reliably using the described formula. Preoperative analysis of TTTG in patients undergoing MOWHTO may prevent unintentional patellofemoral malalignment. III.

Investigation of errors in the representation of 3D models in determining the parameters of parts of devices BY the example of a two-mass rotating system with elastic coupling

Objective. The objective of the study is to consider the application field of modern device development and design methods using the means of 3D modeling and simulation of physical processes. The validity of the application and criteria for the reliability of the results obtained in the sequential design of the mechanical part of the electric drive and the control system are investigated.Methods. Methods of model representation of 3D objects in computer-aided design systems are considered, as well as methods for solving problems of determining strain under the applied load. Using the example of an elementary joint, the obtained results of the dynamic characteristic of an elastic shaft in a computer-aided design system are compared with those calculated analytically.Results. The article defines the basic principles and relations applicable to describing the shape of 3D models. Methods are shown by which it is possible to obtain information about the mass-centering model characteristics. The relations that form the basis of numerical methods for solving problems of determining elastic deformations of bodies are also given. The error that can occur when using insufficiently small elementary volumes in solving the problem of determining elastic deformation is shown.Conclusion. The use of 3D modeling in the design of complex technical systems is justified and speeds up production processes. However, numerical methods cannot always give an accurate result leading to the need to either increase the complexity of calculations or additionally adjust some designed device parameters.

Centered convolutional deep Boltzmann machine for 2D shape modeling

An object shape information plays a vital role in many computer applications. Among these applications, some tasks can allow object shape analysis directly solve the problem. Thus, how to extract shape features and model the shape is a crucial issue. This paper proposes a new shape modeling method utilizing the centered convolutional deep Boltzmann machine to model two-dimensional (2D) shape. The proposed method employs a probabilistic generative model based on deep Boltzmann machine and convolution computation to extract both local and global features of the shape within one framework. In addition, we also propose a bidirectional inference-based training algorithm coupled with the centering method to better learn the probability distribution of shapes without the need for a pre-training procedure. Our experimental results show that the proposed model can achieve best shape modeling performance in qualitative and quantitative evaluation compared with other probabilistic generative models, including the restricted Boltzmann machine, convolutional restricted Boltzmann machine, and deep Boltzmann machine.

ShapeAssembly

Manually authoring 3D shapes is difficult and time consuming; generative models of 3D shapes offer compelling alternatives. Procedural representations are one such possibility: they offer high-quality and editable results but are difficult to author and often produce outputs with limited diversity. On the other extreme are deep generative models: given enough data, they can learn to generate any class of shape but their outputs have artifacts and the representation is not editable. In this paper, we take a step towards achieving the best of both worlds for novel 3D shape synthesis. First, we propose ShapeAssembly, a domain-specific "assembly-language" for 3D shape structures. ShapeAssembly programs construct shape structures by declaring cuboid part proxies and attaching them to one another, in a hierarchical and symmetrical fashion. ShapeAssembly functions are parameterized with continuous free variables, so that one program structure is able to capture a family of related shapes. We show how to extract ShapeAssembly programs from existing shape structures in the PartNet dataset. Then, we train a deep generative model, a hierarchical sequence VAE, that learns to write novel ShapeAssembly programs. Our approach leverages the strengths of each representation: the program captures the subset of shape variability that is interpretable and editable, and the deep generative model captures variability and correlations across shape collections that is hard to express procedurally. We evaluate our approach by comparing the shapes output by our generated programs to those from other recent shape structure synthesis models. We find that our generated shapes are more plausible and physically-valid than those of other methods. Additionally, we assess the latent spaces of these models, and find that ours is better structured and produces smoother interpolations. As an application, we use our generative model and differentiable program interpreter to infer and fit shape programs to unstructured geometry, such as point clouds.

Reliability and quality of statistical shape and deformation models constructed from optical foot scans

The unique shape of modern human feet, and how they change shape when loaded are thought to be integral to effective upright gait. This unique shape, and the natural variations therein, have previously been analysed using a range of methods; from visual assessments, anthropometric measurements, and footprints, to x-ray, ultrasound and magnetic resonance images. However, these methods are often limited by their use of linear two-dimensional measures. Only recently have advances in three-dimensional (3D) scanning technology and statistical shape analysis been applied to studying 3D foot shape variations. Given their novelty, information regarding the reliability and repeatability of 3D foot scanning and shape modelling is lacking. To investigate whether repeated foot scans captured by two examiners give the same 3D shape and produce consistent statistical shape models, 17 healthy adults’ left feet were scanned while bearing half and full bodyweight, as well as minimal weight. Surface to surface distances between corresponding foot meshes and differences between shape model quality criteria were both found to be small and insignificant. The only exception being the specificity criterion for minimally loaded foot scans. Furthermore, Euclidean vectors were used to model the magnitude and direction of deformation that feet undergo as a consequence of increased loading. The deformation models showed that loading a minimally loaded foot results in greater, but less consistent, shape changes than when increasing the load on an already loaded foot. These results show that the utilized methods offer a valuable, reliable and repeatable approach to analysing foot shape and deformation.

Surface roughness of asteroid (162173) Ryugu and comet 67P/Churyumov–Gerasimenko inferred fromin situobservations

ABSTRACTAlteration processes on asteroid and comet surfaces, such as thermal fracturing, (micrometeorite) impacts or volatile outgassing, are complex mechanisms that form diverse surface morphologies and roughness on various scales. These mechanisms and their interaction may differ on the surfaces of different bodies. Asteroid Ryugu and comet 67P/Churyumov–Gerasimenko, both, have been visited by landers that imaged the surfaces in high spatial resolution. We investigate the surface morphology and roughness of Ryugu and 67P/Churyumov–Gerasimenko based on high-resolution in situ images of 0.2 and 0.8 mm pixel resolution over an approximately 25 and 80 cm wide scene, respectively. To maintain comparability and reproducibility, we introduce a method to extract surface roughness descriptors (fractal dimension, Hurst exponent, joint roughness coefficient, root-mean-square slope, hemispherical crater density, small-scale roughness parameter, and Hapke mean slope angle) from in situ planetary images illuminated by LEDs. We validate our method and choose adequate parameters for an analysis of the roughness of the surfaces. We also derive the roughness descriptors from 3D shape models of Ryugu and orbiter camera images and show that the higher spatially resolved images result in a higher roughness. We find that 67P/Churyumov–Gerasimenko is up to 6 per cent rougher than Ryugu depending on the descriptor used and attribute this difference to the different intrinsic properties of the materials imaged and the erosive processes altering them. On 67P/Churyumov–Gerasimenko sublimation appears to be the main cause for roughness, while on Ryugu micrometeoroid bombardment as well as thermal fatigue and solar weathering may play a significant role in shaping the surface.

3D printed gummies: Personalized drug dosage in a safe and appealing way.

Obtention of customized dosage forms is one of the main attractions of 3D printing in pharmaceuticals. In this sense, children are one of the groups within the population with a greater need for drug doses adapted to their requirements (age, weight, pathological state…), but most 3D printed oral dosages are solid forms and, therefore, not suitable for them. This work developed patient-tailored medicinal gummies, an alternative oral dosage form with eye-catching appearance and appropriate organoleptic characteristics. Four inks were formulated, characterised and 3D printed by means of syringe-based extrusion mechanism. Different tests were performed to ensure reproducibility of the process and validate work methodology for dosage unit fabrication applying basic manufacturing standards. Rheological test helped in evaluating inks printability. Visual characterization concluded that drugmies, apart from a high fidelity in the 3D model shape reproduction, had a bright and uniformly coloured appearance and a pleasant aroma, which made them highly appetising and attractive. The printed gummy oral dosages complied comfortably with the mass uniformity assay regardless of the formulated ink used or the 3D model selected for printing. Ranitidine hydrochloride individual contents were determined using uv-vis spectrophotometry, showing successful results both in dose accuracy, uniformity of drug content and dissolution.