3D Training Data Research Articles

3D in-system calibration method for PET detectors.

Light-sharing detector designs for positron emission tomography (PET) systems have sparked interest in the scientific community. Particularly, (semi-)monoliths show generally good performance characteristics regarding 2D positioning, energy-, and timing resolution, as well as readout area. This is combined with intrinsic depth-of-interaction (DOI) capability to ensure a homogeneous spatial resolution across the entire field of view (FoV). However, complex positioning calibration processes limit their use in PET systems, especially in large-scale clinical systems. This work proposes a new 3D positioning in-system calibration method for fast and convenient (re-)calibration and quality control of assembled PET scanners. The method targets all kinds of PET detectors that achieve the best performance with individual calibration, including complex segmented detector designs. The in-system calibration method is evaluated and empirically compared to a state-of-the-art fan-beam calibration for a small-diameter proof of concept (PoC) scanner. A simulation study evaluates the method's applicability to different scanner geometries. A PoC scanner geometry of 120mm inner diameter and 150mm axial extent was set up consisting of five identical finely segmented slab detectors (one detector under test and four collimation detectors). A 2 2Na point source was moved in a circular path inside the FoV. Utilizing virtual collimation and by selecting gamma rays incident approximately perpendicular to the detector normal of the detector under test, training data was created for the training of a 2D positioning model with the machine-learning technique gradient tree boosting (GTB). Data with oblique ray angles was acquired in the same measurement for subsequent angular DOI calibration. For this, a 2D position estimate in the detector under test was calculated first. On this basis, the DOI label was calculated geometrically from the ray path within the detector to finally establish up to 3D training data. With a mean absolute error (MAE) of 0.8 and 1.19mm full-width at half maximum (FWHM) along the planar-monolithic slab dimension, the in-system methods performed similarly within 1% to the fan-beam collimator results. The DOI performance was at ∼90% with 1.13mm MAE and 2.47mm FWHM to the fan-beam collimator. Analytical calculations suggest an improved performance for larger scanner geometries. The functionality of the 3D in-system positioning calibration method was successfully demonstrated with the measurements within a PoC scanner configuration with similar positioning performance as the bench-top fan-beam setup. The in-system calibration method can be used to calibrate and test fully assembled PET systems to enable more complex light-sharing detector architectures in, for example, large PET systems with many detectors. The acquired data can further be used for more complex energy and time calibrations.

Fast reconstruction 3D computed tomography image of stacked cell under faster scanning by dual-branch cross-fusion flat bottom network

Abstract Cone beam computed tomography (CBCT) fast scanning and reconstruction is a key step to achieve rapid detection of internal defects in batteries. In this work, we have achieved a faster CT scanning just in 5 seconds by reducing the X-ray exposure time in sparse view CT. However, the CT data is extremely incomplete by faster scanning; the existing reconstruction methods are difficult to reconstruct a high quality three-dimensional (3D) CT image of stacked cells. To address this issue, we propose a 3D CT image reconstruction network, which can reconstruct higher quality CT images from low quality 3D volume data. The input data of the reconstruction network is not 2D projection data, but 3D volume data. In this network, a high and low resolution dual-branch cross-fusion flat bottom structure is designed. The high resolution flat bottom branch aims to preserve detailed information, while the low resolution flat bottom branch focuses on capturing more semantic information. Cross-fusion between these branches mitigates the loss of semantic details. Additionally, the auxiliary loss function, the main loss function, and the 3D attention module are designed to enhance semantic accuracy and the learning performance of the network. The 3D training data is collected under a fast scanning strategy spanning 5-60 seconds. During the training phase, we use clipping block technology to cut the 3D volume data, enabling direct training on the 3D volume data. Our experimental results demonstrate that our 3D reconstruction network outperforms mainstream algorithms under this faster scanning strategy, which is able to reconstruct higher quality 3D CT images just in 15 seconds. Ablation experiments confirm the positive impact of the dual-branch cross-fusion flat bottom structure, attention module, and loss functions on improving the quality of 3D CT images.

Designing spongy-bone-like cellular materials: Matched topology and anisotropy

Bone is a natural material with properties such as high specific stiffness and strength. These exceptional mechanical properties are attributed to the meso-scale structure and elastic anisotropy of spongy bone. Replicating the topological traits and mechanical properties of spongy bone presents a novel opportunity to develop high-performance cellular materials. To achieve this, we propose an innovative framework for designing biomimetic cellular materials that match the trabecular structure and elastic anisotropy of spongy bone. This framework introduces a forward-flow design process that utilizes gradient-based feature tuning on a low-dimensional feature vector, transforming the complex inverse design problem into an efficient iterative process. A key innovation in our approach is the use of a pre-trained generative model, SliceGAN, to reconstruct 3D unit cells from 2D micro-CT images. This significantly reduces the cost and time associated with traditional layer-by-layer CT scans typically required for 3D training data. Numerical homogenization is then used to determine the effective elastic stiffness matrix, and a Fourier neural operator is trained to predict these matrices efficiently, greatly enhancing the computational efficiency of the design process. Using this framework, we successfully designed unit cells with topological traits and elastic anisotropy that closely approximate those of natural spongy bone. This opens new avenues for developing spongy-bone-mimetic cellular materials with exceptional mechanical properties. Moreover, the framework's versatility allows it to be extended to the design of other bio-inspired cellular materials.

Neural Semantic Surface Maps

AbstractWe present an automated technique for computing a map between two genus‐zero shapes, which matches semantically corresponding regions to one another. Lack of annotated data prohibits direct inference of 3D semantic priors; instead, current state‐of‐the‐art methods predominantly optimize geometric properties or require varying amounts of manual annotation. To overcome the lack of annotated training data, we distill semantic matches from pre‐trained vision models: our method renders the pair of untextured 3D shapes from multiple viewpoints; the resulting renders are then fed into an off‐the‐shelf image‐matching strategy that leverages a pre‐trained visual model to produce feature points. This yields semantic correspondences, which are projected back to the 3D shapes, producing a raw matching that is inaccurate and inconsistent across different viewpoints. These correspondences are refined and distilled into an inter‐surface map by a dedicated optimization scheme, which promotes bijectivity and continuity of the output map. We illustrate that our approach can generate semantic surface‐to‐surface maps, eliminating manual annotations or any 3D training data requirement. Furthermore, it proves effective in scenarios with high semantic complexity, where objects are non‐isometrically related, as well as in situations where they are nearly isometric.

Generating 3D images of material microstructures from a single 2D image: a denoising diffusion approach

Three-dimensional (3D) images provide a comprehensive view of material microstructures, enabling numerical simulations unachievable with two-dimensional (2D) imaging alone. However, obtaining these 3D images can be costly and constrained by resolution limitations. We introduce a novel method capable of generating large-scale 3D images of material microstructures, such as metal or rock, from a single 2D image. Our approach circumvents the need for 3D image data while offering a cost-effective, high-resolution alternative to existing imaging techniques. Our method combines a denoising diffusion probabilistic model with a generative adversarial network framework. To compensate for the lack of 3D training data, we implement chain sampling, a technique that utilizes the 3D intermediate outputs obtained by reversing the diffusion process. During the training phase, these intermediate outputs are guided by a 2D discriminator. This technique facilitates our method’s ability to gradually generate 3D images that accurately capture the geometric properties and statistical characteristics of the original 2D input. This study features a comparative analysis of the 3D images generated by our method, SliceGAN (the current state-of-the-art method), and actual 3D micro-CT images, spanning a diverse set of rock and metal types. The results shown an improvement of up to three times in the Frechet inception distance score, a typical metric for evaluating the performance of image generative models, and enhanced accuracy in derived properties compared to SliceGAN. The potential of our method to produce high-resolution and statistically representative 3D images paves the way for new applications in material characterization and analysis domains.

Cellstitch: 3D cellular anisotropic image segmentation via optimal transport

BackgroundSpatial mapping of transcriptional states provides valuable biological insights into cellular functions and interactions in the context of the tissue. Accurate 3D cell segmentation is a critical step in the analysis of this data towards understanding diseases and normal development in situ. Current approaches designed to automate 3D segmentation include stitching masks along one dimension, training a 3D neural network architecture from scratch, and reconstructing a 3D volume from 2D segmentations on all dimensions. However, the applicability of existing methods is hampered by inaccurate segmentations along the non-stitching dimensions, the lack of high-quality diverse 3D training data, and inhomogeneity of image resolution along orthogonal directions due to acquisition constraints; as a result, they have not been widely used in practice.MethodsTo address these challenges, we formulate the problem of finding cell correspondence across layers with a novel optimal transport (OT) approach. We propose CellStitch, a flexible pipeline that segments cells from 3D images without requiring large amounts of 3D training data. We further extend our method to interpolate internal slices from highly anisotropic cell images to recover isotropic cell morphology.ResultsWe evaluated the performance of CellStitch through eight 3D plant microscopic datasets with diverse anisotropic levels and cell shapes. CellStitch substantially outperforms the state-of-the art methods on anisotropic images, and achieves comparable segmentation quality against competing methods in isotropic setting. We benchmarked and reported 3D segmentation results of all the methods with instance-level precision, recall and average precision (AP) metrics.ConclusionsThe proposed OT-based 3D segmentation pipeline outperformed the existing state-of-the-art methods on different datasets with nonzero anisotropy, providing high fidelity recovery of 3D cell morphology from microscopic images.

Paleokarst caves recognition from seismic response simulation to convolutional neural network detection

Paleokarst systems, found in carbonate rock formations worldwide, have potential for creating vast reservoirs and facilitating hydrocarbon migration. Thus, studying these systems is essential for the exploration and development of carbonate reservoirs. An approach using convolutional neural networks (CNNs) is introduced to automatically and precisely identify cave features within 3D seismic data. An efficient technique is outlined for generating ample amounts of 3D training data, which is comprised of synthetic seismic data and labels for cave features contained in the seismic data, as a solution to bypass the labeling task for training the CNN. This workflow uses point-spread functions to simulate the cave response in the seismic data and allows us to easily generate realistic and diverse synthetic training data sets with different geologic structures and cave features. By training a CNN with these synthetic data sets, it can effectively learn to detect cave features in field seismic volumes. Upon evaluation using multiple examples, this approach outperforms earlier techniques like seismic attributes and other CNN-based paleokarst characterization methods.

A conditional GAN-based approach to build 3D facies models sequentially upwards

This paper presents an alternative way of simulating 3D facies models using conditional generative adversarial networks (GANs) by reconstructing the 3D volume upward with customised thickness from generated 2D slices. To mimic natural fluvial reservoirs, GANs need to learn the complicated facies’ lateral and vertical sequences, connectivity patterns and spatial correlation, e.g. the distribution of channel deposits and lateral accretion packages, including channel lags and point bars in a meandering fluvial system. 3D GAN modelling remains challenging, despite some success in 3D, due to the increased data and GAN model complexity caused by the 3D training data. For example, a 3D facies model with a larger number of voxels introduces more complex geobody shapes and requires 3D convolutions that increases the computational burden compared to 2D convolutions.Our proposed conditional GAN-based framework predicts 2D upper layers slice by slice to build 3D facies models, mimicking the sedimentary process in a purely aggrading system, which is less computationally costly than generating the facies models used as training data and generating facies models directly in 3D from a GAN. Each upper layer is predicted conditionally to a lower layer that contains the facies distributions in all layers below the target using several unique training enhancements. All those methods help prevent the simulated 3D models from gradually losing realism and the preset sedimentary settings vertically while inferring the plausible evolution of the channel system. The results demonstrate that the trained conditional generator, a SPADE generator, can work as a simulator to produce unconditional 3D meandering fluvial facies models by inputting a user-defined spatial correlation strength between layers and the number of layers needed (thickness in metres) without post-processing or retraining. This framework’s performance was evaluated using a connectivity metric towards the range of avulsion scenarios that represent a range of depositional uncertainty.

BARC: Breed-Augmented Regression Using Classification for 3D Dog Reconstruction from Images

The goal of this work is to reconstruct 3D dogs from monocular images. We take a model-based approach, where we estimate the shape and pose parameters of a 3D articulated shape model for dogs. We consider dogs as they constitute a challenging problem, given they are highly articulated and come in a variety of shapes and appearances. Recent work has considered a similar task using the multi-animal SMAL model, with additional limb scale parameters, obtaining reconstructions that are limited in terms of realism. Like previous work, we observe that the original SMAL model is not expressive enough to represent dogs of many different breeds. Moreover, we make the hypothesis that the supervision signal used to train the network, that is 2D keypoints and silhouettes, is not sufficient to learn a regressor that can distinguish between the large variety of dog breeds. We therefore go beyond previous work in two important ways. First, we modify the SMAL shape space to be more appropriate for representing dog shape. Second, we formulate novel losses that exploit information about dog breeds. In particular, we exploit the fact that dogs of the same breed have similar body shapes. We formulate a novel breed similarity loss, consisting of two parts: One term is a triplet loss, that encourages the shape of dogs from the same breed to be more similar than dogs of different breeds. The second one is a breed classification loss. With our approach we obtain 3D dogs that, compared to previous work, are quantitatively better in terms of 2D reconstruction, and significantly better according to subjective and quantitative 3D evaluations. Our work shows that a-priori side information about similarity of shape and appearance, as provided by breed labels, can help to compensate for the lack of 3D training data. This concept may be applicable to other animal species or groups of species. We call our method BARC (Breed-Augmented Regression using Classification). Our code is publicly available for research purposes at https://barc.is.tue.mpg.de/.

MiCellAnnGELo: annotate microscopy time series of complex cell surfaces with 3D virtual reality.

Advances in 3D live cell microscopy are enabling high-resolution capture of previously unobserved processes. Unleashing the power of modern machine learning methods to fully benefit from these technologies is, however, frustrated by the difficulty of manually annotating 3D training data. MiCellAnnGELo virtual reality software offers an immersive environment for viewing and interacting with 4D microscopy data, including efficient tools for annotation. We present tools for labelling cell surfaces with a wide range of applications, including cell motility, endocytosis and transmembrane signalling. MiCellAnnGELo employs the cross-platform (Mac/Unix/Windows) Unity game engine and is available under the MIT licence at https://github.com/CellDynamics/MiCellAnnGELo.git, together with sample data. MiCellAnnGELo can be run in desktop mode on a 2D screen or in 3D using a standard VR headset with a compatible GPU. Supplementary data are available at Bioinformatics online.

Detecting 3D Points of Interest Using Projective Neural Networks

Detecting points of interest on 3D shapes is a fundamental research problem in geometry processing. Due to the complicated relationship between points of interest and their geometric features, detecting points of interest on any given 3D shape remains challenging. Due to the lack of training data, previous data-driven methods for detecting 3D points of interest mainly focus on utilizing hand-crafted geometric features to predict the probabilities of each point being a POI, which greatly limits detection performance. In this paper, we propose a novel algorithm for detecting 3D points of interest by using projective neural networks. Our method first projects the labeled training 3D shapes into multiple 2D views and then learns the required features from the 2D views in an end-to-end fashion. The points of interest on test 3D shapes are then automatically detected by applying the learned neural network and our improved density peak clustering. Our method relies neither on hand-crafted feature descriptors nor a large quantity of expensive 3D training data to obtain satisfactory results. Experimental results show significantly superior detection performance of our method over the state-of-the-art methods.

Generating three-dimensional structures from a two-dimensional slice with generative adversarial network-based dimensionality expansion

Generative adversarial networks (GANs) can be trained to generate three-dimensional (3D) image data, which are useful for design optimization. However, this conventionally requires 3D training data, which are challenging to obtain. Two-dimensional (2D) imaging techniques tend to be faster, higher resolution, better at phase identification and more widely available. Here we introduce a GAN architecture, SliceGAN, that is able to synthesize high-fidelity 3D datasets using a single representative 2D image. This is especially relevant for the task of material microstructure generation, as a cross-sectional micrograph can contain sufficient information to statistically reconstruct 3D samples. Our architecture implements the concept of uniform information density, which ensures both that generated volumes are equally high quality at all points in space and that arbitrarily large volumes can be generated. SliceGAN has been successfully trained on a diverse set of materials, demonstrating the widespread applicability of this tool. The quality of generated micrographs is shown through a statistical comparison of synthetic and real datasets of a battery electrode in terms of key microstructural metrics. Finally, we find that the generation time for a 108 voxel volume is on the order of a few seconds, yielding a path for future studies into high-throughput microstructural optimization. A generative approach called SliceGAN is demonstrated that can construct complex three-dimensional (3D) images from representative two-dimensional (2D) image examples. This is a promising approach in particular for studying microstructured materials where acquiring good-quality 3D data is challenging; 3D datasets can be created with SliceGAN, making use of high-quality 2D imaging techniques that are widely available.

RockFlow: Fast Generation of Synthetic Source Rock Images Using Generative Flow Models

Image-based evaluation methods are a valuable tool for source rock characterization. The time and resources needed to obtain images has spurred development of machine-learning generative models to create synthetic images of pore structure and rock fabric from limited image data. While generative models have shown success, existing methods for generating 3D volumes from 2D training images are restricted to binary images and grayscale volume generation requires 3D training data. Shale characterization relies on 2D imaging techniques such as scanning electron microscopy (SEM), and grayscale values carry important information about porosity, kerogen content, and mineral composition of the shale. Here, we introduce RockFlow, a method based on generative flow models that creates grayscale volumes from 2D training data. We apply RockFlow to baseline binary micro-CT image volumes and compare performance to a previously proposed model. We also show the extension of our model to 2D grayscale data by generating grayscale image volumes from 2D SEM and dual modality nanoscale shale images. The results show that our method underestimates the porosity and surface area on the binary baseline datasets but is able to generate realistic grayscale image volumes for shales. With improved binary data preprocessing, we believe that our model is capable of generating synthetic porous media volumes for a very broad class of rocks from shale to carbonates to sandstone.

Multi-Person Hierarchical 3D Pose Estimation in Natural Videos

Despite the increasing need of analyzing human poses on the street and in the wild, multi-person 3D pose estimation using monocular static or moving camera in real-world scenarios remains a challenge, either requiring large-scale training data or high computation complexity due to the high degrees of freedom in 3D human poses. We propose a novel scheme to effectively track and hierarchically estimate 3D human poses in natural videos in an efficient fashion. Without the need of using labelled 3D training data, we formulate torso estimation as a Perspective-N-Point (PNP) problem, and limb pose estimation as an optimization problem, and hierarchically structure the high dimensional poses to efficiently address the challenge. Experiments show good performance and high efficiency of multi-person 3D pose estimation on real-world videos, including street scenarios and various human daily activities from fixed and moving cameras, resulting in great new opportunities to understand and predict human behaviors.

SaltSeg: Automatic 3D salt segmentation using a deep convolutional neural network

Salt boundary interpretation is important for the understanding of salt tectonics and velocity model building for seismic migration. Conventional methods consist of computing salt attributes and extracting salt boundaries. We have formulated the problem as 3D image segmentation and evaluated an efficient approach based on deep convolutional neural networks (CNNs) with an encoder-decoder architecture. To train the model, we design a data generator that extracts randomly positioned subvolumes from large-scale 3D training data set followed by data augmentation, then feed a large number of subvolumes into the network while using salt/nonsalt binary labels generated by thresholding the velocity model as ground truth labels. We test the model on validation data sets and compare the blind test predictions with the ground truth. Our results indicate that our method is capable of automatically capturing subtle salt features from the 3D seismic image with less or no need for manual input. We further test the model on a field example to indicate the generalization of this deep CNN method across different data sets.