3D Generative Adversarial Networks Research Articles

Automated tree crown labeling with 3D radiative transfer modelling achieves human comparable performances for tree segmentation in semi-arid landscapes

Mapping tree crowns in arid or semi-arid areas, which cover around one-third of the Earth’s land surface, is a key methodology towards sustainable management of trees. Recent advances in deep learning have shown promising results for tree crown segmentation. However, a large amount of manually labeled data is still required. We here propose a novel method to delineate tree crowns from high resolution satellite imagery using deep learning trained with automatically generated labels from 3D radiative transfer modeling, intending to reduce human annotation significantly. The methodological steps consist of 1) simulating images with a 3D radiative transfer model, 2) image style transfer learning based on generative adversarial network (GAN) and 3) tree crown segmentation using U-net segmentation model. The delineation performances of the proposed method have been evaluated on a manually annotated dataset consisting of more than 40,000 tree crowns. Our approach, which relies solely on synthetic images, demonstrates high segmentation accuracy, with an F1 score exceeding 0.77 and an Intersection over Union (IoU) above 0.64. Particularly, it achieves impressive accuracy in extracting crown areas (r2 greater than 0.87) and crown densities (r2 greater than 0.72), comparable to that of a trained dataset with human annotations only. In this study, we demonstrated that the integration of a 3D radiative transfer model and GANs for automatically generating training labels can achieve performances comparable to human labeling, and can significantly reduce the time needed for manual labeling in remote sensing segmentation applications.

Some effects of limited wall-sensor availability on flow estimation with 3D-GANs

In this work we assess the impact of the limited availability of wall-embedded sensors on the full 3D estimation of the flow field in a turbulent channel with Reτ=200\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Re_{\ au }=200$$\\end{document}. The estimation technique is based on a 3D generative adversarial network (3D-GAN). We recently demonstrated that 3D-GANs are capable of estimating fields with good accuracy by employing fully-resolved wall quantities (pressure and streamwise/spanwise wall shear stress on a grid with DNS resolution). However, the practical implementation in an experimental setting is challenging due to the large number of sensors required. In this work, we aim to estimate the flow fields with substantially fewer sensors. The impact of the reduction of the number of sensors on the quality of the flow reconstruction is assessed in terms of accuracy degradation and spectral length-scales involved. It is found that the accuracy degradation is mainly due to the spatial undersampling of scales, rather than the reduction of the number of sensors per se. We explore the performance of the estimator in case only one wall quantity is available. When a large number of sensors is available, pressure measurements provide more accurate flow field estimations. Conversely, the elongated patterns of the streamwise wall shear stress make this quantity the most suitable when only few sensors are available. As a further step towards a real application, the effect of sensor noise is also quantified. It is shown that configurations with fewer sensors are less sensitive to measurement noise.

Generative AI unlocks PET insights: brain amyloid dynamics and quantification.

Studying the spatiotemporal patterns of amyloid accumulation in the brain over time is crucial in understanding Alzheimer's disease (AD). Positron Emission Tomography (PET) imaging plays a pivotal role because it allows for the visualization and quantification of abnormal amyloid beta (Aβ) load in the living brain, providing a powerful tool for tracking disease progression and evaluating the efficacy of anti-amyloid therapies. Generative artificial intelligence (AI) can learn complex data distributions and generate realistic synthetic images. In this study, we demonstrate for the first time the potential of Generative Adversarial Networks (GANs) to build a low-dimensional representation space that effectively describes brain amyloid load and its dynamics. Using a cohort of 1,259 subjects with AV45 PET images from the Alzheimer's Disease Neuroimaging Initiative (ADNI), we develop a 3D GAN model to project images into a latent representation space and generate back synthetic images. Then, we build a progression model on the representation space based on non-parametric ordinary differential equations to study brain amyloid evolution. We found that global SUVR can be accurately predicted with a linear regression model only from the latent representation space (RMSE = 0.08 ± 0.01). We generated synthetic PET trajectories and illustrated predicted Aβ change in four years compared with actual progression. Generative AI can generate rich representations for statistical prediction and progression modeling and simulate evolution in synthetic patients, providing an invaluable tool for understanding AD, assisting in diagnosis, and designing clinical trials. The aim of this study was to illustrate the huge potential that generative AI has in brain amyloid imaging and to encourage its advancement by providing use cases and ideas for future research tracks.

Physically‐Based Lighting for 3D Generative Models of Cars

AbstractRecent work has demonstrated that Generative Adversarial Networks (GANs) can be trained to generate 3D content from 2D image collections, by synthesizing features for neural radiance field rendering. However, most such solutions generate radiance, with lighting entangled with materials. This results in unrealistic appearance, since lighting cannot be changed and view‐dependent effects such as reflections do not move correctly with the viewpoint. In addition, many methods have difficulty for full, 360° rotations, since they are often designed for mainly front‐facing scenes such as faces. We introduce a new 3D GAN framework that addresses these shortcomings, allowing multi‐view coherent 360° viewing and at the same time relighting for objects with shiny reflections, which we exemplify using a car dataset. The success of our solution stems from three main contributions. First, we estimate initial camera poses for a dataset of car images, and then learn to refine the distribution of camera parameters while training the GAN. Second, we propose an efficient Image‐Based Lighting model, that we use in a 3D GAN to generate disentangled reflectance, as opposed to the radiance synthesized in most previous work. The material is used for physically‐based rendering with a dataset of environment maps. Third, we improve the 3D GAN architecture compared to previous work and design a careful training strategy that allows effective disentanglement. Our model is the first that generate a variety of 3D cars that are multi‐view consistent and that can be relit interactively with any environment map.

Synthesising microstructures of a partially frozen salty sand using voxel-based 3D generative adversarial networks

Generating synthetic material microstructures is essential in the numerical modelling of geomaterials. The occurrence of permafrost and saline groundwater overlapping regions is crucial in a series of phenomena, such as carbon emissions and subgrade settlements. The microstructure of geomaterials in these regions is of particular complexity because of the multiphase nature with salty water and ice crystals. This complexity renders existing generative models ineffective in synthesising their microstructures. Traditional generative methods are limited in the sense that require prior knowledge of material descriptors. Recently, machine learning generative models achieved unprecedented levels of performance and realism, but still lack the means to assess posterior error. This work aims to bridge the gap between traditional methods and deep learning generative models by assessing posterior image quality in the latter. A 3D Generative Adversarial Network (GAN) model is trained with image samples from an X-ray CT of a partially frozen salty sand. The metrics retained to assess posterior quality are particle fabric (shape parameter and anisotropy) and homogenised elastic coefficients obtained with Finite Element Method (FEM) simulations. A hyperparametric study on batch size and latent dimension serves to select the best configuration based on particle fabric. FEM simulations determine the deviation in the generated images elastic coefficients being 7.55% on average with respect to real samples. With 803 voxels, generated images are the largest up-to-date in a three-phase material, allowing to reach REV criteria. Applications range from the generation of microscales in double-scale models to the calibration of image processing tools.

GAN-Based Motion Artifact Correction of 3D MR Volumes Using an Image-to-Image Translation Algorithm.

The quality of brain MRI volumes is often compromised by motion artifacts arising from intricate respiratory patterns and involuntary head movements, manifesting as blurring and ghosting that markedly degrade imaging quality. In this study, we introduce an innovative approach employing a 3D deep learning framework to restore brain MR volumes afflicted by motion artifacts. The framework integrates a densely connected 3D U-net architecture augmented by generative adversarial network (GAN)-informed training with a novel volumetric reconstruction loss function tailored to 3D GAN to enhance the quality of the volumes. Our methodology is substantiated through comprehensive experimentation involving a diverse set of motion artifact-affected MR volumes. The generated high-quality MR volumes have similar volumetric signatures comparable to motion-free MR volumes after motion correction. This underscores the significant potential of harnessing this 3D deep learning system to aid in the rectification of motion artifacts in brain MR volumes, highlighting a promising avenue for advanced clinical applications.

Impact of bias field correction on 0.35 T pelvic MR images: evaluation on generative adversarial network-based OARs' auto-segmentation and visual grading assessment.

Magnetic resonance imaging (MRI)-guided radiotherapy enables adaptive treatment plans based on daily anatomical changes and accurate organ visualization. However, the bias field artifact can compromise image quality, affecting diagnostic accuracy and quantitative analyses. This study aims to assess the impact of bias field correction on 0.35 T pelvis MRIs by evaluating clinical anatomy visualization and generative adversarial network (GAN) auto-segmentation performance. 3D simulation MRIs from 60 prostate cancer patients treated on MR-Linac (0.35 T) were collected and preprocessed with the N4ITK algorithm for bias field correction. A 3D GAN architecture was trained, validated, and tested on 40, 10, and 10 patients, respectively, to auto-segment the organs at risk (OARs) rectum and bladder. The GAN was trained and evaluated either with the original or the bias-corrected MRIs. The Dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (HD95th) were computed for the segmented volumes of each patient. The Wilcoxon signed-rank test assessed the statistical difference of the metrics within OARs, both with and without bias field correction. Five radiation oncologists blindly scored 22 randomly chosen patients in terms of overall image quality and visibility of boundaries (prostate, rectum, bladder, seminal vesicles) of the original and bias-corrected MRIs. Bennett's S score and Fleiss' kappa were used to assess the pairwise interrater agreement and the interrater agreement among all the observers, respectively. In the test set, the GAN trained and evaluated on original and bias-corrected MRIs showed DSC/HD95th of 0.92/5.63mm and 0.92/5.91mm for the bladder and 0.84/10.61mm and 0.83/9.71mm for the rectum. No statistical differences in the distribution of the evaluation metrics were found neither for the bladder (DSC: p = 0.07; HD95th: p = 0.35) nor for the rectum (DSC: p = 0.32; HD95th: p = 0.63). From the clinical visual grading assessment, the bias-corrected MRI resulted mostly in either no change or an improvement of the image quality and visualization of the organs' boundaries compared with the original MRI. The bias field correction did not improve the anatomy visualization from a clinical point of view and the OARs' auto-segmentation outputs generated by the GAN.

A Deep-Learning Approach to a Volumetric Radio Environment Map Construction for UAV-Assisted Networks

Providing global coverage for ubiquitous users is a key requirement of the fifth generation (5G) and beyond wireless technologies. This can be achieved by integrating airborne networks, such as unmanned aerial vehicles (UAVs) and satellite networks, with terrestrial networks. However, the deployment of airborne networks in a three-dimensional (3D) or volumetric space requires a new understanding of the propagation channel and its losses in both the areal and altitude dimensions. Despite significant research on radio environment map (REM) construction, much of it has been limited to two-dimensional (2D) contexts. This neglects the altitude-related characteristics of electromagnetic wave propagation and confines REMs to 2D formats, which limits the comprehensive and continuous visualization of propagation environment variation in spatial dimensions. This paper proposes a volumetric REM (VREM) construction approach to compute 3D propagation losses. The proposed approach addresses the limitations of existing approaches by learning the spatial correlation of wireless propagation channel characteristics and visualizing REM in areal and height/altitude dimensions using deep learning models. Specifically, the approach uses two deep learning-based models: volume-to-volume (Vol2Vol) VREM with 3D-generative adversarial networks and sliced VREM with altitude-aware spider-UNets. In both cases, knowledge of the propagation environment and transmitter locations in 3D space is used to capture the spatial and altitude dependency of the propagation channel’s characteristics. We developed the Addis dataset, a large REM dataset comprising 54,000 samples collected from the urban part of Addis Ababa, Ethiopia, to train the proposed models. Each sample of data comprises a 512-meter by 512-meter areal resolution with different 3D obstacles (buildings and terrain), 15 simulated propagation loss maps at every 3-meter altitude resolution, and 80 different 3D transmitter locations. The results of the training and testing of the proposed models reveal that the constructed VREMs are statistically comparable. In particular, the Vol2Vol approach has a minimum L1 loss of 0.01, which further decreases to 0.0084 as the line-of-sight (LoS) probability increases to 0.95.

Auto-segmentation of pelvic organs at risk on 0.35T MRI using 2D and 3D Generative Adversarial Network models

PurposeManual recontouring of targets and Organs At Risk (OARs) is a time-consuming and operator-dependent task. We explored the potential of Generative Adversarial Networks (GAN) to auto-segment the rectum, bladder and femoral heads on 0.35T MRIs to accelerate the online MRI-guided-Radiotherapy (MRIgRT) workflow. Methods3D planning MRIs from 60 prostate cancer patients treated with 0.35T MR-Linac were collected. A 3D GAN architecture and its equivalent 2D version were trained, validated and tested on 40, 10 and 10 patients respectively. The volumetric Dice Similarity Coefficient (DSC) and 95th percentile Hausdorff Distance (HD95th) were computed against expert drawn ground-truth delineations. The networks were also validated on an independent external dataset of 16 patients. ResultsIn the internal test set, the 3D and 2D GANs showed DSC/HD95th of 0.83/9.72 mm and 0.81/10.65 mm for the rectum, 0.92/5.91 mm and 0.85/15.72 mm for the bladder, and 0.94/3.62 mm and 0.90/9.49 mm for the femoral heads. In the external test set, the performance was 0.74/31.13 mm and 0.72/25.07 mm for the rectum, 0.92/9.46 mm and 0.88/11.28 mm for the bladder, and 0.89/7.00 mm and 0.88/10.06 mm for the femoral heads. The 3D and 2D GANs required on average 1.44 s and 6.59 s respectively to generate the OARs’ volumetric segmentation for a single patient. ConclusionsThe proposed 3D GAN auto-segments pelvic OARs with high accuracy on 0.35T, in both the internal and the external test sets, outperforming its 2D equivalent in both segmentation robustness and volume generation time.

Deep learning-enabled design for tailored mechanical properties of SLM-manufactured metallic lattice structures

The lattice structures obtained by combining repetitive light cellular forms provide superior high strength to weight capabilities compared to monolithic solid bodies. These properties of lattice structures can be further enhanced by improving the design and manufacturing process. However, the process of creating aluminum alloy lattice structures with tailored mechanical properties is still challenging due to the wide range of possible designs and their complex structure-property relationships. This paper proposes a new methodology that uses the deep learning-based 3D Generative Adversarial Network (3DGAN) model to solve this complex engineering problem. Unlike previous studies on deep learning-based lattice design, the dataset obtained through parametric design and Simulated Annealing techniques enables GAN to create new 3D lattice structures with improved mechanical strength properties. The designs generated with the GAN algorithm were produced using Selective Laser Melting Additive Manufacturing (SLM-AM) technology. The mechanical properties of SLM-fabricated (SLMed) AlSi10Mg unit cell samples were examined by a series of compression and impact tests. The results reveal that the lattice configurations generated using the generative model exhibit improved mechanical properties (e.g., a remarkable increase in normalized energy absorption and extension capacities of up to 57% and 26%, respectively). This study shows that designs generated with the 3DGAN model in lattice structures improve the design space by improving their mechanical properties. Moreover, the successful integration of deep learning and SLM-AM opens up new possibilities for creating custom parts with improved strength-to-weight ratios, dimensional accuracy, and mechanical performance.

3DGAUnet: 3D Generative Adversarial Networks with a 3D U-Net Based Generator to Achieve the Accurate and Effective Synthesis of Clinical Tumor Image Data for Pancreatic Cancer

Pancreatic ductal adenocarcinoma (PDAC) presents a critical global health challenge, and early detection is crucial for improving the 5-year survival rate. Recent medical imaging and computational algorithm advances offer potential solutions for early diagnosis. Deep learning, particularly in the form of convolutional neural networks (CNNs), has demonstrated success in medical image analysis tasks, including classification and segmentation. However, the limited availability of clinical data for training purposes continues to represent a significant obstacle. Data augmentation, generative adversarial networks (GANs), and cross-validation are potential techniques to address this limitation and improve model performance, but effective solutions are still rare for 3D PDAC, where the contrast is especially poor, owing to the high heterogeneity in both tumor and background tissues. In this study, we developed a new GAN-based model, named 3DGAUnet, for generating realistic 3D CT images of PDAC tumors and pancreatic tissue, which can generate the inter-slice connection data that the existing 2D CT image synthesis models lack. The transition to 3D models allowed the preservation of contextual information from adjacent slices, improving efficiency and accuracy, especially for the poor-contrast challenging case of PDAC. PDAC’s challenging characteristics, such as an iso-attenuating or hypodense appearance and lack of well-defined margins, make tumor shape and texture learning challenging. To overcome these challenges and improve the performance of 3D GAN models, our innovation was to develop a 3D U-Net architecture for the generator, to improve shape and texture learning for PDAC tumors and pancreatic tissue. Thorough examination and validation across many datasets were conducted on the developed 3D GAN model, to ascertain the efficacy and applicability of the model in clinical contexts. Our approach offers a promising path for tackling the urgent requirement for creative and synergistic methods to combat PDAC. The development of this GAN-based model has the potential to alleviate data scarcity issues, elevate the quality of synthesized data, and thereby facilitate the progression of deep learning models, to enhance the accuracy and early detection of PDAC tumors, which could profoundly impact patient outcomes. Furthermore, the model has the potential to be adapted to other types of solid tumors, hence making significant contributions to the field of medical imaging in terms of image processing models.

Segmentation of Brain Tumor Using a 3D Generative Adversarial Network.

Images of brain tumors may only show up in a small subset of scans, so important details may be missed. Further, because labeling is typically a labor-intensive and time-consuming task, there are typically only a small number of medical imaging datasets available for analysis. The focus of this research is on the MRI images of the human brain, and an attempt has been made to propose a method for the accurate segmentation of these images to identify the correct location of tumors. In this study, GAN is utilized as a classification network to detect and segment of 3D MRI images. The 3D GAN network model provides dense connectivity, followed by rapid network convergence and improved information extraction. Mutual training in a generative adversarial network can bring the segmentation results closer to the labeled data to improve image segmentation. The BraTS 2021 dataset of 3D images was used to compare two experimental models.

A conditional GAN-based approach to build 3D facies models sequentially upwards

This paper presents an alternative way of simulating 3D facies models using conditional generative adversarial networks (GANs) by reconstructing the 3D volume upward with customised thickness from generated 2D slices. To mimic natural fluvial reservoirs, GANs need to learn the complicated facies’ lateral and vertical sequences, connectivity patterns and spatial correlation, e.g. the distribution of channel deposits and lateral accretion packages, including channel lags and point bars in a meandering fluvial system. 3D GAN modelling remains challenging, despite some success in 3D, due to the increased data and GAN model complexity caused by the 3D training data. For example, a 3D facies model with a larger number of voxels introduces more complex geobody shapes and requires 3D convolutions that increases the computational burden compared to 2D convolutions.Our proposed conditional GAN-based framework predicts 2D upper layers slice by slice to build 3D facies models, mimicking the sedimentary process in a purely aggrading system, which is less computationally costly than generating the facies models used as training data and generating facies models directly in 3D from a GAN. Each upper layer is predicted conditionally to a lower layer that contains the facies distributions in all layers below the target using several unique training enhancements. All those methods help prevent the simulated 3D models from gradually losing realism and the preset sedimentary settings vertically while inferring the plausible evolution of the channel system. The results demonstrate that the trained conditional generator, a SPADE generator, can work as a simulator to produce unconditional 3D meandering fluvial facies models by inputting a user-defined spatial correlation strength between layers and the number of layers needed (thickness in metres) without post-processing or retraining. This framework’s performance was evaluated using a connectivity metric towards the range of avulsion scenarios that represent a range of depositional uncertainty.

Patch-wise 3D segmentation quality assessment combining reconstruction and regression networks.

General deep-learning (DL)-based semantic segmentation methods with expert level accuracy may fail in 3D medical image segmentation due to complex tissue structures, lack of large datasets with ground truth, etc. For expeditious diagnosis, there is a compelling need to predict segmentation quality without ground truth. In some medical imaging applications, maintaining the quality of segmentation is crucial to the localized regions where disease is prevalent rather than just globally maintaining high-average segmentation quality. We propose a DL framework to identify regions of segmentation inaccuracies by combining a 3D generative adversarial network (GAN) and a convolutional regression network. Our approach is methodologically based on the learned ability to reconstruct the original images identifying the regions of location-specific segmentation failures, in which the reconstruction does not match the underlying original image. We use conditional GAN to reconstruct input images masked by the segmentation results. The regression network is trained to predict the patch-wise Dice similarity coefficient (DSC), conditioned on the segmentation results. The method relies directly on the extracted segmentation related features and does not need to use ground truth during the inference phase to identify erroneous regions in the computed segmentation. We evaluated the proposed method on two public datasets: osteoarthritis initiative 4D (3D + time) knee MRI (knee-MR) and 3D non-small cell lung cancer CT (lung-CT). For the patch-wise DSC prediction, we observed the mean absolute errors of 0.01 and 0.04 with the independent standard for the knee-MR and lung-CT data, respectively. This method shows promising results in localizing the erroneous segmentation regions that may aid the downstream analysis of disease diagnosis and prognosis prediction.

Enhancing cryo-EM maps with 3D deep generative networks for assisting protein structure modeling.

The tertiary structures of an increasing number of biological macromolecules have been determined using cryo-electron microscopy (cryo-EM). However, there are still many cases where the resolution is not high enough to model the molecular structures with standard computational tools. If the resolution obtained is near the empirical borderline (3-4.5 Å), improvement in the map quality facilitates structure modeling. We report EM-GAN, a novel approach that modifies an input cryo-EM map to assist protein structure modeling. The method uses a 3D generative adversarial network (GAN) that has been trained on high- and low-resolution density maps to learn the density patterns, and modifies the input map to enhance its suitability for modeling. The method was tested extensively on a dataset of 65 EM maps in the resolution range of 3-6 Å and showed substantial improvements in structure modeling using popular protein structure modeling tools. https://github.com/kiharalab/EM-GAN, Google Colab: https://tinyurl.com/3ccxpttx.

CG-3DSRGAN: A classification guided 3D generative adversarial network for image quality recovery from low-dose PET images.

Positron emission tomography (PET) is the most sensitive molecular imaging modality routinely applied in our modern healthcare. High radioactivity caused by the injected tracer dose is a major concern in PET imaging and limits its clinical applications. However, reducing the dose leads to inadequate image quality for diagnostic practice. Motivated by the need to produce high quality images with minimum 'low-dose', convolutional neural networks (CNNs) based methods have been developed for high quality PET synthesis from its low-dose counterparts. Previous CNNs-based studies usually directly map low-dose PET into features space without consideration of different dose reduction level. In this study, a novel approach named CG-3DSRGAN (Classification-Guided Generative Adversarial Network with Super Resolution Refinement) is presented. Specifically, a multi-tasking coarse generator, guided by a classification head, allows for a more comprehensive understanding of the noise-level features present in the low-dose data, resulting in improved image synthesis. Moreover, to recover spatial details of standard PET, an auxiliary super resolution network - Contextual-Net - is proposed as a second-stage training to narrow the gap between coarse prediction and standard PET. We compared our method to the state-of-the-art methods on whole-body PET with different dose reduction factors (DRF). Experiments demonstrate our method can outperform others on all DRF.Clinical Relevance- Low-Dose PET, PET recovery, GAN, task driven image synthesis, super resolution.

End-to-End 3D Liver CT Image Synthesis from Vasculature Using a Multi-Task Conditional Generative Adversarial Network

Acquiring relevant, high-quality, and heterogeneous medical images is essential in various types of automated analysis, used for a variety of downstream data augmentation tasks. However, a large number of real image samples are expensive to obtain, especially for 3D medical images. Therefore, there is an urgent need to synthesize realistic 3D medical images. However, the existing generator models have poor stability and lack the guidance of prior medical knowledge. To this end, we propose a multi-task (i.e., segmentation task and generation task) 3D generative adversarial network (GAN) for the synthesis of 3D liver CT images (3DMT-GAN). To the best of our knowledge, this is the first application for a 3D liver CT image synthesis task. Specifically, we utilize a mask of vascular segmentation as the input because it contains structural information about a variety of rich anatomical structures. We use the semantic mask of the liver as prior medical knowledge to guide the 3D CT image generation, reducing the calculation of a large number of backgrounds, thus making the model more focused on the generation of the region of the liver. In addition, we introduce a stable multiple gradient descent algorithm (MGDA) reconstruction method into our model to balance the weights of the multi-task framework. Experiments were conducted on a real dataset, and the experimental results show that the segmentation task achieves a Dice similarity coefficient (DSC) of 0.87, while the synthesis task outperforms existing state-of-the-art methods. This study demonstrates the feasibility of using vascular images to synthesize images of the liver.

Wor(l)d-GAN: Toward Natural-Language-Based PCG in Minecraft

This work presents Wor(l)d-GAN, a method to perform data-driven Procedural Content Generation via Machine Learning in Minecraft from a single example. Based on a 3D Generative Adversarial Network (GAN) architecture, we are able to create arbitrarily sized world snippets from a given sample. Our method applies dense representations used in Natural Language Processing (NLP) in two ways. First, we propose block2vec representations based on word2vec. Secondly, we use the pretrained large language model BERT to generate representations directly from the token names. These representations make Wor(l)d-GAN independent from the number of different blocks, which can vary a lot in Minecraft, and enable the generation of larger levels. We evaluate our approach on creations from the community as well as structures generated with the Minecraft World Generator under several metrics. Wor(l)d-GAN enables its users to generate Minecraft worlds based on parts of their creations.

Speculative hybrids: Investigating the generation of conceptual architectural forms through the use of 3D generative adversarial networks

The process of architectural design aims at solving complex problems that have loosely defined formulations, no explicit basis for terminating the problem-solving activity, and where no ideal solution can be achieved. This means that design problems, as wicked problems, sit in a space between incompleteness and precision. Applying digital tools in general and artificial intelligence in particular to design problems will then mediate solution spaces between incompleteness and precision. In this paper, we present a study where we employed machine learning algorithms to generate conceptual architectural forms for site-specific regulations. We created an annotated dataset of single-family homes and used it to train a 3D Generative Adversarial Network that generated annotated point clouds complying with site constraints. Then, we presented the framework to 23 practitioners of architecture in an attempt to understand whether this framework could be a useful tool for early-stage design. We make a three-fold contribution: First, we share an annotated dataset of architecturally relevant 3D point clouds of single-family homes. Next, we present and share the code for a framework and the results from training the 3D generative neural network. Finally, we discuss machine learning and creative work, including how practitioners feel about the emergence of these tools as mediators between incompleteness and precision in architectural design.

Combining physics-based models with deep learning image synthesis and uncertainty in intraoperative cone-beam CT of the brain.

Image-guided neurosurgery requires high localization and registration accuracy to enable effective treatment and avoid complications. However, accurate neuronavigation based on preoperative magnetic resonance (MR) or computed tomography (CT) images is challenged by brain deformation occurring during the surgical intervention. To facilitate intraoperative visualization of brain tissues and deformable registration with preoperative images, a 3D deep learning (DL) reconstruction framework (termed DL-Recon) was proposed for improved intraoperative cone-beam CT (CBCT) image quality. The DL-Recon framework combines physics-based models with deep learning CT synthesis and leverages uncertainty information to promote robustness to unseen features. A 3D generative adversarial network (GAN) with a conditional loss function modulated by aleatoric uncertainty was developed for CBCT-to-CT synthesis. Epistemic uncertainty of the synthesis model was estimated via Monte Carlo (MC) dropout. Using spatially varying weights derived from epistemic uncertainty, the DL-Recon image combines the synthetic CT with an artifact-corrected filtered back-projection (FBP) reconstruction. In regions of high epistemic uncertainty, DL-Recon includes greater contribution from the FBP image. Twenty paired real CT and simulated CBCT images of the head were used for network training and validation, and experiments evaluated the performance of DL-Recon on CBCT images containing simulated and real brain lesions not present in the training data. Performance among learning- and physics-based methods was quantified in terms of structural similarity (SSIM) of the resulting image to diagnostic CT and Dice similarity metric (DSC) in lesion segmentation compared to ground truth. A pilot study was conducted involving seven subjects with CBCT images acquired during neurosurgery to assess the feasibility of DL-Recon in clinical data. CBCT images reconstructed via FBP with physics-based corrections exhibited the usual challenges to soft-tissue contrast resolution due to image non-uniformity, noise, and residual artifacts. GAN synthesis improved image uniformity and soft-tissue visibility but was subject to error in the shape and contrast of simulated lesions that were unseen in training. Incorporation of aleatoric uncertainty in synthesis loss improved estimation of epistemic uncertainty, with variable brain structures and unseen lesions exhibiting higher epistemic uncertainty. The DL-Recon approach mitigated synthesis errors while maintaining improvement in image quality, yielding 15%-22% increase in SSIM (image appearance compared to diagnostic CT) and up to 25% increase in DSC in lesion segmentation compared to FBP. Clear gains in visual image quality were also observed in real brain lesions and in clinical CBCT images. DL-Recon leveraged uncertainty estimation to combine the strengths of DL and physics-based reconstruction and demonstrated substantial improvements in the accuracy and quality of intraoperative CBCT. The improved soft-tissue contrast resolution could facilitate visualization of brain structures and support deformable registration with preoperative images, further extending the utility of intraoperative CBCT in image-guided neurosurgery.