Uncertainty quantification of compressive stress response in expanded polystyrene foams using evidential neural networks

Uncertainty quantification: Can we trust artificial intelligence in drug discovery?

Purpose: To derive a multinomial probability function and quantitative measures of the data and epistemic uncertainty as direct output of a 3D U-Net segmentation network. Approach: A set of T1 brain MRI images were downloaded from the Connectome Project and segmented using FMRIB's FAST algorithm to be used as ground truth. A 3D U-Net neural network was trained with sample sizes of 200, 500, and 898 T1 brain images using a loss function defined as the negative logarithm of the likelihood based on a derivation of the definition of the multinomial probability function. From this definition, the epistemic and aleatoric uncertainty equations were derived and used to quantify maps of the uncertainty along with tissue segmentations. Results: Maps of the tissue segmentation along with the epistemic and aleatoric uncertainty, per voxel, are presented. The uncertainty decreased based on the increasing number of training data used to train the neural network. The neural network trained with 898 volumes resulted in uncertainty maps that were high primarily in the tissue boundary regions. The epistemic and aleatoric uncertainty were averaged over all test data (connectome and tumor separately), and the epistemic uncertainty showed a decreasing trend, as expected, with increasing numbers of data used to train the model. The aleatoric uncertainty showed a similar trend which was also expected as the aleatoric uncertainty is not expected to be as dependent on the number of training data. Conclusion: The derived uncertainty equations from a multinomial probability distribution were able to quantify the aleatoric and epistemic uncertainty per voxel and are applicable for all two-dimensional and three-dimensional neural networks.

Traditional modeling of mechanical energy absorption due to compressive loadings in expanded polystyrene foams involves mathematical descriptions that are derived from stress/strain continuum mechanics models. Nevertheless, most of those models are either constrained using the strain as the only variable to work at large deformation regimes and usually neglect important parameters for energy absorption properties such as the material density or the rate of the applying load. This work presents a neural-network-based approach that produces models that are capable to map the compressive stress response and energy absorption parameters of an expanded polystyrene foam by considering its deformation, compressive loading rates, and different densities. The models are trained with ground-truth data obtained in compressive tests. Two methods to select neural network architectures are also presented, one of which is based on a Design of Experiments strategy. The results show that it is possible to obtain a single artificial neural networks model that can abstract stress and energy absorption solution spaces for the conditions studied in the material. Additionally, such a model is compared with a phenomenological model, and the results show than the neural network model outperforms it in terms of prediction capabilities, since errors around 2% of experimental data were obtained. In this sense, it is demonstrated that by following the presented approach is possible to obtain a model capable to reproduce compressive polystyrene foam stress/strain data, and consequently, to simulate its energy absorption parameters.

Towards the NASA UQ Challenge 2019: Systematically forward and inverse approaches for uncertainty propagation and quantification

Epistemic uncertainty quantification provides useful insight into both deep and shallow neural networks' understanding of the relationships between their training distributions and unseen instances and can serve as an estimate of classification confidence. Bayesian-based approaches have been shown to quantify this relationship better than softmax probabilities. Those approaches to uncertainty quantification, however, require multiple Monte-Carlo samplings of a neural network, augmenting the neural network to learn distributions for its weights, or utilizing an ensemble of neural networks. Current research has yielded several techniques that estimate uncertainty deterministically but tend to struggle in independent evaluation using real-world data and tasks compared to stochastic, or variational, techniques such as Bayesian dropout. In this work, we propose a technique that allows Bayesian dropout uncertainty to be estimated using learned regression models. We find that, once trained, this allows dropout uncertainty to be effectively and efficiently predicted.

Robust quantification of predictive uncertainty is a critical addition needed for machine learning applied to weather and climate problems to improve the understanding of what is driving prediction sensitivity. Ensembles of machine learning models provide predictive uncertainty estimates in a conceptually simple way but require multiple models for training and prediction, increasing computational cost and latency. Parametric deep learning can estimate uncertainty with one model by predicting the parameters of a probability distribution but does not account for epistemic uncertainty. Evidential deep learning, a technique that extends parametric deep learning to higher-order distributions, can account for both aleatoric and epistemic uncertainties with one model. This study compares the uncertainty derived from evidential neural networks to that obtained from ensembles. Through applications of the classification of winter precipitation type and regression of surface-layer fluxes, we show evidential deep learning models attaining predictive accuracy rivaling standard methods while robustly quantifying both sources of uncertainty. We evaluate the uncertainty in terms of how well the predictions are calibrated and how well the uncertainty correlates with prediction error. Analyses of uncertainty in the context of the inputs reveal sensitivities to underlying meteorological processes, facilitating interpretation of the models. The conceptual simplicity, interpretability, and computational efficiency of evidential neural networks make them highly extensible, offering a promising approach for reliable and practical uncertainty quantification in Earth system science modeling. To encourage broader adoption of evidential deep learning, we have developed a new Python package, Machine Integration and Learning for Earth Systems (MILES) group Generalized Uncertainty for Earth System Science (GUESS) (MILES-GUESS) ( https://github.com/ai2es/miles-guess ), that enables users to train and evaluate both evidential and ensemble deep learning. Significance Statement This study demonstrates a new technique, evidential deep learning, for robust and computationally efficient uncertainty quantification in modeling the Earth system. The method integrates probabilistic principles into deep neural networks, enabling the estimation of both aleatoric uncertainty from noisy data and epistemic uncertainty from model limitations using a single model. Our analyses reveal how decomposing these uncertainties provides valuable insights into reliability, accuracy, and model shortcomings. We show that the approach can rival standard methods in classification and regression tasks within atmospheric science while offering practical advantages such as computational efficiency. With further advances, evidential networks have the potential to enhance risk assessment and decision-making across meteorology by improving uncertainty quantification, a longstanding challenge. This work establishes a strong foundation and motivation for the broader adoption of evidential learning, where properly quantifying uncertainties is critical yet lacking.

Breast cancer is the most common cause of cancer in women. Histopathological imaging data can provide important information on cancer since it preserves the underlying tissue architecture in the preparation process. Accurate and automated classification of breast tissue into malignant or healthy from histological images can be used for the diagnosis of breast cancer. However, publicly available labeled histopathological datasets are limited in size and also biased. For such datasets, existing machine learning classifiers have shown limited success. The goal of the present work is to develop a classification technique using machine learning, which can overcome the challenge posed by small and biased datasets. This technique will reduce the inaccuracy of the image analysis, and quantify the uncertainty in their prediction. The field of computer vision and neural networks are aimed at improving the accuracy of image analysis by various network architectures on algorithms. Plain feed-forward neural networks have been successfully used for pattern recognition, but their performance is good only when the data is sufficiently large. When the data size is limited, neural networks yield erroneous or overfitted results, since they don't take the uncertainty of the dataset into account. These feed-forward neural networks learn their weights as point estimates or a deterministic value. Whereas, a Bayesian neural network learns a probability distribution on the weights. The loss function used in Bayesian neural networks is known as a Variational Free Energy (VFE) or Evidence Lower Bound (ELBO) which is to be optimized. Since the Bayesian approach provides probability distributions on the weights of the neural networks, it is possible to calculate the variance of the predictive posterior probability distribution, which is the sum of aleatoric and the epistemic uncertainty. Uncertainty quantification, along with the point estimate, leads to a more informed decision, improved accuracy, and reduced overfitting. In critical applications, especially medical-imaging applications, uncertainty quantification can potentially reduce the unexpected and incorrect results due to the poor decisions. In this thesis, we have worked on the application of Bayesian neural networks on publicly available histopathological images for the detection of breast cancer and uncertainty quantification of the prediction. We have demonstrated that using the Bayesian CNN, the false-negative predictions can be reduced remarkably, by almost 22%. We have found that the predictions associated with higher epistemic uncertainties have features of both the classes. These findings should improve the state of the art machine learning-based biomedical imaging.

Expanded polystyrene foams are widely used materials for various applications in engineering, including their use for protective designs. For this type of application, in engineering analysis and design, it is required to know the mechanical response to compression of this type of material, since energy parameters that support the analysis of the effectiveness of a design are derived from it. One of these parameters is strain hysteresis, through which it is possible to know how capable a material is of absorbing energy. The modeling and prediction of this parameter is a challenge from the analysis point of view. This contribution presents a method based on feed-forward artificial neural network models that address a modeling approach to derive this parameter from the mechanical response of expanded polystyrene foam. From this, models are constructed that can predict the response of such material to various density and loading rate conditions. The best of a total of 30 neural network models, which are capable of deriving energy parameters such as hysteresis, is chosen. The results show that this approach is valid for the deformation energy analysis of expanded polystyrene foams since results consistent with the material phenomenology and errors of less than 3% with respect to experimental data are obtained.

Although deep learning applicability in various fields of earth sciences is rapidly increasing, shallow multilayer-perceptron neural networks remain widely used for regression problems. Despite many clear distinctions between deep and shallow neural networks, some techniques developed for deep learning may help improve shallow models. Dropout, a simple approach to avoid overfitting by randomly skipping some nodes in a net during each training iteration, is among methodological features that made deep learning networks successful. In this study we give a review of dropout methods and empirically show that, when used together with early-stopping, dropout and its variant dropconnect could improve performance of shallow multi-layer perceptron neural networks. Shallow neural networks are applied to streamwater temperature modelling problems in six catchments, based on air temperature, river discharge and declination of the Sun. We found that when training of a particular neural network architecture that includes at least a few hidden nodes is repeated many times, dropout reduces the number of models that perform poorly on testing data, and hence improves the mean performance. If the number of inputs or hidden nodes is very low, dropout only disturbs training. However, nodes need to be dropped out with a much lower probability than in the case of deep neural networks (about 1%, instead of 10–50% for deep learning), due to a much smaller number of nodes in the network. Larger probabilities of dropping out nodes hinder convergence of the training algorithm and lead to poor results for both calibration and testing data. Dropconnect turned out to be slightly more effective than dropout.

This research presents a method to analyze how neural network models, applied to Expanded Polypropylene and Expanded Polystyrene foams, predict their compressive stress responses. By using SHAP values and Partial Dependence Plots, the study elucidates the models’ decision-making processes. It focuses on three main features for both materials: density, loading rate, and strain, with an additional feature concerning loading and unloading for Expanded Polystyrene foam. The findings highlight that increased density and loading rate are closely correlated with higher compressive responses, and strain emerges as the most influential factor for the response of both materials. Partial Dependence Plots reveal a linear relationship with density, whereas other variables demonstrate non-linear relationships. These results validate the use of neural networks in analyzing material behavior, showing that the models’ outputs are in line with empirical observations. In conclusion, as presented, the integration of interpretability tools with neural network models offers a robust method for material response analysis, contributing to a deeper understanding of material science.

Expanded polystyrene is used in diverse applications, notably for protective and structural purposes. Its cushioning and mechanical strength excel under compressive loads, especially when optimally designed. A key factor influencing its compressive stress is the initial density, which plays a significant role in determining the material’s mechanical properties. This aspect is primarily determined by the bead size distribution. Although there is a vast body of literature on modeling the stress response of expanded polystyrene, there is limited emphasis on predictions that account for this factor, which is also relevant for the manufacturing of the material. Recent literature has emphasized the capability of artificial neural networks in predicting the compressive behaviors of expanded polystyrene, incorporating various factors. In this study, artificial neural network models were used to predict the compressive stress responses of polystyrene foams, with a focus on bead size distribution parameters. Specimens of two distinct initial densities were examined using micrographs to identify bead diameters and distributions, which were then used as model inputs. Compression tests on these specimens were conducted at two different rates. The collected data facilitated the development of predictive models for the material’s compressive behavior. The model predictions closely match experimental findings, with error metrics showing deviations <3% compared to the experimental data. This highlights the utility of artificial neural networks in modeling the compressive behavior of polystyrene foams, particularly when bead size and related parameters are considered.

Exploration is a significant challenge in practical reinforcement learning (RL), and uncertainty-aware exploration that incorporates the quantification of epistemic and aleatory uncertainty has been recognized as an effective exploration strategy. However, capturing the combined effect of aleatory and epistemic uncertainty for decision-making is difficult. Existing works estimate aleatory and epistemic uncertainty separately and consider the composite uncertainty as an additive combination of the two. Nevertheless, the additive formulation leads to excessive risk-taking behavior, causing instability. In this paper, we propose an algorithm that clarifies the theoretical connection between aleatory and epistemic uncertainty, unifies aleatory and epistemic uncertainty estimation, and quantifies the combined effect of both uncertainties for a risk-sensitive exploration. Our method builds on a novel extension of distributional RL that estimates a parameterized return distribution whose parameters are random variables encoding epistemic uncertainty. Experimental results on tasks with exploration and risk challenges show that our method outperforms alternative approaches.

The expanded polystyrene foam is widely used as a protective material in engineering applications where energy absorption is critical for the reduction of harmful dynamic loads. However, to design reliable protective components, it is necessary to predict its nonlinear stress response with a good approximation, which makes it possible to know from the engineering design analysis the amount of energy that a product may absorb. In this work, the hyperfoam constitutive material model was used in a finite element model to approximate the mechanical response of an expanded polystyrene foam of three different densities. Additionally, an experimental procedure was performed to obtain the response of the material at three loading rates. The experimental results show that higher densities at high loading rates allow better energy absorption in the expanded polystyrene. As for the energy dissipation, high dissipation is obtained at higher densities at low loading rates. In the numerical results, the proposed finite element model presented a good performance since root mean square error values below 9% were obtained around the experimental compressive stress/strain curves for all tested material densities. Also, the prediction of energy absorption with the proposed model was around a maximum error of 5% regarding the experimental results. Therefore, the prediction of energy absorption and the compressive stress response of expanded polystyrene foams can be studied using the proposed finite element model in combination with the hyperfoam material model.

We study the approximation capacity of some variation spaces corresponding to shallow ReLU $$^k$$ k neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLU $$^k$$ k neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning Hölder functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.

Lightweight polypropylene/BaTiO3 composite foams with tunable dielectric constant