Surrogate Machine Learning Model Research Articles

Large-Scale Pretraining Improves Sample Efficiency of Active Learning-Based Virtual Screening.

Virtual screening of large compound libraries to identify potential hit candidates is one of the earliest steps in drug discovery. As the size of commercially available compound collections grows exponentially to the scale of billions, active learning and Bayesian optimization have recently been proven as effective methods of narrowing down the search space. An essential component of those methods is a surrogate machine learning model that predicts the desired properties of compounds. An accurate model can achieve high sample efficiency by finding hits with only a fraction of the entire library being virtually screened. In this study, we examined the performance of a pretrained transformer-based language model and graph neural network in a Bayesian optimization active learning framework. The best pretrained model identifies 58.97% of the top-50,000 compounds after screening only 0.6% of an ultralarge library containing 99.5 million compounds, improving 8% over the previous state-of-the-art baseline. Through extensive benchmarks, we show that the superior performance of pretrained models persists in both structure-based and ligand-based drug discovery. Pretrained models can serve as a boost to the accuracy and sample efficiency of active learning-based virtual screening.

Bayesian Proper Orthogonal Decomposition for Learnable Reduced-Order Models With Uncertainty Quantification

Designing and/or controlling complex systems in science and engineering relies on appropriate mathematical modeling of systems dynamics. Classical differential equation based solutions in applied and computational mathematics are often computationally demanding. Recently, the connection between reduced-order models of high-dimensional differential equation systems and surrogate machine learning models has been explored. However, the focus of both existing reduced-order and machine learning models for complex systems has been how to best approximate the high fidelity model of choice. Due to high complexity and often limited training data to derive reduced-order or machine learning surrogate models, it is critical for derived reduced-order models to have reliable uncertainty quantification at the same time. In this paper, we propose such a novel framework of Bayesian reduced-order models naturally equipped with uncertainty quantification as it learns the distributions of the parameters of the reduced-order models instead of their point estimates. In particular, we develop <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">learnable Bayesian proper orthogonal decomposition</i> (BayPOD) that learns the distributions of both the POD projection bases and the mapping from the system input parameters to the projected scores/coefficients so that the learned BayPOD can help predict high-dimensional systems dynamics/fields as quantities of interest in different setups with reliable uncertainty estimates. The developed learnable BayPOD inherits the capability of embedding physics constraints when learning the POD-based surrogate reduced-order models, a desirable feature when studying complex systems in science and engineering applications where the available training data are limited. Furthermore, the proposed BayPOD method is an end-to-end solution, which unlike other surrogate-based methods, does not require separate POD and machine learning steps. The results from a real-world case study of the pressure field around an airfoil have shown the potential of learnable BayPOD as a new family of reduced-order models with reliable uncertainty estimates.

QMugs 1.1: Quantum mechanical properties of organic compounds commonly encountered in reactivity datasets

Here, the Quantum Mechanical Properties of Drug-like Molecules (QMugs) dataset is expanded to facilitate its use as training data for surrogate machine learning models to predict quantum mechanical properties for tasks related to chemical reactivity. Small molecules from reaction databases as well as charged and boron-containing compounds from ChEMBL were added. Each of these compounds was passed through a pipeline of MMFF94s/UFF conformer generation, followed by GFN2-xTB optimization and finally a density functional theory single-point calculation at the ωB97X-D/def2-SVP level of theory. In total, 71,632 new molecules were evaluated in this manner. Steric (SASA) and dispersion (Pint) descriptors were computed at the semiempirical GFN2-xTB level of theory for the lowest energy conformer of all species in the enlarged QMugs dataset. The expanded dataset aims to facilitate the construction of surrogate models of much broader scope than the original QMugs dataset which was limited to biologically active compounds.

A Surrogate Machine Learning Model for the Design of Single-Atom Catalyst on Carbon and Porphyrin Supports towards Electrochemistry

We apply the machine learning (ML) tool to calculate the Gibbs free energy (ΔG) of reaction intermediates rapidly and accurately as a guide for designing porphyrin- and graphene-supported single-atom catalysts (SACs) toward electrochemical reactions. Based on the 2105 DFT calculation data from the literature, we trained a support vector machine (SVR) algorithm. The hyperparameters were optimized using Bayesian optimization along with 10-fold cross-validation to avoid overfitting. Based on the Shapley Additive exPlanation (SHAP) and permutation methods, the feature importance analysis suggests that the most important parameters are the number of pyridinic nitrogen (Npy), the number of d electrons (θd), and the number of valence electrons of reaction intermediates. Inspired by this feature importance analysis and the Pearson correlation coefficient, we found a linear dependent, simple, and general descriptor (φ) to describe ΔG of reaction intermediates (e.g., ΔGOH* = 0.020φ – 2.190). Using the trained SVR algorithm, ΔGOH*, ΔGO*, ΔGOOH*, ΔGOO*, ΔGH*, ΔGCOOH*, ΔGCO*, and ΔGN2* intermediates are predicted for the oxygen reduction reaction (ORR), the oxygen evolution reaction (OER), the hydrogen evolution reaction (HER), and the CO2 reduction reaction (CO2RR). The SVR model predicts an ORR overpotential of 0.51 V and an HER overpotential of 0.22 V for FeN4-SAC. Moreover, we used the SVR algorithm for high-throughput screening of SACs, suggesting new SACs with low ORR overpotentials. This strategy provides a data-driven catalyst design method that significantly reduces the costs of DFT calculations while providing the means for designing SACs for electrocatalysis and beyond.

Bridging the gap between mechanistic biological models and machine learning surrogates.

Mechanistic models have been used for centuries to describe complex interconnected processes, including biological ones. As the scope of these models has widened, so have their computational demands. This complexity can limit their suitability when running many simulations or when real-time results are required. Surrogate machine learning (ML) models can be used to approximate the behaviour of complex mechanistic models, and once built, their computational demands are several orders of magnitude lower. This paper provides an overview of the relevant literature, both from an applicability and a theoretical perspective. For the latter, the paper focuses on the design and training of the underlying ML models. Application-wise, we show how ML surrogates have been used to approximate different mechanistic models. We present a perspective on how these approaches can be applied to models representing biological processes with potential industrial applications (e.g., metabolism and whole-cell modelling) and show why surrogate ML models may hold the key to making the simulation of complex biological systems possible using a typical desktop computer.

Machine learning model for transient exergy performance of a phase change material integrated-concentrated solar thermoelectric generator

Despite the merits of incorporating phase change materials in concentrating solar thermoelectric generating systems, the following research gaps still need to be filled to make the technology viable. Past research efforts focused on pulsed rectangular and sinusoidal heat flux forms which differ from actual transient weather data obtainable under field exposure. Additionally, a comprehensive exergy performance modeling of the system is yet to be presented in the literature, hence, there is no known thermodynamic implication of incorporating phase change materials in thermoelectric systems from a first and second law perspective. Finally, optimization insights obtained for designing these systems are based on inefficient, computationally expensive, and time-consuming numerical solvers or experimental methods which significantly hinder the ease at which useful design insights are obtained. This work aims to fill in these gaps by presenting the first-ever exergy performance investigation of a paraffin wax integrated thermoelectric generating system using data-driven surrogate machine learning models trained with numerically generated data computed from actual 12 h transient timeseries weather data collected every minute and averaged for a 10-year period (2010–2020). Results show that for an optical concentration of 10, the incorporated system with phase change material provides a 25%, 57%, 25%, and 56% improvement in the thermoelectric temperature difference, power generation, exergy efficiency, and thermodynamic irreversibility of the stand-alone system, respectively, without phase change material. Furthermore, the incorporated system generates electricity for 2 more hours into the night when solar irradiance is unavailable. Finally, the cheap surrogate machine learning model accurately predicts the thermodynamic performance of the stand-alone and incorporated systems with an almost zero error and almost unity coefficient of determination in just 5 s after the numerical method has taken 24 h to generate the data, indicating a 17,280-computation time saving.

Machine learning a surrogate model for transmission loss on a Pekeris waveguide

The connection between ocean environmental parameters and transmission loss (TL) can be modeled using sound propagation models. We can analyze and improve these models using information theoretic tools such as the Fisher information matrix (FIM) and methods for model reduction, where we identify parameters that can be removed from a model without sacrificing accuracy. Critical to these methods for model reduction is the ability to evaluate derivatives of the model predictions of TL with respect to model parameters. Calculating derivatives of TL models using finite difference methods with sufficient accuracy is challenging due to the complexity of sound fields in an ocean environment; instead, we train a surrogate machine learning model to which we can apply automatic differentiation (AD). We propose a general method for model reduction in which a ML model is used as a surrogate model for physical sound propagation ocean models, which is then used to find potential simplifications of the original model. We demonstrate this method using the Pekeris waveguide model of the ocean, which calculates underwater acoustic TL in an ocean environment due to seafloor characteristics. [Work supported by Office of Naval Research]

Adapting Data-Driven Techniques to Improve Surrogate Machine Learning Model Performance

We demonstrate the adaption of three established methods to the field of surrogate machine learning model development. These methods are data augmentation, custom loss functions and fine-tuning of pre-trained models. Each of these methods have seen widespread use in the field of machine learning, however, here we apply them specifically to surrogate machine learning model development. The machine learning model that forms the basis behind this work was intended to surrogate a traditional engineering model used in the UK nuclear industry. Previous performance of this model has been hampered by poor performance due to limited training data. Here, we demonstrate that through a combination of additional techniques, model performance can be significantly improved. We show that each of the aforementioned techniques have utility in their own right and in combination with one another. However, we see them best applied when used to fine-tune existing models. Five pre-trained surrogate models produced prior to this study were further trained using an augmented dataset and with our custom loss function. Through the combination of all three techniques, we see an improvement of at least 38% in performance across the five models.

Generative organic electronic molecular design informed by quantum chemistry.

Generative molecular design strategies have emerged as promising alternatives to trial-and-error approaches for exploring and optimizing within large chemical spaces. To date, generative models with reinforcement learning approaches have frequently used low-cost methods to evaluate the quality of the generated molecules, enabling many loops through the generative model. However, for functional molecular materials tasks, such low-cost methods are either not available or would require the generation of large amounts of training data to train surrogate machine learning models. In this work, we develop a framework that connects the REINVENT reinforcement learning framework with excited state quantum chemistry calculations to discover molecules with specified molecular excited state energy levels, specifically molecules with excited state landscapes that would serve as promising singlet fission or triplet-triplet annihilation materials. We employ a two-step curriculum strategy to first find a set of diverse promising molecules, then demonstrate the framework's ability to exploit a more focused chemical space with anthracene derivatives. Under this protocol, we show that the framework can find desired molecules and improve Pareto fronts for targeted properties versus synthesizability. Moreover, we are able to find several different design principles used by chemists for the design of singlet fission and triplet-triplet annihilation molecules.

A multi-objective optimization design to generate surrogate machine learning models in explainable artificial intelligence applications

A multi-objective optimization design to generate surrogate machine learning models in explainable artificial intelligence applications

A predictive computational platform for optimizing the design of bioartificial pancreas devices

The delivery of encapsulated islets or stem cell-derived insulin-producing cells (i.e., bioartificial pancreas devices) may achieve a functional cure for type 1 diabetes, but their efficacy is limited by mass transport constraints. Modeling such constraints is thus desirable, but previous efforts invoke simplifications which limit the utility of their insights. Herein, we present a computational platform for investigating the therapeutic capacity of generic and user-programmable bioartificial pancreas devices, which accounts for highly influential stochastic properties including the size distribution and random localization of the cells. We first apply the platform in a study which finds that endogenous islet size distribution variance significantly influences device potency. Then we pursue optimizations, determining ideal device structures and estimates of the curative cell dose. Finally, we propose a new, device-specific islet equivalence conversion table, and develop a surrogate machine learning model, hosted on a web application, to rapidly produce these coefficients for user-defined devices.

A surrogate machine learning model for advanced gas-cooled reactor graphite core safety analysis

A surrogate machine learning model was developed with the aim of predicting seismic graphite core displacements from crack configurations for the advanced gas-cooled reactor. The model was trained on a dataset generated by a software package which simulates the behaviour of the graphite core during a severe earthquake. Several machine learning techniques, such as the use of convolutional neural networks, were identified as highly applicable to this particular problem. Through the development of the model, several observations and insights were garnered which may be of interest from a graphite core analysis and safety perspective. The best performing model was capable of making 95% of test set predictions within a 20 percentage point margin of the ground truth.

Machine learning based inverse design of complex microstructures generated via hierarchical wrinkling

Hierarchical wrinkling is a buckling-based low-cost technique for fabrication of complex microstructures over large areas. However, predicting and fabricating the desired microstructures is challenging due to the underlying nonlinear mechanics. Although computational predictive tools based on finite elements analysis (FEA) are available, these are impractical for inverse design, i.e., for the prediction of inputs that would generate the desired wrinkles. This is because one must resort to slow and resource-intensive iterations to predict the inputs. Here, we present a prediction technique that is based on machine learning (ML) to overcome this limitation. Our technique comprises: (i) capturing the predictive capability of FEA through computationally-efficient surrogate ML models, (ii) applying the models to map the feasible input design space into the outputs, i.e., the predicted wrinkles, and (iii) searching through the predicted wrinkles for the desired microstructure. We have validated our technique against FEA simulations and have demonstrated its utility by rapidly identifying the inputs that would generate desired hierarchical wrinkles including flat-top sinusoidal patterns. Our approach shortens the exploration and comparison of a million design choices from more than ten days to less than a minute. It can therefore significantly mature wrinkling into a scalable manufacturing process for optics, sensing, and surface texturing applications.

Prediction of the Electron Density of States for Crystalline Compounds with Atomistic Line Graph Neural Networks (ALIGNN)

Machine learning (ML)-based models have greatly enhanced the traditional materials discovery and design pipeline. Specifically, in recent years, surrogate ML models for material property prediction have demonstrated success in predicting discrete scalar-valued target properties to within reasonable accuracy of their DFT-computed values. However, accurate prediction of spectral targets, such as the electron density of states (DOS), poses a much more challenging problem due to the complexity of the target, and the limited amount of available training data. In this study, we present an extension of the recently developed atomistic line graph neural network to accurately predict DOS of a large set of material unit cell structures, trained to the publicly available JARVIS-DFT dataset. Furthermore, we evaluate two methods of representation of the target quantity: a direct discretized spectrum, and a compressed low-dimensional representation obtained using an autoencoder. Through this work, we demonstrate the utility of graph-based featurization and modeling methods in the prediction of complex targets that depend on both chemistry and directional characteristics of material structures.

Machine learning for composite structure optimization

This paper considers the problem of composite material dynamic strength. Intensive shock load applied to a composite specimen is modeled using grid-characteristic numerical method. Three-dimensional patterns of wave propagation and stress distribution are calculated for each set of problem parameters. External load and specimen parameters are varied to study the dynamic strength against different types of load. A fast surrogate machine learning model is built for this problem. This model can be used for fast search and design of the materials with given mechanical properties.

A Surrogate Machine Learning Model for Advanced Gas-Cooled Reactor Graphite Core Safety Analysis

A Surrogate Machine Learning Model for Advanced Gas-Cooled Reactor Graphite Core Safety Analysis

Integrative Evolutionary-Based Method for Modeling and Optimizing Budget Assignment of Bridge Maintenance Priorities

Recently, the number of deteriorating bridges has drastically increased. As such, enormous amounts of resources are invested yearly to maintain the performance of bridges within acceptable levels. This entails the development of bridge management systems to manage the imbalance between the extensive needs for maintenance, repair and rehabilitation actions, and the limited available funds. In this regard, the present study introduces a three-tier platform to model and allocate limited resources in maintenance projects. The first model involves building a discrete event simulation model to mimic the bridge deck replacement process. The second encompasses structuring an efficient and straightforward surrogate machine learning model for mimicking the computationally expensive discrete event simulation model. In the second phase, a novel hybrid Elman neural-network invasive-weed optimization model is developed for predicting the time, cost, greenhouse gases, and utilization rates of resource allocation plans using a database generated from the previous model. The third constitutes the formulation of a multiobjective differential evolution optimization model subject to the utilization rates of the involved resources and their dispersion. Results manifest superiority in cost prediction accuracies, achieving a mean absolute percentage error, mean absolute error, and root-mean-squared error of 4.873%, 78.466%, and 39.515%, respectively. Additionally, the developed multiobjective optimization model significantly outperformed a set of well-performing metaheuristics, yielding a hypervolume indicator, generational distance, spacing, diversity, spread, and coverage of 81.721%, 0.029%, 0.1881%, 0.5229%, 0.9618%, and 0.4087%, respectively. The results also demonstrate the developed multiobjective optimization model accomplished an improvement in the minimization time, cost, and greenhouse gases by 71.01%, 27.87%, and 39.29%, respectively, when compared against a genetic algorithm. The developed models are automated through the hybrid programming of C#.net and MATLAB. It is expected that the developed method can enable the practitioners and transportation agencies to establish timely-efficient, cost-effective, and sustainable resource allocation plans while accommodating the efficacious utilization of resources.

Predicting concrete compressive strength using hybrid ensembling of surrogate machine learning models

This study aims to implement a hybrid ensemble surrogate machine learning technique in predicting the compressive strength (CS) of concrete, an important parameter used for durability design and service life prediction of concrete structures in civil engineering projects. For this purpose, an experimental database consisting of 1030 records has been compiled from the machine learning repository of the University of California, Irvine. The database was used to train and validate four conventional machine learning (CML) models, namely Artificial Neural Network (ANN), Linear and Non-Linear Multivariate Adaptive Regression Splines (MARS-L and MARS-C), Gaussian Process Regression (GPR), and Minimax Probability Machine Regression (MPMR). Subsequently, the predicted outputs of CML models were combined and trained using ANN to construct the Hybrid Ensemble Model (HENSM). It is observed that the proposed HENSM produces higher predictive accuracy compared to the CML models used in the present study. The predictive performance of all models for CS prediction was compared using the testing dataset and it is found that the HENSM model attained the highest predictive accuracy in both phases. Based on the experimental results, the newly constructed HENSM model is very potential to be a new alternative in handling the overfitting issues of CML models and hence, can be used to predict the concrete CS, including the design of less polluting and more sustainable concrete constructions.

A machine learning framework for predicting the shear strength of carbon nanotube-polymer interfaces based on molecular dynamics simulation data

Modern aerospace applications require lightweight materials with exceptionally high strength and stiffness. Carbon nanotube (CNT)-reinforced composites have great potential in addressing these requirements. However, one critical factor limiting the potential of CNT-reinforced composites is the limited load transfer capability between CNTs through a polymer matrix, which arises due to low CNT-polymer interfacial shear strength at a molecular scale. While molecular dynamics (MD) simulations can be employed to investigate the CNT-polymer interface, such simulations are computationally expensive. It is thus intractable to explore a sufficiently large design space for interface modifications and optimization using MD simulations alone, motivating the use of surrogate models to efficiently map CNT-polymer interfacial configurations to interfacial strength. In this paper, we develop a surrogate machine learning model trained with MD models of functionalized CNT-epoxy and the corresponding interfacial shear strength. The proposed ML model consists of (i) a feature representation of CNT-epoxy based on radial distribution functions; (ii) a convolutional neural network model to map feature representations to a metric of interfacial shear strength; and (iii) a data augmentation method to maximize the use of limited training data. The proposed ML model can predict the CNT pullout force quite well despite a large variation associated with the pullout force.

A Novel Hybrid Model Based on a Feedforward Neural Network and One Step Secant Algorithm for Prediction of Load-Bearing Capacity of Rectangular Concrete-Filled Steel Tube Columns.

In this study, a novel hybrid surrogate machine learning model based on a feedforward neural network (FNN) and one step secant algorithm (OSS) was developed to predict the load-bearing capacity of concrete-filled steel tube columns (CFST), whereas the OSS was used to optimize the weights and bias of the FNN for developing a hybrid model (FNN-OSS). For achieving this goal, an experimental database containing 422 instances was firstly gathered from the literature and used to develop the FNN-OSS algorithm. The input variables in the database contained the geometrical characteristics of CFST columns, and the mechanical properties of two CFST constituent materials, i.e., steel and concrete. Thereafter, the selection of the appropriate parameters of FNN-OSS was performed and evaluated by common statistical measurements, for instance, the coefficient of determination (R2), root mean square error (RMSE), and mean absolute error (MAE). In the next step, the prediction capability of the best FNN-OSS structure was evaluated in both global and local analyses, showing an excellent agreement between actual and predicted values of the load-bearing capacity. Finally, an in-depth investigation of the performance and limitations of FNN-OSS was conducted from a structural engineering point of view. The results confirmed the effectiveness of the FNN-OSS as a robust algorithm for the prediction of the CFST load-bearing capacity.