Neural Network Surrogate Model Research Articles

Comprehensive Botnet Detection by Mitigating Adversarial Attacks, Navigating the Subtleties of Perturbation Distances and Fortifying Predictions with Conformal Layers

Botnets are computer networks controlled by malicious actors that present significant cybersecurity challenges. They autonomously infect, propagate, and coordinate to conduct cybercrimes, necessitating robust detection methods. This research addresses the sophisticated adversarial manipulations posed by attackers, aiming to undermine machine learning-based botnet detection systems. We introduce a flow-based detection approach, leveraging machine learning and deep learning algorithms trained on the ISCX and ISOT datasets. The detection algorithms are optimized using the Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) to obtain a baseline detection method. The Carlini & Wagner (C&W) and Generative Adversarial Network (GAN) attacks generate deceptive data with subtle perturbations, targeting each feature used for classification while preserving their semantic and syntactic relationships, which ensures that the adversarial samples retain meaningfulness and realism. An in-depth analysis of the required L2 distance from the original sample for the malware sample to misclassify is performed across various iteration checkpoints, showing different levels of misclassification at different L2 distances of the pertrub sample from the original sample. Our work delves into the vulnerability of various models, examining the transferability of adversarial examples from a Neural Network surrogate model to Tree-based algorithms. Subsequently, models that initially misclassified the perturbed samples are retrained, enhancing their resilience and detection capabilities. In the final phase, a conformal prediction layer is integrated, significantly rejecting incorrect predictions — 58.20% in the ISCX dataset and 98.94% in the ISOT dataset.

Stress-constrained optimization of multiscale structures with parameterized microarchitectures using machine learning

A multiscale topology optimization framework for stress-constrained design is presented. Spatially varying microstructures are distributed in the macroscale where their material properties are estimated using a neural network surrogate model for homogenized constitutive relations. Meanwhile, the local stress state of each microstructure is evaluated with another neural network trained to emulate second-order homogenization. This combination of two surrogate models — one for effective properties, one for local stress evaluation — is shown to accurately and efficiently predict relevant stress values in structures with spatially varying microstructures. An augmented lagrangian approach to stress-constrained optimization is then implemented to minimize the volume of multiscale structures subjected to stress constraints in each microstructure. Several examples show that the approach can produce designs with varied microarchitectures that respect local stress constraints. As expected, the distributed microstructures cannot surpass density-based topology optimization designs in canonical volume minimization problems. Despite this, the stress-constrained design of hierarchical structures remains an important component in the development of multiphysics and multifunctional design. This work presents an effective approach to multiscale optimization where a machine learning approach to local analysis has increased the information exchange between micro- and macroscales.

Study of Draft Tube Optimization Using a Neural Network Surrogate Model for Micro-Francis Turbines Utilized in the Water Supply System of High-Rise Buildings

Study of Draft Tube Optimization Using a Neural Network Surrogate Model for Micro-Francis Turbines Utilized in the Water Supply System of High-Rise Buildings

Real-time channel selection in multi-mode multiplexing optical interconnection implemented by hybrid algorithm and material system

Multi-mode multiplexing optical interconnection (MMOI) has been widely used as a new technology that can significantly expand communication bandwidth. However, the constant-on state of each channel in the existing MMOI systems leads to serious interference for receivers when extracting and processing information, necessitating introducing real-time selective-on function for each channel in MMOI systems. To achieve this goal, combining several practical requirements, we propose a real-time selective mode switch based on phase-change materials, which can individually tune the passing/blocking of different modes in the bus waveguide. We utilize our proposed particle swarm optimization algorithm with embedded neural network surrogate models (NN-in-PSO) to design this mode switch. The proposed NN-in-PSO significantly reduces the optimization cost, enabling multi-dimensional simultaneous optimization. The resulting mode switch offers several advantages, including ultra-compactness, rapid tuning, nonvolatility, and large extinction ratio. Then, we demonstrate the real-time channel selection function by integrating the mode switch into the MMOI system. Finally, we prove the fabricating robustness of the proposed mode switch, which paves the way for its large-scale application.

Artificial neural network-based sequential approximate optimization of metal sheet architecture and forming process

Abstract This paper introduces a sequential approximate optimization method that combines the finite element method (FEM), dynamic differential evolution (DDE), and artificial neural network (ANN) surrogate models. The developed method is applied to address two optimization problems. The first involves metamaterial design optimization for metal sheet architecture with binary design variables. The second pertains to optimizing process parameters in multi-stage metal forming, where the discrete nature arises owing to changing tool geometries across stages. This process is highly nonlinear, accumulating contact, geometric, and material nonlinear effects discretely through forming stages. The efficacy of the proposed optimization method, utilizing ANN surrogate models, is compared with traditionally used polynomial response surface (PRS) surrogate models, primarily based on low-order polynomials. Efficient learning of ANN surrogate models is facilitated through the FEM and Python integration framework. Initial data for surrogate model training is collected via Latin hypercube sampling and FEM simulations. DDE is employed for sequential approximate optimization, optimizing ANN or PRS surrogate models to determine optimal design variables. PRS surrogate models encounter challenges in dealing with nonlinear changes in sequential approximate optimization concerning discrete characteristics such as binary design variables and discrete nonlinear behavior found in multi-stage metal forming processes. Owing to the discrete nature, PRS surrogate models require more data and iterations for optimal design variables. In contrast, ANN surrogate models adeptly predict nonlinear behavior through the activation function's characteristics. In the optimization problem of Metal Sheet Architecture for design target C, the ANN surrogate model required an average of 4.6 times fewer iterations to satisfy stopping criteria compared to the PRS surrogate model. Furthermore, in the optimization of multi-stage deep drawing processes, the ANN surrogate model required an average of 6.1 times fewer iterations to satisfy stopping criteria compared to the PRS surrogate model. As a result, the sequential global optimization method utilizing ANN surrogate models achieves optimal design variables with fewer iterations than PRS surrogate models. Further confirmation of the method's efficiency is provided by comparing Pearson correlation coefficients and locus plots.

Surrogate-model-based optimization of a nonideal polymerization reactor for producing tailored molecular weight distribution

The optimization of polymerization reactors with nonideal mixing has been greatly hindered by a tremendous computation burden. In this study, a global optimization framework to integrate full MWD calculations and detailed reactor dynamics with nonideal mixing behaviors was developed by reducing the considerable computation time required for implementing full MWD. A combination of surrogate-model-based optimization and a computational fluid dynamics (CFD)-compartment reactor model was applied to produce the desired MWD in a nonideal reactor. A neural network (NN) surrogate model was trained using 3,000 MWD data produced from the mechanistic reactor model. The results showed that the model successfully described asymmetric MWD with bimodality resulting from the nonideal mixing behaviors in the reactor. The surrogate model was used to construct operation maps for the asymmetry and bimodality factors, with SHapley Additive exPlanations (SHAP) values showing that the flow rate of the chain transfer agent had the largest impact on the asymmetry factor, whereas temperature significantly influenced the bimodality factor. Finally, the optimization strategy based on the asymmetric and bimodality factors successfully determined operating conditions to produce four tailored MWDs, reported in actual polymerization reactors with R scores of 0.988–0.997.

Data assimilation method and application of shear stress transport turbulence model for complex separation of internal shock boundary layer flow

Data assimilation method and application of shear stress transport turbulence model for complex separation of internal shock boundary layer flow

Bayesian data-driven framework for structural health monitoring of composite structures under limited experimental data

Ultrasonic-guided waves can be used to monitor the health of thin-walled structures. However, the run of experimental damage tests on materials like carbon fiber-reinforced plastics can be impractical and costly. Instead, numerical models can be used to create hybrid datasets to train machine learning algorithms, integrating data from numerical and experimental tests. This paper presents a Bayesian-driven framework to compensate for limited experimental data regarding Lamb wave propagation in composite plates. Using Bayesian inference, the framework updates a numerical finite element model, considering observed uncertainties by sampling posterior probability density functions for input parameters using Markov–Chain Monte Carlo simulations with the Metropolis-Hastings algorithm. A neural network surrogate model speeds-up these simulations, leading to a model that replicates the uncertain experimental setup. This model then generates data to augment true experimental data. Finally, a one-dimensional convolutional neural network is trained on a three different datasets to analyze Lamb wave signals and assess damage. Comparing training strategies shows the hybrid dataset augmented by samples generated by the updated FE model gives the most accurate damage size predictions.

Analysis of slope stochastic fields using a novel deep learning model with attention mechanism

This paper proposes a novel deep learning model incorporating attention mechanisms for the analysis of slope stochastic fields. Initially, a deep learning model is designed to digitally image the stochastic field features of soil strength variability. This is achieved by discretizing the slope soil stochastic field using the Karhunen-Loeve expansion method and transforming the discrete results into digital images. These images are then used to establish a Convolutional Neural Network (CNN) surrogate model that maps the implicit relationship between stochastic field images and slope functional function values, thus calculating the probability of slope failure. The precision of the CNN surrogate model is enhanced through Bayesian optimization and five-fold cross-validation. Moreover, to overcome the limitations of existing data-driven landslide stability prediction models, this study also introduces a Spatial-Temporal Attention (STA) mechanism. By combining the CNN with Long Short-Term Memory (LSTM) networks, the model can accurately approximate the actual results of slope stability calculations in scenarios of high-dimensional representation imaging of stochastic fields and low-probability slope instability. Consequently, this significantly improves the computational efficiency of slope reliability analysis considering stochastic field simulations.

A comprehensive deep learning geometric shape optimization framework with field prediction surrogate and reinforcement learning

The optimization of aerodynamic components' geometric shapes demands a novel technical approach for adaptive and efficient exploration and decision-making within the design space. In this study, we introduce an innovative shape optimization framework that leverages deep reinforcement learning with neural network surrogate models. The field prediction surrogate, realized by two distinct U-net architectures, can efficiently generate holistic field solutions based on the transformed mesh coordinates. Subsequently, an inference engine dynamically calculates the key metric of the flow fields, serving as the objective function for the subsequent geometry-aware Deep Q network (DQN)-based optimization. The framework's efficacy is validated using a rocket nozzle as an illustrative example. During surrogate validation, under both friction and frictionless conditions, the l1 errors of the entire flow field of both the U-net vision transformer (ViT) and U-net convolutional neural network (CNN) architectures are less than 0.4%. The proposed U-net ViT consistently outperforms U-net CNN, and the superiority is particularly evident in complex flow areas, outlet sections, and vacuum thrust prediction. Following training, the DQN model is employed to explore the design variable space. The B-spline defining profile successfully is optimized to a final expanding segment shape with improved thrust. Under frictionless conditions, it closely approaches the theoretical optimum. In the practical condition considering friction, the optimized shape gains a 2.96% thrust improvement. The results demonstrate that the proposed framework, especially when coupled with U-net ViT, exhibits enhanced accuracy and adaptability for shape optimization tasks.

The reduced-order model of 5 × 5 fuel rod bundles

The fuel rod bundles are the core part of pressurized water reactors (PWRs), and its heat transfer characteristics directly impact the safety of PWRs. A computational fluid dynamics (CFD) model of 5 × 5 fuel rod bundles with a spacer grid is established, and the numerical simulation results are in excellent agreement with the experimental results. Then, the effects of four turbulence models, namely shear stress transport model, standard k–ε model, re-normalization group k–ε model, and realizable k–ε model on the thermal-hydraulic characteristics of the 5 × 5 fuel rod bundles are systematically investigated. Furthermore, two data-driven methods, namely proper orthogonal decomposition (POD) and dynamic mode decomposition (DMD), are used to analyze the flow fields of the 5 × 5 fuel rod bundles. The two methods can extract key modes or features to enhance the comprehension and description of the dynamic behaviors within the flow fields of 5 × 5 fuel rod bundles. Finally, two reduced-order models (ROMs), called the POD-radial basis function neural network surrogate model and DMD method, are constructed, which can enable rapid prediction of the flow fields for 5 × 5 fuel rod bundles with high accuracy. The CFD simulation results presented in this paper can provide valuable insights for studying the thermal-hydraulic characteristics of the 5 × 5 fuel rod bundles. The two ROMs proposed in this paper can significantly reduce the computational costs associated with studying the thermal-hydraulic characteristics of 5 × 5 fuel rod bundles.

Machine learning-enhanced model-based scenario optimization for DIII-D

Scenario development in tokamaks is an open area of investigation that can be approached in a variety of different ways. Experimental trial and error has been the traditional method, but this required a massive amount of experimental time and resources. As high fidelity predictive models have become available, offline development and testing of proposed scenarios has become an option to reduce the required experimental resources. The use of predictive models also offers the possibility of using a numerical optimization process to find the controllable inputs that most closely achieve the desired plasma state. However, this type of optimization can require as many as hundreds or thousands of predictive simulation cases to converge to a solution; many of the commonly used high fidelity models have high computational burdens, so it is only reasonable to run a handful of predictive simulations. In order to make use of numerical optimization approaches, a compromise needs to be found between model fidelity and computational burden. This compromise can be achieved using neural networks surrogates of high fidelity models that retain nearly the same level of accuracy as the models they are trained to replicate while reducing the computation time by orders of magnitude. In this work, a model-based numerical optimization tool for scenario development is described. The predictive model used by the optimizer includes neural network surrogate models integrated into the fast Control-Oriented Transport simulation framework. This optimization scheme is able to converge to the optimal values of the controllable inputs that produce the target plasma scenario by running thousands of predictive simulations in under an hour without sacrificing too much prediction accuracy.

Multi-objective optimization of CO2 ejector by combined significant variables recognition, ANN surrogate model and multi-objective genetic algorithm

The ejectors are crucial in enhancing the efficiency of the CO2-based refrigeration cycle. Although the ejector can be optimized by experimental and computational fluid dynamic simulation to improve its performance, these methods are time-consuming and complicated. This research aims to present a fast and systematic approach for optimizing CO2 ejector by combining variance analysis, surrogate model, and non-dominated sorting genetic algorithm. Firstly, the key geometric structures affecting the performance of ejector were determined by variance analysis. Secondly, the influence of key geometric structures and boundary conditions on ejector efficiency and entrainment ratio was analyzed by using computational fluid dynamic simulation, and the database was generated. Next, the artificial neural network surrogate model was established to forecast the ejector performance. Finally, the non-dominated sorting genetic algorithm was involved to optimize the ejector for maximizing ejector efficiency and entrainment ratio. The results show that the efficiencies of the optimized ejectors are above 42%, and the entrainment ratios exceed 0.7. Compared with the basic model, the average efficiencies have increased about 13.20%. When the pressure lift is 0.7 MPa, the entrainment ratios of the optimized ejectors are 0.696 (−20 °C ≤ Te < −15 °C), 0.82(-15 °C ≤ Te < −5 °C), 0.60 (−5 °C ≤ Te < 5 °C) and 0.90 (5 °C ≤ Te < 15 °C), respectively. The above research demonstrates that the success of this approach in addressing time-consuming multi-optimization problems. It offers a fast and systematic approach for the multi-objective optimization of CO2 ejector as a valuable reference for engineering applications.

Flow fields prediction for data-driven model of 5 × 5 fuel rod bundles based on POD-RBFNN surrogate model

The fuel rod bundles are a crucial component of nuclear reactors. Research the flow characteristics of the fuel rod bundles can be computationally expensive, despite some progress made in the past few decades. In this paper, the fast prediction of flow fields for 5 × 5 fuel rod bundles is implemented based on a data-driven algorithm. First, a refined model of the 5 × 5 fuel rod bundles with spacer grid is established, and meshes are generated using a hybrid meshes technology. Then, the flow characteristics of the 5 × 5 fuel rod bundles are extensively analyzed using Computational Fluid Dynamics (CFD) methods. A comparison between the CFD results and experimental data reveals a strong agreement, thereby validating our research findings. Subsequently, a reduced-order model (ROM) called the proper orthogonal decomposition (POD)-radial basis function neural network (RBFNN) surrogate model is proposed. the POD algorithm is utilized to extracting the dominant flow modes, and the RBFNN is employed to predict the POD mode coefficients of non-sample points. By linearly combining the predicted POD mode coefficients with the POD modes, rapid prediction of flow fields for non-sample points can be achieved. Compared to the full-scale CFD simulations, it finds that the proposed POD-RBFNN surrogate model not only significantly improved efficiency, but also maintained a high level of predictive accuracy. It is believed that the research results hold important value for CFD calculations within nuclear reactor cores.

Adaptive neural network surrogate model for solving the nonlinear elastic inverse problem via Bayesian inference

Abstract In this paper, we consider a Bayesian method for nonlinear elastic inverse problems. As a working model, we are interested in the inverse problem of restoring elastic properties from measured tissue displacement. In order to reduce the computational cost, we will use the following multi-fidelity model approach. First, we construct a surrogate low-fidelity DNNs-based model in the prior distribution, then use a certain number of simulations of high fidelity model associated with an adaptive strategy online to update the low-fidelity model locally. Numerical examples show that the proposed method can solve nonlinear elastic inverse problems efficiently and accurately.

A numerical compass for experiment design in chemical kinetics and molecular property estimation

Kinetic process models are widely applied in science and engineering, including atmospheric, physiological and technical chemistry, reactor design, or process optimization. These models rely on numerous kinetic parameters such as reaction rate, diffusion or partitioning coefficients. Determining these properties by experiments can be challenging, especially for multiphase systems, and researchers often face the task of intuitively selecting experimental conditions to obtain insightful results. We developed a numerical compass (NC) method that integrates computational models, global optimization, ensemble methods, and machine learning to identify experimental conditions with the greatest potential to constrain model parameters. The approach is based on the quantification of model output variance in an ensemble of solutions that agree with experimental data. The utility of the NC method is demonstrated for the parameters of a multi-layer model describing the heterogeneous ozonolysis of oleic acid aerosols. We show how neural network surrogate models of the multiphase chemical reaction system can be used to accelerate the application of the NC for a comprehensive mapping and analysis of experimental conditions. The NC can also be applied for uncertainty quantification of quantitative structure–activity relationship (QSAR) models. We show that the uncertainty calculated for molecules that are used to extend training data correlates with the reduction of QSAR model error. The code is openly available as the Julia package KineticCompass.Graphical

Optimizing nanoporous metallic actuators through multiscale calculations and machine learning

Nanoporous materials (NMs) immersed in electrolytes can achieve approximately 1 % deformation at a low operating voltage of about 1 V. The actuation renders them promising artificial muscles. The actuation performance significantly hinges on the structure and size of nanopores and ligaments in NMs. Consequently, designing an optimal configuration is imperative for excellent performance. The actuation mechanism of NMs involves the coupling of multiple fields at various length scales, posing a formidable challenge to conventional simulation and design approaches. To surmount this challenge, we have developed a computational framework capable of conducting concurrent and sequential multiscale calculations. By utilizing artificial neural network (ANN) surrogate models trained on data obtained through the finite element method (FEM), the framework achieves optimized values for both actuation strain and effective Young's modulus within a designated design space. The constitutive model, which establishes the relationship between surface stress and charges in FEM, is derived from the surface eigenstress model and symbolic regression. This involves utilizing data calculated through joint density functional theory. This framework not only ensures the desired properties but also demonstrates its potential for effectively addressing other multiscale optimization problems.

Designing Ti-6Al-4V microstructure for strain delocalization using neural networks

The deformation behavior of Ti-6Al-4V titanium alloy is significantly influenced by slip localized within crystallographic slip bands. Experimental observations reveal that intense slip bands in Ti-6Al-4V form at strains well below the macroscopic yield strain and may serially propagate across grain boundaries, resulting in long-range localization that percolates through the microstructure. These connected, localized slip bands serve as potential sites for crack initiation. Although slip localization in Ti-6Al-4V is known to be influenced by various factors, an investigation of optimal microstructures that limit localization remains lacking. In this work, we develop a novel strategy that integrates an explicit slip band crystal plasticity technique, graph networks, and neural network models to identify Ti-6Al-4V microstructures that reduce the propensity for strain localization. Simulations are conducted on a dataset of 3D polycrystals, each represented as a graph to account for grain neighborhood and connectivity. The results are then used to train neural network surrogate models that accurately predict localization-based properties of a polycrystal, given its microstructure. These properties include the ratio of slip accumulated in the band to that in the matrix, fraction of total applied strain accommodated by slip bands, and spatial connectivity of slip bands throughout the microstructure. The initial dataset is enriched by synthetic data generated by the surrogate models, and a grid search optimization is subsequently performed to find optimal microstructures. Describing a 3D polycrystal with only a few features and a combination of graph and neural network models offer robustness compared to the alternative approaches without compromising accuracy. We show that while each material property is optimized through a unique microstructure solution, elongated grain shape emerges as a recurring feature among all optimal microstructures. This finding suggests that designing microstructures with elongated grains could potentially mitigate strain localization without compromising strength.

Techno-economic ionic liquid-based capturing, electrochemical reduction, and hydrogenation of carbon dioxide in the simultaneous production of formic acid and biomethane

CO2 utilization is vital for mitigating climate change by converting CO2 into valuable products, promoting environmental protection and resource efficiency. Novel pathways for CO2 utilization to produce formic acid are proposed namely solute phase electroreduction, gas phase electroreduction, and hydrogenation, are investigated. Employing multi-objective optimization with a deep neural network surrogate model, this study identifies optimal process conditions balancing capital and operational expenditures. The result shows that CO2 hydrogenation ($868 ton−1) exhibits the lowest production cost followed by gas-phase electroreduction ($986 ton−1) and solute-phase electroreduction ($2103 ton−1). The result also shows that without any intervention at all only hydrogenation can generate profit. Furthermore, an in-depth analysis of CO2 emissions indicates that gas phase electroreduction results in the lowest CO2 emissions (0.7 kg CO2 kgHCOOH−1) among the examined pathways. Insights from our research suggest a minimum current density of 417.3 mA cm−2 is recommended to achieve at least parity with hydrogenation in terms of production cost. To push the commercialization of gas-phase electroreduction, besides current density improvement, electricity cost reduction, and carbon trading mechanism is proposed to reduce the production cost.

Physics-Informed Neural Network surrogate model for bypassing Blade Element Momentum theory in wind turbine aerodynamic load estimation

This paper proposes the use of Artificial Neural Networks (ANNs), specifically Physics-Informed Neural Networks (PINNs), for dynamic surrogate modelling of wind turbines. PINNs offer the flexibility to model complex relationships while incorporating physics-based constraints, enabling accurate representation of wind turbine dynamics. In this paper, a PINN-based surrogate model is developed for the Blade Element Momentum (BEM) aerodynamic model used in state-of-the-art numerical wind turbine simulations. The PINN model replaces the time-consuming root-finding process in BEM with high-dimensional regression, significantly improving computational efficiency. The PINN model is trained using data generated from a numerical model of the IEA-15MW reference wind turbine, and its performance is compared against conventional data-driven Neural Network (DDNN) models. The proposed surrogate model provides more efficient and accurate evaluations of wind turbine responses compared with traditional surrogate modelling approaches. A significant computational advantage is obtained by using the developed surrogate models with a forty-fold speedup demonstrated compared to the BEM model. Replacing the BEM model with the PINN-based surrogate model for load computation in the numerical model used for dynamic analysis results in an overall reduction of 35% in computational time for a complete dynamic simulation. This is a substantial improvement in efficiency without sacrificing accuracy — the maximum Mean Absolute Error (MAE) values for the surrogate models are of the order of 10−2, which shows that the surrogate models can predict the angle of attack at any blade node with a discrepancy of less than 0.5°. The surrogate models significantly reduce computational time while maintaining high accuracy, making them a promising approach for simulating wind turbine dynamics, especially in fields such as reliability analysis or fatigue estimation where many simulations are necessary.