Genetic Programming-based Symbolic Regression Research Articles

A quantitative investigation on pyrolysis behaviors of metal ion-exchanged coal macerals by interpretable machine learning algorithms

Generalizing the rules from complex processes such as catalytic pyrolysis to guide their process control is always a difficult but attractive task. The influences of ion-exchange of metal ions (Na+, K+, Ca2+, Mg2+, Co2+ and Ni2+) on the pyrolysis behavior of vitrinite and inertinite from Shendong coal were investigated by thermogravimetric analyzer-Fourier transform infrared spectrometer (TG-FTIR), fixed-bed reactor (FBR), gas chromatograph-mass spectrometer (GC-MS) and X-ray diffractometer (XRD). A set of machine learning models was successfully constructed based on random forest, support vector machine, and Gaussian process regression, to quantify the relationships between pyrolysis behaviors and the properties of metals and macerals. The leave-one-out-cross validation showed that there are considerable determination coefficients (R2 > 0.9) between predicted and experimental values for most responses (17 out of 29). By combining genetic programming-based symbolic regression with the black-box algorithm, 23 symbolic regression expressions with high confidence were successfully constructed. This work is a pioneering attempt of optimization in small-scale experiments. By utilizing the highly interpretable models, a demand-orientated (multi-)optimization coal pyrolysis can be achieved. The bi-objective optimization was conducted on the yield of tar and the content of light aromatics in tar, and the results show that Co is the optimal loading metal.

Interaction between microcapsules and crack using XFEM and data-driven fracture toughness predictive model

The interaction mechanism between microstructures of materials, including microcrack, inclusion and interface defect, is crucial for studying the microscopic damage of materials. The interaction between microcapsules and cracks in self-healing material is examined using a combination of the extended finite element method (XFEM) and the interaction integral method, which is implemented by a program written in Fortran language. The non-dimensional stress intensity factors (SIFs) of a crack with a single microcapsule, a pair of microcapsules and a cluster of microcapsules are investigated. Moreover, the data-driven fracture toughness predictive model using a genetic programming-based symbolic regression (SR) technique is proposed by training the data from the XFEM program. Results reveal that the non-dimensional SIFs are related to the mechanical property, geometry and interface defect of microcapsule. The influence of microcapsule on the SIF is limited within a distance of five times the microcapsule radius. The predictive model quantifies the contribution of microscopic structures on the macroscopic fracture parameter, highlighting the potential of machine learning in guiding the quantitative design and discovery of high-performance smart materials.

Prediction model of asphalt emulsion evaporation rate based on CFD simulation and genetic programming-based symbolic regression

The evaporation rate of asphalt emulsion is critical to the strength development of asphalt emulsion materials. Its value is dependent on environmental factors, i.e. relative humidity, temperature, and wind speed. However, the quantitative relation of evaporation rate with these environmental factors are unclear. To quantificationally study the effect of environmental factors on the water evaporation rate of asphalt emulsion, a computational fluid dynamics (CFD) simulation model of water evaporation is firstly proposed and validated by experiment. Then, wide range of environmental factors are studied in the CFD simulation model, and genetic programming-based symbolic regression is used to train the simulation results of CFD samples. Finally, a relatively accurate prediction equation of the evaporation rate of asphalt emulsion versus environmental factor is proposed, and the sensitivity of the water evaporation rate under the three environmental factors is analyzed. Results show that the evaporation rate can be accurately simulated by CFD. All the three environmental factors can greatly affect the evaporation rate of asphalt emulsion, while their effect on the sensitivity of the water evaporation rate is ranked as: temperature > relative humidity > wind speed. To ensure an acceptable curing period, asphalt emulsion-based materials should be avoided to be paved in the conditions of low temperature and high relative humidity.

SR-Forest: A Genetic Programming based Heterogeneous Ensemble Learning Method

Ensemble learning methods have been widely used in machine learning in recent years due to their high predictive performance. With the development of genetic programming-based symbolic regression methods, many papers begin to choose a popular ensemble learning method, random forests, as the baseline competitor. Instead of considering them as competitors, an alternative idea might be to consider symbolic regression as an enhancement technique for random forest. Genetic programming-based symbolic regression methods which fit a smooth function are complementary to the piecewise nature of decision trees, as the smooth variation is common in regression problems. In this article, we propose to form an ensemble model with symbolic regression-based decision trees to address this issue. Furthermore, we design a guided mutation operator to speed up the search on high-dimensional problems, a multi-fidelity evaluation strategy to reduce the computational cost and an ensemble selection mechanism to improve predictive performance. Finally, experimental results on a regression benchmark with 120 datasets show that the proposed ensemble model outperforms 25 existing symbolic regression and ensemble learning methods. Moreover, the proposed method can provide notable insights on an XGBoost hyperparameter performance prediction task, which is an important application area of ensemble learning methods.

Complementing a continuum thermodynamic approach to constitutive modeling with symbolic regression

An interpretable machine learning method, physics-informed genetic programming-based symbolic regression (P-GPSR), is integrated into a continuum thermodynamic approach to developing constitutive models. The proposed strategy for combining a thermodynamic analysis with P-GPSR is demonstrated by generating a yield function for an idealized material with voids, i.e., the Gurson yield function. First, a thermodynamic-based analysis is used to derive model requirements that are exploited in a custom P-GPSR implementation as fitness criteria or are strongly enforced in the solution. The P-GPSR implementation improved accuracy, generalizability, and training time compared to the same GPSR code without physics-informed fitness criteria. The yield function generated through the P-GPSR framework is in the form of a composite function that describes a class of materials and is characteristically more interpretable than GPSR-derived equations. The physical significance of the input functions learned by P-GPSR within the composite function is acquired from the thermodynamic analysis. Fundamental explanations of why the implemented P-GPSR capabilities improve results over a conventional GPSR algorithm are provided.

Generalizing the Gurson model using symbolic regression and transfer learning to relax inherent assumptions

To generate material models with fewer limiting assumptions while maintaining closed-form, interpretable solutions, we propose using genetic programming based symbolic regression (GPSR), a machine learning (ML) approach that describes data using free-form symbolic expressions. To maximize interpretability, we start from an analytical, derived material model, the Gurson model for porous ductile metals, and systematically relax inherent assumptions made in its derivation to understand each assumption’s contribution to the GPSR model forms. We incorporate transfer learning methods into the GPSR training process to increase GPSR efficiency and generate models that abide by known mechanics of the system. The results show that regularizing the GPSR fitness function is critical for generating physically valid models and illustrate how GPSR allows a high level of interpretability compared with other ML approaches. The method of systematic assumption relaxation allows the generation of models that address limiting assumptions found in the Gurson model, and the symbolic forms allow conjecture of decreased material strength due to void interaction and non-symmetric void shapes.

Parametrically upscaled crack nucleation model (PUCNM) for fatigue nucleation in titanium alloys containing micro-texture regions (MTR)

Micro-texture regions (MTRs), delineated as the clusters of grains with similar crystallographic orientations in the polycrystalline microstructure, play a significant role in fatigue crack nucleation and life of structures of Ti alloys. This paper develops a parametrically upscaled constitutive and crack nucleation modeling (PUCM/PUCNM) platform for predicting structural-scale fatigue crack nucleation in α/β Ti–6Al–4V alloys, whose polycrystalline microstructures contain MTRs. The PUCM/PUCNM platform bridges micro and macro scales through thermodynamically consistent incorporation of representative aggregated microstructural parameters (RAMPs) in macroscopic constitutive relations. A novel RAMP kMTRθc that captures both the MTR size and contrast in the overall texture is proposed to quantitatively represent the MTR intensity in the microstructure. Geometric analysis and comparison with experimental data establish the effectiveness of kMTRθc in characterizing the MTR distributions. The impact of MTR characteristics on fatigue crack nucleation is evaluated through the support vector regression (SVR)-aided Sobol analysis, based on data from crystal plasticity FE simulations of polycrystalline microstructure volumes with a grain-scale crack nucleation model. The PUCNM is uniquely suitable for the incorporation of the MTR RAMP kMTRθc in its probabilistic framework. A novel functional form is derived using the genetic programming-based symbolic regression (GPSR). The PUCM/PUCNM tool is used to simulate an engine blade under dwell loading conditions. Results exhibit the reduction of nucleation life with a higher level of MTR intensity, despite the same overall textures.

Crack tip enhanced phase-field model for crack evolution in crystalline Ti6Al from concurrent crystal plasticity FE-molecular dynamics simulations

Dislocations nucleating from a crack-tip contribute to the plasticity evolution in the vicinity of crack, as well as to the crack propagation process. This paper systematically develops a method for enhancing the defect energy, accounting for crack-tip nucleated dislocations in the Helmholtz free energy density functional, associated with the phase-field formulation of crack evolution, in a coupled crystal plasticity finite element phase field (CPFE-PF) model. A concurrent crystal plasticity FE- hyper-dynamics accelerated molecular dynamics (CPFE-MD) model is created. The model that identifies, transfers, and propagates dislocations from the atomistic to continuum domain, is used to generate a dataset of crack tip dislocation density evolution along with different state variables. The paper focuses on crack tip plasticity mechanisms for the crystalline alloy Ti6Al. Using a Bayesian inference approach the critical state variables that affect the evolution of crack tip nucleated dislocation density are inferred. A functional form of the evolution of dislocation density in terms of the state variables is derived by employing a genetic programming based symbolic regression (GPSR) approach. The contribution of nucleated dislocation densities to effective plastic strain evolution at the crack tip is validated using the CPFE-MD model simulations. A comparison of the crack path and other state variables in the vicinity of the crack with and without contributions from the nucleated dislocations shows the importance of this augmentation on crack evolution.

Genetic programming-based backbone curve model of reinforced concrete walls

Backbone curve, as a nonlinear response analysis method, can be used for performance assessment of residual resistance and performance prediction during the preliminary design of structures. In this study, a backbone curve model of reinforced concrete (RC) walls based on Genetic programming-based symbolic regression (GP-SR) was proposed, which can help to quickly evaluate the bearing capacity and seismic performance of RC walls. Unlike the black-box characteristic of traditional machine learning models, the GP-SR method can give explicit computational equations, which are more interpretable and easier to be used by researchers and engineers. Experimental data of 388 existing RC walls were used for feature selection, model training, and comparison with the modeling method of ASCE 41-17 to verify its effectiveness for modeling the backbone curves of RC walls with four failure modes (i.e., flexure, flexure-shear, shear, and shear-sliding). The results showed that the accuracy of the GP-SR model was better than that of the prediction of ASCE 41-17. Overall, the GP-SR model described well the backbone curves of RC walls with various design conditions.

Forecasting the Crude Oil Spot Price with Bayesian Symbolic Regression

In this study, the crude oil spot price is forecast using Bayesian symbolic regression (BSR). In particular, the initial parameters specification of BSR is analysed. Contrary to the conventional approach to symbolic regression, which is based on genetic programming methods, BSR applies Bayesian algorithms to evolve the set of expressions (functions). This econometric method is able to deal with variable uncertainty (feature selection) issues in oil price forecasting. Secondly, this research seems to be the first application of BSR to oil price forecasting. Monthly data between January 1986 and April 2021 are analysed. As well as BSR, several other methods (also able to deal with variable uncertainty) are used as benchmark models, such as LASSO and ridge regressions, dynamic model averaging, and Bayesian model averaging. The more common ARIMA and naïve methods are also used, together with several time-varying parameter regressions. As a result, this research not only presents a novel and original application of the BSR method but also provides a concise and uniform comparison of the application of several popular forecasting methods for the crude oil spot price. Robustness checks are also performed to strengthen the obtained conclusions. It is found that the suitable selection of functions and operators for BSR initialization is an important, but not trivial, task. Unfortunately, BSR does not result in forecasts that are statistically significantly more accurate than the benchmark models. However, BSR is computationally faster than the genetic programming-based symbolic regression.

Automated learning of interpretable models with quantified uncertainty

Interpretability and uncertainty quantification in machine learning can provide justification for decisions, promote scientific discovery and lead to a better understanding of model behavior. Symbolic regression provides inherently interpretable machine learning, but relatively little work has focused on the use of symbolic regression on noisy data and the accompanying necessity to quantify uncertainty. A new Bayesian framework for genetic-programming-based symbolic regression (GPSR) is introduced that uses model evidence (i.e., marginal likelihood) to formulate replacement probability during the selection phase of evolution. Model parameter uncertainty is automatically quantified, enabling probabilistic predictions with each equation produced by the GPSR algorithm. Model evidence is also quantified in this process, and its use is shown to increase interpretability, improve robustness to noise, and reduce overfitting when compared to a conventional GPSR implementation on both numerical and physical experiments.

Symbolic regression-based allometric model development of a mangrove forest LAI using structural variables and digital hemispherical photography

The leaf area index (LAI) serves as a proxy to understand the dynamics of plant productivity, energy balance, and gas exchange. Cost-effective and accurate estimation of LAI is essential for under-assessed carbon-rich tropical forests, e.g., mangroves. Here, we developed allometric equations to estimate LAI using a combination of non-destructive, optical measurements through digital hemispherical photographs (DHP), and genetic programming-based Symbolic Regression (SR). We used three structural variables: diameter at breast height (DBH), tree density (TD), and canopy height (Ht) for a mangrove forest in the BhitarKanika Wildlife Sanctuary (BWS), located along the Eastern coast of India. Triplet combination using SR provided the best equation (R2 = 0.51) than any singlet or duplet combination of the variables, and even it was better than Partial Least Square (PLS) based regression (R2 = 0.42). To the best of our knowledge, the current study is the maiden attempt to develop an allometric model to estimate LAI for a mangrove ecosystem in India. In-situ measurements of structural variables such as DBH, Ht, and TD can be used for LAI estimates, as shown here. LAI estimates using cost-effective methods would greatly enhance our understanding of the spatial and temporal dynamics of mangrove ecosystems.

Genetic programming-based symbolic regression for goal-oriented dimension reduction

The majority of dimension reduction techniques are built upon the optimization of an objective function aiming to retain certain characteristics of the projected datapoints: the variance of the original dataset, the distance between the datapoints or their neighbourhood characteristics, etc. Building upon the optimization-based formalization of dimension reduction techniques, the goal-oriented formulation of projection cost functions is proposed. For the optimization of the application-oriented data visualization cost function, a Multi-gene genetic programming (GP)-based algorithm is introduced to optimize the structures of the equations used for mapping high-dimensional data into a two-dimensional space and to select variables that are needed to explore the internal structure of the data for data-driven software sensor development or classifier design. The main benefit of the approach is that the evolved equations are interpretable and can be utilized in surrogate models. The applicability of the approach is demonstrated in the benchmark wine dataset and in the estimation of the product quality in a diesel oil blending technology based on an online near-infrared (NIR) analyzer. The results illustrate that the algorithm is capable to generate goal-oriented and interpretable features, and the resultant simple algebraic equations can be directly implemented into applications when there is a need for computationally cost-effective projections of high-dimensional data as the resultant algebraic equations are computationally simpler than other solutions as neural networks.

A Hybrid Genetic Programming–Gray Wolf Optimizer Approach for Process Optimization of Biodiesel Production

Biodiesel is one the most sought after alternate fuels in the current global need for sustainable and renewable energy sources due to their lower emissions and no major modification requirement to existing engines. However, the performance and productivity of the biodiesel production process are significantly dependent on the process parameters. In this regard, a novel hybrid genetic programming-gray wolf optimizer approach for the process optimization of biodiesel production is proposed in this paper. For an illustration of the proposed approach, kinematic viscosity is expressed as a symbolic regression metamodel to account for the influence of catalyst concentration, reaction temperature, alcohol-to-oil molar ratio, and reaction time. Then, the genetic programming-based symbolic regression metamodel is used as an objective function by the gray wolf optimizer to optimize the process parameters. The obtained results show that the proposed approach is simple, accurate, and robust.

Genetic programming based symbolic regression for shear capacity prediction of SFRC beams

The complexity of shear transfer mechanisms in steel fiber-reinforced concrete (SFRC) has motivated researchers to develop diverse empirical and soft-computing models for predicting the shear capacity of SFRC beams. Yet, such existing methods have been developed based on limited experimental databases, which makes their generalization capability uncertain. To account for the limited experimental data available, this study pioneers a novel approach based on tabular generative adversarial networks (TGAN) to generate 2000 synthetic data examples. A “train on synthetic - test on real” philosophy was adopted. Accordingly, the entire 2000 synthetic data were used for training a genetic programming-based symbolic regression (GP-SR) model to develop a shear strength equation for SFRC beams without stirrups. The model accuracy was then tested on the entire set of 309 real experimental data examples, which thus far are unknown to the model. Results show that the novel GP-SR model achieved superior predictive accuracy, outperforming eleven existing equations. Sensitivity analysis revealed that the shear-span-to-depth ratio was the most influential parameter in the proposed equation. The present study provides an enhanced predictive model for the shear capacity of SFRC beams, which should motivate further research to effectively train evolutionary algorithms using synthetic data when acquiring large and comprehensive experimental datasets is not feasible.

Lattice Thermal Conductivity Prediction Using Symbolic Regression and Machine Learning.

Prediction models of lattice thermal conductivity (κL) have wide applications in the discovery of thermoelectrics, thermal barrier coatings, and thermal management of semiconductors. However, κL is notoriously difficult to predict. Although classic models such as the Debye-Callaway model and the Slack model have been used to approximate the κL of inorganic compounds, their accuracy is far from being satisfactory. Herein we propose a genetic programming-based symbolic regression (SR) approach for finding analytical κL models and compare them with multilayer perceptron neural networks and random forest regression models using a hybrid cross-validation (CV) approach including both K-fold CV and holdout validation. Four formulae have been discovered by our SR approach that outperform the Slack formula as evaluated on our dataset. Through the analysis of our models' performance and the formulae generated, we found that the trained formulae successfully reproduce the correct physical law that governs the lattice thermal conductivity of materials. We also systematically show that currently extrapolative prediction over datasets with different distributions as the training set remains to be a big challenge for both SR and machine learning-based prediction models.

Genetic Programming for Instance Transfer Learning in Symbolic Regression.

Transfer learning has attracted more attention in the machine-learning community recently. It aims to improve the learning performance on the domain of interest with the help of the knowledge acquired from a similar domain(s). However, there is only a limited number of research on tackling transfer learning in genetic programming for symbolic regression. This article attempts to fill this gap by proposing a new instance weighting framework for transfer learning in genetic programming-based symbolic regression. In the new framework, differential evolution is employed to search for optimal weights for source-domain instances, which helps genetic programming to identify more useful source-domain instances and learn from them. Meanwhile, a density estimation method is used to provide good starting points to help the search for the optimal weights while discarding some irrelevant or less important source-domain instances before learning regression models. The experimental results show that compared with genetic programming and support vector regression that learn only from the target instances, and learning from a mixture of instances from the source and target domains without any transfer learning component, the proposed method can evolve regression models which not only achieve notably better cross-domain generalization performance in stability but also reduce the trend of overfitting effectively. Meanwhile, these models are generally much simpler than those generated by the other GP methods.

Symbolic regression in materials science

Abstract

Symbolic regression in dynamic scenarios with gradually changing targets

Symbolic regression is a machine learning task: given a training dataset with features and targets, find a symbolic function that best predicts the target given the features. This paper concentrates on dynamic regression tasks, i.e. tasks where the goal changes during the model fitting process. Our study is motivated by dynamic regression tasks originating in the domain of reinforcement learning: we study four dynamic symbolic regression problems related to well-known reinforcement learning benchmarks, with data generated from the standard Value Iteration algorithm. We first show that in these problems the target function changes gradually, with no abrupt changes. Even these gradual changes, however, are a challenge to traditional Genetic Programming-based Symbolic Regression algorithms because they rely only on expression manipulation and selection. To address this challenge, we present an enhancement to such algorithms suitable for dynamic scenarios with gradual changes, namely the recently introduced type of leaf nodes called Linear Combination of Features. This type of leaf node, aided by the error backpropagation technique known from artificial neural networks, enables the algorithm to better fit the data by utilizing the error gradient to its advantage rather than searching blindly using only the fitness values. This setup is compared with a baseline of the core algorithm without any of our improvements and also with a classic evolutionary dynamic optimization technique: hypermutation. The results show that the proposed modifications greatly improve the algorithm ability to track a gradually changing target.

Intelligent Computing Based Formulas to Predict the Settlement of Shallow Foundations on Cohesionless Soils

Introduction:Although it is a regular duty of geotechnical engineers to evaluate how much shallow foundation settles in the granular soil, there is no well-approved formula for this task. The intent of this research is to develop a formula that is adequately simple to be used in routine geotechnical engineering work but complete enough to address the behavior of granular soil associated with the settlement issue.Methods:Cone penetration test and foundation load test data were used to generate a formula that can predict the settlement. Genetic Programming (GP) based Symbolic Regression (GP-SR) and artificial neural networks were used to develop an optimized formula. Settlements were also calculated using the finite method and compared to the results of the developed formula.Results and Conclusion:Two formulas were developed using SR, and several models were developed using ANN. ANN model 1 has the highest R2value (0.93) and the lowest MSE (0.16) among all developed ANN and GP-SR models. FEM settlements were almost double the measured ones in some instances.