Symbolic Regression Models Research Articles

Choosing function sets with better generalisation performance for symbolic regression models

Supervised learning by means of Genetic Programming (GP) aims at the evolutionary synthesis of a model that achieves a balance between approximating the target function on the training data and generalising on new data. The model space searched by the Evolutionary Algorithm is populated by compositions of primitive functions defined in a function set. Since the target function is unknown, the choice of function set’s constituent elements is primarily guided by the makeup of function sets traditionally used in the GP literature. Our work builds upon previous research of the effects of protected arithmetic operators (i.e. division, logarithm, power) on the output value of an evolved model for input data points not encountered during training. The scope is to benchmark the approximation/generalisation of models evolved using different function set choices across a range of 43 symbolic regression problems. The salient outcomes are as follows. Firstly, Koza’s protected operators of division and exponentiation have a detrimental effect on generalisation, and should therefore be avoided. This result is invariant of the use of moderately sized validation sets for model selection. Secondly, the performance of the recently introduced analytic quotient operator is comparable to that of the sinusoidal operator on average, with their combination being advantageous to both approximation and generalisation. These findings are consistent across two different system implementations, those of standard expression-tree GP and linear Grammatical Evolution. We highlight that this study employed very large test sets, which create confidence when benchmarking the effect of different combinations of primitive functions on model generalisation. Our aim is to encourage GP researchers and practitioners to use similar stringent means of assessing generalisation of evolved models where possible, and also to avoid certain primitive functions that are known to be inappropriate.

Canopy Height Estimation Using Sentinel Series Images through Machine Learning Models in a Mangrove Forest

Canopy height serves as a good indicator of forest carbon content. Remote sensing-based direct estimations of canopy height are usually based on Light Detection and Ranging (LiDAR) or Synthetic Aperture Radar (SAR) interferometric data. LiDAR data is scarcely available for the Indian tropics, while Interferometric SAR data from commercial satellites are costly. High temporal decorrelation makes freely available Sentinel-1 interferometric data mostly unsuitable for tropical forests. Alternatively, other remote sensing and biophysical parameters have shown good correlation with forest canopy height. The study objective was to establish and validate a methodology by which forest canopy height can be estimated from SAR and optical remote sensing data using machine learning models i.e., Random Forest (RF) and Symbolic Regression (SR). Here, we analysed the potential of Sentinel-1 interferometric coherence and Sentinel-2 biophysical parameters to propose a new method for estimating canopy height in the study site of the Bhitarkanika wildlife sanctuary, which has mangrove forests. The results showed that interferometric coherence, and biophysical variables (Leaf Area Index (LAI) and Fraction of Vegetation Cover (FVC)) have reasonable correlation with canopy height. The RF model showed a Root Mean Squared Error (RMSE) of 1.57 m and R2 value of 0.60 between observed and predicted canopy heights; whereas, the SR model through genetic programming demonstrated better RMSE and R2 values of 1.48 and 0.62 m, respectively. The SR also established an interpretable model, which is not possible via any other machine learning algorithms. The FVC was found to be an essential variable for predicting forest canopy height. The canopy height maps correlated with ICESat-2 estimated canopy height, albeit modestly. The study demonstrated the effectiveness of Sentinel series data and the machine learning models in predicting canopy height. Therefore, in the absence of commercial and rare data sources, the methodology demonstrated here offers a plausible alternative for forest canopy height estimation.

Comparative study of measurement and verification (M&V) baseline models for quantifying energy savings in building renovations

Measurement and verification (M&V) is the process of quantifying energy savings originated by one or several energy conservation measures (ECM) in an existing building. The estimation of the savings consist of comparing actual energy consumption to the adjusted baseline model. This paper focuses on comparing three approaches for creating baseline models: linear, symbolic regression (SR) and extreme gradient boost (XGBoost); and discusses the advantages and drawbacks on each of them from a practitioner’s perspective. In this paper, these approaches are assessed qualitatively and quantitatively. The qualitative assessment compares the type of model output, interpretability and calibration time. Linear model excels in all three criteria whereas the XGBoost is the worst option for model output and interpretability. SR model is the worst performing in terms of calibration time, but intermediate model for output and interpretability. Quantitative assessment is done through the quantification of prediction errors in 367 buildings after being calibrated with hourly data for a 12-month period. The XGBoost model has the highest prediction accuracy in terms of CVRMSE. The linear model performs significantly well in terms of NMBE. SR preforms well in terms of CVRMSE and has the best median NMBE overall remaining as the most accurate interpretable option. The results show different benefits and drawbacks of each approach and the implementation of SR model for this application is the main innovation of this paper.

Parameter identification for symbolic regression using nonlinear least squares

In this paper we analyze the effects of using nonlinear least squares for parameter identification of symbolic regression models and integrate it as local search mechanism in tree-based genetic programming. We employ the Levenberg–Marquardt algorithm for parameter optimization and calculate gradients via automatic differentiation. We provide examples where the parameter identification succeeds and fails and highlight its computational overhead. Using an extensive suite of symbolic regression benchmark problems we demonstrate the increased performance when incorporating nonlinear least squares within genetic programming. Our results are compared with recently published results obtained by several genetic programming variants and state of the art machine learning algorithms. Genetic programming with nonlinear least squares performs among the best on the defined benchmark suite and the local search can be easily integrated in different genetic programming algorithms as long as only differentiable functions are used within the models.

Symbolic regression models for predicting viscous dissipation of three-dimensional non-Newtonian flows in single-screw extruders

Modeling the flow of non-Newtonian melts in single-screw extruders generally requires numerical methods. In this study, we analyze viscous dissipation for fully developed flow of non-Newtonian fluids within three-dimensional metering channels of single-screw extruders. Applying the theory of similarity we identified a set of four independent dimensionless influencing parameters that fully describe the underlying physics of the considered problem: the channel height-to-width ratio h/w, the screw-pitch ratio t/Db, the power-law exponent n, and the dimensionless down-channel pressure gradient Πp, z. Based on a comprehensive numerical parametric design study we developed two generalized symbolic regression models predicting dimensionless viscous dissipation – one for given pressure gradient, and the other one for given throughput. As a result of the three-dimensional modeling approach our regression models take the effects of flight flanks, transverse flow, and non-Newtonian fluid behavior into account. The accuracy of these developed models was proven by means of an error analysis using an independent data set. These models allow fast, stable and accurate predictions of viscous dissipation, thus contributing to the optimal design of extruder screws without the need for numerical simulations. Furthermore they are applicable to injection molding machines and for predicting the required drive power.

Data Based Prediction of Blood Glucose Concentrations Using Evolutionary Methods.

Predicting glucose values on the basis of insulin and food intakes is a difficult task that people with diabetes need to do daily. This is necessary as it is important to maintain glucose levels at appropriate values to avoid not only short-term, but also long-term complications of the illness. Artificial intelligence in general and machine learning techniques in particular have already lead to promising results in modeling and predicting glucose concentrations. In this work, several machine learning techniques are used for the modeling and prediction of glucose concentrations using as inputs the values measured by a continuous monitoring glucose system as well as also previous and estimated future carbohydrate intakes and insulin injections. In particular, we use the following four techniques: genetic programming, random forests, k-nearest neighbors, and grammatical evolution. We propose two new enhanced modeling algorithms for glucose prediction, namely (i) a variant of grammatical evolution which uses an optimized grammar, and (ii) a variant of tree-based genetic programming which uses a three-compartment model for carbohydrate and insulin dynamics. The predictors were trained and tested using data of ten patients from a public hospital in Spain. We analyze our experimental results using the Clarke error grid metric and see that 90% of the forecasts are correct (i.e., Clarke error categories A and B), but still even the best methods produce 5 to 10% of serious errors (category D) and approximately 0.5% of very serious errors (category E). We also propose an enhanced genetic programming algorithm that incorporates a three-compartment model into symbolic regression models to create smoothed time series of the original carbohydrate and insulin time series.

Unveiling the properties of structured grammatical evolution

Structured grammatical evolution (SGE) is a new genotypic representation for grammatical evolution (GE). It comprises a hierarchical organization of the genes, where each locus is explicitly linked to a non-terminal of the grammar being used. This one-to-one correspondence ensures that the modification of a gene does not affect the derivation options of other non-terminals. We present a comprehensive set of optimization results obtained with problems from three different categories: symbolic regression, path finding, and predictive modeling. In most of the situations SGE outperforms standard GE, confirming the effectiveness of the new representation. To understand the reasons for SGE enhanced performance, we scrutinize its main features. We rely on a set of static measures to model the interactions between the representation and variation operators and assess how they influence the interplay between the genotype-phenotype spaces. The study reveals that the structured organization of SGE promotes an increased locality and is less redundant than standard GE, thus fostering an effective exploration of the search space.

Prediction of self-compacting concrete elastic modulus using two symbolic regression techniques

Abstract This paper introduces a novel symbolic regression approach, namely biogeographical-based programming (BBP), for the prediction of elastic modulus of self-compacting concrete (SCC). The BBP model was constructed directly from a comprehensive dataset of experimental results of SCC available in the literature. For comparison purposes, another new symbolic regression model, namely artificial bee colony programming (ABCP), was also developed. Furthermore, several available formulas for predicting the elastic modulus of SCC were assessed using the collected database. The results show that the proposed BBP model provides slightly closer results to experiments than ABCP model and existing available formulas. A sensitivity analysis of BBP parameters also shows that the prediction by BBP model improves with the increase of habitat size, colony size, and maximum tree depth. In addition, among all considered empirical and design code equations, Leemann and Hoffmann and ACI 318-08's equations exhibit a reasonable performance but Persson and Felekoglu et al.'s equations are highly inaccurate for the prediction of SCC elastic modulus.

Prognostics and Health Management of an Automated Machining Process

Machine failure modes are presenting a major burden to the operator, the plant, and the enterprise causing significant downtime, labor cost, and reduced revenue. New technologies are emerging over the past years to monitor the machine’s performance, detect and isolate incipient failures or faults, and take appropriate actions to mitigate such detrimental events. This paper addresses the development and application of novel Prognostics and Health Management (PHM) technologies to a prototype machining process (a screw-tightening machine). The enabling technologies are built upon a series of tasks starting with failure analysis, testing, and data processing aimed to extract useful features or condition indicators from raw data, a symbolic regression modeling framework, and a Bayesian estimation method called particle filtering to predict the feature state estimate accurately. The detection scheme declares the fault of a machine critical component with user specified accuracy or confidence and given false alarm rate while the prediction algorithm estimates accurately the remaining useful life of the failing component. Simulation results support the efficacy of the approach and match well the experimental data.

A symbolic regression-based modelling strategy of AC/DC rectifiers for RFID applications

In this paper, we present the use of symbolic regression for the black-box modelling of non-linear dynamic behaviour in the AC/DC rectifiers of radio-frequency identification (RFID) circuits. Typical simulations of an AC/DC rectifier are exceptionally slow when performed using conventional transistor-level simulations. The proposed modelling strategy is capable of modelling the transient process, i.e., the change in the output voltage as a function of time, both accurately and efficiently. The accuracy and the simulation speed are compared to the transistor-level simulations. The results show that the symbolic regression models are accurate and several orders of magnitude faster than the transistor-level models. The proposed strategy is demonstrated for rectifiers designed using 130-nm, 180-nm and 0.35-μm CMOS process. The simulation speed-up factor of the proposed model is shown to be between 1080 and 3.84E6 times for a root-mean-square error (RMSE) below 4.8mV.

Interpretable semi-parametric regression models with defined error bounds

Unreliable extrapolation of data-driven models hinders their applicability not only in safety-related domains. The paper discusses how model interpretability and uncertainty estimates can address this problem. A new semi-parametric approach is proposed for providing an interpretable model with improved accuracy by combining a symbolic regression model with a residual Gaussian Process. While the learned symbolic model is highly interpretable the residual model usually is not. However, by limiting the output of the residual model to a defined range a worst-case guarantee can be given in the sense that the maximal deviation from the symbolic model is always below a defined limit. The limitation of the residual model can include the uncertainty estimate of the Gaussian Process, thus giving the residual model more impact in high-confidence regions. When ranking the accuracy and interpretability of several different approaches on the SARCOS data benchmark the proposed combination yields the best result.

Physiological Complexity of Acute Traumatic Brain Injury in Patients Treated with a Brain Oxygen Protocol: Utility of Symbolic Regression in Predictive Modeling of a Dynamical System

Predictive modeling of emergent behavior, inherent to complex physiological systems, requires the analysis of large complex clinical data streams currently being generated in the intensive care unit. Brain tissue oxygen protocols have yielded outcome benefits in traumatic brain injury (TBI), but the critical physiological thresholds for low brain oxygen have not been established for a dynamical patho-physiological system. High frequency, multi-modal clinical data sets from 29 patients with severe TBI who underwent multi-modality neuro-clinical care monitoring and treatment with a brain oxygen protocol were analyzed. The inter-relationship between acute physiological parameters was determined using symbolic regression (SR) as the computational framework. The mean patient age was 44.4±15 with a mean admission GCS of 6.6±3.9. Sixty-three percent sustained motor vehicle accidents and the most common pathology was intra-cerebral hemorrhage (50%). Hospital discharge mortality was 21%, poor outcome occurred in 24% of patients, and good outcome occurred in 56% of patients. Criticality for low brain oxygen was intracranial pressure (ICP) ≥22.8 mm Hg, for mortality at ICP≥37.1 mm Hg. The upper therapeutic threshold for cerebral perfusion pressure (CPP) was 75 mm Hg. Eubaric hyperoxia significantly impacted partial pressure of oxygen in brain tissue (PbtO2) at all ICP levels. Optimal brain temperature (Tbr) was 34-35°C, with an adverse effect when Tbr≥38°C. Survivors clustered at [Formula: see text] Hg vs. non-survivors [Formula: see text] 18 mm Hg. There were two mortality clusters for ICP: High ICP/low PbtO2 and low ICP/low PbtO2. Survivors maintained PbtO2 at all ranges of mean arterial pressure in contrast to non-survivors. The final SR equation for cerebral oxygenation is: [Formula: see text]. The SR-model of acute TBI advances new physiological thresholds or boundary conditions for acute TBI management: PbtO2≥25 mmHg; ICP≤22 mmHg; CPP≈60-75 mmHg; and Tbr≈34-37°C. SR is congruous with the emerging field of complexity science in the modeling of dynamical physiological systems, especially during pathophysiological states. The SR model of TBI is generalizable to known physical laws. This increase in entropy reduces uncertainty and improves predictive capacity. SR is an appropriate computational framework to enable future smart monitoring devices.

Symbolic regression modeling of noise generation at porous airfoils

Abstract Based on data sets from previous experimental studies, the tool of symbolic regression is applied to find empirical models that describe the noise generation at porous airfoils. Both the self noise from the interaction of a turbulent boundary layer with the trailing edge of an porous airfoil and the noise generated at the leading edge due to turbulent inflow are considered. Following a dimensional analysis, models are built for trailing edge noise and leading edge noise in terms of four and six dimensionless quantities, respectively. Models of different accuracy and complexity are proposed and discussed. For the trailing edge noise case, a general dependency of the sound power on the fifth power of the flow velocity was found and the frequency spectrum is controlled by the flow resistivity of the porous material. Leading edge noise power is proportional to the square of the turbulence intensity and shows a dependency on the fifth to sixth power of the flow velocity, while the spectrum is governed by the flow resistivity and the integral length scale of the incoming turbulence.

Modelling hot rolling manufacturing process using soft computing techniques

Steel making industry is becoming more competitive due to the high demand. In order to protect the market share, automation of the manufacturing industrial process is vital and represents a challenge. Empirical mathematical modelling of the process was used to design mill equipment, ensure productivity and service quality. This modelling approach shows many problems associated to complexity and time consumption. Evolutionary computing techniques show significant modelling capabilities on handling complex non-linear systems modelling. In this research, symbolic regression modelling via genetic programming is used to develop relatively simple mathematical models for the hot rolling industrial non-linear process. Three models are proposed for the rolling force, torque and slab temperature. A set of simple mathematical functions which represents the dynamical relationship between the input and output of these models shall be presented. Moreover, the performance of the symbolic regression models is compared to the known empirical models for the hot rolling system. A comparison with experimental data collected from the Ere[gtilde]li Iron and Steel Factory in Turkey is conducted for the verification of the promising model performance. Genetic programming shows better performance results compared to other soft computing approaches, such as neural networks and fuzzy logic.

Bivariate symbolic regression models for interval-valued variables

Interval-valued variables have become very common in data analysis. Up until now, symbolic regression mostly approaches this type of data from an optimization point of view, considering neither the probabilistic aspects of the models nor the nonlinear relationships between the interval response and the interval predictors. In this article, we formulate interval-valued variables as bivariate random vectors and introduce the bivariate symbolic regression model based on the generalized linear models theory which provides much-needed exibility in practice. Important inferential aspects are investigated. Applications to synthetic and real data illustrate the usefulness of the proposed approach.

Human-competitive results produced by genetic programming

Genetic programming has now been used to produce at least 76 instances of results that are competitive with human-produced results. These human-competitive results come from a wide variety of fields, including quantum computing circuits, analog electrical circuits, antennas, mechanical systems, controllers, game playing, finite algebras, photonic systems, image recognition, optical lens systems, mathematical algorithms, cellular automata rules, bioinformatics, sorting networks, robotics, assembly code generation, software repair, scheduling, communication protocols, symbolic regression, reverse engineering, and empirical model discovery. This paper observes that, despite considerable variation in the techniques employed by the various researchers and research groups that produced these human-competitive results, many of the results share several common features. Many of the results were achieved by using a developmental process and by using native representations regularly used by engineers in the fields involved. The best individual in the initial generation of the run of genetic programming often contains only a small number of operative parts. Most of the results that duplicated the functionality of previously issued patents were novel solutions, not infringing solutions. In addition, the production of human-competitive results, as well as the increased intricacy of the results, are broadly correlated to increased availability of computing power tracked by Moore’s law. The paper ends by predicting that the increased availability of computing power (through both parallel computing and Moore’s law) should result in the production, in the future, of an increasing flow of human-competitive results, as well as more intricate and impressive results.

Engine Emission Modeling Using a Mixed Physics and Regression Approach

This paper presents a novel control-oriented model of the raw emissions of diesel engines. An extended quasistationary approach is developed where some engine process variables, such as combustion or cylinder charge characteristics, are used as inputs. These inputs are chosen by a selection algorithm that is based on genetic-programming techniques. Based on the selected inputs, a hybrid symbolic regression algorithm generates the adequate nonlinear structure of the emission model. With this approach, the model identification efforts can be reduced significantly. Although this symbolic regression model requires fewer than eight parameters to be identified, it provides results comparable to those obtained with artificial neural networks. The symbolic regression model is capable of predicting the behavior of the engine in operating points not used for the model parametrization, and it can be adapted easily to other engine classes. Results from experiments under steady-state and transient operating conditions are used to show the accuracy of the presented model. Possible applications of this model are the optimization of the engine system operation strategy and the derivation of virtual sensor designs.

Improved Genetic Programming Algorithm Applied to Symbolic Regression and Software Reliability Modeling

The present study aims at improving the ability of the canonical genetic programming algorithm to solve problems, and describes an improved genetic programming (IGP). The proposed method can be described as follows: the first inves-tigates initializing population, the second investigates reproduction operator, the third investigates crossover operator, and the fourth investigates mutation operation. The IGP is examined in two domains and the results suggest that the IGP is more effective and more efficient than the canonical one applied in different domains.