Genetic Programming For Symbolic Regression Research Articles

Semantic Linear Genetic Programming for Symbolic Regression.

Symbolic regression (SR) is an important problem with many applications, such as automatic programming tasks and data mining. Genetic programming (GP) is a commonly used technique for SR. In the past decade, a branch of GP that utilizes the program behavior to guide the search, called semantic GP (SGP), has achieved great success in solving SR problems. However, existing SGP methods only focus on the tree-based chromosome representation and usually encounter the bloat issue and unsatisfactory generalization ability. To address these issues, we propose a new semantic linear GP (SLGP) algorithm. In SLGP, we design a new chromosome representation to encode the programs and semantic information in a linear fashion. To utilize the semantic information more effectively, we further propose a novel semantic genetic operator, namely, mutate-and-divide propagation, to recursively propagate the semantic error within the linear program. The empirical results show that the proposed method has better training and test errors than the state-of-the-art algorithms in solving SR problems and can achieve a much smaller program size.

Population diversity and inheritance in genetic programming for symbolic regression

AbstractIn this work we aim to empirically characterize two important dynamical aspects of GP search: the evolution of diversity and the propagation of inheritance patterns. Diversity is calculated at the genotypic and phenotypic levels using efficient similarity metrics. Inheritance information is obtained via a full genealogical record of evolution as a directed acyclic graph and a set of methods for extracting relevant patterns. Advances in processing power enable our approach to handle previously infeasible graph sizes of millions of arcs and vertices. To enable a more comprehensive analysis we employ three closely-related but different evolutionary models: canonical GP, offspring selection and age-layered population structure. Our analysis reveals that a relatively small number of ancestors are responsible for producing the majority of descendants in later generations, leading to diversity loss. We show empirically across a selection of five benchmark problems that each configuration is characterized by different rates of diversity loss and different inheritance patterns, in support of the idea that each new problem may require a unique approach to solve optimally.

Shape-constrained multi-objective genetic programming for symbolic regression

We describe and analyze algorithms for shape-constrained symbolic regression, which allow the inclusion of prior knowledge about the shape of the regression function. This is relevant in many areas of engineering — in particular, when data-driven models, which are based on data of measurements must exhibit certain properties (e.g. positivity, monotonicity, or convexity/concavity). To satisfy these properties, we have extended multi-objective algorithms with shape constraints. A soft-penalty approach is used to minimize both the constraint violations and the prediction error. We use the non-dominated sorting genetic algorithm (NSGA-II) as well as the multi-objective evolutionary algorithm based on decomposition (MOEA/D). The algorithms are tested on a set of models from physics textbooks and compared against previous results achieved with single objective algorithms. Further, we generated out-of-domain samples to test the extrapolation behavior using shape constraints and added a different level of noise on the training data to verify if shape constraints can still help maintain the prediction errors to a minimum and generate valid models. The results showed that the multi-objective algorithms were capable of finding mostly valid models, also when using a soft-penalty approach. Further, we investigated that NSGA-II achieved the best overall ranks on high noise instances.

Semantic schema based genetic programming for symbolic regression

Despite the empirical success of Genetic programming (GP) in various symbolic regression applications, GP is not still known as a reliable problem-solving technique in this domain. Non-locality of GP representation and operators causes ineffectiveness of its search procedure. This study employs semantic schema theory to control and guide the GP search and proposes a local GP called semantic schema-based genetic programming (SBGP). SBGP partitions the semantic search space into semantic schemas and biases the search to the significant schema of the population, which is gradually progressing towards the optimal solution. Several semantic local operators are proposed for performing a local search around the significant schema. In combination with schema evolution as a global search, the local in-schema search provides an efficient exploration–exploitation control mechanism in SBGP. For evaluating the proposed method, we use six benchmarks, including synthesized and real-world problems. The obtained errors are compared to the best semantic genetic programming algorithms, on the one hand, and data-driven layered learning approaches, on the other hand. Results demonstrate that SBGP outperforms all mentioned methods in four out of six benchmarks up to 87% in the first set and up to 76% in the second set of experiments in terms of generalization measured by root mean squared error.

Improving Model-Based Genetic Programming for Symbolic Regression of Small Expressions.

The Gene-pool Optimal Mixing Evolutionary Algorithm (GOMEA) is a model-based EA framework that has been shown to perform well in several domains, including Genetic Programming (GP). Differently from traditional EAs where variation acts blindly, GOMEA learns a model of interdependencies within the genotype, that is, the linkage, to estimate what patterns to propagate. In this article, we study the role of Linkage Learning (LL) performed by GOMEA in Symbolic Regression (SR). We show that the non-uniformity in the distribution of the genotype in GP populations negatively biases LL, and propose a method to correct for this. We also propose approaches to improve LL when ephemeral random constants are used. Furthermore, we adapt a scheme of interleaving runs to alleviate the burden of tuning the population size, a crucial parameter for LL, to SR. We run experiments on 10 real-world datasets, enforcing a strict limitation on solution size, to enable interpretability. We find that the new LL method outperforms the standard one, and that GOMEA outperforms both traditional and semantic GP. We also find that the small solutions evolved by GOMEA are competitive with tuned decision trees, making GOMEA a promising new approach to SR.

Preserving Population Diversity Based on Transformed Semantics in Genetic Programming for Symbolic Regression

Population diversity plays an important role in avoiding premature convergence in evolutionary techniques including genetic programming (GP). Obtaining an adequate level of diversity during the evolutionary process has became a concern of many previous researches in GP. This work proposes a new novelty metric for entropy-based diversity measure for GP. The new novelty metric is based on the transformed semantics of models in GP, where the semantics are the set of outputs of a model on the training data and principal component analysis is used for a transformation of the semantics. Based on the new novelty metric, a new diversity preserving framework, which incorporates a new fitness function and a new selection operator, is proposed to help GP achieve a good balance between the exploration and the exploitation, thus enhancing its learning and generalization performance. Compared with two stat-of-the-art diversity preserving methods, the new method can generalize better and reduce the overfitting trend more effectively in most cases. Further examinations on the properties of the search process confirm that the new framework notably enhances the evolvability and locality of GP.

Rademacher Complexity for Enhancing the Generalization of Genetic Programming for Symbolic Regression.

Model complexity has a close relationship with the generalization ability and the interpretability of the learned models. Simple models are more likely to generalize well and easy to interpret. However, too much emphasis on minimizing complexity can prevent the discovery of more complex yet more accurate solutions. Genetic programming (GP) has a trend of generating overcomplex models that are difficult to interpret while not being able to generalize well. This work proposes a novel complexity measure based on the Rademacher complexity for GP for symbolic regression. The complexity of an evolved model is measured by the maximum correlation between the model and the Rademacher variables on the selected training instances. Taking minimizing the training error and the Rademacher complexity of the models as the two objectives, the proposed GP method has shown to be much superior to the standard GP on generalization performance. Compared with GP equipped with two state-of-the-art complexity measures, the proposed method still has a notable advance on generating a better front consisting of individuals with lower generalization errors and being simpler in the behavioral complexity. Further analyses reveal that compared with the state-of-the-art methods, the proposed GP method evolves models that are much closer to the target models in the model structure, and have better interpretability.

Genetic Programming for Instance Transfer Learning in Symbolic Regression.

Transfer learning has attracted more attention in the machine-learning community recently. It aims to improve the learning performance on the domain of interest with the help of the knowledge acquired from a similar domain(s). However, there is only a limited number of research on tackling transfer learning in genetic programming for symbolic regression. This article attempts to fill this gap by proposing a new instance weighting framework for transfer learning in genetic programming-based symbolic regression. In the new framework, differential evolution is employed to search for optimal weights for source-domain instances, which helps genetic programming to identify more useful source-domain instances and learn from them. Meanwhile, a density estimation method is used to provide good starting points to help the search for the optimal weights while discarding some irrelevant or less important source-domain instances before learning regression models. The experimental results show that compared with genetic programming and support vector regression that learn only from the target instances, and learning from a mixture of instances from the source and target domains without any transfer learning component, the proposed method can evolve regression models which not only achieve notably better cross-domain generalization performance in stability but also reduce the trend of overfitting effectively. Meanwhile, these models are generally much simpler than those generated by the other GP methods.

An empirical equation for the gas compressibility factor, Z

An empirical equation is presented for the gas compressibility factor, Z, as a function of reduced temperature and reduced pressure. The average error of the equation is 2.5%, median error is 1.3%, and the coefficient of determination (R2) is 0.98. The equation is presented in a manner ready for use in software code. Comparison is made to the traditional generalized compressibility factor chart. The equation was found using genetic programming for symbolic regression coupled with the Levenburg-Marquardt algorithm for fitting-parameter optimization.

Improving Generalization of Genetic Programming for Symbolic Regression With Angle-Driven Geometric Semantic Operators

Geometric semantic genetic programming (GP) has recently attracted much attention. The key innovations are inducing a unimodal fitness landscape in the semantic space and providing a theoretical framework for designing geometric semantic operators. The geometric semantic operators aim to manipulate the semantics of programs by making a bounded semantic impact and generating child programs with similar or better behavior than their parents. These properties are shown to be highly related to a notable generalization improvement in GP. However, the potential ineffectiveness and difficulties in bounding the variations in these geometric operators still limits their positive effect on generalization. This paper attempts to further explore the geometry and search space of geometric operators to gain a greater generalization improvement in GP for symbolic regression. To this end, a new angle-driven selection operator and two new angle-driven geometric search operators are proposed. The angle-awareness brings new geometric properties to these geometric operators, which are expected to provide a greater leverage for approximating the target semantics in each operation, and more importantly, be resistant to overfitting. The experiments show that compared with two state-of-the-art geometric semantic operators, our angle-driven geometric operators not only drive the evolutionary process to fit the target semantics more efficiently but also improve the generalization performance. A further comparison between the evolved models shows that the new method generally produces simpler models with a much smaller size and is more likely to evolve toward the correct structure of the target models.

Multiple response optimization: Analysis of genetic programming for symbolic regression and assessment of desirability functions

Multiple responses optimization (MRO) consists in the search for the best settings in an problem with conflicting responses. MRO is performed following the steps: experimental design; experimental data gathering; mathematical models building; statistical validation of models; agglutination of the models responses in only one function to be optimized; optimization of agglutinated function; experimental validation of the best conditions. This work selected two MRO cases from literature aiming to compare two methods of mathematical models building and two agglutinating functions to assess the best one among the four possible combinations. The methods used in mathematical models building were the ordinary least squares performed in Minitab (v. 17) and genetic programming performed in Eureqa Formulize (v. 1.24.0). The assessment of the best method for building mathematical models was performed using the Akaike Information Criterion. The responses agglutination were performed using the desirability and modified desirability functions. In all MRO cases, the optimization step was performed by generalized reduced gradient method on Microsoft ExcelTM software. The average percentage distance between predicted and experimental results was used to both assess the best agglutination function and verify the effect of the method used in the building of the mathematical models about its fitness to estimate the best condition close to that one obtained on experimental validation step. The obtained results suggest as the better strategy for multiple responses optimization the use, jointly, of genetic programming to mathematical models building and the modified desirability function to responses agglutination.

Interpretable policies for reinforcement learning by genetic programming

The search for interpretable reinforcement learning policies is of high academic and industrial interest. Especially for industrial systems, domain experts are more likely to deploy autonomously learned controllers if they are understandable and convenient to evaluate. Basic algebraic equations are supposed to meet these requirements, as long as they are restricted to an adequate complexity. Here we introduce the genetic programming for reinforcement learning (GPRL) approach based on model-based batch reinforcement learning and genetic programming, which autonomously learns policy equations from pre-existing default state–action trajectory samples. GPRL is compared to a straightforward method which utilizes genetic programming for symbolic regression, yielding policies imitating an existing well-performing, but non-interpretable policy. Experiments on three reinforcement learning benchmarks, i.e., mountain car, cart–pole balancing, and industrial benchmark, demonstrate the superiority of our GPRL approach compared to the symbolic regression method. GPRL is capable of producing well-performing interpretable reinforcement learning policies from pre-existing default trajectory data.

Feature Selection to Improve Generalization of Genetic Programming for High-Dimensional Symbolic Regression

When learning from high-dimensional data for symbolic regression (SR), genetic programming (GP) typically could not generalize well. Feature selection, as a data preprocessing method, can potentially contribute not only to improving the efficiency of learning algorithms but also to enhancing the generalization ability. However, in GP for high-dimensional SR, feature selection before learning is seldom considered. In this paper, we propose a new feature selection method based on permutation to select features for high-dimensional SR using GP. A set of experiments has been conducted to investigate the performance of the proposed method on the generalization of GP for high-dimensional SR. The regression results confirm the superior performance of the proposed method over the other examined feature selection methods. Further analysis indicates that the models evolved by the proposed method are more likely to contain only the truly relevant features and have better interpretability.

Competent Geometric Semantic Genetic Programming for Symbolic Regression and Boolean Function Synthesis.

Program semantics is a promising recent research thread in Genetic Programming (GP). Over adozen semantic-aware search, selection, and initialization operators for GP have been proposed to date. Some of these operators are designed to exploit the geometric properties of semantic space, while others focus on making offspring effective, that is, semantically different from their parents. Only asmall fraction of previous works aimed at addressing both of these features simultaneously. In this article, we propose asuite of competent operators that combine effectiveness with geometry for population initialization, mate selection, mutation, and crossover. We present atheoretical rationale behind these operators and compare them experimentally to operators known from literature on symbolic regression and Boolean function synthesis benchmarks. We analyze each operator in isolation as well as verify how they fare together in an evolutionary run, concluding that the competent operators are superior on awide range of performance indicators, including best-of-run fitness, test-set fitness, and programsize.

Studying bloat control and maintenance of effective code in linear genetic programming for symbolic regression

Linear Genetic Programming (LGP) is an Evolutionary Computation algorithm, inspired in the Genetic Programming (GP) algorithm. Instead of using the standard tree representation of GP, LGP evolves a linear program, which causes a graph-based data flow with code reuse. LGP has been shown to outperform GP in several problems, including Symbolic Regression (SReg), and to produce simpler solutions. In this paper, we propose several LGP variants and compare them with a traditional LGP algorithm on a set of benchmark SReg functions from the literature. The main objectives of the variants were to both control bloat and privilege useful code in the population. Here we evaluate their effects during the evolution process and in the quality of the final solutions. Analysis of the results showed that bloat control and effective code maintenance worked, but they did not guarantee improvement in solution quality.

A comparison between parametric and non-parametric soft computing approaches to model the temperature of a metal cutting tool

Modelling the temperature of cutting tools in industry has become a vital assignment for manufacturing process modelling. The quality and the cost of the produced products are significantly affected if the process is inaccurate. However, the process of modelling the temperature is complex because the relationship between the model variables of the temperature is nonlinear. In this article, five soft computing techniques are used to develop mathematical models for two different cutting tools. Genetic Algorithm, Particle Swarm Optimisation and Cuckoo Search are used to estimate the parameters of a common empirical model for the temperature of the cutting tools. This empirical model is also nonlinear model of the parameters. Thus, traditional modelling techniques such as least square estimation will have trouble finding the optimal set of model parameters. The challenging problem which is also tackled in this article is the development of nonlinear model of the cutting tool using Artificial Neural Networks and Multigene Symbolic Regression Genetic Programming (GP). In the case of ANNs, the model structure is hidden inside the network, whereas in the case of Multigene Symbolic Regression GP, the developed model equation shall reveal the relationship between model variables. Obtained models will be validated based on many evaluation criteria.

Influence of Nitrogen-di-Oxide, Temperature and Relative Humidity on Surface Ozone Modeling Process Using Multigene Symbolic Regression Genetic Programming

Automatic monitoring, data collection, analysis and prediction of environmental changes is essential for all living things. Understanding future climate changes does not only helps in measuring the influence on people life, habits, agricultural and health but also helps in avoiding disasters. Giving the high emission of chemicals on air, scientist discovered the growing depletion in ozone layer. This causes a serious environmental problem. Modeling and observing changes in the Ozone layer have been studied in the past. Understanding the dynamics of the pollutants features that influence Ozone is ex-plored in this article. A short term prediction model for surface Ozone is offered using Multigene Symbolic Regression Genetic Programming (GP). The proposed model customs Nitrogen-di-Oxide, Temperature and Relative Humidity as the main features to predict the Ozone level. Moreover, a comparison between GP and Artificial Neural Network (ANN) in modeling Ozone is presented. The developed results show that GP outperform the ANN.