Symbolic Regression Problems Research Articles

Statistical genetic programming for symbolic regression

In this paper, a new genetic programming (GP) algorithm for symbolic regression problems is proposed. The algorithm, named statistical genetic programming (SGP), uses statistical information—such as variance, mean and correlation coefficient—to improve GP. To this end, we define well-structured trees as a tree with the following property: nodes which are closer to the root have a higher correlation with the target. It is shown experimentally that on average, the trees with structures closer to well-structured trees are smaller than other trees. SGP biases the search process to find solutions whose structures are closer to a well-structured tree. For this purpose, it extends the terminal set by some small well-structured subtrees, and starts the search process in a search space that is limited to semi-well-structured trees (i.e., trees with at least one well-structured subtree). Moreover, SGP incorporates new genetic operators, i.e., correlation-based mutation and correlation-based crossover, which use the correlation between outputs of each subtree and the targets, to improve the functionality. Furthermore, we suggest a variance-based editing operator which reduces the size of the trees. SGP uses the new operators to explore the search space in a way that it obtains more accurate and smaller solutions in less time.SGP is tested on several symbolic regression benchmarks. The results show that it increases the evolution rate, the accuracy of the solutions, and the generalization ability, and decreases the rate of code growth.

A comparison of fitness-case sampling methods for genetic programming

Genetic programming (GP) is an evolutionary computation paradigm for automatic program induction. GP has produced impressive results but it still needs to overcome some practical limitations, particularly its high computational cost, overfitting and excessive code growth. Recently, many researchers have proposed fitness-case sampling methods to overcome some of these problems, with mixed results in several limited tests. This paper presents an extensive comparative study of four fitness-case sampling methods, namely: Interleaved Sampling, Random Interleaved Sampling, Lexicase Selection and Keep-Worst Interleaved Sampling. The algorithms are compared on 11 symbolic regression problems and 11 supervised classification problems, using 10 synthetic benchmarks and 12 real-world data-sets. They are evaluated based on test performance, overfitting and average program size, comparing them with a standard GP search. Comparisons are carried out using non-parametric multigroup tests and post hoc pairwise statistical tests. The experimental results suggest that fitness-case sampling methods are particularly useful for difficult real-world symbolic regression problems, improving performance, reducing overfitting and limiting code growth. On the other hand, it seems that fitness-case sampling cannot improve upon GP performance when considering supervised binary classification.

Automatic model construction for the behavior of human crowds

Designing suitable behavioral rules of agents so as to generate realistic behaviors is a fundamental and challenging task in many forms of computational modeling. This paper proposes a novel methodology to automatically generate a descriptive model, in the form of behavioral rules, from video data of human crowds. In the proposed methodology, the problem of modeling crowd behaviors is formulated as a symbolic regression problem and the self-learning gene expression programming is utilized to solve the problem and automatically obtain behavioral rules that match data. To evaluate its effectiveness, we apply the proposed method to generate a model from a video dataset in Switzerland and then test the generality of the model by validating against video data from the United States. The results demonstrate that, based on the observed movement of people in one scenario, the proposed methodology can automatically construct a general model capable of describing the crowd dynamics of another scenario in a different context (e.g., Switzerland vs. U.S.) as long as that the crowd behavior patterns are similar.

Dynamic Programming Inspired Genetic Programming to Solve Regression Problems

The candidate solution in traditional Genetic Pro-graming is evolved through prescribed number of generations using fitness measure. It has been observed that, improvement of GP on different problems is insignificant at later generations. Furthermore, GP struggles to evolve on some symbolic regression problems due to high selective pressure, where input range is very small, and few generations are allowed. In such scenarios stagnation of GP occurs and GP cannot evolve a desired solution. Recent works address these issues by using single run to reduce residual error which is based on semantic concept. A new approach is proposed called Dynamic Decomposition of Genetic Programming (DDGP) inspired by dynamic programing. DDGP decomposes a problem into sub problems and initiates sub runs in order to find sub solutions. The algebraic sum of all the sub solutions merge into an overall solution, which provides the desired solution. Experiments conducted on well known benchmarks with varying complexities, validates the proposed approach, as the empirical results of DDGP are far superior to the standard GP. Moreover, statistical analysis has been conducted using T test, which depicted significant difference on eight datasets. Symbolic regression problems where other variants of GP stagnates and cannot evolve the required solution, DDGP is highly recommended for such symbolic regression problems.

Influence of pRNGs onto GPA-ES behaviors

The main aim of this paper is to investigate if the evolutionary algorithms (EAs) can be influenced by (pseudo) Random Number Generators ((p)RNGs) and if different evolutionary operators applied within EAs requires different features of (p)RNGs. This question is significant especially if genetic programming is applied to symbolic regression task with the aim to produce human expert comparable results because such task requires massive computations. Experiments were performed on GPA-ES algorithm combining genetic programming algorithm (GPA) for structure development and evolutionary strategy (ES) algorithm for parameter optimization. This algorithm is described bellow and it applies extended scale of different evolutionary operators (additional individuals generating, symmetric crossover, mutations, and one point crossover). These experiments solved problem of symbolic regression of dynamic system. The number of iterations needed for required quality of regression was used as the measure of (p)RNG influence. These experiments point that different (p)RNGs fit to different evolutionary operators, that some combinations (p)RNGs are better than others and that some theoretically excellent (p)RNGs produces poor results. Presented experiments point that the efficiency of evolutionary algorithms might be increased by application of more (p)RNGs in one algorithm optimised for each particular evolutionary operator.

Automated Conjecturing of Frobenius Numbers via Grammatical Evolution

Conjecturing formulas and other symbolic relations occurs frequently in number theory and combinatorics. If we could automate conjecturing, we could benefit not only from faster conjecturing but also from finding conjectures previously out of our grasp. Grammatical evolution (GE), a genetic programming technique, can be used for automated conjecturing in mathematics. Concretely, this work describes how one can interpret the Frobenius problem as a symbolic regression problem, and then apply GE to it. In this manner, a few formulas for Frobenius numbers of specific quadruples were found automatically. The sketch of the proof of one conjectured formula, using lattice point enumeration method, is provided as well. The same method can easily be used on other problems to speed up and enhance the research process.

Rainstorm flash flood risk assessment using genetic programming: a case study of risk zoning in Beijing

In this study, a genetic programming (GP) algorithm is introduced to solve the symbolic regression problem for flash flood risk zoning in Beijing. GP operates in simulation of biological revolution and can avoid arbitrariness in risk estimates. Herein, this revolutionary computing searched for an appropriate model to best fit the training samples which comprise the data fields of the predictand of Ripley’s K-values to be the posterior risk, and the predictors of the rainstorm hazard index value (RHIV), physical vulnerability, terrain factor, impervious surface area, and population density. After generations of revolution, the optimal fit regressions for estimating the risk value were determined in the form of function (parse) trees. Also, the grid risk values were calculated using the deduced regression. The risk zoning map indicates that the risk values are higher in urban areas, which is reasonable in comparison with the distribution of historical flash flood events. With an explicit model structure, this symbolic regression manifests that the risk value is mainly determined by the RHIV and impervious land surface and is weakly correlated with the other risk factors, e.g., the physical vulnerability, the terrain factor, and population density. Our research demonstrates that GP in an artificial intelligence manner meets the needs of risk assessment in determining the optimal fit regressions and is a promising technique for future applications. Meanwhile, approaches are still available for improving the GP application in the risk assessment, e.g., considering the historical losses in posterior risk estimations and improvement in the sampling training data.

Self-Learning Gene Expression Programming

In this paper, a novel self-learning gene expression programming (GEP) methodology named SL-GEP is proposed to improve the search accuracy and efficiency of GEP. In contrast to the existing GEP variants, the proposed SL-GEP features a novel chromosome representation in which each chromosome is embedded with subfunctions that can be deployed to construct the final solution. As part of the chromosome, the subfunctions are self-learned or self-evolved by the proposed algorithm during the evolutionary search. By encompassing subfunctions or any partial solution as input arguments of another subfunction, the proposed SL-GEP facilitates the formation of sophisticated, higher-order, and constructive subfunctions that improve the accuracy and efficiency of the search. Further, a novel search mechanism based on differential evolution is proposed for the evolution of chromosomes in the SL-GEP. The proposed SL-GEP is simple, generic and has much fewer control parameters than the traditional GEP variants. The proposed SL-GEP is validated on 15 symbolic regression problems and six even-parity problems. Experimental results show that the proposed SL-GEP offers enhanced performances over several state-of-the-art algorithms in terms of accuracy and search efficiency.

Grammatical evolution using two-dimensional gene for symbolic regression: an advanced improvement with conditional statement grammar

Symbolic regression problems can be solved using grammatical evolution (GE), an evolutionary computation (EC) method, to find a function that coincides satisfactorily with the given datasets. The evolutional approach of GE is based on the grammar learning paradigm, which can translate the genotype (binary digit) into the phenotype (terminals and non-terminals). Unlike traditional codons in a genotype, the fittest codons in phenotype represented by the Backus-Naur form (BNF) are difficult for next generation genes to inherit the traits of parents, accounting for crossover and mutation. For this issue, this article presents a proposal of an advanced improvement to GE using a two-dimensional gene (GE2DG). In contrast to multi-chromosomal GE (GEMC), our proposal not only encloses the two-dimensional gene-expression for symbolic regression, but also introduces one independent gene defined as a conditional statement to express a new BNF grammar of an if-then (-else) branch. In the experiments described herein, continuous/discontinuous non-branch functions and continuous/discontinuous branch functions, four testing patterns, are considered as numerical examples. Results show that GE2DG has better performance than the original GE or GEMC. Especially for the case of branch functions, GE with hybrid chromosome (GEHC), where GE2DG is incorporated with GEMC, has faster convergence in symbolic regression than other methods.

Using Genetic Programming with Prior Formula Knowledge to Solve Symbolic Regression Problem.

A researcher can infer mathematical expressions of functions quickly by using his professional knowledge (called Prior Knowledge). But the results he finds may be biased and restricted to his research field due to limitation of his knowledge. In contrast, Genetic Programming method can discover fitted mathematical expressions from the huge search space through running evolutionary algorithms. And its results can be generalized to accommodate different fields of knowledge. However, since GP has to search a huge space, its speed of finding the results is rather slow. Therefore, in this paper, a framework of connection between Prior Formula Knowledge and GP (PFK-GP) is proposed to reduce the space of GP searching. The PFK is built based on the Deep Belief Network (DBN) which can identify candidate formulas that are consistent with the features of experimental data. By using these candidate formulas as the seed of a randomly generated population, PFK-GP finds the right formulas quickly by exploring the search space of data features. We have compared PFK-GP with Pareto GP on regression of eight benchmark problems. The experimental results confirm that the PFK-GP can reduce the search space and obtain the significant improvement in the quality of SR.

Subtree semantic geometric crossover for genetic programming

The semantic geometric crossover (SGX) proposed by Moraglio et al. has achieved very promising results and received great attention from researchers, but has a significant disadvantage in the exponential growth in size of the solutions. We propose a crossover operator named subtree semantic geometric crossover (SSGX), with the aim of addressing this issue. It is similar to SGX but uses subtree semantic similarity to approximate the geometric property. We compare SSGX to standard crossover (SC), to SGX, and to other recent semantic-based crossover operators, testing on several symbolic regression problems. Overall our new operator out-performs the other operators on test data performance, and reduces computational time relative to most of them. Further analysis shows that while SGX is rather exploitative, and SC rather explorative, SSGX achieves a balance between the two. A simple method of further enhancing SSGX performance is also demonstrated.

Solution Modeling Using Postfix Genetic Programming

This article introduces Postfix Genetic Programming (GP), a postfix notation-based GP, approach to symbolic regression for solving empirical modeling problems. The main features of Postfix-GP are presented. These features include (1) postfix-based, variable-length individual-representation and its stack-based evaluation and (2) subtree crossover operator and semantic-aware subtree crossover operator. The article also presents two different constant creation approaches to evolve useful numeric constants for symbolic regression problems. The first approach uses an explicit list of constants and the second allows the Postfix-GP algorithm to evolve constants. Use of Postfix-GP as a solution modeling tool is demonstrated by solving two function identification problems and two deterministic chaotic time series modeling problems. Effectiveness of the evolved models is tested by statistically analyzing their performance on training and out-of-sample datasets of the problems. Experimental results on test problems suggest that Postfix-GP offers a new possibility for solving empirical modeling problems. We also compared the performance of Postfix-GP with the lil-GP system for all four problems. The results suggest that the quality of solutions found by Postfix-GP was better than that found by lil-GP, for the tested problems.

Improving GP generalization: a variance-based layered learning approach

This paper introduces a new method that improves the generalization ability of genetic programming (GP) for symbolic regression problems, named variance-based layered learning GP. In this approach, several datasets, called primitive training sets, are derived from the original training data. They are generated from less complex to more complex, for a suitable complexity measure. The last primitive dataset is still less complex than the original training set. The approach decomposes the evolution process into several hierarchical layers. The first layer of the evolution starts using the least complex (smoothest) primitive training set. In the next layers, more complex primitive sets are given to the GP engine. Finally, the original training data is given to the algorithm. We use the variance of the output values of a function as a measure of the functional complexity. This measure is utilized in order to generate smoother training data, and controlling the functional complexity of the solutions to reduce the overfitting. The experiments, conducted on four real-world and three artificial symbolic regression problems, demonstrate that the approach enhances the generalization ability of the GP, and reduces the complexity of the obtained solutions.

An improved Gene Expression Programming approach for symbolic regression problems

Gene Expression Programming (GEP) is a powerful evolutionary method for knowledge discovery and model learning. Based on the basic GEP algorithm, this paper proposes an improved algorithm named S_GEP, which is especially suitable for dealing with symbolic regression problems. The major advantages for this S_GEP method include: (1) A new method for evaluating individual without expression tree; (2) a corresponding expression tree construction schema for the new evaluating individual method if required by some special complex problems; and (3) a new approach for manipulating numeric constants so as to improve the convergence. A thorough comparative study between our proposed S_GEP method with the primitive GEP, as well as other methods are included in this paper. The comparative results show that the proposed S_GEP method can significantly improve the GEP performance. Several well-studied benchmark test cases and real-world test cases demonstrate the efficiency and capability of our proposed S_GEP for symbolic regression problems.

遺伝的プログラミングを用いた航空機概念設計手法の構築に向けた研究

The purpose of this paper is creating a method to determine the locations of components applying Genetic Programming (GP) – one of evolutionary computation algorithm which is often applied to solve symbolic regression problems. In the aircraft conceptual design, first, mission requirements are given. Secondly, the parameters of main components such as wing, tail, fuselage and engine are determined to meet these requirements. Finally, the locations of the components are fixed considering the center of gravity, stability and gear layout. Then an initial three-dimensional view is drawn. The wings and the gear locations are mainly determined by the designers' sense based on their experience as there is no well-defined method during the preliminary phase. Therefore, GP is applied to the problem relating to the aircraft conceptual design and the functions between the locations of components and constraints such as stability and gear layout are obtained. These results indicated potential abilities of GP to create a model which would help the aircraft designer to determine the locations of the components during the design process.

Performance comparison of crossover operators for postfix genetic programming

In this article, we present three crossover operators for postfix-GP, a GP system that adopts postfix notation for an individual representation. These crossover operators are: GA-like one-point, sub-tree, and semantic aware sub-tree. The algorithm and implementation details for each of these crossover operators are presented. The operators are applied on a set of real-valued symbolic regression problems. The performance comparison of the crossover operators is carried out using two measures, number of successful runs and mean best adjusted fitness. The significance of the obtained results is tested using statistical test. The results suggest that semantic aware sub-tree crossover outperforms GA-like one-point and sub-tree crossovers on all problems.

Solving symbolic regression problems with uniform design-aided gene expression programming

Gene Expression Programming (GEP) significantly surpasses traditional evolutionary approaches to solving symbolic regression problems. However, existing GEP algorithms still suffer from premature c...

Improvement of Individual Definition in Grammatical Evolution of Symbolic Regression Problem

Improvement of Individual Definition in Grammatical Evolution of Symbolic Regression Problem

Parse-matrix evolution for symbolic regression

Data-driven model is highly desirable for industrial data analysis in case the experimental model structure is unknown or wrong, or the concerned system has changed. Symbolic regression is a useful method to construct the data-driven model (regression equation). Existing algorithms for symbolic regression such as genetic programming and grammatical evolution are difficult to use due to their special target programming language (i.e., LISP) or additional function parsing process. In this paper, a new evolutionary algorithm, parse-matrix evolution (PME), for symbolic regression is proposed. A chromosome in PME is a parse-matrix with integer entries. The mapping process from the chromosome to the regression equation is based on a mapping table. PME can easily be implemented in any programming language and free to control. Furthermore, it does not need any additional function parsing process. Numerical results show that PME can solve the symbolic regression problems effectively.

Semantically-based crossover in genetic programming: application to real-valued symbolic regression

We investigate the effects of semantically-based crossover operators in genetic programming, applied to real-valued symbolic regression problems. We propose two new relations derived from the semantic distance between subtrees, known as semantic equivalence and semantic similarity. These relations are used to guide variants of the crossover operator, resulting in two new crossover operators--semantics aware crossover (SAC) and semantic similarity-based crossover (SSC). SAC, was introduced and previously studied, is added here for the purpose of comparison and analysis. SSC extends SAC by more closely controlling the semantic distance between subtrees to which crossover may be applied. The new operators were tested on some real-valued symbolic regression problems and compared with standard crossover (SC), context aware crossover (CAC), Soft Brood Selection (SBS), and No Same Mate (NSM) selection. The experimental results show on the problems examined that, with computational effort measured by the number of function node evaluations, only SSC and SBS were significantly better than SC, and SSC was often better than SBS. Further experiments were also conducted to analyse the perfomance sensitivity to the parameter settings for SSC. This analysis leads to a conclusion that SSC is more constructive and has higher locality than SAC, NSM and SC; we believe these are the main reasons for the improved performance of SSC.