Tree-based Genetic Programming Research Articles

Multitask Linear Genetic Programming with Shared Individuals and its Application to Dynamic Job Shop Scheduling

Multitask genetic programming methods have been applied to various domains, such as classification, regression, and combinatorial optimization problems. Most existing multitask genetic programming methods are designed based on tree-based structures, which are not good at reusing building blocks since each sub-tree passes its outputs to only one parent. It may limit the design and performance of knowledge sharing in multitask optimization. Different from tree-based genetic programming, building blocks in linear genetic programming can be easily reused by more than one parent. Besides, existing multitask genetic programming methods always allocate each individual to a specific task and have to duplicate genetic materials from task to task in knowledge transfer, which is inefficient and often produces redundancy. Contrarily, it is natural for a linear genetic programming individual to produce multiple distinct outputs, which enables each linear genetic programming individual to solve multiple tasks simultaneously. With this in mind, we propose a new multitask linear genetic programming method that transfers knowledge via multi-output individuals (i.e., shared individuals among tasks). By integrating different solutions into one multi-output individual, the proposed method efficiently reuses common knowledge among tasks and maintains distinct behaviors for each task. The empirical results show that the proposed method has a significantly better test performance than state-of-the-art multitask genetic programming methods. Further analyses verify that the new knowledge transfer mechanism can adjust the transfer rate automatically and thus improves its effectiveness.

Synthesizing Missing Travel Time of P-Wave and S-Wave: A Two-Stage Evolutionary Modeling Approach

Acquiring sonic waves is an essential part of oil and gas exploration as they give critical information about the well’s data and lithography at each well depth progression. However, these measurements are not always accessible, making analysis challenging. As computational power has improved, machine learning methods may now be used to predict these values from other data. Nonetheless, one shortcoming of existing models is that the most of them are not transparent (i.e., black box models). As a result, although promising great performance, they do not offer much insights to petrophysicists and geologists. This research aims to generate mathematical models for predicting compression wave (P-wave) and shear wave (S-wave) readings using a multistage evolutionary modelling approach. In particular, a multistage equation modelling approach using tree-based Genetic Programming (GP) and Adaptive Differential Evolution (ADE) is proposed. The obtained best mathematical models yield R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> of 0.745 and 0.9066 for P-wave and S-wave regression on normalized data, respectively. The average performance of models are R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.90 (P-Wave) and R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.75 (S-Wave). The performance of these mathematical models are comparable with other ”black box” models but with more compact mathematical approach in regression thereby opening opportunities for interpretability and analysis. Finally, the ”white box” models presented in this paper can be fine-tuned further as needed.

Sensor abnormality detection in multistage compressor units: A “white box” approach using tree-based genetic programming

Sensors are crucial in detecting equipment problems in various plant systems. In particular, detecting sensor abnormality is challenging in the case of utilizing the data which are acquired and stored offline (data logs). These data are normally acquired using Internet of Things (IoT) system and stored in a dedicated server. This situation presents both opportunities and challenges for exploration in sensor abnormality detection task. In this paper, we propose a multistage compressor sensor fault detection method using data logs. In the proposed method, the compressor sensor output is modeled as a function of other sensors using static approach. Subsequently, the model output is used for detecting abnormality by observing the residuals. It has been shown that the histogram of residuals offers rich information to predict abnormality of the targeted sensor. In particular, we explore the concept using Genetic Programming (GP) to generate regression model which offers more “white box” solution to the operators. There are various advantages in this approach. Firstly, the conventional “black box” approach lacks model transparency and, thus, is highly undesirable in critical systems. Secondly, equations are more easily applied in Programmable Logic Controller (PLC) if autonomous flagging is required. We also compare the proposed model with Multiple Linear Regression (MLR) and Neural Network Regression (ANN). Results show that the best generated models are comparable with the latter but with more crisp “white box” mathematical equations utilizing lesser feature inputs (four features only). This model yields R2 of 0.991 and RMSE of 2.1×10−2.

Dissipative Arithmetic

Large arithmetic expressions are dissipative: they lose information and are robust to perturbations. Lack of conservation gives resilience to fluctuations. The limited precision of floating point and the mixture of linear and nonlinear operations make such functions anti-fragile and give a largely stable locally flat plateau a rich fitness landscape. This slows long-term evolution of complex programs, suggesting a need for depth-aware crossover and mutation operators in tree-based genetic programming. It also suggests that deeply nested computer program source code is error tolerant because disruptions tend to fail to propagate, and therefore the optimal placement of test oracles is as close to software defects as practical.

Shape-Constrained Symbolic Regression-Improving Extrapolation with Prior Knowledge.

We investigate the addition of constraints on the function image and its derivatives for the incorporation of prior knowledge in symbolic regression. The approach is called shape-constrained symbolic regression and allows us to enforce, for example, monotonicity of the function over selected inputs. The aim is to find models which conform to expected behavior and which have improved extrapolation capabilities. We demonstrate the feasibility of the idea and propose and compare two evolutionary algorithms for shape-constrained symbolic regression: (i) an extension of tree-based genetic programming which discards infeasible solutions in the selection step, and (ii) a two-population evolutionary algorithm that separates the feasible from the infeasible solutions. In both algorithms we use interval arithmetic to approximate bounds for models and their partial derivatives. The algorithms are tested on a set of 19 synthetic and four real-world regression problems. Both algorithms are able to identify models which conform to shape constraints which is not the case for the unmodified symbolic regression algorithms. However, the predictive accuracy of models with constraints is worse on the training set and the test set. Shape-constrained polynomial regression produces the best results for the test set but also significantly larger models.

GP-DMD: a genetic programming variant with dynamic management of diversity

The proper management of diversity is essential to the success of Evolutionary Algorithms. Specifically, methods that explicitly relate the amount of diversity maintained in the population to the stopping criterion and elapsed period of execution, with the aim of attaining a gradual shift from exploration to exploitation, have been particularly successful. However, in the area of Genetic Programming, the performance of this design principle has not been studied. In this paper, a novel Genetic Programming method, Genetic Programming with Dynamic Management of Diversity (GP-DMD), is presented. GP-DMD applies this design principle through a replacement strategy that combines penalties based on distance-like functions with a multi-objective Pareto selection based on accuracy and simplicity. The proposed general method was adapted to the well-established Symbolic Regression benchmark problem using tree-based Genetic Programming. Several state-of-the-art diversity management approaches were considered for the experimental validation, and the results obtained showcase the improvements both in terms of mean square error and size. The effects of GP-DMD on the dynamics of the population are also analyzed, revealing the reasons for its superiority. As in other fields of Evolutionary Computation, this design principle contributes significantly to the area of Genetic Programming.

Multi-Objective Memetic Algorithms with Tree-Based Genetic Programming and Local Search for Symbolic Regression

Symbolic regression is to search the space of mathematical expressions to find a model that best fits a given dataset. As genetic programming (GP) with the tree representation can represent solutions as expression trees, it is popularly-used for regression. However, GP tends to evolve unnecessarily large programs (known as bloat), causing excessive use of CPU time/memory and evolving solutions with poor generalization ability. Moreover, even though the importance of local search has been proved in augmenting the search ability of GP (termed as memetic algorithms), local search is underused in GP-based methods. This work aims to handle the above problems simultaneously. To control bloat, a multi-objective (MO) technique (NSGA-II, Non-dominant Sorting Genetic Algorithm) is selected to incorporate with GP, forming a multi-objective GP (MOGP). Moreover, three mutation-based local search operators are designed and incorporated with MOGP respectively to form three multi-objective memetic algorithms (MOMA), i.e. MOMA_MR (MOMA with Mutation-based Random search), MOMA_MF (MOMA with Mutation-based Function search) and MOMA_MC (MOMA with Mutation-based Constant search). The proposed methods are tested on both benchmark functions and real-world applications, and are compared with both GP-based (i.e. GP and MOGP) and nonGP-based symbolic regression methods. Compared with GP-based methods, the proposed methods can reduce the risk of bloat with the evolved solutions significantly smaller than GP solutions, and the local search strategies introduced in the proposed methods can improve their search ability with the evolved solutions dominating MOGP solutions. In addition, among the three proposed methods, MOMA_MR performs best in RMSE for testing, yet it consumes more training time than others. Moreover, compared with six reference nonGP-based symbolic regression methods, MOMA_MR generally performs better than or similar to them consistently.

Genetic programming hyperheuristic parameter configuration using fitness landscape analysis

Fitness landscape analysis is a tool that can help us gain insight into a problem, determine how hard it is to solve a problem using a given algorithm, choose an algorithm for solving a given problem, or choose good algorithm parameters for solving the problem. In this paper, fitness landscape analysis of hyperheuristics is used for clustering instances of three scheduling problems. After that, good parameters for tree-based genetic programming that can solve a given scheduling problem are calculated automatically for every cluster. Additionally, we introduce tree editing operators which help in the calculation of fitness landscape features in tree based genetic programming. A heuristic is proposed based on introduced operators, and it calculates the distance between any two trees. The results show that the proposed approach can obtain parameters that offer better performance compared to manual parameter selection.

Interpretable policy derivation for reinforcement learning based on evolutionary feature synthesis

Reinforcement learning based on the deep neural network has attracted much attention and has been widely used in real-world applications. However, the black-box property limits its usage from applying in high-stake areas, such as manufacture and healthcare. To deal with this problem, some researchers resort to the interpretable control policy generation algorithm. The basic idea is to use an interpretable model, such as tree-based genetic programming, to extract policy from other black box modes, such as neural networks. Following this idea, in this paper, we try yet another form of the genetic programming technique, evolutionary feature synthesis, to extract control policy from the neural network. We also propose an evolutionary method to optimize the operator set of the control policy for each specific problem automatically. Moreover, a policy simplification strategy is also introduced. We conduct experiments on four reinforcement learning environments. The experiment results reveal that evolutionary feature synthesis can achieve better performance than tree-based genetic programming to extract policy from the neural network with comparable interpretability.

Compositional kernel learning using tree-based genetic programming for Gaussian process regression

Although Gaussian process regression (GPR) is a powerful Bayesian nonparametric regression model for engineering problems, its predictive performance is highly dependent on a kernel for covariance function of GPR. However, choosing a proper kernel is still challenging even for experts. To choose proper kernel automatically, this study proposes a compositional kernel (CPK) learning using tree-based genetic programming (GEP). The optimal structure of the kernel is defined as a compositional representation based on sums and products of eight base-kernels. The CPK can be encoded as a tree-structure, so that tree-based GEP is employed to discover an optimal tree-structure of the CPK. To avoid overly complex solution in GEP, the proposed method introduced a dynamic maximum tree-depth technique. The novelty of the proposed method is to utilize more flexible and efficient learning capability to learn the relationship between input and output than existing methods. To evaluate the learning capability of the proposed method, seven test functions were firstly investigated with various noise levels, and its predictive accuracy was compared with existing methods. Reliability problems in both parallel and series systems were introduced to evaluate the performance of the proposed method for efficient reliability assessment. The results show that the proposed method generally outperforms or performs similarly to the best one among existing methods. In addition, it is also shown that proper kernel function can significantly improve the performance of GPR as the training data increases. Stated differently, the proposed method can learn the function of being fitted efficiently with less training samples than existing methods. In this context, the proposed method can make powerful and automatic predictive modeling based on GPR in engineering problems.

Parameter identification for symbolic regression using nonlinear least squares

In this paper we analyze the effects of using nonlinear least squares for parameter identification of symbolic regression models and integrate it as local search mechanism in tree-based genetic programming. We employ the Levenberg–Marquardt algorithm for parameter optimization and calculate gradients via automatic differentiation. We provide examples where the parameter identification succeeds and fails and highlight its computational overhead. Using an extensive suite of symbolic regression benchmark problems we demonstrate the increased performance when incorporating nonlinear least squares within genetic programming. Our results are compared with recently published results obtained by several genetic programming variants and state of the art machine learning algorithms. Genetic programming with nonlinear least squares performs among the best on the defined benchmark suite and the local search can be easily integrated in different genetic programming algorithms as long as only differentiable functions are used within the models.

Data Based Prediction of Blood Glucose Concentrations Using Evolutionary Methods.

Predicting glucose values on the basis of insulin and food intakes is a difficult task that people with diabetes need to do daily. This is necessary as it is important to maintain glucose levels at appropriate values to avoid not only short-term, but also long-term complications of the illness. Artificial intelligence in general and machine learning techniques in particular have already lead to promising results in modeling and predicting glucose concentrations. In this work, several machine learning techniques are used for the modeling and prediction of glucose concentrations using as inputs the values measured by a continuous monitoring glucose system as well as also previous and estimated future carbohydrate intakes and insulin injections. In particular, we use the following four techniques: genetic programming, random forests, k-nearest neighbors, and grammatical evolution. We propose two new enhanced modeling algorithms for glucose prediction, namely (i) a variant of grammatical evolution which uses an optimized grammar, and (ii) a variant of tree-based genetic programming which uses a three-compartment model for carbohydrate and insulin dynamics. The predictors were trained and tested using data of ten patients from a public hospital in Spain. We analyze our experimental results using the Clarke error grid metric and see that 90% of the forecasts are correct (i.e., Clarke error categories A and B), but still even the best methods produce 5 to 10% of serious errors (category D) and approximately 0.5% of very serious errors (category E). We also propose an enhanced genetic programming algorithm that incorporates a three-compartment model into symbolic regression models to create smoothed time series of the original carbohydrate and insulin time series.

Recursion in tree-based genetic programming

Recursion is a powerful concept that enables a solution to a problem to be expressed as a relatively simple decomposition of the original problem into sub-problems of the same type. We survey previous research about the evolution of recursive programs in tree-based Genetic Programming. We then present an analysis of the fitness landscape of recursive programs, and report results on evolving solutions to a range of problems. We conclude with guidelines concerning the choice of fitness function and variation operators, as well as the handling of the halting problem. The main findings are as follows. The distribution of fitness changes initially as we look at programs of increasing size but once some threshold has been exceeded, it shows very little variation with size. Furthermore, the proportion of halting programs decreases as size increases. Recursive programs exhibit the property of weak causality; small changes in program structure may cause big changes in semantics. Nevertheless, the evolution of recursive programs is not a needle-in-a-haystack problem; the neighbourhoods of optimal programs are populated by halting individuals of intermediate fitness. Finally, mutation-based variation operators performed the best in finding recursive solutions. Evolution was also shown to outperform random search.

Review and comparative analysis of geometric semantic crossovers

This paper provides a structured, unified, formal and empirical perspective on all geometric semantic crossover operators proposed so far, including the exact geometric crossover by Moraglio, Krawiec, and Johnson, as well as the approximately geometric operators. We start with presenting the theory of geometric semantic genetic programming, and discuss the implications of geometric operators for the structure of fitness landscape. We prove that geometric semantic crossover can by construction produce an offspring that is not worse than the fitter parent, and that under certain conditions such an offspring is guaranteed to be not worse than the worse parent. We review all geometric semantic crossover operators presented to date in the literature, and conduct a comprehensive experimental comparison using a tree-based genetic programming framework and a representative suite of nine symbolic regression and nine Boolean function synthesis tasks. We scrutinize the performance (program error and success rate), generalization, computational cost, bloat, population diversity, and the operators’ capability to generate geometric offspring. The experiment leads to several interesting conclusions, the primary one being that an operator’s capability to produce geometric offspring is positively correlated with performance. The paper is concluded by recommendations regarding the suitability of operators for the particular domains of program induction tasks.

Improving the performance of GPU-based genetic programming through exploitation of on-chip memory

Genetic Programming (GP) (Koza, Genetic programming, MIT Press, Cambridge, 1992) is well-known as a computationally intensive technique. Subsequently, faster parallel versions have been implemented that harness the highly parallel hardware provided by graphics cards enabling significant gains in the performance of GP to be achieved. However, extracting the maximum performance from a graphics card for the purposes of GP is difficult. A key reason for this is that in addition to the processor resources, the fast on-chip memory of graphics cards needs to be fully exploited. Techniques will be presented that will improve the performance of a graphics card implementation of tree-based GP by better exploiting this faster memory. It will be demonstrated that both L1 cache and shared memory need to be considered for extracting the maximum performance. Better GP program representation and use of the register file is also explored to further boost performance. Using an NVidia Kepler 670GTX GPU, a maximum performance of 36 billion Genetic Programming Operations per Second is demonstrated.

Automatic programming via iterated local search for dynamic job shop scheduling.

Dispatching rules have been commonly used in practice for making sequencing and scheduling decisions. Due to specific characteristics of each manufacturing system, there is no universal dispatching rule that can dominate in all situations. Therefore, it is important to design specialized dispatching rules to enhance the scheduling performance for each manufacturing environment. Evolutionary computation approaches such as tree-based genetic programming (TGP) and gene expression programming (GEP) have been proposed to facilitate the design task through automatic design of dispatching rules. However, these methods are still limited by their high computational cost and low exploitation ability. To overcome this problem, we develop a new approach to automatic programming via iterated local search (APRILS) for dynamic job shop scheduling. The key idea of APRILS is to perform multiple local searches started with programs modified from the best obtained programs so far. The experiments show that APRILS outperforms TGP and GEP in most simulation scenarios in terms of effectiveness and efficiency. The analysis also shows that programs generated by APRILS are more compact than those obtained by genetic programming. An investigation of the behavior of APRILS suggests that the good performance of APRILS comes from the balance between exploration and exploitation in its search mechanism.

A new genetic programming framework based on reaction systems

This paper presents a new genetic programming framework called Evolutionary Reaction Systems. It is based on a recently defined computational formalism, inspired by chemical reactions, called Reaction Systems, and it has several properties that distinguish it from other existing genetic programming frameworks, making it interesting and worthy of investigation. For instance, it allows us to express complex constructs in a simple and intuitive way, and it lightens the final user from the task of defining the set of primitive functions used to build up the evolved programs. Given that Evolutionary Reaction Systems is new and it has small similarities with other existing genetic programming frameworks, a first phase of this work is dedicated to a study of some important parameters and their influence on the algorithm’s performance. Successively, we use the best parameter setting found to compare Evolutionary Reaction Systems with other well established machine learning methods, including standard tree-based genetic programming. The presented results show that Evolutionary Reaction Systems are competitive with, and in some cases even better than, the other studied methods on a wide set of benchmarks.

Tree-based genetic programming approach to infer microphysical parameters of the DSDs from the polarization diversity measurements

The use of polarization diversity measurements to infer the microphysical parametrization has remained an active goal in the radar remote sensing community. In view of this, the tree-based genetic programming (GP) as a novel approach has been presented for retrieving the governing microphysical parameters of a normalized gamma drop size distribution model D0 (median drop diameter), Nw (concentration parameter), and μ (shape parameter) from the polarization diversity measurements. A large number of raindrop spectra acquired from a Joss–Waldvogel disdrometer has been utilized to develop the GP models, relating the microphysical parameters to the T-matrix scattering simulated polarization measurements. Several functional formulations retrieving the microphysical parameters-D0 [f(ZDR), f(ZH, ZDR)], log10Nw [f(ZH, D0), f(ZH, ZDR, D0), and μ[f(ZDR, D0), f(ZH, ZDR, D0)], where ZH represents reflectivity and ZDR represents differential reflectivity, have been investigated, and applied to a S-band polarimetric radar (CAMRA) for evaluation. It has been shown that the GP model retrieved microphysical parameters from the polarization measurements are in a reasonable agreement with disdrometer observations. The calculated root mean squared errors (RMSE) are noted as 0.23–0.25mm for D0, 0.74–0.85 for log10Nw (Nw in mm−1 mm−3), and 3.30–3.36 for μ. The GP model based microphysical retrieval procedure is further compared with a physically based constrained gamma model for D0 and log10Nw estimates. The close agreement of the retrieval results between the GP and the constrained gamma models supports the suitability of the proposed genetic programming approach to infer microphysical parameterization.

Evolving Distributed Algorithms With Genetic Programming

In this paper, we evaluate the applicability of genetic programming (GP) for the evolution of distributed algorithms. We carry out a large-scale experimental study in which we tackle three well-known problems from distributed computing with six different program representations. For this purpose, we first define a simulation environment in which phenomena such as asynchronous computation at changing speed and messages taking over each other, i.e., out-of-order message delivery, occur with high probability. Second, we define extensions and adaptations of established GP approaches (such as tree-based and linear GP) in order to make them suitable for representing distributed algorithms. Third, we introduce novel rule-based GP methods designed especially with the characteristic difficulties of evolving algorithms (such as epistasis) in mind. Based on our extensive experimental study of these approaches, we conclude that GP is indeed a viable method for evolving non-trivial, deterministic, non-approximative distributed algorithms. Furthermore, one of the two rule-based approaches is shown to exhibit superior performance in most of the tasks and thus can be considered as an interesting idea also for other problem domains.

Stock trading strategy creation using GP on GPU

This paper investigates the speed improvements available when using a graphics processing unit (GPU) for evaluation of individuals in a genetic programming (GP) environment. An existing GP system is modified to enable parallel evaluation of individuals on a GPU device. Several issues related to implementing GP on GPU are discussed, including how to perform tree-based GP on a device without recursion support, as well as the effect that proper memory layout can have on speed increases when using CUDA-enabled nVidia GPU devices. The specific GP implementation is designed to evolve stock trading strategies using technical analysis indicators. The second goal of this research is to investigate the possible improvement in performance when training individuals on a larger number of stocks and training days. This increased training size (nearly 100,000 training points) is enabled due to the speedups realized by GPU evaluation. Several different scenarios were used to test various speed optimizations of GP evaluation on the GPU device, with a peak speedup factor of over 600 (when compared to sequential evaluation on a 2.4 GHz CPU). Also, it is found that increasing the number of stocks and the length of the training period can result in higher out-of-training testing profitability.