Bloat Control Research Articles

SEMANTIC APPROXIMATION BASED OPERATOR FOR REDUCING CODE BLOAT IN GENETIC PROGRAMMING

In Genetic Programming, code bloat is a well-known problem that is the increase in the average program size without a corresponding improvement in fitness. In order to address this problem, we proposed a new operator called Prune and Plant based on Approximate Terminal (shortened as PP-AT). PP-AT aims at reducing GP code bloat. It replaces a random subtree in a parent by an approximate tree of semantics to obtain the first offspring. This subtree is also added to the population as the second offspring. PP-AT is tested on fifteen regression problems and compared to standard GP and three recent bloat control methods. The experimental results demonstrate that PP-AT outperforms standard GP and other bloat control methods under comparison.

Simplification of genetic programs: a literature survey

Genetic programming (GP), a widely used evolutionary computing technique, suffers from bloat—the problem of excessive growth in individuals’ sizes. As a result, its ability to efficiently explore complex search spaces reduces. The resulting solutions are less robust and generalisable. Moreover, it is difficult to understand and explain models which contain bloat. This phenomenon is well researched, primarily from the angle of controlling bloat: instead, our focus in this paper is to review the literature from an explainability point of view, by looking at how simplification can make GP models more explainable by reducing their sizes. Simplification is a code editing technique whose primary purpose is to make GP models more explainable. However, it can offer bloat control as an additional benefit when implemented and applied with caution. Researchers have proposed several simplification techniques and adopted various strategies to implement them. We organise the literature along multiple axes to identify the relative strengths and weaknesses of simplification techniques and to identify emerging trends and areas for future exploration. We highlight design and integration challenges and propose several avenues for research. One of them is to consider simplification as a standalone operator, rather than an extension of the standard crossover or mutation operators. Its role is then more clearly complementary to other GP operators, and it can be integrated as an optional feature into an existing GP setup. Another proposed avenue is to explore the lack of utilisation of complexity measures in simplification. So far, size is the most discussed measure, with only two pieces of prior work pointing out the benefits of using time as a measure when controlling bloat.

Metameric representations on optimization of nano particle cancer treatment

In silico evolutionary optimization of cancer treatment based on multiple nano-particle (NP) assisted drug delivery systems was investigated in this study. The use of multiple types of NPs is expected to increase the robustness of the treatment, due to imposing higher complexity on the solution tackling a problem of high complexity, namely the physiology of a tumor. Thus, the utilization of metameric representations in the evolutionary optimization method was examined, along with suitable crossover and mutation operators. An open-source physics-based simulator was utilized, namely PhysiCell, after appropriate modifications, to test the fitness of possible treatments with multiple types of NPs. The possible treatments could be comprised of up to ten types of NPs, simultaneously injected in an area close to the cancerous tumour. Initial results seem to suffer from bloat, namely the best solutions discovered are converging towards the maximum amount of different types of NPs, however, without providing a significant return in fitness when compared with solutions of fewer types of NPs. As the large diversity of NPs will most probably prove to be quite toxic in lab experiments, we opted for methods to reduce the bloat, thus, resolve to therapies with fewer types of NPs. Namely, the bloat control methods studied here were removing types of NPs from the optimization genome as part of the mutation operator and applying parsimony pressure in the replacement operator. By utilizing these techniques, the treatments discovered are composed of fewer types of NPs, while their fitness is not significantly smaller.

Evolving simple and accurate symbolic regression models via asynchronous parallel computing

In machine learning, reducing the complexity of a model can help to improve its computational efficiency and avoid overfitting. In genetic programming (GP), the model complexity reduction is often achieved by reducing the size of evolved expressions. However, previous studies have demonstrated that the expression size reduction does not necessarily prevent model overfitting. Therefore, this paper uses the evaluation time – the computational time required to evaluate a GP model on data – as the estimate of model complexity. The evaluation time depends not only on the size of evolved expressions but also their composition, thus acting as a more nuanced measure of model complexity than the expression size alone. To discourage complexity, this study employs a novel method called asynchronous parallel GP (APGP) that introduces a race condition in the evolutionary process of GP; the race offers an evolutionary advantage to the simple solutions when their accuracy is competitive. To evaluate the proposed method, it is compared to the standard GP (GP) and GP with bloat control (GP+BC) methods on six challenging symbolic regression problems. APGP produced models that are significantly more accurate (on 6/6 problems) than those produced by both GP and GP+BC. In terms of complexity control, APGP prevailed over GP but not over GP+BC; however, GP+BC produced simpler solutions at the cost of test-set accuracy. Moreover, APGP took a significantly lower number of evaluations than both GP and GP+BC to meet a target training fitness in all tests. Our analysis of the proposed APGP also involved: (1) an ablation study that separated the proposed measure of complexity from the race condition in APGP and (2) the study of an initialisation scheme that encourages functional diversity in the initial population that improved the results for all the GP methods. These results question the overall benefits of bloat control and endorse the employment of both the evaluation time as an estimate of model complexity and the proposed APGP method for controlling it.

Semantic approximation for reducing code bloat in Genetic Programming

Abstract Code bloat is a phenomenon in Genetic Programming (GP) characterized by the increase in individual size during the evolutionary process without a corresponding improvement in fitness. Bloat negatively affects GP performance, since large individuals are more time consuming to evaluate and harder to interpret. In this paper, we propose two approaches for reducing GP code bloat based on a semantic approximation technique. The first approach replaces a random subtree in an individual by a smaller tree of approximate semantics. The second approach replaces a random subtree by a smaller tree that is semantically approximate to the desired semantics. We evaluated the proposed methods on a large number of regression problems. The experimental results showed that our methods help to significantly reduce code bloat and improve the performance of GP compared to standard GP and some recent bloat control methods in GP. Furthermore, the performance of the proposed approaches is competitive with the best machine learning technique among the four tested machine learning algorithms.

Grammatically uniform population initialization for grammar-guided genetic programming

The initial population distribution is an essential issue in evolutionary computation performance. Population initialization methods for grammar-guided genetic programming have some difficulties generating a representative sample of the search space, which negatively affects the overall evolutionary process. This paper presents a grammatically uniform population initialization method to address this issue by improving the initial population uniformity: the equiprobability of obtaining any individual of the search space defined by the context-free grammar. The proposed initialization method assigns and updates probabilities dynamically to the production rules of the grammar to pursue uniformity and includes a code bloat control mechanism. We have conducted empirical experiments to compare the proposed algorithm with a standard initialization approach very often used in grammar-guided genetic programming. The results report that the proposed initialization method approximates very well a uniform distribution of the individuals in the search space. Moreover, the overall evolutionary process that takes place after the population initialization performs better in terms of convergence speed and quality of the final solutions achieved when the proposed method generates the initial population than when the usual approach does. The results also show that these performance differences are more significant when the experiments involve large search spaces.

Preference-driven Pareto front exploitation for bloat control in genetic programming

As one of evolutionary algorithms (EAs), genetic programming (GP) has been applied in a wide range of areas, e.g. bioinformatics and robotics. Different from other EAs, GP can represent problems with variable length (e.g. trees), which makes it more flexible in evolving solutions, yet leads to a serious problem, bloat. It can cause evolving redundant parts and slowing down search. Multi-objective techniques are popularly used for reducing bloat in GP (termed as MOGP). Specifically, MOGP methods evolve trade-off solutions of all objectives, which constitute the so-called Pareto front. Then users select solutions on the front based on their preference for specific tasks. However, existing MOGP methods rarely consider users’ preference during evolution, which wastes computation power and time to search for useless solutions and cannot generate fine-grained interested regions on the Pareto front. Therefore, this paper investigates introducing users’ preference to guide multi-objective techniques to focus on the interested regions on Pareto front during evolution. Specifically, Pareto dominance is an important notion in multi-objective techniques for comparing two solutions. We design two preference-driven Pareto dominance mechanisms, scPd (static constraint Pareto dominance) and dcPd (dynamic constraint Pareto dominance), which are introduced in a base multi-objective technique and then are incorporated with GP respectively to form two new bloat control MOGP methods, i.e. scPd_MOGP and dcPd_MOGP. They are tested on benchmark symbolic regression tasks comparing with GP, two existing bloat control methods (i.e. a parsimony GP method (pGP) and a standard multi-objective GP method (sMOGP)), and four popularly-used symbolic regression methods. Results show that the proposed methods can reduce bloat in GP and outperform pGP in bloat control, and comparison with sMOGP shows that they can search front regions based on users’ preference where the solutions have better functionality, yet relatively larger sizes. In addition, compared with four popularly-used symbolic regression methods, scPd_MOGP is generally better; while dcPd_MOGP achieves varied results, yet it performs better or similar to the reference methods on the majority of the given test functions. Moreover, comparison between the two proposed methods suggests that the constraint in the Pareto dominance of scPd_MOGP is more relaxed than that of dcPd_MOGP.

The impact of lexicographic parsimony pressure for ORDER/MAJORITY on the run time

While many optimization problems work with a fixed number of decision variables and thus a fixed-length representation of possible solutions, genetic programming (GP) works on variable-length representations. A naturally occurring problem is that of bloat, that is, the unnecessary growth of solution lengths, which may slow down the optimization process. So far, the mathematical runtime analysis could not deal well with bloat and required explicit assumptions limiting bloat.In this paper, we provide the first mathematical runtime analysis of a GP algorithm that does not require any assumptions on the bloat. Previous performance guarantees were only proven conditionally for runs in which no strong bloat occurs. Together with improved analyses for the case with bloat restrictions our results show that such assumptions on the bloat are not necessary and that the algorithm is efficient without explicit bloat control mechanism.More specifically, we analyzed the performance of the (1+1) GP on the two benchmark functions Order and Majority. When using lexicographic parsimony pressure as bloat control, we show a tight runtime estimate of O(Tinit+nlog⁡n) iterations both for Order and Majority. For the case without bloat control, the bounds O(Tinitlog⁡Tinit+n(log⁡n)3) and Ω(Tinit+nlog⁡n) (and Ω(Tinitlog⁡Tinit) for n=1) hold for Majority.1

Time and Individual Duration in Genetic Programming

This paper presents a new way of measuring complexity in variable-size-chromosome-based evolutionary algorithms. Dealing with complexity is particularly useful when considering bloat in Genetic Programming. Instead of analyzing size growth, we focus on the time required for individuals’ fitness evaluations, which correlates with size. This way, we consider time and space as two sides of a single coin when devising a more natural method for fighting bloat. We thus view the problem from a perspective that departs from traditional methods applied in Genetic Programming. We have analyzed first the behavior of individuals across generations, taking into account their fitness evaluation times, thus providing clues about the general practice of the evolutionary process when modern parallel and distributed computers are used to run the algorithm. This new perspective allows us to understand that new methods for bloat control can be derived. Moreover, we develop from this framework a first proposal to show the usefulness of the idea: to group individuals in classes according to computing time required for evaluation, automatically accomplished by parallel and distributed systems without any change in the underlying algorithm, when they are only allowed to breed within their classes. Experimental data confirms the strength of the approach: using computing time as a measure of individuals’ complexity allows control of the natural size growth of genetic programming individuals while preserving the quality of solutions in both the parallel and sequential versions of the algorithm.

Genetic programming based feature construction methods for foreground object segmentation

Foreground object segmentation is a crucial preprocessing step for many high-level computer vision tasks, e.g. object recognition. It is still challenging to achieve accurate segmentation, especially for complex images (e.g. with high variations). Feature construction can help to improve the segmentation performance by extracting more distinctive features for foreground/background regions from the original features. However, commonly-used feature construction methods (e.g. principle component analysis) often involve certain assumptions/constraints, and the constructed features cannot be interpreted. To address these problems, genetic programming (GP) is employed in this paper, which is a well-suited feature construction technique.The aim of this work is to design new feature construction methods using GP, and analyse/compare popular GP-based feature construction methods for foreground object segmentation, especially on complex image datasets with high variations. Specifically, one new feature construction method that incorporates the subtree technique in GP is designed, which can construct multiple features simultaneously (called SubtMFC, Subtree Multiple Feature Construction). Moreover, a parsimony pressure technique is introduced to improve SubtMFC for bloat control (a common issue for GP-based methods), which forms the method, PSubtMFC (Parsimony SubtMFC). In addition, comparison of popular GP-based feature construction methods for foreground object segmentation is conducted for the first time. Results show that SubtMFC achieves better or similar performance compared with three reference methods. In addition, compared with SubtMFC that does not control bloat, PSubtMFC can significantly reduce the solution size while maintain similar performance in the segmentation accuracy. The GP-based feature construction framework is further extended for feature representation based knowledge transfer, which can handle the problem of the scare labelled training data. Moreover, after GP is thoroughly investigated on benchmark datasets with one type of foreground objects (i.e. the Weizmann horse dataset and Pascal aeroplane dataset), it is considered whether the GP methods can perform well on datasets containing multiple types of foreground objects. Compared with three other well-performing GP-based feature construction methods, the proposed method achieves better or comparable results for the given segmentation tasks. In addition, this paper thoroughly compares/analyses popular GP-based feature construction methods for complex figure-ground segmentation for the first time. Moreover, further analyses on the input features frequently used by the GP-evolved feature construction functions reflect the effectiveness of the extracted high-level features.

Destructiveness of lexicographic parsimony pressure and alleviation by a concatenation crossover in genetic programming

For theoretical analyses there are two specifics distinguishing GP from many other areas of evolutionary computation: the variable size representations, in particular yielding a possible bloat (i.e. the growth of individuals with redundant parts); and also the role and the realization of crossover, which is particularly central in GP due to the tree-based representation. Whereas some theoretical work on GP has studied the effects of bloat, crossover had surprisingly little share in this work.We analyze a simple crossover operator in combination with randomized local search, where a preference for small solutions minimizes bloat (lexicographic parsimony pressure); we denote the resulting algorithm Concatenation Crossover GP. We consider three variants of the well-studied Majority test function, adding large plateaus in different ways to the fitness landscape and thus giving a test bed for analyzing the interplay of variation operators and bloat control mechanisms in a setting with local optima. We show that the Concatenation Crossover GP can efficiently optimize these test functions, while local search cannot be efficient for all three variants independent of employing bloat control.

Pool-Based Genetic Programming Using Evospace, Local Search and Bloat Control

This work presents a unique genetic programming (GP) approach that integrates a numerical local search method and a bloat-control mechanism within a distributed model for evolutionary algorithms known as EvoSpace. The first two elements provide a directed search operator and a way to control the growth of evolved models, while the latter is meant to exploit distributed and cloud-based computing architectures. EvoSpace is a Pool-based Evolutionary Algorithm, and this work is the first time that such a computing model has been used to perform a GP-based search. The proposal was extensively evaluated using real-world problems from diverse domains, and the behavior of the search was analyzed from several different perspectives. The results show that the proposed approach compares favorably with a standard approach, identifying promising aspects and limitations of this initial hybrid system.

Local search in speciation-based bloat control for genetic programming

This work presents a unique genetic programming (GP) approach that integrates a numerical local search method and a bloat-control mechanism to address some of the main issues with traditional GP. The former provides a directed search operator to work in conjunction with standard syntax operators that perform more exploration in design space, while the latter controls code growth by maintaining program diversity through speciation. The system can produce highly parsimonious solutions, thus reducing the cost of performing the local optimization process. The proposal is extensively evaluated using real-world problems from diverse domains, and the behavior of the search is analyzed from several different perspectives, including how species evolve, the effect of the local search process and the interpretability of the results. Results show that the proposed approach compares favorably with a standard approach, and that the hybrid algorithm can be used as a viable alternative for solving real-world symbolic regression problems.

Semantic tournament selection for genetic programming based on statistical analysis of error vectors

The selection mechanism plays a very important role in the performance of Genetic Programming (GP). Among several selection techniques, tournament selection is often considered the most popular. Standard tournament selection randomly selects a set of individuals from the population and the individual with the best fitness value is chosen as the winner. However, an opportunity exists to enhance tournament selection as the standard approach ignores finer-grained semantics which can be collected during GP program execution. In the case of symbolic regression problems, the error vectors on the training fitness cases can be used in a more detailed quantitative comparison. In this paper we introduce the use of a statistical test into GP tournament selection that utilizes information from the individual’s error vector, and three variants of the selection strategy are proposed. We tested these methods on twenty five regression problems and their noisy variants. The experimental results demonstrate the benefit of the proposed methods in reducing GP code growth and improving the generalisation behaviour of GP solutions when compared to standard tournament selection, a similar selection technique and a state of the art bloat control approach.

Evolving genetic programming classifiers with novelty search

Novelty Search (NS) is a unique approach towards search and optimization, where an explicit objective function is replaced by a measure of solution novelty. However, NS has been mostly used in evolutionary robotics while its usefulness in classic machine learning problems has not been explored. This work presents a NS-based genetic programming (GP) algorithm for supervised classification. Results show that NS can solve real-world classification tasks, the algorithm is validated on real-world benchmarks for binary and multiclass problems. These results are made possible by using a domain-specific behavior descriptor. Moreover, two new versions of the NS algorithm are proposed, Probabilistic NS (PNS) and a variant of Minimal Criteria NS (MCNS). The former models the behavior of each solution as a random vector and eliminates all of the original NS parameters while reducing the computational overhead of the NS algorithm. The latter uses a standard objective function to constrain and bias the search towards high performance solutions. The paper also discusses the effects of NS on GP search dynamics and code growth. Results show that NS can be used as a realistic alternative for supervised classification, and specifically for binary problems the NS algorithm exhibits an implicit bloat control ability.

Studying bloat control and maintenance of effective code in linear genetic programming for symbolic regression

Linear Genetic Programming (LGP) is an Evolutionary Computation algorithm, inspired in the Genetic Programming (GP) algorithm. Instead of using the standard tree representation of GP, LGP evolves a linear program, which causes a graph-based data flow with code reuse. LGP has been shown to outperform GP in several problems, including Symbolic Regression (SReg), and to produce simpler solutions. In this paper, we propose several LGP variants and compare them with a traditional LGP algorithm on a set of benchmark SReg functions from the literature. The main objectives of the variants were to both control bloat and privilege useful code in the population. Here we evaluate their effects during the evolution process and in the quality of the final solutions. Analysis of the results showed that bloat control and effective code maintenance worked, but they did not guarantee improvement in solution quality.

Neat Genetic Programming: Controlling bloat naturally

Bloat is one of the most widely studied phenomena in Genetic Programming (GP), it is normally defined as the increase in mean program size without a corresponding improvement in fitness. Several theories have been proposed in the specialized GP literature that explain why bloat occurs. In particular, the Crossover-Bias Theory states that the cause of bloat is that the distribution of program sizes during evolution is skewed in a way that encourages bloat to appear, by punishing small individuals and favoring larger ones. Therefore, several bloat control methods have been proposed that attempt to explicitly control the size distribution of programs within the evolving population. This work proposes a new bloat control method called neat-GP, that implicitly shapes the program size distribution during a GP run. neat-GP is based on two key elements: (a) the NeuroEvolution of Augmenting Topologies algorithm (NEAT), a robust heuristic that was originally developed to evolve neural networks; and (b) the Flat Operator Equalization bloat control method, that explicitly shapes the program size distributions toward a uniform or flat shape. Experimental results are encouraging in two domains, symbolic regression and classification of real-world data. neat-GP can curtail the effects of bloat without sacrificing performance, outperforming both standard GP and the Flat-OE method, without incurring in the computational overhead reported by some state-of-the-art bloat control methods.

Improving Generalization Ability of Genetic Programming: Comparative Study

In the field of empirical modeling using Genetic Programming (GP), it is important to evolve solution with good generalization ability. Generalization ability of GP solutions get affected by two important issues: bloat and over-fitting. Bloat is uncontrolled growth of code without any gain in fitness and important issue in GP. We surveyed and classified existing literature related to different techniques used by GP research community to deal with the issue of bloat. Moreover, the classifications of different bloat control approaches and measures for bloat are discussed. Next, we tested four bloat control methods: Tarpeian, double tournament, lexicographic parsimony pressure with direct bucketing and ratio bucketing on six different problems and identified where each bloat control method performs well on per problem basis. Based on the analysis of each method, we combined two methods: double tournament (selection method) and Tarpeian method (works before evaluation) to avoid bloated solutions and compared with the results obtained from individual performance of double tournament method. It was found that the results were improved with this combination of two methods.

GP-induced and explicit bloating of the seeds in incremental GP improves evolutionary success

The parsimony control in genetic programming (GP) is one of the limiting factors in the quick evolution of efficient solutions. A variety of parsimony pressure methods have been developed to address this issue. The effects of these methods on the efficiency of evolution are recognized to depend on the characteristics of the applied problem domain. On the other hand, the implications of using parsimony pressure in evolving the seeds for incremental genetic programming (IGP) are still poorly known and remain uninvestigated. In this work we present a study on the cumulative effect of the bloat and the seeding of the initial population on the efficiency of incremental evolution of simulated snake-like robot (Snakebot). In the proposed IGP, the task of coevolving the locomotion gaits and sensing of the bot in a challenging environment is decomposed into two sub-tasks, implemented as two consecutive evolutionary stages. First, to evolve the pools of sensorless Snakebots, we use GP featuring the following three bloat-control methods: (1) linear parametric parsimony pressure, (2) lexicographic parsimony pressure and (3) no bloat control. During the second stage of IGP, we use these pools to seed the initial population of Snakebots applying two methods of seeding: canonical seeding and seeding inspired by genetic transposition (GT).

Bloat free genetic programming: application to human oral bioavailability prediction

Being able to predict the human oral bioavailability for a potential new drug is extremely important for the drug discovery process. This problem has been addressed by several prediction tools, with Genetic Programming providing some of the best results ever achieved. In this paper we use the newest developments of Genetic Programming, in particular the latest bloat control method, Operator Equalisation, to find out how much improvement we can achieve on this problem. We show examples of some actual solutions and discuss their quality, comparing them with previously published results. We identify some unexpected behaviours related to overfitting, and discuss the way for further improving the practical usage of the Genetic Programming approach.