Grammar-guided Genetic Programming Research Articles

Enhancing Program Synthesis with Large Language Models Using Many-Objective Grammar-Guided Genetic Programming

The ability to automatically generate code, i.e., program synthesis, is one of the most important applications of artificial intelligence (AI). Currently, two AI techniques are leading the way: large language models (LLMs) and genetic programming (GP) methods—each with its strengths and weaknesses. While LLMs have shown success in program synthesis from a task description, they often struggle to generate the correct code due to ambiguity in task specifications, complex programming syntax, and lack of reliability in the generated code. Furthermore, their generative nature limits their ability to fix erroneous code with iterative LLM prompting. Grammar-guided genetic programming (G3P, i.e., one of the top GP methods) has been shown capable of evolving programs that fit a defined Backus–Naur-form (BNF) grammar based on a set of input/output tests that help guide the search process while ensuring that the generated code does not include calls to untrustworthy libraries or poorly structured snippets. However, G3P still faces issues generating code for complex tasks. A recent study attempting to combine both approaches (G3P and LLMs) by seeding an LLM-generated program into the initial population of the G3P has shown promising results. However, the approach rapidly loses the seeded information over the evolutionary process, which hinders its performance. In this work, we propose combining an LLM (specifically ChatGPT) with a many-objective G3P (MaOG3P) framework in two parts: (i) provide the LLM-generated code as a seed to the evolutionary process following a grammar-mapping phase that creates an avenue for program evolution and error correction; and (ii) leverage many-objective similarity measures towards the LLM-generated code to guide the search process throughout the evolution. The idea behind using the similarity measures is that the LLM-generated code is likely to be close to the correct fitting code. Our approach compels any generated program to adhere to the BNF grammar, ultimately mitigating security risks and improving code quality. Experiments on a well-known and widely used program synthesis dataset show that our approach successfully improves the synthesis of grammar-fitting code for several tasks.

Nonlinear Model Predictive Control with Evolutionary Data-Driven Prediction Model and Particle Swarm Optimization Optimizer for an Overhead Crane

This paper presents a new approach to the nonlinear model predictive control (NMPC) of an underactuated overhead crane system developed using a data-driven prediction model obtained utilizing the regularized genetic programming-based symbolic regression method. Grammar-guided genetic programming combined with regularized least squares was applied to identify a nonlinear autoregressive model with an exogenous input (NARX) prediction model of the crane dynamics from input–output data. The resulting prediction model was implemented in the NMPC scheme, using a particle swarm optimization (PSO) algorithm as a solver to find an optimal sequence of the control actions satisfying multi-objective performance requirements and input constraints. The feasibility and performance of the controller were experimentally verified using a laboratory crane actuated by AC motors and compared with a discrete-time feedback controller developed using the pole placement technique. A series of experiments proved the effectiveness of the controller in terms of robustness against operating condition variation and external disturbances.

Data-Driven Identification of Crane Dynamics Using Regularized Genetic Programming

The meaningful problem of improving crane safety, reliability, and efficiency is extensively studied in the literature and targeted via various model-based control approaches. In recent years, crane data-driven modeling has attracted much attention compared to physics-based models, particularly due to its potential in real-time crane control applications, specifically in model predictive control. This paper proposes grammar-guided genetic programming with sparse regression (G3P-SR) to identify the nonlinear dynamics of an underactuated crane system. G3P-SR uses grammars to bias the search space and produces a fixed number of candidate model terms, while a local search method based on an l0-regularized regression results in a sparse solution, thereby also reducing model complexity as well as reducing the probability of overfitting. Identification is performed on experimental data obtained from a laboratory-scale overhead crane. The proposed method is compared with multi-gene genetic programming (MGGP), NARX neural network, and Takagi-Sugeno fuzzy (TSF) ARX models in terms of model complexity, prediction accuracy, and sensitivity. The G3P-SR algorithm evolved a model with a maximum mean square error (MSE) of crane velocity and sway prediction of 1.1860 × 10−4 and 4.8531 × 10−4, respectively, in simulations for different testing data sets, showing better accuracy than the TSF ARX and MGGP models. Only the NARX neural network model with velocity and sway maximum MSEs of 1.4595 × 10−4 and 4.8571 × 10−4 achieves a similar accuracy or an even better one in some testing scenarios, but at the cost of increasing the total number of parameters to be estimated by over 300% and the number of output lags compared to the G3P-SR model. Moreover, the G3P-SR model is proven to be notably less sensitive, exhibiting the least deviation from the nominal trajectory for deviations in the payload mass by approximately a factor of 10.

Estimation of Distribution Algorithm for Grammar-Guided Genetic Programming.

Genetic variation operators in grammar-guided genetic programming are fundamental to guide the evolutionary process in search and optimization problems. However, they show some limitations, mainly derived from an unbalanced exploration and local-search trade-off. This article presents an estimation of distribution algorithm for grammar-guided genetic programming to overcome this difficulty and thus increase the performance of the evolutionary algorithm. Our proposal employs an extended dynamic stochastic context-free grammar to encode and calculate the estimation of the distribution of the search space from some promising individuals in the population. Unlike traditional estimation of distribution algorithms, the proposed approach improves exploratory behavior by smoothing the estimated distribution model. Therefore, this algorithm is referred to as SEDA, smoothed estimation of distribution algorithm. Experiments have been conducted to compare overall performance using a typical genetic programming crossover operator, an incremental estimation of distribution algorithm, and the proposed approach after tuning their hyperparameters. These experiments involve challenging problems to test the local search and exploration features of the three evolutionary systems. The results show that grammar-guided genetic programming with SEDA achieves the most accurate solutions with an intermediate convergence speed.

Informed Down-Sampled Lexicase Selection: Identifying productive training cases for efficient problem solving.

Genetic Programming (GP) often uses large training sets and requires all individuals to be evaluated on all training cases during selection. Random down-sampled lexicase selection evaluates individuals on only a random subset of the training cases allowing for more individuals to be explored with the same amount of program executions. However, sampling randomly can exclude important cases from the down-sample for a number of generations, while cases that measure the same behavior (synonymous cases) may be overused. In this work, we introduce Informed Down-Sampled Lexicase Selection. This method leverages population statistics to build down-samples that contain more distinct and therefore informative training cases. Through an empirical investigation across two different GP systems (PushGP and Grammar-Guided GP), we find that informed down-sampling significantly outperforms random down-sampling on a set of contemporary program synthesis benchmark problems. Through an analysis of the created down-samples, we find that important training cases are included in the down-sample consistently across independent evolutionary runs and systems. We hypothesize that this improvement can be attributed to the ability of Informed Down-Sampled Lexicase Selection to maintain more specialist individuals over the course of evolution, while still benefiting from reduced per-evaluation costs.

Soft-Computing-Based Estimation of a Static Load for an Overhead Crane.

Payload weight detection plays an important role in condition monitoring and automation of cranes. Crane cells and scales are commonly used in industrial practice; however, when their installation to the hoisting equipment is not possible or costly, an alternative solution is to derive information about the load weight indirectly from other sensors. In this paper, a static payload weight is estimated based on the local strain of a crane's girder and the current position of the trolley. Soft-computing-based techniques are used to derive a nonlinear input-output relationship between the measured signals and the estimated payload mass. Data-driven identification is performed using a novel variant of genetic programming named grammar-guided genetic programming with sparse regression, multi-gene genetic programming, and subtractive fuzzy clustering method combined with the least squares algorithm on experimental data obtained from a laboratory overhead crane. A comparative analysis of the methods showed that multi-gene genetic programming and grammar-guided genetic programming with sparse regression performed similarly in terms of accuracy and both performed better than subtractive fuzzy clustering. The novel approach was able to find a more parsimonious model with its direct implantation having a lower execution time.

Using mutual information to test from Finite State Machines: Test suite generation

Mutual Information is an information theoretic measure designed to quantify the amount of similarity between two random variables ranging over two sets. In recent work we have use it as a base for a measure, called Biased Mutual Information, to guide the selection of a test suite among different possibilities. In this paper, we adapt this concept and show how it can be used to address the problem of generating a test suite with high fault finding capability, in a black-box scenario and following a maximise diversity approach. Additionally, we present a new Grammar-Guided Genetic Programming Algorithm that uses Biased Mutual Information to guide the generation of such test suites. Our experimental results clearly show the potential value of our measure when used to generate test suites. Moreover, they show that our measure is better in guiding test generation than current state-of-the-art measures, like Test Set Diameter (TSDm) measures. Additionally, we compared our proposal with classical completeness-oriented methods, like the H-Method and the Transition Tour method, and found that our proposal produces smaller test suites with high enough fault finding capability. Therefore, our methodology is preferable in an scenario where a compromise is necessary between fault detection and execution time.

Disease-Ligand Identification Based on Flexible Neural Tree.

In order to screen the disease-related compounds of a traditional Chinese medicine prescription in network pharmacology research accurately, a new virtual screening method based on flexible neural tree (FNT) model, hybrid evolutionary method and negative sample selection algorithm is proposed. A novel hybrid evolutionary algorithm based on the Grammar-guided genetic programming and salp swarm algorithm is proposed to infer the optimal FNT. According to hypertension, diabetes, and Corona Virus Disease 2019, disease-related compounds are collected from the up-to-date literatures. The unrelated compounds are chosen by negative sample selection algorithm. ECFP6, MACCS, Macrocycle, and RDKit are utilized to numerically characterize the chemical structure of each compound collected, respectively. The experiment results show that our proposed method performs better than classical classifiers [Support Vector Machine (SVM), random forest (RF), AdaBoost, decision tree (DT), Gradient Boosting Decision Tree (GBDT), KNN, logic regression (LR), and Naive Bayes (NB)], up-to-date classifier (gcForest), and deep learning method (forgeNet) in terms of AUC, ROC, TPR, FPR, Precision, Specificity, and F1. MACCS method is suitable for the maximum number of classifiers. All methods perform poorly with ECFP6 molecular descriptor.

EvoStencils: a grammar-based genetic programming approach for constructing efficient geometric multigrid methods

For many systems of linear equations that arise from the discretization of partial differential equations, the construction of an efficient multigrid solver is challenging. Here we present EvoStencils, a novel approach for optimizing geometric multigrid methods with grammar-guided genetic programming, a stochastic program optimization technique inspired by the principle of natural evolution. A multigrid solver is represented as a tree of mathematical expressions that we generate based on a formal grammar. The quality of each solver is evaluated in terms of convergence and compute performance by automatically generating an optimized implementation using code generation that is then executed on the target platform to measure all relevant performance metrics. Based on this, a multi-objective optimization is performed using a non-dominated sorting-based selection. To evaluate a large number of solvers in parallel, they are distributed to multiple compute nodes. We demonstrate the effectiveness of our implementation by constructing geometric multigrid solvers that are able to outperform hand-crafted methods for Poisson’s equation and a linear elastic boundary value problem with up to 16 million unknowns on multi-core processors with Ivy Bridge and Broadwell microarchitecture.

Auto-adaptive Grammar-Guided Genetic Programming algorithm to build Ensembles of Multi-Label Classifiers

Multi-label classification has been used to solve a wide range of problems where each example in the dataset may be related either to one class (as in traditional classification problems) or to several class labels at the same time. Many ensemble-based approaches have been proposed in the literature, aiming to improve the performance of traditional multi-label classification algorithms. However, most of them do not consider the data characteristics to build the ensemble, and those that consider them need to tune many parameters to maximize their performance.In this paper, we propose an Auto-adaptive algorithm based on Grammar-Guided Genetic Programming to generate Ensembles of Multi-Label Classifiers based on projections of k labels (AG3P-kEMLC). It creates a tree-shaped ensemble, where each leaf is a multi-label classifier focused on a subset of k labels. Unlike other methods in the literature, our proposal can deal with different values of k in the same ensemble, instead of fixing one specific value. It also includes an auto-adaptive process to reduce the number of hyper-parameters to tune, prevent overfitting and reduce the runtime required to execute it. Three versions of the algorithm are proposed. The first, fixed, uses the same value of k for all multi-label classifiers in the ensemble. The remaining two deal with different k values in the ensemble: uniform gives the same probability to choose each available value of k, and gaussian favors the selection of smaller values of k.The experimental study carried out considering twenty reference datasets and five evaluation metrics, compared with eleven ensemble methods demonstrates that our proposal performs significantly better than the state-of-the-art methods.

Automatic generation of algorithms for robust optimisation problems using Grammar-Guided Genetic Programming

We develop algorithms capable of tackling robust black-box optimisation problems, where the number of model runs is limited. When a desired solution cannot be implemented exactly the aim is to find a robust one, where the worst case in an uncertainty neighbourhood around a solution still performs well. To investigate improved methods we employ an automatic generation of algorithms approach: Grammar-Guided Genetic Programming. We develop algorithmic building blocks in a Particle Swarm Optimisation framework, define the rules for constructing heuristics from these components, and evolve populations of search algorithms for robust problems. Our algorithmic building blocks combine elements of existing techniques and new features, resulting in the investigation of a novel heuristic solution space. We obtain algorithms which improve upon the current state of the art. We also analyse the component level breakdowns of the populations of algorithms developed against their performance, to identify high-performing heuristic components for robust problems.

Reverse engineering gene regulatory network based on complex-valued ordinary differential equation model

BackgroundThe growing researches of molecular biology reveal that complex life phenomena have the ability to demonstrating various types of interactions in the level of genomics. To establish the interactions between genes or proteins and understand the intrinsic mechanisms of biological systems have become an urgent need and study hotspot.ResultsIn order to forecast gene expression data and identify more accurate gene regulatory network, complex-valued version of ordinary differential equation (CVODE) is proposed in this paper. In order to optimize CVODE model, a complex-valued hybrid evolutionary method based on Grammar-guided genetic programming and complex-valued firefly algorithm is presented.ConclusionsWhen tested on three real gene expression datasets from E.coli and Human Cell, the experiment results suggest that CVODE model could improve 20–50% prediction accuracy of gene expression data, which could also infer more true-positive regulatory relationships and less false-positive regulations than ordinary differential equation.

A propositionalization method of multi-relational data based on Grammar-Guided Genetic Programming

The propositionalization process tries to find distinctive features of the examples in a database to transform such relational data into a simpler representation. More informative features have a positive impact on the classification capabilities of the learning algorithms. In this work, we propose a new propositionalization method, which generates complex Boolean attributes using Grammar-Guided Genetic Programming (G3P). The generated attributes are compound formulas that combine word items coming from a Bag-of-Words (BoW) representation using Boolean operators. The proposal was assessed against three state-of-the-art simple-instance and multiple-instance propositionalization methods. The experimental results show that the proposed method achieves an improvement in terms of classification accuracy and a considerable reduction in the dimensionality of the resulting datasets.

Grammatically uniform population initialization for grammar-guided genetic programming

The initial population distribution is an essential issue in evolutionary computation performance. Population initialization methods for grammar-guided genetic programming have some difficulties generating a representative sample of the search space, which negatively affects the overall evolutionary process. This paper presents a grammatically uniform population initialization method to address this issue by improving the initial population uniformity: the equiprobability of obtaining any individual of the search space defined by the context-free grammar. The proposed initialization method assigns and updates probabilities dynamically to the production rules of the grammar to pursue uniformity and includes a code bloat control mechanism. We have conducted empirical experiments to compare the proposed algorithm with a standard initialization approach very often used in grammar-guided genetic programming. The results report that the proposed initialization method approximates very well a uniform distribution of the individuals in the search space. Moreover, the overall evolutionary process that takes place after the population initialization performs better in terms of convergence speed and quality of the final solutions achieved when the proposed method generates the initial population than when the usual approach does. The results also show that these performance differences are more significant when the experiments involve large search spaces.

Grammar Guided Genetic Programming for Network Architecture Search and Road Detection on Aerial Orthophotography

Photogrammetry involves aerial photography of the Earth’s surface and subsequently processing the images to provide a more accurate depiction of the area (Orthophotography). It is used by the Spanish Instituto Geográfico Nacional to update road cartography but requires a significant amount of manual labor due to the need to perform visual inspection of all tiled images. Deep learning techniques (artificial neural networks with more than one hidden layer) can perform road detection but it is still unclear how to find the optimal network architecture. Our main goal is the automatic design of deep neural network architectures with grammar-guided genetic programming. In this kind of evolutive algorithm, all the population individuals (here candidate network architectures) are constrained to rules specified by a grammar that defines valid and useful structural patterns to guide the search process. Grammar used includes well-known complex structures (e.g., Inception-like modules) combined with a custom designed mutation operator (dynamically links the mutation probability to structural diversity). Pilot results show that the system is able to design models for road detection that obtain test accuracies similar to that reached by state-of-the-art models when evaluated over a dataset from the Spanish National Aerial Orthophotography Plan.

Quantum-Inspired Genetic Programming Algorithm for the Crude Oil Scheduling of a Real-World Refinery

Refinery scheduling comprises a group of decisions that aims to optimize asset allocation, activity sequencing, and the time-related realization of those activities. This scheduling must achieve multiple objectives while considering different types of constraints. Uninterrupted processing unit operation, on-time crude oil batch receipts, and tank switchover minimization coexist in the everyday reasoning of a scheduler. However, it is not usual that works encompassing many operational aspects, such as multiple operational objectives, settling time, and an unlimited number of crudes, to blend in any tank. This article proposes a new algorithm that integrates linear and grammar-guided genetic programming concepts with a quantum-inspired approach to create programs that represent a crude oil refinery scheduling solution. The fitness function comprises four objectives that guide the evolution based on importance predefined by the decision maker. We propose a success ratio to evaluate the algorithm performance considering 50 runs for each case. A final solution is considered a success if two more important objectives are optimized. We assessed our approach with five different scenarios of a real refinery and three of them achieved a 100% success ratio.

Multi-level diversity promotion strategies for Grammar-guided Genetic Programming

Grammar-guided Genetic Programming (G3P) is a family of Evolutionary Algorithms that can evolve programs in any language described by a context-free grammar. The most widespread members of this family are based on an indirect representation: a sequence of bits or integers (the genotype) is transformed into a string of the language (the phenotype) by means of a mapping function, and eventually into a fitness value. Unfortunately, the flexibility brought by this mapping is also likely to introduce non-locality phenomena, reduce diversity, and hamper the effectiveness of the algorithm. In this paper, we experimentally characterize how population diversity, measured at different levels, varies for four popular G3P approaches. We then propose two strategies for promoting diversity which are general, independent both from the specific problem being tackled and from the other components of the Evolutionary Algorithm, such as genotype–phenotype mapping, selection criteria, and genetic operators. We experimentally demonstrate their efficacy in a wide range of conditions and from different points of view. The results also confirm the preponderant importance of the phenotype-level analyzes in diversity promotion.

A multi-level grammar approach to grammar-guided genetic programming: the case of scheduling in heterogeneous networks

The scale at which the human race consumes data has increased exponentially in recent years. One key part in this increase has been the usage of smart phones and connected devices by the populous. Multi-level heterogeneous networks are the driving force behind this mobile revolution, but these are constrained with limited bandwidth and over-subscription. Scheduling users on these networks has become a growing issue. In recent years grammar-guided genetic programming (G3P) has shown its capability to evolve beyond human-competitive network schedulers. Despite the performance of the G3P schedulers, a large margin of improvement is demonstrated to still exist. In the pursuit of this goal we recently proposed a multi-level grammar approach to generating schedulers. The complexity of the grammar was increased at various stages during evolution, allowing for individuals to add more complex functions through variation operations. The goal is to evolve good quality solutions before allowing the population to specialise more as the grammar functionality increased in a layered learning way. In this paper the results of this initial study are replicated, and confirmed, and it is seen that this approach improves the quality of the evolved schedulers. However, despite the gain in performance, we notice that the proposed approach comes with an acute sensitivity to the generation at which the grammar complexity is increased. Therefore, we put forward a novel seeding strategy and show that the seeding strategy mitigates the shortcomings of the original approach. The use of the seeding strategy outperforms the original approach in all the studied cases, and thus yields a better overall performance than the state-of-the-art G3P generated schedulers.

A Grammar-Guided Genetic Programing Algorithm for Associative Classification in Big Data

The state-of-the-art in associative classification includes interesting approaches for building accurate and interpretable classifiers. These approaches generally work on four different phases (data discretization, pattern mining, rule mining, and classifier building), some of them being computational expensive. The aim of this work is to propose a novel evolutionary algorithm for efficiently building associative classifiers in Big Data. The proposed model works in only two phases (a grammar-guided genetic programming framework is performed in each phase): (1) mining reliable association rules; (2) building an accurate classifier by ranking and combining the previously mined rules. The proposal has been implemented on different architectures (multi-thread, Apache Spark and Apache Flink) to take advantage of the distributed computing. The experimental results have been obtained on 40 well-known datasets and analyzed through non-parametric tests. Results were compared to multiple approaches in the field and analyzed on three ways: quality of the predictions, level of interpretability, and efficiency. The proposed method obtained accurate and interpretable classifiers in an efficient way even on high-dimensional data, outperforming the state-of-the-art algorithms on three different levels: quality of the predictions, interpretability, and efficiency.

Evolving rule-based classifiers with genetic programming on GPUs for drifting data streams

Designing efficient algorithms for mining massive high-speed data streams has become one of the contemporary challenges for the machine learning community. Such models must display highest possible accuracy and ability to swiftly adapt to any kind of changes, while at the same time being characterized by low time and memory complexities. However, little attention has been paid to designing learning systems that will allow us to gain a better understanding of incoming data. There are few proposals on how to design interpretable classifiers for drifting data streams, yet most of them are characterized by a significant trade-off between accuracy and interpretability. In this paper, we show that it is possible to have all of these desirable properties in one model. We introduce ERulesD2S: evolving rule-based classifier for drifting data Streams. By using grammar-guided genetic programming, we are able to obtain accurate sets of rules per class that are able to adapt to changes in the stream without a need for an explicit drift detector. Additionally, we augment our learning model with new proposals for rule propagation and data stream sampling, in order to maintain a balance between learning and forgetting of concepts. To improve efficiency of mining massive and non-stationary data, we implement ERulesD2S parallelized on GPUs. A thorough experimental study on 30 datasets proves that ERulesD2S is able to efficiently adapt to any type of concept drift and outperform state-of-the-art rule-based classifiers, while using small number of rules. At the same time ERulesD2S is highly competitive to other single and ensemble learners in terms of accuracy and computational complexity, while offering fully interpretable classification rules. Additionally, we show that ERulesD2S can scale-up efficiently to high-dimensional data streams, while offering very fast update and classification times. Finally, we present the learning capabilities of ERulesD2S for sparsely labeled data streams.