Genetic Programming For Classification Research Articles

Ensemble classifiers using multi-objective Genetic Programming for unbalanced data

Genetic Programming (GP) can be used to design effective classifiers due to its built-in feature selection and feature construction characteristics. Unbalanced data distributions affect the classification performance of GP classifiers. Some fitness functions have been proposed to solve the class imbalance problem of GP classifiers. However, with the evolution of GP, single-objective GP classifiers evaluated by a single fitness function have poor generalization ability. Moreover, using the best evolved GP classifier for decision-making can easily lead to the possibility of misclassification. In this paper, multi-objective GP is used to optimize multiple fitness functions including AUC approximation (Wmw), Distance (Dist), and Complexity to evolve ensemble classifiers, which jointly determines the class labels of unknown instances. Experiments on sixteen datasets show that our multi-objective GP can significantly improve classification performance compared with single-objective GP, and our proposed ensemble classifiers evolved by multi-objective GP can further improve the classification performance than the single best GP classifier. Comparisons with six GP-based and five traditional machine learning algorithms show that our proposed approaches can achieve significantly better classification performance on most cases.

Enhancing Local Decisions in Agent-Based Cartesian Genetic Programming by CMA-ES

Cartesian genetic programming is a popular version of classical genetic programming, and it has now demonstrated a very good performance in solving various use cases. Originally, programs evolved by using a centralized optimization approach. Recently, an algorithmic level decomposition of program evolution has been introduced that can be solved by a multi-agent system in a fully distributed manner. A heuristic for distributed combinatorial problem-solving was adapted to evolve these programs. The applicability of the approach and the effectiveness of the used multi-agent protocol as well as of the evolved genetic programs for the case of full enumeration in local agent decisions has already been successfully demonstrated. Symbolic regression, n-parity, and classification problems were used for this purpose. As is typical of decentralized systems, agents have to solve local sub-problems for decision-making and for determining the best local contribution to solving program evolution. So far, only a full enumeration of the solution candidates has been used, which is not sufficient for larger problem sizes. We extend this approach by using CMA-ES as an algorithm for local decisions. The superior performance of CMA-ES is demonstrated using Koza’s computational effort statistic when compared with the original approach. In addition, the distributed modality of the local optimization is scrutinized by a fitness landscape analysis.

Factors affecting the presence of Arctic charr in streams based on a jittered binary genetic programming model

Arctic charr is one of the fish species most sensitive to climate change but studies on their freshwater habitat preferences are limited, especially in the fluvial environment. Machine learning methods offer automatic and objective models for ecohydrological processes based on observed data. However, i) the number of ecological records is often much smaller than hydrological observations, and ii) ecological measurements over the long-term are costly. Consequently, ecohydrological datasets are scarce and imbalanced. To address these problems, we propose jittered binary genetic programming (JBGP) to detect the most dominant ecohydrological parameters affecting the occurrence of Arctic charr across tributaries within the large subarctic Teno River catchment, in northernmost Finland and Norway. We quantitatively assessed the accuracy of the proposed model and compared its performance with that of classic genetic programming (GP), decision tree (DT) and state-of-the-art jittered-DT methods. The JBGP achieves the highest total classification accuracy of 90% and a Heidke skill score of 78%, showing its superiority over its counterparts. Our results showed that the dominant factors contributing to the presence of Arctic charr in Teno River tributaries include i) a higher density of macroinvertebrates, ii) a lower percentage of mires in the catchment and iii) a milder stream channel slope.

A hybrid gene expression programming model for discharge prediction

The head–discharge relationship of an overflow weir is a prerequisite for flow measurement. Conventionally, it is determined by regression methods. With machine learning techniques, data-driven modelling becomes an alternative. However, a standalone model may be inadequate to generate satisfactory results, particularly for a complex system. With the intention of improving the performance of standard gene expression programming (GEP), a hybrid evolutionary scheme is proposed, which is coupled with grey system theory and probabilistic technique. As a gene filter, grey relational analysis (GRA) eliminates noise and simulated annealing (SA) reduces overfitting by optimising the gene weights. The proposed GEP-based model was developed and validated using experimental data of a submerged pivot weir. Compared with standalone GEP, the GRA–GEP–SA model was found to generate more accurate results. Its coefficients of determination and correlation were improved by 3.6% and 1.7%, respectively. The root mean square error was lowered by 24.8%, which is significant. The number of datasets with an error of less than 10% and 20% was increased by 15% and 12%, respectively. The proposed approach outperforms classic genetic programming and shows a comparative error level with the empirical formula. The hybrid procedure also provides a reference for applications in other hydraulic issues.

Wavelet Packet-Genetic Programming: A New Model for Meteorological Drought Hindcasting

This study presents developing procedures and verification of a new hybrid model, namely wavelet packet-genetic programming (WPGP) for short-term meteorological drought forecast. To this end, the multi-temporal standardized precipitation evapotranspiration index (SPEI) has been used as the drought quantifying parameter at two meteorological stations at Ankara province, Turkey. The new WPGP model comprises two main steps. In the first step, the wavelet packet, which is a generalization of the well-known wavelet transform, is used to decompose the SPEI series into deterministic and stochastic sub-signals. Then, classic genetic programming (GP) is applied to formulate the deterministic sub-signal considering its effective lags. To characterize the stochastic component, different theoretical probability distribution functions were assessed, and the best one was selected to integrate with the GP-evolved function. The efficiency of the new model was cross-validated with the first order autoregressive (AR1), GP, and random forest (RF) models developed as the benchmarks in the present study. The results showed that the WPGP is a robust model, superior to AR1 and RF, and significantly increases the predictive accuracy of the standalone GP model.

MSGP-LASSO: An improved multi-stage genetic programming model for streamflow prediction

This paper presents the development and verification of a new multi-stage genetic programming (MSGP) technique, called MSGP-LASSO, which was applied for univariate streamflow forecasting in the Sedre River, an intermittent river in Turkey. The MSGP-LASSO is a practical and cost-neutral improvement over classic genetic programming (GP) that increases modelling accuracy, while decreasing its complexity by coupling the MSGP and multiple regression LASSO methods. The new model uses average mutual information to identify the optimum lags, and root mean-square technique to minimize forecasting error. Based on Nash-Sutcliffe efficiency and bias-corrected Akaike information criterion, MSGP-LASSO is superior to GP, multigene GP, MSGP, and hybrid MSGP-least-square models. It is explicit and promising for real-life applications.

Designing genetic programming classifiers with feature selection and feature construction

Due to the flexibility of Genetic Programming (GP), GP has been used for feature construction, feature selection and classifier construction. In this paper, GP classifiers with feature selection and feature construction are investigated to obtain simple and effective classification rules. During the construction of a GP classifier, irrelevant and redundant features affect the search ability of GP, and make GP easily fall into local optimum. This paper proposes two new GP classifier construction methods to restrict bad impact of irrelevant and redundant features on GP classifier. The first is to use a multiple-objective fitness function that decreases both classification error rate and the number of selected features, which is named as GPMO. The second is to first use a feature selection method, i.e., linear forward selection (LFS) to remove irrelevant and redundant features and then use GPMO to construct classifiers, which is named as FSGPMO. Experiments on twelve datasets show that GPMO and FSGPMO have advantages over GP classifiers with a single-objective fitness function named GPSO in term of classification performance, the number of selected features, time cost and function complexity. The proposed FSGPMO can achieve better classification performance than GPMO on higher dimension datasets, however, FSGPMO may remove potential effective features for GP classifier and achieve much lower classification performance than GPMO on some datasets. Compared with two other GP-based classifiers, GPMO can significantly improve the classification performance. Comparisons with other classification algorithms show that GPMO can achieve better or comparable classification performance on most selected datasets. Our proposed GPMO can achieve better performance than wrapper-based feature construction methods using GP on applications with insufficient instances. Further investigations show that bloat phenomena exists in the process of GP evolution and overfitting phenomena is not obvious. Moreover, the benefits of GP over other machine learning algorithms are discussed.

Benchmarking state-of-the-art symbolic regression algorithms

Symbolic regression (SR) is a powerful method for building predictive models from data without assuming any model structure. Traditionally, genetic programming (GP) was used as the SR engine. However, for these purely evolutionary methods it was quite hard to even accommodate the function to the range of the data and the training was consequently inefficient and slow. Recently, several SR algorithms emerged which employ multiple linear regression. This allows the algorithms to create models with relatively small error right from the beginning of the search. Such algorithms are claimed to be by orders of magnitude faster than SR algorithms based on classic GP. However, a systematic comparison of these algorithms on a common set of problems is still missing and there is no basis on which to decide which algorithm to use. In this paper we conceptually and experimentally compare several representatives of such algorithms: GPTIPS, FFX, and EFS. We also include GSGP-Red, which is an enhanced version of geometric semantic genetic programming, an important algorithm in the field of SR. They are applied as off-the-shelf, ready-to-use techniques, mostly using their default settings. The methods are compared on several synthetic SR benchmark problems as well as real-world ones ranging from civil engineering to aerodynamics and acoustics. Their performance is also related to the performance of three conventional machine learning algorithms: multiple regression, random forests and support vector regression. The results suggest that across all the problems, the algorithms have comparable performance. We provide basic recommendations to the user regarding the choice of the algorithm.

Electrical Energy Demand Prediction: A Comparison Between Genetic Programming and Decision Tree

Several recent studies have used various data mining techniques to obtain accurate electrical energy demand forecasts in power supply systems. This paper, for the first time, compares the efficiency of the decision tree (DT) and classic genetic programming (GP) data mining models developed for electrical energy demand forecasting in Nicosia, Northern Cyprus. The models were trained and tested using daily electricity consumptions measured during the period 2011-2016 and were compared in terms of three statistical performance indices including coefficient of determination, mean absolute percentage error and concordance coefficient. The prediction results showed that the proposed models can be effectively applied to forecasts of electrical energy demand. The results also indicated that the GP is slightly superior to DT in terms of the performance indices.

Data mining and image analysis using genetic programming

Genetic programming (GP) is an artificial intelligence technique that benefits from evolutionary computations allowing computers to solve problems automatically. In this paper, we present an optimized genetic-programming-based classifier that directly solves the multi-class classification problems in data mining and image analysis. A new fitness function is proposed for multiclass classification and brain tumor detection, which is validated by 10-fold cross validation. Instead of defining static thresholds as boundaries to differentiate between multiple labels, our work presents a method of classification where a GP system learns the relationships among experiential data and models them mathematically during the evolutionary process. We propose an optimized GP classifier based on a combination of pruning subtrees and a new fitness function. An orthogonal least squares algorithm is also applied in the training phase to create a robust GP classifier. Our approach has been assessed on six multiclass datasets and on a magnetic resonance imaging (MRI) brain image for tumor detection. The results of data classification for Iris, Wine, Glass, Pima, BUPA Liver and Balance Scale datasets are compared with existing algorithms. The high accuracy of brain tumor classification provided by our GP classifier confirms the strong ability of the developed technique for complicated classification problems. We compared our approach in terms of speed with previous GP algorithms as well. The analyzed results illustrate that the developed classifier produces a productive and rapid method for classification tasks that outperforms the previous methods for more challenging multiclass classification problems.

Multiclass Classification on High Dimension and Low Sample Size Data using Genetic Programming

Multiclass classification is one of the most fundamental tasks in data mining. However, traditional data mining methods rely on the model assumption, they generally can suffer from the overfitting problem on high dimension and low sample size (HDLSS) data. Trying to address multiclass classification problems on HDLSS data from another perspective, we utilize Genetic Programming (GP), an intrinsic evolutionary classification algorithm that can implement feature construction automatically without model assumption. This paper develops an ensemble-based genetic programming classification framework, the Sigmoid-based Ensemble Gene Expression Programming (SEGEP). To relieve the problem of output conflict in GP-based multiclass classifiers, the proposed method employs a flexible probability representation with continuous relaxation to better integrate the output of all the binary classifiers, an effective data division strategy to further enhance the ensemble performance, and a novel sampling strategy to refine the existing GP-based binary classifier. The experiment results indicate that SE-GEP can attain better classification accuracy compared to other GP methods. Moreover, the comparison with other representative machine learning methods indicates that SE-GEP is a competitive method for multiclass classification in HDLSS data.

Multiple genetic programming: a new approach to improve genetic-based month ahead rainfall forecasts.

It is well documented that standalone machine learning methods are not suitable for rainfall forecasting in long lead-time horizons. The task is more difficult in arid and semiarid regions. Addressing these issues, the present paper introduces a hybrid machine learning model, namely multiple genetic programming (MGP), that improves the predictive accuracy of the standalone genetic programming (GP) technique when used for 1-month ahead rainfall forecasting. The new model uses a multi-step evolutionary search algorithm in which high-performance rain-borne genes from a multigene GP solution are recombined through a classic GP engine. The model is demonstrated using rainfall measurements from two meteorology stations in Lake Urmia Basin, Iran. The efficiency of the MGP was cross-validated against the benchmark models, namely standard GP and autoregressive state-space. The results indicated that the MGP statistically outperforms the benchmarks at both rain gauge stations. It may reduce the absolute and relative errors by approximately up to 15% and 40%, respectively. This significant improvement over standalone GP together with the explicit structure of the MGP model endorse its application for 1-month ahead rainfall forecasting in practice.

Ensemble gene expression programming: a new approach for evolution of parsimonious streamflow forecasting model

A precise forecast of streamflow in intermittent rivers is of major difficulties and challenges in watershed management, particularly in arid and semiarid regions. The present research study introduces an ensemble gene expression programming (EGEP) modeling approach to 1- and 2-day ahead streamflow forecasts that meet both accuracy and simplicity criteria of an applied model. Three main components of the proposed EGEP approach which are capable of producing a parsimonious model include (i) creating a population of suitable solutions using classic genetic programming (GP) instead of a single solution, (ii) combining the solutions throughout gene expression programming, and (iii) parsimony selection based upon trade-off analysis between the complexity and accuracy of the best-evolved solutions at the holdout validation set. The EGEP model was trained and verified using the streamflow measurements from the Shahrchay River lying northwest of Iran. Several statistical indicators were computed for verification of the ensemble models’ accuracy with that of classic GP and artificial neural network models developed as the benchmarks. Our results revealed that the EGEP outperforms the benchmarks. It is an explicit, simple, and precise approach and, therefore, worthy to be used in practice.

Interpretable Multiview Early Warning System Adapted to Underrepresented Student Populations

Early warning systems have been progressively implemented in higher education institutions to predict student performance. However, they usually fail at effectively integrating the many information sources available at universities to make more accurate and timely predictions, they often lack decision-making reasoning to motivate the reasons behind the predictions, and they are generally biased toward the general student body, ignoring the idiosyncrasies of underrepresented student populations (determined by socio-demographic factors such as race, gender, residency, or status as a freshmen, transfer, adult, or first-generation students) that traditionally have greater difficulties and performance gaps. This paper presents a multiview early warning system built with comprehensible Genetic Programming classification rules adapted to specifically target underrepresented and underperforming student populations. The system integrates many student information repositories using multiview learning to improve the accuracy and timing of the predictions. Three interfaces have been developed to provide personalized and aggregated comprehensible feedback to students, instructors, and staff to facilitate early intervention and student support. Experimental results, validated with statistical analysis, indicate that this multiview learning approach outperforms traditional classifiers. Learning outcomes will help instructors and policy-makers to deploy strategies to increase retention and improve academics.

Classification of electroencephalogram signal for the detection of epilepsy using Innovative Genetic Programming

AbstractEpilepsy, sometimes called seizure disorder, is a neurological condition that justifies itself as a susceptibility to seizures. A seizure is a sudden burst of rhythmic discharges of electrical activity in the brain that causes an alteration in behaviour, sensation, or consciousness. It is essential to have a method for automatic detection of seizures, as these seizures are arbitrary and unpredictable. A profound study of the electroencephalogram (EEG) recordings is required for the accurate detection of these epileptic seizures. In this study, an Innovative Genetic Programming framework is proposed for classification of EEG signals into seizure and nonseizure. An empirical mode decomposition technique is used for the feature extraction followed by genetic programming for the classification. Moreover, a method for intron deletion, hybrid crossover, and mutation operation is proposed, which are responsible for the increase in classification accuracy and a decrease in time complexity. This suggests that the Innovative Genetic Programming classifier has a potential for accurately predicting the seizures in an EEG signal and hints on the possibility of building a real‐time seizure detection system.

A modified content-based evolutionary approach to identify unsolicited emails

This computational research seeks to classify unsolicited versus legitimate emails. A modified version of an existing genetic programming (GP) classifier—i.e., modified genetic programming (MGP)—is implemented to build an ensemble of classifiers to identify unsolicited emails. The proposed classifier is assessed using informative features extracted from two corpora (Enron and SpamAssassin) with the help of the greedy stepwise feature search method. Further, a comparative study is performed with other popular classifiers, such as Bayesian network, naive Bayes, decision tree, random forest (RF), support vector machine (SVM), and GP. Further the results are validated with 20-fold cross-validation and paired T test. The results prove that the proposed classifier performs better in terms of accuracy and false-positive detection in comparison with the other machine learning classifiers tested in this study. Using different training and testing a set of email files from the Enron corpus, ensemble-based classifiers, such as boosted SVM, boosted Bayesian, boosted naive Bayesian, RF, and the proposed MGP classifier, are tested and compared on all metrics, including training and testing time. The findings suggest that the MGP classifier with the greedy stepwise feature search method offers an improvement over alternative methods in detecting unsolicited emails.

An improved gene expression programming model for streamflow forecasting in intermittent streams

Skilful forecasting of monthly streamflow in intermittent rivers is a challenging task in stochastic hydrology. In this study, genetic algorithm (GA) was combined with gene expression programming (GEP) as a new hybrid model for month ahead streamflow forecasting in an intermittent stream. The hybrid model was named GEP-GA in which sub-expression trees of the best evolved GEP model were rescaled by appropriate weighting coefficients through the use of GA optimizer. Auto-correlation and partial auto-correlation functions of the streamflow records as well as evolutionary search of GEP were used to identify the optimum predictors (i.e., number of lags) for the model. The proposed methodology was demonstrated using monthly streamflow data from the Shavir Creek in Iran. Performance of the GEP-GA was compared to that of classic genetic programming (GP), GEP, multiple linear regression and GEP-linear regression models developed in the present study as the benchmarks. The results showed that the GEP-GA outperforms all the benchmarks and motivated to be used in practice.

Comparison of a genetic algorithm to grammatical evolution for automated design of genetic programming classification algorithms

Genetic Programming (GP) is gaining increased attention as an effective method for inducing classifiers for data classification. However, the manual design of a genetic programming classification algorithm is a non-trivial time consuming process. This research investigates the hypothesis that automating the design of a GP classification algorithm for data classification can still lead to the induction of effective classifiers and also reduce the design time. Two evolutionary algorithms, namely, a genetic algorithm (GA) and grammatical evolution (GE) are used to automate the design of GP classification algorithms. The classification performance of the automated designed GP classifiers i.e. GA designed GP classifiers and GE designed GP classifiers are compared to each other and to manually designed GP classifiers on real-world problems. Furthermore, a comparison of the design times of automated design and manual design is also carried out for the same set of problems. The automated designed classifiers were found to outperform manually designed classifiers across problem domains. Automated design time is also found to be less than manual design time. This study revealed that for the considered datasets GE performs better for binary classification while the GA does better for multiclass classification. Overall the results of the study are in support of the hypothesis.

A new two-stage genetic programming classification algorithm and its applications

This work aims to find a better solution from an improved genetic programming (GP) algorithm for classification problems without increasing computation costs as far as possible. Firstly, the standard GP algorithm is difficult to succeed in finding better individuals since too large search landscape and it is easy to run into local optimum when evolution reaches a certain stage, while, by enlarging the evolutionary generation or the size of population might increase computation complexity. So, a two-stage GP may be a good solution for these. In the first stage, GP is used to induce a relatively simple classifier and construct features; in the second stage, GP is executed on these features to evolve a better classifier. Secondly, in order to improve the convergence, a new initialization (NI) strategy and a new function operator selection method are designed. In this paper, a NI strategy based two-stage GP algorithm (NITGP) is proposed, and compares with the standard GP on a set of artificial, real-world datasets and image edge detection tasks. The experimental results show that our approach can evolve classifiers with better performance.

Classification in the Retrieval Phase of Case-based Reasoning

Case-based reasoning (CBR) is a problem solving technique that uses previous experiences to solve new problems. Among the four phases of CBR, Retrieval is the first and the most important phase, as it lays the foundation of the entire CBR cycle. Retrieval aims to retrieve similar cases from the case-base, given a new situation. CBR systems typically use a strategy called similarity-based retrieval for retrieving cases. One of the derivatives of similarity-based retrieval is k-nearest neighbor (k-NN) algorithm. In this paper, we compare the performances of k-NN, Fuzzy nearest neighbor (Fuzzy NN) and Genetic Programming (GP) classifiers for retrieval of cases. We evaluate these algorithms in WEKA, with benchmark data sets for classification from UCI.