GP-based Methods Research Articles

A hybrid physics-informed machine learning approach for time-dependent reliability assessment of electromagnetic relays

Electromagnetic relays (EMRs) are intricate micro-electromechanical systems characterized by nonlinear behavior and coupling effects between electromagnetic and mechanical forces. Accurately modeling degradation and assessing reliability are crucial yet challenging tasks for ensuring their safe and efficient operation. Current data-driven methods for degradation modeling and reliability assessment often neglect the known physical knowledge regarding EMRs, leading to inaccuracies in modeling and assessment outcomes when data is incomplete. While physics-informed machine learning (PIML) approaches offer a potential solution, common regression models like Gaussian processes (GP) and long short-term memory (LSTM) suffer from underfitting and overfitting, respectively. To address these issues, we presents a hybrid PIML approach for time-dependent reliability assessment based on the emerging variational autoencoder (VAE) framework. This approach combines the advantages of GP-based methods that enable probabilistic representation with deep neural network-based methods that are more flexible and computationally efficient. Finally, we validate our proposed approach using real-world engineering data, demonstrating its superior accuracy and computational efficiency compared to state-of-the-art methods.

A Global and Local Surrogate-Assisted Genetic Programming Approach to Image Classification

Genetic programming (GP) has achieved promising performance in image classification. However, GP-based methods usually require a long computation time for fitness evaluations, posing a challenge to real-world applications. Surrogate models can be efficiently computable approximations of expensive fitness evaluations. However, most existing surrogate methods are designed for evolutionary computation techniques with a vector-based representation consisting of numerical values, thus cannot be directly used for GP with a tree-based representation consisting of functions/operators. The variable sizes of GP trees further increase the difficulty of building the surrogate model for fitness approximations. To address these limitations, we propose a new surrogate-assisted GP approach including global and local surrogate models, which can accelerate the evolutionary learning process and achieve competitive classification performance simultaneously. The global surrogate model can assist GP in exploring the entire search space, while the local surrogate model can speed up convergence and further improve performance. Furthermore, a new surrogate training set is constructed to assist in establishing the relationship between the GP tree and its fitness, and effective surrogate models can be built accordingly. Experimental results on ten datasets of varying difficulty show that the new approach significantly reduces the computational cost of the GP-based method without sacrificing the classification accuracy. The comparisons with other state-of-the-art methods also demonstrate the effectiveness of the new approach. Further analysis reveals the significance of the global and local surrogates and the new surrogate training set on improving or maintaining the performance of the proposed approach while reducing the computational cost.

M5GP: Parallel Multidimensional Genetic Programming with Multidimensional Populations for Symbolic Regression

Machine learning and artificial intelligence are growing in popularity thanks to their ability to produce models that exhibit unprecedented performance in domains that include computer vision, natural language processing and code generation. However, such models tend to be very large and complex and impossible to understand using traditional analysis or human scrutiny. Conversely, Symbolic Regression methods attempt to produce models that are relatively small and (potentially) human-readable. In this domain, Genetic Programming (GP) has proven to be a powerful search strategy that achieves state-of-the-art performance. This paper presents a new GP-based feature transformation method called M5GP, which is hybridized with multiple linear regression to produce linear models, implemented to exploit parallel processing on graphical processing units for efficient computation. M5GP is the most recent variant from a family of feature transformation methods (M2GP, M3GP and M4GP) that have proven to be powerful tools for both classification and regression tasks applied to tabular data. The proposed method was evaluated on SRBench v2.0, the current standard benchmarking suite for Symbolic Regression. Results show that M5GP achieves performance that is competitive with the state-of-the-art, achieving a top-three rank on the most difficult subset of black-box problems. Moreover, it achieves the lowest computation time when compared to other GP-based methods that have similar accuracy scores.

Automatic Feature Construction-Based Genetic Programming for Degraded Image Classification

Accurately classifying degraded images is a challenging task that relies on domain expertise to devise effective image processing techniques for various levels of degradation. Genetic Programming (GP) has been proven to be an excellent approach for solving image classification tasks. However, the program structures designed in current GP-based methods are not effective in classifying images with quality degradation. During the iterative process of GP algorithms, the high similarity between individuals often results in convergence to local optima, hindering the discovery of the best solutions. Moreover, the varied degrees of image quality degradation often lead to overfitting in the solutions derived by GP. Therefore, this research introduces an innovative program structure, distinct from the traditional program structure, which automates the creation of new features by transmitting information learned across multiple nodes, thus improving GP individual ability in constructing discriminative features. An accompanying evolution strategy addresses high similarity among GP individuals by retaining promising ones, thereby refining the algorithm’s development of more effective GP solutions. To counter the potential overfitting issue of the best GP individual, a multi-generational individual ensemble strategy is proposed, focusing on constructing an ensemble GP individual with an enhanced generalization capability. The new method evaluates performance in original, blurry, low contrast, noisy, and occlusion scenarios for six different types of datasets. It compares with a multitude of effective methods. The results show that the new method achieves better classification performance on degraded images compared with the comparative methods.

An Efficient Method for Antenna Design Based on a Self-Adaptive Bayesian Neural Network-Assisted Global Optimization Technique

Gaussian process (GP) is a very popular machine learning method for online surrogate-model-assisted antenna design optimization. Despite many successes, two improvements are important for the GP-based antenna global optimization methods, including: 1) the convergence speed (i.e., the number of necessary electromagnetic (EM) simulations to obtain a high-performance design) and 2) the GP model training cost when there are several tens of design variables and/or specifications. In both aspects, the state-of-the-art GP-based methods show practical but not desirable performance. Therefore, a new method, called the self-adaptive Bayesian neural network surrogate-model-assisted differential evolution (DE) for antenna optimization (SB-SADEA), is presented in this article. The key innovations include: 1) the introduction of the Bayesian neural network (BNN)-based antenna surrogate modeling method into this research area, replacing GP modeling, and 2) a bespoke self-adaptive lower confidence bound (LCB) method for antenna design landscape making use of the BNN-based antenna surrogate model. The performance of SB-SADEA is demonstrated by two challenging design cases, showing considerable improvement in terms of both convergence speed and machine learning cost compared with the state-of-the-art GP-based antenna global optimization methods.

Bayesian Calibration of Traffic Flow Fundamental Diagrams Using Gaussian Processes

Modeling the relationship between vehicle speed and density on the road is a fundamental problem in traffic flow theory. Recent research found that using the least-squares (LS) method to calibrate single-regime speed-density models is biased because of the uneven distribution of samples. This paper explains the issue of the LS method from a statistical perspective: the biased calibration is caused by the correlations/dependencies in regression residuals. Based on this explanation, we propose a new calibration method for single-regime speed-density models by modeling the covariance of residuals via a zero-mean Gaussian Process (GP). Our approach can be viewed as a generalized least-squares (GLS) method with a specific covariance structure (i.e., kernel function) and is a generalization of the existing LS and the weighted least-squares (WLS) methods. Next, we use a sparse approximation to address the scalability issue of GPs and apply a Markov chain Monte Carlo (MCMC) sampling scheme to obtain the posterior distributions of the parameters for speed-density models and the hyperparameters (i.e., length scale and variance) of the GP kernel. Finally, we calibrate six well-known single-regime speed-density models with the proposed method. Results show that the proposed GP-based methods (1) significantly reduce the biases in the LS calibration, (2) achieve a similar effect as the WLS method, (3) can be used as a non-parametric speed-density model, and (4) provide a Bayesian solution to estimate posterior distributions of parameters and speed-density functions.

Bloat-aware GP-based methods with bloat quantification

Bloat-aware GP-based methods with bloat quantification

A Divide-and-Conquer Genetic Programming Algorithm With Ensembles for Image Classification

Genetic programming (GP) has been applied to feature learning in image classification and achieved promising results. However, one major limitation of existing GP-based methods is the high computational cost, which may limit their applications on large-scale image classification tasks. To address this, this article develops a divide-and-conquer GP algorithm with knowledge transfer (KT) and ensembles to achieve fast feature learning in image classification. In the new algorithm framework, a divide-and-conquer strategy is employed to split the training data and the population into small subsets or groups to reduce computational time. A new KT method is proposed to improve GP learning performance. A new fitness function based on log loss and a new ensemble formulation strategy are developed to build an effective ensemble for image classification. The performance of the proposed approach has been examined on 12 image classification datasets of varying difficulty. The results show that the new approach achieves better classification performance in significantly less computation time than the baseline GP-based algorithm. The comparisons with state-of-the-art algorithms show that the new approach achieves better or comparable performance in almost all the comparisons. Further analysis demonstrates the effectiveness of ensemble formulation and KT in the proposed approach.

Multi-Objective Memetic Algorithms with Tree-Based Genetic Programming and Local Search for Symbolic Regression

Symbolic regression is to search the space of mathematical expressions to find a model that best fits a given dataset. As genetic programming (GP) with the tree representation can represent solutions as expression trees, it is popularly-used for regression. However, GP tends to evolve unnecessarily large programs (known as bloat), causing excessive use of CPU time/memory and evolving solutions with poor generalization ability. Moreover, even though the importance of local search has been proved in augmenting the search ability of GP (termed as memetic algorithms), local search is underused in GP-based methods. This work aims to handle the above problems simultaneously. To control bloat, a multi-objective (MO) technique (NSGA-II, Non-dominant Sorting Genetic Algorithm) is selected to incorporate with GP, forming a multi-objective GP (MOGP). Moreover, three mutation-based local search operators are designed and incorporated with MOGP respectively to form three multi-objective memetic algorithms (MOMA), i.e. MOMA_MR (MOMA with Mutation-based Random search), MOMA_MF (MOMA with Mutation-based Function search) and MOMA_MC (MOMA with Mutation-based Constant search). The proposed methods are tested on both benchmark functions and real-world applications, and are compared with both GP-based (i.e. GP and MOGP) and nonGP-based symbolic regression methods. Compared with GP-based methods, the proposed methods can reduce the risk of bloat with the evolved solutions significantly smaller than GP solutions, and the local search strategies introduced in the proposed methods can improve their search ability with the evolved solutions dominating MOGP solutions. In addition, among the three proposed methods, MOMA_MR performs best in RMSE for testing, yet it consumes more training time than others. Moreover, compared with six reference nonGP-based symbolic regression methods, MOMA_MR generally performs better than or similar to them consistently.

Scalable Variational Gaussian Processes for Crowdsourcing: Glitch Detection in LIGO.

In the last years, crowdsourcing is transforming the way classification training sets are obtained. Instead of relying on a single expert annotator, crowdsourcing shares the labelling effort among a large number of collaborators. For instance, this is being applied in the laureate laser interferometer gravitational waves observatory (LIGO), in order to detect glitches which might hinder the identification of true gravitational-waves. The crowdsourcing scenario poses new challenging difficulties, as it has to deal with different opinions from a heterogeneous group of annotators with unknown degrees of expertise. Probabilistic methods, such as Gaussian processes (GP), have proven successful in modeling this setting. However, GPs do not scale up well to large data sets, which hampers their broad adoption in real-world problems (in particular LIGO). This has led to the very recent introduction of deep learning based crowdsourcing methods, which have become the state-of-the-art for this type of problems. However, the accurate uncertainty quantification provided by GPs has been partially sacrificed. This is an important aspect for astrophysicists in LIGO, since a glitch detection system should provide very accurate probability distributions of its predictions. In this work, we first leverage a standard sparse GP approximation (SVGP) to develop a GP-based crowdsourcing method that factorizes into mini-batches. This makes it able to cope with previously-prohibitive data sets. This first approach, which we refer to as scalable variational Gaussian processes for crowdsourcing (SVGPCR), brings back GP-based methods to a state-of-the-art level, and excels at uncertainty quantification. SVGPCR is shown to outperform deep learning based methods and previous probabilistic ones when applied to the LIGO data. Its behavior and main properties are carefully analyzed in a controlled experiment based on the MNIST data set. Moreover, recent GP inference techniques are also adapted to crowdsourcing and evaluated experimentally.

An adaptive GP-based memetic algorithm for symbolic regression

Symbolic regression is a process to find a mathematical expression that represents the relationship between a set of explanatory variables and a measured variable. It has become a best-known problem for GP (genetic programming), as GP can use the tree representation to represent solutions as expression trees. Since the success of memetic algorithms (MAs (Memetic algorithms (MAs) can be regarded as a class of methods that combine population-based global search and local search [6, 30])) has proved the importance of local search in augmenting the global search ability of GP, GP with local search is investigated to solve symbolic regression tasks in this work. An important design issue of MAs is the balance between the global exploration of GP and the local exploitation, which has a great influence on the performance and efficiency of MAs. This work proposes a GP-based memetic algorithm for symbolic regression, termed as aMeGP (a daptive Me metic GP), which can balance global exploration and local exploitation adaptively. Compared with GP, two improvements are made in aMeGP to invoke and stop local search adaptively during evolution. The proposed aMeGP is compared with GP-based and nonGP-based symbolic regression methods on both benchmark test functions and real-world applications. The results show that aMeGP is generally better than both GP-based and nonGP-based reference methods with its evolved solutions achieving lower root mean square error (RMSE) for most test cases. Moreover, aMeGP outperforms the reference GP-based methods in the convergence ability, which can converge to lower RMSE values with faster or similar speeds.

Rethinking Greulich and Pyle: A Deep Learning Approach to Pediatric Bone Age Assessment Using Pediatric Trauma Hand Radiographs.

To develop a deep learning approach to bone age assessment based on a training set of developmentally normal pediatric hand radiographs and to compare this approach with automated and manual bone age assessment methods based on Greulich and Pyle (GP). In this retrospective study, a convolutional neural network (trauma hand radiograph-trained deep learning bone age assessment method [TDL-BAAM]) was trained on 15 129 frontal view pediatric trauma hand radiographs obtained between December 14, 2009, and May 31, 2017, from Children's Hospital of New York, to predict chronological age. A total of 214 trauma hand radiographs from Hasbro Children's Hospital were used as an independent test set. The test set was rated by the TDL-BAAM model as well as a GP-based deep learning model (GPDL-BAAM) and two pediatric radiologists (radiologists 1 and 2) using the GP method. All ratings were compared with chronological age using mean absolute error (MAE), and standard concordance analyses were performed. The MAE of the TDL-BAAM model was 11.1 months, compared with 12.9 months for GPDL-BAAM (P = .0005), 14.6 months for radiologist 1 (P < .0001), and 16.0 for radiologist 2 (P < .0001). For TDL-BAAM, 95.3% of predictions were within 24 months of chronological age compared with 91.6% for GPDL-BAAM (P = .096), 86.0% for radiologist 1 (P < .0001), and 84.6% for radiologist 2 (P < .0001). Concordance was high between all methods and chronological age (intraclass coefficient > 0.93). Deep learning models demonstrated a systematic bias with a tendency to overpredict age for younger children versus radiologists who showed a consistent mean bias. A deep learning model trained on pediatric trauma hand radiographs is on par with automated and manual GP-based methods for bone age assessment and provides a foundation for developing population-specific deep learning algorithms for bone age assessment in modern pediatric populations.Supplemental material is available for this article.© RSNA, 2020See also the commentary by Halabi in this issue.

A hybrid Gaussian process approach to robust economic model predictive control

This paper proposes a hybrid Gaussian process (GP) approach to robust economic model predictive control under unknown future disturbances in order to reduce the conservatism of the controller. The proposed hybrid GP is a combination of two well-known methods, namely, kernel composition and nonlinear auto-regressive. A switching mechanism is employed to select one of these methods for disturbance prediction after analyzing the prediction outcomes. The hybrid GP is intended to detect not only patterns but also unexpected behaviors in the unknown disturbances by using past disturbance measurements. A novel forgetting factor concept is also utilized in the hybrid GP, giving less weight to older measurements, in order to increase prediction accuracy based on recent disturbances values. The detected disturbance information is used to reduce prediction uncertainty in economic model predictive controllers systematically. The simulation results show that the proposed method can improve the overall performance of an economic model predictive controller compared to other GP-based methods in cases when disturbances have discernible patterns.

Evolving semantic object segmentation methods automatically by genetic programming from images and image processing operators

Even though numerous segmentation methods exist, the requirement of prior knowledge or parameter tuning makes them restricted to limited image domains. Without predefining solution models, genetic programming (GP) is able to solve complex problems by evolving computer programs automatically. In this paper, three new GP-based methods are designed to evolve segmentation algorithms automatically from images and primitive image processing operators (e.g., filters and histogram equalization). Specifically, a strongly typed representation, the cooperative coevolution technique and a two-stage evolution are introduced in GP, respectively, to form three new methods that can evolve solutions to conduct image preprocessing, segmentation and postprocessing automatically. The new methods are termed as StronglyGP, CoevoGP and TwostageGP, and standard GP-based algorithm (StandardGP) is employed as a reference method. The proposed methods are tested on two complicated datasets (i.e., Weizmann and Pascal datasets), which contain high variations in both objects and backgrounds. The results show that StronglyGP and StandardGP can evolve effective segmentors for the given complex segmentation tasks, while CoevoGP and TwostageGP perform worse than StronglyGP and StandardGP, which may be caused by the overfitting problem in deriving postprocessing solutions. In addition, compared with StandardGP, StronglyGP achieves better segmentation performance with smaller solution sizes. Moreover, compared with four widely used segmentation methods, StronglyGP and StandardGP can produce satisfactory results consistently on both Weizmann and Pascal datasets.

Genetic programming based feature construction methods for foreground object segmentation

Foreground object segmentation is a crucial preprocessing step for many high-level computer vision tasks, e.g. object recognition. It is still challenging to achieve accurate segmentation, especially for complex images (e.g. with high variations). Feature construction can help to improve the segmentation performance by extracting more distinctive features for foreground/background regions from the original features. However, commonly-used feature construction methods (e.g. principle component analysis) often involve certain assumptions/constraints, and the constructed features cannot be interpreted. To address these problems, genetic programming (GP) is employed in this paper, which is a well-suited feature construction technique.The aim of this work is to design new feature construction methods using GP, and analyse/compare popular GP-based feature construction methods for foreground object segmentation, especially on complex image datasets with high variations. Specifically, one new feature construction method that incorporates the subtree technique in GP is designed, which can construct multiple features simultaneously (called SubtMFC, Subtree Multiple Feature Construction). Moreover, a parsimony pressure technique is introduced to improve SubtMFC for bloat control (a common issue for GP-based methods), which forms the method, PSubtMFC (Parsimony SubtMFC). In addition, comparison of popular GP-based feature construction methods for foreground object segmentation is conducted for the first time. Results show that SubtMFC achieves better or similar performance compared with three reference methods. In addition, compared with SubtMFC that does not control bloat, PSubtMFC can significantly reduce the solution size while maintain similar performance in the segmentation accuracy. The GP-based feature construction framework is further extended for feature representation based knowledge transfer, which can handle the problem of the scare labelled training data. Moreover, after GP is thoroughly investigated on benchmark datasets with one type of foreground objects (i.e. the Weizmann horse dataset and Pascal aeroplane dataset), it is considered whether the GP methods can perform well on datasets containing multiple types of foreground objects. Compared with three other well-performing GP-based feature construction methods, the proposed method achieves better or comparable results for the given segmentation tasks. In addition, this paper thoroughly compares/analyses popular GP-based feature construction methods for complex figure-ground segmentation for the first time. Moreover, further analyses on the input features frequently used by the GP-evolved feature construction functions reflect the effectiveness of the extracted high-level features.

Promoting active learning with mixtures of Gaussian processes

Active learning is an effective methodology to relieve the tedious and expensive work of manual annotation for many supervised learning applications. The active learning framework with good performance usually contains powerful learning models and delicate active learning strategies. Gaussian process (GP)-based active learning was proposed to be one of the most effective methods. However, the single GP suffers from the limitation of not modeling multimodal data well enough, and thus existing active learning strategies based on GPs only make use of limited information from data. In this paper, we propose three novel active learning methods, in which the existing mixture of GP model (MGP) is adjusted as the learning model and three active learning strategies are designed based on the adjusted MGP. Through experiments on multiple data sets, we analyze the performance and characteristics of the three proposed active learning methods, and further compare with popular GP-based methods and some other state-of-the-art methods.

Experimental Validation of Gaussian Process-Based Air-to-Ground Communication Quality Prediction in Urban Environments.

This paper presents a detailed experimental assessment of Gaussian Process (GP) regression for air-to-ground communication channel prediction for relay missions in urban environment. Considering restrictions from outdoor urban flight experiments, a way to simulate complex urban environments at an indoor room scale is introduced. Since water significantly absorbs wireless communication signal, water containers are utilized to replace buildings in a real-world city. To evaluate the performance of the GP-based channel prediction approach, several indoor experiments in an artificial urban environment were conducted. The performance of the GP-based and empirical model-based prediction methods for a relay mission was evaluated by measuring and comparing the communication signal strength at the optimal relay position obtained from each method. The GP-based prediction approach shows an advantage over the model-based one as it provides a reasonable performance without a need for a priori information of the environment (e.g., 3D map of the city and communication model parameters) in dynamic urban environments.

GP-based methods for domain adaptation: using brain decoding across subjects as a test-case

Research on classifier transferability intends that the information gathered in the solution of a given classification problem could be reused in the solution of similar or related problems. We propose the evolution of transferable classifiers based on the use of multi-objective genetic programming and new fitness-functions that evaluate the amount of transferability. We focus on the domain adaptation scenario in which the problem to be solved is the same in the source and target domains, but the distribution of data is different between domains. As a real-world test case we address the brain decoding problem, whose goal is to predict the stimulus presented to a subject from the analysis of his brain activity. Brain decoding across subjects attempts to reuse the classifiers learned from some subjects in the classification of the others. We evolved GP-based classifiers using different variants of the introduced approach to test their effectiveness on data obtained from a brain decoding experiment involving 16 subjects. Our results show that the GP-based classifiers evolved trying to maximize transferability are able to improve classification accuracy over other classical classifiers that incorporate domain adaptation methods. Moreover, after comparing our algorithm to importance-weighted cross validation (in conjunction with many ML methods), we conclude that our approach achieves state of the art results in terms of transferability.

Figure-ground image segmentation using feature-based multi-objective genetic programming techniques

Figure-ground image segmentation is a process of separating regions of interest from a target image. Genetic programming has been employed to evolve segmentors that have the potential to capture high variations of images and conduct accurate segmentation. However, GP-based methods tend to evolve complex segmentors that have large sizes, are computationally expensive and difficult to interpret. Therefore, this work aims to balance the solution functionality with the complexity by applying multi-objective techniques to GP. Specifically, NSGA-II (nondominated sorting genetic algorithm) and SPEA2 (strength Pareto evolutionary algorithm) are selected as the base multi-objective techniques, in which a new Pareto dominance mechanism is designed, thus creating two new multi-objective techniques—INSGA-II (improved NSGA-II) and ISPEA2 (improved SPEA2). By applying the INSGA-II and ISPEA2 to GP, respectively, two novel multi-objective GP (MOGP) methods are proposed—INSGP and ISPGP. Both methods have two objectives: a solution functionality measure (i.e. the classification accuracy) and a solution complexity measure based on an exponential function. The results show that the proposed MOGP methods can evolve solutions with good trade-offs between the functionality and complexity, and INSGP is better at keeping solution diversity than ISPGP for the segmentation tasks in this paper. Moreover, the analyses on the evolved segmentors show that certain discriminatory patterns can be captured.

Characterizing Dynamic Walking Patterns and Detecting Falls with Wearable Sensors Using Gaussian Process Methods.

By incorporating a growing number of sensors and adopting machine learning technologies, wearable devices have recently become a prominent health care application domain. Among the related research topics in this field, one of the most important issues is detecting falls while walking. Since such falls may lead to serious injuries, automatically and promptly detecting them during daily use of smartphones and/or smart watches is a particular need. In this paper, we investigate the use of Gaussian process (GP) methods for characterizing dynamic walking patterns and detecting falls while walking with built-in wearable sensors in smartphones and/or smartwatches. For the task of characterizing dynamic walking patterns in a low-dimensional latent feature space, we propose a novel approach called auto-encoded Gaussian process dynamical model, in which we combine a GP-based state space modeling method with a nonlinear dimensionality reduction method in a unique manner. The Gaussian process methods are fit for this task because one of the most import strengths of the Gaussian process methods is its capability of handling uncertainty in the model parameters. Also for detecting falls while walking, we propose to recycle the latent samples generated in training the auto-encoded Gaussian process dynamical model for GP-based novelty detection, which can lead to an efficient and seamless solution to the detection task. Experimental results show that the combined use of these GP-based methods can yield promising results for characterizing dynamic walking patterns and detecting falls while walking with the wearable sensors.