Soft target and functional complexity reduction: A hybrid regularization method for genetic programming

Abstract

Regularization is frequently used in supervised machine learning to prevent models from overfitting. This paper tackles the problem of regularization in genetic programming. We apply, for the first time, soft target regularization, a method recently defined for artificial neural networks, to genetic programming. Also, we introduce a novel measure of functional complexity of the genetic programming individuals, aimed at quantifying their degree of curvature. We experimentally demonstrate that both the use of soft target regularization, and the minimization of the complexity during learning, are often able to reduce overfitting, but they are never able to eliminate it. On the other hand, we demonstrate that the integration of these two strategies into a novel hybrid genetic programming system can completely eliminate overfitting, for all the studied test cases. Last but not least, consistently with what found in the literature, we offer experimental evidence of the fact that the size of the genetic programming models has no correlation with their generalization ability.