Latent feature learning via autoencoder training for automatic classification configuration recommendation

Abstract

The Combined Algorithm Selection and Hyperparameter Optimization problem, in short, CASH, seeks the most suitable classifiers and hyperparameters for the underlying classification problems. In current literature, the common approaches in dealing with CASH problem are conducted via search-based methods such as sequential model-based optimization (SMBO) along with various active tests. Different from current existing approaches, in this paper, we propose a new method by incorporating the so-called denoising autoencoder (DAE) approach into meta-learning (MtL) for automatic configuration (both algorithms and their hyperparameters) recommendation, which appears to be quite effective compared to standard search-based approaches. More specifically, we set up the configuration search space for CASH and produce the metadata, and generate the classification performance on a set of collected historical datasets. Then both encoder and decoder in the DAE system are trained with the masked metadata as inputs and the unmasked metadata as targets to extract the subtle latent variables of metadata and recover the unmasked inputs subsequently. Under our framework, the performance over the entire configuration space can be predicted effectively through two different settings, and the configuration with the highest predictive performance is thus recommended. The first recommendation approach is by inactivating some inputs and then to recover their entries via the trained encoder and decoder for new problems, while in the second approach, the relationship between the acquired latent variables and the meta-features of historical datasets via kernel multivariate multiple regression (MMR) is enacted, leading to the performance estimation of new datasets being pursued directly through MMR and the decoder of DAE without requiring any new configuration evaluations. An automatic classification configuration recommendation system, including 81 historical problems and 11 common classifiers with a total of 4983 configurations, is established to show the effectiveness of our proposed approach. The comparative results on 45 testing problems demonstrate that our proposed model has the superior recommendation capacity in terms of the baselines for existing MtL as well as other search-based approaches.