Combined Algorithm Selection Research Articles

Do We Really Need Imputation in AutoML Predictive Modeling?

Numerous real-world data contain missing values, while in contrast, most Machine Learning (ML) algorithms assume complete datasets. For this reason, several imputation algorithms have been proposed to predict and fill in the missing values. Given the advances in predictive modeling algorithms tuned in an Automated Machine Learning context (AutoML) setting, a question that naturally arises is to what extent sophisticated imputation algorithms (e.g., Neural Network based) are really needed, or we can obtain a descent performance using simple methods like Mean/Mode (MM). In this article, we experimentally compare six state-of-the-art representatives of different imputation algorithmic families from an AutoML predictive modeling perspective, including a feature selection step and combined algorithm and hyper-parameter selection. We used a commercial AutoML tool for our experiments, in which we included the selected imputation methods. Experiments ran on 25 binary classification real-world incomplete datasets with missing values and 10 binary classification complete datasets in which synthetic missing values are introduced according to different missingness mechanisms, at varying missing frequencies. The main conclusion drawn from our experiments is that the best method on average is the Denoise AutoEncoder on real-world datasets and the MissForest in simulated datasets, followed closely by MM. In addition, binary indicator variables encoding missingness patterns actually improve predictive performance, on average. Last, although there are cases where Neural-Network-based imputation significantly improves predictive performance, this comes at a great computational cost and requires measuring all feature values to impute new samples.

ML2DAC: Meta-Learning to Democratize AutoML for Clustering Analysis

Analysts often struggle with the combined algorithm selection and hyperparameter optimization problem, a.k.a. CASH problem in literature. Typically, they execute several algorithms with varying hyperparameter settings to find configurations that show valuable results. Efficiently finding these configurations is a major challenge. In clustering analyses, analysts face the additional challenge to select a cluster validity index that allows them to evaluate clustering results in a purely unsupervised fashion. Many different cluster validity indices exist and each one has its benefits depending on the dataset characteristics. While experienced analysts might address these challenges using their domain knowledge and experience, especially novice analysts struggle with them. In this paper, we propose a new meta-learning approach to address these challenges. Our approach uses knowledge from past clustering evaluations to apply strategies that experienced analysts would exploit. In particular, we use meta-learning to (a) select a suitable clustering validity index, (b) efficiently select well-performing clustering algorithm and hyperparameter configurations, and (c) reduce the search space to suitable clustering algorithms. In the evaluation, we show that our approach significantly outperforms state-of-the-art approaches regarding accuracy and runtime.

Concrete compressive strength prediction using an explainable boosting machine model

The mixing ratio of the raw materials has a significant impact on concrete compressive strength. Although the compressive strength of concrete can be inferred from the mix ratio, it is frequently challenging to determine how each mix ratio parameter affects the results. In this study, an Explainable Boosting Machine (EBM) is applied to predict the concrete compressive strength and explain the contribution of mix ratio factors on the compressive strength. Meanwhile, the combined algorithm selection and hyperparameter optimization problem in machine learning is implemented by employing a Bayesian optimization technique. A dataset consisting of 1030 concrete compressive strength test data has been used for model building. The results show that the Bayesian optimization algorithm iteratively constructs the algorithmic/hyperparametric model and identifies the optimal point in the space, significantly reducing the time consumption in the ML model building process. In terms of model prediction performance, the EBM algorithm shows excellent results with R2= 0.93, RMSE = 4.33, and MAE = 3.10. The EBM algorithm allows one to fully interpret the contribution of individual features to the prediction results from both global and local aspects, by which the impact of each mix ratio parameter on the compression strength of the concrete can be further determined.

Performance, Efficiency, and Target Setting for Bank Branches: Time Series With Automated Machine Learning

Setting targets for the bank branches and distribution of annual targets to the branches and portfolio managers, quarterly, is a crucial process for decision making and strategic planning in the banking industry. Performance of the bank branches and portfolio managers are also evaluated by the quarterly divided targets to the branches and portfolio managers. In this study, the focus is on performance prediction by using state-of-art machine learning algorithms. A novel automated machine learning approach with combined algorithm selection and hyperparameter optimization is also applied for each of the branches since all the branches might have different customer segmentation and behavior. Moreover, the postconditions are executed to finalize the target calculation and distribution over the performance predictions. The study shows the success of the methodology with a successful application of 98% accuracy in the prediction and majority of branch target calculations. An end-to-end solution found to the seasonality and periodicity problem, which is the biggest problem faced by branches while achieving their goals. Also, the novel approach increases the success of branch targets by 10% in overall. The most significant innovation this study provides to the literature and practitioners is that, unlike classical studies, it solves the seasonality and periodicity problem through multiple time series modeling. The target setting procedure was employed by the largest financial institution in Turkey, Ziraat Bank, to evaluate the operating performance of its branches. The empirical study demonstrates the applicability of the proposed model in the banking sector. The outputs of the study are implemented in real life for all retail branches of Ziraat Bank. In addition, the study awarded the most innovative use of AI/ML, the most innovative project for in-house implementation related to the innovative aspect of the work, by the Global FinTech Innovation Awards 2022.

Latent feature learning via autoencoder training for automatic classification configuration recommendation

The Combined Algorithm Selection and Hyperparameter Optimization problem, in short, CASH, seeks the most suitable classifiers and hyperparameters for the underlying classification problems. In current literature, the common approaches in dealing with CASH problem are conducted via search-based methods such as sequential model-based optimization (SMBO) along with various active tests. Different from current existing approaches, in this paper, we propose a new method by incorporating the so-called denoising autoencoder (DAE) approach into meta-learning (MtL) for automatic configuration (both algorithms and their hyperparameters) recommendation, which appears to be quite effective compared to standard search-based approaches. More specifically, we set up the configuration search space for CASH and produce the metadata, and generate the classification performance on a set of collected historical datasets. Then both encoder and decoder in the DAE system are trained with the masked metadata as inputs and the unmasked metadata as targets to extract the subtle latent variables of metadata and recover the unmasked inputs subsequently. Under our framework, the performance over the entire configuration space can be predicted effectively through two different settings, and the configuration with the highest predictive performance is thus recommended. The first recommendation approach is by inactivating some inputs and then to recover their entries via the trained encoder and decoder for new problems, while in the second approach, the relationship between the acquired latent variables and the meta-features of historical datasets via kernel multivariate multiple regression (MMR) is enacted, leading to the performance estimation of new datasets being pursued directly through MMR and the decoder of DAE without requiring any new configuration evaluations. An automatic classification configuration recommendation system, including 81 historical problems and 11 common classifiers with a total of 4983 configurations, is established to show the effectiveness of our proposed approach. The comparative results on 45 testing problems demonstrate that our proposed model has the superior recommendation capacity in terms of the baselines for existing MtL as well as other search-based approaches.

An automated machine learning approach for earthquake casualty rate and economic loss prediction

This study presents an automated machine learning (AutoML) framework to predict the casualty rate and direct economic loss induced by earthquakes. The AutoML framework enables automated combined algorithm selection and hyperparameter tuning (CASH), reducing the manual works in the model development. The proposed AutoML framework includes 5 modules: data collection, data preprocessing, CASH, loss prediction, and model analysis. The AutoML models are learned from the dataset that is composed of earthquake information and social indicators. The optimal algorithm and hyperparameter setting of models are determined by the CASH module. A two-step model including a classifier and a regression model is designed for the casualty rate to address zero-casualty cases and also minimize their impacts on data distribution. The proposed AutoML framework is implemented on the seismic loss dataset of mainland China to demonstrate its practicability. A comparison study is conducted to show the high predictive abilities of the AutoML model compared with the traditional seismic risk model and other AutoML models. Models learned from the complete dataset achieve the ultimate performance compared with subsets that are composed of partial features. The model interpretation results indicate that earthquake magnitude, position, and population density are leading indicators for loss prediction.

Auto-CASH: A meta-learning embedding approach for autonomous classification algorithm selection

With years of development, machine learning algorithms have excellent performance in some tasks of data analysis and data mining. To apply machine learning to new tasks, suitable algorithm and hyperparameters selection techniques, which is known as Combined Algorithm Selection and Hyperparameter optimization problem, are in demand. In the field of data analysis, how to automate the algorithm selection process has become a hot research topic in recent years. Most of the existing approaches are developed under the background of Automated Machine Learning with high time or space complexity. To alleviate the issue, an approach extracts and learns from prior experience based on meta-learning theory named Auto-CASH is proposed in this paper. One of the major drawbacks of existing meta-learning methods is that they rely too much on human expertise to extract and filter knowledge that guides subsequent training. Auto-CASH can automatically select features of tasks by introducing a reinforcement learning strategy. Thus Auto-CASH becomes less dependent on human expertise. Besides, two pruning strategies when processing Hyperparameter Optimization to improve efficiency are firstly proposed. Extensive experiments on classification tasks are conducted and results demonstrate that Auto-CASH outperforms state-of-the-art CASH approaches and popular AutoML systems with less time cost.

An extensive experimental evaluation of automated machine learning methods for recommending classification algorithms

This paper presents an experimental comparison among four automated machine learning (AutoML) methods for recommending the best classification algorithm for a given input dataset. Three of these methods are based on evolutionary algorithms (EAs), and the other is Auto-WEKA, a well-known AutoML method based on the combined algorithm selection and hyper-parameter optimisation (CASH) approach. The EA-based methods build classification algorithms from a single machine learning paradigm: either decision-tree induction, rule induction, or Bayesian network classification. Auto-WEKA combines algorithm selection and hyper-parameter optimisation to recommend classification algorithms from multiple paradigms. We performed controlled experiments where these four AutoML methods were given the same runtime limit for different values of this limit. In general, the difference in predictive accuracy of the three best AutoML methods was not statistically significant. However, the EA evolving decision-tree induction algorithms has the advantage of producing algorithms that generate interpretable classification models and that are more scalable to large datasets, by comparison with many algorithms from other learning paradigms that can be recommended by Auto-WEKA. We also observed that Auto-WEKA has shown meta-overfitting, a form of overfitting at the meta-learning level, rather than at the base-learning level.

Efficient Automatic CASH via Rising Bandits

The Combined Algorithm Selection and Hyperparameter optimization (CASH) is one of the most fundamental problems in Automatic Machine Learning (AutoML). The existing Bayesian optimization (BO) based solutions turn the CASH problem into a Hyperparameter Optimization (HPO) problem by combining the hyperparameters of all machine learning (ML) algorithms, and use BO methods to solve it. As a result, these methods suffer from the low-efficiency problem due to the huge hyperparameter space in CASH. To alleviate this issue, we propose the alternating optimization framework, where the HPO problem for each ML algorithm and the algorithm selection problem are optimized alternately. In this framework, the BO methods are used to solve the HPO problem for each ML algorithm separately, incorporating a much smaller hyperparameter space for BO methods. Furthermore, we introduce Rising Bandits, a CASH-oriented Multi-Armed Bandits (MAB) variant, to model the algorithm selection in CASH. This framework can take the advantages of both BO in solving the HPO problem with a relatively small hyperparameter space and the MABs in accelerating the algorithm selection. Moreover, we further develop an efficient online algorithm to solve the Rising Bandits with provably theoretical guarantees. The extensive experiments on 30 OpenML datasets demonstrate the superiority of the proposed approach over the competitive baselines.

Weighted Sampling for Combined Model Selection and Hyperparameter Tuning

The combined algorithm selection and hyperparameter tuning (CASH) problem is characterized by large hierarchical hyperparameter spaces. Model-free hyperparameter tuning methods can explore such large spaces efficiently since they are highly parallelizable across multiple machines. When no prior knowledge or meta-data exists to boost their performance, these methods commonly sample random configurations following a uniform distribution. In this work, we propose a novel sampling distribution as an alternative to uniform sampling and prove theoretically that it has a better chance of finding the best configuration in a worst-case setting. In order to compare competing methods rigorously in an experimental setting, one must perform statistical hypothesis testing. We show that there is little-to-no agreement in the automated machine learning literature regarding which methods should be used. We contrast this disparity with the methods recommended by the broader statistics literature, and identify a suitable approach. We then select three popular model-free solutions to CASH and evaluate their performance, with uniform sampling as well as the proposed sampling scheme, across 67 datasets from the OpenML platform. We investigate the trade-off between exploration and exploitation across the three algorithms, and verify empirically that the proposed sampling distribution improves performance in all cases.

Automatic Composition and Optimization of Multicomponent Predictive Systems With an Extended Auto-WEKA

Composition and parameterization of multicomponent predictive systems (MCPSs) consisting of chains of data transformation steps are a challenging task. Auto-WEKA is a tool to automate the combined algorithm selection and hyperparameter (CASH) optimization problem. In this paper, we extend the CASH problem and Auto-WEKA to support the MCPS, including preprocessing steps for both classification and regression tasks. We define the optimization problem in which the search space consists of suitably parameterized Petri nets forming the sought MCPS solutions. In the experimental analysis, we focus on examining the impact of considerably extending the search space (from approximately 22,000 to 812 billion possible combinations of methods and categorical hyperparameters). In a range of extensive experiments, three different optimization strategies are used to automatically compose MCPSs for 21 publicly available data sets. The diversity of the composed MCPSs found is an indication that fully and automatically exploiting different combinations of data cleaning and preprocessing techniques is possible and highly beneficial for different predictive models. We also present the results on seven data sets from real chemical production processes. Our findings can have a major impact on the development of high-quality predictive models as well as their maintenance and scalability aspects needed in modern applications and deployment scenarios.

Reinforcement-based Method for Simultaneous Clustering Algorithm Selection and its Hyperparameters Optimization

A wide range of clustering algorithms exists, most of them expose many hyperparameters, on which clustering partition quality depends. Simultaneous algorithm (model) selection and its hyperparameters optimization is considered to be a sophisticated task, which is known according to some sources as combined algorithm selection and hyperparameter optimization. In this paper, we focus on problem of selecting a clustering algorithm and its hyperparameter vector simultaneously given a dataset in order to achieve the best partition quality. We propose a method for selecting a proper clustering algorithm and its hyperparameter vector using reinforcement learning. Instead of tuning hyperparameters for all available clustering algorithms and selecting one showing the best performance, we make them to compete for time that they can use for optimizing their own hyperparameters. In our algorithm, we use a framework for multi-armed bandit problem, which is a special case of reinforcement learning. Each clustering algorithm is considered as an arm in the multi-armed bandit setting, while assigning a time budget to optimize hyperparameters of a clustering algorithm is considered as playing the corresponding arm. We conducted series of experiments for comparing out reinforcement learning approach to the classical exhaustive search approach. We conducted experiments on 20 datasets from UCI Repository such as Iris, haberman, krvskp, glass and other. We use 19 cluster validity indices to validate the clusters, built by selected and configured algorithm. As a hyperparameter optimization algorithm, we used SMAC. Our approach managed to improve model selection and hyperparameter optimization process, by sustaining the exploration-exploitation trade-off and spending available time budget more wisely.

Hybrid analysis for indicating patients with breast cancer using temperature time series

Breast cancer is the most common cancer among women worldwide. Diagnosis and treatment in early stages increase cure chances. The temperature of cancerous tissue is generally higher than that of healthy surrounding tissues, making thermography an option to be considered in screening strategies of this cancer type. This paper proposes a hybrid methodology for analyzing dynamic infrared thermography in order to indicate patients with risk of breast cancer, using unsupervised and supervised machine learning techniques, which characterizes the methodology as hybrid. The dynamic infrared thermography monitors or quantitatively measures temperature changes on the examined surface, after a thermal stress. In the dynamic infrared thermography execution, a sequence of breast thermograms is generated. In the proposed methodology, this sequence is processed and analyzed by several techniques. First, the region of the breasts is segmented and the thermograms of the sequence are registered. Then, temperature time series are built and the k-means algorithm is applied on these series using various values of k. Clustering formed by k-means algorithm, for each k value, is evaluated using clustering validation indices, generating values treated as features in the classification model construction step. A data mining tool was used to solve the combined algorithm selection and hyperparameter optimization (CASH) problem in classification tasks. Besides the classification algorithm recommended by the data mining tool, classifiers based on Bayesian networks, neural networks, decision rules and decision tree were executed on the data set used for evaluation. Test results support that the proposed analysis methodology is able to indicate patients with breast cancer. Among 39 tested classification algorithms, K-Star and Bayes Net presented 100% classification accuracy. Furthermore, among the Bayes Net, multi-layer perceptron, decision table and random forest classification algorithms, an average accuracy of 95.38% was obtained.

Initializing Bayesian Hyperparameter Optimization via Meta-Learning

Model selection and hyperparameter optimization is crucial in applying machine learning to a novel dataset. Recently, a subcommunity of machine learning has focused on solving this problem with Sequential Model-based Bayesian Optimization (SMBO), demonstrating substantial successes in many applications. However, for computationally expensive algorithms the overhead of hyperparameter optimization can still be prohibitive. In this paper we mimic a strategy human domain experts use: speed up optimization by starting from promising configurations that performed well on similar datasets. The resulting initialization technique integrates naturally into the generic SMBO framework and can be trivially applied to any SMBO method. To validate our approach, we perform extensive experiments with two established SMBO frameworks (Spearmint and SMAC) with complementary strengths; optimizing two machine learning frameworks on 57 datasets. Our initialization procedure yields mild improvements for low-dimensional hyperparameter optimization and substantially improves the state of the art for the more complex combined algorithm selection and hyperparameter optimization problem.