Meta-learning System Research Articles

A scalable AutoML approach based on graph neural networks

AutoML systems build machine learning models automatically by performing a search over valid data transformations and learners, along with hyper-parameter optimization for each learner. Many AutoML systems use meta-learning to guide search for optimal pipelines. In this work, we present a novel meta-learning system called KGpip which (1) builds a database of datasets and corresponding pipelines by mining thousands of scripts with program analysis, (2) uses dataset embeddings to find similar datasets in the database based on its content instead of metadata-based features, (3) models AutoML pipeline creation as a graph generation problem, to succinctly characterize the diverse pipelines seen for a single dataset. KGpip's meta-learning is a sub-component for AutoML systems. We demonstrate this by integrating KGpip with two AutoML systems. Our comprehensive evaluation using 121 datasets, including those used by the state-of-the-art systems, shows that KGpip significantly outperforms these systems.

EW-CACTUs-MAML: A Robust Metalearning System for Rapid Classification on a Large Number of Tasks

This study aims to develop a robust metalearning system for rapid classification on a large number of tasks. The model-agnostic metalearning (MAML) with the CACTUs method (clustering to automatically construct tasks for unsupervised metalearning) is improved as EW-CACTUs-MAML after integrated with the entropy weight (EW) method. Few-shot mechanisms are introduced in the deep network for efficient learning of a large number of tasks. The process of implementation is theoretically interpreted as “gene intelligence.” Validation of EW-CACTUs-MAML on a typical dataset (Omniglot) indicates an accuracy of 97.42%, performing better than CACTUs-MAML (validation accuracy = 97.22%). At the end of this paper, the availability of our thoughts to improve another metalearning system (EW-CACTUs-ProtoNets) is also preliminarily discussed based on a cross-validation on another typical dataset (Miniimagenet).

Cost-sensitive meta-learning framework

PurposeThis paper aims to describe the use of a meta-learning framework for recommending cost-sensitive classification methods with the aim of answering an important question that arises in machine learning, namely, “Among all the available classification algorithms, and in considering a specific type of data and cost, which is the best algorithm for my problem?”Design/methodology/approachThis paper describes the use of a meta-learning framework for recommending cost-sensitive classification methods for the aim of answering an important question that arises in machine learning, namely, “Among all the available classification algorithms, and in considering a specific type of data and cost, which is the best algorithm for my problem?” The framework is based on the idea of applying machine learning techniques to discover knowledge about the performance of different machine learning algorithms. It includes components that repeatedly apply different classification methods on data sets and measures their performance. The characteristics of the data sets, combined with the algorithms and the performance provide the training examples. A decision tree algorithm is applied to the training examples to induce the knowledge, which can then be used to recommend algorithms for new data sets. The paper makes a contribution to both meta-learning and cost-sensitive machine learning approaches. Those both fields are not new, however, building a recommender that recommends the optimal case-sensitive approach for a given data problem is the contribution. The proposed solution is implemented in WEKA and evaluated by applying it on different data sets and comparing the results with existing studies available in the literature. The results show that a developed meta-learning solution produces better results than METAL, a well-known meta-learning system. The developed solution takes the misclassification cost into consideration during the learning process, which is not available in the compared project.FindingsThe proposed solution is implemented in WEKA and evaluated by applying it to different data sets and comparing the results with existing studies available in the literature. The results show that a developed meta-learning solution produces better results than METAL, a well-known meta-learning system.Originality/valueThe paper presents a major piece of new information in writing for the first time. Meta-learning work has been done before but this paper presents a new meta-learning framework that is costs sensitive.

ProMetaUS: A proactive meta-learning uncertainty-based framework to select models for Dynamic Risk Management

Safety managers, practitioners, and researchers can employ different models for estimating and assessing hazards, consequences, likelihoods, risks, and/or mitigation measures in the safety field. The selection of a specific model may depend on the uncertainty associated with its estimation and its impact on the safety-related decision-making process. The recognition of this issue as an example of Algorithm Selection Problem (ASP) allows investigating the applicability of meta-learning principles that are scarcely adopted in the risk and safety literature. Consequently, we propose a novel meta-learning inspired framework to proactively rank a set of candidate models for Dynamic Risk Management (DRM) based on desired uncertainty conditions. We denominate this framework ProMetaUS (Proactive Meta-learning and Uncertainty-based Selection for dynamic risk management). To achieve this purpose, our meta-learning system acquires knowledge that relates the characteristics extracted both directly and indirectly from datasets (e.g. data-based, domain-based, simple and fast uncertainty-based, simple and fast sensitivity-based meta-features) to some performance measures of the models. Performance measures include confidence information, shape measurable quantities, safety decision criteria and threshold limits, and sensitivity analysis outputs. We tested the proposed framework in a case study about Oxygen Deficiency Hazard (ODH) assessment by means of @RISK. For each of the five datasets, single-performance measure rankings and a final ranking of the three models are generated. Such rankings are aggregated to obtain the global recommended ranking.

Boosting meta-learning with simulated data complexity measures

Meta-Learning has been largely used over the last years to support the recommendation of the most suitable machine learning algorithm(s) and hyperparameters for new datasets. Traditionally, a meta-base is created containing meta-features extracted from several datasets along with the performance of a pool of machine learning algorithms when applied to these datasets. The meta-features must describe essential aspects of the dataset and distinguish different problems and solutions. However, if one wants the use of Meta-Learning to be computationally efficient, the extraction of the meta-feature values should also show a low computational cost, considering a trade-off between the time spent to run all the algorithms and the time required to extract the meta-features. One class of measures with successful results in the characterization of classification datasets is concerned with estimating the underlying complexity of the classification problem. These data complexity measures take into account the overlap between classes imposed by the feature values, the separability of the classes and distribution of the instances within the classes. However, the extraction of these measures from datasets usually presents a high computational cost. In this paper, we propose an empirical approach designed to decrease the computational cost of computing the data complexity measures, while still keeping their descriptive ability. The proposal consists of a novel Meta-Learning system able to predict the values of the data complexity measures for a dataset by using simpler meta-features as input. In an extensive set of experiments, we show that the predictive performance achieved by Meta-Learning systems which use the predicted data complexity measures is similar to the performance obtained using the original data complexity measures, but the computational cost involved in their computation is significantly reduced.

Meta-learning in multivariate load demand forecasting with exogenous meta-features

Although many studies have examined various types of single load demand prediction algorithms, it is yet a challenging decision to select the best predictor. Geographical characteristics are one of the most effective factors in load consumption patterns, the performance of the algorithms, and the accuracy of predictions. Therefore, developing a general framework in which meteorological characteristics determine the best predictor can be of importance. Meta-learning models, which are less considered in this area, propose a framework for algorithm selection by using a set of meta-features. In this paper, a meta-learning system is developed using exogenous weather variables as meta-features. The proposed system selects the best predictor among a pool of candidate forecasting algorithms including ARIMA, ARIMAX, MLR, NAR, NARX, and SVR. In statistical approaches, the volatility of the mean and variance of the load demand is modeled by GARCH process. The selection process uses a set of meta-features built on temperature and dew point. The empirical tests were performed on 30 samples of daily load demand taken from ten various geographical zones in Ontario, Canada. The achievements reveal that the characteristics of weather meta-features such as range of variation, skewness, and kurtosis play an important role in assigning the most suitable predictor to each sample set and both groups of statistical and artificial intelligence-based predictors were selected in different sample sets.

Cross-domain Meta-learning for Time-series Forecasting

There are many algorithms that can be used for the time-series forecasting problem, ranging from simple (e.g. Moving Average) to sophisticated Machine Learning approaches (e.g. Neural Networks). Most of these algorithms require a number of user-defined parameters to be specified, leading to exponential explosion of the space of potential solutions. Since the trial-and-error approach to finding a good algorithm for solving a given problem is typically intractable, researchers and practitioners need to resort to a more intelligent search strategy, with one option being to constraint the search space using past experience - an approach known as Meta-learning. Although potentially attractive, Meta-learning comes with its own challenges. Gathering a sufficient number of Meta-examples, which in turn requires collecting and processing multiple datasets from each problem domain under consideration is perhaps the most prominent issue. In this paper, we are investigating the situations in which the use of additional data can improve performance of a Meta-learning system, with focus on cross-domain transfer of Meta-knowledge. A similarity-based cluster analysis of Meta-features has also been performed in an attempt to discover homogeneous groups of time-series with respect to Meta-learning performance. Although the experiments revealed limited room for improvement over the overall best base-learner, the Meta-learning approach turned out to be a safe choice, minimizing the risk of selecting the least appropriate base-learner.

Link Prediction Methods and Their Accuracy for Different Social Networks and Network Metrics

Currently, we are experiencing a rapid growth of the number of social-based online systems. The availability of the vast amounts of data gathered in those systems brings new challenges that we face when trying to analyse it. One of the intensively researched topics is theprediction of social connections between users. Although a lot of effort has been made to develop new prediction approaches, the existing methods are not comprehensively analysed. In this paper we investigate the correlation between network metrics and accuracy of different prediction methods. We selected six time-stamped real-world social networks and ten most widely used link prediction methods. The results of the experiments show that the performance of some methods has a strong correlation with certain network metrics. We managed to distinguish “prediction friendly” networks, for which most of the prediction methods give good performance, as well as “prediction unfriendly” networks, for which most of the methods result in high prediction error. Correlation analysis between network metrics and prediction accuracy of prediction methods may form the basis of a metalearning system where based on network characteristics it will be able to recommend the right prediction method for a given network.

A distributed meta-learning system for Chinese entity relation extraction

Entity relation extraction is an important task for obtaining useful information from multiple text documents. This paper presents a distributed meta-learning method which incorporates the distributed system and the meta-learning strategy for Chinese entity relation extraction. At the basic level of the meta-learning, we construct a learner for each relation type and the basic learners are different with each other on account of different feature sets. Then the communication among these basic learners is set up to improve the performance. At the meta-level, the meta-learner is used to make decision based on the results of each basic learner. Experiments are carried out on Automatic Content Extraction Relation Detection and Characterization (ACE RDC) 2005 Chinese corpus and the results show that the F-score of our distributed meta-learning system is 69.81%, which is higher than that of baseline (the method based on Support Vector Machine (SVM) using composite kernel) by 1.31% and precedes over the state-of-the-art systems.

Cloud based metalearning system for predictive modeling of biomedical data.

Rapid growth and storage of biomedical data enabled many opportunities for predictive modeling and improvement of healthcare processes. On the other side analysis of such large amounts of data is a difficult and computationally intensive task for most existing data mining algorithms. This problem is addressed by proposing a cloud based system that integrates metalearning framework for ranking and selection of best predictive algorithms for data at hand and open source big data technologies for analysis of biomedical data.

Metalearning: a survey of trends and technologies

Metalearning attracted considerable interest in the machine learning community in the last years. Yet, some disagreement remains on what does or what does not constitute a metalearning problem and in which contexts the term is used in. This survey aims at giving an all-encompassing overview of the research directions pursued under the umbrella of metalearning, reconciling different definitions given in scientific literature, listing the choices involved when designing a metalearning system and identifying some of the future research challenges in this domain.

Ranking of Classifiers based on Dataset Characteristics using Active Meta Learning

Classification is a machine learning technique which is used to categorize the different input patterns into different classes. To select the best classifier for a given dataset is one of the critical issues in Classification. Using cross-validation approach, it is possible to apply candidate algorithms on a given dataset and best classifier is selected by considering various evaluation measures of Classification. But computational cost is significant. Meta Learning automates this process by acquiring knowledge in form of Meta-features and performance information of candidate algorithm on datasets and creates a Meta Knowledge Base. Once Meta Knowledge Base is generated, system uses k-Nearest Neighbor as a Meta Learner that identifies the most similar datasets to new dataset. But generation of Meta Example is a costly process due to a large number of candidate algorithms and datasets with different characteristics involved. So Active Learning is incorporated into Meta Learning System that reduces generation of Meta example and at the same time maintaining performance of candidate algorithms. Once the training phase is completed based on Active Meta Learning approach, ranking is provided based on Success Rate Ratio (SRR) method that considers accuracy as a performance evaluation measure.

Load forecasting using a multivariate meta-learning system

Although over a thousand scientific papers address the topic of load forecasting every year, only a few are dedicated to finding a general framework for load forecasting that improves the performance, without depending on the unique characteristics of a certain task such as geographical location. Meta-learning, a powerful approach for algorithm selection has so far been demonstrated only on univariate time-series forecasting. Multivariate time-series forecasting is known to have better performance in load forecasting. In this paper we propose a meta-learning system for multivariate time-series forecasting as a general framework for load forecasting model selection. We show that a meta-learning system built on 65 load forecasting tasks returns lower forecasting error than 10 well-known forecasting algorithms on 4 load forecasting tasks for a recurrent real-life simulation. We introduce new metafeatures of fickleness, traversity, granularity and highest ACF. The meta-learning framework is parallelized, component-based and easily extendable.

Automatic classifier selection for non-experts

Choosing a suitable classifier for a given dataset is an important part of developing a pattern recognition system. Since a large variety of classification algorithms are proposed in literature, non-experts do not know which method should be used in order to obtain good classification results on their data. Meta-learning tries to address this problem by recommending promising classifiers based on meta-features computed from a given dataset. In this paper, we empirically evaluate five different categories of state-of-the-art meta-features for their suitability in predicting classification accuracies of several widely used classifiers (including Support Vector Machines, Neural Networks, Random Forests, Decision Trees, and Logistic Regression). Based on the evaluation results, we have developed the first open source meta-learning system that is capable of accurately predicting accuracies of target classifiers. The user provides a dataset as input and gets an automatically created high-performance ready-to-use pattern recognition system in a few simple steps. A user study of the system with non-experts showed that the users were able to develop more accurate pattern recognition systems in significantly less development time when using our system as compared to using a state-of-the-art data mining software.

Advances in intelligent information processing

Advances in intelligent information processing