A Classification Based Ensemble Pruning Framework with Multi-metric Consideration

Abstract

Machine learning techniques have been an essential driving force for Internet companies while handling daily tasks like fraud detection. Given a pre-designed model, several efforts, such as fine-tuning and ensemble techniques, can be taken to enhance its performance. However, more effective methods, with consideration of task-specific requirements, are always highly demanded. In this paper, we aim at efficiently obtaining an effective framework with consideration of multiple metrics, based on a pre-designed model. To achieve this, we build a framework which utilizes the ensemble pruning technique and adapts a classification based optimization method to perform the model pruning procedure. On one hand, to maintain the performance consistency of the pruned ensemble model between the selection and deployment stage, a calibration strategy is introduced to the model pruning (weight optimization) procedure. On the other hand, to meet the business requirement for multi-metric consideration, multiple evaluation metrics can be equipped in this framework, and we improve the optimization method to simultaneously optimize these metrics. Experiments are first conducted on several benchmark datasets, and the results show that this framework can effectively enhance the performance. We further apply the proposed framework to large-scale fraud detection tasks, which also validates its effectiveness.