Top Decile Lift Research Articles

Investigating the beneficial impact of segmentation-based modelling for credit scoring

Due to its vital role in financial risk management, credit scoring has been investigated extensively in extant information systems studies. However, most credit scoring studies rely on one-size-fits-all classifiers with logistic regression (LR) as a popular benchmark. Moreover, extant literature largely focuses on predictive performance as an evaluation criterion. To find a better balance between predictive performance and interpretability though, the current study investigates the beneficial impact of segmentation-based modelling by benchmarking the logit leaf model (LLM) which is based on LR and decision trees. By a large experimental setup using a real-life credit scoring data set containing 65,536 active customers, we find that LLM is a viable classifier over its constituent parts, i.e., LR and decision trees, and is very competitive to state-of-the-art credit decision making techniques (neural networks, support vector machines, bagging, boosting and random forests) on three evaluation metrics (AUC, top-decile lift and profit). Furthermore, we show its extraordinary interpretability capacities by proposing a new visualization based on the LLM output. In sum, the excellence of the LLM as a classifier for credit decision making problems stems from its ability to combine strong predictive performance with interpretable insights that in turn can inform managerial decisions.

Exploration of machine learning methods for maritime risk predictions

ABSTRACT Maritime applications such as targeting ships for inspections, improved domain awareness, and dynamic risk exposure assessments for strategic planning all benefit from ship-specific incident probabilities. Using a unique and comprehensive global data set, of 1.2 million observations over the period from 2014 to 2020, this study explores the effectiveness and suitability of 144 model variants from the field of machine learning for eight incident endpoints of interest and evaluating over 580 covariates. Furthermore, the importance of covariates is examined and visualized. The results differ for each endpoint of interest but confirm that random forest methods can improve prediction capabilities. Based on out-of-sample evaluations for the year 2020, targeting the top 10% most risky vessels would improve predictions by a factor of 2.7 to 4.9 compared to random selection and based on the top decile lift. Balanced random forests and random forests with balanced training variants outperform regular random forests, for which the selected variants also depend on aggregation types. The most important covariate groups for predicting incident probabilities relate to beneficial ownership, the safety management company, and the size and age of the vessel, while the relevance of these factors remains similar across the different endpoints of interest.

Ensemble Methods in Customer Churn Prediction: A Comparative Analysis of the State-of-the-Art

In the past several single classifiers, homogeneous and heterogeneous ensembles have been proposed to detect the customers who are most likely to churn. Despite the popularity and accuracy of heterogeneous ensembles in various domains, customer churn prediction models have not yet been picked up. Moreover, there are other developments in the performance evaluation and model comparison level that have not been introduced in a systematic way. Therefore, the aim of this study is to perform a large scale benchmark study in customer churn prediction implementing these novel methods. To do so, we benchmark 33 classifiers, including 6 single classifiers, 14 homogeneous, and 13 heterogeneous ensembles across 11 datasets. Our findings indicate that heterogeneous ensembles are consistently ranked higher than homogeneous ensembles and single classifiers. It is observed that a heterogeneous ensemble with simulated annealing classifier selection is ranked the highest in terms of AUC and expected maximum profits. For accuracy, F1 measure and top-decile lift, a heterogenous ensemble optimized by non-negative binomial likelihood, and a stacked heterogeneous ensemble are, respectively, the top ranked classifiers. Our study contributes to the literature by being the first to include such an extensive set of classifiers, performance metrics, and statistical tests in a benchmark study of customer churn.

A decision support framework to incorporate textual data for early student dropout prediction in higher education

Managing student dropout in higher education is critical, considering its substantial impacts on students' lives, academic institutions, and society as a whole. Using predictive modeling can be instrumental for this task, as a means to identify dropouts proactively on the basis of student characteristics and their academic performance. To enhance these predictions, textual student feedback also might be relevant; this article proposes a hybrid decision support framework that combines predictive modeling with student segmentation efforts. A real-life data set from a French higher education institution, containing information of 14,391 students and 62,545 feedback documents, confirms the superior performance of the proposed framework, in terms of the area under the curve and top decile lift, compared with various benchmarks. In contributing to decision support system research, this study (1) proposes a new framework for automatic, data-driven segmentation of students based on textual data; (2) compares multiple text representation methods and confirms that incorporating student textual feedback data improves the predictive performance of student dropout models; and (3) establishes useful insights to help decision-makers anticipate and manage student dropout behaviors.

A Sampling-Based Stack Framework for Imbalanced Learning in Churn Prediction

Churn prediction is gaining popularity in the research community as a powerful paradigm that supports data-driven operational decisions. Datasets related to churn prediction are often skewed with imbalanced class distribution. Data-level solutions, like over-sampling and under-sampling, have been commonly used by researchers to address this problem. There are limited number of case studies that attempt to evolve these data-level solutions by integrating them with computationally advanced frameworks, like ensembles. Ensembles primarily employ algorithmic diversity using a fixed set of training instances to achieve superior performance. This study aims to introduce algorithmic diversity in ensembles by modifying the fixed set of training instances using diverse sampling strategies to increase predictive performance in imbalanced learning. Data is acquired from the world’s largest open hotel commerce platform company. A four-part series of experiments is conducted to analyze the effectiveness of sampling techniques and ensemble solutions on model performance. A new sampling-based stack framework called “Stacking of Samplers for Imbalanced Learning” is proposed. The framework combines the prediction capabilities of sampling solutions to stimulate the information gain of the meta features in ensemble. It is observed that the proposed framework leads to improvement in model performance with AUC of 86.4% and top-decile lift of 4.7 for customers of the hotel technology provider. Additionally, results show that the framework records a higher information gain for meta features used in a stack, compared to commonly used stack frameworks.

Spline-rule ensemble classifiers with structured sparsity regularization for interpretable customer churn modeling

An important business domain that relies heavily on advanced statistical- and machine learning algorithms to support operational decision-making is customer retention management. Customer churn prediction is a crucial tool to support customer retention. It allows an early identification of customers who are at risk to abandon the company and provides the ability to gain insights into why customers are at risk. Hence, customer churn prediction models should complement predictive performance with model insights. Inspired by their ability to reconcile strong predictive performance and interpretability, this study introduces rule ensembles and their extension, spline-rule ensembles, as a promising family of classification algorithms to the customer churn prediction domain. Spline-rule ensembles combine the flexibility of a tree-based ensemble classifier with the simplicity of regression analysis. They do, however, neglect the relatedness between potentially conflicting model components which can introduce unnecessary complexity in the models and compromises model interpretability. To tackle this issue, a novel algorithmic extension, spline-rule ensembles with sparse group lasso regularization (SRE-SGL) is proposed to enhance interpretability through structured regularization. Experiments on fourteen real-world customer churn data sets in different industries (i) demonstrate the superior predictive performance of spline-rule ensembles with sparse group lasso over a set well yet powerful benchmark methods in terms of AUC and top decile lift; (ii) show that spline-rule ensembles with sparse group lasso regularization significantly outperform conventional rule ensembles whilst performing at least as well as conventional spline-rule ensembles; and (iii) illustrate the interpretable nature of a spline-rule ensemble model and the advantage of structured regularization in SRE-SGL by means of a case study on customer churn prediction for a telecommunications company.

Predicting donation behavior: Acquisition modeling in the nonprofit sector using Facebook data

The purpose of this study is to demonstrate the value of Facebook data in predicting first-time donation behavior. More specifically, we provide evidence that Facebook data can be used as a valuable data source for nonprofit organizations in acquiring new donors. To do so, we evaluate three different dimensionality reduction techniques (i.e., singular value decomposition, non-negative matrix factorization, and latent Dirichlet allocation) over seven classification techniques (i.e., logistic regression, k-nearest neighbors, bagged trees, random forest, adaboost, extreme gradient boosting, and artificial neural networks) using five times twofold cross-validation. Next, we assess what type of Facebook data and which predictors are most important. The results indicate that we can predict first-time donation behavior based on Facebook data with high predictive performance. Our benchmark indicates that the combination of singular value decomposition and logistic regression outperforms all other analytical methodologies with an area under the receiver operating characteristic of 0.72 and a top decile lift of 3.33. The results show that Facebook pages and categories of Facebook pages are the most important data types. The most important predictors are dimensions related to age, education, residence, materialism, responsible consumption, and interest in nonprofits. The presented acquisition models can be used by nonprofit organizations to implement a one-to-one targeted marketing campaign towards Facebook fans. To the best of our knowledge, our study is the first to determine the predictive value of Facebook data for nonprofits in a real-life acquisition context.

Social Network Analysis and Churn Prediction in Telecommunications Using Graph Theory.

Due to telecommunications market saturation, it is very important for telco operators to always have fresh insights into their customer’s dynamics. In that regard, social network analytics and its application with graph theory can be very useful. In this paper we analyze a social network that is represented by a large telco network graph and perform clustering of its nodes by studying a broad set of metrics, e.g., node in/out degree, first and second order influence, eigenvector, authority and hub values. This paper demonstrates that it is possible to identify some important nodes in our social network (graph) that are vital regarding churn prediction. We show that if such a node leaves a monitored telco operator, customers that frequently interact with that specific node will be more prone to leave the monitored telco operator network as well; thus, by analyzing existing churn and previous call patterns, we proactively predict new customers that will probably churn. The churn prediction results are quantified by using top decile lift metrics. The proposed method is general enough to be readily adopted in any field where homophilic or friendship connections can be assumed as a potential churn driver.

Customer Churn Prediction with Feature Embedded Convolutional Neural Network: An Empirical Study in the Internet Funds Industry

In this paper, we investigated the customer churn prediction problem in the Internet funds industry. We designed a novel feature embedded convolutional neural networks (FE-CNN) method that can automatically learn features from both the dynamic customer behavioral data and static customer demographic data and can utilize the advantage of convolutional neural networks to automatically learn features that capture the structured information. Our results show that our FE-CNN model outperforms the other traditional machine learning models with hand-crafted features, such as logistic regression (LR), support vector machines (SVM), random forests (RF) and neural networks (NN) in terms of accuracy, area under the receiver operating characteristics curve (AUC) and top-decile lift. Furthermore, we found that after adding the demographic data feature to the basic CNN model, the performance of the FE-CNN model improved. Overall, we found that the FE-CNN is the most powerful way to solve the problem of customer churn prediction in the Internet funds industry. Our FE-CNN method can also be applied to other fields that have both dynamic data and static data.

A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees

Decision trees and logistic regression are two very popular algorithms in customer churn prediction with strong predictive performance and good comprehensibility. Despite these strengths, decision trees tend to have problems to handle linear relations between variables and logistic regression has difficulties with interaction effects between variables. Therefore a new hybrid algorithm, the logit leaf model (LLM), is proposed to better classify data. The idea behind the LLM is that different models constructed on segments of the data rather than on the entire dataset lead to better predictive performance while maintaining the comprehensibility from the models constructed in the leaves. The LLM consists of two stages: a segmentation phase and a prediction phase. In the first stage customer segments are identified using decision rules and in the second stage a model is created for every leaf of this tree. This new hybrid approach is benchmarked against decision trees, logistic regression, random forests and logistic model trees with regards to the predictive performance and comprehensibility. The area under the receiver operating characteristics curve (AUC) and top decile lift (TDL) are used to measure the predictive performance for which LLM scores significantly better than its building blocks logistic regression and decision trees and performs at least as well as more advanced ensemble methods random forests and logistic model trees. Comprehensibility is addressed by a case study for which we observe some key benefits using the LLM compared to using decision trees or logistic regression.

Identifying Soccer Players on Facebook Through Predictive Analytics

This study assesses the feasibility of identifying self-reported sports practitioners (soccer players) on Facebook. The main goal is to develop a system to support marketers with the decision as to which prospects to target for advertising purposes. To do so, we benchmark several algorithms (i.e., random forest, logistic regression, adaboost, rotation forest, neural networks, and kernel factory) using five times twofold cross-validation. To evaluate performance and variable importances, we build a fusion model, which combines the results of the other algorithms using the weighted average. This technique is also referred to as information-fusion sensitivity analysis. The results reveal that Facebook data provide a viable basis to come up with sports predictions as the predictive performance ranges from 72.01% to 80.43% for area under the receiver operating characteristic curve (AUC), from 81.96% to 83.95% for accuracy, and from 2.41 to 3.06 for top-decile lift. Our benchmark study indicates that stochastic adaboost, the fusion model, random forest, rotation forest, and regularized logistic regression are the best-performing algorithms. Furthermore, the results show that the most important variables are the average number of friends that play soccer, membership of a soccer group, and the number of favorite teams. We also assess the impact of our results on profitability by conducting a thorough sensitivity analysis. Our analysis reveals that our approach can be beneficial for a wide range of companies. The analysis and results in this study will assist sports brands with decisions regarding their implementation of targeted marketing approaches.

A comparative analysis of data preparation algorithms for customer churn prediction: A case study in the telecommunication industry

Data preparation is a process that aims to convert independent (categorical and continuous) variables into a form appropriate for further analysis. We examine data-preparation alternatives to enhance the prediction performance for the commonly-used logit model. This study, conducted in a churn prediction modeling context, benchmarks an optimized logit model against eight state-of-the-art data mining techniques that use standard input data, including real-world cross-sectional data from a large European telecommunication provider. The results lead to following conclusions. (i) Analysts better acknowledge that the data-preparation technique they choose actually affects churn prediction performance; we find improvements of up to 14.5% in the area under the receiving operating characteristics curve and 34% in the top decile lift. (ii) The enhanced logistic regression also is competitive with more advanced single and ensemble data mining algorithms. This article concludes with some managerial implications and suggestions for further research, including evidence of the generalizability of the results for other business settings.

An empirical evaluation of rotation-based ensemble classifiers for customer churn prediction

Several studies have demonstrated the superior performance of ensemble classification algorithms, whereby multiple member classifiers are combined into one aggregated and powerful classification model, over single models. In this paper, two rotation-based ensemble classifiers are proposed as modeling techniques for customer churn prediction. In Rotation Forests, feature extraction is applied to feature subsets in order to rotate the input data for training base classifiers, while RotBoost combines Rotation Forest with AdaBoost. In an experimental validation based on data sets from four real-life customer churn prediction projects, Rotation Forest and RotBoost are compared to a set of well-known benchmark classifiers. Moreover, variations of Rotation Forest and RotBoost are compared, implementing three alternative feature extraction algorithms: principal component analysis (PCA), independent component analysis (ICA) and sparse random projections (SRP). The performance of rotation-based ensemble classifier is found to depend upon: (i) the performance criterion used to measure classification performance, and (ii) the implemented feature extraction algorithm. In terms of accuracy, RotBoost outperforms Rotation Forest, but none of the considered variations offers a clear advantage over the benchmark algorithms. However, in terms of AUC and top-decile lift, results clearly demonstrate the competitive performance of Rotation Forests compared to the benchmark algorithms. Moreover, ICA-based Rotation Forests outperform all other considered classifiers and are therefore recommended as a well-suited alternative classification technique for the prediction of customer churn that allows for improved marketing decision making.

Hybrid Models Using Unsupervised Clustering for Prediction of Customer Churn

Churn management is one of the key issues handled by mobile telecommunication operators. Data mining techniques can help in the prediction of churn behavior of customers. Various supervised learning techniques have been used to study customer churn. However, research on the use of unsupervised learning techniques for prediction of churn is limited. In this article, we use two-stage hybrid models consisting of unsupervised clustering techniques and decision trees with boosting on two different data sets and evaluate the models in terms of top decile lift. We examine two different approaches for hybridization of the models for utilizing the results of clustering based on various attributes related to services usage and revenue contribution of customers. The results indicate that the use of clustering led to improved top decile lift for the hybrid models compared with the benchmark case when no clustering is used. It is also shown that using cluster labels as inputs to the decision trees is a preferred method of hybridization. Out of the five unsupervised clustering techniques used, none is found to dominate others. But interesting attributes and rules that can help marketing experts identify churners from the data are obtained from the best hybrid models.