Real-world Classification Problems Research Articles

Probabilistic Confusion Matrix: A Novel Method for Machine Learning Algorithm Generalized Performance Analysis

The paper addresses the issue of classification machine learning algorithm performance based on a novel probabilistic confusion matrix concept. The paper develops a theoretical framework which associates the proposed confusion matrix and the resulting performance metrics with the regular confusion matrix. The theoretical results are verified based on a wide variety of real-world classification problems and state-of-the-art machine learning algorithms. Based on the properties of the probabilistic confusion matrix, the paper then highlights the benefits of using the proposed concept both during the training phase and the application phase of a classification machine learning algorithm.

Image classification with symbolic hints using limited resources.

Typical machine learning classification benchmark problems often ignore the full input data structures present in real-world classification problems. Here we aim to represent additional information as "hints" for classification. We show that under a specific realistic conditional independence assumption, the hint information can be included by late fusion. In two experiments involving image classification with hints taking the form of text metadata, we demonstrate the feasibility and performance of the fusion scheme. We fuse the output of pre-trained image classifiers with the output of pre-trained text models. We show that calibration of the pre-trained models is crucial for the performance of the fused model. We compare the performance of the fusion scheme with a mid-level fusion scheme based on support vector machines and find that these two methods tend to perform quite similarly, albeit the late fusion scheme has only negligible computational costs.

Predicting insurance claims through a variety of data mining techniques: facing lots of missing values and moderate class-imbalanced levels

Abstract This paper copes with a real-world classification problem related to the management of claims received in an insurance company. The way to obtain the classifier is not easy due to the high amount of missing values as well as the inherent imbalanced scenario within class labels. Once the data partition has been done, the training set is submitted to an intensive double grid search in order to obtain the most promising type of missing value imputation approach and then a step ahead is done using the best method and it starts the next round of data mining strategies which now falls into data rebalancing umbrella. Again, a grid search from an undersampling and oversampling family with different settings is done taking into account only seen data. The training data obtained after the first grid search are now submitted to the second step according the second grid search in order to get the ready training set for the further classifier training. The main objective of the work is to find the best combination of data mining techniques that suits the data set with a pipeline containing two types of data preparation methods coming from different families. As an outcome, first the problem of the presence of missing values has been addressed and then the data rebalancing techniques has been applied. The study focuses on obtaining classifiers based on Bayesian and lazy approaches as well as decision trees, evaluated on metrics such as the area under the ROC curve (AUC), Cohen’s kappa, Accuracy and the F-measure, among others. The imputation by the mean the mode is preferable to the Expectation Maximization Imputation in the scenario faced in this paper taking into account that the amount of missing values is higher than a forty percent for many features.

Explainable prediction of node labels in multilayer networks: a case study of turnover prediction in organizations.

In real-world classification problems, it is important to build accurate prediction models and provide information that can improve decision-making. Decision-support tools are often based on network models, and this article uses information encoded by social networks to solve the problem of employer turnover. However, understanding the factors behind black-box prediction models can be challenging. Our question was about the predictability of employee turnover, given information from the multilayer network that describes collaborations and perceptions that assess the performance of organizations that indicate the success of cooperation. Our goal was to develop an accurate prediction procedure, preserve the interpretability of the classification, and capture the wide variety of specific reasons that explain positive cases. After a feature engineering, we identified variables with the best predictive power using decision trees and ranked them based on their added value considering their frequent co-occurrence. We applied the Random Forest using the SMOTE balancing technique for prediction. We calculated the SHAP values to identify the variables that contribute the most to individual predictions. As a last step, we clustered the sample based on SHAP values to fine-tune the explanations for quitting due to different background factors.

Tree boosting methods for balanced and imbalanced classification and their robustness over time in risk assessment

Most real-world classification problems deal with imbalanced datasets, posing a challenge for Artificial Intelligence (AI), i.e., machine learning algorithms, because the minority class, which is of extreme interest, often proves difficult to be detected. This paper empirically evaluates tree boosting methods' performance given different dataset sizes and class distributions, from perfectly balanced to highly imbalanced. For tabular data, tree-based methods such as XGBoost, stand out in several benchmarks due to detection performance and speed. Therefore, XGBoost and Imbalance-XGBoost are evaluated. After introducing the motivation to address risk assessment with machine learning, the paper reviews evaluation metrics for detection systems or binary classifiers. It proposes a method for data preparation followed by tree boosting methods including hyper-parameter optimization. The method is evaluated on private datasets of 1 thousand (K), 10K and 100K samples on distributions with 50, 45, 25, and 5 percent positive samples. As expected, the developed method increases its recognition performance as more data is given for training and the F1 score decreases as the data distribution becomes more imbalanced, but it is still significantly superior to the baseline of precision-recall determined by the ratio of positives divided by positives and negatives. Sampling to balance the training set does not provide consistent improvement and deteriorates detection. In contrast, classifier hyper-parameter optimization improves recognition, but should be applied carefully depending on data volume and distribution. Finally, the developed method is robust to data variation over time up to some point. Retraining can be used when performance starts deteriorating.

UNFIS: A Novel Neuro-Fuzzy Inference System with Unstructured fuzzy rules

An important constraint of Fuzzy Inference Systems (FIS) is their structured rules. Indeed, each rule should evaluate all input variables in its antecedent part. Thus, the length of all fuzzy rules and the number of input variables are equal. However, in many decision-making problems evaluating some conditions on a limited set of input variables is sufficient to decide properly (unstructured rules). Consequently, the FIS’s performance, generalizability, and interpretability are restricted by the aforementioned limitation. This study provides a neuro-fuzzy inference system that can create each fuzzy rule taking into account various sets of input variables in order to handle this problem. To realize this capability, a new fuzzy selector neuron with an adaptive parameter is proposed that can select input variables in the antecedent part of each fuzzy rule. Moreover, the consequent part of the Takagi–Sugeno–Kang FIS is properly changed to consider only the selected set of input variables. To learn the parameters of the proposed architecture, the Levenberg–Marquardt (LM) method is used for regression problems. For classification applications, a trust-region-based learning method (General quasi-Levenberg–Marquardt (GqLM)) is proposed to minimize cross-entropy in multiclass problems. The performance of the proposed method is compared with some related previous approaches in some real-world classification and regression problems. Based on these comparisons, the proposed method has better or very close performance with a parsimonious structure consisting of unstructured fuzzy rules.

Enhancing Energy Efficiency and Classification Modeling Through a Combined Approach of LightGBM and Stratified KFold Cross-Validation

This research article aims to develop a robust and interpretable model for energy efficiency and classification that overcomes the limitations of existing techniques. The study’s objectives are to address the challenges faced by the established methods such as linear regression, random forest, support vector machines, and others. These limitations include issues related to predictive accuracy, model interpretability, and computational efficiency. To bridge these gaps, we introduce a novel model, stratified KFold LightGBM ensemble (SKL-EEFE). SKL-EEFE leverages the strengths of stratified KFold cross-validation, LightGBM gradient boosting, domain-specific feature engineering, and ensemble learning to provide a comprehensive solution to classification tasks. Our key findings reveal that SKL-EEFE outperforms existing methods while maintaining model interpretability. The proposed model offers both accuracy and transparency, making it a valuable addition to the field of classification modeling. This research contributes to advancing the state of knowledge by offering an innovative approach to overcome the limitations of existing models and providing a practical solution for real-world classification problems.

Claude 2.0 large language model: Tackling a real-world classification problem with a new iterative prompt engineering approach

In the last year, Large Language Models (LLMs) have transformed the way of tackling problems, opening up new perspectives in various works and research fields, due to their ability to generate and understand human languages. In this regard, the recent release of Claude 2.0 has contributed to the processing of more complex prompts. In this scenario, the goal of this paper is to evaluate the effectiveness of Claude 2.0 in a specific classification task. In particular, we considered the Forest cover-type problem, concerning the prediction of a cover-type value according to the geospatial characterization of target worldwide areas. To this end, we propose a novel iterative prompt template engineering approach, which integrates files by exploiting prompts and evaluates the quality of responses provided by the LLM. Moreover, we conducted several comparative analyses to evaluate the effectiveness of Claude 2.0 with respect to online and batch learning models. The results demonstrated that, although some online and batch models performed better than Claude 2.0, the new iterative prompt engineering approach improved the quality of responses, leading to better performance with increases ranging from 14% to 32% in terms of accuracy, precision, recall, and F1-score.

Consumer Data Without Compromise: Integrating Differential Privacy and GANs for Privacy-Preserving Digital Marketing

In the rapidly evolving digital marketing landscape, the utilization of consumer data is essential for efficient targeting and personalization of marketing practices. However, the growing concerns regarding user privacy and stringent data protection regulations have created challenges in accessing and using consumer data for marketing purposes. This paper introduces a novel approach that leverages Differential Privacy and Conditional Tabular Generative Adversarial Networks (CTGAN) to address these privacy concerns while maintaining the efficacy of data-driven digital marketing strategies. Our approach amalgamates the strengths of Differential Privacy and CTGAN, applying differential privacy to the original dataset to ensure that extracted data cannot be tied back to individuals. We then train a CTGAN on an open marketing dataset to learn and generate synthetic data of close resemblance. Through extensive empirical analysis, we evaluate the fidelity, utility, and trade-offs of our approach, demonstrating its effectiveness in synthesizing non-Gaussian and multi-modal distributions, and its applicability in real-world classification problems. The research also highlights the complexity of hyperparameter tuning and the importance of a balanced approach in model training. Our findings contribute valuable insights to both the theoretical understanding of generative models and practical guidance for digital marketing practitioners

Learning Ordinal-Hierarchical Constraints for Deep Learning Classifiers.

Real-world classification problems may disclose different hierarchical levels where the categories are displayed in an ordinal structure. However, no specific deep learning (DL) models simultaneously learn hierarchical and ordinal constraints while improving generalization performance. To fill this gap, we propose the introduction of two novel ordinal-hierarchical DL methodologies, namely, the hierarchical cumulative link model (HCLM) and hierarchical-ordinal binary decomposition (HOBD), which are able to model the ordinal structure within different hierarchical levels of the labels. In particular, we decompose the hierarchical-ordinal problem into local and global graph paths that may encode an ordinal constraint for each hierarchical level. Thus, we frame this problem as simultaneously minimizing global and local losses. Furthermore, the ordinal constraints are set by two approaches ordinal binary decomposition (OBD) and cumulative link model (CLM) within each global and local function. The effectiveness of the proposed approach is measured on four real-use case datasets concerning industrial, biomedical, computer vision, and financial domains. The extracted results demonstrate a statistically significant improvement to state-of-the-art nominal, ordinal, and hierarchical approaches.

Many-Objective Jaccard-based Evolutionary Feature Selection for High-Dimensional Imbalanced Data Classification.

Filters and wrappers represent two mainstream approaches to feature selection (FS). Although evolutionary wrapper-based FS outperforms filters in addressing real-world classification problems, extending these methods to high-dimensional, many-objective optimization problems with imbalanced data poses substantial challenges. Overcoming computational costs and identifying suitable performance metrics are vital for navigating search operation complexities. Here, we propose using the Jaccard similarity (JS) in a set-based evolutionary many-objective (JSEMO) FS search, addressing both evolutionary FS and imbalanced classifier choice concurrently. This study highlights the mutual influence between these aspects, impacting overall algorithm performance. JSEMO integrates JS into population initialization, reproduction, and elitism steps, enhancing diversity and avoiding duplicate solutions. The set-based variation operator utilizes intersection and union operators for compatibility with binary coding. We also introduce a double-weighted KNN (KNN2W) classifier with four supportive objectives as a many-objective FS problem to handle imbalanced distributions. Compared with 20 methods on 15 benchmark problems, JSEMO produces distinct optimal features, significantly improving overall accuracy, balance accuracy, and g-mean metrics with comparable feature set size and computational cost. The ablation study underscores the positive impact of all JSEMO components, highlighting the set-based variation operation with JS and KNN2W with relevant evaluation metrics as the most influential aspects.

A high order fractal-based Kullback–Leibler divergence with application in classification

Dempster–Shafer evidence theory (DSET) is extensively employed in multi-source data fusion applications. Nonetheless, when belief probability assignments (BPAs) exhibit considerable conflict, unexpected results can occur. To address this limitation, the high-order fractals are explored and a K-order fractal-based Kullback–Leibler divergence (KO-FKL) is introduced, which defines the K-order as the optimal fractal epoch. This measure is employed to quantify the divergence between BPAs and demonstrates superior performance in assessing the conflict between two BPAs in numerical examples, compared to existing belief divergence methods. To utilize the KO-FKL divergence measure to real-world problems, a novel KO-FKL-based multi-source data fusion (KO-FKL-MSDF) algorithm is designed. Through comparisons with well-known related methods, our proposed KO-FKL-MSDF algorithm demonstrates superiority and enhanced robustness. Lastly, the KO-FKL-MSDF algorithm is applied to real-world classification problems, underlining its high practical applicability.

Hybrid feature ranking and classifier aggregation based on multi-criteria decision-making

This study introduces an ensemble methodology, namely, hybrid feature ranking and classifier aggregation (HyFraCa), to integrate ensemble feature selection and ensemble classification in a composite framework. The proposed HyFraCa is embedded in a multi-criteria decision-making (MCDM)-based scheme for feature ranking and classifier weighting, with an effective aggregation rule that yields a consensus feature ranking from ensembles of heterogeneous classifiers and feature selectors. Experimental evaluations on 20 public UCI datasets demonstrated the superiority of HyFraCa in producing a more accurate and generalizable classification compared with state-of-the-art benchmark ensemble methods. HyFraCa also provides robust and reliable consensus feature rankings, which are favorable for real-world classification problems in which feature interpretability is emphasized.

A classification method of fuzzy semi-supervised support vector machines for the problems of imbalance

Positive instances are often significantly less than negative instances in real-world classification problems. However, positive categories are typically more relevant to the primary focus of categorization tasks. Moreover, obtaining labeled data is often expensive, and the majority of real-life data is unlabeled. Therefore, semi-supervised learning has become a popular approach for addressing imbalanced problems. Traditional support vector machines (SVMs) treat all samples equally and are not suitable for semi-supervised learning. To address this issue, a semi-supervised model called the fuzzy semi-supervised SVM (FS3VM) has been proposed. The FS3VM model uses the degree of entropy-based fuzzy membership to ensure the materiality of positive classes by assigning positive instances to relatively large degrees of fuzzy membership. After introducing the mainstream FS3VM model, the fundamental theory and methods of the model are discussed and expanded upon, including the FS3VM algorithm, which applies the Sequential Minimal Optimization (SMO) algorithm to the dual problem. The proposed FS3VM model is a smooth and continuous optimization problem, and its dual is a standard quadratic programming. Experimental results demonstrate that the proposed FS3VM model outperforms other compared learning algorithms.

MBT: Model-Based Transformer for retinal optical coherence tomography image and video multi-classification

Background and objectiveThe detection of retinal diseases using optical coherence tomography (OCT) images and videos is a concrete example of a data classification problem. In recent years, Transformer architectures have been successfully applied to solve a variety of real-world classification problems. Although they have shown impressive discriminative abilities compared to other state-of-the-art models, improving their performance is essential, especially in healthcare-related problems. MethodsThis paper presents an effective technique named model-based transformer (MBT). It is based on popular pre-trained transformer models, particularly, vision transformer, swin transformer for OCT image classification, and multiscale vision transformer for OCT video classification. The proposed approach is designed to represent OCT data by taking advantage of an approximate sparse representation technique. Then, it estimates the optimal features, and performs data classification. ResultsThe experiments are carried out using three real-world retinal datasets. The experimental results on OCT image and OCT video datasets show that the proposed method outperforms existing state-of-the-art deep learning approaches in terms of classification accuracy, precision, recall, and f1-score, kappa, AUC-ROC, and AUC-PR. It can also boost the performance of existing transformer models, including Vision transformer and Swin transformer for OCT image classification, and Multiscale Vision Transformers for OCT video classification. ConclusionsThis work presents an approach for the automated detection of retinal diseases. Although deep neural networks have proven great potential in ophthalmology applications, our findings demonstrate for the first time a new way to identify retinal pathologies using OCT videos instead of images. Moreover, our proposal can help researchers enhance the discriminative capacity of a variety of powerful deep learning models presented in published papers. This can be valuable for future directions in medical research and clinical practice.

ENSEMBLE LEARNING APPROACHES FOR CLASSIFICATION WITH HIGH-DIMENSIONAL DATA

Classification with high-dimensional data is a significant challenge in machine learning because the abundance of features in high-dimensional data makes it difficult to identify meaningful patterns, which leads to overfitting and reduced classification performance. Moreover, the computational cost of processing high-dimensional data is often prohibitively expensive, requiring specialized hardware or optimized algorithms. Ensemble learning is a powerful machine learning technique that combines multiple models to improve classification accuracy. By aggregating the predictions of multiple models, ensemble learning can reduce overfitting, increase robustness, and improve performance on a wide range of real-world classification problems. Ensemble learning is effective for classification with high-dimensional data because it can combine multiple models to mitigate the effects of the curse of dimensionality, reduce overfitting, and enhance generalization performance. By using different learning algorithms or subsets of features, ensemble learning can improve the diversity of the models, leading to better overall performance on high-dimensional data. This paper proposes two hybrid ensemble machine learning approaches that integrate random subspace ensemble with bagging and boosting to enhance classification performance with high-dimensional data. Experimental results demonstrate that these methods significantly improve classification accuracy with highdimensional data.

A DEBIASING VARIATIONAL AUTOENCODER FOR DEFORESTATION MAPPING

Abstract. Deep Learning (DL) algorithms provide numerous benefits in different applications, and they usually yield successful results in scenarios with enough labeled training data and similar class proportions. However, the labeling procedure is a cost and time-consuming task. Furthermore, numerous real-world classification problems present a high level of class imbalance, as the number of samples from the classes of interest differ significantly. In various cases, such conditions tend to promote the creation of biased systems, which negatively impact their performance. Designing unbiased systems has been an active research topic, and recently some DL-based techniques have demonstrated encouraging results in that regard. In this work, we introduce an extension of the Debiasing Variational Autoencoder (DB-VAE) for semantic segmentation. The approach is based on an end-to-end DL scheme and employs the learned latent variables to adjust the individual sampling probabilities of data points during the training process. For that purpose, we adapted the original DB-VAE architecture for dense labeling in the context of deforestation mapping. Experiments were carried out on a region of the Brazilian Amazon, using Sentinel-2 data and the deforestation map from the PRODES project. The reported results show that the proposed DB-VAE approach is able to learn and identify under-represented samples, and select them more frequently in the training batches, consequently delivering superior classification metrics.

Continuous ordinal patterns: Creating a bridge between ordinal analysis and deep learning.

We introduce a generalization of the celebrated ordinal pattern approach for the analysis of time series, in which these are evaluated in terms of their distance to ordinal patterns defined in a continuous way. This allows us to naturally incorporate information about the local amplitude of the data and to optimize the ordinal pattern(s) to the problem under study. This last element represents a novel bridge between standard ordinal analysis and deep learning, allowing the achievement of results comparable to the latter in real-world classification problems while also retaining the conceptual simplicity, computational efficiency, and easy interpretability of the former. We test this through the use of synthetic time series, generated by standard chaotic maps and dynamical models, data sets representing brain activity in health and schizophrenia, and the dynamics of delays in the European air transport system. We further show how the continuous ordinal patterns can be used to assess other aspects of the dynamics, like time irreversibility.

A fractal belief KL divergence for decision fusion

Dempster–Shafer (D–S) evidence theory is useful in the realm of multi-source data fusion. However, a counterintuitive result may be obtained when the belief probability assignments (BPAs) are highly conflicting. To overcome this flaw, in this paper a symmetric fractal-based belief Kullback–Leibler divergence (FBDSKL) is proposed. It is used to measure the divergence between BPAs, and is more capable than the existing belief divergence methods in measuring the conflict between two BPAs in numerical examples. Furthermore, the proposed FBDSKL is proved to have desirable properties including nonnegativity, nondegeneracy and symmetry. To apply FBDSKL divergence measure to practical problems, a novel FBDSKL-based multi-source data fusion (FBDSKL-MSDF) algorithm is designed. Through comparisons with the well-known related methods, the proposed FBDSKL-MSDF algorithm is validated to be superior and more robust. Finally, the proposed FBDSKL-MSDF is applied to two real-world classification problems to verify its high practicability.

A Comparative Study of the Use of Stratified Cross-Validation and Distribution-Balanced Stratified Cross-Validation in Imbalanced Learning.

Nowadays, the solution to many practical problems relies on machine learning tools. However, compiling the appropriate training data set for real-world classification problems is challenging because collecting the right amount of data for each class is often difficult or even impossible. In such cases, we can easily face the problem of imbalanced learning. There are many methods in the literature for solving the imbalanced learning problem, so it has become a serious question how to compare the performance of the imbalanced learning methods. Inadequate validation techniques can provide misleading results (e.g., due to data shift), which leads to the development of methods designed for imbalanced data sets, such as stratified cross-validation (SCV) and distribution optimally balanced SCV (DOB-SCV). Previous studies have shown that higher classification performance scores (AUC) can be achieved on imbalanced data sets using DOB-SCV instead of SCV. We investigated the effect of the oversamplers on this difference. The study was conducted on 420 data sets, involving several sampling methods and the DTree, kNN, SVM, and MLP classifiers. We point out that DOB-SCV often provides a little higher F1 and AUC values for classification combined with sampling. However, the results also prove that the selection of the sampler-classifier pair is more important for the classification performance than the choice between the DOB-SCV and the SCV techniques.