Unbalanced Training Dataset Research Articles

Improving Training Dataset Balance with ChatGPT Prompt Engineering

The rapid evolution of large language models, in particular OpenAI’s GPT-3.5-turbo and GPT-4, indicates a growing interest in advanced computational methodologies. This paper proposes a novel approach to synthetic data generation and knowledge distillation through prompt engineering. The potential of large language models (LLMs) is used to address the problem of unbalanced training datasets for other machine learning models. This is not only a common issue but also a crucial determinant of the final model quality and performance. Three prompting strategies have been considered: basic, composite, and similarity prompts. Although the initial results do not match the performance of comprehensive datasets, the similarity prompts method exhibits considerable promise, thus outperforming other methods. The investigation of our rebalancing methods opens pathways for future research on leveraging continuously developed LLMs for the enhanced generation of high-quality synthetic data. This could have an impact on many large-scale engineering applications.

Federated learning for IoT devices: Enhancing TinyML with on-board training

The spread of the Internet of Things (IoT) involving an uncountable number of applications, combined with the rise of Machine Learning (ML), has enabled the rapid growth of pervasive and intelligent systems in a variety of domains, including healthcare, environment, railway transportation and Industry 4.0. While this opens up favorable scenarios, it also raises new challenges. The huge amount of data collected and processed by ML applications requires efficient and scalable solutions that contrast with the constrained capabilities of IoT devices as for memory, power consumption, processing and network bandwidth. The TinyML technologies foster the adoption of ML algorithms running locally on IoT devices. However, they typically foresee a remote training process (e.g., on cloud servers) combined with local inference – a strategy not always viable, e.g., for privacy and security issues.We present a technique to enable the on-board training of ML algorithms on IoT devices, through the combination of federated learning (FL) and transfer learning (TL). We experimentally analyze it in classification and regression problems, comparing it to traditional FL solutions, as well as with a consolidated technique based on Tensorflow Lite. Results show that FL with TL reaches accuracy values better than FL without TL in both classification (86.48%) and regression (0.0201). These results are comparable with a model trained on the full dataset. We further analyze training and inference time and power consumption on various devices. Finally, we evaluate how the performance changes with unbalanced training datasets, showing that although they strongly impact accuracy, FL makes models more robust, letting them achieve accuracy comparable to when trained on balanced datasets.

Evaluating the impact of generative adversarial models on the performance of anomaly intrusion detection

AbstractWith the increasing rate and types of cyber attacks against information systems and communication infrastructures, many tools are needed to detect and mitigate against such attacks, for example, Intrusion Detection Systems (IDSs). Unfortunately, traditional Signature‐based IDSs (SIDSs) perform poorly against previously unseen adversarial attacks. Anomaly‐based IDSs (AIDSs) use Machine Learning (ML) and Deep Learning (DL) approaches to overcome these limitations. However, AIDS performance can be poor when trained on imbalanced datasets. To address the challenge of AIDS performance caused by these unbalanced training datasets, generative adversarial models are proposed to obtain adversarial attacks from one side and analyse their quality from another. According to extensive usage and reliability criteria for generative adversarial models in different disciplines, Generative Adversarial Networks (GANs), Bidirectional GAN (BiGAN), and Wasserstein GAN (WGAN) are employed to serve AIDS. The authors have extensively assessed their abilities and robustness to deliver high‐quality attacks for AIDS. AIDSs are constructed, trained, and tuned based on these models to measure their impacts. The authors have employed two datasets: NSL‐KDD and CICIDS‐2017 for generalisation purposes, where ML and DL approaches are utilised to implement AIDSs. Their results show that the WGAN model outperformed GANs and BiGAN models in binary and multiclass classifications for both datasets.

Tackling bias in the data for breast cancer prediction using machine learning-based decision support

In this study, a machine learning (ML)-based decision support approach was developed to identify breast cancer likelihood in patients, based on their background and physiological data. Two ML models, Naïve Bayes and Logistic Regression were used to evaluate the Breast Cancer Surveillance Consortium dataset that had about 9:1 ratio of non-cancer cases (‘Class 0’) to cancer cases (‘Class 1’). We manually built both balanced and unbalanced training datasets and a non-overlapping testing dataset using a stratified sampling method. For each model, we partitioned the prediction results on testing set into two groups, the ‘Agree’ group included cases where balanced and unbalanced ML predictions agreed, and the remaining cases come under ‘Disagree’ group. Sensitivity and Positive Predictive Value were used as the prediction performance measures. For Naïve Bayes, the sensitivity of Class 1 in regular versus ‘Agree’ group increased from 0.687 to 0.936 and for Logistic Regression, it increased from 0.358 to 0.8306. This indicates the ‘Agree’ group predictions were more accurate and could be labeled as high-confidence ML predictions. The ‘Agree’ group consisted of 89% cases in the testing set, so the improved prediction performance was applicable for a large portion of the testing dataset.

A federated approach to Android malware classification through Perm-Maps

In the last decades, mobile-based apps have been increasingly used in several application fields for many purposes involving a high number of human activities. Unfortunately, in addition to this, the number of cyber-attacks related to mobile platforms is increasing day-by-day. However, although advances in Artificial Intelligence science have allowed addressing many aspects of the problem, malware classification tasks are still challenging. For this reason, the following paper aims to propose new special features, called permission maps (Perm-Maps), which combine information related to the Android permissions and their corresponding severity levels. Such features have proven to be very effective in classifying different malware families through the usage of a convolutional neural network. Also, the advantages introduced by the Perm-Maps have been enhanced by a training process based on a federated logic. Experimental results show that the proposed approach achieves up to a 3% improvement in average accuracy with respect to J48 trees and Naive Bayes classifier, and up to 16% compared to multi-layer perceptron classifier. Furthermore, the combined use of Perm-Maps and federated logic allows dealing with unbalanced training datasets with low computational efforts.

Diagnosis of vertebral column pathologies using concatenated resampling with machine learning algorithms

Medical diagnosis through the classification of biomedical attributes is one of the exponentially growing fields in bioinformatics. Although a large number of approaches have been presented in the past, wide use and superior performance of the machine learning (ML) methods in medical diagnosis necessitates significant consideration for automatic diagnostic methods. This study proposes a novel approach called concatenated resampling (CR) to increase the efficacy of traditional ML algorithms. The performance is analyzed leveraging four ML approaches like tree-based ensemble approaches, and linear machine learning approach for automatic diagnosis of inter-vertebral pathologies with increased. Besides, undersampling, over-sampling, and proposed CR techniques have been applied to unbalanced training dataset to analyze the impact of these techniques on the accuracy of each of the classification model. Extensive experiments have been conducted to make comparisons among different classification models using several metrics including accuracy, precision, recall, and F1 score. Comparative analysis has been performed on the experimental results to identify the best performing classifier along with the application of the re-sampling technique. The results show that the extra tree classifier achieves an accuracy of 0.99 in association with the proposed CR technique.

Novel MOA Fault Detection Technology Based on Small Sample Infrared Image

This paper proposes a novel metal oxide arrester (MOA) fault detection technology based on a small sample infrared image. The research is carried out from the detection process and data enhancement. A lightweight MOA identification and location algorithm is designed at the edge, which can not only reduce the amount of data uploaded, but also reduce the search space of cloud algorithm. In order to improve the accuracy and generalization ability of the defect detection model under the condition of small samples, a multi-model fusion detection algorithm is proposed. Different features of the image are extracted by multiple convolutional neural networks, and then multiple classifiers are trained. Finally, the weighted voting strategy is used for fault diagnosis. In addition, the extended model of fault samples is constructed by transfer learning and deep convolutional generative adversarial networks (DCGAN) to solve the problem of unbalanced training data sets. The experimental results show that the proposed method can realize the accurate location of arrester under the condition of small samples, and after the data expansion, the recognition rate of arrester anomalies can be improved from 83% to 85%, showing high effectiveness and reliability.

AILC: Accelerate On-Chip Incremental Learning With Compute-in-Memory Technology

As AI applications become pervasive on edge device, incrementally learning new tasks is demanded for deep neural network (DNN) models. In this article, we proposed AILC, a compute-in-memory (CIM)-based accelerator for on-chip incremental learning using STT-MRAM technology. On the software side, a network-expansion-based low-precision training algorithm is proposed for incremental learning, where the loss function is modified to handle the unbalanced training dataset. On the hardware side, the detailed CIM accelerator design for incremental learning is illustrated. A workload-aware hardware resources assignment protocol is proposed to improve the throughput when the workload of weight gradient calculation is low. The software simulation results on CIFAR-100 dataset show that the proposed algorithm can effectively support incremental learning despite the device conductance variation exists. System-level benchmark shows that AILC could achieve 147×, 3.7×~28.7×, 2.05×~2.9× higher energy efficiency than Nvidia Titan-V GPU, RRAM-based CIM accelerators and edge TPU/GPU, respectively. Compared to the baselines, the throughput of AILC is improved by 2.0×~2.2× on average with the hardware resources assignment protocol, which results in 4.1×~21.4× higher throughput than edge TPU/GPU.

Network traffic classification using deep convolutional recurrent autoencoder neural networks for spatial–temporal features extraction

The right choice of features to be extracted from individual or aggregated observations is an extremely critical factor for the success of modern network traffic classification approaches based on machine learning. Such activity, usually in charge of the designers of the classification scheme is strongly related to their experience and skills, and definitely characterizes the whole approach, implementation strategy as well as its performance. The main aim of this work is supporting this process by mining new and more expressive, meaningful and discriminating features from the basic ones without human intervention. For this purpose, a novel autoencoder-based deep neural network architecture is proposed where multiple autoencoders are embedded with convolutional and recurrent neural networks to elicit relevant knowledge about the relations existing among the basic features (spatial-features) and their evolution over time (temporal-features). Such knowledge, consisting in new properties that are not immediately evident and better represent the most hidden and representative traffic dynamics can be successfully exploited by machine learning-based classifiers. Different network combinations are analyzed both from a theoretical perspective, and through specific performance evaluation experiments on a real network traffic dataset. We show that the traffic classifier obtained by stacking the autoencoder with a fully-connected neural network, achieves up to a 28% improvement in average accuracy over state-of-the-art machine learning-based approaches, up to a 10% over pure convolutional and recurrent stacked neural networks, and 18% over pure feed-forward networks. It is also able to maintain high accuracy even in the presence of unbalanced training datasets.

A Customized Convolutional Neural Network Model Integrated With Acceleration-Based Smart Insole Toward Personalized Foot Gesture Recognition

Foot gestures play an important role in human machine interface and also indicate the lower-limb motor functionalities. Lately, machine learning models have been integrated with different kinds of wearable devices for foot gesture recognition. However, most existing wearable devices lack of robustness and facileness and existing machine learning models suffer from inconsistent recognition accuracy due to limited and unbalanced training datasets and, therefore, do not work well for new users. To address this, a customized convolutional neural network model architecture is designed, and strategies for achieving personalized models are presented, considering potential fairness sensitive attributes of a diverse training group, including age, gender, body mass index (defined by height and weight), shoe size, and health conditions. The proposed model to be embodied into a flexible smart insole with only two accelerometers aims to recognize four types of foot gestures-toe tapping, heel tapping, foot kicking, and foot stepping-in a sitting posture at high accuracies. After the hyperparameters are well-tuned, average recognition accuracy reaches 92% in training sets, 88.05% in validation sets, 85.27% in test sets, and 83.09% in fivefold cross-validation. The evaluation results from both in-group and leave-one-subject-out show that our unified model is easily transformable for individual uses by further fine-tuning the key hyperparameters.

Detección de fraude financiero mediante redes neuronales de clasificación en un caso real español

Este análisis supone una primera aproximación a la implementación de modelos de redes neuronales al trabajo pericial para la detección de operaciones de fraude. Los datos analizados provienen de un caso real de blanqueo de capitales en el que se está colaborando con la Policía Nacional Española. En ellos se cuenta con información de operaciones contables individuales entre las que se cuenta con una proporción de operaciones bien identificadas como fraudulentas con la que es posible entrenar un modelo de clasificación. En este trabajo, tras describir brevemente la metodología utilizada y la estrategia de ajuste se obtiene un modelo con una capacidad predictiva reseñable, incluso con datos de entrenamiento fuertemente desequilibrados. Además, al aplicar técnicas de balanceado de los datos de entrenamiento (SMOTE) se obtiene un resultado que indicaría la viabilidad de este tipo de modelos como herramienta en la planificación y priorización de las tareas de investigación policial, ya que uno de los principales problemas de los investigadores expertos en estos delitos financieros es la incapacidad para traducir la gran cantidad de información que se deriva de las empresas implicadas en patrones de compra de los individuos claramente fraudulentos.

Fault prediction as a service in the smart factory: addressing common challenges for an effective implementation

Abstract Fault prediction in manufacturing systems has consistently been an important theme in engineering research. Data-driven methods to deliver this service are gaining momentum due to developments regarding information and communication technologies. Particularly, fault prediction may be interpreted as a supervised learning classification problem, in which algorithms trained by operational data gathered from the shop-floor are capable of informing managers whether a machine might enter in a failure state or not. Despite the relevance of this approach, implementations are hindered by several challenges. In this work, we review approaches aimed to deal with four of these challenges, namely: limited amount of training data, unbalanced training data sets, uncertainty regarding which variables should be monitored, and uncertainty regarding how exactly historical data should be employed in the algorithm’s training. To deal with training sets with limited size, learning procedures observed to perform well with a lower volume of training data can be used, such as the Random Forests technique. Alternatively, transfer learning techniques can be utilized to adapt models trained in a virtual domain with abundant synthetic data to the real manufacturing system domain. To deal with unbalance among classification classes, cost-sensitive learning methods can be employed to alter the penalties incurred when misclassifications occurs in the minority class. Alternatively, resampling methods can be applied before learning occurs. Lastly, both the decisions regarding which variables to track, and to what extent historical data should be included in the training process, can be addressed through the use of specific feature selection methods.

IRNA-m5C_NB: A Novel Predictor to Identify RNA 5-Methylcytosine Sites Based on the Naive Bayes Classifier

As one of the widespread RNA post-transcriptional modifications (PTCMs), 5-Methylcytosine (m5C) plays vital roles in better understanding of basic biological mechanisms and major disease treatments. In experiments, traditional high-throughput approaches to find m5C sites are usually expensive and laborious. Additionally, facing with a large number of RNA sequences, developing accurate computational methods to distinguish m5C and non-m5C sites is an efficient solution. Here we introduced a novel predictor, called iRNA-m5C_NB, to identify m5C sites in Home sapiens using Naive Bayes (NB) algorithm. In this method, unbalanced dataset Met935 is firstly analyzed using efficient hybrid-sampling strategy SMOTEEEN. Then top 57 features are selected by the ANOVA F-value from four kinds of well-performance feature extraction techniques, including Bi-profile Bayes (BPB), enhanced Nucleic Acid Composition (ENAC), electron-ion interaction pseudopotentials (EIIP) and mMGap_1. Based on the jackknife test, the evaluated recall for the unbalanced training dataset Met935 is up to 82.81% with MCC of 0.63. And for the independent dataset Test1157, the predictor still shows high recall of 70.06% and MCC of 0.34. It is the first m5C predictor constructed using the unbalanced dataset, and the recall scores are increased by 19.82% and 59.23% for jackknife and independent tests compared with the latest tool RNAm5CPred, respectively. We demonstrate that the proposed predictor iRNA-m5C_NB outperforms other state-of-art models, which hopes to be an efficient and reliable method to identify m5C sites.

Animal species recognition in the wildlife based on muzzle and shape features using joint CNN

Monitoring of animal behavior in the wild supposes the reliable techniques for their species recognition using, mainly, visual data captured by camera traps. In this paper, we propose to extent Convolutional Neural Network (CNN) VGG by three branches, two of which are VGG16 for the muzzle and part of shape recognition and one is VGG19 for the whole shape recognition. A necessity of such branched CNN structure is caused by great variety of the animal poses fixed by a camera trap. Also, here we met with an objective problem of the unbalanced dataset due to different behavior of animals in nature. Preliminary categorization procedure of images helps to obtain better recognition results. Experiments were conducted using the dataset obtained from Ergaki national park, Krasnoyarsky Kray, Russia, 2012-2018. The joint CNN shows good accuracy results on the balanced dataset achieving 80.6% Top-1 and 94.1% Top-5, respectively. In the case of the unbalanced training dataset, we obtained 38.7% Top-1 and 54.8% Top-5 accuracy.

Reservoir computing Echo State Network classifier training

Reservoir Computing (RC) is taking attention of neural networks structures developers because of machine learning algorithms are simple at the high level of generalization of the models. The approaches are numerous. RC can be applied to different architectures including recurrent neural networks with irregular connections that are called Echo State Networks (ESN). However, the existence of successful examples of chaotic sequences predictions does not provide successful method of multiple attribute objects classification. In this paper ESN binary classifiers are researched. We show that the reason of low precision of classification is using of unbalanced training dataset. Then the method to solve the problem is proposed. We propose to use stochastic perforation balancing algorithm on training data set and method of data temporalization. The resulting errors matrixes are pretty good. The proposed method is illustrated by the usage on synthetic data set. The features of ESN classifier are demonstrated in the case of anomaly events detection such as based on transaction attributes fraud detection.

Deep variance network: An iterative, improved CNN framework for unbalanced training datasets

Convolutional neural network (CNN) has demonstrated its superior ability to achieve amazing accuracy in computer vision field. Nevertheless, for practical domain-specific image recognition tasks, it still remains difficult to obtain massive high-quality labeled datasets due to the strong requirements for extensive, tedious manual processing. Inspired by the well-known observation that human brain can accurately recognize objects without relying on massive congeneric examples, we propose a novel deep variance network (DVN) to further enhance the generalization ability of CNN in this paper, which could still produce higher recognition accuracy even with unbalanced training datasets than original CNN. The key idea of our DVN is built upon the intrinsic exploitation of inter-class homogeneity and intra-class heterogeneity. Towards such goal, we make the first attempt to incorporate a hierarchical Bayesian model into the powerful CNN framework, which can transfer the joint feature distribution from certain object’s complete training dataset to other object’s incomplete training dataset in an iterative way. In each training cycle, the CNN-resulted features are clustered into discrimination-related subspaces to guide the learning and adaptive adjustment of homogeneity and heterogeneity over unbalanced training datasets. In practice, we furnish several state-of-the-art deep networks with our proposed DVN, and conduct extensive experiments and comprehensive evaluations over CIFAR-10, MNIST, and SVHN benchmarks. The experiments have shown that, most of the furnished deep networks can benefit from our DVN, wherein they gain at most 6.9% accuracy improvement over CIFAR-10 benchmark, 52.83% error reduction over MNIST benchmark, and an improvement of 6.2% over SVHN datasets.

Towards automatic wild animal monitoring: Identification of animal species in camera-trap images using very deep convolutional neural networks

Non-intrusive monitoring of animals in the wild is possible using camera trapping networks. The cameras are triggered by sensors in order to disturb the animals as little as possible. This approach produces a high volume of data (in the order of thousands or millions of images) that demands laborious work to analyze both useless (incorrect detections, which are the most) and useful (images with presence of animals). In this work, we show that as soon as some obstacles are overcome, deep neural networks can cope with the problem of the automated species classification appropriately. As case of study, the most common 26 of 48 species from the Snapshot Serengeti (SSe) dataset were selected and the potential of the Very Deep Convolutional neural networks framework for the species identification task was analyzed. In the worst-case scenario (unbalanced training dataset containing empty images) the method reached 35.4% Top-1 and 60.4% Top-5 accuracy. For the best scenario (balanced dataset, images containing foreground animals only, and manually segmented) the accuracy reached a 88.9% Top-1 and 98.1% Top-5, respectively. To the best of our knowledge, this is the first published attempt on solving the automatic species recognition on the SSe dataset. In addition, a comparison with other approaches on a different dataset was carried out, showing that the architectures used in this work outperformed previous approaches. The limitations of the method, drawbacks, as well as new challenges in automatic camera-trap species classification are widely discussed.

Selected aspects of prior and likelihood information for a Bayesian classifier in a road safety analysis

The development of the Bayesian logistic regression model classifying the road accident severity is discussed. The already exploited informative priors (method of moments, maximum likelihood estimation, and two-stage Bayesian updating), along with the original idea of a Boot prior proposal, are investigated when no expert opinion has been available. In addition, two possible approaches to updating the priors, in the form of unbalanced and balanced training data sets, are presented. The obtained logistic Bayesian models are assessed on the basis of a deviance information criterion (DIC), highest probability density (HPD) intervals, and coefficients of variation estimated for the model parameters. The verification of the model accuracy has been based on sensitivity, specificity and the harmonic mean of sensitivity and specificity, all calculated from a test data set. The models obtained from the balanced training data set have a better classification quality than the ones obtained from the unbalanced training data set. The two-stage Bayesian updating prior model and the Boot prior model, both identified with the use of the balanced training data set, outperform the non-informative, method of moments, and maximum likelihood estimation prior models. It is important to note that one should be careful when interpreting the parameters since different priors can lead to different models.

Source Camera Identification with Imbalanced Training Dataset

In this paper, we address the problem of unbalanced training dataset for source camera identification, namely, there are fewer training examples for some camera models compared to other camera models. A new source camera identification approach is proposed to alleviate the influence of imbalanced training dataset. In the proposed approach, firstly, we treat source camera identification as a multi-class classification problem, and decompose it into binary classification problems. After decomposing, the problem of imbalanced training dataset for multiclass classification is transformed to the problem of imbalanced training dataset for binary classification. Then, we incorporate SMOTE and AdaBoost algorithms to construct SVM ensemble to address the issue of imbalanced training dataset for binary classification. A number of experiments show the proposed approach can deal with the imbalanced training dataset effectively.