Multi-class Classification Task Research Articles

A general maximal margin hyper-sphere SVM for multi-class classification

Traditional SVM algorithms for multi-class (k > 2 classes) classification tasks include “one-against-one”, “one-against-rest”, and “one-against-one-against-rest”, which build k(k−1)/2 or k classifiers for space partitioning and classification decision. However, they may cause a variety of problems, such as an imbalanced problem, a high temporal complexity, and trouble establishing the decision boundary. In this study, we use the notion of minimizing structural risks (SRM) to recognize k classes by designing only one optimization problem, which we call M3HS-SVM. The M3HS-SVM offers numerous benefits. In summary, the following points should be emphasized: (1) Rather than dividing the space with hyper-planes, M3HS-SVM describes the structural characteristics of various classes of data and trains the hyper-sphere classifier of each class based on the data distribution. (2) M3HS-SVM inherits all of the advantages of classical binary SVM, such as the maximization spirit, the use of kernel techniques to solve nonlinear separable problems, and excellent generalization ability. (3) In the dual problem, we develop an SMO algorithm to effectively reduce the complexity of time and space. We eventually validate the preceding statement with comprehensive experiments. The experiment findings show that our method outperforms other mainstream methods in terms of computing time and classification performance on synthetic datasets, UCI datasets, and NDC datasets.

Synthetic Data as a Proxy for Real-World Electronic Health Records in the Patient Length of Stay Prediction

While generative artificial intelligence has gained popularity, e.g., for the creation of images, it can also be used for the creation of synthetic tabular data. This bears great potential, especially for the healthcare industry, where data are often scarce and underlie privacy restrictions. For instance, the creation of synthetic electronic health records (EHR) promises to improve the usage of machine learning algorithms, which usually work with large amounts of data. This also applies for the prediction of the patient length of stay (LOS), a key measure for hospitals. Thereby, the LOS represents one of the core tools for decision makers to plan the allocation of resources. Thus, this paper aims to add to the still-young research concerning the application of generative adversarial nets (GAN) on tabular EHR. It does that with the intention to leverage the advantages of synthetic data for the prediction of the LOS in order to contribute to the efficiency-enhancing and cost-saving aspirations of hospitals and insurance companies. Therefore, the applicability of synthetic data that is generated using GANs as a proxy for scarce real-world EHR for the patient LOS multi-class classification task is examined. In this context, the Conditional Tabular GAN (CTGAN) and the Copula GAN are selected as the underlying models as they are state-of-the-art GAN architectures designed for generating synthetic tabular data. The CTGAN is found to be the superior model for the underlying use case. Nevertheless, the paper shows that there is still room for improvement when applying state-of-the-art GAN architectures to clinical healthcare data.

An adaptive error-correcting output codes algorithm based on gene expression programming and similarity measurement matrix

The multi-class classification task is one of the most common tasks in machine learning. As a typical solution based on a partitioning strategy, Error-Correcting Output Codes (ECOC) can transform a multi-class classification problem into multiple binary classification problems. The key of ECOC is to construct an effective codematrix to represent a set of class decomposition schemes, which transforms a multiclass problem into a group of binary class problems. Consequently, the design of a fast and effective ECOC codematrix generation method is of great research significance and value for solving multi-class classification problems. In ECOC algorithms, the design of codematrix is treated as a combination problem between different code columns, in which the evolutionary algorithm shows a great advantage. Based on this consideration, the Gene Expression Programming (GEP) is applied to search for the codematrix with high performance because its expressive tree structure makes it well represent codematrcies for subsequent optimization operations. This paper proposes an adaptive ECOC algorithm based on Gene Expression Programming (GEP) and similarity measurement matrix, named GEP-ECOC. In our GEP, each individual represents a set of columns to form a random ECOC codematrix, which is optimized in the evolutionary process. Meanwhile, the crossover and mutation operations are modified to include a legality checking process to ensure that the generated codematrix satisfies the ECOC constraints. The GEP-based ECOC codematrix generation algorithm can quickly produce a codematrix with better performance, which ensures the efficiency of the algorithm to a certain extent. In addition, an adaptive algorithm based on a similarity measurement matrix is proposed to add new columns to the current codematrix, aiming to better handle hard classes. Our algorithm is compared with other algorithms on various data sets, and the experimental results confirm that our GEP-ECOC can balance the efficiency and performance of the algorithm and achieve higher performance.

An extremely lightweight CNN model for the diagnosis of chest radiographs in resource-constrained environments.

In recent years, deep learning methods have been successfully used for chest x-ray diagnosis. However, such deep learning models often contain millions of trainable parameters and have high computation demands. As a result, providing the benefits of cutting-edge deep learning technology to areas with low computational resources would not be easy. Computationally lightweight deep learning models may potentially alleviate this problem. We aim to create a computationally lightweight model for the diagnosis of chest radiographs. Our model has only 0.14M parameters and 550 KB size. These make the proposed model potentially useful for deployment in resource-constrainedenvironments. We fuse the concept of depthwise convolutions with squeeze and expand blocks to design the proposed architecture. The basic building block of our model is called Depthwise Convolution In Squeeze and Expand (DCISE) block. Using these DCISE blocks, we design an extremely lightweight convolutional neural network model (ExLNet), a computationally lightweight convolutional neural network (CNN) model for chest x-raydiagnosis. We perform rigorous experiments on three publicly available datasets, namely, National Institutes of Health (NIH), VinBig ,and Chexpert for binary and multi-class classification tasks. We train the proposed architecture on NIH dataset and evaluate the performance on VinBig and Chexpert datasets. The proposed method outperforms several state-of-the-art approaches for both binary and multi-class classification tasks despite having a significantly less number ofparameters. We design a lightweight CNN architecture for the chest x-ray classification task by introducing ExLNet which uses a novel DCISE blocks to reduce the computational burden. We show the effectiveness of the proposed architecture through various experiments performed on publicly available datasets. The proposed architecture shows consistent performance in binary as well as multi-class classification tasks and outperforms other lightweight CNN architectures. Due to a significant reduction in the computational requirements, our method can be useful for resource-constrained clinical environment aswell.

The unified effect of data encoding, ansatz expressibility and entanglement on the trainability of HQNNs

Recent advances in quantum computing and machine learning have brought about a promising intersection of these two fields, leading to the emergence of quantum machine learning (QML). However, the integration of quantum computing and machine learning poses several challenges. One of the prominent challenges lies in the presence of barren plateaus (BP) in QML algorithms, particularly in quantum neural networks (QNNs). Recent studies have successfully identified the fundamental causes underlying the existence of BP in QNNs. This paper presents a framework designed to explore the interplay of multiple factors contributing to the BP problem in quantum neural networks (QNNs), which poses a critical challenge for the practical applications of QML. We focus on the combined influence of data encoding, qubit entanglement, and ansatz expressibility in hybrid quantum neural networks (HQNNs) for multi-class classification tasks. Our framework aims to empirically analyze the joint impact of these factors on the training landscape of HQNNs. Our results show that the occurrence of the BP problem in HQNNs is contingent upon the expressibility of the underlying ansatz and the type of the adopted data encoding technique. Additionally, we observe that qubit entanglement also plays a role in exacerbating the BP problem. Leveraging various evaluation metrics for classification tasks, we systematically evaluate the performance of HQNNs and provide recommendations tailored to different constraint scenarios. Our findings emphasize the significance of our framework in addressing the practical success of QNNs.

Convolutional neural networks applied to microtomy: Identifying the trimming-end cutting routine on paraffin-embedded tissue blocks

In the field of histopathology, the microtomy procedure yields thin sections of tissue embedded in paraffin blocks, then to be further processed for diagnostic purposes. Within microtomy, trimming is an initial but critical process in which the excess paraffin covering the tissue of interest is removed by continuous cutting routines, until the tissue is suitably exposed and ready for sectioning. Trimming is currently a time-consuming process that is manually held by technicians. In this paper, we present a method to automatize this process, by analyzing tissue block surface images resulting from each cyclic cutting routine. Two types of Convolutional Neural Networks (CNNs) were fine-tuned: one for binary segmentation, the other for multi-class classification tasks, by exploring and optimizing lightweight architectures to provide fast analytical results on cost-effective edge computing. Two sequential online conditions followed the CNNs to output the current stage of the block surface and rule if the trimming-end cutting routine was reached. We compared the results obtained through our method with the ones obtained by three skilled technicians processing 75 tissue blocks. The proposed method identified the trimming-end cutting routine approximately as accurate as the technicians did, yielding up to 90% of trimmed blocks of optimal quality and evidencing the potential of this tool in future automated trimming instruments. We deployed our method to an Edge TPU hardware accelerator to showcase its capability to provide immediate and objective results at every microtomy station applied with an ad hoc hardware, potentially guaranteeing a throughput 50% higher than manual trimming.

MAGNETO and DeepInsight: Extended Image Translation with Semantic Relationships for Classifying Attack Data with Machine Learning Models

The translation of traffic flow data into images for the purposes of classification in machine learning tasks has been extensively explored in recent years. However, the method of translation has a significant impact on the success of such attempts. In 2019, a method called DeepInsight was developed to translate genetic information into images. It was then adopted in 2021 for the purpose of translating network traffic into images, allowing the retention of semantic data about the relationships between features, in a model called MAGNETO. In this paper, we explore and extend this research, using the MAGNETO algorithm on three new intrusion detection datasets—CICDDoS2019, 5G-NIDD, and BOT-IoT—and also extend this method into the realm of multiclass classification tasks using first a One versus Rest model, followed by a full multiclass classification task, using multiple new classifiers for comparison against the CNNs implemented by the original MAGNETO model. We have also undertaken comparative experiments on the original MAGNETO datasets, CICIDS17, KDD99, and UNSW-NB15, as well as a comparison for other state-of-the-art models using the NSL-KDD dataset. The results show that the MAGNETO algorithm and the DeepInsight translation method, without the use of data augmentation, offer a significant boost to accuracy when classifying network traffic data. Our research also shows the effectiveness of Decision Tree and Random Forest classifiers on this type of data. Further research into the potential for real-time execution is needed to explore the possibilities for extending this method of translation into real-world scenarios.

Childhood Leukemia Classification via Information Bottleneck Enhanced Hierarchical Multi-Instance Learning.

Leukemia classification relies on a detailed cytomorphological examination of Bone Marrow (BM) smear. However, applying existing deep-learning methods to it is facing two significant limitations. Firstly, these methods require large-scale datasets with expert annotations at the cell level for good results and typically suffer from poor generalization. Secondly, they simply treat the BM cytomorphological examination as a multi-class cell classification task, thus failing to exploit the correlation among leukemia subtypes over different hierarchies. Therefore, BM cytomorphological estimation as a time-consuming and repetitive process still needs to be done manually by experienced cytologists. Recently, Multi-Instance Learning (MIL) has achievedmuch progress in data-efficient medical image processing, which only requires patient-level labels (which can be extracted from the clinical reports). In this paper, we propose a hierarchical MIL framework and equip it with Information Bottleneck (IB) to tackle the above limitations. First, to handle the patient-level label, our hierarchical MIL framework uses attention-based learning to identify cells with high diagnostic values for leukemia classification in different hierarchies. Then, following the information bottleneck principle, we propose a hierarchical IB to constrain and refine the representations of different hierarchies for better accuracy and generalization. By applying our framework to a large-scale childhood acute leukemia dataset with corresponding BM smear images and clinical reports, we show that it can identify diagnostic-related cells without the need for cell-level annotations and outperforms other comparison methods. Furthermore, the evaluation conducted on an independent test cohort demonstrates the high generalizability of our framework.

Hand-Written Analysis and Recognition Based on Deep Learning and Machine Learning Techniques

Abstract: This paper presents a comprehensive comparative analysis of Support Vector Machines (SVM), along with other machine learning models, and Convolutional Neural Networks (CNN) for Handwritten Analysis and Recognition. We investigate the performance of these models on a multi-class pairwise classification task using a large-scale handwritten dataset. Evaluation metrics such as accuracy, precision, and recall are utilized to measure their performance and compare them against the CNN model. The study aims to provide insights into the strengths and weaknesses of SVM, as well as other machine learning models, in comparison to CNN for handwritten analysis and recognition. SVM, known for its ability to handle complex non-linear relationships, offers good generalization and interpretability. However, it may face challenges in capturing intricate patterns in handwritten data. Other machine learning models, such as k-Nearest Neighbors (k-NN) and decision trees, also offer different advantages and limitations in this context. Through extensive experimentation, we compare the performance of SVM, k-NN, decision trees, and other relevant machine learning models with CNN on the same handwritten dataset. We analyze their accuracies, precision, and recall rates to evaluate their effectiveness in multi-class pairwise recognition of handwritten samples. Additionally, we discuss the computational requirements, training times, and interpretability associated with each model. The findings of this study provide valuable insights into the suitability of SVM and other machine learning models when compared to CNN for handwritten analysis and recognition tasks. The results shed light on the trade-offs between traditional machine learning approaches and deep learning architectures in this domain. These insights can guide researchers and practitioners in choosing the most appropriate model based on their specific requirements, computational resources, and interpretability needs.

Sensitivity of Modern Deep Learning Neural Networks to Unbalanced Datasets in Multiclass Classification Problems

One of the critical problems in multiclass classification tasks is the imbalance of the dataset. This is especially true when using contemporary pre-trained neural networks, where the last layers of the neural network are retrained. Therefore, large datasets with highly unbalanced classes are not good for models’ training since the use of such a dataset leads to overfitting and, accordingly, poor metrics on test and validation datasets. In this paper, the sensitivity to a dataset imbalance of Xception, ViT-384, ViT-224, VGG19, ResNet34, ResNet50, ResNet101, Inception_v3, DenseNet201, DenseNet161, DeIT was studied using a highly imbalanced dataset of 20,971 images sorted into 7 classes. It is shown that the best metrics were obtained when using a cropped dataset with augmentation of missing images in classes up to 15% of the initial number. So, the metrics can be increased by 2–6% compared to the metrics of the models on the initial unbalanced data set. Moreover, the metrics of the rare classes’ classification also improved significantly–the True Positive value can be increased by 0.3 or more. As a result, the best approach to train considered networks on an initially unbalanced dataset was formulated.

EEG-based classification combining Bayesian convolutional neural networks with recurrence plot for motor movement/imagery

Electroencephalogram (EEG)-based Motor imagery (MI) is a key topic in the brain-computer interface (BCI). The EEG-based real execution and motor imagery multi-class classification tasks are also crucial, but only a few kinds of literature research it. In addition, classification accuracy still has room for improvement, and the inter-individual variability problems in BCI applications need to be solved. To address these issues, we developed a novel model (RP-BCNNs) that combines the recurrence plot (RP) and Bayesian Convolutional Neural Networks (BCNNs). First, we employ an RP computation for preprocessed EEG signals of each channel and merge all RPs of all channels into one based on the weighted average method. Then, we feed the RP features into BCNNs to classify 2-class, 3-class, 4-class, and 5-class on real/imaginary movements classification tasks. The results show that the RP-BCNNs model outperforms the state-of-the-art methods, achieving average accuracies of 92.86%, 94.12%, 91.37%, 92.61% for real movements and 94.07%, 93.77%, 90.54%, 91.85% for imaginary movements. Our findings suggest that combining complex network methods with deep learning can improve the classification performance of EEG-based BCI systems (e.g., motor imagery, emotion recognition, and epileptic seizure classification).

Generic Paddy Plant Disease Detector (GP2D2)

Rice is the primary food for almost half of the world’s population, especially for the people of Asian countries. There is a demand to improve the quality and increase the quantity of rice production to meet the food requirements of the increasing population. Bulk cultivation and quality production of crops need appropriate technology assistance over manual traditional methods. In this work, six popular Deep-CNN architectures, namely AlexNet, VGG-19, VGG-16, InceptionV3, MobileNet, and ResNet-50, are exploited to identify the diseases in paddy plants since they outperform most of the image classification applications. These CNN models are trained and tested with Plant Village dataset for classifying the paddy plant images into one of the four classes namely, Healthy, Brown Spot, Hispa, or Leaf Blast, based on the disease condition. The performance of the chosen architectures is compared with different hyper parameter settings. AlexNet outperformed other convolutional neural networks (CNNs) in this multiclass classification task, achieving an accuracy of 89.4% at the expense of a substantial number of network parameters, indicating the large model size of AlexNet. For developing mobile applications, the ResNet-50 architecture was adopted over other CNNs, since it has a comparatively smaller number of network parameters and a comparable accuracy of 86.1%. A fine-tuned ResNet-50 architecture supported mobile app, “Generic Paddy Plant Disease Detector (GP2D2)” has been developed for the identification of most commonly occurring diseases in paddy plants. This tool will be more helpful for the new generation of farmers in bulk cultivation and increasing the productivity of paddy. This work will give insight into the performance of CNN architectures in rice plant disease detection task and can be extended to other plants too.

An ensemble machine learning approach for classification tasks using feature generation

Although machine learning classifiers have been successfully used in the medical and engineering fields, there is still room for improving the predictive accuracy of model classification. The higher the accuracy of the classifier, the better suggestions can be provided for the decision makers. Therefore, in this study, we propose an ensemble machine learning approach, called Feature generation-based Ensemble Support Vector Machine (FESVM), for classification tasks. We first apply the feature selection technique to select the most related features. Next, we introduce an ensemble strategy to aggregate multiple base estimators for the final prediction using the meta-classifier SVM. During this stage, we use the classification probabilities obtained from the base classifier to generate new features. After that, the generated features are added to the original data set to form a new data set. Finally, this new data set is utilised to train the meta-classifier SVM to obtain the final classification results. For example, for a binary classification task, each base classifier has two probabilities (p for one class and 1−p for the other class). In this case, two new features are generated from the combination of probabilities based on these base classifiers. One is the sum of p as new feature 1, and the other is the sum of 1−p as new feature 2. These two new features are then added to the original data set to form the new data set. In the same way, our feature generation method can be easily extended for a multi-class task for generating new features, where the number of features depends on the number of classes. Those generated features from the base estimators (first layer) are added to the original data set to form a new data set. This new data set is used as the input to the second layer (meta-classifier) to obtain the final model. Experiments based on the 20 data sets show that our proposed model FESVM has the best performance compared to the other machine learning classifiers under comparison. In addition, our FESVM has better performance than the original stacking method in the multi-class classification tasks. Statistical results based on the Wilcoxon–Holm method also confirms that our FESVM can significantly outperform the other models. These indicate that our FESVM can be a useful tool for classification tasks, especially multi-classification tasks.

Statistically significant features improve binary and multiple Motor Imagery task predictions from EEGs.

In recent studies, in the field of Brain-Computer Interface (BCI), researchers have focused on Motor Imagery tasks. Motor Imagery-based electroencephalogram (EEG) signals provide the interaction and communication between the paralyzed patients and the outside world for moving and controlling external devices such as wheelchair and moving cursors. However, current approaches in the Motor Imagery-BCI system design require effective feature extraction methods and classification algorithms to acquire discriminative features from EEG signals due to the non-linear and non-stationary structure of EEG signals. This study investigates the effect of statistical significance-based feature selection on binary and multi-class Motor Imagery EEG signal classifications. In the feature extraction process performed 24 different time-domain features, 15 different frequency-domain features which are energy, variance, and entropy of Fourier transform within five EEG frequency subbands, 15 different time-frequency domain features which are energy, variance, and entropy of Wavelet transform based on five EEG frequency subbands, and 4 different Poincare plot-based non-linear parameters are extracted from each EEG channel. A total of 1,364 Motor Imagery EEG features are supplied from 22 channel EEG signals for each input EEG data. In the statistical significance-based feature selection process, the best one among all possible combinations of these features is tried to be determined using the independent t-test and one-way analysis of variance (ANOVA) test on binary and multi-class Motor Imagery EEG signal classifications, respectively. The whole extracted feature set and the feature set that contain statistically significant features only are classified in this study. We implemented 6 and 7 different classifiers in multi-class and binary (two-class) classification tasks, respectively. The classification process is evaluated using the five-fold cross-validation method, and each classification algorithm is tested 10 times. These repeated tests provide to check the repeatability of the results. The maximum of 61.86 and 47.36% for the two-class and four-class scenarios, respectively, are obtained with Ensemble Subspace Discriminant among all these classifiers using selected features including only statistically significant features. The results reveal that the introduced statistical significance-based feature selection approach improves the classifier performances by achieving higher classifier performances with fewer relevant components in Motor Imagery task classification. In conclusion, the main contribution of the presented study is two-fold evaluation of non-linear parameters as an alternative to the commonly used features and the prediction of multiple Motor Imagery tasks using statistically significant features.

The Application of Fully Homomorphic Encryption on XGBoost Based Multiclass Classification

Fully Homomorphic Encryption (FHE) is a ground breaking cryptographic technique that allows computations to be performed directly on encrypted data, preserving privacy and security. This paper explores the application of Fully Homomorphic Encryption on Extreme Gradient Boosting (XGBoost) multiclass classification, demonstrating its potential to enable secure and privacy-preserving machine learning. The paper presents a framework for training and evaluating XGBoost models using encrypted data, leveraging FHE operations for encrypted feature engineering, model training, and inference. The experimental results showcase the feasibility of applying Fully Homomorphic Encryption to XGBoost-based multiclass classification tasks while maintaining data confidentiality. The findings highlight the trade-off between computation complexity and model accuracy in FHE-based approaches and provide insights into the challenges and future directions of utilizing Fully Homomorphic Encryption in practical machine learning scenarios. The study underscores the significance of privacy-preserving machine learning techniques and paves the way for secure data analysis in sensitive domains where data privacy is of utmost importance.

Overview of the 2022 n2c2 shared task on contextualized medication event extraction in clinical notes

Background:An accurate medication history, foundational for providing quality medical care, requires understanding of medication change events documented in clinical notes. However, extracting medication changes without the necessary clinical context is insufficient for real-world applications. Methods:To address this need, Track 1 of the 2022 National NLP Clinical Challenges focused on extracting the context for medication changes documented in clinical notes using the Contextualized Medication Event Dataset. Track 1 consisted of 3 subtasks: extracting medication mentions from clinical notes (NER), determining whether a medication change is being discussed (Event), and determining the action, negation, temporality, certainty, and actor for any change events (Context). Participants were allowed to participate in any one or more of the subtasks. Results:A total of 32 teams with participants from 19 countries submitted a total of 211 systems across all subtasks. Most teams formulated NER as a token classification task and Event and Context as multi-class classification tasks, using transformer-based large language models. Overall, performance for NER was high across submitted systems. However, performance for Event and Context were much lower, often due to indirectly stated change events with no clear action verb, events requiring farther textual clues for understanding, and medication mentions with multiple change events. Conclusions:This shared task showed that while NLP research on medication extraction is relatively mature, understanding of contextual information surrounding medication events in clinical notes is still an open problem requiring further research to achieve the end goal of supporting real-world clinical applications.

Deep learning neural network derivation and testing to distinguish acute poisonings

ABSTRACT Introduction Acute poisoning is a significant global health burden, and the causative agent is often unclear. The primary aim of this pilot study was to develop a deep learning algorithm that predicts the most probable agent a poisoned patient was exposed to from a pre-specified list of drugs. Research design & methods Data were queried from the National Poison Data System (NPDS) from 2014 through 2018 for eight single-agent poisonings (acetaminophen, diphenhydramine, aspirin, calcium channel blockers, sulfonylureas, benzodiazepines, bupropion, and lithium). Two Deep Neural Networks (PyTorch and Keras) designed for multi-class classification tasks were applied. Results There were 201,031 single-agent poisonings included in the analysis. For distinguishing among selected poisonings, PyTorch model had specificity of 97%, accuracy of 83%, precision of 83%, recall of 83%, and a F1-score of 82%. Keras had specificity of 98%, accuracy of 83%, precision of 84%, recall of 83%, and a F1-score of 83%. The best performance was achieved in the diagnosis of single-agent poisoning in diagnosing poisoning by lithium, sulfonylureas, diphenhydramine, calcium channel blockers, then acetaminophen, in PyTorch (F1-score = 99%, 94%, 85%, 83%, and 82%, respectively) and Keras (F1-score = 99%, 94%, 86%, 82%, and 82%, respectively). Conclusion Deep neural networks can potentially help in distinguishing the causative agent of acute poisoning. This study used a small list of drugs, with polysubstance ingestions excluded.Reproducible source code and results can be obtained at https://github.com/ashiskb/npds-workspace.git.

COVID-19 Hakkındaki Türkçe Tweetlerde LSTM Ağı Kullanılarak Duygu Sınıflandırması

As Covid-19 pandemic affected everyone in various aspects, people have been expressing their opinions on these aspects mostly on social media platforms because of the pandemic. These opinions play a crucial role in understanding the sentiments towards the pandemic. In this study, Turkish tweets on Covid-19 topic were collected from March 2020 to January 2021 and labelled as positive, negative, or neutral in terms of sentiment using BERT which is a pre-trained text classifier model. Using this labelled dataset, a set of experiments were carried out with SVM, Naive Bayes, K-Nearest Neighbors, and CNN-LSTM model machine learning algorithms for binary and multi-class classification tasks. Results of these experiments have shown that CNN-LSTM model outperforms other machine learning algorithms which are used in this study in both binary classification and multi-class classification tasks.

R1DIT: Privacy-Preserving Malware Traffic Classification With Attention-Based Neural Networks

With the advances in deep learning techniques and the increase in the volume of network traffic data, deep neural networks trained directly with the raw traffic data have become more popular and successful for malware traffic classification without explicit feature extraction. However, most of the existing studies raises privacy concerns when using the payload data and ignore the generalization of the model to the newly emerged traffic such as DDoS detection on TLS 1.3. To overcome these limitations, we introduce a malware traffic classification system, Residual 1-D Image Transformer (R1DIT) model. We first leverage network domain knowledge by carefully parsing IP, HTTP, DNS, and unencrypted TLS record headers as sequences of bytes for input without interfering with IP addresses, port numbers and the payload. Then, we apply raw data transform and attention-based modules in our deep model to classify different malware types and benign traffic. Our results on NetML dataset show that the proposed model delivers 0.972 F1 score, nearly 0.3 higher than the feature-based methods and outperforms state-of-the-art models with 0.9999 F1 score for multi-class malware classification task using CICIDS2017 dataset. The generalization of this model has been proven using the TLS 1.3 traffic obtained from CICDDoS2019 dataset with the detection rate 0.9897 using meta-learning.

Fisher Regularized ε-Dragging for Image Classification

Discriminative least squares regression (DLSR) has been shown to achieve promising performance in multi-class image classification tasks. Its key idea is to force the regression labels of different classes to move in opposite directions by means of the ε-dragging technique, yielding discriminative regression model exhibiting wider margins. However, the ε-dragging technique ignores an important problem: its relaxation matrix is dynamically updated in optimization, which means the dragging values can also cause the labels from the same class to be uncorrelated. In order to learn a more powerful projection, as well as regression labels, we propose a Fisher regularized ε-dragging framework (Fisher-ε) for image classification by constraining the relaxed labels using the Fisher criterion. On one hand, the Fisher criterion improves the intra-class compactness of the relaxed labels during relaxation learning. On the other hand, it is expected further to enhance the inter-class separability of ε-dragging. Fisher-ε for the first time ever attempts to integrate the Fisher criterion and ε-dragging technique into a unified model because they are complementary in learning discriminative projection. Extensive experiments on various datasets demonstrate that the proposed Fisher-ε method achieves performance that is superior to other state-of-the-art classification methods. The Matlab codes are available at https://github.com/chenzhe207/Fisher-epsilon.