Traditional Supervised Learning Algorithms Research Articles

A novel approach to intrusion detection using zero-shot learning hybrid partial labels

Computer networks have become the backbone of our interconnected world in today's technologically driven landscape. Unauthorized access or malicious activity carried out by threat actors to acquire control of network resources, exploit vulnerabilities, or undermine system integrity are examples of network intrusion. ZSL(Zero-Shot Learning) is a machine learning paradigm that addresses the problem of detecting and categorizing objects or concepts that were not present in the training data. . Traditional supervised learning algorithms for intrusion detection frequently struggle with insufficient labeled data and may struggle to adapt to unexpected assault patterns. In this article We have proposed a unique zero-shot learning hybrid partial label model suited to a large image-based network intrusion dataset to overcome these difficulties. The core contribution of this study is the creation and successful implementation of a novel zero-shot learning hybrid partial label model for network intrusion detection, which has a remarkable accuracy of 99.12%. The suggested system lays the groundwork for future study into other feature selection techniques and the performance of other machine learning classifiers on larger datasets. Such research can advance the state-of-the-art in intrusion detection and improve our ability to detect and prevent the network attacks. We hope that our research will spur additional research and innovation in this critical area of cybersecurity.

MSFSS: A whale optimization-based multiple sampling feature selection stacking ensemble algorithm for classifying imbalanced data

<abstract> <p>Learning from imbalanced data is a challenging task in the machine learning field, as with this type of data, many traditional supervised learning algorithms tend to focus more on the majority class while damaging the interests of the minority class. Stacking ensemble, which formulates an ensemble by using a meta-learner to combine the predictions of multiple base classifiers, has been used for solving class imbalance learning issues. Specifically, in the context of class imbalance learning, a stacking ensemble learning algorithm is generally considered to combine with a specific sampling algorithm. Such an operation, however, might suffer from suboptimization problems as only using a sampling strategy may make it difficult to acquire diverse enough features. In addition, we also note that using all of these features may damage the meta-learner as there may exist noisy and redundant features. To address these problems, we have proposed a novel stacking ensemble learning algorithm named MSFSS, which divides the learning procedure into two phases. The first stage combined multiple sampling algorithms and multiple supervised learning approaches to construct meta feature space by means of cross combination. The adoption of this strategy satisfied the diversity of the stacking ensemble. The second phase adopted the whale optimization algorithm (WOA) to select the optimal sub-feature combination from the meta feature space, which further improved the quality of the features. Finally, a linear regression classifier was trained as the meta learner to conduct the final prediction. Experimental results on 40 benchmarked imbalanced datasets showed that the proposed MSFSS algorithm significantly outperformed several popular and state-of-the-art class imbalance ensemble learning algorithms. Specifically, the MSFSS acquired the best results in terms of the F-measure metric on 27 datasets and the best results in terms of the G-mean metric on 26 datasets, out of 40 datasets. Although it required consuming more time than several other competitors, the increment of the running time was acceptable. The experimental results indicated the effectiveness and superiority of the proposed MSFSS algorithm.</p> </abstract>

SWSEL: Sliding Window-based Selective Ensemble Learning for class-imbalance problems

For class-imbalance problems, traditional supervised learning algorithms tend to favor majority instances (also called negative instances). Therefore, it is difficult for them to accurately identify the minority instances (also called positive instances). Ensemble learning is a common method to solve the class-imbalance problem. They build multiple classifier systems on the training dataset to improve the recognition accuracy of minority instances. Sliding window is a commonly used method for processing data stream. Few researchers have used sliding windows to select majority instances and construct ensemble learning models. Traditional ensemble learning methods use some or all of the majority instances for modeling by oversampling or undersampling. However, they also inherit the drawbacks of the preprocessing methods. Therefore, in this paper, we try to use similarity mapping to construct pseudo-sequences of majority instances. Then, according to the sliding window idea, we fully use all existing majority instances, and a novel sliding window-based selective ensemble learning method (SWSEL) is proposed to deal with the class-imbalance problem. This method uses the idea of distance alignment in multi-view alignment to align the centers of the minority instances with the majority instances, and slide to select the majority instances on the sequence of pseudo-majority instances. In addition, to prevent too many classifiers from leading to long running times, we use distance metric to select a certain number of base classifiers to build the final ensemble learning model. Extensive experimental results on various real-world datasets show that using SVM, MLP and RF as the base classifier, SWSEL achieves a statistically significant performance improvement on two evaluation metrics, AUC and G-mean, compared to state-of-the-art methods.

Improving performance of decision threshold moving-based strategies by integrating density-based clustering technique

<abstract> <p>Class imbalance learning (CIL), which aims to addressing the performance degradation problem of traditional supervised learning algorithms in the scenarios of skewed data distribution, has become one of research hotspots in fields of machine learning, data mining, and artificial intelligence. As a postprocessing CIL technique, the decision threshold moving (DTM) has been verified to be an effective strategy to address class imbalance problem. However, no matter adopting random or optimal threshold designation ways, the classification hyperplane could be only moved parallelly, but fails to vary its orientation, thus its performance is restricted, especially on some complex and density variable data. To further improve the performance of the existing DTM strategies, we propose an improved algorithm called CDTM by dividing majority training instances into multiple different density regions, and further conducting DTM procedure on each region independently. Specifically, we adopt the well-known DBSCAN clustering algorithm to split training set as it could adapt density variation well. In context of support vector machine (SVM) and extreme learning machine (ELM), we respectively verified the effectiveness and superiority of the proposed CDTM algorithm. The experimental results on 40 benchmark class imbalance datasets indicate that the proposed CDTM algorithm is superior to several other state-of-the-art DTM algorithms in term of G-mean performance metric.</p> </abstract>

Bayesian network-based over-sampling method (BOSME) with application to indirect cost-sensitive learning

Traditional supervised learning algorithms do not satisfactorily solve the classification problem on imbalanced data sets, since they tend to assign the majority class, to the detriment of the minority class classification. In this paper, we introduce the Bayesian network-based over-sampling method (BOSME), which is a new over-sampling methodology based on Bayesian networks. Over-sampling methods handle imbalanced data by generating synthetic minority instances, with the benefit that classifiers learned from a more balanced data set have a better ability to predict the minority class. What makes BOSME different is that it relies on a new approach, generating artificial instances of the minority class following the probability distribution of a Bayesian network that is learned from the original minority classes by likelihood maximization. We compare BOSME with the benchmark synthetic minority over-sampling technique (SMOTE) through a series of experiments in the context of indirect cost-sensitive learning, with some state-of-the-art classifiers and various data sets, showing statistical evidence in favor of BOSME, with respect to the expected (misclassification) cost.

A Travel Mode Identification Framework Based on Cellular Signaling Data

The rapid development of telecommunication network has produced a large amount of spatial-temporal information of mobile phone users. GPS data are typically collected by smartphones apps, which are restricted to small samples of the population. Cellular signaling data (CSD) are usually collected by mobile network operators, which enables researchers to conduct travel behavior analysis of the entire population at a relatively low cost compared to GPS. However, extracting travel mode information from CSD is particularly challenging due to the noise data and low positioning accuracy. This paper proposes a travel mode identification framework based on CSD, which includes data cleansing and travel mode identification. In terms of data cleansing, oscillation sequence and drift data are mainly cleansed. For the oscillation sequence, this paper proposes a detection algorithm based on time window. For the drift data, this paper proposes a detection algorithm based on distance, velocity and frequency. In terms of travel mode identification, the task is divided into two dichotomous problems: motor and non-motor transport identification and public and private transport identification. Each dichotomous problem proposes an algorithm that does not rely on the ground truth dataset for model training. Finally, a ground truth dataset is constructed to verify all algorithms. The result shows that, in terms of data cleansing, the similarity between CSD after cleansing and the actual trajectory according to DTW improved by 101.13% on average. In terms of travel mode identification, the proposed method can achieve similar or even better accuracy than traditional supervised-learning algorithms (94% in motor and non-motor transport identification, 83.5% in public and private transport identification), which can be directly applied to large-scale population analysis scenarios.

Enhancing keyphrase extraction from academic articles with their reference information

With the development of Internet technology, the phenomenon of information overload is becoming more and more obvious. It takes a lot of time for users to obtain the information they need. However, keyphrases that summarize document information highly are helpful for users to quickly obtain and understand documents. For academic resources, most existing studies extract keyphrases through the title and abstract of papers. We find that title information in references also contains author-assigned keyphrases. Therefore, this article uses reference information and applies two typical methods of unsupervised extraction methods (TF*IDF and TextRank), two representative traditional supervised learning algorithms (Naïve Bayes and Conditional Random Field) and a supervised deep learning model (BiLSTM-CRF), to analyze the specific performance of reference information on keyphrase extraction. It is expected to improve the quality of keyphrase recognition from the perspective of expanding the source text. The experimental results show that reference information can increase precision, recall, and F1 of automatic keyphrase extraction to a certain extent. This indicates the usefulness of reference information on keyphrase extraction of academic papers and provides a new idea for the research on automatic keyphrase extraction.

SVDD-based weighted oversampling technique for imbalanced and overlapped dataset learning

Imbalanced dataset classification issue poses a major challenge on machine learning domain. Traditional supervised learning algorithms usually bias towards the majority class when handling imbalanced datasets, thus leading to poor classification results on the minority class. The learning task would become crucially difficult when there are overlapping and within-imbalance issues in imbalanced datasets, which are often the case and have been proven to severely deteriorate the classification performance relative to between-class imbalance. In this paper, we propose a novel SVDD boundary-based weighted oversampling approach (SVDDWSMOTE) for handling imbalanced and overlapped data. The proposed approach first applies support vector data description (SVDD) model with greater penalty constant for the minority class than the majority class to generate the class boundary, and then identifies those misclassified majority or few minority instances by the generated class boundary as potential overlapped or noisy ones and eliminates them. To address the within-balance issues, we propose a weight assignment strategy based on densities and the distances to the SVDD class boundary, which facilitates simultaneously combating between-class and within-class imbalance issues caused by complicated distribution. In addition, such a strategy also favors generating more synthetic minority instances for borderline and sparser instances which are usually informative to the later learning tasks. Finally, oversampling is performed by the weighed SMOTE scheme based on SVDD boundary to not only counteract the within imbalance but also avoid the generation of any noisy or overlapped synthetic instance. Extensive comparison results on various datasets show that the proposed approach achieves statistically significant improvements in terms of different classification performance metrics relative to state-of-the-art ones.

Classification of the Acoustics of Loose Gravel.

Road condition evaluation is a critical part of gravel road maintenance. One of the assessed parameters is the amount of loose gravel, as this determines the driving quality and safety. Loose gravel can cause tires to slip and the driver to lose control. An expert assesses the road conditions subjectively by looking at images and notes. This method is labor-intensive and subject to error in judgment; therefore, its reliability is questionable. Road management agencies look for automated and objective measurement systems. In this study, acoustic data on gravel hitting the bottom of a car was used. The connection between the acoustics and the condition of loose gravel on gravel roads was assessed. Traditional supervised learning algorithms and convolution neural network (CNN) were applied, and their performances are compared for the classification of loose gravel acoustics. The advantage of using a pre-trained CNN is that it selects relevant features for training. In addition, pre-trained networks offer the advantage of not requiring days of training or colossal training data. In supervised learning, the accuracy of the ensemble bagged tree algorithm for gravel and non-gravel sound classification was found to be 97.5%, whereas, in the case of deep learning, pre-trained network GoogLeNet accuracy was 97.91% for classifying spectrogram images of the gravel sounds.

Generalisations of stochastic supervision models

When the labelling information is not deterministic, traditional supervised learning algorithms cannot be applied. In this case, stochastic supervision models provide a valuable alternative to classification. However, these models are restricted in several aspects, which critically limits their applicability. In this paper, we provide four generalisations of stochastic supervision models, extending them to asymmetric assessments, multiple classes, feature-dependent assessments and multi-modal classes, respectively. Corresponding to these generalisations, we derive four new EM algorithms. We show the effectiveness of our generalisations through illustrative examples of simulated datasets, as well as real-world examples of three famous datasets, the MNIST dataset, the CIFAR-10 dataset and the EMNIST dataset.

Partial label metric learning by collapsing classes

Partial label learning (PLL) is a weakly supervised learning framework proposed recently, in which the ground-truth label of training sample is not precisely annotated but concealed in a set of candidate labels, which makes the accuracy of the existing PLL algorithms is usually lower than that of the traditional supervised learning algorithms. Since the accuracy of a learning algorithm is usually closely related to its distance metric, the metric learning technologies can be employed to improve the accuracy of the existing PLL algorithms. However, only a few PLL metric learning algorithms have been proposed up to the present. In view of this, a novel PLL metric learning algorithm is proposed by using the collapsing classes model in this paper. The basic idea is first to take each training sample and its neighbor with shared candidate labels as a similar pair, while each training sample and its neighbor without shared candidate labels as a dissimilar pair, then two probability distributions are defined based on the distance and label similarity of these pairs, respectively, finally the metric matrix is obtained via minimizing the Kullback–Leibler divergence of these two probability distributions. Experimental results on six UCI data sets and four real-world PLL data sets show that the proposed algorithm can obviously improve the accuracy of the existing PLL algorithms.

Heterogeneous tree structure classification to label Java programmers according to their expertise level

Open-source code repositories are a valuable asset to creating different kinds of tools and services, utilizing machine learning and probabilistic reasoning. Syntactic models process Abstract Syntax Trees (AST) of source code to build systems capable of predicting different software properties. The main difficulty of building such models comes from the heterogeneous and compound structures of ASTs, and that traditional machine learning algorithms require instances to be represented as n-dimensional vectors rather than trees. In this article, we propose a new approach to classify ASTs using traditional supervised-learning algorithms, where a feature learning process selects the most representative syntax patterns for the child subtrees of different syntax constructs. Those syntax patterns are used to enrich the context information of each AST, allowing the classification of compound heterogeneous tree structures. The proposed approach is applied to the problem of labeling the expertise level of Java programmers. The system is able to label expert and novice programs with an average accuracy of 99.6%. Moreover, other code fragments such as types, fields, methods, statements and expressions could also be classified, with average accuracies of 99.5%, 91.4%, 95.2%, 88.3% and 78.1%, respectively.

Supervised PU Learning for Cyber Security Event Prioritization

To ensure the cyber security of an enterprise, typically a SIEM (Security Information and Event Management) system is in place to alert security events and assign each of them a severity score based on some pre-determined rules. Analysts in the security operations center (SOC) investigate the high severity events to decide if they are truly malicious or not. However, generally, the number of events is overwhelmingly large, far exceeding the SOC’s budget to handle them. There is a great need for a machine learning system to assist for the accurate detection of malicious events. Traditional supervised learning algorithms cannot be directly applied to this problem because there are only a very small percentage of events that are verified and labelled malicious events (called positively labelled) by SOC analysts and vast majority of events are unlabeled as they are not even investigated. In this paper, we propose to aggregate security events to host level and use supervised PU (Positive Unlabeled) learning technique to accurately detect the high risk hosts. We use Support Vector Machine with radial basis kernel for label propagation and classification and achieve a high classification accuracy of AUC (area under the curve) of 0.96 and lift of 18 relative to the current rule-based alerting mechanism.

An Experimental Comparison of Semi-supervised Learning Algorithms for Multispectral Image Classification

Semi-Supervised Learning (SSL) method has recently caught much attention in the fields of machine learning and computer vision owing to its superiority in classifying abundant unlabelled samples using a few labeled samples. The goal of this paper is to provide an experimental efficiency comparison between graph based SSL algorithms and traditional supervised learning algorithms (e.g., support vector machines) for multispectral image classification. This research shows that SSL algorithms generally outperform supervised learning algorithms in both classification accuracy and anti-noise ability. In the experiments carried out on two data sets (hyperspectral image and Landsat image), the mean overall accuracies (OAs) of supervised learning algorithms are 15 percent and 86 percent, while the mean OAS of SSL algorithms are 26 percent and 99 percent. To overcome the polynomial complexity of SSL algorithms, we also developed a linear-complexity algorithm by employing multivariate Taylor Series Expansion (TSE) and Woodbury Formula.

Self-Training using a K-Nearest Neighbor as a Base Classifier Reinforced by Support Vector Machines

In supervised learning, algorithms infer a general prediction model based on previously labeled data. However, in many real-world machine learning problems, the number of labeled data is small, while the unlabeled data are abundant. Obviously, the reliability of the learned model depends essentially on the size of the training set (labeled data). Indeed, if the amount of labeled data is not high enough, the generalization errors of learned model may be important. In such situation, semi supervised learning algorithm may improve the generalization performance of this model by integrating unlabeled data in the learning process. One of the most classical methods of the semi-supervised learning is the self-training. An advantage of this method is that several traditional supervised learning algorithms are used to build the model in the self-training process. In this paper, the k-Nearest Neighbors (k-NN) classifier was chosen in making decision during the self-training process. We also propose to reinforce self-training strategy by using a Support vector machines (SVM) classifier that can help the kNN to label the unlabeled data. Experimental results showed that Self-training based on k-NN and SVM can outperform the results with the Self-training based on k-NN classifier only. .

Channel-Robust Classifiers

A key assumption underlying traditional supervised learning algorithms is that labeled examples used to train a classifier are drawn i.i.d. from the same distribution as test samples. This assumption is violated when classifying a test sample whose statistics differ from the training samples because the test signal is the output of a noisy linear time-invariant system, e.g., from channel propagation or filtering. We assume that the channel impulse response is unknown, but can be modeled as a random channel with finite first and second-order statistics that can be estimated from sample impulse responses. We present two kernels, the expected and projected RBF kernels, that account for the stochastic channel. Compared to the strategy of virtual examples, an SVM trained with the proposed kernels requires dramatically less training time, and may perform better in practice. We also extend the joint quadratic discriminant analysis (joint QDA) classifier, which also accounts for a stochastic channel, to a local version that reduces model bias. Results show the proposed methods achieve state-of-the-art performance and significantly faster training times.

COG: local decomposition for rare class analysis

Given its importance, the problem of predicting rare classes in large-scale multi-labeled data sets has attracted great attention in the literature. However, rare class analysis remains a critical challenge, because there is no natural way developed for handling imbalanced class distributions. This paper thus fills this crucial void by developing a method for classification using local clustering (COG). Specifically, for a data set with an imbalanced class distribution, we perform clustering within each large class and produce sub-classes with relatively balanced sizes. Then, we apply traditional supervised learning algorithms, such as support vector machines (SVMs), for classification. Along this line, we explore key properties of local clustering for a better understanding of the effect of COG on rare class analysis. Also, we provide a systematic analysis of time and space complexity of the COG method. Indeed, the experimental results on various real-world data sets show that COG produces significantly higher prediction accuracies on rare classes than state-of-the-art methods and the COG scheme can greatly improve the computational performance of SVMs. Furthermore, we show that COG can also improve the performances of traditional supervised learning algorithms on data sets with balanced class distributions. Finally, as two case studies, we have applied COG for two real-world applications: credit card fraud detection and network intrusion detection.