Count Vector Research Articles

Analysis of statistical properties of variables in log data for advanced anomaly detection in cyber security

Log lines consist of static parts that characterize their structure and enable assignment of event types, and event parameters, i.e., variable parts that provide specific information on system processes, such as host and user names, IP addresses, and file operations. Many detection approaches only focus on anomalous event type occurrences, i.e., they parse log lines to derive unique event identifiers and subsequently detect anomalies in event sequences or event count vectors, but neglect variable parts of log lines entirely during analysis. This is especially problematic, when monitoring strongly structured log data that contains only a small number of distinct event types, for example, logs that consist of strict key value pairs, i.e., parameters that occur consistently throughout all log lines, such as it is case in access and audit logs. Thus, novel approaches are required, which focus on analysis of log lines' variable parts. In this paper, we propose the variable type detector (VTD), a novel unsupervised approach that autonomously analyzes variable log line parts to enable anomaly detection. It assigns data types to each variable, which also include probability distributions for discrete and continuous variables. The VTD raises an alarm if a variable's data type changes. Furthermore, it implements a robust indicator function that reduces false positives by tracking the data type history of each variable and reports only significant data type changes. Additionally, an event indicator enables event-based anomaly detection by taking into account the data types of all variables of a single event type. The evaluation conducted on open-source log data, demonstrates the effectiveness of the VTD compared to conventional anomaly detection approaches, such as time series analysis and PCA. Consequently, the VTD acts as a solution that extends the intrusion detection capabilities of security information and event management (SIEM) and integrates with modern concepts of endpoint detection and response (EDR) and extended detection and responses (XDR), while simultaneously serving as an asset for process monitoring that supports user and entity behavior analytics (UEBA).

Solving the imbalanced data issue: automatic urgency detection for instructor assistance in MOOC discussion forums

In MOOCs, identifying urgent comments on discussion forums is an ongoing challenge. Whilst urgent comments require immediate reactions from instructors, to improve interaction with their learners, and potentially reducing drop-out rates—the task is difficult, as truly urgent comments are rare. From a data analytics perspective, this represents a highly unbalanced (sparse) dataset. Here, we aim to automate the urgent comments identification process, based on fine-grained learner modelling—to be used for automatic recommendations to instructors. To showcase and compare these models, we apply them to the first gold standard dataset for Urgent iNstructor InTErvention (UNITE), which we created by labelling FutureLearn MOOC data. We implement both benchmark shallow classifiers and deep learning. Importantly, we not only compare, for the first time for the unbalanced problem, several data balancing techniques, comprising text augmentation, text augmentation with undersampling, and undersampling, but also propose several new pipelines for combining different augmenters for text augmentation. Results show that models with undersampling can predict most urgent cases; and 3X augmentation + undersampling usually attains the best performance. We additionally validate the best models via a generic benchmark dataset (Stanford). As a case study, we showcase how the naïve Bayes with count vector can adaptively support instructors in answering learner questions/comments, potentially saving time or increasing efficiency in supporting learners. Finally, we show that the errors from the classifier mirrors the disagreements between annotators. Thus, our proposed algorithms perform at least as well as a ‘super-diligent’ human instructor (with the time to consider all comments).

Kurdish Fake News Detection Based on Machine Learning Approaches

The widespread use of social media platforms and the internet has increased information sharing, including both true and false news. Detecting fake news is challenging, and several studies have been conducted to automate this process for popular languages such as English and Arabic. However, more research must be done on detecting fake news in low-resource languages such as Kurdish. This gap was addressed, and a publicly available Kurdish fake news dataset (KDFND) was used, comprising 100962 news articles, among which 50751 are real, and 50211 are fake news labeled as Real and Fake. In this study, three techniques were implemented to extract features from news texts, including word embedding, term frequency-inverse document frequency, and count vector, and three various machine learning and deep learning classifiers were used (Random Forest, Support Vector Machine, and Convolutional Neural Networks) to identify the fake news dataset. The results showed that fake news with textual content could be identified, especially when convolutional neural networks are used. According to the experimental results of the study, CNN performs better than the other models, with an F1-score of 95% and an accuracy of more than 91% percent. These findings indicate that machine learning methods can efficiently detect fake news in low-resource languages like Kurdish, even in complex environments.

A machine learning tool for collecting and analyzing subjective road safety data from Twitter

Several measures have been proposed in the road safety field to measure safety objectively. However, studies focused on measuring subjective road safety (i.e., the cognitive safety of road environment by drivers) are limited due to the lack and difficulty of collecting subjective safety data. Still, subjective safety is important and should be monitored as much as objective safety as it can negatively impact road users’ mobility and road agencies draw information on it in determining policies (ultimately affecting collision occurrence). Nowadays, crowdsourcing big data related to subjective road safety can be done in social media platforms like Twitter. Therefore, this research aims at developing a tool for extracting, classifying, and studying drivers’ affective states from road safety-related tweets using keyword filtering, geo-boundaries, natural language processing, and machine-learning (ML) classification. The ML classification algorithms used in this study were naive Bayes (NB), logistic regression (LR), support vector machine (SVM), random forest (RF), and extreme gradient boosting (XGBoost). Also, word count, word level TF-IDF, N-gram level TF-IDF, and character level TF-IDF were used as features. Metro Vancouver (Canada) was selected as the geographic region for tweet extraction and creation of training and test data sets. The study focused on the year 2019, where 13,226 unique tweets were extracted after removing duplicates. The performance of proposed ML models was compared by estimating accuracy, precision, recall, and F1-scores. The results showed that the trained RF model with count vector, and SVM classifier with word-level TF-IDF performed best in separating road safety-related from unrelated tweets (accuracy = 0.935, F1-score = 0.937) and determining the proposed classification tags (accuracy = 0.881, F1-score = 0.879), respectively. Finally, sentiment analysis was conducted to investigate the polarity of tweets in each group of the proposed classification to better understand drivers’ affective states.

Arabic Cyberbullying Detection Using Support Vector Machine with Cuckoo Search

Cyberbullying is one of the biggest electronic problems that takes multiple forms of harassment using various social media. Currently, this phenomenon has become very common and is increasing, especially for young people and adolescents. Negative comments have a significant and dangerous impact on society in general and on adolescents in particular. Therefore, one of the most successful prevention methods is to detect and block harmful messages and comments. In this research, negative Arabic comments that refer to cyberbullying will be detected using a support vector machine algorithm. The term frequency-inverse document frequency vectorizer and the count vectorizer methods were used for feature extraction, and the results were improved using the cuckoo search algorithm. The resulting accuracy before and after optimizing the support vector machine’s hyperparameters is 85.8% and 87.1%, respectively.

Detection of IOT botnet attack based on machine learning

The paper discussed the process of data processing and algorithm selection for three different scenarios in order to improve accuracy in detecting DDOS attacks, SPAM emails, and malware. It provided detailed descriptions of each process involved in the simulation. For the DDOS attack detection simulation, three different datasets were used, and missing data was removed to ensure the quality of the data. In addition, features were processed to make sure they could be applied to specific algorithms. Both decision tree and random forest algorithms were selected and tuned to obtain maximum accuracy. Similarly, for the SPAM email detection simulation, binary was used to represent whether an email was spam or not, and Count Vectorizer function was applied to convert mail contents into feature vectors. The KNN and decision tree algorithms were chosen, and emphasis was given on parameter adjustment to eliminate overfitting and ensure optimal model accuracy. The paper also discussed the importance of considering multiple factors when selecting and tuning algorithms, such as accuracy, complexity, and computational efficiency. These factors must be balanced to achieve the best overall performance. Overall, the paper provided a comprehensive overview of the methods and processes involved in data processing and algorithm selection to improve detection accuracy for DDOS attacks, SPAM emails, and malware. This research can greatly benefit organizations that are looking to enhance their security measures and minimize the risks associated with these cyber threats.

Multivariate central limit theorems for random clique complexes

Motivated by open problems in applied and computational algebraic topology, we establish multivariate normal approximation theorems for three random vectors which arise organically in the study of random clique complexes. These are: the vector of critical simplex counts attained by a lexicographical Morse matching,the vector of simplex counts in the link of a fixed simplex, andthe vector of total simplex counts. The first of these random vectors forms a cornerstone of modern homology algorithms, while the second one provides a natural generalisation for the notion of vertex degree, and the third one may be viewed from the perspective of U-statistics. To obtain distributional approximations for these random vectors, we extend the notion of dissociated sums to a multivariate setting and prove a new central limit theorem for such sums using Stein’s method.

Comparative Performance of Data Mining Techniques for Cyberbullying Detection of Arabic Social Media Text

Cyberbullying has spread like a virus on social media platforms and is getting out of control. According to psychological studies on the subject, the victims are increasingly suffering, sometimes to the point of committing suicide among the victims. The issue of cyberbullying on social media is spreading around the world. Social media use is growing, and it can have useful and negative implications when you take into account how social media platforms are abused through different forms of cyberbullying. Although there is a lot of cyberbullying detection in English, there are few studies in the Arabic language. Data Mining techniques are often used to solve and detect this problem. In this study, different data mining algorithms were used to detect cyberbullying in Arabic texts.. Our study was conducted The Bullying datasets consisted of 26,000 comments written in Arabic and were collected from kaggle.com, the Cyber_2021 dataset consisted of 13,247 comments collected via github.com, and the Data 2022 dataset consisted of 47,224 comments collected via Instagram. Various extraction features CountVectorizer and Tf-Idf were used Accuracy, precision, recall, and the F1 score were used to evaluate classifier performance. In the study, Bagging Classifier achieve high results of Bullying dataset from Kaggle Accuracy 96.04, F1-Score 95.98, Recall 96.04, Precision 95.95, SVC model gave the highest results of Cyber_2021 dataset from Github an Accuracy 98.49, F1-Score 98.49, Recall 98.49, Precision 98.50, while Data 2022 dataset from (Instagram) achieving an Accuracy of 77.51, F1-Score 76.60, Recall 77.51, and Precision 77.24. Were achieved for Tf-Idf Vectorizer. Tf-Idf Vectorizer the best to all results than count Vectorizer .

Emotion Classification in Indonesian Language: A CNN Approach with Hyperband Tuning

In today's world, there is a high demand for accurate techniques to classify emotions in various fields. This study proposed utilizing a Convolutional Neural Network (CNN) optimized with a Hyperband Tuner (HT) to perform the Emotion Classification task in the Indonesian language effectively. Various feature extraction techniques experiments were conducted to explore the best combinations of feature extraction and CNN for the data set, including CountVectorizer (CV), TF-IDF, and Keras Tokenizer (KT). Last, the proposed methodology was evaluated and compared to the stateof-the-art techniques, including K-Nearest Neighbors (KNN), Decision Tree (DT), Naive Bayes (NB), and Boosting SVM. The experimental results revealed that the proposed method in this research outperforms the existing technique as evidenced by the accuracy, precision, recall, and F1-score metrics, which respectively reached 71.5655%, 71.5483%, 71.5655%, and 71.0041%.

Comparison of Artificial Decision Techniques for Detection of Sarcastic News Headlines

Newspapers are a rich informational source. A headline of an article sparks an interest in the reader. So, news providing agencies tend to create catchy headlines to attract the reader's attention onto them, and this is how sarcasm manages to find its way into news headlines. Sarcasm employs the use of words that carry opposite meaning with respect to what needs to be conveyed. This leads to the need of developing methods by which we can correctly predict whether a piece of text, or news for that matter, truthfully means what it says or is simply being sarcastic about it. Here, the authors have used a dataset containing 55,329 tuples consisting of news headlines from The Onion and the Huffington Post, which was taken from Kaggle, on which they applied feature extraction techniques such as Count Vectorizer, TF-IDF, Hashing Vectorizer, and Global Vectorizer (GloVe). Then they applied seven classifiers on the obtained dataset. The experimental results showed that the highest accuracies among the ML models were 81.39% for LR model with Count Vectorizer, 79.2% for LR model with TF-IDF Vectorizer, and 78% for SVM model with Count Vectorizer. They also obtained the best accuracy of 90.7% using the Bi-LSTM Deep Learning Model. They have trained the seven models and compared them based on their respective accuracies and F1-Scores.

Experimental of vectorizer and classifier for scrapped social media data

In this study, we used several classifiers and vectorizers to see their effect on processing social media data. In this study, the classifiers used were random forest, logistic regression, Bernoulli Naive Bayes (NB), and support vector clustering (SVC). Random forests are used to reduce spatial complexity, and also to minimize errors. Logistic regression is a method with a statistical model whose basic form uses a logistic function to represent the binary dependent variable. Then, the Naive Bayes function uses binary elements and SVC which has so far given good results rivals other guided learning. Our tests use social media data. Based on the tests that have been carried out on classifier variations and vectorizer variations, it was found that the best classifier is a linear regression algorithm based on predictive adaptive compared to the random forest method based on decision trees, probability-based Bernoulli NB and SVC which work by clustering. Meanwhile, from the test results on the count vectorizer, term frequency-inverse document frequency (TFIDF), and hashing, the best accuracy is achieved on the TFIDF vectorizer. In this case, it means that the TFIDF vectorizer has a better value in presenting word feature dimensions.

A State-Equation-Based Backward Approach to a Legal Firing Sequence Existence Problem in Petri Nets

Reachability is the basis for studying other dynamic properties of Petri nets (PNs). When a state equation is used to determine the reachability of a marking, we need to judge whether there is a corresponding legal firing sequence (LFS) for a non-negative integer solution (NIS), i.e., a firing count vector, of the state equation. The search for an LFS is an NP-hard problem, and previous work cannot always find an LFS for any NISs. This article proposes that transition-dependent circuits or firing-dependent circuits are the root cause that a state equation has an NIS but the marking is nonreachable, i.e., there is no LFS corresponding to an NIS in PNs. Based on this, we propose a state-equation-based backward algorithm (SBA) to determine whether there is an LFS corresponding to an NIS of the state equation in a PN. The correctness and effectiveness of SBA are verified by a case study on a PN-based flexible manufacturing system and through simulation on an S <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> PR net. The experimental results show that the time required for SBA to determine the existence of an LFS increases linearly with the transition firing count in NISs. When the number of NISs of a state equation is finite, we can efficiently determine the reachability of a marking. This represents an important result in theory and applications of PNs.

Structural Analysis of URL For Malicious URL Detection Using Machine Learning

Malicious websites are intentionally created websites that aid online criminals in carrying out illicit actions. They commit crimes like installing malware on the victim's computer, stealing private data from the victim's system, and exposing the victim online. Malicious codes can also be found on legitimate websites. Therefore, locating such a website in cyberspace is a difficult operation that demands the utilization of an automated detection tool. Currently, machine learning/deep learning technologies are employed to detect such malicious websites. However, the problem persists since the attack vector is constantly changing. Most research solutions use a limited number of URL lexical features, DNS information, global ranking information, and webpage content features. Combining several derived features involves computation time and security risk. Additionally, the dataset's minimal features don't maximize its potential. This paper exclusively uses URLs to address this problem and blends linguistic and vectorized URL features. Complete potential of the URL is utilized through vectorization. Six machine learning algorithms are examined. The results indicate that the proposed approach performs better for the count vectorizer with random forest algorithm

Text Algorithms in Economics

This article provides an overview of the methods used for algorithmic text analysis in economics, with a focus on three key contributions. First, we introduce methods for representing documents as high-dimensional count vectors over vocabulary terms, for representing words as vectors, and for representing word sequences as embedding vectors. Second, we define four core empirical tasks that encompass most text-as-data research in economics and enumerate the various approaches that have been taken so far to accomplish these tasks. Finally, we flag limitations in the current literature, with a focus on the challenge of validating algorithmic output.

Cycles in Mallows random permutations

AbstractWe study cycle counts in permutations of drawn at random according to the Mallows distribution. Under this distribution, each permutation is selected with probability proportional to , where is a parameter and denotes the number of inversions of . For fixed, we study the vector where denotes the number of cycles of length in and is sampled according to the Mallows distribution. When the Mallows distribution simply samples a permutation of uniformly at random. A classical result going back to Kolchin and Goncharoff states that in this case, the vector of cycle counts tends in distribution to a vector of independent Poisson random variables, with means . Here we show that if is fixed and then there are positive constants such that each has mean and the vector of cycle counts can be suitably rescaled to tend to a joint Gaussian distribution. Our results also show that when there is a striking difference between the behavior of the even and the odd cycles. The even cycle counts still have linear means, and when properly rescaled tend to a multivariate Gaussian distribution. For the odd cycle counts on the other hand, the limiting behavior depends on the parity of when . Both and have discrete limiting distributions—they do not need to be renormalized—but the two limiting distributions are distinct for all . We describe these limiting distributions in terms of Gnedin and Olshanski's bi‐infinite extension of the Mallows model. We investigate these limiting distributions further, and study the behavior of the constants involved in the Gaussian limit laws. We for example show that as the expected number of 1‐cycles tends to —which, curiously, differs from the value corresponding to . In addition we exhibit an interesting “oscillating” behavior in the limiting probability measures for and odd versus even.

Policy-Based Spam Detection of Tweets Dataset

Spam communications from spam ads and social media platforms such as Facebook, Twitter, and Instagram are increasing, making spam detection more popular. Many languages are used for spam review identification, including Chinese, Urdu, Roman Urdu, English, Turkish, etc.; however, there are fewer high-quality datasets available for Urdu. This is mainly because Urdu is less extensively used on social media networks such as Twitter, making it harder to collect huge volumes of relevant data. This paper investigates policy-based Urdu tweet spam detection. This study aims to collect over 1,100,000 real-time tweets from multiple users. The dataset is carefully filtered to comply with Twitter’s 100-tweet-per-hour limit. For data collection, the snscrape library is utilized, which is equipped with an API for accessing various attributes such as username, URL, and tweet content. Then, a machine learning pipeline consisting of TF-IDF, Count Vectorizer, and the following machine learning classifiers: multinomial naïve Bayes, support vector classifier RBF, logical regression, and BERT, are developed. Based on Twitter policy standards, feature extraction is performed, and the dataset is separated into training and testing sets for spam analysis. Experimental results show that the logistic regression classifier has achieved the highest accuracy, with an F1-score of 0.70 and an accuracy of 99.55%. The findings of the study show the effectiveness of policy-based spam detection in Urdu tweets using machine learning and BERT layer models and contribute to the development of a robust Urdu language social media spam detection method.

Comparison analysis of Bangla news articles classification using support vector machine and logistic regression

In the information age, Bangla news articles on the internet are fast-growing. For organizing, every news site has a particular structure and categorization. News article classification is a method to determine a document’s classification based on various predefined categories. This research discusses the classification of Bangla news articles on the online platform and tries to make constructive comparison using several classification algorithms. For Bangla news articles classification, term frequencyinverse document frequency (TF-IDF) weighting and count vectorizer have been used as a feature extraction process, and two common classifiers named support vector machine (SVM) and logistic regression (LR) employed for classifying the documents. It is clear that the accuracy of the experimental results by applying SVM is 84.0% and LR is 81.0% for twelve categories of news articles. In this research work, when we have made comparison two renowned classification algorithms applied on the Bangla news articles, LR was outperformed by SVM.

Comparison of the TF-IDF Method with the Count Vectorizer to Classify Hate Speech

Hate speech is a form of expression used to spread hatred and commit acts of violence and discrimination against a person or group of people for various reasons. Cases of hate speech are very common in social media, one of which is Twitter. The goal to be achieved is to create a system that can classify a tweet on Twitter into hate speech (HS) or non-hate speech (NONHS) classes. The method used is Support Vector Machine by comparing the features of TF-IDF and Count Vectorizer. And the parameters compared are seen from accuracy, precision, recall, and f1-score. Results obtained, overall, by using the TF-IDF feature, the Support Vector Machine algorithm gets high results compared to the Count Vectorizer feature, with an accuracy value of 88.77%, 87.45% precision, 88.77% recall, and f1-score of 87.81%.

Automotive Movie Recommendation System based on Natural Language Processing

People are puzzled about which movie to watch these days because there are so many movies available on various OTT platforms. A recommender system would solve this problem by recommending the best movie to the user based on his genre, actor, director, and rating preferences. The cosine similarity principle would be used to guide the recommendation system. Apart from that, we will use the Tfidftransformer and count vectorizer from the sci-kit-learn library in Python in this work. In this study work, all of the approaches' constraints have been described. All of this work was done using datasets from several OTT platforms that were available on Kaggle.

Optimal Statistical Analyses of Bell Experiments

We show how both smaller and more reliable p-values can be computed in Bell-type experiments by using statistical deviations from no-signalling equalities to reduce statistical noise in the estimation of Bell’s S or Eberhard’s J. Further improvement was obtained by using the Wilks likelihood ratio test based on the four tetranomially distributed vectors of counts of the four different outcome combinations, one 4-vector for each of the four setting combinations. The methodology was illustrated by application to the loophole-free Bell experiments of 2015 and 2016 performed in Delft and Munich, at NIST, and in Vienna, respectively, and also to the earlier (1998) Innsbruck experiment of Weihs et al. and the recent (2022) Munich experiment of Zhang et al., which investigates the use of a loophole-free Bell experiment as part of a protocol for device-independent quantum key distribution (DIQKD).