Malicious URLs Research Articles

Web of shadows: Investigating malware abuse of internet services

Internet Web and cloud services are routinely abused by malware, but the breadth of this abuse has not been thoroughly investigated. In this work, we quantitatively investigate this abuse by leveraging data from the Cyber Threat Alliance (CTA), where 36 security vendors share threat intelligence. We analyze CTA data collected over 4 years from January 2020 until December 2023 comprising over one billion cyber-security observations from where we extract 7.7M URLs and 1.8M domains related to malware. We complement this dataset with an active measurement where we periodically attempt to download the content pointed out by 33,876 recently reported malicious URLs. We investigate the following questions. How generalized is malware abuse of Internet services? How do domains of abused Internet services differ? For what purpose are Internet services abused? and How long do malicious resources remain active? Among others, we uncover a broad abuse affecting 22K domains of Internet services, that Internet services are largely abused for enabling malware distribution, and that malicious content in Internet services remains active longer than on malicious domains.

New Heuristics Method for Malicious URLs Detection Using Machine Learning

Malicious URLs are a very prominent, dangerous form of cyber threats in view of the fact that they can enable many evils like phishing attacks, malware distribution, and several other kinds of cyber fraud. The techniques of detection conventionally applied are based on blacklisting and heuristic analyses, which are gradually becoming inefficient against sophisticated, rapidly evolving threats. In this paper, the authors present various machine learning techniques applied in malicious URL detection. In the present paper, we will look at three machine learning models: Logistic Regression, Random Forest, and Support Vector Machines. We used a methodology that involved collecting data and feature extraction, training a model, then evaluating its performance with different metrics such as accuracy, precision, recall, and F1-score. We implemented and optimized three models—Logistic Regression, Random Forest, and Support Vector Machines (SVM)—based on the literature available that indicates the effectiveness of these models. Logistic Regression shows promising results to detect the malicious URLs, according to Vanitha and Vinodhini. Random Forest models are found to be very robust and accurate according to Cui et al. and Vanhoenshoven et al., SVM models are evidenced to have very high accuracy according to Manjeri et al., Further works on deep learning models emphasized their potentials. In our study, the optimized Random Forest model in our case showed the best performance, and its training accuracy was 99%, while validation accuracy was 90.5%, also logistic Regression and SVM achieved training accuracy was 89.31%, while validation accuracy was 90.5%. All the optimization processes, model performances, and integration into the real-time cybersecurity infrastructures, along with the strengths and limitations, are discussed in this paper. The paper will, therefore, discuss the benefits and challenges for each model in this aspect—emphasizing continuous updating of the models and integrating them into real-time cybersecurity infrastructures.

PMANet: Malicious URL detection via post-trained language model guided multi-level feature attention network

The expansion of the Internet has led to the widespread proliferation of malicious URLs, becoming a primary vector for cyber threats. Detecting malicious URLs is now essential for improving network security. The technological revolution spurred by pre-trained language models holds great promise for advancing the detection of malicious URLs. However, current research applying these models to URLs fails to address several crucial factors, including the lack of domain-specific adaptability, the omission of character-level information, and the neglect of both local detail extraction and low-order encoding information. In this paper, we propose PMANet, a pre-trained Language Model-Guided multi-level feature attention network, for addressing these issues. To facilitate a smooth transition of the pre-trained Transformer into the URL domain and to enable it to effectively capture information at both subword and character levels, we propose a post-training program that continues training the model on URLs using three self-supervised learning objectives: masked language model, noisy language model, and domain discrimination task. Subsequently, we develop a module to capture the output of each encoding layer, thus extracting hierarchical representations of URLs spanning from low-level to high-level. In addition, we propose a layer-wise attention mechanism that dynamically assigns weight coefficients to these feature layers based on their relevance. Finally, we apply spatial pyramid pooling to perform multi-scale down-sampling in order to obtain both local features and global context. PMANet achieves multifaceted integration in URL feature extraction, including capturing information at both the lexical and character levels, extracting features from low to high order, and discerning patterns at both global and local scales. We evaluate PMANet against challenging real-world scenarios, such as small-scale data, class imbalance, cross-dataset, adversarial attacks, and case studies on active malicious URLs. All experiments demonstrate that PMANet exhibits superiority over both the previous state-of-the-art pre-trained models and conventional deep learning models. Specifically, PMANet still achieves a 0.9941 AUC under adversarial attacks and correctly identifies all 20 actively malicious URLs in the case study. The code and data for our research are available at: https://github.com/Alixyvtte/Malicious-URL-Detection-PMANet.

TransURL: Improving malicious URL detection with multi-layer Transformer encoding and multi-scale pyramid features

While machine learning progress is advancing the detection of malicious URLs, advanced Transformers applied to URLs face difficulties in extracting local information, character-level information, and structural relationships. To address these challenges, we propose a novel approach for malicious URL detection, named TransURL, that is implemented by co-training the character-aware Transformer with three feature modules—Multi-Layer Encoding, Multi-Scale Feature Learning, and Spatial Pyramid Attention. This special Transformer allows TransURL to extract embeddings that contain character-level information from URL token sequences, with three feature modules contributing to the fusion of multi-layer Transformer encodings and the capture of multi-scale local details and structural relationships. The proposed method is evaluated across several challenging scenarios, including class imbalance learning, multi-classification, cross-dataset testing, and adversarial sample attacks. The experimental results demonstrate a significant improvement compared to the best previous methods. For instance, it achieved a peak F1-score improvement of 40% in class-imbalanced scenarios, and exceeded the best baseline result by 14.13% in accuracy in adversarial attack scenarios. Additionally, we conduct a case study where our method accurately identifies all 30 active malicious web pages, whereas two pior SOTA methods miss 4 and 7 malicious web pages respectively. The codes and data are available at: https://github.com/Vul-det/TransURL/.

큐싱 공격 탐지를 위한 AutoML 머신러닝 기반 악성 URL 분류 기술 연구 및 서비스 구현

In recent trends, there has been an increase in 'Qshing' attacks, a hybrid form of phishing that exploits fake QR (Quick Response) codes impersonating government agencies to steal personal and financial information. Particularly, this attack method is characterized by its stealthiness, as victims can be redirected to phishing pages or led to download malicious software simply by scanning a QR code, making it difficult for them to realize they have been targeted. In this paper, we have developed a classification technique utilizing machine learning algorithms to identify the maliciousness of URLs embedded in QR codes, and we have explored ways to integrate this with existing QR code readers. To this end, we constructed a dataset from 128,587 malicious URLs and 428,102 benign URLs, extracting 35 different features such as protocol and parameters, and used AutoML to identify the optimal algorithm and hyperparameters, achieving an accuracy of approximately 87.37%. Following this, we designed the integration of the trained classification model with existing QR code readers to implement a service capable of countering Qshing attacks. In conclusion, our findings confirm that deriving an optimized algorithm for classifying malicious URLs in QR codes and integrating it with existing QR code readers presents a viable solution to combat Qshing attacks.

An integrated model based on deep learning classifiers and pre-trained transformer for phishing URL detection

The unique nature of website URLs has made phishing detection a challenging task. Unlike natural language, URLs have an unstructured nature with non-linear and sophisticated correlations. Therefore, they should be handled as both natural language and unstructured data sequences. However, the current solutions for phishing URL detection only focused on a single aspect of web page URLs. In this concern, this paper proposes an integrated model based on DL classifiers and pre-trained transformer to examine both the unique nature and the natural language structure of URL sequences simultaneously. The proposed model consists of three modules: RasNet (Keras-ResNet), TCMA (TCN-MHSA), and MPNet (Masked and Permuted Pre-training for Language Understanding). Considering the unique nature of the input data, RasNet combines two Keras embedding techniques to obtain the feature representations of URLs and then fuses them using a Residual Network (ResNet) to balance the weight distribution among the character-level and word-level information. Additionally, TCMA integrates the Temporal Convolutional Network (TCN) with the Multi-Head Self-Attention (MHSA) mechanism to optimize feature extraction and improve classification accuracy. Concurrently, MPNet joins the advantages and eliminates the drawbacks of Masked Language Modelling and Permuted Language Modelling to examine the nature language structure of web page URLs. The proposed model was trained and tested on four different datasets, including Ebbu2017, PhishCrawl, 420K-PD, and 1M-PD. The experimental results indicated that the proposed solution outperformed other models in classifying malicious URLs with the highest detection rate of 99.71% on the 1M-PD dataset, improving the performance accuracy of the state-of-the-art approaches by 1.37% to 2.01%.

A Large Scale Study and Classification of VirusTotal Reports on Phishing and Malware URLs

VirusTotal (VT) is a widely used scanning service for researchers and practitioners to label malicious entities and predict new security threats. Unfortunately, it is little known to the end-users how VT URL scanners decide on the maliciousness of entities and the attack types they are involved in (e.g., phishing or malware-hosting websites). In this paper, we conduct a systematic comparative study on VT URL scanners' behavior for different attack types of malicious URLs, in terms of 1) detection specialties, 2) stability, 3) correlations between scanners, and 4) lead/lag behaviors. Our findings highlight that the VT scanners commonly disagree with each other on their detection and attack type classification, leading to challenges in ascertaining the maliciousness of a URL and taking prompt mitigation actions according to different attack types. This motivates us to present a new highly accurate classifier that helps correctly identify the attack types of malicious URLs at the early stage. This in turn assists practitioners in performing better threat aggregation and choosing proper mitigation actions for different attack types.

Generative adversarial network-based phishing URL detection with variational autoencoder and transformer

Phishing attacks pose a constant threat to online security, necessitating the development of efficient tools for identifying malicious URLs. In this article, we propose a novel approach to detect phishing URLs employing a generative adversarial network (GAN) with a variational autoencoder (VAE) as the generator and a transformer model with self-attention as the discriminator. The VAE generator is trained to produce synthetic URLs. In contrast, the transformer discriminator uses its self-attention mechanism to focus on the different parts of the input URLs to extract crucial features. Our model uses adversarial training to distinguish between legitimate and phishing URLs. We evaluate the effectiveness of the proposed method using a large set of one million URLs that incorporate both authentic and phishing URLs. Experimental results show that our model is effective, with an impressive accuracy of 97.75%, outperforming the baseline models. This study significantly improves online security by offering a novel and highly accurate phishing URL detection method.

Ransomware Threat Analysis Using Machine Learning

With the advancement and easy accessibility of computer and internet technology, network security has become vulnerable to hacker threats, including ransomware attacks targeting smartphone operating systems (e.g., Android) and applications. Despite efforts to detect malicious URLs, lexical features sometimes need to be improved. This document provides an overview of ransomware, a timeline of assaults, and comprehensive research on methods for identifying, avoiding, minimising, and recovering from ransomware attacks. Analysing studies between 2017 and 2024 offers up-to- date knowledge on developments in ransomware detection and advancements in combating ransomware attacks, highlighting unanswered concerns and potential research challenges. KEYWORDS: Machine Learning; Ransomware, Cybers-attacks; Android; Malware; Hacker; Ransomware detection.

URL Detection Based on Quantum Long Short-Term Memory Neural Network

In today's era, with the continuous development of the internet, surfing the internet has become an inseparable part of our lives. For those who frequently roam the internet, URLs are information that we are constantly exposed to. And YRL is also divided into malicious and normal. Clicking on malicious URLs can cause certain losses to our internet tools, and in severe cases, it can even lead to money theft. Malicious URL detection is an important task in the field of network security, aimed at identifying and preventing potential network attacks and malicious activities. Traditional malicious URL detection methods, such as Blacklist based detection and Machine learning based on feature extraction, have achieved certain results, but face challenges in real-time, accuracy, and data requirements. Therefore, this article proposes the use of Quantum Long Short-Term Memory Neural Network (QLSTM) to solve this problem. By observing the accuracy, recall, and F1 values obtained from training a large amount of data on the QLSTM network, it is shown that this method is feasible.

SaleSurf - Strengthening the Web's Armour, One URL at a Time - Malicious URL Detection using Machine Learning Models

One of the most common cybersecurity vulnerabilities involves malicious websites or URLs. Every year, individuals and organizations suffer major financial losses from using harmful content such as spam, malware, inappropriate advertising, and scams that encourage visitors to cheat. These malicious URLs are often promoted through emails, advertisements, web search results, or links to other websites. Considering how many users click on these malicious URLs, there is an urgent need for a reliable system that can classify and identify dangerous URLs; In particular, phishing, spam and malware attacks are increasing. Data volume, updated attack models and strategies, correlation between URL features, lack of data, inconsistent data and the presence of outliers make the division of labor very difficult. In our research, we focus on negative URL search to gain more insight. Our information is divided into four main categories: phishing, harmless (safe), tampering, and malware. We have collected a large database of 651,191 URLs to support our application. To achieve the goal of identifying and identifying malicious URLs, we use three machine learning algorithms: Random Forest, LightGBM, XGBoost, Logistic Regression, CNN and Ensemble Model.

Ensemble Learning-Based Browser Extension for Mitigating Cyber Attacks Carried out using Malicious Short URLs

As digitization continues to expand and cybercrimes become more prevalent, making it is crucial to prioritize the implementation of robust security measures. Malicious short URLs are frequently utilized as a vector for cyber-attacks on online forums and social media platforms. To address this issue, a plugin-based solution that uses ensemble learning to combine random forest, k-neighbors classifier, and logistic regression into a stacked model, was developed. The model was trained over the combination of three most popular kaggle datasets, with over 1081195 URLs. Additionally, gradient boosting was applied to further enhance the model's performance, resulting in a 92% accuracy in the detection. We developed the browser extension with Flask and JavaScript that identifies URLs as malicious or safe, for facilitation of the proposed solution. The work emphasizes the need for effective measures to mitigate cyber-attack risks, particularly those involving malicious short URLs.

Malicious Social Bot Using Twitter Network Analysis in Django

Malicious social bots generate fake tweets and automate their social relationships either by pretending to be a followers or by creating multiple fake accounts with malicious activities. Moreover, malicious social bots post shortened malicious URLs in the tweets to redirect the requests of online social networking participants to some malicious servers. Hence, distinguishing malicious social bots from legitimate users is one of the most important tasks in the Twitter network. To detect malicious social bots, extracting URL-based features (such as URL redirection, frequency of shared URLs, and spam content in URL) consumes less amount of time in comparison with social graph-based features (which rely on the social interactions of users). Furthermore, malicious social bots cannot easily manipulate URL redirection chains. In this article, learning automata-based malicious social bot detection (LA-MSBD) algorithm is proposed by integrating a trust computation model with URL-based features for identifying trustworthy participants (users) in the Twitter network. The proposed trust computation model contains two parameters, namely, direct trust and indirect trust. Moreover, the direct trust is derived from Bayes’ theorem, and the indirect trust is derived from the Dempster– Shafer theory (DST) to determine the trustworthiness of each participant accurately. Finally, we shown the user tweet data in terms of graph visualization of bar chart and pie chart of the system. Experimental results shown the better performance of the system.

URL Security Assessment Using Machine Learning

The significance of the World Wide Web has expanded considerably over time. However, alongside technological progress, there has been a rise in the complexity of methods aimed at exploiting users. These efforts often involve infecting users' computers with malware or directing them to unfriendly websites for purposes such as peddling counterfeit goods or exposing sensitive information, leading to financial fraud. Malicious URLs pose a significant threat to potential victims by hosting a range of unwanted content. Therefore, there is a pressing need for a rapid and effective detection approach. This thesis addresses the challenge of identifying hazardous URLs by leveraging URL data and machine learning techniques. Various machine learning algorithms, including XGBoost, LightGBM, and Random Forest, will be utilized for this purpose. Key Words: Malicious URLs, XGBoost, LightGBM, Random Forest

Detecting Malicious Twitter Bots using Machine Learning

Abstract: Twitter play a significant role in our daily lives, offering a wide range of opportunities to their users. However, Twitter and online social networks (OSNs) in general are increasingly being utilized by automated accounts, commonly known as bots, as they continue to gain immense popularity across various user demographics. Malicious twitter Bots detection is required to detect real users from fraudulent users because it leads to spreading of spam messages and engage in fraudulent activities. To overcome this, we are going to differentiate bots from legitimate users using feature extraction techniques and find malicious bots and tweets using machine learning algorithm and deep learning architecture known as VGG19 which is combined with the convolutional neural network (CNN) in order to identify whether the tweets are posted by bots or real users and also identifying malicious twitter bots along with malicious URLs. By following these techniques, we can identify the account as bots or real user and prevent spreading of malicious content in the society. The deep learning architecture combined with convolutional neural network to evaluate over a series of experiments using two large real Twitter datasets and compare the experimental results with other machine learning algorithms and provide advantages over other existing techniques like logistic regression targeting the identification of malicious users in social media.

Phishing URL Detection using Machine Learning

Abstract: Phishing attacks pose a significant threat in cyberspace, Cybercrimes such as Phishing means accessing personal data and violating security through the Internet and its main aim is to steal the information from the users using different techniques in that the primary one is Phishing which demands effective detection mechanisms. This study evaluates the performance of Gradient Boosting Classifier, Random Forest, and Decision Tree machine learning models in conjunction with feature selection techniques namely SelectKBest and Chi-Square. Initially, a comprehensive feature set of 30 attributes achieved a baseline accuracy of 97.4%. Through SelectKBest and Chi-Square feature selection methods, 13 key features were identified, leading to a slightly decreased accuracy of 95.6% upon model retraining. This research highlights the importance of feature selection in enhancing phishing detection accuracy while maintaining model interpretability. Technologies such as NumPy, Pandas, Matplotlib, Scikit-learn, and Flask drive. The project, emphasising the exploration of ML models, EDA (Exploratory Data Analysis) on phishing data, and understanding feature importance. The Machine Learning Models like Gradient Boosting Classifier, Random Forest and Decision tree are used to detect whether the given URL is Malicious URL or Legitimate URL using Comprehensive Feature set and Key Feature set. This Models and Feature sets are compared based on the Performance Metrics namely Accuracy, Precision, Recall and F1- score. This study contributes to advancing cyber defence mechanism through the fusion of sophisticated Machine Learning algorithms and meticulous Feature Selection methodologies.

Toward More Generalized Malicious URL Detection Models

This paper reveals a data bias issue that can profoundly hinder the performance of machine learning models in malicious URL detection. We describe how such bias can be diagnosed using interpretable machine learning techniques and further argue that such biases naturally exist in the real world security data for training a classification model. To counteract these challenges, we propose a debiased training strategy that can be applied to most deep-learning based models to alleviate the negative effects of the biased features. The solution is based on the technique of adversarial training to train deep neural networks learning invariant embedding from biased data. Through extensive experimentation, we substantiate that our innovative strategy fosters superior generalization capabilities across both CNN-based and RNN-based detection models. The findings presented in this work not only expose a latent issue in the field but also provide an actionable remedy, marking a significant step forward in the pursuit of more reliable and robust malicious URL detection.

Detection of malicious URLs using machine learning

AbstractThe detection of fraudulent URLs that lead to malicious websites using addresses similar to those of legitimate websites is a key form of defense against phishing attacks. Currently, in the case of Internet of Things devices is especially relevant, because they usually have access to the Internet, although in many cases they are vulnerable to these phishing attacks. This paper offers an overview of the most relevant techniques for the accurate detection of fraudulent URLs, from the most widely used machine learning and deep learning algorithms, to the application, as a proof of concept, of classification models based on quantum machine learning. Starting from an essential data preparation phase, special attention is paid to the initial comparison of several traditional machine learning models, evaluating them with different datasets and obtaining interesting results that achieve true positive rates greater than 90%. After that first approach, the study moves on to the application of quantum machine learning, analysing the specificities of this recent field and assessing the possibilities it offers for the detection of malicious URLs. Given the limited available literature specifically on the detection of malicious URLs and other cybersecurity issues through quantum machine learning, the research presented here represents a relevant novelty on the combination of both concepts in the form of quantum machine learning algorithms for cybersecurity. Indeed, after the analysis of several algorithms, encouraging results have been obtained that open the door to further research on the application of quantum computing in the field of cybersecurity.

AntiPhishStack: LSTM-Based Stacked Generalization Model for Optimized Phishing URL Detection

The escalating reliance on revolutionary online web services has introduced heightened security risks, with persistent challenges posed by phishing despite extensive security measures. Traditional phishing systems, reliant on machine learning and manual features, struggle with evolving tactics. Recent advances in deep learning offer promising avenues for tackling novel phishing challenges and malicious URLs. This paper introduces a two-phase stack generalized model named AntiPhishStack, designed to detect phishing sites. The model leverages the learning of URLs and character-level TF-IDF features symmetrically, enhancing its ability to combat emerging phishing threats. In Phase I, features are trained on a base machine learning classifier, employing K-fold cross-validation for robust mean prediction. Phase II employs a two-layered stacked-based LSTM network with five adaptive optimizers for dynamic compilation, ensuring premier prediction on these features. Additionally, the symmetrical predictions from both phases are optimized and integrated to train a meta-XGBoost classifier, contributing to a final robust prediction. The significance of this work lies in advancing phishing detection with AntiPhishStack, operating without prior phishing-specific feature knowledge. Experimental validation on two benchmark datasets, comprising benign and phishing or malicious URLs, demonstrates the model’s exceptional performance, achieving a notable 96.04% accuracy compared to existing studies. This research adds value to the ongoing discourse on symmetry and asymmetry in information security and provides a forward-thinking solution for enhancing network security in the face of evolving cyber threats.

Phishing attack detection using gradient boosting

Phishing is a prevalent cyber attack that uses deceptive websites to trick individuals into revealing personal information. These sites mimic legitimate ones to steal data such as usernames, passwords, and financial details. Detecting phishing is crucial, and machine learning algorithms are effective tools for this task. Attackers favor phishing due to its effectiveness in tricking victims with authentic-looking yet malicious links, which can breach security measures. This method employs machine learning to innovate phishing website detection. However, attackers can manipulate features like HTML, DOM, and URLs using web scraping and scripting languages. A new approach using machine learning classifiers tackles these threats by analyzing internet URLs and domain names. A dataset sourced from globally recognized intelligence services and organizations facilitates streamlined feature extraction, reducing processing overhead by prioritizing URL and domain name traits. The Gradient Boosting Classifier is used on an 11,055-instance dataset with thirty-two features to classify phishing URLs, demonstrating superior accuracy compared to methods like Random Forest. Gradient boosting is highly effective across various machine learning tasks, leveraging aggregated weak learners such as decision trees for strong predictive accuracy. Its suitability for handling imbalanced datasets makes it particularly effective for phishing detection, which is crucial for distinguishing between legitimate and malicious URLs. This method enhances accuracy by extracting and comparing distinct characteristics of legitimate and phishing URLs. By focusing on URL and domain name attributes, a more effective approach to identifying phishing attempts in cybersecurity is proposed.