Web Robot Research Articles

Exploiting web content semantic features to detect web robots from weblogs

Nowadays, web robots are predominantly used for auto-accessing web content, sharing almost one-third of the total web traffic and often posing threats to various web applications’ security, privacy, and performance. Detecting these robots is essential, and both online and offline methods are employed. One popular offline method is the use of weblog feature-based automated learning. However, this method alone cannot accurately identify web robots that continuously evolve and camouflage. Web content features combined with weblog features are used to detect such robots based on the assumption that human users exhibit specific interests while robots randomly navigate web pages. State-of-the-art web content-based feature methods lack the ability to generate coherent topics, which can confound the performance of classification models. Therefore, we propose a new content semantic feature extraction method that uses the LDA2Vec topic model, combining the strengths of LDA and the Word2Vec model to produce more semantically coherent topics by exploiting website content for a web session. To effectively detect web robots, web resource content semantic features are combined with log-based features in the proposed web robot detection approach. The proposed approach is evaluated in an e-commerce website access logs and content data. The F-score, balanced accuracy, G-mean, and Jaccard similarity are used for performance measures, and the coherence score metric is used to determine the number of topics for a session. Experimental results demonstrate that a combination of weblogs and content semantic features is effective in web robot detection.

Evaluating Machine Learning Techniques for Web Robot Detection

A web robot, also known as a web crawler, is an automated script that systematically browses the World Wide Web in a methodical manner. Web robots are commonly used by search engines to index web pages, gather information, and update search engine databases. They follow hyperlinks from one web page to another, collecting data and information for various purposes such as indexing, data mining, and website monitoring. Web robot detection is the process of identifying and distinguishing between human users and web robots on websites, which are a major source of web traffic. This process is vital to prevent malicious web robots from having a negative effect on web servers’ traffic and their users’ privacy. Companies such as Imperva, Inc. use machine learning models to identify malicious bots and prevent them from having unauthorized access to an organization’s server. To understand the benefit that machine learning models bring to web robot detection, we used pre-published server log data to construct machine learning models on Orange (a Data Mining Software) and Python that can distinguish between malicious and benign web robots. We evaluated the performance of three well-known machine learning algorithms: kNN, neural network, and decision tree. Based on our experimental results, the neural network gains the highest precision and the lowest false-positive and false-negative percentage of web robots. However, the neural network requires more time to generate the desired output.

Phenotypic Environmental Sensitivity and Mental Health During Pregnancy and Post Partum: Protocol for the Experiences of Pregnancy Longitudinal Cohort Study.

Mental health problems during pregnancy and post partum are common and associated with negative short- and long-term impacts on pregnant individuals, obstetric outcomes, and child socioemotional development. Socio-environmental factors are important predictors of perinatal mental health, but the effects of the environment on mental health are heterogeneous. The differential susceptibility theory and the environmental sensitivity framework suggest that individuals differ in their degree of sensitivity to positive and negative environments, which can be captured by individual phenotypes such as temperament and personality. While there is strong evidence for these models in childhood, few studies examined them in adults, and they were not examined in pregnancy. The primary objective of the Experiences of Pregnancy study is to explore whether childhood and current environments are associated with mental health and well-being in pregnancy and whether these effects depend on individual sensitivity phenotypes (personality). This study also aims to gather important psychosocial and health data for potential secondary data analyses and integrative data analyses. We will conduct a longitudinal cohort study. The study was not registered elsewhere, other than this protocol. Participants will be recruited through social media advertisements linking to the study website, followed by an eligibility call on Zoom (Zoom Video Communications). Participants must be aged 18 years or older, currently residing in the United States as citizens or permanent residents, and currently planning to continue the pregnancy. A minimum of 512 participants will be recruited based on power analyses for the main objectives. Since the data will also be a resource for secondary analyses, up to 1000 participants will be recruited based on the available budget. Participants will be in their first trimester of pregnancy, and they will be followed at each trimester and once post partum. Data will be obtained through self-reported questionnaires assessing demographic factors; pregnancy-related factors; delivery, labor, and birth outcomes; early infant feeding; individual personality factors; childhood and current environments; mental health and well-being; attachment; and infant temperament. A series of measures were taken to safeguard the study from web robots and fraudulent participants, as well as to reduce legal and social risks for participants following Dobbs v. Jackson. The study received ethics approval in April 2023 from the University of Oklahoma-Norman Campus Institutional Review Board. Recruitment occurred from May to August 2023, with 3 follow-ups occurring over 10 months. The Experiences of Pregnancy study will extend theories of environmental sensitivity, mainly applied in children to the perinatal period. This will help better understand individual sensitivity factors associated with risk, resilience, plasticity, and receptivity to negative and positive environmental influences during pregnancy for pregnant individuals. PRR1-10.2196/49243.

FRS-SIFS: fuzzy rough set session identification and feature selection in web robot detection

FRS-SIFS: fuzzy rough set session identification and feature selection in web robot detection

Participant Preferences for the Development of a Digitally Delivered Gardening Intervention to Improve Diet, Physical Activity, and Cardiovascular Health: Cross-sectional Study.

Low dietary intake of fruits and vegetables and physical inactivity are 2 modifiable risk factors for cardiovascular disease. Fruit and vegetable gardening can provide access to fresh produce, and many gardening activities are considered moderate physical activity. This makes gardening interventions a potential strategy for cardiovascular disease risk reduction. Previously developed gardening interventions have relied on in-person delivery models, which limit scalability and reach. The purpose of this study was to ascertain participant insight on intervention components and topics of interest to inform a digitally delivered, gardening-focused, multiple health behavior change intervention. A web-based survey was delivered via Amazon Mechanical Turk (MTurk), including quantitative and open-ended questions. Eligible participants were aged ≥20 years, could read and write in English, were US residents, and had at least a 98% MTurk task approval rating. A multilevel screening process was used to identify and exclude respondents with response inattention, poor language fluency, or suspected automated web robots (bots). Participants were asked about their interest in gardening programming, their preferences for intervention delivery modalities (1-hour expert lectures, a series of brief <5-minute videos, or in-person meetings), and what information is needed to teach new gardeners. Comparisons were made between never gardeners (NG) and ever gardeners (EG) in order to examine differences in perceptions based on prior experience. Quantitative data were summarized, and differences between groups were tested using chi-square tests. Qualitative data were coded and organized into intervention functions based on the Behavior Change Wheel. A total of 465 participants were included (n=212, 45.6% NG and n=253, 54.4% EG). There was a high level of program interest overall (n=355, 76.3%), though interest was higher in EG (142/212, 67% NG; 213/253, 84.2% EG; P<.001). The majority of participants (n=282, 60.7%) preferred a series of brief <5-minute videos (136/212, 64.2% NG; 146/253, 57.7% EG; P=.16) over 1-hour lectures (29/212, 13.7% NG; 50/253, 19.8% EG; P=.08) or in-person delivery modes (47/212, 22.2% NG; 57/253, 22.5% EG; P=.93). Intervention functions identified were education and training (performing fundamental gardening and cooking activities), environmental restructuring (eg, social support), enablement (provision of tools or seeds), persuasion (offering encouragement and highlighting the benefits of gardening), and modeling (using content experts and participant testimonials). Content areas identified included the full lifecycle of gardening activities, from the fundamentals of preparing a garden site, planting and maintenance to harvesting and cooking. In a sample of potential web-based learners, participants were interested in a digitally delivered gardening program. They preferred brief videos for content delivery and suggested content topics that encompassed how to garden from planting to harvesting and cooking. The next step in this line of work is to identify target behavior change techniques and pilot test the intervention to assess participant acceptability and preliminary efficacy.

Web Robot Detection Based On Fuzzy System and PSO Algorithm

Web Robot Detection Based On Fuzzy System and PSO Algorithm

ReMouse Dataset: On the Efficacy of Measuring the Similarity of Human-Generated Trajectories for the Detection of Session-Replay Bots

Session-replay bots are believed to be the latest and most sophisticated generation of web bots, and they are also very difficult to defend against. Combating session-replay bots is particularly challenging in online domains that are repeatedly visited by the same genuine human user(s) in the same or similar ways—such as news, banking or gaming sites. In such domains, it is difficult to determine whether two look-alike sessions are produced by the same human user or if these sessions are just bot-generated session replays. Unfortunately, to date, only a handful of research studies have looked at the problem of session-replay bots, with many related questions still waiting to be addressed. The main contributions of this paper are two-fold: (1) We introduce and provide to the public a novel real-world mouse dynamics dataset named ReMouse. The ReMouse dataset is collected in a guided environment, and, unlike other publicly available mouse dynamics datasets, it contains repeat sessions generated by the same human user(s). As such, the ReMouse dataset is the first of its kind and is of particular relevance for studies on the development of effective defenses against session-replay bots. (2) Our own analysis of ReMouse dataset using statistical and advanced ML-based methods (including deep and unsupervised neural learning) shows that two different human users cannot generate the same or similar-looking sessions when performing the same or a similar online task; furthermore, even the (repeat) sessions generated by the same human user are sufficiently distinguishable from one another.

An Improved Text-Based and Image-Based CAPTCHA Based on Solving and燫esponse Time

CAPTCHA is an acronym that stands for Completely Automated Public Turing Test to tell Computers and Humans Apart (CAPTCHA), it is a good example of an authentication system that can be used to determine the true identity of any user. It serves as a security measure to prevent an attack caused by web bots (automatic programs) during an online transaction. It can come as text-based or image-based depending on the project and the programmer. The usability and robustness, as well as level of security, provided each of the varies and call for the development of an improved system. Hence, this paper studied and improved two different CAPTCHA systems (the text-based CAPTCHA and image-based CAPTCHA). The text-based and image-based CAPTCHA were designed using JavaScript. Response time and solving time are the two metrics used to determine the effectiveness and efficiency of the two CAPTCHA systems. The inclusion of response time and solving time improved the shortfall of the usability and robustness of the existing system. The developed system was tested using 200 students from the Federal College of Animal Health and Production Technology. The results of each of the participants, for the two CAPTCHAs, were extracted from the database and subjected to analysis using SPSS. The result shows that text-based CAPTCHA has the lowest average solving time (21.3333 s) with a 47.8% success rate while image-based CAPTCHA has the highest average solving time was 23.5138 s with a 52.8% success rate. The average response time for the image-based CAPTCHA was 2.1855 s with a 37.9% success rate lower than the text-based CAPTCHA response time (3.5561 s) with a 62.1% success rate. This indicates that the text-based CAPTCHA is more effective in terms of usability tests while image-based CAPTCHA is more efficient in terms of system responsiveness and recommended for potential users.

Free Innovative Tool to Address Memory Concerns and Brain Health for All Communities

AbstractBackgroundOver 60% of dementia cases are undetected, often due to fear, stigma, and lack of education about memory and brain health. To address this unmet need, UsAgainstAlzheimer’s developed BrainGuide, a free, web‐based consumer education platform designed to empower individuals to take the best next steps in managing their own or a loved one’s brain health.MethodBrainGuide was built with nonprofit, advocacy, industry, medical and technology experts. It offers two memory questionnaires created with input from medical experts and adapted from validated tools in the healthcare community. The self‐assessment is based on MIS/MIS‐T and the caregiver assessment is based on the AD8. Both were adapted for use with a voice and web bot. BrainGuide is not intended to diagnose or provide treatments. Six personas representing individuals concerned about their or a loved one’s brain health, caregivers, and individuals living with Alzheimer’s, were created to curate resources for users to browse after completing a questionnaire. Twenty‐five original tailored factsheets and 70+ links to external resources are also available. BrainGuide’s development was informed by market research including qualitative and quantitative feedback from users. It runs on Amazon Web Services making it available to anyone with telephone or Internet access and was developed with in‐kind and financial support by Biogen.ResultSince launching in March 2021, over 283,000 individuals have visited BrainGuide and 160,000 have completed a memory questionnaire; 79% of users are women, 78% of users are 60+ years and 17% are non‐white. 79% of individuals are taking the memory questionnaire for themselves, and the platform generates a summary to bring to a provider. 74% of users who took the questionnaire indicated that they felt more prepared to discuss brain health with a doctor through our post‐experience user survey.ConclusionData collection is ongoing, to measure consumer engagement and behavior change. To date, we have collected data including consumer confidence in raising the topic of memory with loved ones and providers, the adoption among different audience segments and the ongoing engagement of users. This data provides insights for the field as we work to address challenges to early detection and diagnosis.

Advanced Development of a Prayer Schedule Bot Application on Telegram Using PHP

A bot is a web robot or an assistant with computer intelligence. The usefulness of a bot is very much starting from the web, messenger, and others. One application possessing a provider to make a bot is a Telegram. The development of technology, such as social media applications, has made them prevalent for the public, including most Muslims. Sometimes, people will be negligent in worshiping when using the application. Therefore, creating a prayer schedule bot is necessary to prevent people from doing so. A framework creation aimed to facilitate the stages of making a bot using a PHP programming language. The bot-making results could be utilized in various countries with the calculation method found in a bot, providing a preference for users. Telegram bot is preferred for smartphone users wanting storage efficiency and having an internet connection. In contrast, the conventional prayer scheduler application does not care much about the storage taken and does not have an internet connection.

Web Bot Detection System Based on Divisive Clustering and K-Nearest Neighbor Using Biostatistics Features Set

Web bots are destructive programs that automatically fill the web form and steal the data from web sites. According to numerous web bot traffic reports, web bots traffic comprises of more than fifty percent of the total web traffic. An effective guard against the stealing of the data from web sites and automated web form is to identify and confirm the human user presence on web sites. In this paper, an efficient k-Nearest Neighbor algorithm using hierarchical clustering for web bot detection is proposed. Proposed technique exploits a novel taxonomy of web bot features known as Biostatistics Features. Numerous attack scenarios for web bot attacks such as automatic account registration, automatic form filling, bulk message posting, and web scrapping are created to imitate the zero-day web bot attacks. The proposed technique is evaluated with number of experiments using standard evaluation parameters. The experimental result analysis demonstrates that the proposed technique is extremely efficient in differentiating human users from web bots.

A Quantum-Inspired Classifier for Early Web Bot Detection

This paper introduces a novel approach, inspired by the principles of Quantum Computing, to address web bot detection in terms of real-time classification of an incoming data stream of HTTP request headers, in order to ensure the shortest decision time with the highest accuracy. The proposed approach exploits the analogy between the intrinsic correlation of two or more particles and the dependence of each HTTP request on the preceding ones. Starting from the a-posteriori probability of each request to belong to a particular class, it is possible to assign a Qubit state representing a combination of the aforementioned probabilities for all available observations of the time series. By leveraging the underlying mathematical details of superposition and entanglement on specific subsequences, it is possible to devise a measure of membership to each class, thus enabling the system to take a reliable decision when a sufficient level of confidence is met or to continue with additional observations. The results reported in this paper objectively show the effectiveness of our quantum-inspired algorithm which outperforms other state-of-the-art approaches, including our own one based on the Sequential Probability Ratio Test.

Performance Analysis of Indie Gaming Projects on Crowdfunding Platforms: Evidence from Kickstarter.com

Research question: The aim of this paper is to examine the financial success of projects in game development industry in comparison with to projects in other industries hosted on crowdfunding platform Kickstarter.com. Motivation: We live in the world of technology where companies arise and disappear on daily basis. The traditional way of financing was expanded with alternative online platforms. Our goal was to conduct an empirical analysis of one crowdfunding platform (Kickstarter.com) in order to understand if technology projects are doing better than other projects. If they do, what are the key factors for their success? Idea: Our goal was to better understand how crowdfunding model supports small indie development projects and projects in other industries. Data: We used the data of 148.510 companies applied for crowdfunding financing between 2015 and 2020, published on the web crawler platform Web Robots. Tools: All statistical analyses were performed using statistical software IBM® SPSS® Statistics v.21. The data were presented using standard methods of non-parametric descriptive statistics (absolute and relative frequencies for medians and interquartile ranges for numeric outcomes). For testing of statistical significance of difference between two groups we used the Pearson´s Chi-Quadrat test and Mann-Whitney test, where appropriate. The effect size for the 2x2 analyses was estimated using Odds ratios. Findings: The paper analyses the financial success of gaming and not gaming projects and tries to identify key factors for successful funding. We found a statistically significant higher prevalence of successful financing in game development projects, with 2.3 times higher odds of successful funding compared to not-gaming industry. Our analysis of quantitative indicators such as the number of backers, goal amount, pledged amount, pledged amount per backer and pledged to goal ratio also showed that projects in game development statistically outperformed projects in other categories. Promising game projects were supported by three times more backers on average and attained almost as double funds as other projects, while still sporting more modest pledged amounts per backer. These findings support the notion of crowdfunding being a viable modality of financing independent game development in emerging economies. Contribution: This paper expands the existing research related to the crowdfunding platforms and indie development companies and formulates key factors for successful financing for technology startup firms.

Web Bot Detection System Based on Divisive Clustering and K-Nearest Neighbor Using Biostatistics Features Set

Web bots are destructive programs that automatically fill the web form and steal the data from web sites. According to numerous web bot traffic reports, web bots traffic comprises of more than fifty percent of the total web traffic. An effective guard against the stealing of the data from web sites and automated web form is to identify and confirm the human user presence on web sites. In this paper, an efficient k-Nearest Neighbor algorithm using hierarchical clustering for web bot detection is proposed. Proposed technique exploits a novel taxonomy of web bot features known as Biostatistics Features. Numerous attack scenarios for web bot attacks such as automatic account registration, automatic form filling, bulk message posting, and web scrapping are created to imitate the zero-day web bot attacks. The proposed technique is evaluated with number of experiments using standard evaluation parameters. The experimental result analysis demonstrates that the proposed technique is extremely efficient in differentiating human users from web bots.

Research on Web Robot Detection Technology for Concept Drift

Web robots have long been significant participants of the internet and the detection is deemed to be essential. Contemporary web robot detection techniques lack the ability to detect when the characteristics of web robots drastically change over time, namely a phenomenon called concept drift. Such change of web robot’s behavior may lead to the deterioration of detection model performance. In order to maintain high detection performance over time, in this paper, we propose a novel web robot detection framework that consists of a model pool, a reinforcement learning algorithm integrating all the models and a concept drift detection module. First, we employ reinforcement learning to integrate a number of detection models that targets different types of web robots, by dynamically adjusting model weights. Then we apply the drift detection method (DDM) to monitor concept drift and identify the need to retrain the model over time. Experiments are conducted using real website datasets under concept drift. The results demonstrate that our model significantly outperforms the state-of-the-art approaches.

Detection of Advanced Web Bots by Combining Web Logs with Mouse Behavioural Biometrics

Web bots vary in sophistication based on their purpose, ranging from simple automated scripts to advanced web bots that have a browser fingerprint, support the main browser functionalities, and exhibit a humanlike behaviour. Advanced web bots are especially appealing to malicious web bot creators, due to their browserlike fingerprint and humanlike behaviour that reduce their detectability. This work proposes a web bot detection framework that comprises two detection modules: (i) a detection module that utilises web logs, and (ii) a detection module that leverages mouse movements. The framework combines the results of each module in a novel way to capture the different temporal characteristics of the web logs and the mouse movements, as well as the spatial characteristics of the mouse movements. We assess its effectiveness on web bots of two levels of evasiveness: (a) moderate web bots that have a browser fingerprint and (b) advanced web bots that have a browser fingerprint and also exhibit a humanlike behaviour. We show that combining web logs with visitors’ mouse movements is more effective and robust toward detecting advanced web bots that try to evade detection, as opposed to using only one of those approaches.

Educational Assistance Bot

An Internet bot performs automated tasks over the Internet. This software application is also known as a web robot or robot or simply bot, Typically, bots runs simple and structurally repetitive tasks, at a much higher rate than would be possible for a human alone. This effort is to help the users by providing all the basic student’s academic details like subjects, faculty names, books, notes and routine according to their need using bot. Users can also get other information like fees structure, library inventory list, calendar etc. Additional information like top educational sites, most trending educational videos etc will be regularly updated by the admin. This is an all-in-one application which can also be used to fetch and update information which is purely related to COVID-19.

Visibility and Ethical Considerations of Pakistani Universities Researchers on Google Scholar

Maximizing visibility by using academic profiling sites is very crucial in the academic world to improve the readership of research papers published and constant evaluation of research quality. In this article, the authors focused on the visibility of Pakistani University scholars on Google Scholar (GS). An intelligent Web Bot (MAKGBOT) was developed to collect the scholarly data of all Pakistani scholars, whose data is publicly available on Google Scholar. The findings of this research show that 87% of Pakistani universities have a presence on Google Scholar. It analyzes the research performance of scholars based on the last five years’ data from 2016 to 2020. Furthermore, the analysis reports the level of scholarly activities of all provinces and autonomous areas of Pakistan. This paper concludes by discussing the ethical issue of misrepresentation of information on the public profile and its consequences on the rankings of legitimate scholars.

Discovering social bots on Twitter: a thematic review

The onset of online social networks (OSN) like Twitter became a predominant platform for social expression and public relations. Twitter had 330 million monthly active users by the year 2019. With the gain in popularity, the ratio of virulent and automated accounts has also increased. It is estimated that 48 million of its functioning accounts are bots. Precisely, Twitter bots or Sybil accounts are kinds of automated web robot software that regulate activities like the tweet, retweet, like or follow via Twitter API. These bots misguide and delude genuine users by spreading spurious content. Hence, uncovering malicious bots from authentic users is obligatory to ensure a safe environment on Twitter. In this paper, the nature of Twitter bots and features of bot detection are discussed with their efficiency. Various bot detection approaches are classified based on the attributes and techniques used. Strong and weak aspects of distinct features and techniques are discussed. Key challenges and future research directions in detecting social bots are also presented. Special reference has been emphasised to contemporary emerging trends.

Sliding Puzzle CAPTCHA Analysis

Today a number of everyday activities are done through the Internet. To perform such web services users must register in relation to websites or fill some form. In such websites, some hackers write malicious programs called bots that destroy website resources by creating fake registrations or form submissions. This false registration may adversely affect the performance of websites. Therefore, it is necessary to distinguish between actual human users and Web bots (or computer programs) via tests known as CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart). Most of the conventional CAPTCHA challenges have become an easy nut to crack for bots, hence calling a need for better CAPTCHA alternatives. This paper studies the infamous 'Sliding Puzzle' as an alternate CAPTCHA challenge and analysing the complexity to solve it via bot script.