Access Log Files Research Articles

An innovative data collection method to eliminate the preprocessing phase in web usage mining

The underlying data source for web usage mining (WUM) is commonly thought to be server logs. However, access log files ensure quite limited data about the clients. Identifying sessions from this messy data takes considerable effort, and operations performed for this purpose do not always yield excellent results. Also, this data cannot be used for web analytics efficiently. This study proposes a method for user tracking, session management, and collecting web usage data. The method is mainly based on an innovative approach for using collected data for web analytics extraction as the data source in web usage mining. An application-based API has been developed with a different strategy from conventional client-side methods to obtain and process log data. The log data has been successfully gathered by integrating the technique into an enterprise web application. The results reveal that the homogeneous data collected and stored with this method is more convenient to browse, filter, and process than web server logs. This structured data can be used effortlessly as a reliable data source for high-performance web usage mining activity, real-time web analytics, machine learning algorithms, or a functional recommendation system.

Data Lake Architecture for Storing and Transforming Web Server Access Log Files

Web server access log files are text files containing important data about server activities, client requests addressed to a server, server responses, etc. Large-scale analysis of these data can contribute to various improvements in different areas of interest. The main problem lies in storing these files in their raw form, over long time, to allow analysis processes to be run at any time enabling information to be extracted as foundation for high quality decisions. Our research focuses on offering an economical, secure, and high-performance solution for the storage of large amount of raw log files. Proposed system implements a Data Lake (DL) architecture in cloud using Azure Data Lake Storage Gen2 (ADLS Gen2) for extract–load–transform (ELT) pipelines. This architecture allows large volumes of data to be stored in their raw form. Afterwards they can be subjected to transformation and advanced analysis processes without the need of a structured writing scheme. The main contribution of this paper is to provide a solution that is affordable and more accessible to perform web server access log data ingestion, storage and transformation over the newest technology, Data Lake. As derivative contribution, we proposed the use of Azure Blob Trigger Function to implement the algorithm of transforming log files into parquet files leading to 90% reduction in storage space compared to their original size. That means much lower storage costs than if they had been stored as log files. A hierarchical data storage model has also been proposed for shared access to data over different layers in the DL architecture, on top of which Data Lifecycle Management (DLM) rules have been proposed for storage cost efficiency. We proposed ingesting log files into a Data Lake deployed in cloud due to ease of deployment and low storage costs. The aim is to maintain this data in the long term, to be used in future advanced analytics processes by cross-referencing with other organizational or external data. That could bring important benefits. While the proposed solution is explicitly based on ADLS Gen2, it represents an important benchmark in approaching a cloud DL solution offered by any other vendor.

Web Personalized Recommendation Model Using Temporal Fuzzy Association Rule Mining

Abstract: A web access log file contains timely sequenced log entries which include essential fields to indicate user activities. Analysis of these patterns provides valuable information for web designer to quickly respond to their individual needs. Many industries are struggling to retain regular interested customers for the improvement of customer relationship. Retrieval of relevant information automatically from these log files for interested group of users is a difficult process, since acquiring interested user profiles which evolves continuously with respect to time are not so easy. The paper presents a novel Web Personalized Recommendation Model (WPRM) using temporal fuzzy association rule mining technique. Temporal Fuzzy Association Rule Mining (FTARM) technique is proposed and applied on a focused set of interested users to provide intelligent recommendations. The proposed model results in less execution time and reduced memory utilization with high accuracy.

Trends in user-initiated health information exchange in the inpatient, outpatient, and emergency settings

Prior research on health information exchange (HIE) typically measured provider usage through surveys or they summarized the availability of HIE services in a healthcare organization. Few studies utilized user log files. Using HIE access log files, we measured HIE use in real-world clinical settings over a 7-year period (2011-2017). Use of HIE increased in inpatient, outpatient, and emergency department (ED) settings. Further, while extant literature has generally viewed the ED as the most relevant setting for HIE, the greatest change in HIE use was observed in the inpatient setting, followed by the ED setting and then the outpatient setting. Our findings suggest that in addition to federal incentives, the implementation of features that address barriers to access (eg, Single Sign On), as well as value-added services (eg, interoperability with external data sources), may be related to the growth in user-initiated HIE.

Data Pre-Processing on Web Server Access Logs of University for User Interaction Patterns

In the current digital Era, websites are developed and organized into multifaceted in nature. It is essential to distinguish user sessions/intent and browsing behavior from logs in order to recommend appropriate content for the web designers and administrators. This paper focuses on data preprocessing of the weblogs received from Kannada University Hampi, Vidyaranya Karnataka state are cleaned viably by applying various pre-processing methodologies. The work identifies the superior quality of data to discover user interactions, user sessions, the specific web pages, and the regularly visited Uri’s, most visited pages, most time spent on pages and incorrect webpages served to users. These pre-processed webserver access log files will be utilized to discover patterns, fine grained analysis and study. This paper also focuses on challenges of log file analysis.

RETRACTED ARTICLE: Normal discriminant deep convolution neural classification based web behavioral pattern mining for user identification

Mining the user behavior patterns from huge log files plays a major role in discovering the web user identity. Several data mining techniques describe the way of predicting the user identity using web access behavior patterns. Accurate prediction of the user identity is still a challenging issue. The proposed work introduces a normal discriminant Tanimoto similarity based deep convolution feedforward neural learning classification (NDTS-DCFNLC) technique to improve the user identifications with their web access behavior patterns. The NDTS-DCFNLC technique comprises multiple layers. Normal discriminant preprocessing is carried out in layer one to remove the unwanted patterns from the web access log files. Similarity among the relevant web pattern is calculated using Tanimoto similarity at layer two. Sigmoid activation function is used in output layer to classify the frequently accessed patterns with higher accuracy and minimum error based on similarity threshold. Experimental testing is done using apache weblog with different number of patterns and the diverse constraints such as classification accuracy, false positive rate, space requirements and execution time. Results were compared with linear temporal logic and KNN classification methods. The outcome shows NDTS-DCFNLC technique increases classification accuracy, decreases the false positives, execution time and the space requirements.

Using blockchain to log genome dataset access: efficient storage and query

BackgroundGenomic variants are considered sensitive information, revealing potentially private facts about individuals. Therefore, it is important to control access to such data. A key aspect of controlled access is secure storage and efficient query of access logs, for potential misuse. However, there are challenges to securing logs, such as designing against the consequences of “single points of failure”. A potential approach to circumvent these challenges is blockchain technology, which is currently popular in cryptocurrency due to its properties of security, immutability, and decentralization. One of the tasks of the iDASH (Integrating Data for Analysis, Anonymization, and Sharing) Secure Genome Analysis Competition in 2018 was to develop time- and space-efficient blockchain-based ledgering solutions to log and query user activity accessing genomic datasets across multiple sites, using MultiChain.MethodsMultiChain is a specific blockchain platform that offers “data streams” embedded in the chain for rapid and secure data storage. We devised a storage protocol taking advantage of the keys in the MultiChain data streams and created a data frame from the chain allowing efficient query. Our solution to the iDASH competition was selected as the winner at a workshop held in San Diego, CA in October 2018. Although our solution worked well in the challenge, it has the drawback that it requires downloading all the data from the chain and keeping it locally in memory for fast query. To address this, we provide an alternate “bigmem” solution that uses indices rather than local storage for rapid queries.ResultsWe profiled the performance of both of our solutions using logs with 100,000 to 600,000 entries, both for querying the chain and inserting data into it. The challenge solution requires 12 seconds time and 120 Mb of memory for querying from 100,000 entries. The memory requirement increases linearly and reaches 470 MB for a chain with 600,000 entries. Although our alternate bigmem solution is slower and requires more memory (408 seconds and 250 MB, respectively, for 100,000 entries), the memory requirement increases at a slower rate and reaches only 360 MB for 600,000 entries.ConclusionOverall, we demonstrate that genomic access log files can be stored and queried efficiently with blockchain. Beyond this, our protocol potentially could be applied to other types of health data such as electronic health records.

Adaptive and Optimization of Personalized Information Retrieval Model in Semantic Web

The recognition of user’s visited set of web pages for the prediction of web page is a key drawback. Thus the work is employed with the web access log files which is stored in the server. To understand the user interest patterns, the web access log files are extracted that depicts the user behavior. Various applications can be employed to predicting user’s behavior while serving the web. During this work, the proposed framework analyze the user usage, reinforced the content and the content retrieved with the semantic manner. The semantic information retrieval supported the user access pages are preprocessed and the web log data of the particular user is analyzed to identify the user profile. Then the retrieved information is graded with clustering the semantic content based results. The ranked content is then analyzed with the user profile to produce an optimized search results for the users based on the user classification.

A Study of Educational Data Mining: Evidence from a Thai University

Educational data mining provides a way to predict student academic performance. A psychometric factor like time management is one of the major issues affecting Thai students’ academic performance. Current data sources used to predict students’ performance are limited to the manual collection of data or data from a single unit of study which cannot be generalised to indicate overall academic performance. This study uses an additional data source from a university log file to predict academic performance. It investigates the browsing categories and the Internet access activities of students with respect to their time management during their studies. A single source of data is insufficient to identify those students who are at-risk of failing in their academic studies. Furthermore, there is a paucity of recent empirical studies in this area to provide insights into the relationship between students’ academic performance and their Internet access activities. To contribute to this area of research, we employed two datasets such as web-browsing categories and Internet access activity types to select the best outcomes, and compared different weights in the time and frequency domains. We found that the random forest technique provides the best outcome in these datasets to identify those students who are at-risk of failure. We also found that data from their Internet access activities reveals more accurate outcomes than data from browsing categories alone. The combination of two datasets reveals a better picture of students’ Internet usage and thus identifies students who are academically at-risk of failure. Further work involves collecting more Internet access log file data, analysing it over a longer period and relating the period of data collection with events during the academic year.

Log files Analysis Using MapReduce to Improve Security

Log files are a very useful source of information to diagnose system security and to detect problems that occur in the system, and are often very large and can have complex structure. In this paper, we provide a methodology of security analysis that aims to apply Big Data techniques, such as MapReduce, over several system log files in order to locate and extract data probably related to attacks made by malicious users whose intends to compromise a system. These data will lead, through a process of learning, to identify, predict attacks or detect intrusions. We have clarified this approach with a concrete case study on exploiting access log files of web apache servers to predict and detect SQLI and DDOS attacks. The obtained results are promising, we are able to extract malicious indicators and events that characterize the intrusions, which help us to make an accurate diagnosis of the security and supervise state of the system, and subsequently in the learning process.

A Proposed Methodology on Predicting Visitor’s Behavior based on Web Mining Technique

The evolution of the internet in recent decades enlarge the website's reports with the records of user’s activities and behaviors that registered in the web server which can be created automatically in the web access log file. The feedback concerning the user’s activities, performance and any problem that may be occur including the cyber security approaches of the web server represents the principal raison of applying the web mining technique. In this paper, we proposed a methodology on predicting users behavior based on the web mining technique by creating and executing analysis applications using a Deep Log Analyzer tool that applied on the web server access log of our faculty website. Furthermore, an associated programmed application has been developed which employs the extracted data into dynamic visualizations reports(tables, graphs, charts) in order to help the web system administrator to increase the web site effectiveness, we had creating a suitable access patterns that permits to identify the interacting users behaviors and the interesting usage patterns such as the occurred errors, potential visitors, navigation activities, behavioral analysis, diagnostic study, and security alerts for intrusion prevention. Moreover, the obtained results achieved the aim of producing a dynamic monitoring by extracting investigation summaries which analyses the discovered access patterns that registered in the faculty web server in order to improve the web site usability by tracking the user’s behaviors and the browsing activities. Our proposed tool will highlight providing a security alerts against the malicious users by predicting the malicious behaviors taking into consideration all the discovered vulnerabilities by detecting the corrupted links used by the abnormal visitors.

A soft computing approach for benign and malicious web robot detection

The accurate detection of web robot sessions from a web server log is essential to take accurate traffic-level measurements and to protect the performance and privacy of information on a Web server. Moreover, the irrecoverable risks of visits from malicious robots that intentionally try to evade web server intrusion detection systems, covering-up their visits with fabricated fields in their http request packets, cannot be ignored. To separate both types of robots from humans in practice, analysts turn to heuristic methods or state-of-the-art soft computing approaches that have only been tuned to the specification of a kind of web server. Noting that the landscape of web robot agents is ever changing, and that behavioral patterns and characteristics vary across different web servers, both options are lacking. To overcome this challenge, this paper presents SMART, a soft computing system that simultaneously detects benign and malicious types of robot agents from web server logs and can automatically adapt to the session characteristics of a web server. The results of experiments over some access log file servers, each servicing different domains of the web, demonstrate outperformance of the proposed method on state-of-the-art ones for benign and malicious robot detection.

Discovering User Pattern Analysis from Web Log Data using Weblog Expert

Objective: This article tries to discover the hidden knowledge and identifying user behavior on the web by using the web data sources. With the help of this knowledge, the overall performance of future accesses, the typical browsing behavior of a user and subsequently to predict desired pages, a user wants to access in future. Methods/Statistical Analysis: The user pattern is analyzed by using the modified Web Log Expert tool from the web access log file collected from the organization. This modified tool tries to conduct a web mining in a domain independent manner. This algorithm consists of three parts: 1. Given an input entity, extracting a set of IP addresses and visor lists and then ranking them according to comparability, 2. Extracting the domains in which the given entity takes part and 3. Identifying and summarizing the competitive evidence that details the organization’s strength. Findings: The main aim of the research work is extracting of user frequent access page using web log data, which is based on user session time, IP addresses, browser details, operating system, top user. This complete analysis work has been implemented in the Web Log Expert tool. The experimental results provide an easier way to navigate the website and improve the website design architecture. This work deliberates the detailed results of a website in a specific education domain application. We investigate the statistics of hourly based, daily based, week and monthly based report of the web usage patterns. The goal is to capture, model and analyze the behavioral patterns and profiles of users interacting with a website. The knowledge about users and their behavior on the web helps the organization benefits and leads directly to profit increase.

Efficient Extraction for Mobile Web Access Log with Caching Strategy

Mobile web access log file plays an important role in the analysis about demand of mobile terminal market or user behavior. However, the log file data is highly dimensional, disorganized and semi-structured, which heightens the difficulty of data extracting accuracy; while it generates and transmits continuously, which poses an extracting efficiency challenge. It is highly desirable to extract the information embedded in log files as disorder or hidden situation efficiently and accurately. This paper proposes an efficient extracting method for mobile web access log with cache strategy. Firstly of all, data dictionary sets are built for each kind of complex field before extracting. Then, data is extracted based on the dictionaries, and the dictionaries will be completed simultaneously. Furthermore, with the discovery of the distribution of some data in the log generally following Zipf-like distribution, cache strategy is considered to be an auxiliary way to reduce mapping time. In addition, the classical cache strategy LFU is chosen. Ultimately, the experiment shows that the data could be extracted from the log accurately, and the extracting efficiency speeds up remarkably with LFU cache strategy.

POUŽITÍ GENETICKÝCH ALGORITMŮ PRO ANALÝZU NÁVŠTĚVNOSTI WWW PORTÁLU

The article proposes a new method suitable for advanced analysis of web portal visits. This is part of retrieving information and knowledge from web usage data (web usage mining). Such information is necessary in order to gain better insight into visitor’s needs and generally consumer behaviour. By le­ve­ra­ging this information a company can optimize the organization of its internet presentations and offer a better end-user experience. The proposed approach is using Grammatical evolution which is computational method based on genetic algorithms. Grammatical evolution is using a context-free grammar in order to generate the solution in arbitrary reusable form. This allows us to describe visitors’ behaviour in different manners depending on desired further processing. In this article we use description with a procedural programming language. Web server access log files are used as source data.The extraction of behaviour patterns can currently be solved using statistical analysis – specifically sequential analysis based methods. Our objective is to develop an alternative algorithm.The article further describes the basic algorithms of two-level grammatical evolution; this involves basic Grammatical Evolution and Differential Evolution, which forms the second phase of the computation. Grammatical evolution is used to generate the basic structure of the solution – in form of a part of application code. Differential evolution is used to find optimal parameters for this solution – the specific pages visited by a random visitor. The grammar used to conduct experiments is described along with explanations of the links to the actual implementation of the algorithm. Furthermore the fitness function is described and reasons which yield to its’ current shape. Finally the process of analyzing and filtering the raw input data is described as it is vital part in obtaining reasonable results.

A Web Page Recommendation system using GA based biclustering of web usage data

The World Wide Web store, share, and distribute information in the large scale. There is large number of internet users on the web. They are facing many problems like information overload due to the significant and rapid growth in the amount of information and the number of users. As a result, how to provide web users with more exactly needed information is becoming a critical issue in web applications. Web mining extracts interesting pattern or knowledge from web data. It is classified into three types as web content mining, web structure, and web usage mining. Web usage mining is the process of extracting useful knowledge from the server logs. This useful knowledge can be applied to target marketing and in the design of web portals. It may give information that is useful for improving the services offered by web portals and information access and retrieval tools. In this paper we are introducing a new approach for web page recommendation and user profile generation. This approach makes use of evolutionary biclustering technique for web page recommendation. We have applied it on two different datasets. One is clickstream data and other is web access log file of KSV University. The final results are generated from optimal biclusters obtained from evolutionary biclustering.

Optimized Agglomeration Algorithmic rule to Uncover User Pattern Applying on Weblog Knowledge

These days, event logs contain immense amounts of knowledge that may simply overwhelm a personality's. Consequently, mining content from the phenomenon of logs is accepted as a crucial function for the system management. This paper presents a completely unique clump rule for log file information sets that helps one to sight frequent patterns from log files, to create log file profiles, and to spot abnormal log file lines. To attain this, clump tool is employed for playacting 2 phases. First, a military operation is finished on internet group action information. Second, AN analysis is finished on internet access log files to research the user group action behavior and user’s log-in approach. During this paper, the cluster medical specialty and verification tool is employed to find hidden relationships among the net server information and access patterns.

Differential Impact of Web Technology on Various Dimensions of Students’ Academic and NonAcademic Activities (a Case Study)

Internet and computing have been advocated as one shot solution to many existing constraints and difficulties of education system. Our work aims to study exhaustively differential impact of Web technology on various dimensions of students' academic performance, co-curricular, extra-curricular and non-academic activities. We analyzed the Internet usage behavior of students in terms of average time spent in Internet per day and category of visited Websites by them along with the effects of these Internet usage behaviors on their academic performance (CPI) and other co-curricular and extra-curricular activities. For this analysis we used proxy server access log files of an engineering college in India which are collected during 30 months continually. We have classified student activities (with extending ODP classification scheme) under four categories: Curricular, Co-Curricular, Extra-Curricular and Non-Curricular. Only 59% of female students used Internet continually as against 81% of male students. This analysis led us to conclude that a significant gender gap exists in terms of Internet usage.

Differential Impact of Web Technology on Various Dimensions of Students' Academic and Non-academic Activities (a Case Study)

Internet and computing have been advocated as one shot solution to many existing constraints and difficulties of education system. Our work aims to study exhaustively differential impact of Web technology on various dimensions of students' academic performance, co-curricular, extra-curricular and non-academic activities. We analyzed the Internet usage behavior of students in terms of average time spent in Internet per day and category of visited Websites by them along with the effects of these Internet usage behaviors on their academic performance (CPI) and other co-curricular and extra-curricular activities. For this analysis we used proxy server access log files of an engineering college in India which are collected during 30 months continually. We have classified student activities (with extending ODP classification scheme) under four categories: Curricular, Co-Curricular, Extra-Curricular and Non-Curricular. Only 59% of female students used Internet continually as against 81% of male students. This analysis led us to conclude that a significant gender gap exists in terms of Internet usage.

Perceived Internet Usage Behaviours as Predictor to Outlier Detection in Students' Communities in Academic Environments

It is important to provide perspectives about the effects of Internet usage on students' personal and social behaviours along with the impacts of these usages on their academic performances. To explore students' Internet usage behaviors and predicting outliers in student's community, we have developed Web based data mining tool named Education Data Miner (EDMiner), which provides user friendly interface for different stockholders of the system including professors and deans. This research study was conducted with a sample of 5210 students from one engineering college in India during 36 months continually. The primary focus of this study is to extract Internet usage pattern of students by exploring proxy server access log files. These patterns were then used for identifying outliers in students' community. We have applied centroid and density based clustering methods to identify outliers. Further, the relationship between Internet usage behaviours and various Academic and Non-academic activities were explored. Based on our results the majority of visited Websites, 35 percent, belongs to Websites under Extra-Curricular category whereas for curricular Websites it is 24 percent. Further, our results also contradict the perception that the Internet usage adversary affects the academic performance. Moreover, our analysis results show higher average time spent on Internet did result into nonparticipation in other activities, which are very essential for the growth of these students. This nonparticipation in other activities may prove to be an indicator for loneliness of these individuals.