Large Amount Of Log Data Research Articles

An Intelligent Framework for Log Anomaly Detection Based on Log Template Extraction

Log anomaly detection holds great significance in computer systems and network security. A large amount of log data is generated in the background of various information systems and equipment, so automated methods are required to identify abnormal behavior that may indicate security threats or system malfunctions. The traditional anomaly detection methods usually rely on manual statistical discovery, or match by regular expression which are complex and time-consuming. To prevent system failures, minimize troubleshooting time, and reduce service interruptions, a log template-based anomaly detection method has been proposed in this context. This approach leverages log template extraction, log clustering, and classification technology to timely detect abnormal events within the information system. The effectiveness of this method has been thoroughly tested and compared against traditional log anomaly detection systems. The results demonstrate improvements in log analysis depth, event recognition accuracy, and overall efficiency.

A study of the performance of general compressors on log files

Large-scale software systems and cloud services continue to produce a large amount of log data. Such log data is usually preserved for a long time (e.g., for auditing purposes). General compressors, like the LZ77 compressor used in gzip, are usually used in practice to compress log data to reduce the cost of long-term storage. However, such general compressors do not consider the unique nature of log data. In this paper, we study the performance of general compressors on compressing log data relative to their performance on compressing natural language data. We used 12 widely used general compressors to compress nine log files that are collected based on surveying prior literature on text compression, log compression and log analysis. We observe that log data is more repetitive than natural language data, and that log data can be compressed and decompressed faster with higher compression ratios. Besides, the compressor with the highest compression ratio for natural language data is rarely the one for log data. Nevertheless, the compressors with the highest compression ratio for log data are rarely adopted in practice by current logging libraries and log management tools. We also observe that the peak compression and decompression speeds of general compressors on log data is often achieved with a small data size, while such size may not be used by log management tools. Finally, we observe that the optimal compression performance (measured by a combined compression performance score) of log data usually requires the compression level to be configured higher than the default level. Our findings call for careful consideration of choosing general compressors and their associated compression levels for log data in practice. In addition, our findings shed lights on the opportunities for future research on compressors that better suit the characteristics of log data.

Exploring the Relationships between Reading Behavior Patterns and Learning Outcomes Based on Log Data from E-Books: A Human Factor Approach

ABSTRACTOnline learning environments presently accumulate large amounts of log data. Analysis of learning behaviors from these log data is expected to benefit instructors and learners. This study was intended to identify effective measures from e-book materials used at Kyushu University and to employ these measures for analyzing learning behavioral patterns. In an evaluation, students were grouped into four clusters using k-means clustering, and their learning behavioral patterns were analyzed. We examined whether the learning behavioral patterns exhibited relations with the learning outcomes. The results reveal that the learning behavior of “backtrack” style reading exerts a significant positive influence on learning effectiveness, which can aid students to learn more efficiently.

Analyzing the power consumption behavior of a large scale data center

The aim of this paper is to illustrate the use of application and system level logs to better understand scientific data center behavior and energy-spending. Analyzing a data center log of 900 nodes (Sandy Bridge and Haswell), we study node power consumption and describe approaches to estimate and forecast it. Our results include methods to cluster nodes based on different vmstat and RAPL measurements as well as Gaussian and GAM models for estimating the plug power consumption. We also analyze failed jobs and find that non-successfully terminated jobs consume around 40% of computing time. While the actual numbers are likely to vary in different data centers at different times, the purpose of the paper is to share ideas of what can be found by statistical and machine learning analysis of large amount of log data.

Optimized Prefetching Scheme to Support VCR-Like Operations in P2P VoD Applications with Guided Seeks

In Peer-to-Peer (P2P) Video-on-Demand (VoD) streaming systems, supporting free VCR operations is challenging. Prefetching is a good way to improve user experience of VCR interactivity. But most existing P2P VoD prefetching schemes are proposed aimed at those popular videos with large amount of log data, without considering the situation that the videos are unpopular or the popular videos are in their initial phase of release. In this situation, these schemes cannot support user VCR interactivity very well. To address this issue, we propose a new optimized prefetching scheme, called Hybrid Anchor Scheme (HAS), in which fixed anchors and dynamic anchors are merged together. The dynamic anchors are generated based on association rule and segments popularity. Through combining the way of sequential prefetching according to weight of segments and the way of several rounds prefetching, we implement HAS effectively. Extensive simulations validate the proposed prefetching scheme provide shorter seeking latency compared to other prefetching schemes.

A New Algorithm for Web Log Mining

The enormous content of information on the World Wide Web makes it obvious candidate for data mining research. Data Mining Technique application is used to the World Wide Web referred as Web mining where this term has been used in three distinct ways; , Web Structure Mining, Web Content Mining and Web Usage Mining. Web Log Mining is one of the Web based application where it will facing with large amount of log data. In order to produce the web log through portal usage patterns and user behaviors, this research work implements the high level process of Web Log mining technique using basic rules. Web Log Mining consists of three main phases, namely Data Preprocessing, Pattern filtering and Pattern Analysis. As we know that server log files become a set of raw data where it’s must go through with all the Web Log Mining phases to producing the final results. Here, Web Log mining, approach has been combining with the basic rules, to optimize the total execution time. Finally, this work wills present an overview of results analysis and Web Log Mining can use the findings for the suitable valuable actions.

Discovering sequential concept patterns for behavioural diagnosis by interpreting web‐page contents

As web‐based courses become more popular, web systems can accumulate large amounts of log data. Because this data is generated by learners’ behaviour while studying web‐based courses, many researchers agree that analysing web logs could benefit learners, instructors and web‐site managers. This paper describes a study that used data‐mining techniques in order to analyse learners’ online behaviour for the purpose of extracting concept patterns—by transforming general web logs to a content perspective. An important advantage of this approach is that it can be used to identify concept patterns based on learning behaviour—rather than the more conventional browsing patterns based on browsing behaviour.

Mobile Digital Assistants for Community Support

We applied mobile computing to community support and explored mobile computing with a large number of terminals. This article reports on the Second International Conference on Multiagent Systems (ICMAS'96) Mobile Assistant Project that was conducted at an actual international conference for multiagent systems using 100 personal digital assistants (PDAs) and cellular telephones. We supported three types of service: (1) communication services such as e-mail and net news; (2) information services such as conference, personal, and tourist information; and (3) community support services such as forum and meeting arrangements. After the conference, we analyzed a large amount of log data and obtained the following results: It appears that people continuously used PDAs in their hotel rooms after dinner; e-mail services were used independently of the conference structure, but the load on information services reflected the schedule of the conference. Postquestionnaire data showed that our trial was considered interesting, although people were not fully satisfied with the PDAs and services provided. Participants showed a deep interest in mobile computing for community support.

Computerized Log Analysis for Efficient Evaluation of Gas Wells and Gas Storage Reservoirs

SCHMIDT, A.W., SCHLUMBERGER-DOLL RESEARCH CENTER, RIDGEFIELD, CONN. TINCH, D.H., SCHLUMBERGER-DOLL RESEARCH CENTER, RIDGEFIELD, CONN.MEMBERS AIME CARPENTER, B.N., SCHLUMBERGER-DOLL RESEARCH CENTER, RIDGEFIELD, CONN. HOYLE, W.R., SCHLUMBERGER-DOLL RESEARCH CENTER, RIDGEFIELD, CONN. Abstract Open-hole log interpretation methods are employed to provide effective porosity data that are then used with cased-hole neutron logs to compute gas saturations in storage reservoirs. A method of using density and neutron logs to solve the problem of porosity determination in shallowly invaded gas reservoirs is developed. A system of digital magnetic tape and a computer was used to process the large amounts of log data. Line printer frequency plots and customized "Computed Logs" are graphical techniques developed to display the computer output in analyzable form. The computed logs were used in constructing cross-sections as well as porosity and hydrocarbon fraction maps. The computer was used to integrate these maps into total reservoir and gas storage volumes. Introduction In the evaluation of gas-producing or gas-storage reservoirs a primary necessity is the accurate determination of porosity. When the lithology is known and the formation is fully liquid saturated, porosity usually may be determined with a single log-density, neutron, or sonic. However, porosity determination from logs is frequently complicated by unknown lithological characteristics. The presence of gas further complicates accurate porosity determination. These difficulties are overcome by combining several porosity-sensitive devices; for example, the interpretation of porosity in the presence of gas is done with the combination of open-hole density and neutron logs. Such multiple-log interpretations can be handled by interpretation charts and cross-plot techniques. If made over many levels, the work becomes long and tedious. For gas storage, logs are used to monitor gas in place. This is done by combining open-hole porosity data with cased-hole neutron logs.' The neutron logs are run at periodic intervals or after periods of large gas injection or withdrawal. Evaluation of these monitoring logs on a fieldwide basis has tended to be qualitative rather than quantitative because of the manpower required for manually processing the large amount of data. The labor of these techniques and the necessity for handling large amounts of log data invited the application of digital computer techniques to these methods. This requires that the data be in computer-compatible digitized form, preferably on magnetic tape. Work on large-scale projects has been greatly expedited by the introduction of equipment for recording logging signals on digital magnetic tape at the well site. This equipment, introduced experimentally in the Illinois basin in 1965, is now in general use in the U.S. Along with appropriate computer programs, graphic techniques were developed to display as logs the computed results from these tapes.' In computed log form it is easy to scan quickly the zones within a well and to compare or correlate between wells. In this paper we will discuss in particular the application of computer processing to:The use of open-hole logs for determining porosity in gas-bearing zones. Attention also will be given to determining the amount of gas originally present in new wells.The evaluation of neutron logs through casing in order to monitor changes in amounts of gas in place.The application of these results from individual wells to field-wide studies. A few comments will be made on the data processing equipment and techniques. Reservoir Evaluation from Open-Hole Logs Two log combinations are used to illustrate computerized techniques in gas reservoir evaluation.For clean formations: density and neutron logs for porosity, plus an induction log for water saturation. This combination is used to handle the case of shallow invasion in which gas is present within the range of investigation of density and neutron logs.For shaly sands: sonic, density, and gamma ray logs. This combination yields values of effective porosity and shaliness. Shallow Invasion of Gas-Bearing Formations It is worthwhile at this point to review how the interpretation of the different logs may be complicated by the presence of gas close to the borehole. JPT P. 959ˆ