Log Analysis Tool Research Articles

A framework for detection and mitigation of cyber criminal activities using university networks in Kazakhstan

Abstract. The increasing number of information security incidents in higher education underscores the urgent need for robust cybersecurity measures. This paper proposes a comprehensive framework designed to analyze the illegal use of internet resources in university networks in Kazakhstan. The subject of this article is the detection and mitigation of cybercriminal activities using university networks in Kazakhstan. The goal is to develop a comprehensive framework that integrates multiple educational organizations to enhance collaborative security efforts by monitoring network activity and categorizing texts using machine learning techniques. The tasks to be solved are: to formalize the procedure of integrating multiple educational organizations into a collaborative cybersecurity framework; developing a log analysis tool tailored for monitoring network activities within university networks; creating a novel dictionary of extremist terms in the Kazakh language for text categorization; to implement advanced machine learning models for network traffic classification. The methods used are: log analysis tools for real-time monitoring and anomaly detection in network activities, Natural language processing (NLP) techniques to develop a specialized dictionary of extremist terms in Kazakh, Machine learning models to classify network traffic and detect potential cyber threats, and collaborative architecture design to integrate network security efforts across multiple institutions. The following results were obtained: a comprehensive log analysis tool was developed and implemented, providing real-time monitoring of network activities in university networks; a dictionary of extremist terms in Kazakh was created, facilitating the categorization and analysis of texts related to potential security threats; advanced machine learning models were successfully applied to classify network traffic, enhancing the detection and mitigation of cyber threats; and an experimental architecture integrating multiple educational organizations was established, fostering collaborative efforts in cybersecurity. Conclusions. The scientific novelty of the results obtained is as follows: 1) a robust framework for collaborative cybersecurity in educational institutions was developed, leveraging log analysis and machine learning techniques; 2) the creation of a specialized dictionary of extremist terms in Kazakh significantly improved the accuracy of text categorization related to cybersecurity; 3) the application of advanced machine learning models to network traffic classification provided a methodological approach to effectively managing and securing network infrastructure effectively; 4) the experimental architecture demonstrated the potential for enhanced security through collaboration among educational organizations, offering strategic recommendations for improving information security in academic environments. The outcomes of this research contribute to the broader cybersecurity field by providing a structured approach to detecting and mitigating cyber threats in educational contexts. The proposed framework has potential applications extending to global security frameworks, aiming to foster a safer internet usage environment and reduce the risks associated with cyber threats and unauthorized data access.

Forensic readiness of industrial control systems under stealthy attacks

Cyberattacks against Industrial Control Systems (ICS) can have harmful physical impacts. Investigating such attacks can be difficult, as evidence could be lost to physical damage. This is especially true with stealthy attacks; i.e., attacks that can evade detection. In this paper, we aim to engineer Forensic Readiness (FR) in safety-critical, geographically distributed ICS, by proactively collecting potential evidence of stealthy attacks. The collection of all data generated by an ICS at all times is infeasible due to the large volume of such data. Hence, our approach only triggers data collection when there is the possibility for a potential stealthy attack to cause damage. We determine the conditions for such an event by performing predictive, model-based, safety checks. Furthermore, we use the geographical layout of the ICS and the safety predictions to identify data that is at risk of being lost due to damage, i.e., relevant data. Finally, to reduce the control performance overhead resulting from real-time data collection, we select a subset of relevant data to collect by performing a trade-off between expected impact of the attack and the estimated cost of collection. We demonstrate these ideas using simulations of the widely-used Tennessee–Eastman Process (TEP) benchmark. We show that the proposed approach does not miss relevant data and results in a reduced control performance overhead compared to the case when all data generated by the ICS is collected. We also showcase the applicability of our approach in improving the efficiency of existing ICS forensic log analysis tools.

Scalable Architectural Pattern for Integrating Syslog Servers with Splunk

An enterprise infrastructure consists of several devices. The devices emit event notifications representing their current state. The devices without storage such as printers and routers are configured to send the event notifications in the form of syslogs to one or more remote syslog servers over the network. Depending on the size and usage of the enterprise infrastructure, millions of syslogs may be emitted per second. These syslogs are used by the system administrators to detect and address the anomalies in the infrastructure. The system administrators often integrate the syslog servers with Log Analysis tools that offer aggregation, analytics, and visualisation capabilities. Splunk is one such popular tool that can be integrated with syslog servers. This paper proposes an architectural pattern for syslog servers that are to be integrated with Splunk for better performance, scalability and resilience.

Petrophysical evaluation of well log data for reservoir characterization in Titas gas field, Bangladesh: a case study

The petrophysical evaluation of the Titas gas field (Titas-15) has been carried out for its reservoir characterization and to quantify the reservoir hydrocarbon prospects and gas productivity. The Quick Log Analysis Tool (QLAT) of PE2 Essentials software has been utilized for the computer-processed interpretation inputting the various well log data such as gamma-ray (GR), density log (RHOB), density-porosity (DPHI), neutron porosity (NDHI), resistivity logs, etc. Among the three reservoir sand groups, three gas-bearing sub-zones have been identified in the upper sand group, which are primarily producing gas from the least net-pay of 9.5 m, 39.5 m and 28.5 m, respectively. The shale volume is calculated based on GR, caliper logs and neutron-density (N-D) cross-plot (minimum shale volume) methods whose average value is comprised between 26.3 and 33.9%. The range of effective porosity is 18.4–18.6% indicating a good reservoir quality; permeability is 14.11–25.95 mD; recognizing as the well-sorted nature of sands. Additionally, the hydrocarbon saturation in the respective zones varies from 48% to 60%, suggesting a good hydrocarbon reservoir, whereas the water saturation is 40–52%. Moreover, the range of measured ‘gas initially in place’ in upper, middle and lower gas sands are 2371.6–2579.1, 10330.7–11206.1 and 6373.1–7110.5 MMscf, respectively, implying an adequate gas reserve on commercial criteria. From the evaluation of petrophysical properties, the Titas reservoir has a higher ability to enhance hydrocarbon accumulation and production.

Usage of "Techniment" Technical Sentiment Value to Determine Whichlogs to Send for Log Analysis Tools to Save Not Only Bandwidth But Also Compute

Usage of "Techniment" Technical Sentiment Value to Determine Whichlogs to Send for Log Analysis Tools to Save Not Only Bandwidth But Also Compute

Exploring Properties and Correlations of Fatal Events in a Large-Scale HPC System

In this paper, we explore potential correlations of fatal system events for one of the most powerful supercomputers—IBM Blue Gene/Q Mira, which is deployed at Argonne National Laboratory, based on its 5-year reliability, availability, and serviceability (RAS) log. Our contribution is two-fold. (1) We design an efficient log analysis tool, namely LogAider, with a novel filtering method to effectively extract fatal events from masses of system messages that are heavily duplicated in the log. LogAider exhibits a very precise detection of temporal-correlation with a high similarity (up to 95 percent) to the ground-truth (i.e., compared to the failure records reported by the administrators). The total number of fatal events can be reduced to about 1,255 compared with originally 2.6 million duplicated fatal messages. (2) We analyze the 5-year RAS log of the MIRA system using LogAider, and summarize six important “takeaways” which can help system vendors and administrators better understand an extreme-scale system's fatal events. Specifically, we find that the distribution or proportion of the fatal system events follow a Pareto-like principle in general. The temporal correlation among fatal events is much stronger than that of warn messages and info messages, and the correlated events tend to constitute a few clusters. The mean time between fatal events (MTBFE) of the Mira system is about 1.3 days from the perspective of the system, and the MTTI is 2–4 days from the perspective of users. The most error-prone item value with respect to any key attribute appears likely in the log every 2–10 days. Weibull, Gamma, and Pearson6 are the three best-fit distributions for the fatal event intervals. The overall correlation of fatal events on the 5D torus network is not prominent, whereas the small-region locality correlation (e.g., the fatal events inside racks) is relatively strong. We believe our work will be interesting to large-scale HPC system administrators and vendors and to fault tolerance researchers, enabling them to better understand fatal events and mitigate such events accordingly.

Design of Smart Platform-based Event-Log Analysis and Parameter-modifying Software for Naval Combat Systems

In this paper, a smart platform-based log analysis and parameter-modifying tool for naval combat systems (NCSs) is proposed. To support the development of NCSs, the log-analysis and parameter-modifying tool provides several other functions, such as node information gathering and remote log gathering. Applications called smartWatcher (for smart devices) and smartAgent (for nodes) communicate to gather node information and log files. The developer of the NCS can modify parameters to resolve errors in the NCS based on results of the log analysis. Simulation results show the availability and effectiveness of the smart platform for NCSs.

Operational-Log Analysis for Big Data Systems: Challenges and Solutions

Big data systems (BDSs) are complex, consisting of multiple interacting hardware and software components, such as distributed computing nodes, databases, and middleware. Any of these components can fail. Finding the failures' root causes is extremely laborious. Analysis of BDS-generated logs can speed up this process. The logs can also help improve testing processes, detect security breaches, customize operational profiles, and aid with any other tasks requiring runtime-data analysis. However, practical challenges hamper log analysis tools' adoption. The logs emitted by a BDS can be thought of as big data themselves. When working with large logs, practitioners face seven main issues: scarce storage, unscalable log analysis, inaccurate capture and replay of logs, inadequate log-processing tools, incorrect log classification, a variety of log formats, and inadequate privacy of sensitive data. Some practical solutions exist, but serious challenges remain. This article is part of a special issue on Software Engineering for Big Data Systems.

Complex log file synthesis for rapid sandbox-benchmarking of security- and computer network analysis tools

Today Information and Communications Technology (ICT) networks are a dominating component of our daily life. Centralized logging allows keeping track of events occurring in ICT networks. Therefore a central log store is essential for timely detection of problems such as service quality degradations, performance issues or especially security-relevant cyber attacks. There exist various software tools such as security information and event management (SIEM) systems, log analysis tools and anomaly detection systems, which exploit log data to achieve this. While there are many products on the market, based on different approaches, the identification of the most efficient solution for a specific infrastructure, and the optimal configuration is still an unsolved problem. Today׳s general test environments do not sufficiently account for the specific properties of individual infrastructure setups. Thus, tests in these environments are usually not representative. However, testing on the real running productive systems exposes the network infrastructure to dangerous or unstable situations. The solution to this dilemma is the design and implementation of a highly realistic test environment, i.e. sandbox solution, that follows a different – novel – approach. The idea is to generate realistic network event sequence (NES) data that reflects the actual system behavior and which is then used to challenge network analysis software tools with varying configurations safely and realistically offline. In this paper we define a model, based on log line clustering and Markov chain simulation to create this synthetic log data. The presented model requires only a small set of real network data as an input to understand the complex real system behavior. Based on the input׳s characteristics highly realistic customer specified NES data is generated. To prove the applicability of the concept developed in this work, we conclude the paper with an illustrative example of evaluation and test of an existing anomaly detection system by using generated NES data.

DigLA – A Digsby log analysis tool to identify forensic artifacts

Since the inception of Web 2.0, instant messaging, e-mailing, and social networking have emerged as cheap and efficient means of communication over the Web. As a result, a number of communication platforms like Digsby have been developed by various research groups to facilitate access to multiple e-mail, instant messaging, and social networking sites using a single credential. Although such platforms are advantageous for end-users, they present new challenges to digital forensic examiners because of their illegitimate use by anti-social elements. To identify digital artifacts from Digsby log data, an examiner is assumed to have knowledge of the whereabouts of Digsby traces before starting an investigation process. This paper proposes a design for a user-friendly GUI-based forensic tool, DigLA, which provides a unified platform for analyzing Digsby log data at different levels of granularity. DigLA is also equipped with password decryption methods for both machine-specific and portable installation versions of Digsby. By considering Windows registry and Digsby log files as dynamic sources of evidence, specifically when Digsby has been used to commit a cyber crime, this paper presents a systematic approach to analyzing Digsby log data. It also presents an approach to analyzing RAM and swap files to collect relevant traces, specifically the login credentials of Digsby and IM users. An expected insider attack from a server security perspective is also studied and discussed in this paper.

Web Analytics: A Picture of the Academic Library Web Site User

This article describes the usefulness of Web analytics for understanding the users of an academic library Web site. Using a case study, the analysis describes how Web analytics can answer questions about Web site user behavior, including when visitors come, the duration of the visit, how they get there, the technology they use, and the most popular content using an open-source Web log analysis tool. The author offers suggestions for future research into user motivations to complement the findings possible using Web analytics.

Getting to the Source: a Survey of Quantitative Data Sources Available to the Everyday Librarian: Part I: Web Server Log Analysis

This is the first part of a two-part article that provides a survey of data sources which are likely to be immediately available to the typical practitioner who wishes to engage in statistical analysis of collections and services within his or her own library. Part I outlines the data elements which can be extracted from web server logs, and discusses web log analysis tools. Part II looks at logs, reports, and data sources from proxy servers, resource vendors, link resolvers, federated search engines, institutional repositories, electronic reference services, and the integrated library system.

Getting to the Source: a Survey of Quantitative Data Sources Available to the Everyday Librarian: Part II: Data Sources from Specific Library Applications

This is the second part of a two-part article that provides a survey of data sources which are likely to be immediately available to the typical practitioner who wishes to engage in statistical analysis of collections and services within his or her own library. Part I outlines the data elements which can be extracted from web server logs, and discusses web log analysis tools. Part II looks at logs, reports, and data sources from proxy servers, resource vendors, link resolvers, federated search engines, institutional repositories, electronic reference services, and the integrated library system.

데이타베이스 튜닝을 위한 로그 분석 도구

Database logs contain various information on database operations, but they are used to recover database systems from failures generally. This paper proposes a log analysis tool that provides useful information for database tuning. This tool provides users with information on work-load organization, database schemas, and resources usages of queries. This paper describes the tool in views of its architecture, functions, implementation, and verification. The tool is verified by running the TPC-W benchmark, and representative analysis results are also presented.

Using interactive visualizations of WWW log data to characterize access patterns and inform site design

HTTP server log files provide Web site operators with substantial detail regarding the visitors to their sites. Interest in interpreting this data has spawned an active market for software packages that summarize and analyze this data, providing histograms, pie graphs, and other charts summarizing usage patterns. While useful, these summaries obscure useful information and restrict users to passive interpretation of static displays. Interactive visualizations can be used to provide users with greater abilities to interpret and explore web log data. By combining two-dimensional displays of thousands of individual access requests, color and size coding for additional attributes, and facilities for zooming and filtering, these visualizations provide capabilities for examining data that exceed those of traditional web log analysis tools. We introduce a series of interactive visualizations that can be used to explore server data across various dimensions. Sample visualizations of server data from two web sites are presented. Coordinated, snap-together visualizations (STVs) of log data are introduced as a means of gaining additional expressive power. Possible uses of these visualizations are discussed, and difficulties of data collection, presentation, and interpretation are explored.

Using interactive visualizations of WWW log data to characterize access patterns and inform site design

HTTP server log files provide Web site operators with substantial detail regarding the visitors to their sites. Interest in interpreting this data has spawned an active market for software packages that summarize and analyze this data, providing histograms, pie graphs, and other charts summarizing usage patterns. Although useful, these summaries obscure useful information and restrict users to passive interpretation of static displays. Interactive visualizations can be used to provide users with greater abilities to interpret and explore Web log data. By combining two-dimensional displays of thousands of individual access requests, color, and size coding for additional attributes, and facilities for zooming and filtering, these visualizations provide capabilities for examining data that exceed those of traditional Web log analysis tools. We introduce a series of interactive visualizations that can be used to explore server data across various dimensions. Possible uses of these visualizations are discussed, and difficulties of data collection, presentation, and interpretation are explored.