ADAL-NN: Anomaly Detection and Localization Using Deep Relational Learning in Distributed Systems

Abstract

Modern distributed systems that operate concurrently generate interleaved logs. Identifiers (ID) are always associated with active instances or entities in order to track them in logs. Consequently, log messages with similar IDs can be categorized to aid in the localization and detection of anomalies. Current methods for achieving this are insufficient for overcoming the following obstacles: (1) Log processing is performed in a separate component apart from log mining. (2) In modern software systems, log format evolution is ongoing. It is hard to detect latent technical issues using simple monitoring techniques in a non-intrusive manner. Within the scope of this paper, we present a reliable and consistent method for the detection and localization of anomalies in interleaved unstructured logs in order to address the aforementioned drawbacks. This research examines Log Sequential Anomalies (LSA) for potential performance issues. In this study, IDs are used to group log messages, and ID relation graphs are constructed between distributed components. In addition to that, we offer a data-driven online log parser that does not require any parameters. By utilizing a novel log parser, the bundled log messages undergo a transformation process involving both semantic and temporal embedding. In order to identify instance–granularity anomalies, this study makes use of a heuristic searching technique and an attention-based Bi-LSTM model. The effectiveness, efficiency, and robustness of the paper are supported by the research that was performed on real-world datasets as well as on synthetic datasets. The neural network improves the F1 score by five percent, which is greater than other cutting-edge models.