Federated Anomaly Detection on System Logs for the Internet of Things: A Customizable and Communication-Efficient Approach

Abstract

Runtime log-based anomaly detection is one of several key building blocks in ensuring system security, as well as post-incident forensic investigations. However, existing log-based anomaly detection approaches that are implemented on large-scale Internet of Things (IoT) systems generally upload local data from edge devices to a centralized (cloud) server for processing and analysis. Such a workflow incurs significant communication and computation overheads, with potential privacy implications. Hence, in this paper, we propose a customizable and communication-efficient federated anomaly detection scheme (hereafter referred to as FedLog), designed to facilitate the identification of abnormal log patterns in large-scale IoT systems. Specifically, we first craft a Temporal Convolutional Network-Attention Mechanism-based Convolutional Neural Network (TCN-ACNN) model, to effectively extract fine-grained features from system logs. Second, we develop a new federated learning framework to support IoT devices in establishing a comprehensive anomaly detection model in a collaborative and privacy-preserving manner. Third, a lottery ticket hypothesis based masking strategy is designed to achieve customizable and communication-efficient federated learning in handling non-Independent and Identically Distributed (non-IID) log datasets. We then evaluate the performance of our proposed scheme with those of DeepLog (published in CCS, 2017) and Loganomaly (published in IJCAI, 2019) in both centralized learning and federated learning settings, using two publicly available and widely used real-world datasets (i.e., HDFS and BGL). The findings demonstrate the utility of the proposed FedLog scheme, in terms of log-based anomaly detection.