Decentralized Network Anomaly Detection via a Riemannian Cluster Approach

Abstract

Network anomaly detection in large scale sensor networks is a fundamental task in many Internet of Things (IoT) applications. Given an incomplete set of corrupted observations of the network matrices, the problem of network matrix recovery and anomaly detection can be formulated as a low-rank matrix completion problem with a fraction of the observed entries being corrupted by outliers. Although many centralized algorithms have been proposed to solve the low-rank matrix completion problem, they generally require the observations to be centrally available, which can incur many problems in practical applications, e.g., power budget constraints, single point failure and privacy concern. Recently, a decentralized nuclear-norm minimization-based algorithm was developed to solve the low-rank matrix completion problem in a mesh network. In this paper, we consider a two-tier network and propose a new decentralized approach based on the Riemannian optimization. Our proposed clustering and consensus sharing method achieves a balance between the performance guarantee and the computational cost: the proposed method distributes the computational burden over the agent nodes, while exhibits a recovery performance close to its centralized counterpart. In addition, our Riemannian optimization-based approach scales well with the size of the problem, hence it is more favoured for handling large data sets in IoT than the nuclear-norm minimization-based algorithm. Numerical simulations are performed to demonstrate the effectiveness of the proposed approach, which is able to solve problems of size 2000×2000 of rank 5 with 50 agent nodes in 10 seconds.