Machine Learning for NetFlow Anomaly Detection With Human-Readable Annotations

Abstract

We propose a framework for anomaly detection in communication network logs along with automated extraction of human-readable annotations that explain the decision logic underlying each anomaly detection. For this purpose, we develop a machine learning methodology formulated in terms of a model comprised of an OR-combination of multiple Boolean logic based sentences. Each sentence is an empirically learned set of inequality conditions involving subsets of features. The feature set, which comprises the “alphabet” for human-readable annotations, is constructed using dynamic graph based spatio-temporal aggregation to extract human-understandable aggregates of network activity. These aggregates are constructed both in terms of computers (nodes in dynamic graph) and communications between computers (edges in dynamic graph). From the alphabet, the learned model identifies subsets of features that relate to each anomaly type and the combinations of conditions in terms of the feature subsets for detection of the specific anomaly type. Given a data point that the learned model detects as anomalous, the model identifies the specific features and their combinations related to the anomaly detection. These human-readable annotations provide a cyber-security analyst a transparent view into the decision logic underlying an anomaly detection.