Real-Time Cyber Analytics Data Collection Framework

Abstract

In cyber security, it is critical that event data is collected in as near real time as possible to enable early detection and response to threats. Performing analytics from event logs stored in databases slows down the response time due to the time cost of database insertion and retrieval operations. The authors present a data collection framework that minimizes the need for long-term storage. Events are buffered in memory, up to a configurable threshold, before being streamed in real time using live streaming technologies. The framework deploys virtualized data collecting agents that ingest data from multiple sources including threat intelligence. The framework enables the correlation of events from various sources, improving detection precision. The authors have tested the framework in a real time, machine-learning-based threat detection system. The results show a time gain of 300 milliseconds in transmission time from event capture to analytics system, compared with storage-based collection frameworks. Threat detection was measured at 95%, which is comparable to the benchmark snort IDS.