Entropy-based robust PCA for communication network anomaly detection

Abstract

Principal component analysis (PCA) has received increasing attention as a method to distinguish network traffic anomalies from normal data instances based on its orthogonal linear transformation characteristics and dimensionality reduction technique. To address the issue of parameter sensitivity in the classical PCA, we propose modifications to the classical PCA, called robust PCA in this paper, which exhibits greater flexibility in detecting outliers for different traffic distributions. First, the robust PCA utilizes the Mahalanobis distance function which generates more flexible results than that of the Euclidean distance used in the classical PCA. The second modification to the classical PCA is to take into account the temporal effect of network traffic data by considering the neighbors' corresponding values. Temporal correlation is a practically important feature for network traffic, which the classical PCA does not consider. In addition, the proposed robust PCA also adopts entropy calculation to cope with both numerical and categorical data, as both data types exist in real traffic traces. Finally, using the robust PCA, our experimental results demonstrate the effectiveness in identifying network anomalies.