Development of anomalous computer behavior detection method based on probabilistic automaton

Abstract

The paper proposes a method for identifying the anomalous behavior of a computer system based on probabilistic automaton. The main components of the method are the model of generation of the structure of the automaton and its modification procedure. The defining feature of the method is adaptation of automaton structure generation procedure for detecting scenarios of the same type, by restructuring the structure of the automaton upon a match and by recalculation of the state transition probabilities. Input data of the automaton consist of discrete events (system calls, process IDs or sections of code instructions), typical for a certain class of anomalous behavior, and grouped by type. The automaton structure is first created in accordance with one of the instances of a class, and then restructured during the analysis of other instances. Possibility of state transition depends on the input state and transition probability value. Generated automaton structure is used to detect anomalous computer system behavior. Automaton structure can be updated, if an anomaly occurs with different scenarios. Proposed method allows to speed up detecting anomalous computer behavior, as well as to detect computer system anomalies, scenario profiles of which only partially match with instances used for generation the structure of the automaton. Obtained research results allow us to conclude about the possibility of using this method in heuristic analyzers of anomaly detection systems.