Abstract: Anomaly detection is an active research topic in network intrusion detection. This paper presents a novel method for detecting anomalous user behavior based on shell commands and hidden Markov models (HMM). The method constructs a specific HMM to represent the normal behavior profile of a network user, and associates classes of user behavior patterns with states of the HMM. The HMM parameters are calculated with a sequence matching algorithm which is much simpler than the classical Baum-Welch algorithm. This reduces computational complexity to a great extent. At the detection stage, a decision rule based on probabilities of short state sequences is adopted, and more than one threshold are used to classify the user behavior. Performance of the method is tested in computer simulation, showing high detection accuracy and efficiency.

Key words: intrusion detection, hidden Markov model, anomaly detection, shell command