AbstractThe Session Initiation Protocol (SIP) is a text‐based protocol, which defines the messaging between the SIP entities to establish, maintain, and terminate a multimedia session. Because of the text‐ and transaction‐based nature of the SIP protocol, it encounters various types of malformed message and resource depletion attacks. In this paper, we study the security concerns of the SIP‐based systems, and propose a feature set for it. Engineered features are derived from the SIP header fields in real time detecting the deviation of the input traffic from normal state. These features are built at three levels: packet, transaction, and dialog. The designed features can accurately detect the SIP known attacks. Moreover, because we successfully model the state machine of SIP during its normal behavior, we can also identify the unknown attacks. To study the effectiveness of the engineered feature set, we employ them in a sample one‐class support vector machine classifier. We evaluate the engineered features on three different datasets with various types of attack scenarios including resource depletion and authentication and brute force attacks. The impact of these attack scenarios on the designed features are shown in different test cases to demonstrate the effectiveness of our proposed feature set. Copyright © 2014 John Wiley & Sons, Ltd.