Protocol Message Format Research Articles

Specification-Based Symbolic Execution for Stateful Network Protocol Implementations in IoT

IoT devices offer insufficient protections against exploitation of critical programming errors (such as buffer overflows) it is therefore paramount to sufficiently test IoT software before deployment. A central source of these errors are implementations of stateful network protocols used in the IoT (e.g. MQTT-SN). Unfortunately, comprehensive automated testing of such protocol implementations is challenging due to their large state space. Furthermore, prior work on automated testing of protocol implementations focuses on conventional desktop systems and does thus not address challenges specific to the IoT domain (e.g. handling of peripheral interactions). In this publication, we leverage symbolic execution (an automated software testing technique) for testing stateful network protocol implementation in the IoT domain. To mitigate issues regarding state space explosion, we propose a novel symbolic execution approach which is based on created protocol state and message format specifications and specifically tailored to the IoT domain. We employ our specification-based symbolic execution approach to test protocol implementations provided by two popular IoT operating systems. Our experiments indicate that a significant increase in code coverage can be achieved using our automated software testing approach. We were further able to discover three critical and previously unknown bugs in network protocol implementations provided by the RIOT operating system.

R-gram: Inferring message formats of service protocols with relative positional n-grams

Automatically discovering message formats of unknown service or system protocols from network traces has become important for a variety of applications, such as emulating the behavior of an unknown protocol in service virtualization, or enabling deep packet inspection in network security. Among existing schemes, the keyword extraction based approaches have been shown to be effective. Inspired by the template structure of protocol messages, recent works leverage the positions of keywords to extract message keywords more accurately. However, these methods are deficient for messages with large variations in length. To address this problem, we propose R-gram, which exploits the relative positions of keywords in messages, allowing the keywords to be robustly detected in variable length messages. It first extracts the common template of the messages in a given message trace with a fast sampling technique, and segments each message into blocks according to the relative positions of the common keywords in the template. It then identifies message keywords in each block by using a new concept and technique — relative positional n-gram (r-gram in short). Finally, the message keywords are used to separate all the messages into type-specific clusters and consequently derive the message format for each cluster. We have implemented and evaluated R-gram on real-world service traces containing either textual or binary protocol messages. Our experimental results show that R-gram is more accurate and robust than existing state-of-the-art tools in protocol message format extraction. Furthermore, R-gram is efficient for processing large-scale message traces.

Protocol format extraction based on an improved CFSM algorithm

As the information technology rapidly develops, many network applications appear and their communication protocols are unknown. Although many protocol keyword recognition based protocol reverse engineering methods have been proposed, most of the keyword recognition algorithms are time consuming. This paper firstly uses the traffic clustering method F-DBSCAN to cluster the unknown protocol traffic. Then an improved CFSM (Closed Frequent Sequence Mining) algorithm is used to mine closed frequent sequences from the messages and identify protocol keywords. Finally, CFGM (Closed Frequent Group Mining) algorithm is proposed to explore the parallel, sequential and hierarchical relations between the protocol keywords and obtain accurate protocol message formats. Experimental results show that the proposed protocol formats extraction method is better than Apriori algorithm and Sequence alignment algorithm in terms of time complexity and it can achieve high keyword recognition accuracy. Additionally, based on the relations between the keywords, the method can obtain accurate protocol formats. Compared with the protocol formats obtained from the existing methods, our protocol format can better grasp the overall structure of target protocols and the results perform better in the application of protocol reverse engineering such as fuzzing test.

GrAMeFFSI: Graph Analysis Based Message Format and Field Semantics Inference For Binary Protocols, Using Recorded Network Traffic

Protocol specifications describe the interaction between different entities by defining message formats and message processing rules. Having access to such protocol specifications is highly desirable for many tasks, including the analysis of botnets, building honeypots, defining network intrusion detection rules, and fuzz testing protocol implementations. Unfortunately, many protocols of interest are proprietary, and their specifications are not publicly available. Protocol reverse engineering is an approach to reconstruct the specifications of such closed protocols. Protocol reverse engineering can be tedious work if done manually, so prior research focused on automating the reverse engineering process as much as possible. Some approaches rely on access to the protocol implementation, but in many cases, the protocol implementation itself is not available or its license does not permit its use for reverse engineering purposes. Hence, in this paper, we focus on reverse engineering protocol specifications relying solely on recorded network traffic. More specifically, we propose GrAMeFFSI, a method based on graph analysis that can infer protocol message formats as well as certain field semantics for binary protocols from network traces. We demonstrate the usability of our approach by running it on packet captures of two known protocols, Modbus and MQTT, then comparing the inferred specifications to the official specifications of these protocols.

IPv6-Based Architecture of Community Medical Internet of Things

The effective integration of Community Medical Internet of Things (CMIoT) and the core technology IPv6 about the next generation of Internet will create a new pattern for the medical field. However, as a heterogeneous network system, the interconnection of different CMIoT components is the primary problem, which needs to be solved. Traditional protocol conversion gateway in the Internet of Things only implements the packet format conversion. When the network environment changes, it is difficult to effectively implement the data path. We propose a CMIoT architecture with the assist of the communication auxiliary gateway, and design the simplified protocol message format under the premise of satisfying the functional requirements. At the same time, the CMIoT architecture model is built based on the Ptolemy II modeling environment and implemented in a community, and it is proved that the interconnection between the IPv6 network and the physical network can be realized more effectively.

Bit‐oriented format extraction approach for automatic binary protocol reverse engineering

Protocol message format extraction is a principal process of automatic network protocol reverse engineering when target protocol specifications are not available. However, binary protocol reverse engineering has been a new challenge in recent years for approaches that traditionally have dealt with text-based protocols rather than binary protocols. In this study, the authors propose a novel approach called PRE-Bin that automatically extracts binary-type fields of binary protocols based on fine-grained bits. First, a silhouette coefficient is introduced into the hierarchical clustering to confirm the optimal clustering number of binary frames. Second, a modified multiple sequence alignment algorithm, in which the matching process and back-tracing rules are redesigned, is also proposed to analyse binary field features. Finally, a Bayes decision model is invoked to describe field features and determine bit-oriented field boundaries. The maximum a posteriori criterion is leveraged to complete an optimal protocol format estimation of binary field boundaries. The authors implemented a prototype system of PRE-Bin to infer the specification of binary protocols from actual traffic traces. Experimental results indicate that PRE-Bin effectively extracts binary fields and outperforms the existing algorithms.

A Semantics-Aware Approach to the Automated Network Protocol Identification

Traffic classification, a mapping of traffic to network applications, is important for a variety of networking and security issues, such as network measurement, network monitoring, as well as the detection of malware activities. In this paper, we propose Securitas, a network trace-based protocol identification system, which exploits the semantic information in protocol message formats. Securitas requires no prior knowledge of protocol specifications. Deeming a protocol as a language between two processes, our approach is based upon the new insight that the n-grams of protocol traces, just like those of natural languages, exhibit highly skewed frequency-rank distribution that can be leveraged in the context of protocol identification. In Securitas, we first extract the statistical protocol message formats by clustering n-grams with the same semantics, and then use the corresponding statistical formats to classify raw network traces. Our tool involves the following key features: 1) applicable to both connection oriented protocols and connection less protocols; 2) suitable for both text and binary protocols; 3) no need to assemble IP packets into TCP or UDP flows; and 4) effective for both long-live flows and short-live flows. We implement Securitas and conduct extensive evaluations on real-world network traces containing both textual and binary protocols. Our experimental results on BitTorrent, CIFS/SMB, DNS, FTP, PPLIVE, SIP, and SMTP traces show that Securitas has the ability to accurately identify the network traces of the target application protocol with an average recall of about 97.4% and an average precision of about 98.4%. Our experimental results prove Securitas is a robust system, and meanwhile displaying a competitive performance in practice.

Analyzing Network Protocols of Application Layer Using Hidden Semi-Markov Model

With the rapid development of Internet, especially the mobile Internet, the new applications or network attacks emerge in a high rate in recent years. More and more traffic becomes unknown due to the lack of protocol specifications about the newly emerging applications. Automatic protocol reverse engineering is a promising solution for understanding this unknown traffic and recovering its protocol specification. One challenge of protocol reverse engineering is to determine the length of protocol keywords and message fields. Existing algorithms are designed to select the longest substrings as protocol keywords, which is an empirical way to decide the length of protocol keywords. In this paper, we propose a novel approach to determine the optimal length of protocol keywords and recover message formats of Internet protocols by maximizing the likelihood probability of message segmentation and keyword selection. A hidden semi-Markov model is presented to model the protocol message format. An affinity propagation mechanism based clustering technique is introduced to determine the message type. The proposed method is applied to identify network traffic and compare the results with existing algorithm.

CoAP Protocol for Constrained Networks

This paper discusses the Constrained Application Protocol (CoAP), an application layer protocol introduced by Internet Engineering Task Force (IETF) Constrained RESTful environment (CoRE) Working Group for use with low power and resource constrained nodes in the internet of things. The paper discusses the CoAP architecture supported with suitable examples. CoAP protocol message format, types, methods and security feature is also discussed. In this paper, we present a CoAP based car parking system implemented using Contiki, an open source operating system for resource constrained device. The Cooper (Cu) CoAP user agent for Firefox browser is used for handling CoAP URI scheme.

Instruction Clustering Analysis for Unknown Network Protocol's Abnormal Behavior

Protocol's abnormal behavior analysis is an important task in protocol reverse analysis. Traditional protocol reverse analysis focus on the protocol message format, but protocol behavior especially the abnormal behavior is rare studied. In this paper, protocol behavior is represented by the labeled behavior instruction sequences. Similar behavior instruction sequences mean the similar protocol behavior. Using our developed virtual analysis platform HiddenDisc, we can capture a variety of known or unknown protocols' behavior instruction sequences. All kinds of executed or unexecuted instruction sequences can automatic clustering by our designed instruction clustering algorithm. Thereby we can distinguish and mine the unknown protocols' potential abnormal behavior. The mined potential abnormal behavior instruction sequences are executed, monitored and analyzed on HiddenDisc to determine whether it is an abnormal behavior and what is the behavior's nature. Using the instruction clustering algorithm, we have analyzed 1297 protocol samples, mined 193 potential abnormal instruction sequences, and determined 187 malicious abnormal behaviors by regression testing. Experimental results show that our proposed instruction clustering algorithm has high efficiency and accuracy, can mine unknown protocols' abnormal behaviors effectively, and enhance the initiative defense capability of network security.

Position-based automatic reverse engineering of network protocols

Automatic protocol reverse engineering is a process of extracting protocol message formats and protocol state machine without access to the specification of target protocol. Protocol reverse engineering is useful for addressing many problems of network management and security, such as network management, honey-pot systems, intrusion detection, Botnet detection and prevention, and so on. Currently, protocol reverse engineering is mainly a manual and painstaking process which is time-consuming and error-prone. In this paper, we present a novel approach for automatic reverse engineering application-layer network protocols. We extract protocol keywords from network traces based on their support rates and variances of positions, reconstruct message formats, and infer protocol state machines. We implement our approach in a prototype system called AutoReEngine and evaluate it over four text-based protocols (HTTP, POP3, SMTP and FTP) and two binary protocols (DNS and NetBIOS). The results show that our AutoReEngine outperforms the existing algorithms.