Unknown Protocol Research Articles

From Unknown to Similar: Unknown Protocol Syntax Analysis for Network Flows in IoT

Internet of Things (IoT) is the development and extension of computer, Internet, and mobile communication network and other related technologies, and in the new era of development, it increasingly shows its important role. To play the role of the Internet of Things, it is especially important to strengthen the network communication information security system construction, which is an important foundation for the Internet of Things business relying on Internet technology. Therefore, the communication protocol between IoT devices is a point that cannot be ignored, especially in recent years; the emergence of a large number of botnet and malicious communication has seriously threatened the communication security between connected devices. Therefore, it is necessary to identify these unknown protocols by reverse analysis. Although the development of protocol analysis technology has been quite mature, it is impossible to identify and analyze the unknown protocols of pure bitstreams with zero a priori knowledge using existing protocol analysis tools. In this paper, we make improvements to the existing protocol analysis algorithm, summarize and learn from the experience and knowledge of our predecessors, improve the algorithm ideas based on the Apriori algorithm idea, and perform feature string finding under the idea of composite features of CFI (Combined Frequent Items) algorithm. The advantages of existing algorithm ideas are combined together to finally propose a more efficient OFS (Optimal Feature Strings) algorithm with better performance in the face of bitstream protocol feature extraction problems.

Nowhere to Hide: A Novel Private Protocol Identification Algorithm

In recent years, with the rapid development of mobile Internet and 5G technology, great changes have been brought to our lives, and human beings have stepped into the era of big data. These new features and techniques in 5G support many different types of mobile applications for users, which makes network security extremely challenging. Among them, more and more applications involve users’ private data, such as location information, financial information, and biological information. In order to prevent users’ privacy disclosure, most applications choose to use private protocols. However, such private protocols also provide a means for malware and malicious applications to steal users’ privacy and confidential data. From a more secure point of view, we need to provide a way for users to know how many private protocols are running on their mobile phones and distinguish which are authorized applications and which are not. Therefore, the analysis and identification of private protocols have become a hot topic in current research. How to extract the characteristics of network protocol effectively and identify the private protocol accurately becomes the most important part of this research. In this paper, we combine genetic algorithm and association rule algorithm and then propose a set of feature extraction algorithm and protocol recognition algorithm for unknown protocols. The experimental analysis based on the actual data shows that these methods can effectively solve the problems of feature extraction and recognition for unknown protocols and can greatly improve the accuracy of private protocol recognition.

Application of Convolutional Neural Networks in Radio Station Link Establishment Behaviors Recognition

For the problem that it is difficult to recognize the type of link establishment (LE) behaviors of radio station whose communication protocol is unknown, a method is proposed to solve the problem by using convolutional neural network (CNN) to recognize LE behaviors. Themethod processes physical layer signals directly and breaks through limitation of unknown protocol standard. Three classical CNN models are optimized through experiments so that they become more suitable for recognition of one-dimensional time series signals. Experimental results show CNN can effectively recognize the different link establishment behaviors with a large number of training samples. Moreover, DenseNet whose recognition accuracy can reach 96% when the signal-to-noise ratio (SNR) is 0dB has the best performance compared to GoogLeNet and ResNet.

Protocol format extraction based on an improved CFSM algorithm

As the information technology rapidly develops, many network applications appear and their communication protocols are unknown. Although many protocol keyword recognition based protocol reverse engineering methods have been proposed, most of the keyword recognition algorithms are time consuming. This paper firstly uses the traffic clustering method F-DBSCAN to cluster the unknown protocol traffic. Then an improved CFSM (Closed Frequent Sequence Mining) algorithm is used to mine closed frequent sequences from the messages and identify protocol keywords. Finally, CFGM (Closed Frequent Group Mining) algorithm is proposed to explore the parallel, sequential and hierarchical relations between the protocol keywords and obtain accurate protocol message formats. Experimental results show that the proposed protocol formats extraction method is better than Apriori algorithm and Sequence alignment algorithm in terms of time complexity and it can achieve high keyword recognition accuracy. Additionally, based on the relations between the keywords, the method can obtain accurate protocol formats. Compared with the protocol formats obtained from the existing methods, our protocol format can better grasp the overall structure of target protocols and the results perform better in the application of protocol reverse engineering such as fuzzing test.

A message keyword extraction approach by accurate identification of field boundaries

SummaryWith the recent exponential increase in internet speeds, the traditional network environment is evolving into a high‐capacity network environment. Network traffic usage is also increasing exponentially, as are new malicious behaviors and related applications. Most of these applications and malicious behaviors use unknown protocols for which the structure is inaccessible; hence, protocol reverse engineering is receiving increasing attention in the field of network management. Various approaches have been proposed, but they still suffer from misidentification of field boundaries. To understand message structures properly, it is important to identify accurately the boundaries of the fields constituting the protocol message; accurate keyword extraction based on this approach leads to the correct inference of message types, semantics, and state machine. In this study, we propose a message keyword extraction method using accurate identification of field boundaries from delimiter inference and statistical analysis. Through the identification of field boundaries, messages can be subdivided into fields. We evaluate the efficacy of the proposed method by applying it to several textual and binary protocols. The proposed method showed better results than did other previous studies for both textual and binary protocols.

Unknown Protocol Identification Based on Improved K-Means++ Algorithm

In recent years, the gradual popularization of mobile terminals and the vigorous development of the network have spawned the birth of a new Internet structure and promoted the growth of network traffic. Behind such a large network, effective supervision of network traffic is the cornerstone of network security protection. At present, many studies on the direction of network supervision focus on the analysis of unknown network protocol types. The protocol identification method combined with machine learning is a hot topic in this kind of research. This method extracts data stream features and builds data sets, using machine learning algorithms. The model analyzes unknown network traffic and can obtain better recognition results than traditional network protocol analysis methods. Aiming at the problem of unknown traffic identification, this paper proposes a reasonable unknown traffic identification algorithm. The feature normalization preprocessing, feature selection, LOF outlier analysis, etc. are introduced. The clustering process uses the K-Means++ algorithm, and the maximum local reachable density point in the outlier analysis is used to realize the initial cluster center point. Accurate positioning.

Classification and identification of unknown network protocols based on CNN and T-SNE

With the continuous development of users’ demands and network technology, more and more new network protocols emerge, which poses great challenges to network protocol classification and identification. An artificial intelligence method was used to explore autonomous classification and identification of unknown network protocols in this paper in order to reduce the time and labor cost of network protocol classification and identification. In this paper, firstly, the network traffic was converted into grayscale images, and through transfer learning, the Convolutional Neural Networks (CNN) pre-trained model was used to extract the protocol features, so as to reduce the time and the amount of labeled data needed for the artificial neural network training. Finally, with the improved unsupervised hybrid clustering algorithm based on T-SNE and K-means, the types and number of protocols were autonomously identified and the network traffic was classified simultaneously. In this way, we can identify unknown protocols without prior knowledge and the protocol identification adaptability for big data was also greatly improved. Experimental results show this method has high accuracy and robustness in identifying unknown network protocols.

Clustering of unknown protocol messages based on format comparison

As a solution to detect and analyse unknown or proprietary protocols, Protocol Reverse Engineering(PRE) has been developed swiftly in recent years. In this field, message clustering aimed at protocol format serves as a fundamental solution for differentiating of unknown protocol messages. This paper works on the problem of format-oriented message clustering of unknown protocols, including messages from proprietary or non-cooperative network environments with their specifications unknown. By introducing basic rules of ABNF, we define Token Format Distance (TFD) and Message Format Distance (MFD) to represent format similarity of tokens and messages, and introduce Jaccard Distance and an optimized sequence alignment algorithm (MFD measurement) to compute them. Then, a distance matrix is built by MFD and we feed it to DBSCAN algorithm to cluster unknown protocol messages into classes with different formats. In this process, we design an unsupervised clustering strategy with Silhouette Coefficient and Dunn Index applied to parameter selecting of DBSCAN. In experiment on two datasets, the harmonic average v-measures of homogeneity and completeness on result clusters are both above 0.91, with fmis and coverages no less than 0.97. Together with iqr of v-measure and fmi bellow 0.1 and 0.03 separately in boxplot analyses, this method is proved to have remarkable validity and stability. Comprehensive analyses and comparisons on these indexes also show considerable advantages of our method over previous work.

An unknown Protocol improved k-means clustering algorithm based on Pearson distance

An unknown Protocol improved k-means clustering algorithm based on Pearson distance

An unknown protocol syntax analysis method based on convolutional neural network

AbstractIn recent years, a large number of botnets and dark networks rely on command and control channels of unknown protocol formats for communication, and with the development of Internet of Things technology, this problem becomes more prominent. The syntax analysis of the unknown protocol is helpful to measure the boundary of Botnet in the environment of Internet of things, so as to protect the network security. Based on the analysis of the characteristics of the current bitstream protocol data format, this article proposes an unknown protocol syntax analysis method based on convolutional neural network (CNN). First, the protocol data are preprocessed, and then the image is transformed. Next, the converted image is input to the convolution layer for convolution. After convolution, the data are flattened. Then the flattened data are put into the fully connected neural network. Finally, the unknown protocol is analyzed and predicted. The experimental results show that compared with the traditional feature extraction combine frequent item algorithm (CFI) and other neural network deep neural networks, CNN is 15% more accurate than CFI in the analysis of unknown protocol syntax, and it can accurately analyze and identify the unknown protocol.

Deep neural network-based automatic unknown protocol classification system using histogram feature

The protocol reverse engineering technique can be used to extract the specification of an unknown protocol. However, there is no standardized method, and in most cases, the extracting process is executed manually or semiautomatically. Since only frequently seen values are extracted as fields from the messages of a protocol, it is difficult to understand the complete specification of the protocol. Therefore, if the information about the structure of an unknown protocol could be acquired in advance, it would be easy to conduct reverse engineering. As such, one of the most important techniques for classifying unknown protocols is a feature extraction algorithm. In this paper, we propose a new feature extraction algorithm based on average histogram for classification of an unknown protocol and design unknown protocol classifier using deep belief networks, one of deep learning algorithms. In order to verify the performance of the proposed system, we performed the training using eight open protocols to evaluate the performance using unknown data. Experimental results show that the proposed technique gives significantly more reliable results of about 99% classification performance, regardless of the strength of the modification of the protocol.

Blind identification of network protocols based on improved Apriori algorithm

With the development and application of network technology and private protocols, more and more unknown protocols are found in communication networks. Analysis and identification of unknown protocols has become an important and difficult problem in obtaining valuable information from data. Based on the traditional Apriori algorithm, a reduced bit string algorithm and a weight compression algorithm are proposed to identify the unknown protocols according to the characteristics of network unknown protocol data in this paper. These proposed algorithms make use of location information, matrix compression, bit string and weight compression to gradually improve the identification accuracy and the processing efficiency of unknown protocols. Experiments on simulation data and actual data demonstrate the performance of the improved algorithms in terms of identification accuracy and time efficiency.

Overview of clustering analysis algorithms in unknown protocol recognition

In the process of identifying unknown protocol of bit stream, the clustering of data sets of bit stream in the protocol is the basis of further identifying unknown protocol. Therefore, on the one hand, this paper analyses the classical clustering algorithms used in unknown protocol recognition from three perspectives: the whole process of clustering analysis, similarity measurement and clustering result evaluation. On the other hand, the development trend of clustering algorithm in unknown protocol recognition is summarized, and other problems in unknown protocol recognition can be solved by clustering algorithm according to the characteristics of bit stream data set, which can provide reference for future research work. Finally, the challenges faced by the Research Institute and the prospects for future work are given.

Unknown Protocol Data Frame Classification Algorithm based on Improved K-Means

Unknown Protocol Data Frame Classification Algorithm based on Improved K-Means

Unsupervised field segmentation of unknown protocol messages

In network security systems working on intrusion detection, deep packet inspection, and protocol fuzzing, protocol specifications analyzed by Protocol Reverse Engineering(PRE) play an important role as fundamental input. For binary protocols having fixed-length fields, the location of those field boundaries has great impact on the subsequent analysis as well as the final performance. In this paper, we discuss the field segmentation problem formally, and develop a reasonable method ProSeg by introducing and optimize statistics(self-information and mutual information) from Information Theory. By analyzing the format structure of messages from unknown protocol vertically, the boundaries of fixed-length fields could be located by an expert voting strategy successfully. In experiments and analysis on several common protocols, our method turns out to be effective relatively and the results of ProSeg are consistent with standard segmentations to a great extent.

Statistical network protocol identification with unknown pattern extraction

Network traffic classification is an enabling technique for network security and management for both traditional networks and emerging networks such as Internet of Things. Due to the decreasing effectiveness of traditional port-based and payload-based methods, lots of research attentions are devoted to an alternative approach based on flow and packet-level traffic characteristics. A variety of statistical classification schemes are proposed in this context, but most of them embody an implicit assumption that all protocols are known in advance and well presented in the training data. This assumption is unrealistic because real-world networks constantly witness emerging traffic patterns and protocols that are previously unknown. In this paper, we revisit the problem by proposing a learning scheme with unknown pattern extraction for statistical protocol identification. The scheme is designed with a more realistic setting, in which we assume that the training data only consists of labeled samples from a limited number of protocols, and the goal is to identify these known patterns out of arbitrary traffic mixture of both known and unknown protocols. Our experiments based on real-world traffic show that the proposed scheme outperforms previous approaches by accurately identifying both known and unknown protocols.

Design and synthesis of [1,2,4]-triazolo isoquinoline derivatives via 1, 3-dipolar [3 + 2] cycloaddition: Reaction of azomethine imine with ethyl cyanoformate as unknown protocol

A series of novel 1, 2, 4-triazolo isoquinoline derivatives (8a-8p) were synthesized starting from 2-phenylethan-1-ol derivative (1) in a 7 step synthetic sequence. The key step in the scheme involves 1, 3-dipolar [3 + 2] cycloaddition of azomethine imine and ethyl cyanoformate as unknown reaction protocol. We have demonstrated the hydrolysis of both ester and benzoyl group in a single step and the corresponding carboxylic acid derivatives were coupled with various amines to generate unknown 1, 2, 4-triazolo isoquinoline derivatives The structures of the compounds (8a-8p) were characterized by 1H NMR, 13C NMR and HRMS analysis.

Protocol Specification Extraction Based on Contiguous Sequential Pattern Algorithm

As the amount of Internet traffic increases due to newly emerging applications and their malicious behaviors, the amount of traffic that must be analyzed is rapidly increasing. Many protocols that occur under these situations are unknown and undocumented. For efficient network management and security, a deep understanding of these protocols is required. Although many protocols reverse engineering methods have been introduced in the literature, there is still no single standardized method to completely extract a protocol specification, and each of the existing methods has some limitations. In this paper, we propose a novel protocol reverse engineering method to extract an intuitive and clear protocol specification. The proposed method extracts field formats, message formats, and flow formats as protocol syntax by using a contiguous sequential pattern algorithm three times hierarchically and defining four types of the field formats. Moreover, the proposed methods can extracts protocol semantics and a protocol finite state machine. The proposed method sufficiently compresses input messages into a small number of message formats in order to easily identify the intuitive structure of an unknown protocol. We implemented our method in a prototype system and evaluated the method to infer message formats of HTTP (a text protocol) and DNS (a binary protocol). The experimental results show that the proposed method infers HTTP with 100% correctness and 99% coverage. For DNS, the proposed method achieves 100% correctness and coverage.

Dynamic Combined with Static Analysis for Mining Network Protocol's Hidden Behavior

Unknown protocol's hidden behavior is becoming a new challenge in network security. This paper takes the captured messages and the binary code that implement the protocol both as the studied object. Dynamic Taint Analysis combined with Static Analysis is used for protocol analyzing. Firstly, monitor and analyze the process of protocol program parses the message in the virtual platform HiddenDisc prototype system developed by the authors, record the protocol's public behavior, then based on the authors' proposed Hidden Behavior Perception and Mining algorithm, static analyze the protocol's hidden behavior trigger conditions and hidden behavior instruction sequences. According to the hidden behavior trigger conditions, new protocol messages with the sensitive information are generated, and the hidden behaviors are executed by dynamic triggering. HiddenDisc prototype system can sense, trigger and analyze the protocol's hidden behaviors. According to the statistical analysis results, the authors propose the evaluation method of Protocol Execution Security. The experimental results show that the present method can accurately mining the protocol's hidden behaviors, and can evaluate unknown protocol's execution security.

‘The Sole Anti-Democratic Federation in the Entire Olympic Movement’: Early International Association of Athletics Federations Development Initiatives Between Commercialization and Democratization, 1974–1987

Studies on the development of Olympic Solidarity as a tool of the International Olympic Committee (IOC) to enable National Olympic Committees of developing countries access to resources and influence in the Olympic Movement exist. However, historical scholarship has relatively neglected the development of aid programs by International Federations to explain how they made use of resources to gain influence in international sport politics. Based on extensive multi-national and multi-lingual archival research in the archives of the International Association of Athletics Federation (IAAF) and the German Sport University Cologne, this article explores the establishment and development of the IAAF‘s Technical Aid Program, which had been installed in 1974. Referencing a large amount of previously unknown protocols and written correspondence, the paper critically discusses the IAAF‘s development activities in light of two parallel occurring processes that shaped the federation’s character in the 1970s and 1980s decisively: its increasing commercialization and its path towards democratization in its voting system. It is argued that the IAAF development programs served as a tool to enforce commercial and sport political interests whilst the nature of support remained without clear guidance until the mid-1980s.