GENDA: A Graph Embedded Network Based Detection Approach on encryption algorithm of binary program

Abstract

The cryptographic techniques are commonly used in software protection against malicious re-engineering. How to efficiently detect encryption algorithms used in the software to determine if they meet protection requirements is an interesting and significant task. However, existing encryption algorithm detection methods suffer from a high alarm rate or low efficiency as they fail to extract the complete program structure and semantic features of the encryption algorithms. In this article, we proposed GENDA, a graph embedding network-based detection method on encrypted binary code. We first analyze the characteristics of various encryption algorithms and construct the program graph for each encryption algorithm. Then the program graph is recursively embedded into the graph neural network as a basic unit, and the vector representation of the encryption algorithm graph is obtained. Finally, the type of encryption algorithm is determined by comparing the distance between these vectors. To evaluate GENDA, we collected a number of cryptographic libraries and real application programs from the open-source software. The experimental results show that GENDA can reach over a detection success rate of 92%. We also compared GENDA to existing state-of-the-art detection methods. The comparison results show that GENDA outperforms most of the existing methods.