Binary Code Analysis Research Articles

Pitfalls in Machine Learning for Computer Security

With the growing processing power of computing systems and the increasing availability of massive datasets, machine learning algorithms have led to major breakthroughs in many different areas. This development has influenced computer security, spawning a series of work on learning-based security systems, such as for malware detection, vulnerability discovery, and binary code analysis. Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance and render learning-based systems potentially unsuitable for security tasks and practical deployment. In this paper, we look at this problem with critical eyes. First, we identify common pitfalls in the design, implementation, and evaluation of learning-based security systems. We conduct a study of 30 papers from top-tier security conferences within the past 10 years, confirming that these pitfalls are widespread in the current security literature. In an empirical analysis, we further demonstrate how individual pitfalls can lead to unrealistic performance and interpretations, obstructing the understanding of the security problem at hand. As a remedy, we propose actionable recommendations to support researchers in avoiding or mitigating the pitfalls where possible. Furthermore, we identify open problems when applying machine learning in security and provide directions for further research.

Classification of malware for security improvement in IoT using heuristic aided adaptive multi-scale and dilated ResneXt with gated recurrent unit

The rise of malware in the Internet of Things (IoT) realm exploits sensitive IoT devices that lead to extensive malicious attacks that pose a significant danger to the integrity of the Internet ecosystem. Addressing this threat effectively requires a robust system for classifying and attributing IoT malware, serving as vital initial steps toward implementing countermeasures for attack prevention and mitigation. So, in this work, a new malware classification technique using deep learning is developed for solving different kinds of malware in IoT. The required images for the malware classification process are gathered from online databases and given to the developed Adaptive Multi-scale and Dilated ResneXt with Gated Recurrent Unit (AMDR-GRU)-based malware classification process. Here, the last layer of the ResneXt is modified with GRU, and several parameters are optimized in the suggested AMDR-GRU model with the support of the designed Intensified Random Parameter-based Chameleon Swarm Algorithm (IRPCSA) to enhance the classification performance. Finally, the developed AMDR-GRU model offered the classified outcome and the obtained classification results are compared with different existing malware classification models to prove the effectiveness of the developed model over others. The developed model offered 99.59 % of precision. The result proved that the developed model is used to classify different malware without reverse engineering, binary code analysis, and feature engineering. Moreover, by effectively classifying malware, this technique can contribute to enhancing the security of IoT devices, protecting them from potential threats and vulnerabilities. This has significant implications for ensuring the privacy, integrity, and overall safety of IoT systems.

Analysis of ChatGPT's capabilities for solving problems of reverse-engineering of software

The article presents the results of the analysis and evaluation of the capabilities of the open version of the chatbot with artificial intelligence ChatGPT 3.5 for solving typical problems of software reverse engineering. Three classes of reverse engineering tasks were selected for analysis: source code analysis, binary code analysis, and data models analysis. In each class of tasks, the most typical tasks were selected taking into account the limitations of ChatGPT regarding the processing of graphical models and the amount of input and output data, and sets of test tasks were developed for each task. As an approach to assessment, an approach similar to the assessment of competencies of higher education applicants after studying the relevant discipline was chosen. The following were considered as criteria for evaluating answers to test tasks: correctness (rightness, coincidence with expectations); completeness (obtaining the final result); accuracy (solving the task without additional questions); reasonableness (availability of explanations and answers to questions). The following scale was used and described for evaluations: excellent, very good, good, satisfactory, sufficient, unsatisfactory. During the testing for each test task, tasks statement and all necessary data were entered through the ChatGPT interface. As a result of the analysis, it was determined that ChatGPT better solves the problems of source code analysis (excellent and very good grades were obtained for semantic and structural analysis, restoration of mathematical support, quality assessment, security audit and refactoring, as well as for conversion to another programming language), tasks of decompilation of IDA pseudocode into complete C source code, tasks of reverse engineering of relational databases, and tasks of generating YARA rules for recognizing file formats. Unsatisfactory grades were obtained for dynamic analysis of assembly code and determination of binary file structures of non-standard formats. ChatGPT solves the rest of the problems well and satisfactorily, but requires checking the results, clarifying queries and prompts, as well as manual error correction in some cases. ChatGPT errors were observed when analyzing binary data represented by hexadecimal symbols, as well as errors in generated scripts for programming in IDA. On the basis of the set grades, conclusions were made regarding the expediency, possibility, or impracticality of using ChatGPT to solve each type of software reverse engineering problems, and appropriate recommendations were also provided. Prospects for further research include testing new versions of ChatGPT and other similar artificial intelligence systems regarding the capabilities of analyzing and synthesizing graphical models of software.

An Inclusive Report on Robust Malware Detection and Analysis for Cross-Version Binary Code Optimizations

Numerous practices exist for binary code similarity detection (BCSD), such as Control Flow Graph, Semantics Scrutiny, Code Obfuscation, Malware Detection and Analysis, vulnerability search, etc. On the basis of professional knowledge, existing solutions often compare particular syntactic aspects retrieved from binary code. They either have substantial performance overheads or have inaccurate detection. Furthermore, there aren't many tools available for comparing cross-version binaries, which may differ not only in programming with proper syntax but also marginally in semantics. This Binary code similarity detection is existing for past 10 years, but this research area is not yet systematically analysed. The paper presents a comprehensive analysis on existing Cross-version Binary Code Optimization techniques on four characteristics: 1. Structural analysis, 2. Semantic Analysis, 3. Syntactic Analysis, 4. Validation Metrics. It helps the researchers to best select the suitable tool for their necessary implementation on binary code analysis. Furthermore, this paper presents scope of the area along with future directions of the research.

Binary Code Representation With Well-Balanced Instruction Normalization

The recovery of contextual meanings on a machine code is required by a wide range of binary analysis applications, such as bug discovery, malware analysis, and code clone detection. To accomplish this, advancements on binary code analysis borrow the techniques from natural language processing to automatically infer the underlying semantics of a binary, rather than replying on manual analysis. One of crucial pipelines in this process is instruction normalization, which helps to reduce the number of tokens and to avoid an out-of-vocabulary (OOV) problem. However, existing approaches often substitutes operands with a common token ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., callee target → FOO), inevitably resulting in the loss of important information. In this paper, we introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">well-balanced instruction normalization</i> (WIN), a novel approach that retains rich code information while minimizing the downsides of code normalization.With large swaths of binary code, our finding shows that the instruction distribution follows Zipf’s Law like a natural language, a function conveys contextually meaningful information, and the same instruction at different positions may require diverse code representations. To show the effectiveness of WIN, we present DeepSemantic that harnesses the BERT architecture with two training phases: pre-training for generic assembly code representation, and fine-tuning for building a model tailored to a specialized task. We define a downstream task of binary code similarity detection, which requires underlying code semantics. Our experimental results show that our binary similarity model with WIN outperforms two state-of-the-art binary similarity tools, DeepBinDiff and SAFE, with an average improvement of 49.8% and 15.8%, respectively.

Return Instruction Classification in Binary Code Using Machine Learning

Binary code analysis is vital in source code unavailable cases, such as malware analysis and software vulnerability mining. Its first step could be function identification. Most function identification methods are based on function prologs/epilogs. However, functions may not have standard prologs/epilogs. To identify these functions, we need to use other methods. One approach is to identify return instructions first and then identify the start of a function. Currently, the multi-layer perceptron model is exploited to identify and validate a return instruction at a specific location. On this basis, a new approach is proposed to improve accuracy and provide more details. Specifically, a return instruction is classified into three classes: (1) false return instruction, (2) true return instruction inner a function but not the last instruction, and (3) true return instruction at the end of a function. The evaluation is performed on 5782 real-world binaries. Meanwhile, common classifiers including fully connected neural network, Two-layer Bidirectional Recurrent Neural Network (TBRNN), Two-layer Bidirectional Gate Recurrent Unit (TBGRU), Two-layer Bidirectional Long Short-term Memory Network (TBLSTM), Decision Tree, Random Forest, XGBoost, and Support Vector Machine (SVM) are evaluated on the same data set. The result shows that TBLSTM achieves an accuracy of 99.78%, which is higher than that of other classifiers in the evaluation, including the state-of-the-art tool IDA Pro 7.7.

Efficient Binary Static Code Data Flow Analysis Using Unsupervised Learning

The ever-increasing need to ensure that code is reliably, efficiently and safely constructed has fueled the evolution of popular static binary code analysis tools. In identifying potential coding flaws in binaries, tools such as IDA Pro are used to disassemble the binaries into an opcode/ assembly language format in support of manual static code analysis. Because of the highly manual and resource-intensive nature involved with analyzing large binaries, the probability of overlooking potential coding irregularities and inefficiencies is quite high. In this paper, a light-weight, unsupervised data flow methodology is described which uses highly correlated data flow graph (CDFGs) to identify coding irregularities such that analysis time and required computing resources are minimized. Such analysis accuracy and efficiency gains are achieved by using a combination of graph analysis and unsupervised machine learning techniques which allows an analyst to focus on the most statistically significant flow patterns while performing binary static code analysis.

Disassemble Byte Sequence Using Graph Attention Network

Disassembly is the basis of static analysis of binary code and is used in malicious code detection, vulnerability mining, software optimization, etc. Disassembly of arbitrary suspicious code blocks (e.g., for suspicious traffic packets intercepted by the network) is a difficult task. Traditional disassembly methods require manual specification of the starting address and cannot automate the disassembly of arbitrary code blocks. In this paper, we propose a disassembly method based on code extension selection network by combining traditional linear sweep and recursive traversal methods. First, each byte of a code block is used as the disassembly start address, and all disassembly results (control flow graphs) are combined into a single flow graph. Then a graph attention network is trained to pick the correct subgraph (control flow graph) as the final result. In the experiment, the compiler-generated executable file, as well as the executable file generated by hand-written assembly code, the data file and the byte sequence intercepted by the code segment were tested, and the disassembly accuracy was 93%, which can effectively distinguish the code from the data.

MCFT-CNN: Malware classification with fine-tune convolution neural networks using traditional and transfer learning in Internet of Things

With ever-increasing, internet-connected devices provide an opportunity to fulfil the attacker’s malicious intention. They use malicious programs to compromise the devices and use them to infect others also. The security researchers are straggling to develop a technique that detects all the malware accurately because of the use of invincible techniques in the development of malware such as strong encryption, obfuscation, polymorphic and metamorphic engine. In this context, this paper proposes a novel malware classification with fine-tune convolution neural networks (MCFT-CNN) model. The MCFT-CNN model detects the unknown malware sample without feature engineering and prior knowledge of binary code analysis or reverse engineering, even the advanced evading techniques used to develop the malware. The model uses deep transfer learning to classify the malware images to their respective malware families. The proposed model enhances the ResNet50 model by altering the last layer with a fully connected dense layer. The output of fully connected dense layer and knowledge of ImageNet model are supplied to softmax layer for malware classification. The model is trained with MalImg malware datasets. The proposed model reported 99.18% accuracy and 5.14ms prediction time. The model also shows consistent performance with a relatively larger dataset (Microsoft malware challenge dataset, approximately 500GB) with 98.63% accuracy and 5.15ms prediction time. The proposed model shows consistent efficacy with two benchmark datasets that clarify the model’s generalisability to perform on the diverse datasets.

Binary Linear Codes With Optimal Scaling: Polar Codes With Large Kernels

We prove that, for the binary erasure channel (BEC), the polar-coding paradigm gives rise to codes that not only approach the Shannon limit but do so under the best possible scaling of their block length as a function of the gap to capacity. This result exhibits the first known family of binary codes that attain both optimal scaling and quasi-linear complexity of encoding and decoding. Our proof is based on the construction and analysis of binary polar codes with large kernels. When communicating reliably at rates within ε > 0 of capacity, the code length n often scales as O(1/ε <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">μ</sup> ), where the constant μ is called the scaling exponent. It is known that the optimal scaling exponent is μ = 2, and it is achieved by random linear codes. The scaling exponent of conventional polar codes (based on the 2×2 kernel) on the BEC is μ = 3.63. This falls far short of the optimal scaling guaranteed by random codes. Our main contribution is a rigorous proof of the following result: for the BEC, there exist <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> × <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> binary kernels, such that polar codes constructed from these kernels achieve scaling exponent μ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> ) that tends to the optimal value of 2 as <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> grows. We furthermore characterize precisely how large <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> needs to be as a function of the gap between μ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> ) and 2. The resulting binary codes maintain the recursive structure of conventional polar codes, and thereby achieve construction complexity O(n) and encoding/decoding complexity O(nlogn).

Практическая абстрактная интерпретация бинарного кода

The mathematical foundations of abstract interpretation provide a unified method of formalization and research of program analysis algorithms for a broad spectrum of practical problems. However, its practical usage for binary code analysis faces several challenges, of both scientific and engineering nature. In this paper we address some of those challenges. We describe an intermediate representation that is tailored to binary code analysis; unlike some other IRs it is still useable in system code analysis. To achieve this, we take into account the low-level specifics of how CPUs work; on the IR level this mostly pertains to modeling main memory in that accesses can fail, and addresses can alias. Further, we propose an infrastructure for carrying out abstract interpretation on top of the IR. The user needs to implement the abstract state and the transfer functions, and the infrastructure handles the rest: two executors are currently implemented, one for analysis of a single path, and one for fixed point analysis. Both executors handle interprocedural analysis internally, via inlining or using summaries, so the interpretations only consider only procedure at a time, which greatly simplifies implementation. The IR and the abstract interpretation framework are used together to define a model pipeline for a target instruction set architecture, consisting of a fetch stage, a decode stage, and an execute stage. A distinct fetch stage allows to model delay slots, hardware loops, etc. We currently have limited implementations for RISC-V and x86. The x86 implementation is evaluated in two experiments where concolic execution is used to automatically analyze a «crackme» program, both in dynamic (execution trace) and static (executable image) setting. In conclusion, we outline the future directions of our project.

X64Unpack: Hybrid Emulation Unpacker for 64-bit Windows Environments and Detailed Analysis Results on VMProtect 3.4

In spite of recent remarkable advances in binary code analysis, malware developers are still using complex anti-reversing techniques to make analysis difficult. To protect malware, they use packers, which are (commercial) tools that contain various anti-reverse engineering techniques such as code encryption, anti-debugging, and code virtualization. In this paper, we present x64Unpack: a hybrid emulation scheme that makes it easier to analyze packed executable files and automatically unpacks them in 64-bit Windows environments. The most distinguishable feature of x64Unpack compared to other dynamic analysis tools is that x64Unpack and the target program share virtual memory to support both instruction emulation and direct execution. Emulation runs slow but provides detailed information, whereas direct execution of the code chunk runs very fast and can handle complex cases regarding to operating systems or hardware devices. With x64Unpack, we can monitor major API (Application Programming Interface) function calls or conduct fine-grained analysis at the instruction-level. Furthermore, x64Unpack can detect anti-debugging code chunks, dump memory, and unpack the packed files. To verify the effectiveness of x64Unpack, experiments were conducted on the obfuscation tools: UPX 3.95, MPRESS 2.19, Themida 2.4.6, and VMProtect 3.4. Especially, VMProtect and Themida are considered as some of the most complex commercial packers in 64-bit Windows environments. Experimental results show that x64Unpack correctly emulates the packed executable files and successfully produces the unpacked version. Based on this, we provide the detailed analysis results on the obfuscated executable file that was generated by VMProtect 3.4.

Hybrid Neural Network Model for Protection of Dynamic Cyber Infrastructure

The paper considers a combination of modern artificial neural networks (ANN) that solves the security relative task of intrusion prevention and vulnerabilities detection in cybernetic infrastructure with dynamic network topology. Self-organizing networks, WSN, m2m networks, IIoT, mesh networks are faced with the cyberthreats of specific character: dynamic routing failures, node isolation, DDoS attacks, traffic lack, etc. Most of them are caused by cybersecurity weaknesses: the software vulnerabilities and architectural features of dynamically reconfigured network. The existing methods of binary code analysis and intrusion detection can work with a small number of data sets, are designed for either code inspection or network checking, and are targeted for static networks with regular topology. The proposed neural model demonstrates an universal approach that deals with the cybersecurity weakness as a systems genuine property and attempts to approximate it using a hybrid deep ANN. The new ANN detects both the network security defects and binary code vulnerabilities at once with high accuracy (more than 0.97). It also shows good performance capacity processing big data of the undercontrolled network.

Next-Generation Intermediate Representations for Binary Code Analysis

Many binary code analysis tools rely on intermediate representation (IR) derived from a binary code, instead of working directly with machine instructions. In this paper, we first consider binary code analysis problems that benefit from IR and compile a list of requirements that the IR suitable for solving these problems should meet. Generally speaking, a universal binary analysis platform requires two principal components. The first component is a retargetable instruction decoder that utilizes external specifications to describe target instruction sets. External specifications facilitate maintenance and allow one to quickly implement support for new instruction sets. We analyze some of the most popular instruction set architectures (ISAs), including those used in microcontrollers, and from that compile a list of requirements for the retargetable decoder. We then overview existing multi-ISA decoders and propose our vision of a more generic approach, based on a multi-layer directed acyclic graph that describes the decoding process in universal terms. The second component of the analysis platform is the actual architecture-neutral IR. In this paper, we describe such IRs and propose Pivot 2, an IR that is low-level enough to be easily constructed from decoded machine instructions, also being easy to analyze. The main features of Pivot 2 are explicit side effects, SSA variables, simpler alternative to phi-functions, and extensible elementary operation set at the core. This IR also supports machines that have multiple memory address spaces. Finally, we propose a way to tie the decoder and the IR together to fit them to most of the binary code analysis tasks through abstract interpretation on top of the IR. The proposed scheme takes into account various aspects of target architectures that are overlooked in many other works, including pipeline specifics (handling of delay slots, hardware loop support, etc.), exception and interrupt management, and generic address space model, in which accesses may have arbitrary side effects due to memory-mapped devices or other non-trivial behavior of the memory system.

Parallelization of Implementations of Purely Sequential Algorithms

The work is dedicated to the parallelization of programs in especially difficult cases when the used algorithm is purely sequential, there are no parallel alternatives to this algorithm, and its execution time is unacceptably high. Various parallelization methods for software implementations of such algorithms and resulting computational load balancing are considered that make it possible to considerably accelerate the execution of application programs using purely sequential algorithms. The proposed methods are illustrated with examples of their application to two algorithms used in a dynamic binary code analysis toolset. The main goal of this paper is to show that the use of a purely sequential algorithm in a software implementation does not necessarily imply that its execution is inevitably sequential. The proposed methods of parallelizing implementations of such algorithms and balancing the resulting computational load can help develop efficient parallel programs that fully utilize the hardware capabilities of modern computers.

Machine Learning-Based Analysis of Program Binaries: A Comprehensive Study

Binary code analysis is crucial in various software engineering tasks, such as malware detection, code refactoring, and plagiarism detection. With the rapid growth of software complexity and the increasing number of heterogeneous computing platforms, binary analysis is particularly critical and more important than ever. Traditionally adopted techniques for binary code analysis are facing multiple challenges, such as the need for cross-platform analysis, high scalability and speed, and improved fidelity, to name a few. To meet these challenges, machine learning-based binary code analysis frameworks attract substantial attention due to their automated feature extraction and drastically reduced efforts needed on large-scale programs. In this paper, we provide the taxonomy of machine learning-based binary code analysis, describe the recent advances and key findings on the topic, and discuss the key challenges and opportunities. Finally, we present our thoughts for future directions on this topic.

Декодирование машинных команд в задаче абстрактной интерпретации бинарного кода

Not only developers require tools that work with binary code: it is impossible to achieve sufficient security in contemporary software without inspecting its properties at this level. A key component of binary code analysis toolset is the instruction decoder. Different instruction set architectures give rise to decoders that are differently structured, the decoding results are incompatible, and maintenance is hindered because of the ubiquitous practice of implementing decoders as cascades of conditional operators. Further binary code analysis (control and data flows, symbolic interpretation, etc.) cannot easily be ported from one target architecture to another because of limitations and peculiarities of decoder implementations. In this paper, we propose an approach to decoding machine instructions that is based on external specifications. The main distinction is an original way of representing the decoder instruction universally, i.e. in a way that does not differ from one architecture to another. The decoding process is handled by an abstract stack machine we have developed. Despite the inevitable overhead stemming from the approach’s universality, an implementation prototype displays only 1.5-2.5 times slowdown compared to traditional decoders; the measurements include time required to parse the specification and build the required data structures. The proposed approach to organizing decoding would allow, in the long run, to establish a unified stack of binary code analysis tools that would be applicable to different instruction set architectures. The paper further discusses questions of translating the decoded instructions into a machine-neutral internal representation for analyzing their operational semantics and carrying out abstract interpretation. We give examples of practically useful interpretations: the concrete interpretation and a “directing” interpretation that allows to tie the idea of abstract interpretation with the problem of deeper automatic analysis of individual paths in a program.

Automatic Detection and Bypassing of Anti-Debugging Techniques for Microsoft Windows Environments

In spite of recent remarkable advances in binary code analysis, adversaries are still using diverse anti-reversing techniques for obfuscating code and making analysis difficult. Unlike ...

О новом поколении промежуточных представлений, применяемом для анализа бинарного кода

A lot of binary code analysis tools do not work directly with machine instructions, instead relying on an intermediate representation from the binary code. In this paper, we first analyze problems in binary code analysis that benefit from such an IR and construct a list of requirements that an IR suitable for solving these problems must meet. Generally speaking, a universal binary analysis platform requires two principal components. The first component is a retargetable instruction decoder that utilizes external specifications for describing target instruction sets. External specifications facilitate maintainability and allow for quickly adding support for new instruction sets. We analyze some of the more common ISAs, including those used in microcontrollers, and from that produce a list of requirements for a retargetable decoder. We then survey existing multi-ISA decoders and propose our vision of a more generic approach, based on a multi-layered directed acyclic graph describing the decoding process in universal terms. The second component of an analysis platform is the actual architecture-neutral IR. In this paper we describe such existing IRs, and propose Pivot 2, an IR that is low-level enough to be easily constructed from decoded machine instructions, and at the same time is also easy to analyze. The main features of Pivot 2 are explicit side effects, SSA variables, a simpler alternative to phi-functions, and an extensible elementary operation set at the core. The IR also supports machines that have multiple memory address spaces. Finally, we propose a way to tie the decoder and the IR together to fit them to most binary code analysis tasks through abstract interpretation on top of the IR. The proposed scheme takes into account various aspects of target architectures that are overlooked in many other works, including pipeline specifics (handling of delay slots, hardware loop support, etc.), exception and interrupt management, and a generic address space model where accesses may have arbitrary side effects due to memory-mapped devices or other non-trivial behavior of the memory system.

Платформа межпроцедурного статического анализа бинарного кода

Платформа межпроцедурного статического анализа бинарного кода