Reverse Engineering Tasks Research Articles

Analysis of ChatGPT's capabilities for solving problems of reverse-engineering of software

The article presents the results of the analysis and evaluation of the capabilities of the open version of the chatbot with artificial intelligence ChatGPT 3.5 for solving typical problems of software reverse engineering. Three classes of reverse engineering tasks were selected for analysis: source code analysis, binary code analysis, and data models analysis. In each class of tasks, the most typical tasks were selected taking into account the limitations of ChatGPT regarding the processing of graphical models and the amount of input and output data, and sets of test tasks were developed for each task. As an approach to assessment, an approach similar to the assessment of competencies of higher education applicants after studying the relevant discipline was chosen. The following were considered as criteria for evaluating answers to test tasks: correctness (rightness, coincidence with expectations); completeness (obtaining the final result); accuracy (solving the task without additional questions); reasonableness (availability of explanations and answers to questions). The following scale was used and described for evaluations: excellent, very good, good, satisfactory, sufficient, unsatisfactory. During the testing for each test task, tasks statement and all necessary data were entered through the ChatGPT interface. As a result of the analysis, it was determined that ChatGPT better solves the problems of source code analysis (excellent and very good grades were obtained for semantic and structural analysis, restoration of mathematical support, quality assessment, security audit and refactoring, as well as for conversion to another programming language), tasks of decompilation of IDA pseudocode into complete C source code, tasks of reverse engineering of relational databases, and tasks of generating YARA rules for recognizing file formats. Unsatisfactory grades were obtained for dynamic analysis of assembly code and determination of binary file structures of non-standard formats. ChatGPT solves the rest of the problems well and satisfactorily, but requires checking the results, clarifying queries and prompts, as well as manual error correction in some cases. ChatGPT errors were observed when analyzing binary data represented by hexadecimal symbols, as well as errors in generated scripts for programming in IDA. On the basis of the set grades, conclusions were made regarding the expediency, possibility, or impracticality of using ChatGPT to solve each type of software reverse engineering problems, and appropriate recommendations were also provided. Prospects for further research include testing new versions of ChatGPT and other similar artificial intelligence systems regarding the capabilities of analyzing and synthesizing graphical models of software.

New demand on assembly language proficiency in performing binary reverse engineering tasks

Cybersecurity encompasses a wide field of disciplines and as cyber threat landscape changes, there is a need for tools, techniques and skills to provide safe and secure internet environment. The cyber space industry introduced new roles for reverse engineers, malware analysts, digital forensic experts, exploit engineers, etc which demand the new skill set, and in this context, the proficiency in assembly language programming is highly essential. This paper presents an observation on the training programme for software developers from a software integration company to attain the skills of reverse engineers, application penetration testers as well as application security analysts. In preparing for the new generation of reverse engineers and other new roles that are related to cybersecurity, it would be a good step if assembly language could be taught as a separate programming subject, and it would be highly recommended for higher education institutions to collaborate with the industry to undertake co-teaching in supporting the new roles within the realm of cybersecurity. It has been well observed that being endowed with a working knowledge of developing application from scratch using assembly language would offer a foundation to be a good binary reverse engineer.

A review of data abstraction.

It is well-known that Artificial Intelligence (AI), and in particular Machine Learning (ML), is not effective without good data preparation, as also pointed out by the recent wave of data-centric AI. Data preparation is the process of gathering, transforming and cleaning raw data prior to processing and analysis. Since nowadays data often reside in distributed and heterogeneous data sources, the first activity of data preparation requires collecting data from suitable data sources and data services, often distributed and heterogeneous. It is thus essential that providers describe their data services in a way to make them compliant with the FAIR guiding principles, i.e., make them automatically Findable, Accessible, Interoperable, and Reusable (FAIR). The notion of data abstraction has been introduced exactly to meet this need. Abstraction is a kind of reverse engineering task that automatically provides a semantic characterization of a data service made available by a provider. The goal of this paper is to review the results obtained so far in data abstraction, by presenting the formal framework for its definition, reporting about the decidability and complexity of the main theoretical problems concerning abstraction, and discuss open issues and interesting directions for future research.

Malware-on-the-Brain: Illuminating Malware Byte Codes With Images for Malware Classification

Malware is a piece of software that was written with the intent of doing harm to data, devices, or people. Since a number of new malware variants can be generated by reusing codes, malware attacks can be easily launched and thus become common in recent years, incurring huge losses in businesses, governments, financial institutes, health providers, etc. To defeat these attacks, malware classification is employed, which plays an essential role in anti-virus products. However, existing works that employ either static analysis or dynamic analysis have major weaknesses in complicated reverse engineering and time-consuming tasks. In this paper, we propose a visualized malware classification framework called VisMal, which provides highly efficient categorization with acceptable accuracy. VisMal converts malware samples into images and then applies a contrast-limited adaptive histogram equalization algorithm to enhance the similarity between malware image regions in the same family. We provided a proof-of-concept implementation and carried out an extensive evaluation to verify the performance of our framework. The evaluation results indicate that VisMal can classify a malware sample within 4.0 ms and have an average accuracy of 96.0%. Moreover, VisMal provides security engineers with a simple visualization approach to further validate its performance.

SMA-Net: Deep learning-based identification and fitting of CAD models from point clouds

Identification and fitting is an important task in reverse engineering and virtual/augmented reality. Compared to the traditional approaches, carrying out such tasks with a deep learning-based method have much room to exploit. This paper presents SMA-Net (Spatial Merge Attention Network), a novel deep learning-based end-to-end bottom-up architecture, specifically focused on fast identification and fitting of CAD models from point clouds. The network is composed of three parts whose strengths are clearly highlighted: voxel-based multi-resolution feature extractor, spatial merge attention mechanism and multi-task head. It is trained with both virtually-generated point clouds and as-scanned ones created from multiple instances of CAD models, themselves obtained with randomly generated parameter values. Using this data generation pipeline, the proposed approach is validated on two different data sets that have been made publicly available: robot data set for Industry 4.0 applications, and furniture data set for virtual/augmented reality. Experiments show that this reconstruction strategy achieves compelling and accurate results in a very high speed, and that it is very robust on real data obtained for instance by laser scanner and Kinect.

Binary code traceability of multigranularity information fusion from the perspective of software genes

Binary code traceability aims to use the relevant characteristics of anonymous binary codes to identify concealed authors or teams and replace error-prone and time-consuming manual reverse engineering tasks with automated systems. Although significant progress has been made in source code traceability technology, research on tracking binary files is still limited. Hence, we propose a feature extraction method and deep learning model that exploit the sequence and structure information of binary codes to identify the authors of anonymous and malicious binary codes and their relations with other known binary code families. We further propose a new multigranularity information fusion feature based on biological genes oriented to the traceability of binary codes. The evaluations conducted on the Google Code Jam (GCJ) dataset indicate that our method can accurately trace the binary code from 1000 people to the target author with an accuracy rate of 71%. Further, experimental results verify the robustness of the proposed model. For malicious code datasets, in particular, the proposed method achieved a stable traceability accuracy rate for malicious samples using only a small number of training samples. For the problem of malicious code tracking, in 300 team organizations, the proposed method achieved a code-tracing accuracy rate of 82%.

Метод восстановления протокольных автоматов по бинарному коду

Security analysis of network programs includes set of reverse engineering tasks of network protocols. Data formats restoring and implemented protocol automaton are the previous task issues. Unlike quite researched problem of formats restoring where there are lots of scientist’s papers, finding out the protocol's automaton program implementation looks like terra incognita and the cornerstone is a protocol state description currently undefined. There are two general ways to retrieve the implemented protocol automaton: an analysis of the network traces and looking into binary trace of the target application. This article offers a second one method. The first aim of the paper is the way to describe a mathematical model of a protocol automaton and a method for projecting it onto an executing application binary code. The second is concept of the protocol state definition and a principle to detect the states transitions based on some "global" binary trace objects, are described. Thirdly, there is suggested a protocol automaton precising manner by in-memory fuzzing based on a "floating" fork-server to manage states transitions. Finally, developed toolset's scheme and experiments on its using with a real VPN client, are shown.

Application of Selected Reverse Engineering Procedures Based on Specific Requirements

Abstract The paper describes a solution of reverse engineering tasks using the equipment of the laboratory of the Technical University in Zvolen. The first task was linked to evaluation of a wear rate of the tool used to for mulching and elimination of unwanted wood or weed vegetation. The obtained data will be used as a base line for determination of the time interval for the tool replacement. We developed a prototype of components and their drawing documentation for a single-purpose machine for cutting barley. We created the 3D model using the method of photogrammetry. The finished drawing documentation and prototypes were handed over to the company KRUP. Finally, we identified the most suitable procedures for creating a model of a part with a complex shape. We tested the techniques of editing the point cloud processing, as well as of smoothing the surfaces and automating the creation of partial geometric elements of the model. Based on the properties of the component, which is the starting point of the entire reverse engineering process and from the achieved results, we set recommendations for the selection of appropriate procedures.

Adabot: Fault-Tolerant Java Decompiler (Student Abstract)

Reverse Engineering has been an extremely important field in software engineering, it helps us to better understand and analyze the internal architecture and interrealtions of executables. Classical Java reverse engineering task includes disassembly and decompilation. Traditional Abstract Syntax Tree (AST) based disassemblers and decompilers are strictly rule defined and thus highly fault intolerant when bytecode obfuscation were introduced for safety concern. In this work, we view decompilation as a statistical machine translation task and propose a decompilation framework which is fully based on self-attention mechanism. Through better adaption to the linguistic uniqueness of bytecode, our model fully outperforms rule-based models and previous works based on recurrence mechanism.

Systematic Approach to Malware Analysis (SAMA)

Malware threats pose new challenges to analytic and reverse engineering tasks. It is needed for a systematic approach to that analysis, in an attempt to fully uncover their underlying attack vectors and techniques and find commonalities between them. In this paper, a method of malware analysis is described, together with a report of its application to the case of Flame and Red October. The method has also been used by different analysts to analyze other malware threats like ‘Stuxnet’, ‘Dark Comet’, ‘Poison Ivy’, ‘Locky’, ‘Careto’, and ‘Sofacy Carberp’. The method presented in this work is a systematic and methodological process of analysis, whose main objective is the acquisition of knowledge as well as to gain a full understanding of a particular malware. Using the proposed method to analyze two well-known malware as ‘Flame’ and ‘Red October’ will help to understand the added value of the method.

CPA: Accurate Cross-Platform Binary Authorship Characterization Using LDA

Binary authorship characterization refers to the process of identifying stylistic characteristics that are related to the author of an anonymous binary code. The aim is to automate the laborious and error-prone reverse engineering task of discovering information related to the author(s) of binary code. This paper presents CPA , a novel approach for characterizing the authors of program binaries. Instead of using generic features such as n-grams, CPA proposes a set of new features based on collections of various aspects of author style, including author code traits, code structure characteristics, and author expertise in solving coding tasks. It employs the Latent Dirichlet Allocation (LDA) algorithm to generate author style signatures to help identify similar author style characteristics in other binaries. We evaluated CPA on large datasets extracted from selected open-source C/C++ projects in GitHub and Google Code Jam events, and it successfully attributed a large number of authors with a significantly higher $F_{1}$ score: around 91% when the number of authors was 1,500. In addition, the false positive rate was low, around 1.5%. When the code was subjected to refactoring techniques or code transformation or was processed using different compilers/compilation settings, there was no significant drop in accuracy, demonstrating the robustness of our tool. Finally, in the case of code written by multiple authors, CPA was able to identify the authors with a high $F_{1}$ score, around 89%.

On the feasibility of binary authorship characterization

This work aims to develop an automatic tool that can perform the laborious and error-prone reverse engineering task of binary authorship characterization, i.e., determining clues related to the author(s) of a piece of binary code. Software code written by human programmers reflects the author's educational background, level of expertise, and coding traits. Accordingly, these may be characterized by identifying meaningful features and examining them. Binary authorship characterization reveals information that can be extremely useful for security applications such as digital forensics, malware triage, and binary vulnerability tracking. This paper proposes a system, BinChar, that capture various aspects of author style, including code trait characteristics, code structure characteristics, and code behavior characteristics. For the purpose of detection, a Convolutional Neural Network (CNN) is used. The results generated by the CNN are evaluated more precisely using Bayesian calibration. We tested BinChar in identifying the characteristics of the authors of program binaries. Also, we applied it to almost 500 GB of malware samples provided by the Kaggle Microsoft Malware Classification Challenge, to demonstrate that BinChar is an appropriate tool for characterizing malware families. As an illustration, we report a case study in which we determine the author characteristics of the Mirai botnet and compare them with the author characteristics of 360,000 malware samples.

An extended framework for knowledge modelling and reuse in reverse engineering projects

Reverse engineering is when a real part is analysed in detail in order to create a numerical or virtual model. Reverse engineering allows for multiple redesign possibilities, including changes in the material, the shape and the parameters of the part. Reverse engineering is mostly a manual activity for companies and is thus time consuming. Indeed, measurements must be done in scanned files in order to fit sketches on a mesh and to finally rebuild the computer-aided design/bill of material. This manual process is acceptable when reverse engineering is exceptional. But it is considered as a non-value task when reverse engineering is routine. This non-value task could be automated, at least partially. To make it possible, a capitalization of the company’s part catalogue is a necessary step to proceed. The use of this capitalization can then drive the reverse engineering tasks to enable faster redesign possibilities. The aim of this contribution is thus to propose a knowledge model to support reverse engineering activities in order to integrate the reversed parts quickly into the new product’s detailed design. An extended knowledge framework based on the core product model is proposed, and a use case is shown to validate the feasibility of the proposal of the reverse engineering methodology called PHENIX.

Incremental Decompilation of Loop-Free Binary Code: Erlang

Decompiling byte code to a human readable format is an important research field. A proper decompiler can be used to recover lost source code, helps in different reverse engineering tasks and also enhances static analyzer tools by refining the calculated static semantic information. In an era with a lot of advancement in areas such as incremental algorithms and boolean satisfiability (SAT) solvers, the question of how to properly structure a decompilation tool to function in a completely incremental manner has remained an interesting problem. This paper presents a concise algorithm and structuring design pattern for byte code which has a loop-free representation, as is seen in the Erlang language. The algorithms presented in this paper were implemented and verified during the decompilation of the Erlang/OTP library.

FOSSIL

Identifying free open-source software (FOSS) packages on binaries when the source code is unavailable is important for many security applications, such as malware detection, software infringement, and digital forensics. This capability enhances both the accuracy and the efficiency of reverse engineering tasks by avoiding false correlations between irrelevant code bases. Although the FOSS package identification problem belongs to the field of software engineering, conventional approaches rely strongly on practical methods in data mining and database searching. However, various challenges in the use of these methods prevent existing function identification approaches from being effective in the absence of source code. To make matters worse, the introduction of obfuscation techniques, the use of different compilers and compilation settings, and software refactoring techniques has made the automated detection of FOSS packages increasingly difficult. With very few exceptions, the existing systems are not resilient to such techniques, and the exceptions are not sufficiently efficient. To address this issue, we propose FOSSIL , a novel resilient and efficient system that incorporates three components. The first component extracts the syntactical features of functions by considering opcode frequencies and applying a hidden Markov model statistical test. The second component applies a neighborhood hash graph kernel to random walks derived from control-flow graphs, with the goal of extracting the semantics of the functions. The third component applies z-score to the normalized instructions to extract the behavior of instructions in a function. The components are integrated using a Bayesian network model, which synthesizes the results to determine the FOSS function. The novel approach of combining these components using the Bayesian network has produced stronger resilience to code obfuscation. We evaluate our system on three datasets, including real-world projects whose use of FOSS packages is known, malware binaries for which there are security and reverse engineering reports purporting to describe their use of FOSS, and a large repository of malware binaries. We demonstrate that our system is able to identify FOSS packages in real-world projects with a mean precision of 0.95 and with a mean recall of 0.85. Furthermore, FOSSIL is able to discover FOSS packages in malware binaries that match those listed in security and reverse engineering reports. Our results show that modern malware binaries contain 0.10--0.45 of FOSS packages.

Metrological aspects of reverse engineering of standardized products

The aim of the work is the development of an algorithm of reverse engineering for standardized products. To create metrological models, and even more to solve the complex task of reverse engineering, it is necessary to have not only metrological measurements and calculations using CAD, but also heuristic techniques that allow to recreate the conceptual model and design a new product instead of the damaged one taking into account real technological capabilities. These heuristic techniques are the selection of elements, the establishment of interrelations between elements, the identification of interrelated and independent groups of elements, the definition of inputs and outputs of the system, the use of models in the form of matrices and graphs. The relationship between realistic, conceptual, ideal and normative models is found in the case when the input data are the actual dimensions of the existing product, which are measured with certain errors. As a result of the work it is obtained the heuristic algorithm of the reverse engineering process for gear transmission; it is established that in order to determine the parameters of an ideal model, along with metrological measurements that allow us to find a realistic experimental model, it is necessary to use heuristic techniques and CAD; for complex engineering products with a high degree of standardization, reverse engineering is connected first of all with the use of databases of standard parts, elements of parts, steel grades, and the standard methods of calculation contained in CAE modules

Using Sub-Network Combinations to Scale Up an Enumeration Method for Determining the Network Structures of Biological Functions.

Deduction of biological regulatory networks from their functions is one of the focus areas of systems biology. Among the different techniques used in this reverse-engineering task, one powerful method is to enumerate all candidate network structures to find suitable ones. However, this method is severely limited by calculation capability: due to the brute-force approach, it is infeasible for networks with large number of nodes to be studied using traditional enumeration method because of the combinatorial explosion. In this study, we propose a new reverse-engineering technique based on the enumerating method: sub-network combinations. First, a complex biological function is divided into several sub-functions. Next, the three-node-network enumerating method is applied to search for sub-networks that are able to realize each of the sub-functions. Finally, complex whole networks are constructed by enumerating all possible combinations of sub-networks. The optimal ones are selected and analyzed. To demonstrate the effectiveness of this new method, we used it to deduct the network structures of a Pavlovian-like function. The whole Pavlovian-like network was successfully constructed by combining robust sub-networks, and the results were analyzed. With sub-network combination, the complexity has been largely reduced. Our method also provides a functional modular view of biological systems.

Multi-objective reverse engineering of variability-safe feature models based on code dependencies of system variants

Maintenance of many variants of a software system, developed to supply a wide range of customer-specific demands, is a complex endeavour. The consolidation of such variants into a Software Product Line is a way to effectively cope with this problem. A crucial step for this consolidation is to reverse engineer feature models that represent the desired combinations of features of all the available variants. Many approaches have been proposed for this reverse engineering task but they present two shortcomings. First, they use a single-objective perspective that does not allow software engineers to consider design trade-offs. Second, they do not exploit knowledge from implementation artifacts. To address these limitations, our work takes a multi-objective perspective and uses knowledge from source code dependencies to obtain feature models that not only represent the desired feature combinations but that also check that those combinations are indeed well-formed, i.e. variability safe. We performed an evaluation of our approach with twelve case studies using NSGA-II and SPEA2, and a single-objective algorithm. Our results indicate that the performance of the multi-objective algorithms is similar in most cases and that both clearly outperform the single-objective algorithm. Our work also unveils several avenues for further research.

A novel method for 3D reconstruction: Division and merging of overlapping B-spline surfaces

B-spline surfaces, extracted from scanned sensor data, are usually required to represent objects in inspection, surveying technology, metrology and reverse engineering tasks. In order to express a large object with a satisfactory accuracy, multiple scans, which generally lead to overlapping patches, are always needed due to, inter-alia, practical limitations and accuracy of measurements, uncertainties in measurement devices, calibration problems as well as skills of the experimenter. In this paper, we propose an action sequence consisting of division and merging. While the former divides a B-spline surface into many patches with corresponding scanned data, the latter merges the scanned data and its overlapping B-spline surface patch. Firstly, all possible overlapping cases of two B-spline surfaces are enumerated and analyzed from a view of the locations of the projection points of four corners of one surface in the interior of its overlapping surface. Next, the general division and merging methods are developed to deal with all overlapping cases, and a simulated example is used to illustrate aforementioned detailed procedures. In the sequel, two scans obtained from a three-dimensional laser scanner are simulated to express a large house with B-spline surfaces. The simulation results show the efficiency and efficacy of the proposed method. In this whole process, storage space of data points is not increased with new obtained overlapping scans, and none of the overlapping points are discarded which increases the representation accuracy. We believe the proposed method has a number of potential applications in the representation and expression of large objects with three-dimensional laser scanner data.

Capturing Uncertainty Information and Categorical Characteristics for Network Payload Grouping in Protocol Reverse Engineering

As a promising tool to recover the specifications of unknown protocols, protocol reverse engineering has drawn more and more attention in research over the last decade. It is a critical task of protocol reverse engineering to extract the protocol keywords from network trace. Since the messages of different types have different sets of protocol keywords, it is an effective method to improve the accuracy of protocol keyword extraction by clustering the network payload of unknown traffic into clusters and analyzing each clusters to extract the protocol keywords. Although the classic algorithms such asK-means and EM can be used for network payload clustering, the quality of resultant traffic clusters was far from satisfactory when these algorithms are applied to cluster application layer traffic with categorical attributes. In this paper, we propose a novel method to improve the accuracy of protocol reverse engineering by applying a rough set-based technique for clustering the application layer traffic. This technique analyze multidimension uncertain information in multiple categorical attributes based on rough sets theory to cluster network payload, and apply the Minimum Description Length criteria to determine the optimal number of clusters. The experiments show that our method outperforms the existing algorithms and improves the results of protocol keyword extraction.