Similarity Detection Techniques Research Articles

BinCodex: A comprehensive and multi-level dataset for evaluating binary code similarity detection techniques

The binary code similarity detection (BCSD) technique can quantitatively measure the differences between two given binaries and give matching results at predefined granularity (e.g., function), and has been widely used in multiple scenarios including software vulnerability search, security patch analysis, malware detection, code clone detection, etc. With the help of deep learning, the BCSD techniques have achieved high accuracy in their evaluation. However, on the one hand, their high accuracy has become indistinguishable due to the lack of a standard dataset, thus being unable to reveal their abilities. On the other hand, since binary code can be easily changed, it is essential to gain a holistic understanding of the underlying transformations including default optimization options, non-default optimization options, and commonly used code obfuscations, thus assessing their impact on the accuracy and adaptability of the BCSD technique. This paper presents our observations regarding the diversity of BCSD datasets and proposes a comprehensive dataset for the BCSD technique. We employ and present detailed evaluation results of various BCSD works, applying different classifications for different types of BCSD tasks, including pure function pairing and vulnerable code detection. Our results show that most BCSD works are capable of adopting default compiler options but are unsatisfactory when facing non-default compiler options and code obfuscation. We take a layered perspective on the BCSD task and point to opportunities for future optimizations in the technologies we consider.

Transformer induced enhanced feature engineering for contextual similarity detection in text

Availability of large data storage systems has resulted in digitization of information. Question and answering communities like Quora and stack overflow take advantage of such systems to provide information to users. However, as the amount of information stored gets larger, it becomes difficult to keep track of the existing information, especially information duplication. This work presents a similarity detection technique that can be used to identify similarity levels in textual data based on the context in which the information was provided. This work presents the transformer based contextual similarity detection (TCSD), which uses a combination of bidirectional encoder representations from transformers (BERT) and similarity metrics to derive features from the data. The derived features are used to train the ensemble model for similarity detection. Experiments were performed using the Quora question similarity data set. Results and comparisons indicate that the proposed model exhibits similarity detection with an accuracy of 92.5%, representing high efficiency.

Rapid detection of identity-by-descent tracts for mega-scale datasets

The ability to identify segments of genomes identical-by-descent (IBD) is a part of standard workflows in both statistical and population genetics. However, traditional methods for finding local IBD across all pairs of individuals scale poorly leading to a lack of adoption in very large-scale datasets. Here, we present iLASH, an algorithm based on similarity detection techniques that shows equal or improved accuracy in simulations compared to current leading methods and speeds up analysis by several orders of magnitude on genomic datasets, making IBD estimation tractable for millions of individuals. We apply iLASH to the PAGE dataset of ~52,000 multi-ethnic participants, including several founder populations with elevated IBD sharing, identifying IBD segments in ~3 minutes per chromosome compared to over 6 days for a state-of-the-art algorithm. iLASH enables efficient analysis of very large-scale datasets, as we demonstrate by computing IBD across the UK Biobank (~500,000 individuals), detecting 12.9 billion pairwise connections.

A Scalable Code Similarity Detection with Online Architecture and Focused Comparison for Maintaining Academic Integrity in Programming

<p>Many code similarity detection techniques have been developed to maintain academic integrity in programming. However, most of them assume that the student programs are locally available, and the computation can be run on any computer specification. Further, their comparison in raising suspicion is time-consuming as the student programs are pairwise compared to one another. This paper proposes a scalable code similarity detection with online architecture and focused comparison. The former enables student programs shared among lecturers and guarantees that the computation is runnable. The latter shortens the execution time as only some students are considered, with inclusion criteria determined by the lecturers. To boost up the scalability, the similarity algorithm is cosine correlation, which computation is linear time. Our evaluation shows that focused comparison leads to fewer comparisons and cosine correlation leads to shorter execution time.</p>

Empirical comparison of text-based mobile apps similarity measurement techniques

ContextCode-free software similarity detection techniques have been used to support different software engineering tasks, including clustering mobile applications (apps). The way of measuring similarity may affect both the efficiency and quality of clustering solutions. However, there has been no previous comparative study of feature extraction methods used to guide mobile app clustering.ObjectiveIn this paper, we investigate different techniques to compute the similarity of apps based on their textual descriptions and evaluate their effectiveness using hierarchical agglomerative clustering.MethodTo this end we carry out an empirical study comparing five different techniques, based on topic modelling and keyword feature extraction, to cluster 12,664 apps randomly sampled from the Google Play App Store. The comparison is based on three main criteria: silhouette width measure, human judgement and execution time.ResultsThe results of our study show that using topic modelling, in addition to collocation-based and dependency-based feature extractors perform similarly in detecting app-feature similarity. However, dependency-based feature extraction performs better than any other in finding application domain similarity (ρ = 0.7,p − value < 0.01).ConclusionsCurrent categorisation in the app store studied does not exhibit a good classification quality in terms of the claimed feature space. However, a better quality can be achieved using a good feature extraction technique and a traditional clustering method.

A comparison of code similarity analysers

Copying and pasting of source code is a common activity in software engineering. Often, the code is not copied as it is and it may be modified for various purposes; e.g. refactoring, bug fixing, or even software plagiarism. These code modifications could affect the performance of code similarity analysers including code clone and plagiarism detectors to some certain degree. We are interested in two types of code modification in this study: pervasive modifications, i.e. transformations that may have a global effect, and local modifications, i.e. code changes that are contained in a single method or code block. We evaluate 30 code similarity detection techniques and tools using five experimental scenarios for Java source code. These are (1) pervasively modified code, created with tools for source code and bytecode obfuscation, and boiler-plate code, (2) source code normalisation through compilation and decompilation using different decompilers, (3) reuse of optimal configurations over different data sets, (4) tool evaluation using ranked-based measures, and (5) local + global code modifications. Our experimental results show that in the presence of pervasive modifications, some of the general textual similarity measures can offer similar performance to specialised code similarity tools, whilst in the presence of boiler-plate code, highly specialised source code similarity detection techniques and tools outperform textual similarity measures. Our study strongly validates the use of compilation/decompilation as a normalisation technique. Its use reduced false classifications to zero for three of the tools. Moreover, we demonstrate that optimal configurations are very sensitive to a specific data set. After directly applying optimal configurations derived from one data set to another, the tools perform poorly on the new data set. The code similarity analysers are thoroughly evaluated not only based on several well-known pair-based and query-based error measures but also on each specific type of pervasive code modification. This broad, thorough study is the largest in existence and potentially an invaluable guide for future users of similarity detection in source code.

Event triggered intelligent video recording system using MS-SSIM for smart home security

Abstract This paper presents an intelligent system for event-triggered video recording for smart home applications. Video recording is triggered through a collaborative sensing strategy. PIR motion detectors are used for both directing the master wireless IP-camera for recording in a specific direction in the entrance hall or initiating other wireless IP-cameras for recording inside the rooms. An activated wireless camera starts video recording only during a targeted motion interval. Motion detection for initiation of the recording process is based on an enhanced Multi-Scale Structural Similarity detection technique. RFID tags are used in all rooms to identify persons entering these rooms. When the moving object shifts to another location at home, the local PIR sends a signal to the Gateway which initiates another video camera. Sensors collaborate for identification of the area to be monitored and the events which are to be recorded. The proposed system helps cover all smart home areas, save the required storage space and speeds-up video event analysis.

Viewing functions as token sequences to highlight similarities in source code

The detection of similarities in source code has applications not only in software re-engineering (to eliminate redundancies) but also in software plagiarism detection. This later can be a challenging problem since more or less extensive edits may have been performed on the original copy: insertion or removal of useless chunks of code, rewriting of expressions, transposition of code, inlining and outlining of functions, etc. In this paper, we propose a new similarity detection technique not only based on token sequence matching but also on the factorization of the function call graphs. The factorization process merges shared chunks (factors) of codes to cope, in particular, with inlining and outlining. The resulting call graph offers a view of the similarities with their nesting relations. It is useful to infer metrics quantifying similarity at a function level.

A protocol-independent technique for eliminating redundant network traffic

We present a technique for identifying repetitive information transfers and use it to analyze the redundancy of network traffic. Our insight is that dynamic content, streaming media and other traffic that is not caught by today's Web caches is nonetheless likely to derive from similar information. We have therefore adapted similarity detection techniques to the problem of designing a system to eliminate redundant transfers. We identify repeated byte ranges between packets to avoid retransmitting the redundant data. We find a high level of redundancy and are able to detect repetition that Web proxy caches are not. In our traces, after Web proxy caching has been applied, an additional 39% of the original volume of Web traffic is found to be redundant. Moreover, because our technique makes no assumptions about HTTP protocol syntax or caching semantics, it provides immediate benefits for other types of content, such as streaming media, FTP traffic, news and mail.

A stochastic model of TCP/IP with stationary random losses

We present a technique for identifying repetitive information transfers and use it to analyze the redundancy of network traffic. Our insight is that dynamic content, streaming media and other traffic that is not caught by today's Web caches is nonetheless likely to derive from similar information. We have therefore adapted similarity detection techniques to the problem of designing a system to eliminate redundant transfers. We identify repeated byte ranges between packets to avoid retransmitting the redundant data.