Duplicate Detection Algorithms Research Articles

More extreme duplication in FDA Adverse Event Reporting System detected by literature reference normalization and fuzzy string matching.

Literature reports of adverse drug events can be replicated across multiple companies, resulting in extreme duplication (defined as a majority of reports being duplicates) in the FDA Adverse Event Reporting System (FAERS) database because they can escape legacy duplicate detection algorithms routinely deployed on that data source. Literature reference field, added to in 2014, could potentially be utilized to identify replicated reports. FAERS does not enforce adherence to the Vancouver referencing convention, thus the same article may be referenced differently leading to duplication. The objective of this analysis is to determine if variations of the same literature references observed in FAERS can be resolved with text normalization and fuzzy string matching. We normalized the literature references recorded in the FAERS database through the first quarter of 2021 with a rule-based algorithm so that they better conform to the Vancouver convention. Levenshtein distance was then utilized to merge sufficiently similar normalized literature references together. Normalization of literature references increases the percentage that can be parsed into author, title, and journal from 61.74% to 93.93%. We observe that about 98% of pairs within groups do have a Levenshtein similarity of the title above the threshold. The extreme duplication ranged from 66% to 87% with a median of 72% of reports being duplicates and often involved addictovigilance scenarios. We have shown that these normalized references can be merged via fuzzy string matching to improve enumeration of all the individual case safety reports that refer to the same article. Inclusion of the PubMed ID and adherence to the Vancouver convention could facilitate identification of duplicates in the FAERS dataset. Awareness of this phenomenon may improve disproportionality analysis, especially in areas such as addictovigilance.

An algorithm for detecting RFID duplicates

The problem of using duplicate RFID tags by attackers is becoming more and more actual with the expansion of RFID technology for marking imported goods. The duplicate may contain information about goods, which differs from their actual characteristics. This paper proposes an algorithm for detecting RFID duplicates as a method for achieving the integrity of information that enters the information systems of international goods transportation. The relevance of creating an algorithm deals with the need to reduce the risk of creating and using RFID duplicates by importers during the cross-border movement of marked goods. Existing duplicate detection algorithms are unsuitable for use in the RFID-marking system of goods imported into Russia. The algorithm hinders an attacker from reading data from the original RFID tag, which is necessary to create an RFID duplicate. The proposed algorithm is based on dividing the EPC memory area of an RFID tag into parts and using the tag self-destruction command (kill) to prevent unauthorized readings. The authors considered the scenarios for implementing the algorithm and identified the risks of using the algorithm. The algorithm is presented as a graphical model based on BPMN notation. The efficiency of the proposed algorithm was evaluated using the hypergeometric probability formula. The results of a selective check of RFID tags by the customs authorities were taken as the initial data. It is shown that, in comparison with the existing approach, the implementation of the algorithm in software and hardware complex increases the probability of detecting RFID duplicates, provided that control is carried out only in relation to high-risk declarants. The use of the algorithm reduces the risk of receiving distorted or inaccurate data in the information systems dealing with international goods transportation and increases the validity of legal and economic decisions taken in the information systems of customs authorities.

An Experimental Simulation of Addressing Auto-Configuration Issues for Wireless Sensor Networks

Applications of Wireless Sensor devices are widely used by various monitoring sections such as environmental monitoring, industrial sensing, habitat modeling, healthcare and enemy movement detection systems. Researchers were found that 16 bytes packet size (payload) requires Media Access Control (MAC) and globally unique network addresses overheads as more as the payload itself which is not reasonable in most situations. The approach of using a unique address isn't preferable for most Wireless Sensor Networks (WSNs) applications as well. Based on the mentioned drawbacks, the current work aims to fill the existing gap in the field area by providing two strategies. First, name/address solutions that assign unique addresses locally to clustered topology-based sensor devices, reutilized in a spatial manner, and reduce name/address size by a noticeable amount of 2.9 based on conducted simulation test. Second, name/address solutions that assign reutilizing of names/addresses to location-unaware spanning-tree topology in an event-driven WSNs case (that is providing minimal low latencies and delivering addressing packet in an efficient manner). Also, to decline the approach of needing both addresses (MAC and network) separately, it discloses how in a spatial manner to reutilize locally unique sensor device name approach and could be utilized in both contexts and providing an energy-efficient protocol for location unawareness clustered based WSNs. In comparison, an experimental simulation test performed and given the addresses solution with less overhead in the header and 62 percent fair payload efficiency that outperforms 34 percent less effective globally unique addresses. Furthermore, the proposed work provides addresses uniqueness for network-level without using network-wide Duplicate Address Detection (DAD) algorithm. Consequently, the current study provides a roadmap for addressing/naming scheme to help researchers in this field of study. In general, some assumptions were taken during the work phases of this study such as number of Cluster Head (CH) nodes is 6% of entire sensor nodes, location unawareness for entire sensor network and 4 bits per node address space which considered as the limitation of the study.

Frame Duplication Forgery Detection and Localization Algorithm Based on the Improved Levenshtein Distance

In this digital era of technology and software development tools, low-cost digital cameras and powerful video editing software (such as Adobe Premiere, Microsoft Movie Maker, and Magix Vegas) have become available for any common user. Through these softwares, editing the contents of digital videos became very easy. Frame duplication is a common video forgery attack which can be done by copying and pasting a sequence of frames within the same video in order to hide or replicate some events from the video. Many algorithms have been proposed in the literature to detect such forgeries from the video sequences through analyzing the spatial and temporal correlations. However, most of them are suffering from low efficiency and accuracy rates and high computational complexity. In this paper, we are proposing an efficient and robust frame duplication detection algorithm to detect duplicated frames from the video sequence based on the improved Levenshtein distance. Extensive experiments were performed on some selected video sequences captured by stationary and moving cameras. In the experimental results, the proposed algorithm showed efficacy compared with the state-of-the-art techniques.

Security vulnerability and encryption technology of computer information technology data under big data environment

In recent years, the traditional encryption storage technology and management methods have been difficult to satisfy the demand of big data in terms of speed, capacity, storage efficiency and security, along with the rapid increase of data volume. Data security is facing great challenges and challenges. To solve this problem, this paper proposes a de-redundancy method based big data encryption algorithm. Firstly, bloom filter technology is used to decrease the dimension of big data. Based on its remarkable advantages, bloom filter space and query efficiency and variable-length block detection scheme with less additional overhead, duplicate data detection and deletion algorithm suitable for large data is designed and implemented. Secondly, based on eliminating redundancy of big data, combined with the characteristics of the elliptic curve encryption algorithm and AES block cipher mode in terms of computing speed, parallelism and security, an encryption approach based on data de-redundancy technology is presented. According to the evaluation system of the encryption algorithm, the encryption time, keyspace and security of the new algorithm are evaluated. Experimental results reveal that the scheme has better security, and the presented approach can increase the speed of the encryption process, and decrease the encryption time and consumption.

Large-Scale Data Pollution with Apache Spark

Because of the increasing volume of autonomously collected data objects, duplicate detection is an important challenge in today's data management. To evaluate the efficiency of duplicate detection algorithms with respect to big data, large test data sets are required. Existing test data generation tools, however, are either not able to produce large test data sets or are domain-dependent which limits their usefulness to a few cases. In this paper, we describe a new framework that can be used to pollute a clean, homogeneous and large data set from an arbitrary domain with duplicates, errors and inhomogeneities. To prove its concept, we implemented a prototype which is built upon the cluster computing framework Apache Spark and evaluate its performance in several experiments.

Copy-Move Forgery Detection Using Scale Invariant Feature and Reduced Local Binary Pattern Histogram

Because digitized images are easily replicated or manipulated, copy-move forgery techniques are rendered possible with minimal expertise. Furthermore, it is difficult to verify the authenticity of images. Therefore, numerous efforts have been made to detect copy-move forgeries. In this paper, we present an improved region duplication detection algorithm based on the keypoints. The proposed algorithm utilizes the scale invariant feature transform (SIFT) and the reduced local binary pattern (LBP) histogram. The LBP values with 256 levels are obtained from the local window centered at the keypoint, which are then reduced to 10 levels. For a keypoint, a 138-dimensional is generated to detect copy-move forgery. We test the proposed algorithm on various image datasets and compare the detection accuracy with those of existing methods. The experimental results demonstrate that the performance of the proposed scheme is superior to that of other tested copy-move forgery detection methods. Furthermore, the proposed method exhibits a uniform detection performance for various types of test datasets.

Analysis of Progressive Duplicate Data Detection

Duplicate detection is the process of identifying multiple representations of same real world entities. Today, duplicate detection methods need to process ever larger datasets in ever shorter time: maintaining the quality of a dataset becomes increasingly difficult. We present two novel, progressive duplicate detection algorithms that significantly increase the efficiency of finding duplicates if the execution time is limited, They maximize the gain of the overall process within the time available by reporting most results much earlier than traditional approaches. Comprehensive experiments show that our progressive algorithms can double the efficiency over time of traditional duplicate detection and significantly improve upon related work.

Transforming Pairwise Duplicates to Entity Clusters for High-quality Duplicate Detection

Duplicate detection algorithms produce clusters of database records, each cluster representing a single real-world entity. As most of these algorithms use pairwise comparisons, the resulting (transitive) clusters can be inconsistent: Not all records within a cluster are sufficiently similar to be classified as duplicate. Thus, one of many subsequent clustering algorithms can further improve the result. We explain in detail, compare, and evaluate many of these algorithms and introduce three new clustering algorithms in the specific context of duplicate detection. Two of our three new algorithms use the structure of the input graph to create consistent clusters. Our third algorithm, and many other clustering algorithms, focus on the edge weights, instead. For evaluation, in contrast to related work, we experiment on true real-world datasets, and in addition examine in great detail various pair-selection strategies used in practice. While no overall winner emerges, we are able to identify best approaches for different situations. In scenarios with larger clusters, our proposed algorithm, Extended Maximum Clique Clustering (EMCC), and Markov Clustering show the best results. EMCC especially outperforms Markov Clustering regarding the precision of the results and additionally has the advantage that it can also be used in scenarios where edge weights are not available.

Efficient hash function–based duplication detection algorithm for data Deduplication deduction and reduction

SummaryData Deduplication is the foremost technique used for data compression by removing redundant data. It is broadly utilized for cloud storage server, which reduces the un‐utilized memory space and optimized bandwidth. The confidentiality of sensitive information maintains during Deduplication detection. The Efficient Hash Function–based Duplication Detection (EHFDD) algorithm is proposed to encrypt the content before outsourcing in cloud environment. This paper is an attempt to enhance the data security and notify the issues of approved data Deduplication detections. Furthermore, it also presents several novel Deduplication improvisations for helping certified duplicate verification in hybrid cloud. The EHFDD method enables duplicate detection scheme with minimal overhead compared to existing functions. Based on experimental evaluations, proposed EHFDD algorithm reduces the average delay in 28.7 milliseconds, memory utilization in 8%, computation time in 457 milliseconds, communications overhead in 900 milliseconds, and improves the success rate in 3.13%.

A Novel Region Duplication Detection Algorithm Based on Hybrid Approach

The digital images from various sources are ubiquitous due to easy availability of high bandwidth Internet. Digital images are easy to tamper with good or bad intentions. Non-availability of pre-embedded information in digital images makes the tampering detection process more difficult in case of digital forensics. Thus, passive image tampering is difficult to detect. There are various algorithms available for detecting image tampering. However, these algorithms have some drawbacks, due to which all types of tampering cannot be detected. In this paper researchers intend to present the types of image tampering and its detection techniques with example based approach. This paper also illustrates insights into the various existing algorithms and tries to find out efficient algorithm out of them.

A NOVEL TWO-PHASE PAGE FEATURE AND KTH KEYPHRASE FINGERPRINT BASED DUPLICATE DETECTION TECHNIQUE

The World Wide Web is a huge repository of network-accessible information including text, image, audio, video and metadata. With rapid increase in information resources available via WWW and users of the Internet, it is becoming difficult to manage and access the desired information on the web. Therefore, majority of users use information retrieval tools like search engines to find the desired information from the WWW. Web search engines work by storing information about many web pages, which they retrieve from the WWW itself.. Many of the pages stored in search engine repository are duplicates and near duplicates of other pages. These duplicate and near duplicate web pages require more space for storage, which increase the cost of serving results and also frustrates the users. To help search engines provide quality and redundant free, distinct results duplicate and near duplicate detection algorithms are used. The proposed duplicate detection approach detects near duplicate web pages efficiently and quickly thereby improving search effectiveness and storage efficiency of search engine.

Authentication of Surveillance Videos: Detecting Frame Duplication Based on Residual Frame.

Nowadays, surveillance systems are used to control crimes. Therefore, the authenticity of digital video increases the accuracy of deciding to admit the digital video as legal evidence or not. Inter-frame duplication forgery is the most common type of video forgery methods. However, many existing methods have been proposed for detecting this type of forgery and these methods require high computational time and impractical. In this study, we propose an efficient inter-frame duplication detection algorithm based on standard deviation of residual frames. Standard deviation of residual frame is applied to select some frames and ignore others, which represent a static scene. Then, the entropy of discrete cosine transform coefficients is calculated for each selected residual frame to represent its discriminating feature. Duplicated frames are then detected exactly using subsequence feature analysis. The experimental results demonstrated that the proposed method is effective to identify inter-frame duplication forgery with localization and acceptable running time.

Robust Region Duplication Detection on Log-Polar Domain Using Band Limitation

Region duplication is one of the image-tampering techniques in which a part of image is copied and pasted into another region of the same image. In this paper, a robust duplication detection algorithm is proposed against severe degradations such as illumination changes, blurring, large scaling, noise contamination and JPEG compression. We introduce an adaptive phase correlation scheme in the log-polar domain, which is effective to locate the most salient local feature of an image patch on frequency band. By using the information collected from the band limitation, duplicated regions can be correctly located. Our contribution is to present a robust image duplication detection algorithm which can handle large scaling manipulation while preserving detection performance under other manipulations or degradations. We perform degradation and comparison test on various tampered images, and experimental results show that the proposed algorithm achieves satisfactory performance.

The Efficacy of Epidemic Algorithms on Detecting Node Replicas in Wireless Sensor Networks

A node replication attack against a wireless sensor network involves surreptitious efforts by an adversary to insert duplicate sensor nodes into the network while avoiding detection. Due to the lack of tamper-resistant hardware and the low cost of sensor nodes, launching replication attacks takes little effort to carry out. Naturally, detecting these replica nodes is a very important task and has been studied extensively. In this paper, we propose a novel distributed, randomized sensor duplicate detection algorithm called Discard to detect node replicas in group-deployed wireless sensor networks. Our protocol is an epidemic, self-organizing duplicate detection scheme, which exhibits emergent properties. Epidemic schemes have found diverse applications in distributed computing: load balancing, topology management, audio and video streaming, computing aggregate functions, failure detection, network and resource monitoring, to name a few. To the best of our knowledge, our algorithm is the first attempt at exploring the potential of this paradigm to detect replicas in a wireless sensor network. Through analysis and simulation, we show that our scheme achieves robust replica detection with substantially lower communication, computational and storage requirements than prior schemes in the literature.

Progressive Duplicate Detection

Duplicate detection is the process of identifying multiple representations of same real world entities. Today, duplicate detection methods need to process ever larger datasets in ever shorter time: maintaining the quality of a dataset becomes increasingly difficult. We present two novel, progressive duplicate detection algorithms that significantly increase the efficiency of finding duplicates if the execution time is limited: They maximize the gain of the overall process within the time available by reporting most results much earlier than traditional approaches. Comprehensive experiments show that our progressive algorithms can double the efficiency over time of traditional duplicate detection and significantly improve upon related work.

XNDDF: Towards a Framework for Flexible Near-Duplicate Document Detection Using Supervised and Unsupervised Learning

Abstract The WWW has witnessed the exponential growth of web documents. People of all walks of life depend on the electronic superhighway, Internet, for retrieving information. Search engines retrieve data. Detecting near duplicate documents and handling them can help search engines to improve performance. In this paper, we proposed two algorithms. The first algorithm is meant for unsupervised probabilistic clustering of documents while the second algorithm is to detect near duplicates that can handle in offline processing of search engines. The clustered documents can avoid unnecessary comparisons while near duplicate detection algorithm involve local feature selection in are given document based on weights assigned to terms. A classifier is built to have supervised learning for discriminating documents. We proposed a framework named eXtensible Near Duplicate Detection Framework (XNDDF) which provides various components that provide room for flexible duplicate detection solutions besides showing offline and online processing required by a search engine. Our future work is to implement the framework components through a prototype application.

Streaming quotient filter

The unparalleled growth and popularity of the Internet coupled with the advent of diverse modern applications such as search engines, on-line transactions, climate warning systems, etc., has catered to an unprecedented expanse in the volume of data stored world-wide. Efficient storage, management, and processing of such massively exponential amount of data has emerged as a central theme of research in this direction. Detection and removal of redundancies and duplicates in real-time from such multi-trillion record-set to bolster resource and compute efficiency constitutes a challenging area of study. The infeasibility of storing the entire data from potentially unbounded data streams, with the need for precise elimination of duplicates calls for intelligent approximate duplicate detection algorithms. The literature hosts numerous works based on the well-known probabilistic bitmap structure, Bloom Filter and its variants.In this paper we propose a novel data structure, Streaming Quotient Filter, (SQF) for efficient detection and removal of duplicates in data streams. SQF intelligently stores the signatures of elements arriving on a data stream, and along with an eviction policy provides near zero false positive and false negative rates. We show that the near optimal performance of SQF is achieved with a very low memory requirement, making it ideal for real-time memory-efficient de-duplication applications having an extremely low false positive and false negative tolerance rates. We present detailed theoretical analysis of the working of SQF, providing a guarantee on its performance. Empirically, we compare SQF to alternate methods and show that the proposed method is superior in terms of memory and accuracy compared to the existing solutions. We also discuss Dynamic SQF for evolving streams and the parallel implementation of SQF.

Robust Duplicate Detection of 2D and 3D Objects

In this paper, the authors analyze their graph-based approach for 2D and 3D object duplicate detection in still images. A graph model is used to represent the 3D spatial information of the object based on the features extracted from training images to avoid explicit and complex 3D object modeling. Therefore, improved performance can be achieved in comparison to existing methods in terms of both robustness and computational complexity. Different limitations of this approach are analyzed by evaluating performance with respect to the number of training images and calculation of optimal parameters in a number of applications. Furthermore, effectiveness of object duplicate detection algorithm is measured over different object classes. The authors’ method is shown to be robust in detecting the same objects even when images with objects are taken from different viewpoints or distances.

An Introduction to Duplicate Detection

With the ever increasing volume of data, data quality problems abound. Multiple, yet different representations of the same real-world objects in data, duplicates, are one of the most intriguing data quality problems. The effects of such duplicates are detrimental; for instance, bank customers can obtain duplicate identities, inventory levels are monitored incorrectly, catalogs are mailed multiple times to the same household, etc. Automatically detecting duplicates is difficult: First, duplicate representations are usually not identical but slightly differ in their values. Second, in principle all pairs of records should be compared, which is infeasible for large volumes of data. This lecture examines closely the two main components to overcome these difficulties: (i) Similarity measures are used to automatically identify duplicates when comparing two records. Well-chosen similarity measures improve the effectiveness of duplicate detection. (ii) Algorithms are developed to perform on very large volumes of data in search for duplicates. Well-designed algorithms improve the efficiency of duplicate detection. Finally, we discuss methods to evaluate the success of duplicate detection. Table of Contents: Data Cleansing: Introduction and Motivation / Problem Definition / Similarity Functions / Duplicate Detection Algorithms / Evaluating Detection Success / Conclusion and Outlook / Bibliography