Duplicate Detection Method Research Articles

An image storage duplication detection method using recurrent learning for smart application services

An image storage duplication detection method using recurrent learning for smart application services

Missing values compensation in duplicates detection using hot deck method

Duplicate record is a common problem within data sets especially in huge volume databases. The accuracy of duplicate detection determines the efficiency of duplicate removal process. However, duplicate detection has become more challenging due to the presence of missing values within the records where during the clustering and matching process, missing values can cause records deemed similar to be inserted into the wrong group, hence, leading to undetected duplicates. In this paper, duplicate detection improvement was proposed despite the presence of missing values within a data set through Duplicate Detection within the Incomplete Data set (DDID) method. The missing values were hypothetically added to the key attributes of three data sets under study, using an arbitrary pattern to simulate both complete and incomplete data sets. The results were analyzed, then, the performance of duplicate detection was evaluated by using the Hot Deck method to compensate for the missing values in the key attributes. It was hypothesized that by using Hot Deck, duplicate detection performance would be improved. Furthermore, the DDID performance was compared to an early duplicate detection method namely DuDe, in terms of its accuracy and speed. The findings yielded that even though the data sets were incomplete, DDID was able to offer a better accuracy and faster duplicate detection as compared to DuDe. The results of this study offer insights into constraints of duplicate detection within incomplete data sets.

Detecting Duplicate Questions in Community Based Websites Using Machine Learning

Searching answers to the queries is easy nowadays because of the most popular community based websites like Quora and stack overflow. These websites maintain Q&A database which consists of millions of questions with answers. But the practical problem with these databases is the generation of duplicate questions. Users are generating duplicate questions at a higher in spite of providing clear guidelines for the users to search for the question before posting any new question. Developing an automatic method for duplicate question detection is an on-going research from many years. In this paper we have made an effort to develop such a method using machine learning techniques such as Logistic regression, decision tree, decision tree with baggingrandom forest and Adaboost classifier. We were able to achieve impressive results with 79.21%, 79.29%, 81.70%, 81.72% and 81.73% of classification accuracy from the mentioned machine learning algorithms respectively. The results were compared with the similar study in the state of the art literature. Our results showed good performance in the comparison.

APFA: Automated product feature alignment for duplicate detection

To keep up with the growing interest of using Web shops for product comparison, we have developed a method that targets the problem of product duplicate detection. If duplicates can be discovered correctly and quickly, customers can compare products in an efficient manner. We build upon the state-of-the-art Multi-component Similarity Method (MSM) for product duplicate detection by developing an automated pre-processing phase that occurs before the similarities between products are calculated. Specifically, in this prior phase the features of products are aligned between Web shops, using metrics such as the data type, coverage, and diversity of each key, as well as the distribution and used measurement units of their corresponding values. With this information, the values of these keys can be more meaningfully and efficiently employed in the process of comparing products. Applying our method to a real-world dataset of 1629 TV’s across 4 Web shops, we find that we increase the speed of the product similarity phase by roughly a factor 3 due to fewer meaningless comparisons, an improved brand analyzer, and a renewed title analyzer. Moreover, in terms of quality of duplicate detection, we significantly outperform MSM with an F1-measure of 0.746 versus 0.525.

SiameseQAT: A Semantic Context-Based Duplicate Bug Report Detection Using Replicated Cluster Information

In large-scale software development environments, defect reports are maintained through bug tracking systems (BTS) and analyzed by domain experts. Different users may create bug reports in a non-standard manner and may report a particular problem using a particular set of words due to stylistic choices and writing patterns. Therefore, the same defect can be reported with very different descriptions, generating non-trivial duplicates. To avoid redundant work for the development team, an expert needs to look at all new reports while trying to label possible duplicates. However, this approach is neither trivial nor scalable and directly impacts bug fix correction time. Recent efforts to find duplicate bug reports tend to focus on deep neural approaches that consider hybrid representations of bug reports, using both structured and unstructured information. Unfortunately, these approaches ignore that a single bug can have multiple previously identified duplicates and, therefore, multiple textual descriptions, titles, and categorical information. In this work, we propose SiameseQAT, a duplicate bug report detection method that considers information on individual bugs as well as information extracted from bug clusters. The SiameseQAT combines context and semantic learning on structured and unstructured features and corpus topic extraction-based features, with a novel loss function called Quintet Loss, which considers the centroid of duplicate clusters and their contextual information. We validated our approach on the well-known open-source software repositories Eclipse, NetBeans, and Open Office, comprised of more than 500 thousand bug reports. We evaluated both the retrieval and classification of duplicates, reporting a Recall@25 mean of 85% for retrieval and 84% AUROC for classification tasks, results that were significantly superior to previous works.

Using FDAD to Prevent DAD Attack in SEcure Neighbor Discovery Protocol

The It is very important for the corresponding author to have a linked ORCID (Open Researcher and Contributor ID) account on MTS. To register a linked ORCID account, please go to the Account Update page (http://mts.hindawi.com/update/) in our Manuscript Tracking System and after you have logged in click on the ORCID link at the top of the page. This link will take you to the ORCID website where you will be able to create an account for yourself. Once you have done so, your new ORCID will be saved in our Manuscript Tracking System automatically."?>SEND uses CGA as its address configuration method. CGA binds the IPv6 address with multiple auxiliary parameters, thereby making the dependency relationship between IPv6 address and host provable, which prevents address embezzlement. Owing to the considerable overhead in CGA parameter verification, the malicious host can use this point to carry out DoS attacks. To prevent DoS, the paper proposes a new duplicate address detection method in an SDN environment called FDAD. Two additional mechanisms are added to the FDAD, namely, query and feedback; messages used by the new mechanisms are also designed. Through these two mechanisms, on the one hand, the host can query the MAC address of the suspect host to the controller. On the other hand, if the CGA parameter verification fails, the controller will use feedback information to suppress malicious host from its source port in order to prevent subsequent attacks. Experiments show that the CPU overhead of FDAD is much lower than the normal CGA when suffering Denial of Service attack. The increased CPU consumption and memory overhead of the controller are also within acceptable range, and the network communication overhead is greatly reduced.

Video Forgery Detection Method based on Local Difference Binary

Recently, the rapid development of video editing software has made video forgery applicable. Researchers have proposed methods to detect forged video frames. These methods utilize codec properties, motion artifacts, noise effect and frame similarity to detect forgery. Execution time and low detection accuracy are the two main drawbacks of forgery detection methods reported in the literature. In this study, a new frame duplication detection method using Local Difference Binary (LDB) is proposed to extract features from the frames. Distance between similar frames that have similar feature vectors are is used by the method to estimate Distance of Forgery and to determine the exact location of duplicated frames. PSNR between similar frames are is then used to group them into three classes, and rule-based mechanism reports forged frames according to the membership to classes. Experimental results indicate that the proposed method has lower execution time with higher accuracy than similar works.

Analysis of Progressive Duplicate Data Detection

Duplicate detection is the process of identifying multiple representations of same real world entities. Today, duplicate detection methods need to process ever larger datasets in ever shorter time: maintaining the quality of a dataset becomes increasingly difficult. We present two novel, progressive duplicate detection algorithms that significantly increase the efficiency of finding duplicates if the execution time is limited, They maximize the gain of the overall process within the time available by reporting most results much earlier than traditional approaches. Comprehensive experiments show that our progressive algorithms can double the efficiency over time of traditional duplicate detection and significantly improve upon related work.

Frame duplication detection based on BoW model

Duplicated sequence of frames in a video to cover up or replicate a scene is a video forgery. There are methods to authenticate video files, but embedding authentication information into videos requires extra hardware or software. It is possible to detect frame duplication forgery by carefully inspecting the content to discover high correlation among group of frames. A new frame duplication detection method based on Bag-of-Words (BoW) model is proposed in this paper. BoW is a model used in textual analysis first and image and video retrieval later by researchers. We used BoW to create visual words and build a dictionary from Scale Independent Feature Transform (SIFT) keypoints of frames in video. Frame features, i.e., visual word representations at keypoints, are used to detect sequence of duplicated parts in the video. The method computes thresholds depending on the content to improve both robustness and performance. The proposed method is tested on 31 test videos selected from Surrey University Library for Forensic Analysis (SULFA) and from various movies. Experimental results show a better detection performance and reduced run time compared to similar methods reported in the literature.

Region duplication detection based on image segmentation and keypoint contexts

In this paper, a novel region duplication detection method is proposed based on image segmentation and keypoint contexts. The proposed method includes the primary region duplication detection based on keypoints and the supplementary region duplication detection based on blocks. In the primary region duplication detection, an image is divided into non-overlapped patches by using SLIC. Furthermore, the keypoints are matched and clustered within the same patch as patch feature. Then the patches are matched and an affine transformation is tried to be estimated from a pair of patches. When the estimation fails, in the supplementary region duplication detection, a transformation matrix is tried to be estimated from a pair of keypoints by the proposed Keypoint Contexts (KC) approach. The experimental results indicate that the proposed method can achieve much better comprehensive performances than the state-of-the-art methods on the public databases, even under various challenging conditions.

Benchmarks for measurement of duplicate detection methods in nucleotide databases.

: https://bitbucket.org/biodbqual/benchmarks.

Duplicates, redundancies and inconsistencies in the primary nucleotide databases: a descriptive study.

GenBank, the EMBL European Nucleotide Archive and the DNA DataBank of Japan, known collectively as the International Nucleotide Sequence Database Collaboration or INSDC, are the three most significant nucleotide sequence databases. Their records are derived from laboratory work undertaken by different individuals, by different teams, with a range of technologies and assumptions and over a period of decades. As a consequence, they contain a great many duplicates, redundancies and inconsistencies, but neither the prevalence nor the characteristics of various types of duplicates have been rigorously assessed. Existing duplicate detection methods in bioinformatics only address specific duplicate types, with inconsistent assumptions; and the impact of duplicates in bioinformatics databases has not been carefully assessed, making it difficult to judge the value of such methods. Our goal is to assess the scale, kinds and impact of duplicates in bioinformatics databases, through a retrospective analysis of merged groups in INSDC databases. Our outcomes are threefold: (1) We analyse a benchmark dataset consisting of duplicates manually identified in INSDC—a dataset of 67 888 merged groups with 111 823 duplicate pairs across 21 organisms from INSDC databases – in terms of the prevalence, types and impacts of duplicates. (2) We categorize duplicates at both sequence and annotation level, with supporting quantitative statistics, showing that different organisms have different prevalence of distinct kinds of duplicate. (3) We show that the presence of duplicates has practical impact via a simple case study on duplicates, in terms of GC content and melting temperature. We demonstrate that duplicates not only introduce redundancy, but can lead to inconsistent results for certain tasks. Our findings lead to a better understanding of the problem of duplication in biological databases.Database URL: the merged records are available at https://cloudstor.aarnet.edu.au/plus/index.php/s/Xef2fvsebBEAv9w

Supervised Learning for Detection of Duplicates in Genomic Sequence Databases.

MotivationFirst identified as an issue in 1996, duplication in biological databases introduces redundancy and even leads to inconsistency when contradictory information appears. The amount of data makes purely manual de-duplication impractical, and existing automatic systems cannot detect duplicates as precisely as can experts. Supervised learning has the potential to address such problems by building automatic systems that learn from expert curation to detect duplicates precisely and efficiently. While machine learning is a mature approach in other duplicate detection contexts, it has seen only preliminary application in genomic sequence databases.ResultsWe developed and evaluated a supervised duplicate detection method based on an expert curated dataset of duplicates, containing over one million pairs across five organisms derived from genomic sequence databases. We selected 22 features to represent distinct attributes of the database records, and developed a binary model and a multi-class model. Both models achieve promising performance; under cross-validation, the binary model had over 90% accuracy in each of the five organisms, while the multi-class model maintains high accuracy and is more robust in generalisation. We performed an ablation study to quantify the impact of different sequence record features, finding that features derived from meta-data, sequence identity, and alignment quality impact performance most strongly. The study demonstrates machine learning can be an effective additional tool for de-duplication of genomic sequence databases. All Data are available as described in the supplementary material.

Benchmarks for measuring duplicate detection methods for biological databases

Benchmarks for measuring duplicate detection methods for biological databases

A STUDY AND SURVEY ON VARIOUS PROGRESSIVE DUPLICATE DETECTION MECHANISMS

One of the serious problems faced in several applications with personal details management, customer affiliation management, data mining, etc is duplicate detection. This survey deals with the various duplicate record detection techniques in both small and large datasets. To detect the duplicity with less time of execution and also without disturbing the dataset quality, methods like Progressive Blocking and Progressive Neighborhood are used. Progressive sorted neighborhood method also called as PSNM is used in this model for finding or detecting the duplicate in a parallel approach. Progressive Blocking algorithm works on large datasets where finding duplication requires immense time. These algorithms are used to enhance duplicate detection system. The efficiency can be doubled over the conventional duplicate detection method using this algorithm. Several different methods of data analysis are studied here with various approaches for duplicate detection.

Keypoints-Based Image Passive Forensics Method for Copy-Move Attacks

A novel image passive forensics method for copy-move forgery detection is proposed. The proposed method combines block matching technology and feature point matching technology, and breaks away from the general framework of the visual feature-based approach that used local visual feature such as SIFT and followed by a clustering procedure to group feature points that are spatially close. In our work, image keypoints are extracted using Harris detector, and the statistical features of keypoint neighborhoods are used to generate forensics features. Then we proposed a new forensics features matching approach, in which, a region growth technology and a mismatch checking approach are developed to reduce mismatched keypoints and improve detected accuracy. We also develop a duplicate region detection method based on the distance frequency of corresponding keypoint pairs. The proposed method can detect duplicate regions for high resolution images. It has higher detection accuracy and computation efficiency. Experimental results show that the proposed method is robust for content-preserving manipulations such as JPEG compression, gamma adjustment, filtering, luminance enhancement, blurring, etc.

Identification of approximately duplicate material records in ERP systems

ABSTRACTThe quality of master data is crucial for the accurate functioning of the various modules of an enterprise resource planning (ERP) system. This study addresses specific data problems arising from the generation of approximately duplicate material records in ERP databases. Such problems are mainly due to the firm’s lack of unique and global identifiers for the material records, and to the arbitrary assignment of alternative names for the same material by various users. Traditional duplicate detection methods are ineffective in identifying such approximately duplicate material records because these methods typically rely on string comparisons of each field. To address this problem, a machine learning-based framework is developed to recognise semantic similarity between strings and to further identify and reunify approximately duplicate material records – a process referred to as de-duplication in this article. First, the keywords of the material records are extracted to form vectors of discriminating words. Second, a machine learning method using a probabilistic neural network is applied to determine the semantic similarity between these material records. The approach was evaluated using data from a real case study. The test results indicate that the proposed method outperforms traditional algorithms in identifying approximately duplicate material records.

Progressive Duplicate Detection

Duplicate detection is the process of identifying multiple representations of same real world entities. Today, duplicate detection methods need to process ever larger datasets in ever shorter time: maintaining the quality of a dataset becomes increasingly difficult. We present two novel, progressive duplicate detection algorithms that significantly increase the efficiency of finding duplicates if the execution time is limited: They maximize the gain of the overall process within the time available by reporting most results much earlier than traditional approaches. Comprehensive experiments show that our progressive algorithms can double the efficiency over time of traditional duplicate detection and significantly improve upon related work.

EDDDS: An Efficient Duplicate Data Detection System

Duplicate Detection is critical task of any database of any organization. Duplicates are nothing but the same real time entities or objects are presented in the form of different structure and in the different formats. We can find out the duplicates in relational data, in complex data and hierarchical data like XML. There are lots of works already presented in the past for finding the duplicates in the relational data. But nowadays there is more focus on finding duplicates in the XML data. Because of XML is very popular for data storing and extensively used for data exchange between the organizations. Here we have done an extensive literature survey on this topic and proposed a duplicate detection method that incorporates some of the existing paper's ideas and some of our original ideas. In addition to improving the efficiency and effectiveness, we also checks for its typographical errors when comparing the two XML elements. To test the correctness of our method, we are comparing it with existing duplicate detection system, and giving more focus on how we get higher precision and recall values in the various datasets we have used.

A case study for understanding the nature of redundant entities in bibliographic digital libraries

Purpose – The purpose of this paper is to collect and understand the nature of real cases of author name variants that have often appeared in bibliographic digital libraries (DLs) as a case study of the name authority control problem in DLs. Design/methodology/approach – To find a sample of name variants across DLs (e.g. DBLP and ACM) and in a single DL (e.g. ACM), the approach is based on two bipartite matching algorithms: Maximum Weighted Bipartite Matching and Maximum Cardinality Bipartite Matching. Findings – First, the authors validated the effectiveness and efficiency of the bipartite matching algorithms. The authors also studied the nature of real cases of author name variants that had been found across DLs (e.g. ACM, CiteSeer and DBLP) and in a single DL. Originality/value – To the best of the authors knowledge, there is less research effort to understand the nature of author name variants shown in DLs. A thorough analysis can help focus research effort on real problems that arise when the authors perform duplicate detection methods.