Duplicate Detection Research Articles

Optimization of PDCCH blind detection method in LTE-A system

In LTE-A system, physical downlink control channel (PDCCH) plays a key role in system scheduling. In order to overcome the high complexity of the conventional blind detection method which leads to the low efficiency of the terminal receiving PDCCH, the improvement is made on the basis of the conventional blind detection method. Using the channel indication information reported by UE terminal to get the current channel condition, and the optimal blind detection level is selected according to the current channel condition, and the blind detection range can be narrowed. When the downlink control information is successfully detected, mark its location, Same PDCCH candidate can be avoided duplicate detection, and optimized the condition of blind multiple DCI detection. By simulation, the improved me effectively reduces the number of times required for blind detection, and improves the efficiency of blind detection.

Region duplication detection based on hybrid feature and evaluative clustering

Digital image are easy to be tampered by the photo editing software. Therefore, digital image forensics which aims at validating the authenticity of the digital image are received wide public concern. Region duplication is a commonly used operation in digital image forgeries. The main aims of the region duplication are to overemphasize or conceal some contents by duplicating some regions on the image. Most of the region duplication methods can be categorized into two main classes:block-based and keypoint-based methods. In this paper, a novel region duplication detection scheme is proposed based on hybrid feature and evaluative clustering. The proposed scheme is divided into two stages: the rough matching and the exact matching. In the rough matching, first, hybrid keypoints are extracted from the input image, and those keypoints are described by the unified descriptors. Second, those keypoints are matched by the g2NN strategy. Third, those matched keypoints are grouped by the proposed clustering based on evaluation. Fourth, affine transformations are estimated between these groups, and Bag of Word is used to filter inaccuracy affine transformations to improve the results of pixel level. When no affine transformation is obtained, in the exact matching, each suspicious region is handled separately. Experimental results indicate that the proposed scheme outperforms the state-of-the-art methods under various conditions.

Efficient hash function–based duplication detection algorithm for data Deduplication deduction and reduction

SummaryData Deduplication is the foremost technique used for data compression by removing redundant data. It is broadly utilized for cloud storage server, which reduces the un‐utilized memory space and optimized bandwidth. The confidentiality of sensitive information maintains during Deduplication detection. The Efficient Hash Function–based Duplication Detection (EHFDD) algorithm is proposed to encrypt the content before outsourcing in cloud environment. This paper is an attempt to enhance the data security and notify the issues of approved data Deduplication detections. Furthermore, it also presents several novel Deduplication improvisations for helping certified duplicate verification in hybrid cloud. The EHFDD method enables duplicate detection scheme with minimal overhead compared to existing functions. Based on experimental evaluations, proposed EHFDD algorithm reduces the average delay in 28.7 milliseconds, memory utilization in 8%, computation time in 457 milliseconds, communications overhead in 900 milliseconds, and improves the success rate in 3.13%.

An enhanced secure content de-duplication identification and prevention (ESCDIP) algorithm in cloud environment

In cloud computing, de-duplication plays an essential role in detecting the de-duplication of encoded data with minimal computation and cost. De-duplication cleans the cloud datacentre’s unwanted storage and helps to identify the right owner of the content in the cloud. Even if there is only one copy of each data file stored in the cloud, the cloud has huge quantity of cloud users who own such data file. The existing method discussed a convergent encryption technique to solve the de-duplication problem. It also developed a system which does not allow storing any duplicate data in the cloud. However, the method does not assure consistency, reliability and confidentiality in cloud. Similar or different cloud users could store duplicated file in the cloud server, where cloud storage utilises high volume of storage. To find a solution for the above problems, the paper introduces enhanced secure content de-duplication identification and prevention (ESCDIP) algorithm to enhance the file-level and content-level de-duplication detection of encoded data with reliability in cloud environment. Every cloud user’s files contain an independent master key for encryption using ESCDIP technique and outsourcing them into the cloud. It reduces the overheads that are associated with the interactive duplication detection and query processes. The proposed method identifies the unique data chunking to store in the cloud. Based on experimental result, the ESCDIP method reduces 2.3 data uploading time in seconds, 2.31 data downloading time in seconds and 32.66% communication cost compared to existing approaches.

Quality Management of Workers in an In-House Crowdsourcing-Based Framework for Deduplication of Organizations’ Databases

While organizations in the current era of big data are generating massive volumes of data, they also need to ensure that its quality is maintained for it to be useful in decision-making purposes. The problem of dirty data plagues every organization. One aspect of dirty data is the presence of duplicate data records that negatively impact the organization's operations in many ways. Many existing approaches attempt to address this problem by using traditional data cleansing methods. In this paper, we address this problem by using an in-house crowdsourcing-based framework, namely, DedupCrowd. One of the main obstacles of crowdsourcing-based approaches is to monitor the performance of the crowd, by which the integrity of the whole process is maintained. In this paper, a statistical quality control-based technique is proposed to regulate the performance of the crowd. We apply our proposed framework in the context of a contact center, where the Customer Service Representatives are used as the crowd to assist in the process of deduplicating detection. By using comprehensive working examples, we show how the different modules of the DedupCrowd work not only to monitor the performance of the crowd but also to assist in duplicate detection.

Image Retrieval using Visual Phrases

Keypoint based descriptors are widely used for various computer vision applications. During this process, key-points are initially detected from the given images which are later represented by some robust and distinctive descriptors like scale-invariant feature transform (SIFT). Keypoint based image-to-image matching has gained significant accuracy for image retrieval type of applications like image copy detection, similar image retrieval and near duplicate detection. Local keypoint descriptors are quantized into visual words to reduce the feature space which makes image-to-image matching possible for large scale applications. Bag of visual word quantization makes it efficient at the cost of accuracy. In this paper, the bag of visual word model is extended to detect frequent pair of visual words which is known as frequent item-set in text processing, also called visual phrases. Visual phrases increase the accuracy of image retrieval without increasing the vocabulary size. Experiments are carried out on benchmark datasets that depict the effectiveness of proposed scheme.

Автоматизация обнаружения и анализа ошибок в гиперконвергентных системах

The paper is devoted to the problem of early error detection and analysis in hyperconverged systems. One approach to organizing hyperconverged systems is to install on each physical server a separate instance of an operating system (OS) that carries virtualization tools and tools for administering and using a distributed data warehouse. Errors can occur both at the level of a single OS instance and at the level of the entire cluster. For example, incorrect control element commands from one infrastructure node can cause software failure on another node. In addition, errors from the subsystems of the cluster can provoke abnormal situations inside virtual machines. The complexity of the architecture of hyperconverged systems makes it difficult to analyze the errors that occur in them. To simplify such an analysis and increase its effectiveness, it is necessary to automate the process of detecting problems and collecting data necessary for their study and correction. Existing approaches for automation of error detection are described and various improvements are suggested to adopt them for systems where distributed storage and virtualization technologies are actively used. Improvements include log collection from the whole cluster just after the error occurred, additional analysis of guest operating system behaviour inside virtual machines, usage of a knowledge base for automated crash recovery and duplicate detection. Finally, a real-life scenario of error handling process in Virtuozzo company products is described starting from error detection and ending with fix deployment.

A Novel Region Duplication Detection Algorithm Based on Hybrid Approach

The digital images from various sources are ubiquitous due to easy availability of high bandwidth Internet. Digital images are easy to tamper with good or bad intentions. Non-availability of pre-embedded information in digital images makes the tampering detection process more difficult in case of digital forensics. Thus, passive image tampering is difficult to detect. There are various algorithms available for detecting image tampering. However, these algorithms have some drawbacks, due to which all types of tampering cannot be detected. In this paper researchers intend to present the types of image tampering and its detection techniques with example based approach. This paper also illustrates insights into the various existing algorithms and tries to find out efficient algorithm out of them.

A data cleaning method for heterogeneous attribute fusion and record linkage

In big data era, massive heterogeneous data are generated from various data sources, the cleaning of dirty data is critical for reliable data analysis. Existing rule-based methods are generally developed in single data source environment, issues like data standardisation and duplication detection for different data type attributes, are not fully studied. In order to address these challenges, we introduce a method based on dynamic configurable rules which can integrate data detection, modification and transformation together. Secondly, we propose a type-based blocking and a varying window size selection mechanism based on classic sorted-neighbourhood algorithm. We present a reference implementation of our method in a real-life data fusion system and validate its effectiveness and efficiency using recall and precision metrics. Experimental results indicate that our method is suitable in the scenario of multiple data sources with heterogeneous attribute properties.

Comparison of Image Duplication Detection Using the Polar Coordinates System and Histogram of Oriented Gradients Methods

Comparison of Image Duplication Detection Using the Polar Coordinates System and Histogram of Oriented Gradients Methods

A Method For Detecting Duplicate And Near-Duplicate Images Penetration

A method for detection of duplicate or near-duplicate image penetration from images in the similar group by distribution of color and other attributes of the image. Distinctive sceneries of the images penetration are identified. Each couple of images penetration with at least one distictive scenary is mutual; the distictive scenary of each image penetration is allied to normalize whether the couple is duplicates or near-duplicates.

B Chromosomes of the Asian Seabass (Lates calcarifer) Contribute to Genome Variations at the Level of Individuals and Populations.

The Asian seabass (Lates calcarifer) is a bony fish from the Latidae family, which is widely distributed in the tropical Indo-West Pacific region. The karyotype of the Asian seabass contains 24 pairs of A chromosomes and a variable number of AT- and GC-rich B chromosomes (Bchrs or Bs). Dot-like shaped and nucleolus-associated AT-rich Bs were microdissected and sequenced earlier. Here we analyzed DNA fragments from Bs to determine their repeat and gene contents using the Asian seabass genome as a reference. Fragments of 75 genes, including an 18S rRNA gene, were found in the Bs; repeats represented 2% of the Bchr assembly. The 18S rDNA of the standard genome and Bs were similar and enriched with fragments of transposable elements. A higher nuclei DNA content in the male gonad and somatic tissue, compared to the female gonad, was demonstrated by flow cytometry. This variation in DNA content could be associated with the intra-individual variation in the number of Bs. A comparison between the copy number variation among the B-related fragments from whole genome resequencing data of Asian seabass individuals identified similar profiles between those from the South-East Asian/Philippines and Indian region but not the Australian ones. Our results suggest that Bs might cause variations in the genome among the individuals and populations of Asian seabass. A personalized copy number approach for segmental duplication detection offers a suitable tool for population-level analysis across specimens with low coverage genome sequencing.

A Filter of Minhash for Image Similarity Measures

Image similarity measures play an important role in nearest neighbor search and duplicate detection for large-scale image datasets. Recently, Minwise Hashing (or Minhash) and its related hashing algorithms have achieved great performances in large-scale image retrieval systems. However, there are a large number of comparisons for image pairs in these applications, which may spend a lot of computation time and affect the performance. In order to quickly obtain the pairwise images that theirs similarities are higher than the specific thresholdT(e.g., 0.5), we propose a dynamic threshold filter of Minwise Hashing for image similarity measures. It greatly reduces the calculation time by terminating the unnecessary comparisons in advance. We also find that the filter can be extended to other hashing algorithms, on when the estimator satisfies the binomial distribution, such as b-Bit Minwise Hashing, One Permutation Hashing, etc. In this pager, we use the Bag-of-Visual-Words (BoVW) model based on the Scale Invariant Feature Transform (SIFT) to represent the image features. We have proved that the filter is correct and effective through the experiment on real image datasets.

Region duplication detection in digital images based on Centroid Linkage Clustering of key–points and graph similarity matching

Region duplication or copy–move forgery is an attack in which a region of an image is copied and pasted onto another location of the same image. In the recent state–of–the–art, a number of key–point based methods have been proposed for copy–move forgery detection in digital images. Though the problems of re–scaling and rotation in region duplication, have been sufficiently investigated using key–point based methods, post-processing based attacks such as flip, blur, brightness and noise, remain an open challenge in this field. In this paper, we address the problem of copy–move forgery detection in images, plus aim to identify copied regions, having undergone different geometric (such as rotation, re–scale), and post–processing attacks (such as Gaussian noise, blurring and brightness adjustment). In the proposed algorithm we introduce a region based key–point selection concept, which is considerably more discriminative than single SIFT key–point extraction. In this work, we apply Centroid Linkage Clustering, to identify duplicated regions in an image, from matched key-points. Also, we introduce a Graph Similarity Matching algorithm, to optimize false matches. Our experimental results demonstrate the efficiency of the proposed method in terms of forgery detection and localization efficiency, for a wide range of geometric and post-processing based attacks in region duplication.

Deciphering evolutionary dynamics of SWEET genes in diverse plant lineages

SWEET/MtN3/saliva genes are prevalent in cellular organisms and play diverse roles in plants. These genes are widely considered as evolutionarily conserved genes, which is inconsistent with their extensive expansion and functional diversity. In this study, SWEET genes were identified from 31 representative plant species, and exhibited remarkable expansion and diversification ranging from aquatic to land plants. Duplication detection indicated that the sharp increase in the number of SWEET genes in higher plants was largely due to tandem and segmental duplication, under purifying selection. In addition, phylogeny reconstruction of SWEET genes was performed using the maximum-likelihood (ML) method; the genes were grouped into four clades, and further classified into 10 monocot and 11 dicot subfamilies. Furthermore, selection pressure of SWEET genes in different subfamilies was investigated via different strategies (classical and Bayesian maximum likelihood (Datamonkey/PAML)). The average dN/dS for each group were lower than one, indicating purifying selection. Individual positive selection sites were detected within 4 of the 21 sub-families by both two methods, including two monocot subfamilies in Clade III, harboring five rice SWEET homologs characterized to confer resistance to rice bacterial blight disease. Finally, we traced evolutionary fate of SWEET genes in clade III for functional characterization in future.

Deep Constrained Siamese Hash Coding Network and Load-Balanced Locality-Sensitive Hashing for Near Duplicate Image Detection.

We construct a new efficient near duplicate image detection method using a hierarchical hash code learning neural network and load-balanced locality-sensitive hashing (LSH) indexing. We propose a deep constrained siamese hash coding neural network combined with deep feature learning. Our neural network is able to extract effective features for near duplicate image detection. The extracted features are used to construct a LSH-based index. We propose a load-balanced LSH method to produce load-balanced buckets in the hashing process. The load-balanced LSH significantly reduces the query time. Based on the proposed load-balanced LSH, we design an effective and feasible algorithm for near duplicate image detection. Extensive experiments on three benchmark data sets demonstrate the effectiveness of our deep siamese hash encoding network and load-balanced LSH.

Interactive feature selection for efficient customer recognition in contact centers: Dealing with common names

We propose an interactive decision-making framework to assist a Customer Service Representative (CSR) in the efficient and effective recognition of customer records in a database with many ambiguous entries. Our proposed framework consists of three integrated modules. The first module focuses on the detection and resolution of duplicate records to improve effectiveness and efficiency in customer recognition. The second module determines the level of ambiguity in recognizing an individual customer when there are multiple records with the same name. The third module recommends the series of feature-related questions that the CSR should ask the customer to enable rapid recognition, based on that level of ambiguity. In the first module, the F-Swoosh approach for duplicate detection is used, and in the second module a dynamic programming-based technique is used to determine the level of ambiguity within the customer database for a given name. In the third module, Levenshtein edit distance is used for feature selection in combination with weights based on the Inverse Document Frequency (IDF) of terms. The algorithm that requires the minimum number of questions to be put to the customer to achieve recognition is the algorithm that is chosen. We evaluate the proposed framework on a synthetic dataset and demonstrate how it assists the CSR to rapidly recognize the correct customer.

A Hybrid Intelligent Algorithm Based on the Method to Realize the Vehicle Tracking System

Aiming at the problem in accident vehicle tracking system and the flaws of low detection precision and slow speed in the traditional algorithm designed a vehicle that based on hybrid intelligent algorithm. First getting moving objects by using frame differential method and picking up accurate target image by using differential background image, then abstracting the light stream of the target and marking object the area of interested to overcome the problem of empty, duplicate detection and illumination influence, reduce the time complexity to improve the reliability of target tracking and reduce the test time. According to the result of the experiment, compared with the traditional algorithms, the system can quickly and accurately track the vehicle, with strong validity and practicality.

Credibility Analysis for Online Product Reviews

With the prevalence of e-commerce, online product reviews are increasingly considered crowd-sourced consumer opinions that significantly influence customer purchasing decisions and product rankings. It is therefore important to ensure the truthfulness of reviews by detecting and filtering out fake/spam reviews. This article presents an effective framework to analyze review credibility for spam detection and opinion mining. It incorporates three methods: duplicated review detection, anomaly detection, and incentivized review detection, that complement each other to produce statistical credibility scores indicating review credibility. A practical end-to-end system is designed and developed accordingly, and is equipped with high-level data visualization for easy interpretation and summarization of the analysis results. Experiments on an Amazon review dataset demonstrate its efficiency, scalability and accuracy. This system could help e-commerce and consumers identify fake reviews, refine product rankings, and constrain vendors and spammers from engaging in dishonest practices.

Hierarchical one permutation hashing: efficient multimedia near duplicate detection

With advances in multimedia technologies and the proliferation of smart phone, digital cameras, storage devices, there are a rapidly growing massive amount of multimedia data collected in many applications such as multimedia retrieval and management system, in which the data element is composed of text, image, video and audio. Consequently, the study of multimedia near duplicate detection has attracted significant concern from research organizations and commercial communities. Traditional solution minwish hashing (\minwise) faces two challenges: expensive preprocessing time and lower comparison speed. Thus, this work first introduce a hashing method called one permutation hashing (\oph) to shun the costly preprocessing time. Based on \oph, a more efficient strategy group based one permutation hashing (\goph) is developed to deal with the high comparison time. Based on the fact that the similarity of most multimedia data is not very high, this work design an new hashing method namely hierarchical one permutation hashing (\hoph) to further improve the performance. Comprehensive experiments on real multimedia datasets clearly show that with similar accuracy \hoph is five to seven times faster than