Duplicate Detection Research Articles

A novel pairwise based convolutional neural network for image preprocessing enhancement

Wildfires are untamable and devastating forces that impact both urban and rural regions. While predicting wildfires is challenging, efforts are made to mitigate the damage they inflict. The previous researches have limitations such as not being able to find a small region of fire in the dataset. In this research, pairwise region-based convolutional neural network (PR-CNN) is proposed for wildfire detection. The dataset used for wildfire detection is the fire luminosity airborne-based machine learning evaluation (FLAME) dataset that is pre-processed through normalization and hue, saturation, and lightness (HSV) color space to improve the image quality. Pre-processed images are taken as input to region-based convolutional neural network (R-CNN) for detection, the R-CNN has a region proposal layer that is enhanced by pairwise region and named PR-CNN. These wrapped images are fed into CNN architecture to extract and features to detect wildfire. Additionally, post processing technique like soft-non-maximum suppression (NMS) is utilized to eliminate the duplicate detection from PR-CNN for enhancing the detection accuracy. The proposed method achieves a higher accuracy of 97.44%, a precision of 97.32%, recall of 97.31%, and f1-score of 96.67%, which is comparatively superior to the existing algorithms like recurrent neural network (RNN), and R-CNN.

Optimized Fault Classification in Electric Vehicle Drive Motors Using Advanced Machine Learning and Data Transformation Techniques

The increasing use of electric vehicles has made fault diagnosis in electric drive motors, particularly in variable speed drives (VSDs) using three-phase induction motors, a critical area of research. This article presents a fault classification model based on machine learning (ML) algorithms to identify various faults under six operating conditions: normal operating mode (NOM), phase-to-phase fault (PTPF), phase-to-ground fault (PTGF), overloading fault (OLF), over-voltage fault (OVF), and under-voltage fault (UVF). A dataset simulating real-world operating conditions, consisting of 39,034 instances and nine key motor features, was analyzed. Comprehensive data preprocessing steps, including missing value removal, duplicate detection, and data transformation, were applied to enhance the dataset’s suitability for ML models. Yeo–Johnson and Hyperbolic Sine transformations were used to reduce skewness and improve the normality of the features. Multiple ML algorithms, including CatBoost, Random Forest (RF) Classifier, AdaBoost, and quadratic discriminant analysis (QDA), were trained and evaluated using Bayesian optimization with cross-validation. The CatBoost model achieved the best performance, with an accuracy of 94.1%, making it the most suitable model for fault classification in electric vehicle drive motors.

Two–Stage Detection and Localization of Inter–Frame Tampering in Surveillance Videos Using Texture and Optical Flow

Surveillance cameras provide security and protection through real-time monitoring or through the investigation of recorded videos. The authenticity of surveillance videos cannot be taken for granted, but tampering detection is challenging. Existing techniques face significant limitations, including restricted applicability, poor generalizability, and high computational complexity. This paper presents a robust detection system to meet the challenges of frame duplication (FD) and frame insertion (FI) detection in surveillance videos. The system leverages the alterations in texture patterns and optical flow between consecutive frames and works in two stages; first, suspicious tampered videos are detected using motion residual–based local binary patterns (MR–LBPs) and SVM; second, by eliminating false positives, the precise tampering location is determined using the consistency in the aggregation of optical flow and the variance in MR–LBPs. The system is extensively evaluated on a large COMSATS Structured Video Tampering Evaluation Dataset (CSVTED) comprising challenging videos with varying quality of tampering and complexity levels and cross–validated on benchmark public domain datasets. The system exhibits outstanding performance, achieving 99.5% accuracy in detecting and pinpointing tampered regions. It ensures the generalization and wide applicability of the system while maintaining computational efficiency.

IPv6 addressing strategy with improved secure duplicate address detection to overcome denial of service and reconnaissance attacks

With technology development, the growing self-communicating devices in IoT networks require specific naming and identification, mainly provided by IPv6 addresses. The IPv6 address in the IoT network is generated by using the stateless auto address configuration (SLAAC) mechanism, and its uniqueness is ensured by the DAD protocol. Recent research suggests that IPv6 deployment can be a risky decision due to the existing SLAAC-based addressing scheme and the DAD protocol being prone to reconnaissance and denial of service (DoS) attacks. This research paper proposes a new IPv6 generation scheme with an improved secure DAD mechanism to address these problems. The proposed addressing scheme generates IPv6 addresses by taking a hybrid approach based on vendor id of medium access control (MAC) address, physical location, and arbitrary random numbers, which mitigates reconnaissance attacks by malicious nodes. To prevent the DAD process from DoS attacks, hybrid values of interface identifier (IID) are multicast instead of actual values. The proposed scheme is evaluated under reconnaissance and DoS attacks in the presence of malicious nodes. The evaluation results demonstrate that the proposed method effectively mitigates reconnaissance and DoS attacks, outperforming the EUI-64 and SEUI-64 schemes in terms of address success rate (ASR), energy consumption, and communication overhead. Specifically, the proposed method significantly reduces the average probing rate for scanning the existence of an IPv6 address, with only a 1% probing rate compared to SEUI-64’s 5% and EUI-64’s 100%. Furthermore, the additional communication overhead introduced by the proposed method is less than 13% and 11% compared to EUI-64 and SEUI-64, respectively. Additionally, the energy consumption required to assign an IPv6 address using the proposed method is lower by 12% and 5% when compared to EUI-64 and SEUI-64, respectively. These findings highlight the effectiveness of the proposed method in enhancing security and optimizing resource utilization in IPv6 addressing.

Food and fluid intake during labor in maternity wards: A scoping review protocol.

During the 1940s, aspiration of stomach contents was recognized as a serious problem during labor, which is why fluid and food restriction was adopted for women who would undergo general anesthesia. Currently, the practice of generalized food restriction is a routine that is being discontinued. This review aims to map the evidence on the effects of the intake of foods, supplements and drinks in women on labor outcomes. To that end, documents investigating this topic in pregnant women admitted for uncomplicated deliveries in maternity wards, published from 2013 onwards, will be assessed. This interval was defined based on the publication by Singata et al., who carried out a systematic review on the benefits and harms of oral fluid intake or food restriction during labor. The scoping review methods of the JBI and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Review (PRISMA) were followed. Firstly, a preliminary search was carried out to identify the existence of similar scoping reviews or protocols, as well as the keywords and MeSH descriptors in the titles and abstracts, with a view to developing a complete search strategy. Subsequently, a search will be carried out in the Cochrane Library, Medline/PubMed, Embase, SCOPUS and Web of Science databases. The search strategy will be adapted for each of these databases. Finally, a reverse search will be carried out using the references of the included studies. The obtained documents will be imported into Rayyan for duplicate detection and removal. Two independent reviewers will read the titles and abstracts, observing the inclusion and exclusion criteria. The data extraction from each included study will be carried out independently by two reviewers, using the extraction form created for this purpose. In order to report results, we will follow the PRISMA checklist and report descriptive statistics and a narrative summary.

Big data classification using SpinalNet-Fuzzy-ResNeXt based on spark architecture with data mining approach

In the modern networking topology, big data is highly essential for several domains like e-commerce, healthcare, and finance. Big data classification has offered effectual performance in several applications. Still, big data classification is highly difficult and the recognized classification approaches require a longer duration and numerous resources for executing the accessible data. For resolving such issues, the spark-based classification approach is required. In this work, the hybrid SpinalNet-Fuzzy-ResNeXt model called SFResNeXt is implemented to classify the big data. Here, the SpinalNet and ResNeXt are merged, where the layers are fused with the fuzzy concept. The initial process is the outlier detection. The Holoentrophy method is used to detect the outlier data, and it is removed. Moreover, duplicate detection is performed by fingerprinting approach to detect the repeated data. The, Association Rule Mining (ARM) method is employed for feature selection. The big data is classified by the SFResNeXt. Furthermore, the SFResNeXt-based big data classification offered the accuracy, sensitivity, and specificity of 0.905, 0.914, and 0.922 using the heart disease dataset.

From SSDs Back to HDDs: Optimizing VDO to Support Inline Deduplication and Compression for HDDs as Primary Storage Media

Deduplication and compression are powerful techniques to reduce the ratio between the quantity of logical data stored and the physical amount of consumed storage. Deduplication can impose significant performance overheads, as duplicate detection for large systems induces random accesses to the backend storage. These random accesses have led to the concern that deduplication for primary storage and HDDs are not compatible. Most inline data reduction solutions are therefore optimized for SSDs and discourage their use for HDDs, even for sequential workloads. In this work, we show that these concerns are valid if and only if the lessons learned from deduplication research are not applied. We have therefore investigated data reduction solutions for primary storage based on the RedHat Virtual Disk Optimizer (VDO) and show that directly applying them can decrease sequential write performance for HDDs by 36×. We then show that slight modifications to VDO plus the integration of a very small SSD area significantly improve performance even beyond the performance without data reduction enabled, making HDDs more cost-efficient for a wide range of mostly sequential cloud workloads than SSDs. Additionally, these VDO optimizations do not require to maintain different code bases for HDDs and SSDs, and we therefore provide the first data reduction solution applicable to both storage media.

Duplicate image detection using deep learning modified SVM and k-NN classification method for multimedia application

Duplicate image detection using deep learning modified SVM and k-NN classification method for multimedia application

Clonal Dynamics and Relapse Risk Revealed by High-Sensitivity FLT3-Internal Tandem Duplication Detection in Acute Myeloid Leukemia

The ability to detect low-level disease is key to our understanding of clonal heterogeneity in acute myeloid leukemia (AML) and residual disease that elude conventional assays and seed relapse. We developed a high-sensitivity next-generation sequencing (HS-NGS) clinical assay, able to reliably detect low levels (1 × 10−5) of FLT3-ITD, a frequent, therapeutically targetable and prognostically relevant mutation in AML. By applying this assay to 289 longitudinal samples from 62 patients at initial diagnosis and/or clinical follow-up (mean follow-up of 22 months), we reveal the frequent occurrence of FLT3-ITD subclones at diagnosis and demonstrate a significantly decreased relapse risk when FLT3-ITD is cleared after induction or thereafter. We perform pairwise sequencing of diagnosis and relapse samples from 23 patients to uncover more detailed patterns of FLT3-ITD clonal evolution at relapse than is detectable by less-sensitive assays. Finally, we show that rising ITD level during consecutive biopsies is a harbinger of impending relapse. Our findings corroborate the emerging clinical utility of high-sensitivity FLT3-ITD testing and expands our understanding of clonal dynamics in FLT3-ITD–positive AML.

Proposed threshold-based and rule-based approaches to detecting duplicates in bibliographic database

Bibliographic databases are used to measure the performance of researchers, universities and research institutions. Thus, high data quality is required and data duplication is avoided. One of the weaknesses of the threshold-based approach in duplication detection is the low accuracy level. Therefore, another approach is required to improve duplication detection. This study proposes a method that combines threshold-based and rule-based approaches to perform duplication detection. These two approaches are implemented in the comparison stage. The cosine similarity function is used to create weight vectors from the features. Then, the comparison operator is used to determine whether the pair of records are grouped as duplication or not. Three research databases: Web of Science (WoS), Scopus, and Google Scholar (GS) on the Science and Technology Index (SINTA) database are investigated. Rule 4 and Rule 5 provide the best performance. For WoS dataset, the accuracy, precision, recall, and F1-measure values were 100.00%. For Scopus dataset, the accuracy and precision values were 100.00%, recall: 98.00%, and the F1-measure value is 98.00%. For GS dataset, the accuracy value was 100.00%, precision: 99.00%, recall: 97.00%, and the F1-measure value is 98.00%. The proposed method is potential tool for accurate detection on duplication records in publication databases.

Deduplication Methods Using Levenshtein Distance Algorithm

The study aimed to propose methods to improve the data integrity of the Relational databases such as MS SQL, MySQL and PostgreSQL via record duplication detection. The FODORS and ZAGAT Restaurant database benchmark datasets have been utilized to facilitate the processes involved in preparing and delivering high-quality data. Furthermore, the Levenshtein distance algorithm was used to propose three (3) methods namely: default, eliminating equal string, and knowledge-based libraries to cut duplicate records in the database. In the 70% selected threshold, the average detected duplicate records of 88 out of 112 records in the restaurant dataset. Finally, to efficiently detect duplicate records in the database, depend on the data being analyzed and threshold selected.

IPv6 addressing with hidden duplicate address detection to mitigate denial of service attacks in the internet of drone

SummaryThe fusion of drones with the Internet, termed the Internet of Drones (IoD), resembles the Internet of Things (IoT) paradigm. In IoD, drones necessitate unique identifiers similar to IPv6 addresses. This paper presents a novel approach for securing the duplicate address detection (DAD) process in the IoD environment. This paper focuses on addressing the security challenges posed by malicious drones in the DAD process. The proposed solution introduces a hidden DAD (H‐DAD) process, to mitigate attacks on the DAD process. In this scheme, drones transmit a transformed portion of their interface identifier (IID) while concealing the full IID, thus thwarting potential disruptions by malicious entities. We evaluated the effectiveness of the H‐DAD scheme through experimental analysis using Python with SimPy, comparing it with the Improved DAD, Secure DAD, and Standard DAD schemes. Our results demonstrate a notable improvement, with the Hidden DAD scheme achieving a 100% address success rate under attack. Moreover, it exhibits approximately 5% and 12% lower overhead and energy consumption, respectively, compared to the other schemes.

Hybrid deduplication system with content-based cache for cloud environment

Primary storage deduplication systems are performance sensitive. Their performance depends upon two factors — metadata access for duplicate detection and strategy for elimination of duplicate data. Various approaches for duplicate detection through suitable caching mechanisms have been proposed in the literature. Most of the approaches assumed that the primary workloads exhibit strong temporal locality. Whereas, this cannot be assumed in the context of Cloud as the workloads locality does not exist with interferences among different workloads on the same system. Duplicate content among the data blocks with different addresses lead to an inefficient utilization of the data cache. In this context, applying deduplication causes sharing of the data blocks among the clients with different access patterns and frequencies. In this situation, LRU cache, which considers only the recency of the references, is not appropriate. In this paper, Hybrid Deduplication System (HDS) containing the content-based cache with a new replacement policy — Modified Adaptive Replacement Cache (ARC), is proposed. The proposed system is simulated in the Linux environment using three different types of FIU traces. Effectiveness of the system is compared with a full deduplication system. Experimental results show that the system has performed consistently better than the full deduplication system in reducing the metadata overhead for all of the three data sets.

Duplicate Detection on Cloud Data with Re-Encryption Based Data Sharing

Cloud data storage has gained immense popularity in recent years due to its scalability and accessibility. However, the efficient management of data in the cloud, especially with the ever-increasing volume of data, remains a significant challenge. Data deduplication is a crucial technique to reduce storage costs and enhance data management efficiency. It also concerns the secure cloud data storage and sharing. Here implement a novel approach to de duplication-based cloud data storage using chunk-based partitioning. In this approach, data is divided into fixed-size chunks, and a deduplication mechanism is applied to identify and eliminate redundant chunks. The process of duplicate detection leading to improved storage efficiency and reduced data redundancy. It not only conserves storage space but also enhances data retrieval performance by reducing the volume of data to be transferred over the network. Proposed system also enhance the security of cloud data storage by employing Base64 encoding for data representation and implementing re-encryption techniques based on the Advanced Encryption Standard (AES) for secure data sharing. The Base64 encoding methodology is employed to transform binary data into a text format, rendering it less susceptible to unintended manipulations during data transmission and storage. This encoding mechanism ensures data integrity and facilitates its secure storage in cloud environments. By utilizing AES encryption, data shared among users is re-encrypted with unique keys, ensuring that only authorized recipients can access and decrypt the shared information. The proposed approach significantly enhances data confidentiality and mitigates risks associated with unauthorized access, there by fortifying the security of cloud-based data sharing.

Navigating duplication in pharmacovigilance databases: a scoping review

ObjectivesPharmacovigilance databases play a critical role in monitoring drug safety. The duplication of reports in pharmacovigilance databases, however, undermines their data integrity. This scoping review sought to provide a comprehensive...

Multi-attribute, graph-based approach for duplicate cattle removal and counting in large pasture areas from multiple aerial images

In this study, we address the challenges of counting cattle in large pasture areas using Unmanned Aerial Vehicles (UAVs) equipped with high-resolution cameras. Traditional manual counting methods are laborious and error-prone, while existing automated approaches struggle with duplicate animal detection. To overcome these limitations, we propose a novel graph-based method that incorporates multiple-attributes, including velocity, direction, state (lying down or standing), color, and distance, to improve duplicate removal and counting accuracy. We conducted extensive experiments involving automated hyper-parameter learning to effectively integrate these attributes into our method. By employing a Ford–Fulkerson graph algorithm, we detect and remove duplicated cattle based on their multiple attributes. An ablation study validates the contribution of each attribute. Additionally, we provide new datasets of drone images captured in large pastures to support research in this field. Our results demonstrate that our proposed method outperforms state-of-the-art techniques, achieving an average percentage error of 2.34%. Comparisons with mosaic-based and other graph-based methods validate the effectiveness of our approach, which contributes to more efficient cattle counting practices and enhances livestock management in agriculture.

Connected Components for Scaling Partial-order Blocking to Billion Entities

In entity resolution, blocking pre-partitions data for further processing by more expensive methods. Two entity mentions are in the same block if they share identical or related blocking-keys . Previous work has sometimes related blocking keys by grouping or alphabetically sorting them, but—as was shown for author disambiguation—the respective equivalences or total orders are not necessarily well-suited to model the logical matching-relation between blocking keys. To address this, we present a novel blocking approach that exploits the subset partial order over entity representations to build a matching-based bipartite graph, using connected components as blocks. To prevent over- and underconnectedness, we allow specification of overly general and generalization of overly specific representations. To build the bipartite graph, we contribute a new parallellized algorithm with configurable time/space tradeoff for minimal element search in the subset partial order. As a job-based approach, it combines dynamic scalability and easier integration to make it more convenient than the previously described approaches. Experiments on large gold standards for publication records, author mentions, and affiliation strings suggest that our approach is competitive in performance and allows better addressing of domain-specific problems. For duplicate detection and author disambiguation, our method offers the expected performance as defined by the vector-similarity baseline used in another work on the same dataset and the common surname, first-initial baseline. For top-level institution resolution, we have reproduced the challenges described in prior work, strengthening the conclusion that for affiliation data, overlapping blocks under minimal elements are more suitable than connected components.

Duplicate Records in WorldCat for 20th-Century American, British, and Canadian Books: A Comparison of Duplication Rates and Causes

The bibliographic record duplication rates in WorldCat for 20th-century books cataloged in English and published in New York, London, and Montreal were estimated by sampling records created by Concordia University. Duplicate sets were identified according to OCLC WorldCat record merging guidelines. New York and London records had similar duplication rates, higher than those for Montreal. Changing descriptive cataloging standards, brief cataloging, and both typographical and MARC coding errors caused failures in duplicate record detection. Records for editions, reproductions, fine arts titles, and conference publications were particularly problematic. The earlier cataloging standards were reviewed to uncover how differences have caused duplicates.

Search for duplicates of showers in the IAU MDC database

Context. The meteor shower database of the IAU Meteor Data Center (MDC) is used by the whole community of meteor astronomers. Observers submit both new and known meteor shower parameters to the MDC. Two types of problems may arise during the submission process: If a new observation of an already-known meteor shower is submitted as the discovery of a new shower, a duplicate shower will appear in the MDC. If the submission of a new set of parameters for an existing shower is incorrect, a false duplicate of a known meteor shower will appear in the MDC. The MDC database contains such duplicates and false duplicates, so it is desirable to detect them among the streams already in the database and those delivered to the database as new streams. Aims. We aim to develop a method for objective detection of duplicates among meteor showers and to apply it to the MDC. The method will also enable us to verify whether various sets of parameters of the same shower are compatible and thus reveal the false duplicates. Methods. We suggest two methods based on cluster analyses and two similarity functions among geocentric and heliocentric shower parameters collected in the MDC. Results. We obtained a number of results of varying significance. Seven new showers represented by two or more parameter sets were discovered, revealing the duplicates we searched for. We found full agreement between our results and those reported in the MDC database for 30 showers. The multiple sets of parameters defining these showers are correct since they were identified as duplicates. For 20 showers, the same duplicates as given in the MDC were found only by one method. We found 27 showers for which the number of parameter sets found by both methods is close to the corresponding number in the MDC database. However, we found 56 showers listed in the MDC by more than one set of parameters for which no duplicates were found by either of the applied methods. These showers have false duplicates among their sets of parameters. Conclusions. The obtained results confirm the effectiveness of the proposed approach of identifying duplicates. We have shown that in order to detect and verify duplicate meteor showers, it is possible to apply the objective proposal instead of the subjective approach used so far. We consider the identification of 83 problematic cases in the MDC database, among which at least some duplicates were misclassified, to be a particularly important result. The correction of these cases will significantly improve the content of the MDC database.

Perbandingan Kinerja Sistem Operasi Network Attached Storage: Studi Kasus Truenas dan Xigmanas

Network Attached Storage (NAS) is a data storage device that operates through a computer network and provides an efficient and cost-effective storage solution due to its open-source foundation. This research compares the performance of TrueNAS and XigmaNAS as NAS operating systems in terms of data transfer speed and resource utilization over a period of 7 days. The tested parameters include file copy, file classification, file duplicate detection, CPU usage, and memory usage. The research results indicate that the average file transfer speed from server to client in TrueNAS is 7.4 Mbps, while in XigmaNAS it is 7.36 Mbps. The average file classification rate in TrueNAS is 863 files/s, whereas in XigmaNAS it is 813 files/s. The average speed of duplicate file detection in TrueNAS is 1.23 files/s, while in XigmaNAS it is 1.13 files/s. The average CPU usage in TrueNAS is 30.08%, compared to 29.38% in XigmaNAS. The average available memory for TrueNAS is 6.13 GiB, and for XigmaNAS it is 5.41 GiB. Despite the small variations and differences observed in the test results, overall, TrueNAS outperforms XigmaNAS in the analyzed aspects.