Record Pairs Research Articles

Efficient set containment join

In this paper, we study the problem of set containment join. Given two collections \(\mathcal {R}\) and \(\mathcal {S}\) of records, the set containment join \(\mathcal {R} \bowtie _\subseteq \mathcal {S}\) retrieves all record pairs \(\{(r,s)\} \in \mathcal {R}\times \mathcal {S}\) such that \(r \subseteq s\). This problem has been extensively studied in the literature and has many important applications in commercial and scientific fields. Recent research focuses on the in-memory set containment join algorithms, and several techniques have been developed following intersection-oriented or union-oriented computing paradigms. Nevertheless, we observe that two computing paradigms have their limits due to the nature of the intersection and union operators. Particularly, intersection-oriented method relies on the intersection of the relevant inverted lists built on the elements of \(\mathcal {S}\). A nice property of the intersection-oriented method is that the join computation is verification free. However, the number of records explored during the join process may be large because there are multiple replicas for each record in \(\mathcal {S}\). On the other hand, the union-oriented method generates a signature for each record in \(\mathcal {R}\) and the candidate pairs are obtained by the union of the inverted lists of the relevant signatures. The candidate size of the union-oriented method is usually small because each record contributes only one replica in the index. Unfortunately, union-oriented method needs to verify the candidate pairs, which may be cost expensive especially when the join result size is large. As a matter of fact, the state-of-the-art union-oriented solution is not competitive compared to the intersection-oriented ones. In this paper, we propose a new union-oriented method, namely TT-Join, which not only enhances the advantage of the previous union-oriented methods but also integrates the goodness of intersection-oriented methods by imposing a variant of prefix tree structure. We conduct extensive experiments on 20 real-life datasets and synthetic datasets by comparing our method with 7 existing methods. The experiment results demonstrate that TT-Join significantly outperforms the existing algorithms on most of the datasets and can achieve up to two orders of magnitude speedup. Furthermore, to support large scale of datasets, we extend our techniques to distributed systems on top of MapReduce framework. With the help of careful designed load-aware distribution mechanisms, our distributed join algorithm can achieve up to an order of magnitude speedup than the baselines methods.

Improved scalability in mining using ontology record linkage algorithm

Record linkage offers wide role in record identification and relevant datasets matching. The conventional researchers use probabilistic approach to identify reliable and unique datasets. Record linkage with probabilistic approach exploits data, which are common to an individual record pair. Classical methods have equality based record linkage in common fields. Therefore, errors associated with record linkage reduce the scalability. In this paper, a similarity between individual values of record pairs is improved using ontology-based semantic similarity model. Semantic similarity between the records is tested successfully using angle based neighborhood graph. To validate the proposed approach, a conventional record linkage algorithm is compared with angle based neighborhood ontology record linkage technique, which achieves improved accuracy and scalability. Finally, the accuracy of identifying similar semantic matches is more scalable in proposed technique than conventional methods.

Permutation Jaccard Distance-Based Hierarchical Clustering to Estimate EEG Network Density Modifications in MCI Subjects.

In this paper, a novel electroencephalographic (EEG)-based method is introduced for the quantification of brain-electrical connectivity changes over a longitudinal evaluation of mild cognitive impaired (MCI) subjects. In the proposed method, a dissimilarity matrix is constructed by estimating the coupling strength between every pair of EEG signals, Hierarchical clustering is then applied to group the related electrodes according to the dissimilarity estimated on pairs of EEG recordings. Subsequently, the connectivity density of the electrodes network is calculated. The technique was tested over two different coupling strength descriptors: wavelet coherence (WC) and permutation Jaccard distance (PJD), a novel metric of coupling strength between time series introduced in this paper. Twenty-five MCI patients were enrolled within a follow-up program that consisted of two successive evaluations, at time T0 and at time T1, three months later. At T1, four subjects were diagnosed to have converted to Alzheimer's Disease (AD). When applying the PJD-based method, the converted patients exhibited a significantly increased PJD (p < 0.05), i.e., a reduced overall coupling strength, specifically in delta and θ bands and in the overall range (0.5-32 Hz). In addition, in contrast to stable MCI patients, converted patients exhibited a network density reduction in every subband (delta, θ, alpha, and beta). When WC was used as coupling strength descriptor, the method resulted in a less sensitive and specific outcome. The proposed method, mixing nonlinear analysis to a machine learning approach, appears to provide an objective evaluation of the connectivity density modifications associated to the MCI-AD conversion, just processing noninvasive EEG signals.

FEDERAL: A Framework for Distance-Aware Privacy-Preserving Record Linkage

In privacy-preserving record linkage, a number of data custodians encode their records and submit them to a trusted third-party who is responsible for identifying those records that refer to the same real-world entity. In this paper, we propose FEDERAL, a novel record linkage framework that implements methods for anonymizing both string and numerical data values, which are typically present in data records. These methods rely on a strong theoretical foundation for rigorously specifying the dimensionality of the anonymization space, into which the original values are embedded, to provide accuracy and privacy guarantees under various models of privacy attacks. A key component of the applied embedding process is the threshold that is required by the distance computations, which we prove can be formally specified to guarantee accurate results. We evaluate our framework using three real-world data sets with varying characteristics. Our experimental findings show that FEDERAL offers a complete and effective solution for accurately identifying matching anonymized record pairs (with recall rates constantly above 93 percent) in large-scale privacy-preserving record linkage tasks.

Duplicates Detection Within Incomplete Data Sets Using Blocking and Dynamic Sorting Key Methods

In database records duplicate detection, blocking method is commonly used to reduce the number of comparisons between the candidate record pairs. The main procedure in this method requires selecting attributes that will be used as sorting keys. Selection accuracy is essential in clustering candidates records that are likely matched in the same block. Nevertheless, the presence of missing values affects the creation of sorting keys and this is particularly undesirable if it involves the attributes that are used as the sorting keys. This is because, consequently, records that are supposed to be included in the duplicate detection procedure will be excluded from being examined. Thus, in this paper, we propose a method that can deal with the impact of missing values by using a dynamic sorting key. Dynamic sorting is an extension of blocking method that essentially works on two functions namely uniqueness calculation function (UF) (to choose unique attributes) and completeness function (CF) (to search for missing values). We experimented a particular blocking method called as sorted neighborhood with a dynamic sorting key on a restaurant data set (that consists of duplicate records) obtained from earlier research in order to evaluate the method’s accuracy and speed. Hypothetical missing values were applied to testing data set used in the experiment, where we compare the results of duplicate detection with (and without) dynamic sorting key. The result shows that, even though missing values are present, there is a promising improvement in the partitioning of duplicate records in the same block.

Multiple data linkage using SSCCT for direct and semantic matching pair

Data Linking is a method of integrating multiple data items located on different sources and establishing links among entities of the same type or semantically relevant type. It is necessary to develop the data linkage techniques for different and semantically related items for interconnecting multiple data sources. In this paper, multiple data linkage is used to establish relation between matching entities of different types that are semantically related. The proposed method used Semantically Similar Class Clustering Tree (SSCCT) for implementing multiple data linkage. The SSCCT is built in such a way that it is easy to understand and can be transformed into association rules which are verified using WordNet ontology. The data source properties are represented as tree and the inner node, which consists of features from the first data set. The leaves of the tree represent features from the second data set that is matching with the first data set entities. The proposed method used semantic similarity estimation for pre-pruning process which is used to create Semantically Similar Class Clustering Tree effectively. Threshold value is used for decision making either the record pair is match or non-match.

Record linkage and deduplication using traditional blocking

Record Linkage and Deduplication are the two process that are used in matching records. Matching of records is done to remove the duplicate records. These duplicate records highly influence the outputs of data mining and data processing. If the matching of records is done on the single database, it is called Deduplication. In Deduplication we check for the duplicate records in the single database. Unlike deduplication if the matching of the records is done on the several databases it is called as record linkage. In this paper we also discuss about the indexing technique called as traditional blocking which is used to remove non matching pairs that leads to the less number of record pair to be compared.

A new computationally efficient algorithm for record linkage with field dependency and missing data imputation

Record linkage algorithms aim to identify pairs of records that correspond to the same individual from two or more datasets. In general, fields that are common to both datasets are compared to determine which record-pairs to link. The classic model for probabilistic linkage was proposed by Fellegi and Sunter and assumes that individual fields common to both datasets are completely observed, and that the field agreement indicators are conditionally independent within the subsets of record pairs corresponding to the same and differing individuals. Herein, we propose a novel record linkage algorithm that is independent of these two baseline assumptions. We demonstrate improved performance of the algorithm in the presence of missing data and correlation patterns between the agreement indicators. The algorithm is computationally efficient and can be used to link large databases consisting of millions of record pairs. An R-package, corlink, has been developed to implement the new algorithm and can be downloaded from the CRAN repository.

A Probabilistic Record Linkage Model for Survival Data

ABSTRACTIn the absence of a unique identifier, combining information from multiple files relies on partially identifying variables (e.g., gender, initials). With a record linkage procedure, these variables are used to distinguish record pairs that belong together (matches) from record pairs that do not belong together (nonmatches). Generally, the combined strength of the partially identifying variables is too low causing imperfect linkage; some true nonmatches are identified as match and, on the other hand, some true matches as nonmatch. To avoid bias in further analyses, it is necessary to correct for imperfect linkage. In this article, pregnancy data from the Perinatal Registry of the Netherlands were used to estimate the associations between the (baseline) characteristics from the first delivery and the time to a second delivery. Because of privacy regulations, no unique identifier was available to determine which pregnancies belonged to the same woman. To deal with imperfect linkage in a time-to-event setting, where we have a file with baseline characteristics and a file with event times, we developed a joint model in which the record linkage procedure and the time-to-event analysis are performed simultaneously. R code and example data are available as online supplemental material.

Biometric verification by cross-correlation analysis of 12-lead ECG patterns: Ranking of the most reliable peripheral and chest leads

BackgroundElectrocardiogram (ECG)-based biometrics relies on the most stable and unique beat patterns, i.e. those with maximal intra-subject and minimal inter-subject waveform differences seen from different leads. We investigated methodology to evaluate those differences, aiming to rank the most prominent single and multi-lead ECG sets for biometric verification across a large population. MethodsA clinical standard 12-lead resting ECG database, including 460 pairs of remote recordings (distanced 1year apart) was used. Inter-subject beat waveform differences were studied by cross-correlation and amplitude relations of average PQRST (500ms) and QRS (100ms) patterns, using 8 features/lead in 12-leads. Biometric verification models based on stepwise linear discriminant classifier were trained on the first half of records. True verification rate (TVR) on the remaining test data was further reported as a common mean of the correctly verified equal subjects (true acceptance rate) and correctly rejected different subjects (true rejection rate). Results and conclusionsIn single-lead ECG human identity applications, we found maximal TVR (87–89%) for the frontal plane leads (I, −aVR, II) within (0–60°) sector. Other leads were ranked: inferior (85%), lateral to septal (82–81%), with intermittent V3 drop (77.6%), suggesting anatomical landmark displacements. ECG pattern view from multi-lead sets improved TVR: chest (91.3%), limb (94.6%), 12-leads (96.3%).

Private Blocking Technique for Multi-party Privacy-Preserving Record Linkage

The process of matching and integrating records that relate to the same entity from one or more datasets is known as record linkage, and it has become an increasingly important subject in many application areas, including business, government and health system. The data from these areas often contain sensitive information. To prevent privacy breaches, ideally records should be linked in a private way such that no information other than the matching result is leaked in the process, and this technique is called privacy-preserving record linkage (PPRL). With the increasing data, scalability becomes the main challenge of PPRL, and many private blocking techniques have been developed for PPRL. They are aimed at reducing the number of record pairs to be compared in the matching process by removing obvious non-matching pairs without compromising privacy. However, most of them are designed for two databases and they vary widely in their ability to balance competing goals of accuracy, efficiency and security. In this paper, we propose a novel private blocking approach for PPRL based on dynamic k-anonymous blocking and Paillier cryptosystem which can be applied on two or multiple databases. In dynamic k-anonymous blocking, our approach dynamically generates blocks satisfying k-anonymity and more accurate values to represent the blocks with varying k. We also propose a novel similarity measure method which performs on the numerical attributes and combines with Paillier cryptosystem to measure the similarity of two or more blocks in security, which provides strong privacy guarantees that none information reveals even collusion. Experiments conducted on a public dataset of voter registration records validate that our approach is scalable to large databases and keeps a high quality of blocking. We compare our method with other techniques and demonstrate the increases in security and accuracy.

Evoked potentials in final epoch of self-initiated hand movement: A study in patients with depth electrodes

Comparison between the intended and performed motor action can be expected to occur in the final epoch of a voluntary movement. In search for electrophysiological correlates of this mental process the purpose of the current study was to identify intracerebral sites activated in final epoch of self-paced voluntary movement. Intracerebral EEG was recorded from 235 brain regions of 42 epileptic patients who performed self-paced voluntary movement task. Evoked potentials starting at 0 to 243ms after the peak of averaged, rectified electromyogram were identified in 21 regions of 13 subjects. The mean amplitude value of these late movement potentials (LMP) was 56.4±27.5μV. LMPs were observed in remote regions of mesiotemporal structures, cingulate, frontal, temporal, parietal, and occipital cortices. Closely before the LMP onset, a significant increase of phase synchronization was observed in all EEG record pairs in 9 of 10 examined subjects; p<0.001, Mann-Whitney U test. In conclusion, mesiotemporal structures, cingulate, frontal, temporal, parietal, and occipital cortices seem to represent integral functionally linked parts of network activated in final epoch of self-paced voluntary movement. Activation of this large-scale neuronal network was suggested to reflect a comparison process between the intended and actually performed motor action. Our results contribute to better understanding of neural mechanisms underlying goal-directed behavior crucial for creation of agentive experience.

Measuring morbidity and mortality attributable to rheumatic heart disease in Fiji: a prototype study of disease burden in a developing country by data linkage

ABSTRACT&#x0D; ObjectivesRheumatic heart disease remains a major public health concern in developing countries. Motivated by the lack of up-to-date epidemiologic data from endemic settings, we sought to quantity morbidity and mortality attributable the condition in Fiji, a middle-income country where a high prevalence has consistently been reported. Having resolved to undertake the analysis using the existing routine clinical and administrative data at our disposal, we first set out to develop a data linkage procedure robust to the inherent limitations of data from low resource settings.&#x0D; ApproachRecords were available from four sources: an electronic patient information system, a database of death certificates, a disease control register, and echocardiography clinic registers. All referred to 2008-2012.&#x0D; Throughout the design and calibration process we used 1,406 known duplications in the patient information system from which we calculated the sensitivity and specificity. After cleaning, standardisation and preliminary blocking, we categorised identifiers including names, dates and demographics into agreement, partial agreement, disagreement or missing, accounting for issues such as out of order or misspelt names. After concentrating true matches by further blocking, we estimated match and nonmatch probabilities using expectation maximisation under the Fellegi-Sunter model of record linkage. We then derived the posterior match probability taking into consideration the size of block and prior information about the probability a match be present given the demographics of the individual concerned. In its final configuration, with record pairs considered a match if they achieved a posterior probability of over 50%, our procedure identified the known duplications with sensitivity of 91.4% and specificity of 99.9%.&#x0D; ResultsHaving identified 2,619 cases from the 1,773,999 records available, we used the linked data to make population-based estimates of prevalence using capture-recapture analyses and cause-specific mortality using relative survival methods, the first such estimates for a developing country. Moreover, in sensitivity analyses, we found that changing posterior probability threshold above which record pairs were considered a match had limited impact on the results.&#x0D; ConclusionAlthough data linkage is widely used for epidemiologic research in high-income settings, its application to developing countries has been limited. We developed and validated a data linkage procedure that can be used to turn largely unstudied routine clinical and administrative data into robust estimates of disease burden. With the growing availability of computerized data, we propose our approach has strong potential to assist the production of disease burden statistics in developing countries where civil registration systems are weak.

Building a high quality data linkage spine using a targeted approach to clerical review

ABSTRACTObjectivesThe Tasmanian Data Linkage Unit (TDLU) was established through the University of Tasmania in 2011 with the first dataset imported to its Master Linkage Map (MLM) during 2014. Tasmania an island state of Australia, has a population of approximately 516,000. From the TDLU’s earliest inception, it was deemed important to build a high quality linkage spine comprising key administrative data representative of significant state health and related datasets to support quality population level research.ApproachThe TDLU has embraced a model of continual quality and process enhancement as a determined strategy to support ongoing business improvement. Initial linkage approaches utilised ‘traditional’ methods of reviewing record pairs within an upper and lower confidence range. This approach resulted in false record pairs with high confidence levels being linked (false positives) and true record pairs at lower confidence levels not linked (false negatives). To improve linkage quality, the TDLU has continually refined and modified its clerical review methodology with a specialist software module developed to identify specific record attributes within groups that require the group to be manually reviewed and resolved. A range of SQL queries have also been developed to identify incorrect links and further enhance the linkage quality of the MLM.ResultsThe linkage quality tools implemented have led to improved clerical review and quality assurance processes which in turn have increased the overall quality of the linkage spine. The ‘targeted’ method of clerical review provides easy identification of false positive records, particularly those with high confidence scores such as twins and husband/wife combinations. The review of groups at lower confidence levels has minimised the rate of false negative pairs however further refinement of tools is required to minimise the time spent on reviewing these groups. The clerical review software module has equipped staff with the necessary information to make informed and timely decisions when reviewing groups of records. Detailed documentation is maintained for each linkage project providing continual feedback for system and process improvements as the linkage spine increases in size.ConclusionThe process of clerical review and quality assurance requires a commitment to continual refinement of tools and techniques resulting in a higher quality linkage spine and a reduction in the total time and resource required to link datasets.

The impact of administrative dataset characteristics on clerical review: A Tasmanian case study

ABSTRACTObjectivesThe Tasmanian Data Linkage Unit (TDLU) routinely collects detailed statistical data specific to time spent preparing datasets for linkage, performing clerical review and quality assurance for all datasets added to its Master Linkage Map (MLM). This information is used for a range of functions including resource planning, performance monitoring, preparation of data linkage quotes, data quality reporting and identifying potential process improvements.ApproachDetailed metadata for each dataset imported to the MLM is collected to determine likely linkage quality and to estimate clerical review resource requirements. Attributes including the time period covered, data completeness, data quality and accuracy, the population characteristics and an understanding of how the data was collected are documented. In addition to this metadata, clerical reviewer staff record the number of record pairs and groups reviewed within confidence ranges and the time taken to complete review. ResultsThe data collected provides strong evidence that administrative datasets with more problematic characteristics take longer to prepare, clerically review and quality assure. Tasmanian public hospital emergency presentations and admitted patient datasets, which are high in quality and completeness, have required less clerical review (8% and 6% respectively of total records) compared with perinatal baby (30% of total records clerically reviewed and significant additional time spent on quality assurance) and a small cohort of ambulance patients where 78% of all records were manually reviewed. The ambulance and perinatal baby datasets contained incomplete name, address and hospital identifier fields, and spelling and data errors were evident in the ambulance dataset as a result of the data capture method (over the phone with critical time constraints).ConclusionThe underlying dataset quality and completeness impacts to a significant degree the time taken to clerically review a dataset, and in turn the quality of links made in a MLM. Additionally, each new dataset added to the MLM introduces an additional layer of complexity necessitating additional processing time due to the number of record comparisons made. Having an agreed set of formal processes to prepare administrative data prior to data linkage, and understanding how source data is collected and processed, is crucial to obtaining high levels of quality within the MLM.

Match quality of a linkage strategy based on the combined use of a statistical linkage key and the Levenshtein distance to link birth to death records in Brazil.

ABSTRACTObjectivesTo assess the match quality of a linkage strategy based on the combined use of a statistical linkage key and the Levenshtein distance to link birth to death records in Brazil. ApproachFirst we evaluated the discrimination power of a statistical linkage key adapted from the Australian SLK-581. The modified statistical linkage key (MSLK-781) was based on the concatenation of the 2nd, 3rd and 5th letters of the mother's family name, the 2nd and 3rd letters of the mother's given name, the 2nd and 3rd letters of the mother's middle name, the child's date of birth and sex. We calculated the proportion of records that have a unique value for the MSLK-781 within the 2013 live births (N=224,038 records) and mortality (N=132,646 records) databases for Rio de Janeiro state. We also calculated the joint unique proportion measure based on the product of these two proportions. Second we evaluated the match quality of a linkage strategy based on the combined use of the MSLK-781 and the Levenshtein distance of the mother's name to link the live births database to death records of singleton children younger than one year of age (N=1,488). To assess the match quality we calculated the sensitivity, the predictive positive value (PPV) and the F-measure. ResultsThe proportion of records that have a unique value for the MSLK-781 within the live birth and the mortality databases were, respectively, 97.5% and 98.8%, which yields a joint unique proportion of 96.1%. The match quality measures of the linkage strategy based only on the MSLK-781 were: sensitivity=83.6%; PPV=98.3%; F-measure=90.4%. Combining the agreement on the MSLK-781 with a Levenshtein distance of the mother's name of less than 4 for the record pairs classification eliminated the false-positive matches (PPV=100%) with a small decline in the sensitivity (81.7%) and the F-measure (89.9%). ConclusionUsing the MSLK-781 combined with the Levenshtein distance can be used as a first pass for linking birth to death records in Brazil without having to send pairs of records to clerical review.

Linkage of health insurance data to the Netherlands Twin Register

ABSTRACTObjectivesWe compared four different approaches to record linkage, demonstrating its potential in enriching the Netherlands Twin Register (NTR) with healthcare data from a large health insurance provider (AHD), which covers approximately 26% of the population. Challenges included the fact that young twins share most of their linkage variables and overlapping identification numbers were absent. We extensively validated the linkage results and complemented them with an indicator of representativeness.ApproachSubjects born since 1986 were selected in the NTR (N=30,383) and AHD (N=1,532,675). Linkage variables were cleaned and harmonised and linkage keys were chosen that were strong enough to uniquely identify most subjects in either dataset. The four linkages were deterministic, probabilistic, probabilistic with Jaro-Winkler distance, and probabilistic using encrypted identifiers.Validation took place by reviewing linkage variables and information like family relations; linked record pairs were categorised into correct, possible or false links. Furthermore, the consistency of linked data was reviewed focusing on Attention Deficit Hyperactivity Disorder (ADHD), a prevalent condition among children. Finally, we quantified the impact of the linkage on representativeness of the target population.ResultsProbabilistic linkage with distance calculation linked 7944 NTR subjects to the AHD with 82% correct, 17% possible, and 1% false links. It encompassed nearly all links identified by any of the other methods, which were more conservative.Information about medication was available in the NTR dataset for 40% of the linked subjects. Validation of the information about ADHD inside linked records revealed few discrepancies between the two datasets. Linkage had a modest effect on the representativeness of the population. The current linkage has confirmed a number of known ADHD cases and identified 146 new subjects using methylphenidate and 101 receiving psychiatric care for ADHD, who were not previously recognized as ADHD cases within the NTR. ConclusionRecord linkage with sources of data such as a health insurance database can be an efficient way of data collection in cohort research. Our study shows that record linkage is also possible for twin pairs. Although a minority of all NTR subjects were covered by the AHD, the linkage resulted in an enrichment of the NTR dataset. Exact information about the dosage and frequency of drugs was obtained without contacting subjects with detailed questionnaires. With a total of almost eight thousand retrieved records, the size of the linked dataset is sufficiently large for epidemiological research of non-rare conditions.

Bayesian Estimation of Bipartite Matchings for Record Linkage

ABSTRACT The bipartite record linkage task consists of merging two disparate datafiles containing information on two overlapping sets of entities. This is nontrivial in the absence of unique identifiers and it is important for a wide variety of applications given that it needs to be solved whenever we have to combine information from different sources. Most statistical techniques currently used for record linkage are derived from a seminal article by Fellegi and Sunter in 1969. These techniques usually assume independence in the matching statuses of record pairs to derive estimation procedures and optimal point estimators. We argue that this independence assumption is unreasonable and instead target a bipartite matching between the two datafiles as our parameter of interest. Bayesian implementations allow us to quantify uncertainty on the matching decisions and derive a variety of point estimators using different loss functions. We propose partial Bayes estimates that allow uncertain parts of the bipartite matching to be left unresolved. We evaluate our approach to record linkage using a variety of challenging scenarios and show that it outperforms the traditional methodology. We illustrate the advantages of our methods merging two datafiles on casualties from the civil war of El Salvador. Supplementary materials for this article are available online.

A scaling approach to record linkage.

With increasing availability of large datasets derived from administrative and other sources, there is an increasing demand for the successful linking of these to provide rich sources of data for further analysis. Variation in the quality of identifiers used to carry out linkage means that existing approaches are often based upon 'probabilistic' models, which are based on a number of assumptions, and can make heavy computational demands. In this paper, we suggest a new approach to classifying record pairs in linkage, based upon weights (scores) derived using a scaling algorithm. The proposed method does not rely on training data, is computationally fast, requires only moderate amounts of storage and has intuitive appeal. Copyright © 2017 John Wiley & Sons, Ltd.

Unsupervised blocking and probabilistic parallelisation for record matching of distributed big data

Record Matching refers to identifying pairs of records that relate to the same entities across different data sources. In many applications of data mining, record matching is usually associated to quadratic complexity. In practice, the number of non-matching record pairs always far exceeds the number of matching pairs, and this is called imbalance problem. Blocking is a technique of data reduction, which can filter unlikely matching pairs before record matching. However, for big data there is no fast and effective blocking algorithm yet. In this paper, we report on big data infrastructure to improve efficiency of blocking. Our approach runs blocking process independently and distributedly on the partitions of whole data. To improve efficiency, we adopt a probabilistic technique to balance the speed and the effect of the algorithm that we proposed for distributed blocking. Our experimental analysis endorses the superiority of our technique and shows its novel scalability.