Merged Data Files Research Articles

Multifile Partitioning for Record Linkage and Duplicate Detection

Merging datafiles containing information on overlapping sets of entities is a challenging task in the absence of unique identifiers, and is further complicated when some entities are duplicated in the datafiles. Most approaches to this problem have focused on linking two files assumed to be free of duplicates, or on detecting which records in a single file are duplicates. However, it is common in practice to encounter scenarios that fit somewhere in between or beyond these two settings. We propose a Bayesian approach for the general setting of multifile record linkage and duplicate detection. We use a novel partition representation to propose a structured prior for partitions that can incorporate prior information about the data collection processes of the datafiles in a flexible manner, and extend previous models for comparison data to accommodate the multifile setting. We also introduce a family of loss functions to derive Bayes estimates of partitions that allow uncertain portions of the partitions to be left unresolved. The performance of our proposed methodology is explored through extensive simulations. Supplementary materials for this article are available online.

GazeR: A Package for Processing Gaze Position and Pupil Size Data

Eye-tracking is widely used throughout the scientific community, from vision science and psycholinguistics to marketing and human-computer interaction. Surprisingly, there is little consistency and transparency in preprocessing steps, making replicability and reproducibility difficult. To increase replicability, reproducibility, and transparency, a package in R (a free and widely used statistical programming environment) called gazeR was created to read and preprocess two types of data: gaze position and pupil size. For gaze position data, gazeR has functions for reading in raw eye-tracking data, formatting it for analysis, converting from gaze coordinates to areas of interest, and binning and aggregating data. For data from pupillometry studies, the gazeR package has functions for reading in and merging multiple raw pupil data files, removing observations with too much missing data, eliminating artifacts, blink identification and interpolation, subtractive baseline correction, and binning and aggregating data. The package is open-source and freely available for download and installation: https://github.com/dmirman/gazer . We provide step-by-step analyses of data from two tasks exemplifying the package's capabilities.

Taxamat: Automated biodiversity data management tool - Implications for microbiome studies.

Working with biodiversity data is a computationally intensive process. Numerous applications and services provide options to deal with sequencing and taxonomy data. Professional statistics software are also available to analyze these type of data. However, in-between the two processes there is a huge need to curate biodiversity sample files. Curation involves creating summed abundance values for chosen taxonomy ranks, excluding certain taxa from analysis, and finally merging and downsampling data files. Very few tools, if any, offer a solution to this problem, thus we present Taxamat, a simple data management application that allows for curation of biodiversity data files before they can be imported to other statistics software. Taxamat is a downloadable application for automated curation of biodiversity data featuring taxonomic classification, taxon filtering, sample merging, and downsampling. Input and output files are compatible with most widely used programs. Taxamat is available on the web at http://www.taxamat.com either as a single executable or as an installable package for Microsoft Windows platforms.

A3.24 Distinct profile of cytokine production by th17 cells in systemic sclerosis

BackgroundThe pathogenesis of systemic sclerosis (SSc) is largely unknown, although proinflammatory cytokines are considered to play an important role.AimsInvestigate the pattern of expression of proinflammatory cytokines by peripheral blood (PB)...

Merging Large-Scale Assessment Data for Secondary Analysis: Experiences with EQAO’s Data

This paper is a narrative of our experience in analyzing and merging data files provided to us by the Education Quality and Accountability Office (EQAO). In the paper, we propose a scheme of merging data files by means of Structured Query Language (SQL, pronounced as “sequel”). Although, the narrative of our experiences using this merging scheme could have been extended to any number of data files, the aim of this work was to merge only three EQAO data files. Via this merge process, we were able to gain meaningful information and facilitate the analysis of EQAO data to answer our research questions. By using SQL queries, our approach was not only to analyze the available data files but also to construct a narrative about viewing and handling data contained in the files.

Navigating Complex Sample Analysis Using National Survey Data

The National Center for Health Statistics conducts the National Health and Nutrition Examination Survey and other national surveys with probability-based complex sample designs. Goals of national surveys are to provide valid data for the population of the United States. Analyses of data from population surveys present unique challenges in the research process but are valuable avenues to study the health of the United States population. The aim of this study was to demonstrate the importance of using complex data analysis techniques for data obtained with complex multistage sampling design and provide an example of analysis using the SPSS Complex Samples procedure. Illustration of challenges and solutions specific to secondary data analysis of national databases are described using the National Health and Nutrition Examination Survey as the exemplar. Oversampling of small or sensitive groups provides necessary estimates of variability within small groups. Use of weights without complex samples accurately estimates population means and frequency from the sample after accounting for over- or undersampling of specific groups. Weighting alone leads to inappropriate population estimates of variability, because they are computed as if the measures were from the entire population rather than a sample in the data set. The SPSS Complex Samples procedure allows inclusion of all sampling design elements, stratification, clusters, and weights. Use of national data sets allows use of extensive, expensive, and well-documented survey data for exploratory questions but limits analysis to those variables included in the data set. The large sample permits examination of multiple predictors and interactive relationships. Merging data files, availability of data in several waves of surveys, and complex sampling are techniques used to provide a representative sample but present unique challenges. In sophisticated data analysis techniques, use of these data is optimized.

A synthetic rainbow trout linkage map provides new insights into the salmonid whole genome duplication and the conservation of synteny among teleosts

BackgroundRainbow trout is an economically important fish and a suitable experimental organism in many fields of biology including genome evolution, owing to the occurrence of a salmonid specific whole-genome duplication (4th WGD). Rainbow trout is among some of the most studied teleosts and has benefited from substantial efforts to develop genomic resources (e.g., linkage maps. Here, we first generated a synthetic map by merging segregation data files derived from three independent linkage maps. Then, we used it to evaluate genome conservation between rainbow trout and three teleost models, medaka, stickleback and zebrafish and to further investigate the extent of the 4th WGD in trout genome.ResultsThe INRA linkage map was updated by adding 211 new markers. After standardization of marker names, consistency of marker assignment to linkage groups and marker orders was checked across the three different data sets and only loci showing consistent location over all or almost all of the data sets were kept. This resulted in a synthetic map consisting of 2226 markers and 29 linkage groups spanning over 3600 cM. Blastn searches against medaka, stickleback, and zebrafish genomic databases resulted in 778, 824 and 730 significant hits respectively while blastx searches yielded 505, 513 and 510 significant hits. Homology search results revealed that, for most rainbow trout chromosomes, large syntenic regions encompassing nearly whole chromosome arms have been conserved between rainbow trout and its closest models, medaka and stickleback. Large conserved syntenies were also found between the genomes of rainbow trout and the reconstructed teleost ancestor. These syntenies consolidated the known homeologous affinities between rainbow trout chromosomes due to the 4th WGD and suggested new ones.ConclusionsThe synthetic map constructed herein further highlights the stability of the teleost genome over long evolutionary time scales. This map can be easily extended by incorporating new data sets and should help future rainbow trout whole genome sequence assembly. Finally, the persistence of large conserved syntenies across teleosts should facilitate the identification of candidate genes through comparative mapping, even if the occurrence of intra-chromosomal micro-rearrangement may hinder the accurate prediction their genomic location.

CARINA TCO<sub>2</sub> data in the Atlantic Ocean

Abstract. Water column data of carbon and carbon-relevant hydrographic and hydrochemical parameters from 188 cruises in the Arctic Mediterranean Seas, Atlantic and Southern Ocean have been retrieved and merged in a new data base: the CARINA (CARbon IN the Atlantic) Project. These data have gone through rigorous quality control (QC) procedures so as to improve the quality and consistency of the data as much as possible. Secondary quality control, which involved objective study of data in order to quantify systematic differences in the reported values, was performed for the pertinent parameters in the CARINA data base. Systematic biases in the data have been tentatively corrected in the data products. The products are three merged data files with measured, adjusted and interpolated data of all cruises for each of the three CARINA regions (Arctic Mediterranean Seas, Atlantic and Southern Ocean). Ninety-eight cruises were conducted in the "Atlantic" defined as the region south of the Greenland-Iceland-Scotland Ridge and north of about 30° S. Here we report the details of the secondary QC which was done on the total dissolved inorganic carbon (TCO2) data and the adjustments that were applied to yield the final data product in the Atlantic. Procedures of quality control – including crossover analysis between stations and inversion analysis of all crossover data – are briefly described. Adjustments were applied to TCO2 measurements for 17 of the cruises in the Atlantic Ocean region. With these adjustments, the CARINA data base is consistent both internally as well as with GLODAP data, an oceanographic data set based on the WOCE Hydrographic Program in the 1990s, and is now suitable for accurate assessments of, for example, regional oceanic carbon inventories, uptake rates and model validation.

CARINA data synthesis project: pH data scale unification and cruise adjustments

Abstract. Data on carbon and carbon-relevant hydrographic and hydrochemical parameters from 188 previously non-publicly available cruise data sets in the Artic Mediterranean Seas (AMS), Atlantic Ocean and Southern Ocean have been retrieved and merged to a new database: CARINA (CARbon IN the Atlantic Ocean). These data have gone through rigorous quality control (QC) procedures to assure the highest possible quality and consistency. The data for most of the measured parameters in the CARINA database were objectively examined in order to quantify systematic differences in the reported values. Systematic biases found in the data have been corrected in the data products, three merged data files with measured, calculated and interpolated data for each of the three CARINA regions; AMS, Atlantic Ocean and Southern Ocean. Out of a total of 188 cruise entries in the CARINA database, 59 reported pH measured values. All reported pH data have been unified to the Sea-Water Scale (SWS) at 25 °C. Here we present details of the secondary QC of pH in the CARINA database and the scale unification to SWS at 25 °C. The pH scale has been converted for 36 cruises. Procedures of quality control, including crossover analysis between cruises and inversion analysis are described. Adjustments were applied to the pH values for 21 of the cruises in the CARINA dataset. With these adjustments the CARINA database is consistent both internally as well as with the GLODAP data, an oceanographic data set based on the World Hydrographic Program in the 1990s. Based on our analysis we estimate the internal consistency of the CARINA pH data to be 0.005 pH units. The CARINA data are now suitable for accurate assessments of, for example, oceanic carbon inventories and uptake rates, for ocean acidification assessment and for model validation.

Quality control procedures and methods of the CARINA database

Abstract. Data on the carbon and carbon relevant hydrographic and hydrochemical parameters from previously not publicly available cruises in the Arctic, Atlantic and Southern Ocean have been retrieved and merged to a new data base: CARINA (CARbon IN the Atlantic). These data have gone through rigorous quality control (QC) procedures to assure the highest possible quality and consistency. All CARINA data were subject to primary QC; a process in which data are studied in order to identify outliers and obvious errors. Additionally, secondary QC was performed for several of the measured parameters in the CARINA data base. Secondary QC is a process in which the data are objectively studied in order to quantify systematic differences in the reported values. This process involved crossover analysis, and as a second step the offsets derived from the crossover analysis were used to calculate corrections of the parameters measured on individual cruises using least square models. Significant biases found in the data have been corrected in the data products, i.e. three merged data files containing measured, calculated and interpolated data for each of the three regions (i.e. Arctic Mediterranean Seas, Atlantic, and Southern Ocean). Here we report on the technical details of the quality control and on tools that have been developed and used during the project, including procedures for crossover analysis and least square models. Furthermore, an interactive website for uploading of results, plots, comments etc. was developed and was of critical importance for the success of the project, this is also described here.

Atlantic Ocean CARINA data: overview and salinity adjustments

Abstract. Water column data of carbon and carbon-relevant hydrographic and hydrochemical parameters from 188 previously non-publicly available cruise data sets in the Arctic Mediterranean Seas, Atlantic and Southern Ocean have been retrieved and merged into a new database: CARINA (CARbon dioxide IN the Atlantic Ocean). The data have gone through rigorous quality control procedures to assure the highest possible quality and consistency. The data for the pertinent parameters in the CARINA database were objectively examined in order to quantify systematic differences in the reported values, i.e. secondary quality control. Systematic biases found in the data have been corrected in the three data products: merged data files with measured, calculated and interpolated data for each of the three CARINA regions, i.e. the Arctic Mediterranean Seas, the Atlantic and the Southern Ocean. These products have been corrected to be internally consistent. Ninety-eight of the cruises in the CARINA database were conducted in the Atlantic Ocean, defined here as the region south of the Greenland-Iceland-Scotland Ridge and north of about 30° S. Here we present an overview of the Atlantic Ocean synthesis of the CARINA data and the adjustments that were applied to the data product. We also report the details of the secondary QC (Quality Control) for salinity for this data set. Procedures of quality control – including crossover analysis between stations and inversion analysis of all crossover data – are briefly described. Adjustments to salinity measurements were applied to the data from 10 cruises in the Atlantic Ocean region. Based on our analysis we estimate the internal consistency of the CARINA-ATL salinity data to be 4.1 ppm. With these adjustments the CARINA data products are consistent both internally as well as with GLODAP data, an oceanographic data set based on the World Hydrographic Program in the 1990s, and is now suitable for accurate assessments of, for example, oceanic carbon inventories and uptake rates and for model validation.

CARINA alkalinity data in the Atlantic Ocean

Abstract. Data on carbon and carbon-relevant hydrographic and hydrochemical parameters from previously non-publicly available cruise data sets in the Arctic, Atlantic and Southern Ocean have been retrieved and merged to a new database: CARINA (CARbon IN the Atlantic). These data have gone through rigorous quality control (QC) procedures to assure the highest possible quality and consistency. The data for most of the measured parameters in the CARINA data base were objectively examined in order to quantify systematic differences in the reported values, i.e. secondary quality control. Systematic biases found in the data have been corrected in the data products, i.e. three merged data files with measured, calculated and interpolated data for each of the three CARINA regions; Arctic, Atlantic and Southern Ocean. Out of a total of 188 cruise entries in the CARINA database, 98 were conducted in the Atlantic Ocean and of these, 75 cruises report alkalinity values. Here we present details of the secondary QC on alkalinity for the Atlantic Ocean part of CARINA. Procedures of quality control, including crossover analysis between cruises and inversion analysis of all crossover data are briefly described. Adjustments were applied to the alkalinity values for 16 of the cruises in the Atlantic Ocean region. With these adjustments the CARINA database is consistent both internally as well as with GLODAP data, an oceanographic data set based on the World Hydrographic Program in the 1990s. Based on our analysis we estimate the internal accuracy of the CARINA-ATL alkalinity data to be 3.3 μmol kg−1. The CARINA data are now suitable for accurate assessments of, for example, oceanic carbon inventories and uptake rates and for model validation.

CARINA: nutrient data in the Atlantic Ocean

Abstract. Data on carbon and carbon-relevant hydrographic and hydrochemical parameters from previously non-publicly available cruise data sets in the Arctic, Atlantic and Southern Ocean have been retrieved and merged to a new database: CARINA (CARbon IN the Atlantic). These data have gone through rigorous quality control (QC) procedures to assure the highest possible quality and consistency. The data for most of the measured parameters in the CARINA data base were objectively examined in order to quantify systematic differences in the reported values, i.e. secondary quality control. Systematic biases found in the data have been corrected in the data products, i.e. three merged data files with measured, calculated and interpolated data for each of the three CARINA regions; Arctic Mediterranean Seas, Atlantic and Southern Ocean. Out of a total of 188 cruise entries in the CARINA database, 98 were conducted in the Atlantic Ocean and of these 84 cruises report nitrate values, 79 silicate, and 78 phosphate. Here we present details of the secondary QC for nutrients for the Atlantic Ocean part of CARINA. Procedures of quality control, including crossover analysis between cruises and inversion analysis of all crossover data are briefly described. Adjustments were applied to the nutrient values for 43 of the cruises in the Atlantic Ocean region. With these adjustments the CARINA database is consistent both internally as well as with GLODAP data, an oceanographic data set based on the World Hydrographic Program in the 1990s (Key et al., 2004). Based on our analysis we estimate the internal accuracy of the CARINA-ATL nutrient data to be: nitrate 1.5%; phosphate 2.6%; silicate 3.1%. The CARINA data are now suitable for accurate assessments of, for example, oceanic carbon inventories and uptake rates and for model validation.

The polarisation method for merging data files and analysing loyalty to product attributes, prices and brands in revealed preference

The polarisation method for merging data files and analysing loyalty to product attributes, prices and brands in revealed preference

Effects on accidents of periodic motor vehicle inspection in Norway

An extensive programme of periodic motor vehicle inspection was introduced in Norway after 1995, when the treaty between Norway and the European Union (EU) granting Norway (not a member of the EU) access to the EU inner market took effect (The EEA treaty). This paper evaluates the effects on accidents of periodic inspections of cars. Trucks and buses were not included in the study. Negative binomial regression models were fitted to data on accidents and inspections created by merging data files provided by a major insurance company and by the Public Roads Administration. Technical defects prior to inspection were associated with an increased accident rate. Inspections were found to strongly reduce the number of technical defects in cars. Despite this, no effect of inspections on accident rate were found. This finding is inconsistent with the fact that technical defects appear to increase the accident rate; one would expect the repair of such defects to reduce the accident rate. Potential explanations of the findings in terms of behavioural adaptation among car owners are discussed. It is suggested that car owners adapt driving behaviour to the technical condition of the car and that the effect attributed to technical defects before inspection may in part be the result of a tendency for owners who are less concerned about safety to neglect the technical condition of their cars. These car owners might have had a higher accident rate than other car owners irrespective of the technical condition of the car.

Rising trends in cesarean section rates in Egypt.

Cesarean section rates have been increasing worldwide, but little research exists on trends of cesarean section delivery for any country in the Arab world. The purpose of this study was to document recent levels and trends of cesarean section rates in Egypt, and to estimate trends in cesarean section by type of hospital from three population-based national surveys. This descriptive study used merged data files from three nationally representative samples of ever-married women aged 15 to 49 years. A significant rise in cesarean deliveries occurred for all births, from a low of 4.6 percent in 1992 to 10.3 percent in 2000. However, hospital-based cesarean deliveries were much higher in 1987-1988 (13.9%), increasing to 22.0 percent in 1999-2000. Although the cesarean section rate was slightly higher in private hospitals, the rate also increased consistently in public hospitals. The high and unprecedented increase in cesarean section rates reported in this study may be partly due to cesarean sections that are not medically indicated, and suggest that physician practice patterns, financial incentives or other profitability factors, and patient preferences should be explored.

Residential vacancy chain models of an urban housing market. Exercises in impact and needs assessment

Our purpose is to demonstrate the application of vacancy chain models to housing needs assessment and residential development impact assessment. The empirical base consists primarily of merged data files from the 1975 and 1980 Swedish Census for Population and Housing for the Municipality of Gavle. Our objectives are to calibrate a Markov vacancy chain model for impact assessment, disaggregate that model to identify development impacts on the mobility of specific population subgroups, calibrate a Leontief (input/output‐like) vacancy chain model and demonstrate its use in assessing the need for new housing given a vector of anticipated immigrants and newly formed households. The article concludes with a discussion of the range and limitations of vacancy chain models in applied contexts.