Range Of Data Quality Research Articles

PHA4GE quality control contextual data tags: standardized annotations for sharing public health sequence datasets with known quality issues to facilitate testing and training.

As public health laboratories expand their genomic sequencing and bioinformatics capacity for the surveillance of different pathogens, labs must carry out robust validation, training, and optimization of wet- and dry-lab procedures. Achieving these goals for algorithms, pipelines and instruments often requires that lower quality datasets be made available for analysis and comparison alongside those of higher quality. This range of data quality in reference sets can complicate the sharing of sub-optimal datasets that are vital for the community and for the reproducibility of assays. Sharing of useful, but sub-optimal datasets requires careful annotation and documentation of known issues to enable appropriate interpretation, avoid being mistaken for better quality information, and for these data (and their derivatives) to be easily identifiable in repositories. Unfortunately, there are currently no standardized attributes or mechanisms for tagging poor-quality datasets, or datasets generated for a specific purpose, to maximize their utility, searchability, accessibility and reuse. The Public Health Alliance for Genomic Epidemiology (PHA4GE) is an international community of scientists from public health, industry and academia focused on improving the reproducibility, interoperability, portability, and openness of public health bioinformatic software, skills, tools and data. To address the challenges of sharing lower quality datasets, PHA4GE has developed a set of standardized contextual data tags, namely fields and terms, that can be included in public repository submissions as a means of flagging pathogen sequence data with known quality issues, increasing their discoverability. The contextual data tags were developed through consultations with the community including input from the International Nucleotide Sequence Data Collaboration (INSDC), and have been standardized using ontologies - community-based resources for defining the tag properties and the relationships between them. The standardized tags are agnostic to the organism and the sequencing technique used and thus can be applied to data generated from any pathogen using an array of sequencing techniques. The tags can also be applied to synthetic (lab created) data. The list of standardized tags is maintained by PHA4GE and can be found at https://github.com/pha4ge/contextual_data_QC_tags. Definitions, ontology IDs, examples of use, as well as a JSON representation, are provided. The PHA4GE QC tags were tested, and are now implemented, by the FDA's GenomeTrakr laboratory network as part of its routine submission process for SARS-CoV-2 wastewater surveillance. We hope that these simple, standardized tags will help improve communication regarding quality control in public repositories, in addition to making datasets of variable quality more easily identifiable. Suggestions for additional tags can be submitted to PHA4GE via the New Term Request Form in the GitHub repository. By providing a mechanism for feedback and suggestions, we also expect that the tags will evolve with the needs of the community.

Developing a Global Data Breach Database and the Challenges Encountered

If the mantra “data is the new oil” of our digital economy is correct, then data leak incidents are the critical disasters in the online society. The initial goal of our research was to present a comprehensive database of data breaches of personal information that took place in 2018 and 2019. This information was to be drawn from press reports, industry studies, and reports from regulatory agencies across the world. This article identified the top 430 largest data breach incidents among more than 10,000 data breach incidents. In the process, we encountered many complications, especially regarding the lack of standardization of reporting. This article should be especially interesting to the readers of JDIQ because it describes both the range of data quality and consistency issues found as well as what was learned from the database created. The database that was created, available at https://www.databreachdb.com, shows that the number of data records breached in those top 430 incidents increased from around 4B in 2018 to more than 22B in 2019. This increase occurred despite the strong efforts from regulatory agencies across the world to enforce strict rules on data protection and privacy, such as the General Data Protection Regulation (GDPR) that went into effect in Europe in May 2018. Such regulatory effort could explain the reason why there is such a large number of data breach cases reported in the European Union when compared to the U.S. (more than 10,000 data breaches publicly reported in the U.S. since 2018, while the EU reported more than 160,000 1 data breaches since May 2018). However, we still face the problem of an excessive number of breach incidents around the world. This research helps to understand the challenges of proper visibility of such incidents on a global scale. The results of this research can help government entities, regulatory bodies, security and data quality researchers, companies, and managers to improve the data quality of data breach reporting and increase the visibility of the data breach landscape around the world in the future.

Evaluation of approximate comparison methods on Bloom filters for probabilistic linkage.

IntroductionThe need for increased privacy protection in data linkage has driven the development of privacy-preserving record linkage (PPRL) techniques. A popular technique using Bloom filters with cryptographic analyses, modifications, and hashing variations to optimise privacy has been the focus of much research in this area. With few applications of Bloom filters within a probabilistic framework, there is limited information on whether approximate matches between Bloom filtered fields can improve linkage quality.ObjectivesIn this study, we evaluate the effectiveness of three approximate comparison methods for Bloom filters within the context of the Fellegi-Sunter model of recording linkage: Sørensen–Dice coefficient, Jaccard similarity and Hamming distance.MethodsUsing synthetic datasets with introduced errors to simulate datasets with a range of data quality and a large real-world administrative health dataset, the research estimated partial weight curves for converting similarity scores (for each approximate comparison method) to partial weights at both field and dataset level. Deduplication linkages were run on each dataset using these partial weight curves. This was to compare the resulting quality of the approximate comparison techniques with linkages using simple cut-off similarity values and only exact matching.ResultsLinkages using approximate comparisons produced significantly better quality results than those using exact comparisons only. Field level partial weight curves for a specific dataset produced the best quality results. The Sørensen-Dice coefficient and Jaccard similarity produced the most consistent results across a spectrum of synthetic and real-world datasets.ConclusionThe use of Bloom filter similarity comparisons for probabilistic record linkage can produce linkage quality results which are comparable to Jaro-Winkler string similarities with unencrypted linkages. Probabilistic linkages using Bloom filters benefit significantly from the use of similarity comparisons, with partial weight curves producing the best results, even when not optimised for that particular dataset

Cosmology and astrophysics from relaxed galaxy clusters – I. Sample selection

This is the first in a series of papers studying the astrophysics and cosmology of massive, dynamically relaxed galaxy clusters. Here we present a new, automated method for identifying relaxed clusters based on their morphologies in X-ray imaging data. While broadly similar to others in the literature, the morphological quantities that we measure are specifically designed to provide a fair basis for comparison across a range of data quality and cluster redshifts, to be robust against missing data due to point-source masks and gaps between detectors, and to avoid strong assumptions about the cosmological background and cluster masses. Based on three morphological indicators - Symmetry, Peakiness and Alignment - we develop the SPA criterion for relaxation. This analysis was applied to a large sample of cluster observations from the Chandra and ROSAT archives. Of the 361 clusters which received the SPA treatment, 57 (16 per cent) were subsequently found to be relaxed according to our criterion. We compare our measurements to similar estimators in the literature, as well as projected ellipticity and other image measures, and comment on trends in the relaxed cluster fraction with redshift, temperature, and survey selection method. Code implementing our morphological analysis will be made available on the web.

COMPARISONS OF THE INTERSTELLAR MAGNETIC FIELD DIRECTIONS OBTAINED FROM THEIBEXRIBBON AND INTERSTELLAR POLARIZATIONS

Variations in the spatial configuration of the interstellar magnetic field (ISMF) near the Sun can be constrained by comparing the ISMF direction at the heliosphere found from the Interstellar Boundary Explorer spacecraft (IBEX) observations of a 'Ribbon' of energetic neutral atoms (ENAs), with the ISMF direction derived from optical polarization data for stars within ~40 pc. Using interstellar polarization observations towards ~30 nearby stars within 90 deg of the heliosphere nose, we find that the best fits to the polarization position angles are obtained for a magnetic pole directed towards ecliptic coordinates of lambda, beta 263 deg, 37 deg (or galactic coordinates of L,B 38 deg, 23deg), with uncertainties of +/- 35 deg, based on the broad minimum of the best fits and the range of data quality. This magnetic pole is 33 deg from the magnetic pole that is defined by the center of the arc of the ENA Ribbon. The IBEX ENA ribbon is seen in sightlines that are perpendicular to the ISMF as it drapes over the heliosphere. The similarity of the polarization and Ribbon directions for the local ISMF suggest that the local field is coherent over scale sizes of tens of parsecs. The ISMF vector direction is nearly perpendicular to the flow of local ISM through the local standard of rest, supporting a possible local ISM origin related to an evolved expanding magnetized shell. The local ISMF direction is found to have a curious geometry with respect to the cosmic microwave background dipole moment.

Aeromagnetic regional survey of onshore Australia

The Australian Bureau of Mineral Resources (BMR) is responsible for the National Airborne Magnetic Database. This data base consists of results from approximately 3 500 000 line‐km of regional survey flying carried out over 35 years, recording total magnetic intensity. The magnetic data base is one of the most important geophysical data bases for Australia and is used extensively by the minerals and petroleum exploration industries. First‐pass coverage of onshore Australia is aimed for completion in 1992. This coverage contains data from surveys with a wide range of specifications, resulting in a wide range of data quality; some of the areas covered by poorer quality data may be reflown later. For the most part, the intention has been to acquire data at a continuous ground clearance of 150 m and with a line spacing of 1500 m. However, over some sedimentary basins, the line spacing is in excess of 3200 m. New color and grey‐scale (image processed type) digital magnetic maps (pixel maps) are in preparation; these will supersede the 1976 digital magnetic map of Australia, which was gridded on a 1.2 minute mesh (2000 m) mostly by digitizing contours on maps. The new map, produced from flight‐line data, will have a grid size of 0.25 minutes. Initially, a series of maps will be produced with each one covering a block of 4 degrees latitude by 6 degrees longitude, coinciding with standard 1 : 1 000 000 map sheets. An example included for the Adelaide 1 : 1 000 000 map sheet in Southern Australia shows a dramatic increase in the number of anomalies over those that were evident in earlier contour presentations.