Data Quality Problems Research Articles

Limits on the reproducibility of marker associations with southern leaf blight resistance in the maize nested association mapping population.

BackgroundA previous study reported a comprehensive quantitative trait locus (QTL) and genome wide association study (GWAS) of southern leaf blight (SLB) resistance in the maize Nested Association Mapping (NAM) panel. Since that time, the genomic resources available for such analyses have improved substantially. An updated NAM genetic linkage map has a nearly six-fold greater marker density than the previous map and the combined SNPs and read-depth variants (RDVs) from maize HapMaps 1 and 2 provided 28.5 M genomic variants for association analysis, 17 fold more than HapMap 1. In addition, phenotypic values of the NAM RILs were re-estimated to account for environment-specific flowering time covariates and a small proportion of lines were dropped due to genotypic data quality problems. Comparisons of original and updated QTL and GWAS results confound the effects of linkage map density, GWAS marker density, population sample size, and phenotype estimates. Therefore, we evaluated the effects of changing each of these parameters individually and in combination to determine their relative impact on marker-trait associations in original and updated analyses.ResultsOf the four parameters varied, map density caused the largest changes in QTL and GWAS results. The updated QTL model had better cross-validation prediction accuracy than the previous model. Whereas joint linkage QTL positions were relatively stable to input changes, the residual values derived from those QTL models (used as inputs to GWAS) were more sensitive, resulting in substantial differences between GWAS results. The updated NAM GWAS identified several candidate genes consistent with previous QTL fine-mapping results.ConclusionsThe highly polygenic nature of resistance to SLB complicates the identification of causal genes. Joint linkage QTL are relatively stable to perturbations of data inputs, but their resolution is generally on the order of tens or more Mbp. GWAS associations have higher resolution, but lower power due to stringent thresholds designed to minimize false positive associations, resulting in variability of detection across studies. The updated higher density linkage map improves QTL estimation and, along with a much denser SNP HapMap, greatly increases the likelihood of detecting SNPs in linkage with causal variants. We recommend use of the updated genetic resources and results but emphasize the limited repeatability of small-effect associations.Electronic supplementary materialThe online version of this article (doi:10.1186/1471-2164-15-1068) contains supplementary material, which is available to authorized users.

The association between waiting for psychological therapy and therapy outcomes as measured by the CORE-OM.

To investigate the impact of waiting for psychological therapy on client well-being as measured by the Clinical Outcomes in Routine Evaluation-Outcome Measure (CORE-OM) global distress (GD) score. Global distress scores were retrieved for all clients referred for psychological therapy in a secondary care mental health service between November 2006 and May 2013 and who had completed a CORE-OM at assessment and first session. GD scores for a subgroup of 103 clients who had completed a CORE-OM during the last therapy session were also reviewed. The study sample experienced a median wait of 41.14 weeks between assessment and first session. The relationship between wait time from referral acceptance to assessment, and assessment GD score was not significant. During the period between assessment and first session no significant difference in GD score was observed. Nevertheless 29.1% of the sample experienced reliable change; 16.0% of clients reliably improved and 13.1% reliably deteriorated whilst waiting for therapy. Demographic factors were not found to have a significant effect on the change in GD score between assessment and first session. Waiting time was associated with post-therapy outcomes but not to a degree which was meaningful. The majority of individuals (54.4%), regardless of whether they improved or deteriorated whilst waiting for therapy, showed reliable improvement at end of therapy as measured by the CORE-OM. The majority of GD scores remained stable while waiting for therapy; however, 29.1% of secondary care clients experienced either reliable improvement or deterioration. Irrespective of whether they improved, deteriorated or remained unchanged whilst waiting for therapy, most individuals who had a complete end of therapy assessment showed reliable improvements following therapy. There was no significant difference in GD score between assessment and first session recordings. A proportion of clients (29.1%) showed reliable change, either improvement or deterioration, as measured by the GD score while waiting for therapy. Of the individuals with last session CORE-OMs (54.4%) showed significant improvement following therapy regardless of whether or not they experienced change while waiting for therapy. Limitations include: Problems of data quality, the data were from a routine data set and data were lost at each stage of the analysis. A focus on the CORE-OM limits exploration of the subjective experience of waiting for psychotherapy and the impact this has on psychological well-being.

Reach for gold

Duplicates in a database are one of the prime causes of poor data quality and are at the same time among the most difficult data quality problems to alleviate. To detect and remove such duplicates, many commercial and academic products and methods have been developed. The evaluation of such systems is usually in need of pre-classified results. Such gold standards are often expensive to come by (much manual classification is necessary), not representative (too small or too synthetic), and proprietary and thus preclude repetition (company-internal data). This lament has been uttered in many papers and even more paper reviews. The proposed annealing standard is a structured set of duplicate detection results, some of which are manually verified and some of which are merely validated by many classifiers. As more and more classifiers are evaluated against the annealing standard, more and more results are verified and validation becomes more and more confident. We formally define gold, silver, and the annealing standard and their maintenance. Experiments show how quickly an annealing standard converges to a gold standard. Finally, we provide an annealing standard for 750,000 CDs to the duplicate detection community.

Clinical data quality problems and countermeasure for real world study.

Real world study (RWS) has become a hotspot for clinical research. Data quality plays a vital role in research achievement and other clinical research fields. In this paper, the common quality problems in the RWS of traditional Chinese medicine are discussed, and a countermeasure is proposed.

Ocean Data Impacts in Global HYCOM*

Abstract The impact of the assimilation of ocean observations on reducing global Hybrid Coordinate Ocean Model (HYCOM) 48-h forecast errors is presented. The assessment uses an adjoint-based data impact procedure that characterizes the forecast impact of every observation assimilated, and it allows the observation impacts to be partitioned by data type, geographic region, and vertical level. The impact cost function is the difference between HYCOM 48- and 72-h forecast errors computed for temperature and salinity at all model levels and grid points. It is shown that routine assimilation of large numbers of observations consistently reduces global HYCOM 48-h forecast errors for both temperature and salinity. The largest error reduction is due to the assimilation of temperature and salinity profiles from the tropical fixed mooring arrays, followed by Argo, expendable bathythermograph (XBT), and animal sensor data. On a per-observation basis, the most important global observing system is Argo. The beneficial impact of assimilating Argo temperature and salinity profiles extends to all depths sampled, with salinity impacts maximum at the surface and temperature impacts showing a subsurface maximum in the 100–200-m-depth range. The reduced impact of near-surface Argo temperature profile levels is due to the vertical covariances in the assimilation that extend the influence of the large number of sea surface temperature (SST) observations to the base of the mixed layer. Application of the adjoint-based data impact system to identify a data quality problem in a geostationary satellite SST observing system is also provided.

Data Quality in Ontology-based Data Access: The Case of Consistency

Ontology-based data access (OBDA) is a new paradigm aiming at accessing and managing data by means of an ontology, i.e., a conceptual representation of the domain of interest in the underlying information system. In the last years, this new paradigm has been used for providing users with abstract (independent from technological and system-oriented aspects), effective, and reasoning-intensive mechanisms for querying the data residing at the information system sources. In this paper we argue that OBDA, besides querying data, provides the right principles for devising a formal approach to data quality. In particular, we concentrate on one of the most important dimensions considered both in the literature and in the practice of data quality, namely consistency. We define a general framework for data consistency in OBDA, and present algorithms and complexity analysis for several relevant tasks related to the problem of checking data quality under this dimension, both at the extensional level (content of the data sources), and at the intensional level (schema of the data sources).

Spatial inequalities in infant survival at an early stage of the longevity revolution

Background: Spatial inequalities in human development are of great concern to international organisations and national governments. Demographic indicators like the infant mortality rate are important measures for determining these inequalities. Using demographic indicators over long time periods at relatively high levels of geographical detail, we can examine the long-term continuities and changes in spatial inequalities. Objective: This paper presents the initial outcomes of a larger project that aims to analyse spatial variation in infant survival across Europe over the last 100 years. In this paper, we focus on spatial disparities in infant survival in 1910. At that time, the longevity revolution was still at an early stage. We look at general spatial variation patterns within and across countries, and discuss some of the challenges related to the comparativeness of the data. Methods: We link official infant mortality data from more than 5,000 European regions and localities for the period around 1910 to a European historical GIS of administrative boundaries. The data are analysed using descriptive spatial analysis techniques. Results: In 1910, a number of countries in northern and western Europe led the longevity revolution in Europe, with the area of low infant mortality also extending into the northwestern parts of the German Empire. Other areas with low infant mortality levels included the Belgian region of Wallonia, most parts of Switzerland, as well as central and south-western France. In eastern and southern Europe, we find significant variation within and across countries, which might stem in part from data quality problems

Data Quality Control in Biodiversity Informatics: The Case of Species Occurrence Data

For fighting the current environment sustainability crisis, several studies on biodiversity and the environment have been conducted. These studies are based on the assessment and monitoring of biodiversity by means of the collection, storage, analysis, simulation, modeling, visualization and sharing of a significant volume of biodiversity data in broad temporal and spatial scale. Species occurrences data are a particularly important type of biodiversity data because they are widely used in various studies. Nevertheless, for the analysis and modeling obtained from these data to be reliable, the data used must be high-quality. Thus, to improve the Data Quality (DQ) of species occurrences, the aim of this work was to conduct a study about DQ applied to species occurrences data that allowed assessing and improving DQ, using mechanisms to prevent errors. For the most important data domains identified (taxonomic, geospatial and location), a study on DQ Assessment was performed, in which important DQ dimensions (aspects) and problems that affect theses dimensions were identified, defined and interrelated. Based upon this study, DQ mechanisms were identified that would allow improving the DQ by reducing errors. Using the error-preventing approach, 13 mechanisms to support the prevention of 8 DQ problems were identified, thus providing an improvement of accuracy, precision, completeness, consistency and credibility of source of taxonomic, geospatial and location data of species occurrences. This work showed that with the development of certain computing mechanisms, preventing errors reduces DQ problems. As a result of reducing some problems in particular, the DQ in specific data domains is improved for certain DQ dimensions.

User Perception of Data and Medical Record Personalities

Data quality is an important and costly topic in modern society, and data quality and contextual data quality are important components of the human-computer relationship. As a long-term design imperative, the user interface needs to be improved to support better data quality. To design for better data quality, this exploratory study characterizes physicians’ opinions on their own data and the data produced by their colleagues. Through a regional survey and semi-structured interviews, the issue of data style and medical record personality was a dominant theme. A record’s personality impacts an individual user’s willingness to accept the data it contains. Future work will incorporate these results into modeling the data quality problem with cognitive work analysis, and will explore design elements to support the design for better data quality.

Data quality for data science, predictive analytics, and big data in supply chain management: An introduction to the problem and suggestions for research and applications

Today׳s supply chain professionals are inundated with data, motivating new ways of thinking about how data are produced, organized, and analyzed. This has provided an impetus for organizations to adopt and perfect data analytic functions (e.g. data science, predictive analytics, and big data) in order to enhance supply chain processes and, ultimately, performance. However, management decisions informed by the use of these data analytic methods are only as good as the data on which they are based. In this paper, we introduce the data quality problem in the context of supply chain management (SCM) and propose methods for monitoring and controlling data quality. In addition to advocating for the importance of addressing data quality in supply chain research and practice, we also highlight interdisciplinary research topics based on complementary theory.

Mining a database of single amplified genomes from Red Sea brine pool extremophiles-improving reliability of gene function prediction using a profile and pattern matching algorithm (PPMA).

Reliable functional annotation of genomic data is the key-step in the discovery of novel enzymes. Intrinsic sequencing data quality problems of single amplified genomes (SAGs) and poor homology of novel extremophile's genomes pose significant challenges for the attribution of functions to the coding sequences identified. The anoxic deep-sea brine pools of the Red Sea are a promising source of novel enzymes with unique evolutionary adaptation. Sequencing data from Red Sea brine pool cultures and SAGs are annotated and stored in the Integrated Data Warehouse of Microbial Genomes (INDIGO) data warehouse. Low sequence homology of annotated genes (no similarity for 35% of these genes) may translate into false positives when searching for specific functions. The Profile and Pattern Matching (PPM) strategy described here was developed to eliminate false positive annotations of enzyme function before progressing to labor-intensive hyper-saline gene expression and characterization. It utilizes InterPro-derived Gene Ontology (GO)-terms (which represent enzyme function profiles) and annotated relevant PROSITE IDs (which are linked to an amino acid consensus pattern). The PPM algorithm was tested on 15 protein families, which were selected based on scientific and commercial potential. An initial list of 2577 enzyme commission (E.C.) numbers was translated into 171 GO-terms and 49 consensus patterns. A subset of INDIGO-sequences consisting of 58 SAGs from six different taxons of bacteria and archaea were selected from six different brine pool environments. Those SAGs code for 74,516 genes, which were independently scanned for the GO-terms (profile filter) and PROSITE IDs (pattern filter). Following stringent reliability filtering, the non-redundant hits (106 profile hits and 147 pattern hits) are classified as reliable, if at least two relevant descriptors (GO-terms and/or consensus patterns) are present. Scripts for annotation, as well as for the PPM algorithm, are available through the INDIGO website.

Quality in Hospital Administrative Databases

The clinical content of administrative databases includes, among others, patient demographic characteristics, and codes for diagnoses and procedures. The data in these databases is standardized, clearly defined, readily available, less expensive than collected by other means, and normally covers hospitalizations in entire geographic areas. Although with some limitations, this data is often used to evaluate the quality of healthcare. Under these circumstances, the quality of the data, for instance, errors, or it completeness, is of central importance and should never be ignored. Both the minimization of data quality problems and a deep knowledge about this data (e.g., how to select a patient group) are important for users in order to trust and to correctly interpret results. In this paper we present, discuss and give some recommendations for some problems found in these administrative databases. We also present a simple tool that can be used to screen the quality of data through the use of domain specific data quality indicators. These indicators can significantly contribute to better data, to give steps towards a continuous increase of da ta quality and, certainly, to better informed decision-making.

About Sequence Quality: Impact on Clinical Applications

Sanger DNA sequencing is a robust and flexible technology and should have a crucial role in clinical practice for a long time. Nevertheless, in routine application of DNA sequencing, we are regularly confronted with sequence data quality problems. Surprisingly, we found that the definition of sequence quality is fuzzy and too empirical for many clinical applications. There are few studies or guidelines that directly address quality issues for Sanger sequencing in clinical situations. In addition, these use several combined parameters to ensure the sequence quality; this is too complicated to apply for daily use. Our heuristic analysis of nearly 46,000 sequence traces demonstrated that a combination of three basic parameters (average quality value, average sequence intensity, and electropherogram profile) was necessary and sufficient to determine accurately the quality of any sequence, even when deletions, insertions, and/or repeat sequence regions were present in a sequence trace. Therefore, we propose a simple and practical method with a diagram and decision making table for sequence quality determination in clinical sequencing.

Applying Control Chart Methods to Enhance Data Quality

As the volume and variety of available data continue to proliferate, organizations increasingly turn to analytics in order to enhance business decision-making and ultimately, performance. However, the decisions made as a result of the analytics process are only as good as the data on which they are based. In this article, we examine the data quality problem and propose the use of control charting methods as viable tools for data quality monitoring and improvement. We motivate our discussion using an integrated case study example of a real aircraft maintenance database. We include discussions of the measures of multiple data quality dimensions in this online process. We highlight the lack of appropriate statistical methods for the analysis of this type of problem and suggest opportunities for research in control chart methods within the data quality environment. This article has supplementary material online.

Data Quality in Data warehouse: problems and solution

In recent years, corporate scandals, regulatory changes, and the collapse of major financial institutions have brought much warranted attention to the quality of enterprise data if we can better understand the problems of quality issues, then we can develop a plan of action to address the problem that is both proactive and strategic. Each instance of a quality issue presents challenges in both identifying where problems exist and in quantifying the extent of the problems. Quantifying the issues is important in order to determine where our efforts should be focused. It is reported that more than $2 billion of U.S. federal loan money had been lost because of poor data quality at a single agency. It also reported that manufacturing companies spent over 25% of their sales on wasteful practices. Over the period of time many researchers have contributed to the data quality issues, but no research has collectively gathered all the causes of data quality problems at all the phases of data warehousing along with their possible solution. problems in different phase of data warehouse i.e.; data sources, data integration & data profiling, Data staging and ETL, data warehouse modeling & schema design are discussed in this paper. The purpose of the paper is to identify the reasons for data deficiencies, non-availability or reach ability problems at all the aforementioned stages of data warehousing and to give some classification of these causes as well as solution for improving data quality through Statistical Process Control (SPC),Quality engineering management . etc I have identified possible set of causes of data quality issues from the extensive literature review and with consultation of the data warehouse practitioners working in renowned IT company on India. I hope this will help developers & Implementers of warehouse to examine and analyze these issues before moving ahead for data integration and data warehouse solutions for quality decision oriented and business intelligence oriented applications.

A classification of data quality assessment and improvement methods

Data quality (DQ) assessment and improvement in larger information systems would often not be feasible without using suitable ‘DQ methods’, which are algorithms that can be automatically executed by computer systems to detect and/or correct problems in datasets. Currently, these methods are already essential, and they will be of even greater importance as the quantity of data in organisational systems grows. This paper provides a review of existing methods for both DQ assessment and improvement and classifies them according to the DQ problem and problem context. Six gaps have been identified in the classification, where no current DQ methods exist, and these show where new methods are required as a guide for future research and DQ tool development.

Exploring on-line societal risk perception for harmonious society measurement

The nearly 30-year economic growth miracle brings the consequent tremendous poor-rich gap leading strong drives for social transformation in current China. Chinese top leaders have realized to increase the peoples’ income, improve quality of life and construct a “harmonious society” as key missions especially in recent 10 years. How to measure a harmonious society is one important topic as different measures may lead to different development policies. This paper outlines over 10 indices relevant to measure a harmonious society. Some are global indicators, while some are contributed by domestic researchers and arouse debates. Most of those indicators require conducting surveys on social attitudes under micro levels, which is always time consuming with problem of data quality. As Internet technology advances provide ways to record and disseminate fresh community ideas and thoughts conveniently, detecting topics or emotions from on-line public opinions is becoming a trend or one supplement way to overcome those data acquisition problems. This paper discusses one approach to on-line societal risk perception using hot search words and BBS posts. Such a trial aims to provide another way to societal risk perception different from those in traditional socio psychology studies. Challenges are also indicated.

An observational study of alcohol-related readmission dynamics in young people

BackgroundAlcohol-related harm among young people in England is a substantial public health concern. Hospital Episode Statistics records for the NHS in England are a powerful yet flawed source of information about the development of problematic drinking careers of young people. The aim of this study was to assess whether these careers could be summarised and investigated after novel processing techniques had been applied to these records. These techniques attempted to account for problems in the data, specifically coding errors and splitting of relevant admission records between multiple annual datasets. MethodsA cohort of 15 203 young people aged 10–25 years in England who had an index alcohol-related emergency admission between April, 2003, and March, 2004, were tracked for subsequent alcohol-related readmission through the use of a pseudo-anonymised identification number for 3·75 years (data collection ended March, 2008). International Classification of Diseases 10 codes used in initial sample selection focused on those that indicated the immediate effects of excessive alcohol consumption (F10.0, F10.1, R78.0, T51.-, X45.-, X65.-, Y15.-, Y90.6, Y90.7, Y90.8, Y91.1, Y91.2, Y91.3, and Z72.1). Mean age of the cohort was 18·4 years (56·5% male). For those for whom ethnicity was known, just over 95% were white, although missing data makes this finding uncertain. Novel processing techniques were used to prepare data for analysis, whereby algorithms undertook record-by-record processing, with the intention of quality checking and appropriately combining incomplete or duplicate admission entries. The algorithms accomplished this preparation by applying standard data quality checks to each record, checking for repetition or incomplete or faulty records. By stepping through the dataset line-by-line, the algorithms were able to analyse a record in the context of adjacent, associated records. This method eliminated much uncertainly in the dataset and allowed a detailed picture of subsequent drinking careers to emerge. Confounding factors are inevitable in a cohort of this nature (in particular, dual diagnosis relating to mental health, self-harm, injury, assault, and other substance use) and were accounted for by incorporating them into the analysis as independent variables, along with other patient demographic variables. The resulting analysis dataset was subject to analysis in an attempt to distinguish the determinants of readmission. FindingsAlmost 13% (1950 of 15 203) of the cohort was readmitted during the following 3·75 years, and almost 5% (707 of 15 203) of the cohort had two or more alcohol-related readmissions. The characteristics of each return were investigated. On average, each subsequent admission featured a lengthened hospital stay, was progressively more likely to occur on a non-traditional drinking day, and occurred after a progressively smaller number of days since previous admission. The likelihood increased for a patient, as they cycled through subsequent readmissions, to be living in a neighbourhood of a lower average wealth than that where they lived during their index admission. InterpretationThe use of an automated method of summarising admission sequences proved feasible. The resulting findings can be used to inform the process of hospital screening for referral to alcohol treatment. Additionally, the deprivation finding can be interpreted as showing that poverty is both a part cause of, as well as being a result of, a problematic drinking career. Potential limitations of the study include (1) data quality problems that may have persisted despite the algorithms; (2) an absence of data about actual level of alcohol consumption; and (3) a focus on data that only capture people at specific periods when their drinking (often briefly) emerges in a clinical setting. Associated with this last point, an alternative explanation of the results is that the cohort offers an unrepresentative picture of problem drinking among young people, because the methodology excludes a group of unknown size who develop drinking problems, but who are never admitted (or readmitted) into hospital. FundingNone.

Analysis of data quality issues in real-world industrial data


 
 
 In large industries usage of advanced technological methods and modern equipment comes with the problem of storing, interpreting and analyzing huge amount of information. Handling information becomes more complicated and important at the same time. So, data quality is one of major challenges considering a rapid growth of information, fragmentation of information systems, incorrect data formatting and other issues. The aim of this paper is to describe industrial data processing and analytics on the real- world use case. The most crucial data quality issues are described, examined and classified in terms of Data Quality Dimensions. Factual industrial information supports and illustrates each encountered data deficiency. In addition, we describe methods for elimination data quality issues and data analysis techniques, which are applied after cleaning data procedure. In addition, an approach to address data quality problems in large-scale industrial datasets is proposed. This techniques and methods comprise several well-known techniques, which come from both worlds of mathematical logic and also statistics, improving data quality procedure and cleaning results.
 
 

Assessment of data quality in accounting data with association rules

Business rules are an effective way to control data quality. Business experts can directly enter the rules into appropriate software without error prone communication with programmers. However, not all business situations and possible data quality problems can be considered in advance. In situations where business rules have not been defined yet, patterns of data handling may arise in practice. We employ data mining to accounting transactions in order to discover such patterns. The discovered patterns are represented in form of association rules. Then, deviations from discovered patterns can be marked as potential data quality violations that need to be examined by humans. Data quality breaches can be expensive but manual examination of many transactions is also expensive. Therefore, the goal is to find a balance between marking too many and too few transactions as being potentially erroneous. We apply appropriate procedures to evaluate the classification accuracy of developed association rules and support the decision on the number of deviations to be manually examined based on economic principles.