Data Quality Problems Research Articles

Discovery of MicroDependencies

Data quality rules compose a class of frequently used tools whose purpose is to improve data quality: examples of these rules include functional dependences, conditional functional dependences, and editing rules, among others. Dependences with stronger expressivity can help to detect more data errors and impute more missing values. To the best of our knowledge, most existing rules consider each attribute as an inseparable whole. In many applications, however, a part of a value contains useful information, and more powerful rules can thus be formed to handle data quality problems. In this paper, we aim to discover the rules of this type, namely, microDependences. The left-hand side of a microDependence is a star-free regular expression, along with the positions of the partial information. This indicates that if a string-type attribute value matches the regular expression, elements at the specified positions can determine another attribute's value. To discover microDependences, strings with similar forms should be clustered together. Moreover, similar strings should be aligned vertically to shift elements with a similar meaning to the same position. Then, microDependences can be discovered directly or by the existing methods. Both the clustering and aligning tasks are challenging and play key roles in discovering microDependences. A greedy bottom-up framework is proposed to do the clustering and aligning work simultaneously. For efficiency, several pruning strategies are proposed to reduce the time consumed. In the experimental study, our methods' performances are verified on both real and synthetic data sets.

Prior Management of Temporal Data Quality in a Data Mining Process : an Implementation Architecture

In Data Mining (DM) projects, specifically in the Data Understanding and the Data Preparation phases, several techniques defined in the literature are used to detect and handle noisy data (missing data, outliers, etc.). However, the application of such techniques remains complex and time-consuming due to the lack of anticipation in the detection and resolution of data quality problems. As a result, more than half of DM projects’ effort is spent on data preparation. This work supports an approach for reducing the time devoted to data preparation by advocating an early management of data quality problems. It’s an improvement of a previous work and proposes an implementation architecture based on an anticipative approach for managing temporal data quality.

Rule-Based Data Quality Assessment and Monitoring System in Healthcare Facilities.

Measuring and managing data quality in healthcare has remained largely uncharted territory with few notable exceptions. A rules-based approach to data error identification was explored through compilation of over 6,000 data quality rules used with healthcare data. The rules were categorized based on topic and logic yielding twenty-two rule templates and associated knowledge tables used by the rule templates. This work provides a scalable framework with which data quality rules can be organized, shared among facilities and reused. The ten most frequent data quality problems based on the initial rules results are identified. While there is significant additional work to be done in this area, the exploration of the rule template and associated knowledge tables approach here shows rules-based data quality assessment and monitoring to be possible and scalable.

Tool for validation and import in herbarium database

Abstract Many biological collections databases feature data quality problems. On the existing computational resources, we present an import tool and data validation. The program applies filters to data submitted through a spreadsheet at the time of data import, streamlining the error-checking process. The validations presented were divided into three categories according to the taxonomic, geographical and general specimen collection data. Its implementation eliminated the errors in the data entry of new vouchers in the Herbarium of the Botanical Garden of Rio de Janeiro.

VafusQ: A methodology to build visual analysis applications with data quality features

The use of interactive applications to support the decision-making process is more common every day. However, a huge amount of data is required in order to make more informed decisions. Fortunately, with the arrival of new technologies there are many data sources available. This requirement of data causes heterogeneity and data quality problems. A set of data quality problems are reduced in the preprocessing stage. However, many data quality issues persist after the preprocessing stage. For this reason, we proposed a methodology to take the data quality problems, to represent them and simultaneously support the analysis process. In addition, an application is developed as a use case of the methodology by analyzing the public transport system in Bogotá. Furthermore, a case study is performed to test the usefulness of the developed application. As a result, the methodology made possible the development of interactive visualizations that constitute an application that is useful to achieve the analysis tasks by including data quality features.

Quality of input data in emergency department simulations: Framework and assessment techniques

Operations research techniques are widely used to analyse and optimise emergency department operations. The complex and stochastic nature of an emergency department makes simulation a suitable and frequently used technique. Simulation can provide valuable insights to hospital managers on how to improve the efficiency of an emergency department. However, the output of the simulation study is only as reliable as the input data used as basis for simulation modelling. As a result, high quality input data are essential for the construction of a realistic simulation model. This paper provides a data quality framework that categorises possible data quality problems in electronic healthcare records of emergency departments. Electronic healthcare records are a common source of input data for emergency department simulations, but often suffer from data quality issues. For the data quality problems identified in the framework, data quality assessment techniques are described. These techniques enable researchers and practitioners to identify and quantify the potential data quality issues present in input data. In order to facilitate data quality assessment, an implementation to automate this process is developed and applied to a real-life case study. This case study demonstrates the need for thorough and structured data quality assessment. Possible ways to deal with identified data quality problems are also described.

Poor Quality of Data in Africa: What Are the Issues?

AbstractOne of the problems that scholars encounter when conducting research is that data from Africa are poor. Indeed, they either do not exist in a complete sense or they are not of good quality in the sense of lacking validity and reliability. Using various sources of available data on corruption and tourism, we show that lack of quality data in Africa not only limits the continent’s ability to generate a pertinent body of knowledge but also prevents analysts from generating the evidence that policy makers need to make proper decisions in influencing the development of the continent. We go on to argue that the problem of data quality in Africa is due to the lack of research culture rather than just scarcity of resources, as argued in the literature. It is in this context that we advocate for inculcation of research culture in Africa for development of the continent.Related Articles Asongu, Simplice A., and Jacinta C. Nwachukwu. 2016. “.” Politics & Policy 44 (): 916‐944. https://doi.org/10.1111/polp.12171 Pelizzo, Riccardo, and Abel Kinyondo. 2014. “.” Politics & Policy 42 (): 77‐102. https://doi.org/10.1111/polp.12062 Pelizzo, Riccardo, and Zim Kinyondo. 2016. “.” Politics & Policy 44 (): 1017‐1052. https://doi.org/10.1111/polp.12188 Related Media . 2013. “.” https://www.youtube.com/watch?v=ypGikIFf8LM . 2018. “.” https://www.afrobarometer.org/ . 2018. “.” https://allafrica.com/ . 2018. “.” https://mo.ibrahim.foundation/

An empirical examination of the determinants of audit report delay in France

PurposeThis study aims to investigate the determinants of delay in publishing audited reports.Design/methodology/approachThe research is conducted on a sample of French listed companies, covering the period of five years (from 2010 to 2014). The authors use pooled ordinary least squares regression analysis, modeling audit delay as a function of the following explanatory variables relating to the attributes of companies and their auditors.FindingsA statistically significant association is found between audit delay and type of audit firm, audit opinion, firm size, the month of year-end and profitability. The results suggest that audit delay lag is reduced by appointing an international audit firm but is extended by aspects of qualified audit opinion.Originality/valueThe contribution of this paper is to investigate audit report in a developed capital market by taking advantage of access to proprietary data on audit timing and audit opinion. This allowed to overcome some of the problems of data quality that inhibit importing research methods from France to other advanced capital markets.

Performance Evaluation of Various Missing Data Treatments in Crash Severity Modeling

Data quality, including record inaccuracy and missingness (incompletely recorded crashes and crash underreporting), has always been of concern in crash data analysis. Limited efforts have been made to handle some specific aspects of crash data quality problems, such as using weights in estimation to take care of unreported crash data and applying multiple imputation (MI) to fill in missing information of drivers’ status of attention before crashes. Yet, there lacks a general investigation of the performance of different statistical methods to handle missing crash data. This paper is intended to explore and evaluate the performance of three missing data treatments, which are complete-case analysis (CC), inverse probability weighting (IPW) and MI, in crash severity modeling using the ordered probit model. CC discards those crash records with missing information on any of the variables; IPW includes weights in estimation to adjust for bias, using complete records’ probability of being a complete case; and MI imputes the missing values based on the conditional distribution of the variable with missing information on the observed data. Those missing data treatments provide varying performance in model estimations. Based on analysis of both simulated and real crash data, this paper suggests that the choice of an appropriate missing data treatment should be based on sample size and data missing rate. Meanwhile, it is recommended that MI is used for incompletely recorded crash data and IPW for unreported crashes, before applying crash severity models on crash data.

Efficient histogram-based range query estimation for dirty data

In recent years, data quality issues have attracted wide attentions. Data quality problems are mainly caused by dirty data. Currently, many methods for dirty data management have been proposed, and one of them is entity-based relational database in which one tuple represents an entity. The traditional query optimizations are not suitable for the new entity-based model. Then new query optimizations need to be developed. In this paper, we propose a new query selectivity estimation strategy based on histogram, and focus on solving the overestimation which traditional methods lead to. We prove our approaches are unbiased. The experimental results on both real and synthetic data sets show that our approaches can give good estimates with low error.

Data Quality in Banking System: Case of Azerbaijan

Data quality in banking and financial sector is one of the most researched topics nowadays. With the increasing regulatory burden and increased importance of targeted sales, data quality directly influences funds and performance of banking system. In this paper, the author is aiming to define universal reasons and causes of data quality problem and apply the case to local Azerbaijani banks taking into account local managers’ personal view based on their banking experience. Key finding of the research is that unintegrated software, wrong data insertion, aging of data with the growing speed of market, corporate governance and inability to calculate true costs of low data quality to the local banks are the reasons of data quality issue in the local banks. Moreover, main costs of the data quality issue are time and money, appearance of hidden data factories, obstacles to apply and measure KPIs, uncorrelations in sensitivity analysis and ineffective marketing strategies.

A game theory approach for cooperative control to improve data quality and false data detection in WSN

SummaryNew solutions are required for the management of heterogeneous distributed sensor networks in order to address the problem of data quality and false data detection in wireless sensor networks (WSNs). In this paper, we present a nonlinear cooperative control algorithm based on game theory. Here, a new model is proposed for the automatic processing and management of information in heterogeneous distributed WSNs. We apply our algorithm to a case study with the aim of improving the quality of temperature data collected from indoor surfaces by a WSN. Unlike the classic unsupervised methods, in the proposed algorithm, it is not necessary to define the number of clusters beforehand. Once the game reaches the game equilibrium, the resulting number of clusters can be used as input for the unsupervised classification analysis. Anomalous temperature values are corrected according to their neighborhood, without modifying the temperature clusters.

Recent advances in driving mechanisms of the Arctic amplification: A review

Under the influence of recent global warming, the Arctic climate is continuing to experience unprecedented changes. Surface temperatures in the Arctic are increasing at a rate double that of the global average, known as the Arctic amplification. The Arctic amplification has led to a series of consequences locally such as the accelerated sea ice retreat, the continuous permafrost melting and methane release, and the marine and terrestrial ecological dynamics. It also might cause an increased frequency of extreme weather event in mid-latitudes via perturbing the strength of the polar vortex. The Arctic warming is mainly resulted from the imbalance of energy budget at both surface and top of atmosphere. In order to explore the driving mechanism of Arctic amplification, lots of efforts have been made based on model simulations and observations. Several major competing theories and related evidences have been proposed to explain their contributions to the imbalance of energy budget over the Arctic. The sea-ice albedo feedback, caused by the continuously shrinking summer sea ice potentially increasing the absorption of shortwave radiation, is believed to play a critical role in recent Arctic amplification. But certain model simulations have indicated that sea-ice albedo feedback was likely not the dominant factor—robust warming amplification still occurs in the Arctic in the absence of albedo feedback. The positive lapse rate feedback, is also thought to cause greater warming at the Arctic surface through damping the deep convection process at the bottom atmosphere because of the strong temperature inversion in the cold Arctic. However, whether the lapse rate feedback plays a dominant role in the Arctic amplification has not been widely accepted. The Planck feedback—cold region requires larger temperature increase than warm area to balance certain external radiative forcing, is also believed to contribute to the Arctic amplification. But the magnitude of its contribution remains to be further quantified. An enhanced Atlantic meridional overturning circulation (AMOC) transporting extraordinary amounts of heat northward to the Arctic Ocean has been hypothesized to substantially amplify Arctic warming and accelerate summer sea-ice melting in this region. However, evidence has instead shown significant slowing of the AMOC over the last few decades. The enhanced greenhouse effect, resulted from the increased water vapor and cloudiness, has been suggested as an important driver of the Arctic winter warming and summer sea-ice dynamics. However, whether the increased water vapor comes from the strengthened sea surface evaporation or from an enhanced atmospheric northward transportation, is still unclear. This paper elucidates the theoretical basis of all potential mechanisms, and systematically reviews the research progress of these theories. Based on the comprehensive analysis of these studies, we point out that due to the limitation of current study approaches, the data quality problems such as scarce in situ measurements, difficulty in discriminating clouds from snow cover for some remote sensing products, problematic parameterization schemes and model inputs in some reanalsysis and GCM projects, and the lack of systematic analysis for investigating the internal relations between different climate variables, the conclusions from different studies are discrepant, or even contradictory. The causes of the unprecedented climate change in the Arctic remain unclear and are still under heated debate. Therefore, the development of high quality, long-term radiative product, the optimization of the researching approaches and more accurate quantification of these feedback mechanisms, and the systematic analysis of the potential linkage between different climate variables, will help to advance the study of the Arctic amplification and climate change.

A Data Quality Control Method for Seafloor Observatories: The Application of Observed Time Series Data in the East China Sea.

With the construction and deployment of seafloor observatories around the world, massive amounts of oceanographic measurement data were gathered and transmitted to data centers. The increase in the amount of observed data not only provides support for marine scientific research but also raises the requirements for data quality control, as scientists must ensure that their research outcomes come from high-quality data. In this paper, we first analyzed and defined data quality problems occurring in the East China Sea Seafloor Observatory System (ECSSOS). We then proposed a method to detect and repair the data quality problems of seafloor observatories. Incorporating data statistics and expert knowledge from domain specialists, the proposed method consists of three parts: a general pretest to preprocess data and provide a router for further processing, data outlier detection methods to label suspect data points, and a data interpolation method to fill up missing and suspect data. The autoregressive integrated moving average (ARIMA) model was improved and applied to seafloor observatory data quality control by using a sliding window and cleaning the input modeling data. Furthermore, a quality control flag system was also proposed and applied to describe data quality control results and processing procedure information. The real observed data in ECSSOS were used to implement and test the proposed method. The results demonstrated that the proposed method performed effectively at detecting and repairing data quality problems for seafloor observatory data.

Linked Data: what happens to heterogeneity and interoperability

Linked Data – LD, allows the construction of the Web of Data, which use basic principles that contribute to the growth of the Web. The process developed by LD faces different scenarios that seem to limit the linking process. This paper exposes the problems of heterogeneity, interoperability and data quality, as factors that restrict the process of data linkage. For this, the methodological design performs a bibliographic exploration; the results examine and discuss the postures about the mentioned problems, and the possible incidence of these in the quality of the linked data. In conclusion, speaking about heterogeneity is intrinsic, since the information will always be heterogeneous. Data models may be different but if the data format is common, interoperability is possible. Data models must be known and public, in order to avoid problems that prevent the development of the Web Data.

Adequate life-expectancy reconstruction for adult human mortality data.

Mortality information of populations is aggregated in life tables that serve as a basis for calculation of life expectancy and various life disparity measures. Conventional life-table methods address right-censoring inadequately by assuming a constant hazard in the last open-ended age group. As a result, life expectancy can be substantially distorted, especially in the case when the last age group in a life table contains a large proportion of the population. Previous research suggests addressing censoring in a gamma-Gompertz-Makeham model setting as this framework incorporates all major features of adult mortality. In this article, we quantify the difference between gamma-Gompertz-Makeham life expectancy values and those published in the largest publicly available high-quality life-table databases for human populations, drawing attention to populations for which life expectancy values should be reconsidered. We also advocate the use of gamma-Gompertz-Makeham life expectancy for three reasons. First, model-based life-expectancy calculation successfully handles the problem of data quality or availability, resulting in severe censoring due to the unification of a substantial number of deaths in the last open-end age group. Second, model-based life expectancies are preferable in the case of data scarcity, i.e. when data contain numerous age groups with zero death counts: here, we provide an example of hunter-gatherer populations. Third, gamma-Gompertz-Makeham-based life expectancy values are almost identical to the ones provided by the major high-quality human mortality databases that use more complicated procedures. Applying a gamma-Gompertz-Makeham model to adult mortality data can be used to revise life-expectancy trends for historical populations that usually serve as input for mortality forecasts.

Life prediction of the tensile damage progress for high-speed train gearbox shell based on acoustic emission sensor and an automatic optimization method

The high-speed train gearbox is one of the key components of the train system, and its working state is related to passengers’ safety. The gearbox shell is the protective shell for gears under harsh and complex service environment. Instantaneous impact of hard material during failure can lead to the rapid damage accumulation and fracture of the shell material. In this article, the accumulation of tensile damage of the high-speed train gearbox shell material has been studied. For the short tensile process leading to significant amount of damage, a real-time and non-destructive detection method has been used to monitor the tensile damage progression and predict the residual life of the material. An automatic optimization algorithm has been proposed to deal with data quality problems such as fluctuations, imbalances, and large intervals. In addition, an automatic life prediction optimization model for material tensile process has been established. The life prediction errors are controlled within 50 s, and the majority of errors are less than 20 s. For the accelerated test, in respect to real situation, a 6 h time slot is designed for disaster control such as passenger evacuation and train failure prevention.

Data Quality Guarantee for Credible Caching Device Selection in Mobile Crowdsensing Systems

Mobile crowdsensing systems (MCSs) present a flexible and economical alternative to traditional infrastructure-based large-scale sensing through the recruitment of personal mobile devices as data sources. As this becomes a popular sensing approach, it will impact the capacity of typical centralized cellular communication infrastructures widely adopted by MCS applications and any costs accrued. Following the trend toward edge processing, mobile edge caching offloads data and services from the system core to reduce service latency and bandwidth occupation. However, in the MCS case the edge devices are owned by the general public and are therefore more vulnerable to data or calculation manipulation by the user. We now better understand sensor data and user trustworthiness but have no way to determine which devices can also be trusted (i.e., act as a credible caching device). In this article, we treat the quality of sensing data reported by each user as an indication of their possibility of providing credible caching services. Specifically, we conduct a comprehensive study of the data quality problem with regard to cache-enabled MCSs, and develop an incentivization method to encourage users to actively provide high-quality data. That is, quality-aware behavior evaluation is core to the credible caching device selection process. Results of extensive simulations based on realworld data verify the effectiveness of our design. We also highlight several promising research directions that remain open for further elaboration.

A person-centred approach in nursing: Validity and reliability of the Carer Support Needs Assessment Tool

The Carer Support Needs Assessment Tool (CSNAT) was developed for use among family caregivers in palliative care for assessment of their support needs. The purpose of this study was to translate and evaluate the validity and reliability of the CSNAT in a sample of Swedish family caregivers and nurses in a palliative care context. Data for this validation study was collected during 2016 in the context of palliative home care in two larger Swedish cities. The study was conducted in three stages to reach conceptual, semantic, operational and measurement equivalence between the original UK version and the Swedish version. Stage I consisted of translation to Swedish. In Stage II, cognitive interviews were performed with 8 family caregivers and 10 nurses. Data were analyzed based on relevance, clarity and sensitivity. In Stage III, the CSNAT and related self-rating measures (caregiver burden, preparedness for caregiving and quality of life) were completed by 118 family caregivers. Data quality, construct validity and test-retest reliability were evaluated. The CSNAT items were considered relevant and useful to identify areas of support needs. The Swedish CSNAT showed sound psychometric properties with satisfactory data quality and few problems with missing data across items (1.8%-6.1%). All items except one correlated as expected (rho>0.3) with caregiver burden, supporting construct validity. All items had satisfactory test-retest reliability (κw=0.45-0.75). This study further adds to the validity of the CSNAT and shows in addition that it is reliable and stable for use among family caregivers in palliative care.

The impact of routine data quality assessments on electronic medical record data quality in Kenya.

BackgroundRoutine Data Quality Assessments (RDQAs) were developed to measure and improve facility-level electronic medical record (EMR) data quality. We assessed if RDQAs were associated with improvements in data quality in KenyaEMR, an HIV care and treatment EMR used at 341 facilities in Kenya.MethodsRDQAs assess data quality by comparing information recorded in paper records to KenyaEMR. RDQAs are conducted during a one-day site visit, where approximately 100 records are randomly selected and 24 data elements are reviewed to assess data completeness and concordance. Results are immediately provided to facility staff and action plans are developed for data quality improvement. For facilities that had received more than one RDQA (baseline and follow-up), we used generalized estimating equation models to determine if data completeness or concordance improved from the baseline to the follow-up RDQAs.Results27 facilities received two RDQAs and were included in the analysis, with 2369 and 2355 records reviewed from baseline and follow-up RDQAs, respectively. The frequency of missing data in KenyaEMR declined from the baseline (31% missing) to the follow-up (13% missing) RDQAs. After adjusting for facility characteristics, records from follow-up RDQAs had 0.43-times the risk (95% CI: 0.32–0.58) of having at least one missing value among nine required data elements compared to records from baseline RDQAs. Using a scale with one point awarded for each of 20 data elements with concordant values in paper records and KenyaEMR, we found that data concordance improved from baseline (11.9/20) to follow-up (13.6/20) RDQAs, with the mean concordance score increasing by 1.79 (95% CI: 0.25–3.33).ConclusionsThis manuscript demonstrates that RDQAs can be implemented on a large scale and used to identify EMR data quality problems. RDQAs were associated with meaningful improvements in data quality and could be adapted for implementation in other settings.