Data Quality Problems Research Articles

Conducting conservation social science surveys online.

The coronavirus (COVID‐19) pandemic is affecting the environment and conservation research in fundamental ways. Many conservation social scientists are now administering survey questionnaires online, but they must do so while ensuring rigor in data collection. Further, they must address a suite of unique challenges, such as the increasing use of mobile devices by participants and avoiding bots or other survey fraud. We reviewed recent literature on online survey methods to examine the state of the field related to online data collection and dissemination. We illustrate the review with examples of key methodological decisions made during a recent national study of people who feed wild birds, in which survey respondents were recruited through an online panel and a sample generated via a project participant list. Conducting surveys online affords new opportunities for participant recruitment, design, and pilot testing. For instance, online survey panels can provide quick access to large and diverse samples of people. Based on the literature review and our own experiences, we suggest that to ensure high‐quality online surveys one should account for potential sampling and nonresponse error, design survey instruments for use on multiple devices, test the instrument, and use multiple protocols to identify data quality problems. We also suggest that research funders, journal editors, and policy makers can all play a role in ensuring high‐quality survey data are used to inform effective conservation programs and policies.

A task recommendation framework for heterogeneous mobile crowdsensing

Aiming at the problems of low data quality and high incentive costs caused by the low enthusiasm of participants in mobile crowd sensing, a new task recommendation framework is proposed in this paper. First, the participants' historical behaviors are analyzed, assuming that user behaviors can be quantified as the user's willingness to participate, and the cosine similarity theorem is used to calculate the similarity between participants, thereby constructing a user-hybrid model. Secondly, probabilistic matrix factorization is developed to predict the willingness of participants, and a ranking model is obtained through learn-to-rank algorithm. Finally, a task recommendation list is generated according to the ranking model, which serves as the target participant's preferred task list for sensing task recommendation. The experiment in this paper is carried out on the MATLAB platform based on two real check-in datasets, Gowalla and Brightkite. The results show that the average allocation precision rate can reach 96%, and the sensing user participation rate is about 97%. Meantime, the user's mobile cost is reduced, and the overall goal of maximizing accuracy and minimizing perceived cost is achieved.

Sea surface temperature prediction using a cubic B-spline interpolation and spatiotemporal attention mechanism

ABSTRACT Sea surface temperature (SST) prediction plays an important role in planning marine operations and forecasting climate. With the rapid development of remote sensing technology, there are plenty of SST data available for scientific research. However, most previous studies ignored the quality of SST dataset and the importance of data at each time step, which limited the performance of prediction. Therefore, in order to fully exploit the features of SST data, we propose a model which combines cubic B-spline interpolation, attention mechanism and Long Short Term Memory network (LSTM), named CBSA-LSTM. In this model, we use cubic B-spline interpolation to enhance input data and make SST trend curve continuous derivative in time dimension, adjust time attention mechanism to let it more suitable for SST prediction, refine spatial attention mechanism to mainly focus on latitude, and then combine them with LSTM to predict daily SST. To our knowledge, it is the first attempt to use cubic B-spline interpolation to solve the problem of data quality for SST prediction. The experiment results indicate that the proposed model significantly improve the prediction performance and is reliable for SST prediction with high performance for wide time range and large spatial scope.

Usability of India’s government statistics1

The quality of metadata is a crucial determinant of usability/interpretability of data. This paper draws attention to the poor quality of India’s government statistics and the paucity of metadata necessary to understand data quality problems. The paper suggests that there has been a decline in India both in terms of the availability and quality of metadata for key government sources of information including maps, decennial population censuses and National Sample Surveys amidst growing sophistication in the understanding of metadata. The poor quality of metadata impairs cross-sectional as well as inter-temporal comparisons and policymaking apart from concealing biases and lapses of government statisticians. The paper draws on the experience of three states – erstwhile Jammu and Kashmir, Manipur and Nagaland – where government statistics have been affected by serious errors that are not well-understood due to the lack of adequate metadata.

A beginner’s guide and best practices for using crowdsourcing platforms for survey research: The Case of Amazon Mechanical Turk (MTurk)

A beginner’s guide and best practices for using crowdsourcing platforms for survey research: The Case of Amazon Mechanical Turk (MTurk)

TGT: A Novel Adversarial Guided Oversampling Technique for Handling Imbalanced Datasets

With the volume of data increasing exponentially, there is a growing interest in helping people to benefit from their data regardless of its poor quality. One of the major data quality problems is the imbalanced distribution of different categories existing in the data. Such problem would affect the performance of any possible of analysis and mining on the data. For instance, data with an imbalanced distribution hasa negative effect on the performance achieved by most traditional classification techniques. This paper proposes TGT (Train Generate Test), a novel oversampling technique for handling imbalanced datasets problem. Using different learning strategies, TGT guarantees that the generated synthetic samples reside in minority regions. TGT showed a high improvement in performance of different classification techniques when was experimented with five imbalanced datasets of different types.

Towards an adapted PHM approach: Data quality requirements methodology for fault detection applications

Increasingly, extracting knowledge from data has become an important task in organizations for performance improvements. To accomplish this task, data-driven Prognostics and Health Management (PHM) is introduced as an asset performance management framework for data management and knowledge extraction. However, acquired data come generally with quality issues that affect the PHM process. In this context, data quality problems in the PHM context still an understudied domain. Indeed, the quality of the used data, their quantification, their improvement techniques and their adequacy to the desired PHM tasks are marginalized in the majority of studies. Moreover, many PHM applications are based on the development of very sophisticated data analysis algorithms without taking into account the adaptability of the used data to the fixed objectives. This paper aims to propose a set of data quality requirements for PHM applications and in particular for the fault detection task. The conducted developments in this study are applied to Scoder enterprise, which is a French SME. The feedback on the first results is reported and discussed.

Solution Ideas and Practices for Data Governance Engineering in Colleges and universities

Effectively improving and enhancing the data quality of colleges and universities is the most fundamental goal of university data governance. In this paper, first of all, the causes of data quality problems in colleges and universities are analyzed, and then the general idea of improving data quality is put forward, in which introduces how to identify data quality problems, how to solve data quality problems through technical and business means and how to make the data quality management of colleges and universities form a long-term mechanism through the construction of the system.

Понятие, классификация и методология измерения социально-экономического неравенства: проблемы и решения

Relevance. The challenges standing against the world economy because of the COVID-19 pandemic have required regulators to use unconventional financial policies to stimulate economic growth. This has had a certain impact on the dynamics of poverty and economic inequality. From this perspective, the topic is particularly relevant today and requires a large number of new questions to be answered. Research on socio-economic inequality involves many different terms and indicators, which are sometimes used incorrectly, creating some terminological and methodological confusion in the scientific discussion. In this regard, it is important to study all these theoretical problems and clarify the concept, classification of socio-economic inequalities, as well as issues related to the quality of data and its methods of assessment. The purpose of the work is to solve a number of theoretical and methodological problems related to the essence, classification and methodology of assessing the socio-economic inequality of households. Objectives: define the socio-economic inequality of households; propose a classification of types of socio-economic household inequality; justify the problem of completeness, quality and comparability of data in analysis and international comparisons of the level and dynamics of socio-economic inequality of households and suggest ways to solve it. Methodology. The study used methods of analysis, and synthesis of scientific papers and graphical analysis of statistical data. Results. the definition and classification of types of socio-economic inequality have been clarified; the problem of data quality, which makes it difficult to form an objective assessment of socio-economic inequality was justified: the problem of the complexity of international comparisons, going from the usage of different methodologies for assessing the subject of the study in different countries, is also justified. Conclusions. It is proposed to continue developing and enhancing the system of statistics and analysis of socio-economic inequality of households on the base of international standards and in closer international cooperation between countries, as this will allow to get a deeper understanding of economic inequality problem and the wok out adequate policies of state financial regulation.

A Novel Rank Aggregation-Based Hybrid Multifilter Wrapper Feature Selection Method in Software Defect Prediction.

The high dimensionality of software metric features has long been noted as a data quality problem that affects the performance of software defect prediction (SDP) models. This drawback makes it necessary to apply feature selection (FS) algorithm(s) in SDP processes. FS approaches can be categorized into three types, namely, filter FS (FFS), wrapper FS (WFS), and hybrid FS (HFS). HFS has been established as superior because it combines the strength of both FFS and WFS methods. However, selecting the most appropriate FFS (filter rank selection problem) for HFS is a challenge because the performance of FFS methods depends on the choice of datasets and classifiers. In addition, the local optima stagnation and high computational costs of WFS due to large search spaces are inherited by the HFS method. Therefore, as a solution, this study proposes a novel rank aggregation-based hybrid multifilter wrapper feature selection (RAHMFWFS) method for the selection of relevant and irredundant features from software defect datasets. The proposed RAHMFWFS is divided into two stepwise stages. The first stage involves a rank aggregation-based multifilter feature selection (RMFFS) method that addresses the filter rank selection problem by aggregating individual rank lists from multiple filter methods, using a novel rank aggregation method to generate a single, robust, and non-disjoint rank list. In the second stage, the aggregated ranked features are further preprocessed by an enhanced wrapper feature selection (EWFS) method based on a dynamic reranking strategy that is used to guide the feature subset selection process of the HFS method. This, in turn, reduces the number of evaluation cycles while amplifying or maintaining its prediction performance. The feasibility of the proposed RAHMFWFS was demonstrated on benchmarked software defect datasets with Naïve Bayes and Decision Tree classifiers, based on accuracy, the area under the curve (AUC), and F-measure values. The experimental results showed the effectiveness of RAHMFWFS in addressing filter rank selection and local optima stagnation problems in HFS, as well as the ability to select optimal features from SDP datasets while maintaining or enhancing the performance of SDP models. To conclude, the proposed RAHMFWFS achieved good performance by improving the prediction performances of SDP models across the selected datasets, compared to existing state-of-the-arts HFS methods.

A Two-Stream Dynamic Pyramid Representation Model for Video-Based Person Re-Identification.

Video-based person re-identification (Re-ID) leverages rich spatio-temporal information embedded in sequence data to further improve the retrieval accuracy comparing with single image Re-ID. However, it also brings new difficulties. 1) Both spatial and temporal information should be considered simultaneously. 2) Pedestrian video data often contains redundant information and 3) suffers from data quality problems such as occlusion, background clutter. To solve the above problems, we propose a novel two-stream Dynamic Pyramid Representation Model (DPRM). DPRM mainly consists of three sub-models, i.e., Pyramidal Distribution Sampling Method (PDSM), Dynamic Pyramid Dilated Convolution (DPDC) and Pyramid Attention Pooling (PAP). PDSM is applied for more effective data pre-processing according to sequence semantic distribution. DPDC and PAP can be considered as two streams to describe the motion context and static appearance of a video sequence, respectively. By fusing the two-stream features together, we finally achieve comprehensive spatio-temporal representation. Notably, dynamic pyramid strategy is applied throughout the whole model. This strategy exploits multi-scale features under attention mechanism to maximally capture the most discriminative features and mitigate the impact of video data quality problems such as partial occlusion. Extensive experiments demonstrate the outperformance of DPRM. For instance, it achieves 83.0% mAP and 89.0% Rank-1 accuracy on MARS dataset and reaches state of the art.

Calidad de Datos en Sistemas de Gestión Académica Universitaria basado en ISO/IEC 25012

This article proposes a data quality evaluation model developed on the primary basis of the ISO / IEC 25012 standard, applied to a University Academic Management System, to improve data quality. The proposed model is developed from the perspective of the data consumer and the vision of inherent data quality. The sample consisted of the data stored in the Academic Management System of the Universidad Nacional Micaela Bastidas, Apurímac, Perú, with 22 tables, 154 attributes, and 319,685 records. The model begins with data quality requirements as the main input for its evaluation and ends with an improvement plan, which is automatically implemented using data cleaning tools and SQL code. The characteristics that affect data quality problems are accuracy, consistency, compliance, and timeliness. Finally, it is concluded that it is possible to improve the quality of data by applying the proposed model, which can be used to create and generate value through the exploration, exploitation, and analysis of data for the benefit of university academic

Data quality-aware genomic data integration

Genomic data are growing at unprecedented pace, along with new protocols, update polices, formats and guidelines, terminologies and ontologies, which are made available every day by data providers. In this continuously evolving universe, enforcing quality on data and metadata is increasingly critical. While many aspects of data quality are addressed at each individual source, we focus on the need for a systematic approach when data from several sources are integrated, as such integration is an essential aspect for modern genomic data analysis. Data quality must be assessed from many perspectives, including accessibility, currency, representational consistency, specificity, and reliability.In this article we review relevant literature and, based on the analysis of many datasets and platforms, we report on methods used for guaranteeing data quality while integrating heterogeneous data sources. We explore several real-world cases that are exemplary of more general underlying data quality problems and we illustrate how they can be resolved with a structured method, sensibly applicable also to other biomedical domains. The overviewed methods are implemented in a large framework for the integration of processed genomic data, which is made available to the research community for supporting tertiary data analysis over Next Generation Sequencing datasets, continuously loaded from many open data sources, bringing considerable added value to biological knowledge discovery.

Balancing Practicality and Validity of Elder Abuse Measures: Data From Adult Protective Services Investigations

Research in Adult Protective Services suffers from the problems of small sample size, low completion rates, and poor data quality. To address the “small sample size” issue, data were generated on 1,472 elder abuse cases over six months in California using the Identification, Services and Outcomes (ISO) Matrix. The problem of poor data quality was addressed by testing psychometric properties of Short-Forms on each type of abuse. Good reliability and predictive validity were found for all measures except those that were very rare. Finally, even shorter Mini-Forms were developed and tested in order to contribute to improved completion rates. Mini-Form results were mixed, and some will require further research on their reliability and validity, as well as examination of their ability to improve completion rates. Ongoing adoption by two California counties and the State of Montana demonstrate sustainability and increasing sample sizes using the ISO Matrix for research and practice.

Anomalous Phylogenetic Behavior of Ribosomal Proteins in Metagenome-Assembled Asgard Archaea.

Metagenomic studies permit the exploration of microbial diversity in a defined habitat, and binning procedures enable phylogenomic analyses, taxon description, and even phenotypic characterizations in the absence of morphological evidence. Such lineages include asgard archaea, which were initially reported to represent archaea with eukaryotic cell complexity, although the first images of such an archaeon show simple cells with prokaryotic characteristics. However, these metagenome-assembled genomes (MAGs) might suffer from data quality problems not encountered in sequences from cultured organisms due to two common analytical procedures of bioinformatics: assembly of metagenomic sequences and binning of assembled sequences on the basis of innate sequence properties and abundance across samples. Consequently, genomic sequences of distantly related taxa, or domains, can in principle be assigned to the same MAG and result in chimeric sequences. The impacts of low-quality or chimeric MAGs on phylogenomic and metabolic prediction remain unknown. Debates that asgard archaeal data are contaminated with eukaryotic sequences are overshadowed by the lack of evidence indicating that individual asgard MAGs stem from the same chromosome. Here, we show that universal proteins including ribosomal proteins of asgard archaeal MAGs fail to meet the basic phylogenetic criterion fulfilled by genome sequences of cultured archaea investigated to date: These proteins do not share common evolutionary histories to the same extent as pure culture genomes do, pointing to a chimeric nature of asgard archaeal MAGs. Our analysis suggests that some asgard archaeal MAGs represent unnatural constructs, genome-like patchworks of genes resulting from assembly and/or the binning process.

Data quality problems troubling business and financial researchers: A literature review and synthetic analysis

The data quality of commercial business and financial databases greatly affects research quality and reliability. The presence of data quality problems can not only distort research results, destroy a research effort but also seriously damage management decisions based upon such research. Although library literature rarely discusses data quality problems, business literature reports a wide range of data quality issues, many of which have been systematically tested with statistical methods. This article reviews a collection of the business literature that provides a critical analysis on the data quality of the most frequently used business and finance databases including the Center for Research in Security Prices (CRSP), Compustat, S&P Capital IQ, I/B/E/S, Datastream, Worldscope, Securities Data Company (SDC) Platinum, and Bureau van Dijk (BvD) Orbis and identifies 11 categories of common data quality problems, including missing values, data errors, discrepancies, biases, inconsistencies, static header data, standardization, changes in historic data, lack of transparency, reporting time issues and misuse of data. Finally, the article provides some practical advice for librarians to facilitate their scholarly communications with researchers on data quality problems.

Homogenization of daily time series climatological data in the Eastern Nile basin, Ethiopia

A major challenge to climate-related research is the lack of a well-documented historical climate dataset. Unfortunately, most of the climate databases in developing countries have a critical data quality problem (missing data in time series and inhomogeneity). This study aimed to prepare a complete and homogenized climate dataset for the Tekeze river basin in Ethiopia and to assess the impact of detected inhomogeneity in major climate characteristics. Satellite and reanalysis climate products were used to fill in the missing data in the time series after a thorough evaluation and bias correction to the latter dataset. Together with other quality control approaches, the Multiple Analysis of Series for Homogenization (MASH) software was used for climate data homogenization. Results show that both minimum temperature (Tmin) and maximum temperature (Tmax) have a large number of breakpoints compared to rainfall. After homogenization, the original raw climate data was compared with the homogenized data. As the results show, the detected inhomogeneity causes a significant error to the region’s climate characteristics. For instance, following homogenization process, some stations have shown a change in trend magnitude from – 122 mm/decade to 0.2 mm/decade, from 1.2 °C/decade to 0.3 °C/decade, and from 1.5 °C/decade to 0.6 °C/decade for rainfall, Tmin, and Tmax respectively. The temperature trend resulted from the homogenized dataset is by far in agreement with trend results provided at the country level (nearly 0.3 °C to 0.4 °C/decade). Therefore, there is an urgent need of documenting a complete and homogenized historical climate dataset as a prerequisite for reliable climate-related studies.

Rule-based Data Quality Intelligent Monitoring System

The rapid development of Internet information technology brought great convenience to people’s lives and work, but it also brought the problem of information overload[l]. The data center is the middle layer that mediates the contradiction between the foreground and the background[2]. It provides a reusable, standardized, and agile multifunctional platform through the modeling of background data and the aggregation of data services to support the foreground for the demand[3] of quickly change according to market changes. Through the data center, we have opened up the data islands of each business system. However, when each business system uses data, it found that the data quality is poor, unstable, not timely, and inconsistent. Traditional cluster data quality monitoring has the characteristics of inconsistent data quality requirements for various services, difficult development, low processing efficiency, difficult maintenance, and insufficient intelligence.Based on this situation, we design a data quality monitoring system based on rules and artificial intelligence. General users perform simple configuration on the interface to complete real-time online monitoring of data, and real-time statistics on the number of data quality problems and a detailed list for users to trace the cause of the problem and identify abnormal scenarios in advance. The comparison between experiments and mainstream products proves that the system has greatly improved in terms of user use and flexibility.

Automatic Deep Inference of Procedural Cities from Global-scale Spatial Data

Recent advances in big spatial data acquisition and deep learning allow novel algorithms that were not possible several years ago. We introduce a novel inverse procedural modeling algorithm for urban areas that addresses the problem of spatial data quality and uncertainty. Our method is fully automatic and produces a 3D approximation of an urban area given satellite imagery and global-scale data, including road network, population, and elevation data. By analyzing the values and the distribution of urban data, e.g., parcels, buildings, population, and elevation, we construct a procedural approximation of a city at a large-scale. Our approach has three main components: (1) procedural model generation to create parcel and building geometries, (2) parcel area estimation that trains neural networks to provide initial parcel sizes for a segmented satellite image of a city block, and (3) an optional optimization that can use partial knowledge of overall average building footprint area and building counts to improve results. We demonstrate and evaluate our approach on cities around the globe with widely different structures and automatically yield procedural models with up to 91,000 buildings, and spanning up to 150 km 2 . We obtain both a spatial arrangement of parcels and buildings similar to ground truth and a distribution of building sizes similar to ground truth, hence yielding a statistically similar synthetic urban space. We produce procedural models at multiple scales, and with less than 1% error in parcel and building areas in the best case as compared to ground truth and 5.8% error on average for tested cities.

Treatment of Bad Big Data in Research Data Management (RDM) Systems

Databases such as research data management systems (RDMS) provide the research data in which information is to be searched for. They provide techniques with which even large amounts of data can be evaluated efficiently. This includes the management of research data and the optimization of access to this data, especially if it cannot be fully loaded into the main memory. They also provide methods for grouping and sorting and optimize requests that are made to them so that they can be processed efficiently even when accessing large amounts of data. Research data offer one thing above all: the opportunity to generate valuable knowledge. The quality of research data is of primary importance for this. Only flawless research data can deliver reliable, beneficial results and enable sound decision-making. Correct, complete and up-to-date research data are therefore essential for successful operational processes. Wrong decisions and inefficiencies in day-to-day operations are only the tip of the iceberg, since the problems with poor data quality span various areas and weaken entire university processes. Therefore, this paper addresses the problems of data quality in the context of RDMS and tries to shed light on the solution for ensuring data quality and to show a way to fix the dirty research data that arise during its integration before it has a negative impact on business success.