Data Quality Problems Research Articles

Evaluating User Acceptance of Mobile Payment Systems in Uganda

Purpose: This research evaluates user acceptance of mobile payment systems. Methodology: This study adopted a desk methodology. A desk study research design is commonly known as secondary data collection. This is basically collecting data from existing resources preferably because of its low cost advantage as compared to a field research. Our current study looked into already published studies and reports as the data was easily accessed through online journals and libraries. Findings: Combining machine learning with DW systems provides significant advantages in different fields. This synergy boosts analytical aptitudes, allowing the organization to go a notch higher than descriptive analytics in predictive and prescriptive analytics. However, such a decision is not simple as it has implementation matters such as data quality problems, scalability, and real-time processing problems. Integration best practices include in-database machine learning processing, a feature store, and proper MLOps practices. Real-life examples from the healthcare industry, banking and financial services, retail, and manufacturing industries show that this integration brings operational enhancements for the business and positive effects on customers and overall organizational performance. Recommendations: Mobile payment providers should prioritize user-centric design principles, focusing on simplicity, usability, and intuitive interfaces. Policymakers should develop regulatory frameworks that establish clear standards for mobile payment systems, focusing on data protection, privacy, and security. Keywords: User Acceptance, Mobile Payment Systems

Non-negative least-squares variance and covariance component estimation using the positive-valued function for errors-in-variables models

Abstract Although (co)variance component estimation has been widely applied in the errors-in-variables (EIV) model, the occurrence of negative variance components is still a major issue in the estimated variance components. This problem may be due to the following unfavorable factors: 1) unreasonable selection of initial variance values; 2) low redundancy in the EIV functional model; 3) improper design in the EIV stochastic model, and 4) other data quality problems. Many attempts have been made to prevent the appearance of negative variance components. In this contribution, a novel and efficient non-negative least-squares variance component estimation using the PVF (PVF-NLS-VCE) is introduced, which can simultaneously estimate the unknown (co)variance components in the EIV stochastic model and the parameters in the EIV functional model. Its principle is to implicitly impose a non-negative constraint by replacing the variance component with the positive-valued function (PVF) whose range is the set of positive real numbers. Two numerical examples using real and simulated data are presented. The numerical results of linear regression are identical to those obtained based on least-squares variance component estimation (LS-VCE) with positive initial values of variance components. The numerical results of two-dimensional affine transformation are shown to prevent negative variance components and precede those obtained by LS-VCE with a negative initial value of variance component. Both numerical examples verify the effectiveness of the PVF-NLS-VCE method whether the initial values of variance components are positive or negative. The proposed PVF-NLS-VCE is a simple, convenient and flexible method to achieve the non-negative estimates of variance components, which can reduce sensitivity to initial value selection and automatically guarantee a non-negative definite covariance matrix.

The use of locum doctors in the NHS: understanding and improving the quality and safety of care.

The use of locum doctors in the National Health Service is widely believed to have increased, and there have been widespread and sustained concerns among policy-makers, healthcare providers, professional associations and professional regulators about the quality/safety, cost and effective use of locum doctors. To provide evidence on the extent, quality and safety of medical locum practice and the implications of medical locum working for health service organisation and delivery in primary and secondary care in the English National Health Service, to support policy and practice. Four interlinked work packages involving surveys of National Health Service trusts and of general practices in England; semistructured interviews and focus groups across 11 healthcare organisations in England; analysis of existing routine data sets on the medical workforce in primary care and in National Health Service trusts in England from National Health Service Digital and National Health Service Improvement; and analysis of data from the Clinical Practice Research Datalink in primary care and of electronic patient record data from two National Health Service hospitals in secondary care. In primary care, about 6% of general practice medical consultations were undertaken by locums in 2010 and this had risen slightly to about 7.1% in 2021. In National Health Service trusts (mostly secondary care and mental health), about 4.4% of medical staff full-time equivalent was provided by locum doctors. But those overall national rates of locum use hide a great deal of variation. In primary care, we found the National Health Service Digital workforce returns showed the rate of locum use by Clinical Commissioning Group varied from 1% to almost 31%. Among National Health Service trusts, the reported rate of locum use varied from < 1% to almost 16%. We found that there was poor awareness of and adherence to national guidance on locum working arrangements produced by National Health Service England. Our research showed that locum working can have adverse consequences for the quality and safety of care, but that such consequences were probably more likely to result from the organisational setting and the working arrangements than they were from the locum doctors themselves and their competence, clinical practice or behaviours. Our research was hampered in some respects by the COVID pandemic which both resulted in some delays and other challenges. Our efforts to use electronic patient record data in secondary care to explore locum doctor working were stymied by the problems of data access and quality. Locum doctors are a key component of the medical workforce in the National Health Service, and provide necessary flexibility and additional capacity for healthcare organisations and services. We found that the extent of reliance on locum doctors varied considerably, but that an over-reliance on locums for service provision was undesirable. Some differences in practice and performance between locum and permanent doctors were found, but these seemed often to arise from organisational characteristics. We found that patients were more concerned with the clinical expertise and skills of the doctor they saw than whether they were a locum or not. Organisational arrangements for locum working could be improved in many respects. This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: NIHR128349) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 37. See the NIHR Funding and Awards website for further award information.

Modeling the spatiotemporal dynamics of electric power consumption in China from 2000 to 2020 based on multisource remote sensing data and machine learning

China's rising electricity power consumption (EPC) is strongly correlated with carbon emissions. Timely and accurate analysis of the spatiotemporal dynamics of EPC is important to the realization of carbon peaking and carbon neutrality goals in China. However, due to data quality problems and limitations of modeling methods, the estimation accuracy warrants further improvement. In this study, the EPC in China from 2000 to 2020 was estimated based on multisource remote sensing data using the Random Forest (RF) model. Compared with previous studies, the accuracy of this study was improved by 39%–47%. The reasons were combining multisource remote sensing data can mitigate the quality issues of nighttime light (NTL) data, and the RF can capture the nonlinear relations between remote sensing data and EPC. In addition, the spatial pattern of the average EPC in China was dominated by the low-level EPC, as well as showing an obvious increasing trend. We also found that in the middle reaches of the Yellow River and northern coastal China, the low-speed increase in EPC led to high carbon emissions and emission intensity. We suggest optimizing the fuel fix of energy and adjusting the industrial structure, combining them with scientific and rational spatial planning.

The role of data governance in a high-level approach of data migration to open data

Data quality issues are still a challenge in data migration. Existing methods mainly focus on technical issues. However, the problem of data quality is not only a technical problem, but the main one is the non-technical aspect. This paper aims to identify the role of data governance in the data migration process. The qualitative research method uses two primary references, namely the high-level data migration approach from Kazimir et al. and the DAMA-DMBOK framework. Both are highly compatible with data migration processes focusing on data quality and referencing governance roles. The case study was conducted in a central government working on data migration to the new One Data application. The results show that eight data governance roles are involved in 20 migration processes (65%). Non-technical activities related to data quality involve more than half of the data migration process. This paper proposes the role of data governance for non-technical aspects of the migration process. This paper presents a case study of the high-level data migration process and how data governance comes into play. This research can help the government and practitioners overcome problems that often occur in data migration, especially related to data quality.

The challenges and opportunities of continuous data quality improvement for healthcare administration data.

Various data quality issues have prevented healthcare administration data from being fully utilized when dealing with problems ranging from COVID-19 contact tracing to controlling healthcare costs. (i) Describe the currently adopted approaches and practices for understanding and improving the quality of healthcare administration data. (ii) Explore the challenges and opportunities to achieve continuous quality improvement for such data. We used a qualitative approach to obtain rich contextual data through semi-structured interviews conducted at a state health agency regarding Medicaid claims and reimbursement data. We interviewed all data stewards knowledgeable about the data quality issues experienced at the agency. The qualitative data were analyzed using the Framework method. Sixteen themes emerged from our analysis, collected under 4 categories: (i) Defect characteristics: Data defects showed variability, frequently remained obscure, and led to negative outcomes. Detecting and resolving them was often difficult, and the work required often exceeded the organizational boundaries. (ii) Current process and people issues: The agency adopted primarily ad-hoc, manual approaches to resolving data quality problems leading to work frustration. (iii) Challenges: Communication and lack of knowledge about legacy software systems and the data maintained in them constituted challenges, followed by different standards used by various organizations and vendors, and data verification difficulties. (iv) Opportunities: Training, tool support, and standardization of data definitions emerged as immediate opportunities to improve data quality. Our results can be useful to similar agencies on their journey toward becoming learning health organizations leveraging data assets effectively and efficiently.

Modeling the Effect of Random Environmental Perturbation on Data Precision in Niger Delta Crude Oil Production

Decision-makers most times rely on data precision to support their decision-making processes. There is also a strong belief, however, that data quality or data precision problems are widespread in practice and that reliance on poor data quality or wrong data sets can lead to devastating consequences on crude oil production and financing. This study applies the Runge-Kutta numerical method of order 45 (ordinary differential equation of order 45) to examine the effect of data precision index on the Niger Delta crude oil production due to the variation of the initial condition. The study has found out that the initial condition four (4), (IC 4) with a data precision index value of 0.0118 ranked best in the first crude oil well, the initial condition (IC) 15 with a data precision index value of 0.0021 ranked best in the second crude oil well, the initial condition (IC) 17 with a data precision index value of 0.0021 ranked best in the third crude oil well and the initial condition (IC) 20 with a data precision index value of 0.0234 ranked best in the fourth crude oil well. The study has observed that each crude oil data is associated with a best-fit-value of the data precision index.

A review: Data visualization for COVID-19

As COVID-19 raged across the globe, it seriously disrupted social health security, daily work, and the rest of the people, which aroused great panic among all classes. These facts extensively affected the regional circulation and economic growth of the epidemic area. Data visualization, an advanced technology, can help humans understand epidemic transmission trends, identify high-risk areas of outbreak, and evaluate prevention and control policies. Previous essays immediately established databases to organize and build datasets. Other researchers built new visualization models to analyze the causes of virus transmission and pointed out the low efficiency of the existing COVID-19 system response. Although various research and data visualization methods have been used on COVID-19, there has not been a comprehensive, systematic essay that covers the entire span of COVID-19 and its data visualization usage. Therefore, this paper will provide a new systematic and comprehensive perspective to explore the analysis of data visualization to solve COVID-19 data challenges and provide research methods and applications for COVID-19. To be specific, this paper first introduces the characteristics and challenges of COVID-19 epidemic data people may face during the data visualization process, which includes data complexity problems, data timeliness problems, data regional problems, and data quality problems. Then, this paper discusses data visualization methods used by predecessors in previous research. After that, this paper describes the urgent data visualization analysis requirements of COVID-19, including regional differences in virus spread, the influence of temperature on the virus, transmission trends, and death rates for different genders and ages, etc. Finally, this paper reviews applications of data visualization in COVID-19 analysis, followed by earlier-stage prevention, mid-term containment, and later-term reduction of impact. This paper provides an important reference for further research and application of data visualization technology in COVID-19 prevention and control, which brings new ideas for strengthening future epidemic surveillance and data visualization for global epidemic prevention and control.

The data quality problem (in the European Financial Data Space)

Abstract By creating the European Financial Data Space (EFDS), the European Commission aims to foster the development of data-driven innovation in the financial sector. Artificial intelligence (AI) is the most notable innovation that could benefit from enhanced access to data through the EFDS. However, the current policy framework establishing the EFDS suffers from a major limitation—it neglects the question of data quality or whether the data shared in the EFDS will be fit for the purpose of the development of AI applications. The AI Act and General Data Protection Regulation could play a role in fostering data quality in this area, but the two frameworks are also unlikely to do that to a satisfactory extent. To address the data quality problem, this article proposes policymakers should develop a framework for data quality that encompasses both a conceptual and an organizational dimension aimed at promoting the free flow of high-quality data in the EFDS.

Experimental Study on Using Synthetic Images as a Portion of Training Dataset for Object Recognition in Construction Site

The application of Artificial Intelligence (AI) across various industries necessitates the acquisition of relevant environmental data and the implementation of AI recognition learning based on this data. However, the data available in real-world environments are limited and difficult to obtain. Construction sites represent dynamic and hazardous environments with a significant workforce, making data acquisition challenging and labor-intensive. To address these issues, this experimental study explored the potential of generating synthetic data to overcome the challenges of obtaining data from hazardous construction sites. Additionally, this research investigated the feasibility of hybrid dataset in securing construction-site data by creating synthetic data for scaffolding, which has a high incidence of falls but low object recognition rates due to its linear object characteristics. We generated a dataset by superimposing scaffolding objects, from which the backgrounds were removed, onto various construction site background images. Using this dataset, we produced a hybrid dataset to assess the feasibility of synthetic data for construction sites and to evaluate improvements in object recognition performance. By finding the optimal composition ratio with real data and conducting model training, the highest accuracy was achieved at an 8:2 ratio, with a construction object recognition accuracy of 0.886. Therefore, this study aims to reduce the risk and labor associated with direct data collection at construction sites through a hybrid dataset, achieving data generation at a low cost and high efficiency. By generating synthetic data to find the optimal ratio and constructing a hybrid dataset, this research demonstrates the potential to address the problems of data scarcity and data quality on construction sites. The improvement in recognition accuracy of the construction safety management system is anticipated, suggesting that the creation of synthetic data for constructing a hybrid dataset can reduce construction safety-accident issues.

Data Cleaning: Challenges and Existing Solutions

We classify data quality problems that are addressed by data cleaning and provide an overview of the main solution approaches. Data cleaning is especially required when integrating heterogeneous data sources and should be addressed together with schema-related data transformations. In data warehouses, data cleaning is a major part of the so-called ETL process. We also discuss current tool support for data cleaning.

Cleenex: Support for User Involvement during an Iterative Data Cleaning Process

The existence of large amounts of data increases the probability of occurring data quality problems. A data cleaning process that corrects these problems is usually an iterative process, because it may need to be re-executed and refined to produce high-quality data. Moreover, due to the specificity of some data quality problems and the limitation of data cleaning programs to cover all problems, often a user has to be involved during the program executions by manually repairing data. However, there is no data cleaning framework that appropriately supports this involvement in such an iterative process, a form of human-in-the-loop, to clean structured data. Moreover, data preparation tools that somehow involve the user in data cleaning processes have not been evaluated with real users to assess their effort. Therefore, we propose Cleenex, a data cleaning framework with support for user involvement during an iterative data cleaning process, and conduct two data cleaning experimental evaluations: an assessment of the Cleenex components that support the user when manually repairing data with a simulated user; and a comparison, in terms of user involvement, of data preparation tools with real users. Results show that Cleenex components reduce the user effort when manually cleaning data during a data cleaning process, for example, the number of tuples visualized is reduced in 99%. Moreover, when performing data cleaning tasks with Cleenex, real users need less time/effort (e.g., half the clicks) and, based on questionnaires, prefer it to the other tools used for comparison, OpenRefine and Pentaho Data Integration.

Causal inference of diachronic semantic maps from cross-linguistic synchronic polysemy data

Semantic maps are used in lexical typology to summarize cross-linguistic implicational universals of co-expression between meanings in a domain. They are defined as networks which, using as few links as possible, connect the meanings so that every isolectic set (i.e., set of meanings that can be expressed by the same word in some language) forms a connected component. Due to the close connection between synchronic polysemies and semantic change, semantic maps are often interpreted diachronically as encoding potential pathways of semantic extension. While semantic maps are traditionally generated by hand, there have been attempts to automate this complex and non-deterministic process. I explore the problem from a new algorithmic angle by casting it in the framework of causal discovery, a field which explores the possibility of automatically inferring causal structures from observational data. I show that a standard causal inference algorithm can be used to reduce cross-linguistic polysemy data into minimal network structures which explain the observed polysemies. If the algorithm makes its link deletion decisions on the basis of the connected component criterion, the skeleton of the resulting causal structure is a synchronic semantic map. The arrows which are added to some links in the second stage can be interpreted as expressing the main tendencies of semantic extension. Much of the existing literature on semantic maps implicitly assumes that the data from the languages under analysis is correct and complete, whereas in reality, semantic map research is riddled by data quality and sparseness problems. To quantify the uncertainty inherent in the inferred diachronic semantic maps, I rely on bootstrapping on the language level to model the uncertainty caused by the given language sample, as well as on random link processing orders to explore the space of possible semantic maps for a given input. The maps inferred from the samples are then summarized into a consensus network where every link and arrow receives a confidence value. In experiments on cross-linguistic polysemy data of varying shapes, the resulting confidence values are found to mostly agree with previously published results, though challenges in directionality inference remain.

Comparative Analysis of ML-Based Outlier Detection Techniques for IoT-Based Smart Energy Management Systems

With the development and advancement of ICST, data-driven technology such as the Internet of Things (IoT) and Smart Technology including Smart Energy Management Systems (SEMS) has become a trend in many regions and around the globe. There is no doubt that data quality and data quality problems are among the most vital topics to be addressed for a successful application of IoT-based SEMS. Poor data in such major yet delicate systems will affect the quality of life (QoL) of millions, and even cause destruction and disruption to a country. This paper aims to tackle this problem by searching for suitable outlier detection techniques from the many developed ML-based outlier detection methods. Three methods are chosen and analyzed for their performances, namely the K-Nearest Neighbour (KNN)+ Mahalanobis Distance (MD), Minimum Covariance Determinant (MCD), and Local Outlier Factor (LOF) models. Three sensor-collected datasets that are related to SEMS and with different data types are used in this research, they are pre-processed and split into training and testing datasets with manually injected outliers. The training datasets are then used for searching the patterns of the datasets through training of the models, and the trained models are then tested with the testing datasets, using the found patterns to identify and label the outliers in the datasets. All the models can accurately identify the outliers, with their average accuracies scoring over 95%. However, the average execution time used for each model varies, where the KNN+MD model has the longest average execution time at 12.99 seconds, MCD achieving 3.98 seconds for execution time, and the LOF model at 0.60 seconds, the shortest among the three.

Methods for evaluating and improving manual process data acquisition in one-of-a-kind productions

The One-of-a-kind production is characterized by manual and specialized manufacturing processes. Human adaptability is essential here. As a result, production data acquisition is mainly manual. In many cases, the recorded data are inaccurate or even subject to errors. Consequently, the opportunities for process optimization are impeded. This article therefore analyzes data quality problems and their causes. Subsequently, methods for the evaluation of process data such as process mining and performance indicators are considered and applied to production data sets. The comparison enables the derivation of organizational measures. The result is a practice-oriented method for continuously checking the data quality of manual production data acquisition.

Data Streaming Pipelines in Life Sciences: Improving Data Integrity and Compliance in Clinical TrialS

Data streaming pipelines have become a revolutionary tool in the field of life sciences, providing novel functionalities for managing and examining the large volumes of data produced in clinical trials. By using real-time data processing, these pipelines improve data integrity and guarantee adherence to regulatory requirements, therefore effectively tackling some of the most crucial obstacles encountered in clinical research. Preserving the integrity and precision of data is of utmost importance in clinical studies. Conventional data management systems sometimes have difficulties in keeping up with the fast flow of data from many sources, resulting in delays, discrepancies, and even problems with compliance. In order to provide continuous, real-time processing of data as it is created, data streaming pipelines offer a solution. This methodology guarantees the prompt validation, cleansing, and processing of data, therefore preserving a consistent degree of precision and minimising the likelihood of mistakes. An inherent advantage of data streaming pipelines is their capacity to manage fast-moving data streams originating from diverse sources, including electronic health records (EHRs), wearable devices, and laboratory equipment. Through the integration of multiple data sources into a cohesive pipeline, researchers may get a holistic perspective of trial results, therefore enabling more informed decision-making and prompt interventions. The real-time characteristic of streaming pipelines also facilitates proactive monitoring, enabling the timely identification of abnormalitie r data quality problems that may affect the results of trials.

Empirical analysis of tree-based classification models for customer churn prediction

Customer churn is a vital and reoccurring problem facing most business industries, particularly the telecommunications industry. Considering the fierce competition among telecommunications firms and the high expenses of attracting and gaining new subscribers, keeping existing loyal subscribers becomes crucial. Early prediction of disgruntled subscribers can assist telecommunications firms in identifying the reasons for churn and in deploying applicable innovative policies to boost productivity, maintain market competitiveness, and reduce monetary damages. Controlling customer churn through the development of efficient and dependable customer churn prediction (CCP) solutions is imperative to attaining this goal. According to the outcomes of current CCP research, several strategies, including rule-based and machine-learning (ML) processes, have been proposed to handle the CCP phenomenon. However, the lack of flexibility and robustness of rule based CCP solutions is a fundamental shortcoming, and the lopsided distribution of churn datasets is deleterious to the efficacy of most traditional ML techniques in CCP. Regardless, ML-based CCP solutions have been reported to be more effective than other forms of CCP solutions. Unlike linear-based, instance-based, and function-based ML classifiers, tree-based ML classifiers are known to generate predictive models with high accuracy, high stability, and ease of interpretation. However, the deployment of tree-based classifiers for CCP is limited in most cases to the decision tree (DT) and random forest (RF). Hence, this research investigated the effectiveness of tree-based classifiers with diverse computational properties in CCP. Specifically, the CCP performances of diverse tree-based classifiers such as the single, ensemble, enhanced, and hybrid tree-based classifiers are investigated. Also, the effects of data quality problems such as the class imbalance problem (CIP) on the predictive performances of tree-based classifiers and their homogeneous ensemble variants on CCP were assessed. From the experimental results, it was observed that the investigated tree-based classifiers outperformed other forms of classifiers such as linear-based (Support Vector Machine (SVM)), instance-based (K-Nearest Neighbour (KNN)), Bayesian-based (Naïve Bayes (NB)) and function-based (MultiLayer Perceptron (MLP)) classifiers in most cases with or without the CIP. Also, it was observed that the CIP has a significant effect on the CCP performances of investigated tree-based classifiers, but the combination of a data sampling technique and a homogeneous ensemble method can be an effective solution to CIP and also generate efficient CCP models.

Classification and quantification of timestamp data quality issues and its impact on data quality outcome

Abstract Timestamps play a key role in process mining because it determines the chronology of which events occurred and subsequently how they are ordered in process modelling. The timestamp in process mining gives an insight on process performance, conformance, and modelling. This therefore means problems with the timestamp will result in misrepresentations of the mined process. A few articles have been published on the quantification of data quality problems but just one of the articles at the time of this paper is based on the quantification of timestamp quality problems. This article evaluates the quality of timestamps in event log across two axes using eleven quality dimensions and four levels of potential data quality problems. The eleven data quality dimensions were obtained by doing a thorough literature review of more than fifty process mining articles which focus on quality dimensions. This evaluation resulted in twelve data quality quantification metrics and the metrics were applied to the MIMIC-III dataset as an illustration. The outcome of the timestamp quality quantification using the proposed typology enabled the user to appreciate the quality of the event log and thus makes it possible to evaluate the risk of carrying out specific data cleaning measures to improve the process mining outcome.

The Advantages and Challenges Faced by Business Analytics in the Context of Big Data

In recent years, with the booming development of big data technology, data acquisition and data analysis have become more convenient and efficient, and business data analysis is gradually being widely used in various industries. This paper analyses how business analytics can seize the opportunities for better development in the era of big data, and also explains the challenges faced by business analytics in the era of big data and provides corresponding solutions. The article finds that big data can help business analytics better predict market trends, optimise marketing strategies, provide visual analysis reports, and detect fraud risks; at the same time, the problems of data quality, data security, data processing speed, and data processing capacity arise; finally, the article puts forward ways to solve these problems from the perspectives of improving the quality of data, improving the level of protection, improving the speed of processing, and improving the capacity of analytics, which provides diversified ideas for people.

On some aspects of medical data quality

This paper examines the specific problem of the quality of medical data of patients when they are hospitalized for treatment through the lens of the general problem of data quality. The need to apply international standards in data quality is discussed, taking into account the specifics of the medical field and the national standards and regulations. This necessity is considered with respect to the problem of medical institutions switching to IT systems with electronic medical records and electronic health records. The specifics of the procedures for filling in the files of hospitalized patients are highlighted.