Data Quality Issues Research Articles

Enhancing Reliability in Automated Pavement Condition Data with a Data Quality Check Approach for Highway Agencies

Automated methods have been widely used by highway agencies to collect pavement condition data. However, there are still accuracy and precision issues associated with the reliability of the existing automated data collection methods. Therefore, this research aims to develop data quality check procedures to improve the reliability of automated pavement condition data for highway agencies. The study comprises three main components: identification of data quality check indexes; establishment of data quality thresholds; and implementation of data quality check procedures. Annual pavement rating data collected by a vendor using automated technologies and manual audit data from an independent third party were utilized to develop thresholds and test the procedures. Three districts, including two urban and one rural districts in Texas, were selected for the data quality check implementation. The results indicate that the proposed procedures, along with defined indexes and thresholds, efficiently identify pavements with data quality issues at both the section level and the county level. By pinpointing problematic areas, highway agencies can allocate resources for quick quality checks, enhancing the accuracy and precision of automated pavement condition data. The study can help highway agencies enhance the accuracy and precision of automated pavement condition data, ultimately leading to improvements in their pavement conditions.

Artificial Intelligence in Central Banking

Abstract The paper uses qualitative research to investigate the potential uses of artificial intelligence in the field of central banking. The analysis shows that monetary policy, prudential supervision and the oversight of payments are the areas where the use artificial intelligence is most likely to bring benefits. Monetary policy calibration involves working with long time series of data for various parameters and making the necessary analysis and forecasts, an activity in which artificial neural networks may prove useful. Bank supervision can benefit from the use natural language processing algorithms that can read documents and extract the relevant information. Such algorithms can read all of the required documents (not just those that the supervisor selected) and return all of the sentences that contain a certain predefined expression. In the field of the oversight of payments, the capabilities of machine learning to identify new patterns or anomalies in the data that could indicate fraud or money laundering will boost the efforts to combat them. In terms of challenges associated with the use of artificial intelligence in central banking, perhaps the two biggest challenges are that some of the models do not allow for a reasonable level of explainability of the algorithm(s) through which they arrive at the result (especially relevant for bank supervision) and data availability. With respect to the latter, although the issue of quantity of data can be dismissed as a shortcoming given the huge amounts of data available, the issue of data quality seems to be more pronounced, as deficiencies such as data measured incompletely or incorrectly, scarcity and regulatory barriers that impede data sharing may be difficult to surpass.

A design theory for data quality tools in data ecosystems: Findings from three industry cases

Data ecosystems are a novel inter-organizational form of cooperation. They require at least one data provider and one or more data consumers. Existing research mainly addresses generativity mechanisms in this relationship, such as business models or role models for data ecosystems. However, an essential prerequisite for thriving data ecosystems is high data quality in the shared data. Without sufficient data quality, sharing data might lead to negative business consequences, given that the information drawn from them or services built on them might be incorrect or produce fraudulent results. We tackle this issue precisely since we report on a multi-case study deploying data quality tools in data ecosystem scenarios. From these cases, we derive generalized prescriptive design knowledge as a design theory to make the knowledge available for others designing data quality tools for data sharing. Subsequently, our study contributes to integrating the issue of data quality in data ecosystem research and provides practitioners with actionable guidelines inferred from three real-world cases.

Challenges and the Way Forward in Demand-Forecasting Practices within the Ethiopian Public Pharmaceutical Supply Chain.

This study delves into the challenges of pharmaceutical forecasting within the Ethiopian public pharmaceutical supply chain, which is vital for ensuring medicine availability and optimizing healthcare delivery. t It aims to identify and analyze key hindrances to pharmaceutical forecasting in Ethiopia, employing qualitative analysis through semi-structured interviews with stakeholders. Thematic analysis using NVIVO 14 software reveals challenges including finance-related constraints, workforce shortages, and data quality issues. Financial challenges arise from funding uncertainties, causing delayed procurement and stockouts. Workforce shortages hinder accurate forecasting, while data quality issues result from incomplete and untimely reporting. Recommendations include prioritizing healthcare financing, investing in workforce development, and improving data quality through technological advancements and enhanced coordination among stakeholders.

MODEL MONITORING WITH GRAFANA AND DYNATRACE: A COMPREHENSIVE FRAMEWORK FOR ENSURING ML MODEL PERFORMANCE

In the rapidly evolving landscape of Machine Learning (ML), ensuring the continued efficacy and reliability of deployed models is paramount. This paper introduces a robust framework for model monitoring, highlighting the capabilities of Grafana and Dynatrace. Grafana provides powerful visualization and alerting capabilities, while Dynatrace offers deep insights into system and application performance.By integrating these tools, organization can establish a holistic monitoring solution that tracks key performance indicators (KPIs), detects anomalies, and triggers alerts in real-time. This framework enables proactive management of model drift, data quality issues and resource utilization, thereby bolstering the trustworthiness of ML applications in production.Case studies and practical implementation guidelines illustrate the effectiveness of this approach across various industry domains. The proposed methodology represents a pivotal advancement in the field of ML model operations, facilitating enhanced decision-making, reduced downtime and heightened user satisfaction.

Enhancing Criminal Analysis through Multi-Model Integration: Addressing Challenges and Ensuring Ethical Implementation

Abstract: Law enforcement agencies face numerous challenges in criminal analysis, from data quality issues to ethical concerns. To address these challenges, this research proposes a novel approach: integrating multiple modelling techniques into a unified framework. By combining regression analysis, decision trees, support vector machines, neural networks, and ensemble methods, this approach aims to provide more robust and accurate solutions. The research seeks to develop a framework for integration, evaluate effectiveness using real-world data, and explore ethical implications. Ultimately, the goal is to advance criminal analysis, promote multi-model integration in law enforcement, and ensure ethical implementation for justice.

Estimation and probabilistic projection of age- and sex-specific mortality rates across Brazilian municipalities between 2010 and 2030.

Mortality rate estimation in small areas can be difficult due the low number of events/exposure (i.e. stochastic error). If the death records are not completed, it adds a systematic uncertainty on the mortality estimates. Previous studies in Brazil have combined demographic and statistical methods to partially overcome these issues. We estimated age- and sex-specific mortality rates for all 5,565 Brazilian municipalities in 2010 and forecasted probabilistic mortality rates and life expectancy between 2010 and 2030. We used a combination of the Tool for Projecting Age-Specific Rates Using Linear Splines (TOPALS), Bayesian Model, Spatial Smoothing Model and an ad-hoc procedure to estimate age- and sex-specific mortality rates for all Brazilian municipalities for 2010. Then we adapted the Lee-Carter model to forecast mortality rates by age and sex in all municipalities between 2010 and 2030. The adjusted sex- and age-specific mortality rates for all Brazilian municipalities in 2010 reveal a distinct regional pattern, showcasing a decrease in life expectancy in less socioeconomically developed municipalities when compared to estimates without adjustments. The forecasted mortality rates indicate varying regional improvements, leading to a convergence in life expectancy at birth among small areas in Brazil. Consequently, a reduction in the variability of age at death across Brazil's municipalities was observed, with a persistent sex differential. Mortality rates at a small-area level were successfully estimated and forecasted, with associated uncertainty estimates also generated for future life tables. Our approach could be applied across countries with data quality issues to improve public policy planning.

SCSQ: A sample cooperation optimization method with sample quality for recurrent neural networks

Time series forecasting holds significant value across various application scenarios. However, practical implementations often encounter challenges related to low-quality time series data resulting from system failures or external interference. Unfortunately, existing forecasting methods primarily focus on optimizing model architecture for accurate predictions and overlook the importance of addressing data quality issues. In this paper, we propose optimizing data utilization to enhance model performance based on data quality and introduce the Sample Cooperation with Sample Quality (SCSQ) method to facilitate recurrent neural networks training. Firstly, we define sample quality as the matching degree between samples and model, and suggest using the attention entropy to calculate the sample quality through an attention mechanism. Secondly, we optimize the model's gradient vector based on sample quality. To address optimization training involving samples of different qualities effectively, we propose a more reasonable objective function within our proposed sample gradient conflict optimization module and devise a novel algorithm leveraging approximation techniques along with ADMM algorithm implementation. Through experiments conducted on six datasets, the results demonstrate that SCSQ significantly improves LSTM's performance. In certain cases, it even surpasses the state-of-the-art models. Additionally, SCSQ enhances the anti-interference ability of LSTM against low-quality samples.

Quality Inspection and Problem Analysis of Satellite Image Data in Land Use Survey

Abstract. In order to improve the data quality of land use remote sensing monitoring images, this article introduces the process of generating satellite image data, elaborates on the content of satellite image data verification in land use remote sensing monitoring, and proposes quality issues and improvement measures for satellite image data. Taking the discovered satellite image data quality issues as an example, compared with quality inspection standards, it was found that the main problems in the results were projection parameter errors, image color distortion, image blurring, and position accuracy exceeding limits. It is recommended to check the above issues during the image production stage, analyze the reasons for exceeding the position accuracy limit, image distortion, and embossing, and provide relevant suggestions. Provided strong technical support for land use surveys.

Challenges and Prospects of Artificial Intelligence in Nigerian Journalism Practice: A Narrative Review

The integration of artificial intelligence (AI) into Nigerian journalism practices presents both challenges and prospects for the industry. This paper explores the dynamic interplay between AI technologies and journalism in Nigeria, addressing key conceptual clarifications, theoretical frameworks, challenges, and prospects associated with AI utilization in the field. Drawing upon existing literature, the study examines the multifaceted nature of AI, ranging from its conceptual underpinnings to its practical applications in journalism. It utilizes the Technology-Organization-Environment (TOE) framework to analyze the factors influencing AI adoption within Nigerian newsrooms, considering technological, organizational, and environmental contexts. Identifying challenges such as infrastructural limitations, financial constraints, cultural resistance, and ethical considerations, the paper elucidates the complexities inherent in AI integration in Nigerian journalism. These challenges encompass issues of data quality, transparency, bias, and accountability, underscoring the need for responsible AI implementation and robust ethical frameworks. Additionally, the paper discusses the prospects offered by AI in enhancing journalistic efficiency, facilitating data-driven reporting, personalizing news delivery, and combating misinformation. By leveraging AI for fact-checking, multilingual reporting, predictive analytics, and automated content generation, Nigerian journalists can innovate and adapt to the evolving media landscape. Moreover, AI presents opportunities for improving audience engagement, storytelling, collaboration, and addressing bias and ethics concerns. Embracing AI in journalism also creates educational pathways for aspiring journalists, fostering skill development and innovation in the industry.

The Causes and Consequences of Refugee Flows: A Contemporary Reanalysis

The world faces a forced displacement crisis. Tens of millions of individuals have been forced across international boundaries worldwide. Therefore, the causes and consequences of refugee flows are the subjects of significant social science inquiry. Unfortunately, the historical lack of reliable data on actual refugee flows, country-specific data reporting timelines, and more general pre-2000 data quality issues have significantly limited empirical inferences on these topics. We replicate 28 articles on these topics using data newly released after a multiyear collaboration with the United Nations on annual dyadic flows. We observe major inconsistencies between the newly released flow numbers and the stock-based flow estimates upon which decades of research are based; we also find widespread inappropriate treatment of missing historical values. When we replicate the existing literature using the newly introduced flow data, correcting the treatment of missing historical values, and temporally extending/restricting the study periods, we produce significantly different results.

Machine learning insights into HIV outbreak predictions in Sub-Saharan Africa

Predicting and preventing HIV outbreaks in Sub-Saharan Africa, a region disproportionately affected by the epidemic remains a significant challenge. This review explores the effectiveness and challenges of using machine learning (ML) for forecasting HIV spread in high-risk areas. ML models have shown promise in identifying patterns and trends in HIV data, enabling more accurate predictions and targeted interventions. ML insights into HIV outbreak predictions leverage various data sources, including demographic, epidemiological, and behavioural data. By analysing these data, ML algorithms can identify high-risk populations and geographical areas susceptible to HIV transmission. This information is crucial for public health authorities to allocate resources efficiently and implement preventive measures effectively. Despite the potential benefits, several challenges exist in using ML for HIV outbreak predictions. These include data quality issues, such as incomplete or inaccurate data, which can affect the reliability of predictions. Additionally, the complexity of HIV transmission dynamics and the need for real-time data pose challenges for ML models. To address these challenges, researchers and practitioners are exploring innovative approaches, such as integrating multiple data sources and using advanced ML techniques. Collaborations between researchers, public health officials, and technology experts are also crucial for developing robust ML models for HIV outbreak predictions. In conclusion, while ML offers valuable insights into HIV outbreak predictions in Sub-Saharan Africa, addressing challenges such as data quality and model complexity is essential for its effective use. By overcoming these challenges, ML has the potential to significantly improve HIV prevention efforts and ultimately reduce the burden of the epidemic in the region. Keywords: Machine Learning, AI, HIV Outbreaks: Predictions, Insights.

Diabetes Prediction Using Machine Learning

Diabetes, characterized by high blood sugar levels over time, poses challenges for accurate prediction due to limited labelled data and data quality issues like outliers and missing values. To address these challenges, our research introduces a robust prediction framework. This framework integrates techniques such as outlier rejection, filling missing values, standardizing data, selecting relevant features, and employing K-fold cross-validation. We utilize a range of machine learning algorithms including k-nearest Neighbour, Decision Trees, Random Forest, AdaBoost, Naive Bayes, XGBoost, and Multilayer Perceptron (MLP). Additionally, we propose a weighted ensembling technique to enhance prediction accuracy. The weights for ensembling are determined based on the performance of each classifier using the Area Under ROC Curve (AUC) metric. We optimize model performance through hyperparameter tuning using grid search. Our experiments are conducted using the Pima Indian Diabetes Dataset under uniform conditions. The ensembling classifier we propose achieves superior performance with a sensitivity of 0.789, specificity of 0.934, false omission rate of 0.092, diagnostic odds ratio of 66.234, and AUC of 0.950. This outperforms existing methods by 2.00% in terms of AUC. Our framework surpasses other approaches discussed in the literature and shows promise for improving diabetes prediction. We have released our source code for diabetes prediction to the public.

Regulating the Rational Use of Antimicrobial Drugs and Improving the Quality of Healthcare Big Data: A Development and Usability Study as A Prelude to AI Assisted Healthcare

Background: Healthcare big data has become an important strategic resource, and the study of applying it to the rational application of antimicrobial drugs is of great significance in improving and optimizing the various tasks of medicine and health and promoting the development of social health. Based on its own characteristics, the issue of data quality has important research value in promoting data output and application. Objective: To optimize the status quo of antimicrobial drug use in medical institutions, to explore the value of healthcare big data, and to explore the data governance methods in the era of digital intelligence in order to build a good ecosystem for the application of healthcare big data. Methods: This paper constructs a set of intelligent full-closed-loop antimicrobial rational application management platform integrated with data quality, adopts PDCA cycle management mode, analyzes the problems in the process of applying healthcare big data and gives the corresponding solutions, authorizes according to the personnel's duties, and guarantees the data security. Results: Since the platform went online in January 2021, the antimicrobial drug use intensity indicator of a hospital has decreased year by year from 38 to 34.4 as of June 2023, effectively standardizing the use of antimicrobial drugs. The time consumed for the statistics of antimicrobial use intensity related data has decreased from an average of 30min to 2min, which has significantly improved the work efficiency.Healthcare professionals are able to monitor indicator data in real time, with or without abnormal cases, and fully grasp the trend of indicators. At the same time, the patient's medication can be viewed at any time, and the condition can be grasped in a timely manner. Conclusions: The platform not only effectively solves the problem of information barriers, but also accurately determines the data source and statistical logic of indicator collection, reflecting the actual clinical diagnosis and treatment, and at the same time greatly reduces labor costs and optimizes the workflow.The practical experience of platform construction has important reference significance for the governance and application of healthcare big data, which provides data support and technical guarantee for the scientific, standardized and refined management of hospitals.

High impedance fault classification in microgrids using a transformer-based model with time series harmonic synchrophasors under data quality issues

High impedance fault classification in microgrids using a transformer-based model with time series harmonic synchrophasors under data quality issues

Epidemiological anomaly detection in Philippine public health surveillance data through Newcomb-Benford analysis.

Public health surveillance is vital for monitoring and controlling disease spread. In the Philippines, an effective surveillance system is crucial for managing diverse infectious diseases. The Newcomb-Benford Law (NBL) is a statistical tool known for anomaly detection in various datasets, including those in public health. Using Philippine epidemiological data from 2019 to 2023, this study applied NBL analysis. Diseases included acute flaccid paralysis, diphtheria, measles, rubella, neonatal tetanus, pertussis, chikungunya, dengue, leptospirosis and others. The analysis involved Chi-square tests, Mantissa Arc tests, Mean Absolute Deviation (MAD) and Distortion Factor calculations. Most diseases exhibited nonconformity to NBL, except for measles. MAD consistently indicated nonconformity, highlighting potential anomalies. Rabies consistently showed substantial deviations, while leptospirosis exhibited closer alignment, especially in 2021. Annual variations in disease deviations were notable, with acute meningitis encephalitis syndrome in 2019 and influenza-like illness in 2023 having the highest deviations. The study provides practical insights for improving Philippine public health surveillance. Despite some diseases showing conformity, deviations suggest data quality issues. Enhancing the PIDSR, especially in diseases with consistent nonconformity, is crucial for accurate monitoring and response. The NBL's versatility across diverse domains emphasizes its utility for ensuring data integrity and quality assurance.

Data analytics for predicting disease outbreaks: A review of models and tools

The burgeoning field of data analytics has emerged as a pivotal force in the realm of public health, particularly in the context of predicting and mitigating disease outbreaks. This comprehensive review delves into the diverse landscape of models and tools employed in data analytics for disease outbreak prediction. With a focus on synthesizing existing knowledge, the paper aims to provide a nuanced understanding of the strengths, limitations, and future directions within this dynamic field. The review begins with an exploration of various models utilized for disease outbreak prediction, ranging from statistical approaches to machine learning models and epidemiological frameworks. Each model category is scrutinized for its efficacy in capturing the complexities inherent in infectious disease dynamics. Simultaneously, the paper investigates the array of tools and technologies leveraged in disease outbreak prediction, encompassing Geographic Information Systems (GIS), data visualization tools, and big data analytics platforms. A critical aspect of the review lies in the examination of diverse data sources contributing to predictive analytics. Epidemiological data, environmental factors, and the burgeoning influence of social media and web data are dissected for their roles in enhancing the accuracy and timeliness of outbreak predictions. Amidst the promises of data analytics, the paper navigates the challenges inherent in predicting disease outbreaks. Issues of data quality and availability, model complexity, interpretability, and ethical considerations are dissected, providing a holistic view of the hurdles that practitioners encounter. Drawing upon case studies and real-world applications, the review showcases instances where data analytics has proven successful in predicting disease outbreaks, shedding light on both triumphs and setbacks. The implications for public health, lessons learned from challenges, and the evolving role of data analytics in shaping global health preparedness are thoroughly discussed. As the paper unfolds, it illuminates future trends and innovations in the field, foreseeing the integration of advanced technologies, global collaboration for information sharing, and the adaptation of predictive analytics for emerging diseases. The review culminates in a comprehensive conclusion, summarizing key findings and emphasizing the potential transformative impact of data analytics on the landscape of disease outbreak prediction.

An ontology for maintenance activities and its application to data quality

Maintenance of assets is a multi-million dollar cost each year for asset intensive organisations in the defence, manufacturing, resource and infrastructure sectors. These costs are tracked though maintenance work order (MWO) records. MWO records contain structured data for dates, costs, and asset identification and unstructured text describing the work required, for example ‘replace leaking pump’. Our focus in this paper is on data quality for maintenance activity terms in MWO records (e.g. replace, repair, adjust and inspect). We present two contributions in this paper. First, we propose a reference ontology for maintenance activity terms. We use natural language processing to identify seven core maintenance activity terms and their synonyms from 800,000 MWOs. We provide elucidations for these seven terms. Second, we demonstrate use of the reference ontology in an application-level ontology using an industrial use case. The end-to-end NLP-ontology pipeline identifies data quality issues with 55% of the MWO records for a centrifugal pump over 8 years. For the 33% of records where a verb was not provided in the unstructured text, the ontology can infer a relevant activity class. The selection of the maintenance activity terms is informed by the ISO 14224 and ISO 15926-4 standards and conforms to ISO/IEC 21838-2 Basic Formal Ontology (BFO). The reference and application ontologies presented here provide an example for how industrial organisations can augment their maintenance work management processes with ontological workflows to improve data quality.

Optimization of English Classroom Interaction Models Incorporating Machine Learning

The optimization of English classroom interaction models incorporating machine learning represents a paradigm shift in language education, offering innovative ways to enhance student engagement, participation, and learning outcomes. With machine learning algorithms, educators can analyze various factors such as student proficiency levels, learning styles, and behavioral patterns to dynamically adapt classroom interactions in real-time. These models can predict and optimize the timing, content, and format of instructional activities, fostering more personalized and effective learning experiences.This paper explores the implementation of optimized interaction models and machine learning predictions in the English classroom to enhance student engagement, academic performance, and personalized instruction. Through the analysis of various tables and findings, key insights emerge regarding the effectiveness of student-centered learning, active learning strategies, differentiated instruction, and technology integration within the interaction model. Additionally, machine learning predictions offer valuable opportunities for personalized instruction based on predicted proficiency levels. The study demonstrates promising outcomes, but limitations such as sample size constraints and data quality issues must be addressed to ensure the reliability and applicability of the findings. Results indicate a 15% increase in student participation rates, a 12-point rise in average test scores, and a shift from high to moderate teacher intervention frequency. Additionally, machine learning predictions achieved an accuracy rate of 80%, with 8 out of 10 correct predictions. Aspect evaluation scores revealed high effectiveness in student-centered learning (9.2), formative assessment (9.3), and teacher facilitation (9.0). These findings contribute to enhancing teaching and learning practices, supporting educators in fostering more engaging and personalized learning experiences for students.

Healthcare insurance fraud detection using data mining

BackgroundHealthcare programs and insurance initiatives play a crucial role in ensuring that people have access to medical care. There are many benefits of healthcare insurance programs but fraud in healthcare continues to be a significant challenge in the insurance industry. Healthcare insurance fraud detection faces challenges from evolving and sophisticated fraud schemes that adapt to detection methods. Analyzing extensive healthcare data is hindered by complexity, data quality issues, and the need for real-time detection, while privacy concerns and false positives pose additional hurdles. The lack of standardization in coding and limited resources further complicate efforts to address fraudulent activities effectively.MethodolgyIn this study, a fraud detection methodology is presented that utilizes association rule mining augmented with unsupervised learning techniques to detect healthcare insurance fraud. Dataset from the Centres for Medicare and Medicaid Services (CMS) 2008-2010 DE-SynPUF is used for analysis. The proposed methodology works in two stages. First, association rule mining is used to extract frequent rules from the transactions based on patient, service and service provider features. Second, the extracted rules are passed to unsupervised classifiers, such as IF, CBLOF, ECOD, and OCSVM, to identify fraudulent activity.ResultsDescriptive analysis shows patterns and trends in the data revealing interesting relationship among diagnosis codes, procedure codes and the physicians. The baseline anomaly detection algorithms generated results in 902.24 seconds. Another experiment retrieved frequent rules using association rule mining with apriori algorithm combined with unsupervised techniques in 868.18 seconds. The silhouette scoring method calculated the efficacy of four different anomaly detection techniques showing CBLOF with highest score of 0.114 followed by isolation forest with the score of 0.103. The ECOD and OCSVM techniques have lower scores of 0.063 and 0.060, respectively.ConclusionThe proposed methodology enhances healthcare insurance fraud detection by using association rule mining for pattern discovery and unsupervised classifiers for effective anomaly detection.