Data Quality Issues Research Articles

Exploring Artificial Intelligence and Machine Learning in Precision Agriculture: A Pathway to Improved Efficiency and Economic Outcomes in Crop Production

This review analyzes secondary data from academic databases, research articles, and case studies to explore the role of new technologies for precision agriculture (PA) and investigates the value addition that Artificial Intelligence (AI) and Machine Learning (ML) provide to resource use, crop yield, and economic performance. Accordingly, the most of the key applications of AI in PA were related to crop yield prediction, disease detection, and effective water usage. Operating models through AI will analyze much data in real time, thus providing insight into informed decision making by farmers for proactive action against crop challenges like drought or pest attack. Furthermore, IoT devices and remote sensing support continuous monitoring in the delivery of correct data about optimizations of resources with minimal environmental impact. AI-driven robotics further automates all tasks related to planting, harvesting, and pesticide application, improving labor productivity and operational efficiency. This would involve in other issues like implementation costs, data privacy, and general unawareness among farmers of developing areas. Equally important will be ethical issues like ownership of data and loss of jobs. Various case studies in India, China, the United States, and Africa reveal how AI could transform the future of agriculture if integrated into agricultural systems properly to gain higher productivity and sustainability. Improvements in data quality and ethical issues, and increased access by smallholder farmers, will also be part of future research. Eventually, integrating AI with IoT, robotics, and big data analytics could provide high potential to meet global food demand in a sustainable manner.

Comparative Analysis of Machine Learning, Decision Trees, and K-Nearest Neighbors for Heart Disease Prediction

Abstract. This study investigates the efficacy of Machine Learning (ML), Decision Trees, and K-Nearest Neighbors (KNN) techniques in predicting heart disease, aiming to identify their strengths and limitations. ML models are effective in detecting complex patterns and delivering evolving predictions from large datasets but require high-quality data and can be challenging to interpret. Decision Trees provide clear, understandable decision rules, which is beneficial in clinical settings, yet they are susceptible to overfitting and instability. KNN, valued for its simplicity and flexibility, classifies heart disease based on similarity but struggles with high computational costs and sensitivity to noisy data. Experimental results indicate that each model has distinct advantages: ML excels in pattern recognition, Decision Trees offer interpretability, and KNN handles diverse data effectively. However, each also faces challenges that impact performance, such as data quality issues for ML, overfitting for Decision Trees, and computational demands for KNN. The study suggests that balancing these strengths and weaknesses is crucial for optimizing heart disease prediction models. Future research should explore hybrid approaches that combine these models' advantages while addressing their respective limitations to improve predictive accuracy and practical application in real-world scenarios.

DQM: Data Quality Metrics for AI Components in the Industry

In industrial settings, measuring the quality of data used to represent an intended domain of use and its operating conditions is crucial and challenging. Thus, this paper aims to present a set of metrics addressing this data quality issue in the form of a library, named DQM (Data Quality Metrics), for Machine Learning (ML) use. Additional metrics specific to industrial application are developed in the proposed library. This work aims also to assess various data and datasets types. Those metrics are used to characterize the training and evaluating datasets involved in the process of building ML models for industrial use cases. Two categories of metrics are implemented in DQM: inherent data metrics, are the ones evaluating the quality of a given dataset independently from the ML model such as statistical proprieties and attributes, and model dependent metrics which are those implemented to measure the quality of the dataset by considering the ML model outputs such the gap between two datasets in regards to a given ML model. DQM is used in the scope of the Confiance.ai program to evaluate datasets used for industrial purposes such as autonomous driving.

HANZE v2.1: an improved database of flood impacts in Europe from 1870 to 2020

Abstract. Assessing long-term trends in flood losses and attributing them to climatic and socioeconomic changes requires comprehensive and systematic collection of historical information. Here, we present flood impact data for Europe that are part of the HANZE (Historical Analysis of Natural HaZards) database. The dataset covers riverine, pluvial, coastal, and compound floods that have occurred in 42 European countries between 1870 and 2020. The data were obtained by extensive data collection from more than 800 sources ranging from news reports through government databases to scientific papers. The dataset includes 2521 events characterized by at least one impact statistic: area inundated, fatalities, persons affected. or economic loss. Economic losses are presented both in the original currencies and price levels and with the inflation and exchange rate adjusted to the 2020 value of the euro. The spatial footprint of affected areas is consistently recorded using more than 1400 subnational units corresponding, with minor exceptions, to the European Union's Nomenclature of Territorial Units for Statistics (NUTS) level 3. Daily start and end dates, information on causes of the events, notes on data quality issues or associated non-flood impacts, and full bibliography of each record supplement the dataset. Apart from the possibility of downloading the data (https://doi.org/10.5281/zenodo.8410025; Paprotny, 2023a), the database can be viewed, filtered, and visualized online at https://naturalhazards.eu (last access: 4 November 2024). The dataset is designed to be complementary to HANZE-Exposure, a high-resolution model of historical exposure changes (such as population and asset values) and be easily usable in statistical and spatial analyses, including multi-hazard studies.

Pioneering Museum Data Transformation: Unifying Legacy Systems and Enhancing Data Integrity

In the digital era, museums confront the challenge of modernising legacy data systems to align with current standards. Part of the RECODE (Rethinking Collections Data Ecosystems) Programme (Dupont et al. 2022) examines the complex transition from disparate departmental object models to a unified system, reflecting the broader museum community's struggle with data standardisation. The case study centres on consolidating five distinct data models accumulated over two decades, each tailored to different departmental needs under various constraints. The data mapping process identifies and aligns similar fields across models, facilitating the integration of analogous data points and exposing redundancies and unique practices embedded over years. Major challenges include data complexity, diversity, and quality issues (cleaning, standardizing, and deduplicating), compounded by the massive scale of the data: currently, the Natural History Museum London (NHM) has 262,831,590 records across 8,175 fields, and a total volume of 12 TB of data, which define both the ongoing data modelling work and our future data migration. A key innovation in our process is the immediate visibility of data issues that became apparent upon migrating to Amazon Web Services (AWS). AWS serves as the staging environment for our transition to the new NHM Collections Management System (CMS) and offers an unprecedented platform for directly addressing these challenges. It enables us to query all our data—across all departments and fields—in just seconds. This capability provides a comprehensive view that was previously unattainable. The core of this presentation is sharing "lessons learned" from navigating the intricacies of an ongoing CMS transition within a museum. The endeavor to untangle and unify diverse data models is a common challenge (IEEE Big Data Governance and Metadata Management Industry Connections Activity 2020,Wu et al. 2022, Wu et al. 2021,Little et al. 2022,Woodburn et al. 2022) highlighting the importance of community engagement and knowledge exchange. Our findings underscore the necessity of cross-departmental collaboration and the benefits of a data-driven data modelling approach. By sharing our journey and the developed comprehensive object model (Collier and Woodburn 2022), we aim to contribute valuable insights to museums undergoing similar transformations. The RECODE Programme sheds light on the practical aspects of CMS modernisation through the analytics and knowledge derived from over two decades of collections data and data use.

Addressing Data Availability And Quality Issues In LGU Educational Systems For Effective Machine Learning Integration

Addressing Data Availability And Quality Issues In LGU Educational Systems For Effective Machine Learning Integration

AI in Agriculture : Revolutionizing Precision Farming and Sustainable Crop Management

This article explores the transformative impact of Artificial Intelligence (AI) on agriculture, focusing on precision farming and sustainable crop management. It highlights how AI-driven technologies are revolutionizing agricultural practices by optimizing resource utilization, enhancing productivity, and promoting sustainability. The article discusses key applications of AI in agriculture, including crop health monitoring, predictive analytics, resource optimization, smart irrigation systems, variable rate application of chemicals, and pest and disease detection. It presents case studies and statistical data demonstrating the significant improvements in crop yields, resource efficiency, and environmental sustainability achieved through AI adoption. The article also addresses the challenges facing widespread AI implementation in agriculture, such as cost barriers, lack of technical expertise, data quality issues, and infrastructure limitations, while exploring future directions including the integration of AI with robotics and blockchain technology.

Advances and Challenges in Weather Nowcasting : A Comprehensive Review of Modern Techniques and Models

Weather nowcasting, the short-term forecasting of weather conditions, is a critical area of study due to its impact on safety and planning. Recent advances in machine learning (ML) and deep learning (DL) have significantly improved nowcasting accuracy, particularly in precipitation prediction. This review examines modern techniques and models used for weather nowcasting, focusing on their advancements and challenges. Motivation for this study arises from the need for more accurate and timely weather predictions to mitigate adverse effects on daily life and infrastructure. Despite advancements, current methods face limitations such as high computational costs, data quality issues, and the challenge of generalizing models across different geographic regions. The aim of this review is to provide a comprehensive overview of recent developments in nowcasting techniques, highlight their strengths and weaknesses, and discuss future directions for research. The objective is to synthesize the latest findings, assess the performance of various models, and propose potential improvements. This review underscores the importance of continued innovation and integration of new technologies, including transfer learning and hybrid models, to enhance the effectiveness of weather nowcasting systems.

Addressing data quality issues to assess clinical and epidemiological risk factors for COVID-19 among documented cases in Liberia: a single-centre, retrospective, observational study

IntroductionIdentification of risk factors for severe COVID-19 can help to inform case management in resource-constrained settings. We aimed to rigorously but retroactively address data issues to identify risk factors for...

Lessons From 3 Longitudinal Sensor-Based Human Behavior Assessment Field Studies and an Approach to Support Stakeholder Management: Content Analysis.

Pervasive technologies are used to investigate various phenomena outside the laboratory setting, providing valuable insights into real-world human behavior and interaction with the environment. However, conducting longitudinal field trials in natural settings remains challenging due to factors such as low recruitment success and high dropout rates due to participation burden or data quality issues with wireless sensing in changing environments. This study gathers insights and lessons from 3 real-world longitudinal field studies assessing human behavior and derives factors that impacted their research success. We aim to categorize challenges, observe how they were managed, and offer recommendations for designing and conducting studies involving human participants and pervasive technology in natural settings. We developed a qualitative coding framework to categorize and address the unique challenges encountered in real-life studies related to influential factor identification, stakeholder management, data harvesting and management, and analysis and interpretation. We applied inductive reasoning to identify issues and related mitigation actions in 3 separate field studies carried out between 2018 and 2022. These 3 field studies relied on gathering annotated sensor data. The topics involved stress and environmental assessment in an office and a school, collecting self-reports and wrist device and environmental sensor data from 27 participants for 3.5 to 7 months; work activity recognition at a construction site, collecting observations and wearable sensor data from 15 participants for 3 months; and stress recognition in location-independent knowledge work, collecting self-reports and computer use data from 57 participants for 2 to 5 months. Our key extension for the coding framework used a stakeholder identification method to identify the type and role of the involved stakeholder groups, evaluating the nature and degree of their involvement and influence on the field trial success. Our analysis identifies 17 key lessons related to planning, implementing, and managing a longitudinal, sensor-based field study on human behavior. The findings highlight the importance of recognizing different stakeholder groups, including those not directly involved but whose areas of responsibility are impacted by the study and therefore have the power to influence it. In general, customizing communication strategies to engage stakeholders on their terms and addressing their concerns and expectations is essential, while planning for dropouts, offering incentives for participants, conducting field tests to identify problems, and using tools for quality assurance are relevant for successful outcomes. Our findings suggest that field trial implementation should include additional effort to clarify the expectations of stakeholders and to communicate with them throughout the process. Our framework provides a structured approach that can be adopted by other researchers in the field, facilitating robust and comparable studies across different contexts. Constantly managing the possible challenges will lead to better success in longitudinal field trials and developing future technology-based solutions.

The role of big data technology in project schedule management: the case of Amazon

With the rapid development of the Internet, big data technology has received widespread attention in the field of management. Traditional project schedule management methods cannot adapt to the complex and changing environment, and some researchers believe that big data technology has become a key tool to improve and optimize project management. However, there is a lack of mature theoretical systems and practical experience to guide the application of big data technology. Based on the case of Amazon, this paper analyses how the company uses big data technology to optimize project schedule management. It is found that the application of big data technology in project management brings significant benefits. It is able to provide empirical and innovative support for project schedule management. This paper discusses the application of big data technology in project schedule management, highlighting its capability to enable timely adjustments. Real-time monitoring and forecasting facilitate proactive management of project directions and potential risks. Moreover, it leverages real-time data to optimize resource allocation and enhance resource utilization. Furthermore, the paper addresses the challenges and proposes countermeasures associated with big data technology in this context. To tackle the issue of uneven data quality, a robust data collection mechanism should be established.

AI for Climate Action: Leveraging Artificial Intelligence to Address Climate Change Challenges

This comprehensive article explores the transformative role of Artificial Intelligence (AI) in addressing the global climate crisis, focusing on its applications in climate modeling, renewable energy optimization, and disaster response. Through an extensive literature review, case studies, and expert interviews, we examine how AI technologies are revolutionizing climate prediction accuracy, enhancing data processing capabilities, and enabling sophisticated climate scenario simulations. The article delves into AI's contributions to renewable energy forecasting, smart grid management, and energy storage optimization, highlighting its potential to accelerate the transition to sustainable energy systems. We also investigate AI-driven approaches to disaster response and resilience, including early warning systems, resource allocation during climate-related disasters, and post-disaster recovery planning. While acknowledging the significant advancements AI brings to climate action, this study also addresses the challenges and limitations, such as data quality issues, ethical considerations, and technical barriers. Looking ahead, we discuss emerging AI technologies, their integration with other solutions, and the policy implications for effective climate change mitigation and adaptation. This research underscores the critical role of AI in combating climate change, while emphasizing the need for responsible development and deployment of these technologies to ensure a sustainable and resilient future.

Methods, Challenges, and Ethical Considerations in Data Collection of Corpus Compilation

Corpus compilation is a critical process in linguistics that involves gathering and organizing large datasets for language analysis and model training. This article examines key aspects of corpus compilation, with a particular focus on data collection. It explores the sources of data, strategies for ensuring representativeness, and challenges such as copyright constraints and data quality issues. Ethical considerations, such as anonymization and consent, are also discussed. By understanding these factors, researchers can build effective and ethically sound corpora for linguistic research and computational applications.

Machine Learning Applications in Salesforce CPQ Transforming Sales

This paper aims to understand the innovative disposals of machine learning applications on Salesforce Configure, Price, Quote (CPQ) applications more precisely how these innovations are revolutionizing software development. Salesforce CPQ is an ideal solution for organizations that want a modern tool to help them with their sales processes, and adding machine learning elements significantly improves the system. For example, the study analyses different cases where and how machine learning algorithm is utilised in the application of the following areas: pricing strategy, product configuration and sales forecasting. Upon collection of historical data, machine learning models are able to make computations of patterns that enables businesses to make quick analysis of the best strategies to use in an offer of various prices to the customers in an effort to enhance customer satisfaction. Furthermore, the application of I/A in the context of CPQ decreases the time engaged in manual configurations, liberating software development teams to work on other priorities essentially. These difficulties and strategies are enumerated in the paper: data quality issues and the impossibility of the Machine learning successful implementation without cooperation with other departments, specifically with Salesforce CPQ teams. In addition, it shares its vision on the tendencies in application of machine learning in CPQ systems with reference to software development paradigms. Finally, the scope of this research is as follows: The study will examine how the utilization of machine learning applications in Salesforce CPQ has changed traditional software development practices in today’s dynamic market environment by increasing efficiency and accuracy effectively, improving competitiveness within the market.

A Supervised Ml Algorithm for Detecting and Predicting Fraud Credit Card Transactions

A Credit card fraud presents an escalating threat in today's digital economy, leading to significant financial losses for consumers, businesses, and financial institutions. Traditional detection methods are increasingly inadequate due to the rise in online transactions and the evolving complexity of fraudulent activities. This research aims to create a reliable supervised machine learning model designed to detect and predict fraudulent credit card transactions in real-time. Our approach leverages advanced algorithms and extensive datasets to enhance transaction security, thereby mitigating financial risks. Current systems, like FICO's Fraud Detection System (FDS), utilize a range of supervised learning techniques, including Decision Trees, Random Forests, and Neural Networks. However, they face limitations, including data quality issues, high false positive rates, and the need for continuous retraining to adapt to evolving fraud patterns. To address these challenges, we propose an innovative anomaly detection method based on the Isolation Forest algorithm. Unlike traditional methods that rely on distance and density measures, Isolation Forests isolate anomalies by randomly selecting features and split values, making it more efficient in identifying fraudulent transactions, especially in imbalanced datasets. The proposed system boasts low linear time complexity and minimal memory requirements, enabling the construction of high-performing models even with large datasets. The research demonstrates that the Isolation Forest approach significantly outperforms traditional models in detecting credit card fraud, offering a promising solution for enhancing security in the financial ecosystem.

Intrinsic data quality dimensions: expanding on Wand and Wang’s data quality model

PurposeStudies show that data quality (DQ) issues are extremely costly for companies. To address such issues, as a starting point, there is a need to understand what DQ is. In his context, the 1996 paper “Anchoring data quality dimensions ontological foundations” by Wand and Wang has been highly influential on the understanding of DQ. However, the present study demonstrates that some of the assumptions made in their paper can be challenged. On this basis, this study seeks to develop clearer definitions.Design/methodology/approachThe assumptions behind Wand and Wang’s DQ classification are discussed, on which basis three counter-propositions are formulated. These are investigated through a representation theoretical approach involving analyses of deficient mappings between real-world and information system states. On this basis, an intrinsic DQ classification is derived. A case study is conducted to investigate the value of the developed DQ classification.FindingsThe representation theoretical analysis and the case study support the three propositions. These give rise to the development of a DQ classification that includes three primary intrinsic DQ dimensions (accuracy, completeness and conciseness), which are associated with six primary value-level DQ deficiencies (inaccuracy, incorrectness, meaninglessness, incompleteness, absence and redundancy). The case study supports the value of extending Wand and Wang’s DQ classification with the three additional data deficiencies.Research limitations/implicationsBy improving the conceptual clarity of DQ, this study provides future research with an improved basis for studies and discussions of DQ.Originality/valueThe study advances the understanding of DQ by providing additional clarity.

AI-Powered Information Governance: Balancing Automation and Human Oversight for Optimal Organization Productivity

This study employs a mixed-methods approach to examine the optimal balance between AI-powered automation and human oversight in information governance frameworks, aiming to enhance organizational productivity, efficiency, and compliance. Quantitative data collected from 384 respondents were analyzed using Pearson correlation, regression models, and Structural Equation Modeling (SEM). The results reveal strong positive correlations between AI automation levels and both organization size (r = 0.55, p < .01) and AI adoption duration (r = 0.62, p < .01). Regression analysis indicates that higher levels of AI automation significantly improve error reduction (β = 1.12, p < .001) and compliance (β = 1.05, p < .001), especially in larger organizations with longer AI adoption periods. SEM findings highlight that human oversight positively impacts error reduction (β = 0.65, p < .001) and compliance improvement (β = 0.72, p < .001), and the interaction between human oversight and AI automation further enhances these outcomes (error reduction: β = 0.32, p < .001; compliance improvement: β = 0.35, p < .001). The qualitative analysis, involving thematic extraction from industry reports, reveals ethical challenges such as data quality issues, algorithmic bias, and privacy concerns. Hence, it is necessary to integrate human oversight to ensure ethical standards and build stakeholder trust in AI-driven systems. The study concludes with practical recommendations for organizations: establishing transparent AI governance frameworks, investing in continuous training for employees, and regularly auditing AI processes to mitigate risks. By addressing both the technological and ethical dimensions, organizations can implement AI-powered information governance that not only boosts productivity and efficiency but also ensures compliance and ethical integrity.

AI in Business and Finance: A Micro-to-Macro Perspective on Its Impact

This comprehensive survey examines the transformative impact of machine learning (ML), deep learning (DL), and natural language processing (NLP) across diverse business and finance sectors. From micro-level operational efficiencies to macro-level strategic decision-making, we delve into the multifaceted applications of AI and its influence on traditional practices. Our analysis highlights the increasing prominence of ML in enhancing decision-making and achieving greater accuracy and efficiency. We also identify key challenges such as data quality, modelinterpretability, and linguistic information processing. By addressing these challenges, we can further unlock the potential of ML in revolutionizing business and finance. We conducted a systematic literature review using the PRISMA technique to identify relevant research published within the last five years. Our findings reveal the growing adoption of ML techniques in various business and finance applications, such as fraud detection, risk assessment, customer relationship management, and algorithmic trading. We identified relevant publications through comprehensive searches in leading academic databases, including Scopus, Web of Science, and Google Scholar. We also discuss the challenges associated with implementing ML models in these domains, including data quality issues, model interpretability, and ethical considerations. Byunderstanding the applications, challenges, and opportunities associated with AI, businesses, policymakers, researchers, and investors can make informed decisions and leverage AI to drive innovation and growth in the business and finance sectors.

A Decision Risk Assessment and Alleviation Framework under Data Quality Challenges in Manufacturing

The ability and rapid access to execution data and information in manufacturing workshops have been greatly improved with the wide spread of the Internet of Things and artificial intelligence technologies, enabling real-time unmanned integrated control of facilities and production. However, the widespread issue of data quality in the field raises concerns among users about the robustness of automatic decision-making models before their application. This paper addresses three main challenges relative to field data quality issues during automated real-time decision-making: parameter identification under measurement uncertainty, sensor accuracy selection, and sensor fault-tolerant control. To address these problems, this paper proposes a risk assessment framework in the case of continuous production workshops. The framework aims to determine a method for systematically assessing data quality issues in specific scenarios. It specifies the preparation requirements, as well as assumptions such as the preparation of datasets on typical working conditions, and the risk assessment model. Within the framework, the data quality issues in real-time decision-making are transformed into data deviation problems. By employing the Monte Carlo simulation method to measure the impact of these issues on the decision risk, a direct link between sensor quality and risks is established. This framework defines specific steps to address the three challenges. A case study in the steel industry confirms the effectiveness of the framework. This proposed method offers a new approach to assessing safety and reducing the risk of real-time unmanned automatic decision-making in industrial settings.

Machine learning-based outlier detection for pipeline in-line inspection data

Pipeline companies are facing challenges in maintaining the integrity and reliability of their pipelines. They are working towards predictive maintenance using machine learning-based approaches to predicting anomalies. Training machine learning models requires sufficient data. Data quality is therefore becoming important because inaccurate data will lead to an inaccurate or wrong decision on pipeline condition assessment and the following management. This research paper intends to address the data quality issues of pipeline inspection data such as in-line inspection (ILI) data using machine learning models. Different machine learning models developed by random forest regression, linear regression, and nearest neighbors’ methods were tested to detect outliers in the ILI data. In this paper, the ILI data collected from an oil pipeline over a period of 22 years was applied to testing and analysis. To verify the outlier detection results of machine learning models, we used statistical analysis including Z-score method to check and find if there are any gaps in the analysis. It verifies that all these methods show almost the same or very similar results for the detection of the outliers. Hence, this study presents a robust method for the field applications in the pipeline industry.