Data Quality Management System Research Articles

Review of the development of hydrological data quality control in Typhoon Committee Members

Nowadays, with the continual development of the science and technology applied in data observation, monitoring and collection, human has more and more means and channels to obtain various data, consequently, the amount of collected and stored data is also getting bigger and bigger. In recent years, hydro-meteorological data have multiplied in some Typhoon Committee (TC) Members. Data-based advanced technology applications in TC, such as application of Artificial Intelligent (AI) and impact-based typhoon disaster forecasting and early warning, has emerged one after another. A consistent and integrated data quality management system is crucial for ensuring accurate hydrological and meteorological analysis and prediction. Considering the importance and urgent necessary, TC working group on hydrology (WGH) conducted a cooperation project on data quality management in the past years with the major objective of improving the capacity of TC Members on integrated data quality control and processing. Despite the significant improvements, the uncertainties and difficulties in processing the full-elements of hydro-meteorological data still persist in hydro-meteorological data. To tackle these challenges and further enhance the data quality management system, the integration of AI technology shows great promise. By examining the data quality management system at World Meteorological Organization (WMO) as a starting point, this paper explored how related organizations in China, Japan, Malaysia, Philippines and Republic of Korea, manage the quality of hydro-meteorological data; reviewed the current status of hydro-meteorological data quality control in TC Members, and discussed the potential areas to be enhanced in future.

Automated Data Quality Control System in Health and Demographic Surveillance System

The automated data quality management system serves as a comprehensive solution developed to enhance the precision, dependability, and uniformity of data within data-driven organizations. Such systems play an important role in eliminating the shortcomings associated with manual data quality management, which is prevalent in health and demographic surveillance systems (HDSS). The ongoing difficulty of ensuring data quality through manual processing hinders the HDSS's capacity to optimize data quality effectively. To address this challenge, our study adopted design science methodologies to provide guidelines for the design and implementation of the automated data quality control system. The open source technologies (Pentaho data integration, R Studio, SQL, Windows task scheduler) were used to facilitate the automation and validation of the incoming and database resident data. The quality of data has vastly improved since the implementation of the proposed system. The findings suggest that the automated data quality control system exhibited superior performance compared to the manual methods, thereby minimizing errors and time-wasting efforts.

Challenges and Responses to ESG Risk Management

As Environmental, Social, and Governance (ESG) considerations gradually gain global prominence, the demand for ESG risk management by corporations is rapidly increasing. This is not only a manifestation of social responsibility, but also a prerequisite for sustainable corporate development. ESG risk management is not only crucial for the long-term interests of corporations themselves, but also influences investment decisions and public perceptions of corporations. However, in the implementation process, challenges such as uncertainty in data quality, regulatory compliance, the diversification of stakeholder needs, and the complexity of strategic integration have been encountered. To effectively address these issues, corporations need to establish and enhance ESG data quality management systems, adapt to and proactively respond to regulatory changes, strengthen communication with investors and other stakeholders, and promote deep integration of ESG strategies across various departments and levels within the corporation. This article will explore the challenges faced by ESG risk management and propose corresponding strategies and recommendations.

Challenges and Opportunities of Information Technology Management in Big Data Era

This paper comprehensively explores the challenges and opportunities in Information Technology management during the era of big data. It begins by delving into the definition, characteristics, and sources of big data, highlighting its paramount significance for enterprises and organizations. Addressing issues such as data security, privacy protection, processing capacity bottlenecks, and data quality and trustworthiness, effective strategies are proposed. Recommendations include establishing a robust data governance system, leveraging real-time computing, and employing efficient analytical tools, emphasizing reinforced data security measures and enhanced employee training. For tackling data quality and trustworthiness concerns, the paper underscores the importance of establishing a comprehensive data quality management system. Furthermore, a detailed examination of the application of big data in predictive analysis and decision support is provided, along with suggestions to foster a data-driven decision-making culture and enhance employee data analytics skills. In looking ahead, the paper encourages continuous attention to technological innovation, advocating adjustments in IT management strategies to align with the evolving demands of the big data era.

Development of Integrated Data Quality Management System for Observational Medical Outcomes Partnership Common Data Model.

The amount of research on the gathering and handling of healthcare data keeps growing. To support multi-center research, numerous institutions have sought to create a common data model (CDM). However, data quality issues continue to be a major obstacle in the development of CDM. To address these limitations, a data quality assessment system was created based on the representative data model OMOP CDM v5.3.1. Additionally, 2,433 advanced evaluation rules were created and incorporated into the system by mapping the rules of existing OMOP CDM quality assessment systems. The data quality of six hospitals was verified using the developed system and an overall error rate of 0.197% was confirmed. Finally, we proposed a plan for high-quality data generation and the evaluation of multi-center CDM quality.

Survey and Mapping Geographic Information Archive Informationization Management Construction Analysis

With the rapid development of science and technology and the rapid accumulation of geographic information data, the management of surveying and mapping geographic information archives is facing increasing challenges. The traditional manual management mode makes it difficult to meet the needs of large-scale data processing and sharing, so information management has become a key strategy to improve the efficiency of data management and optimize the use of resources. In this context, this paper discusses the construction of informatization management around surveying and mapping geographic information archives. The key steps in the construction of information management include the nodes of technical infrastructure, data standardization and organization, digital conversion of archives, archive retrieval and query system, security and authority management, and continuous updating and maintenance of archives. At the same time, to promote the construction of information management, work can be started from the formulation of a comprehensive information management plan, the establishment of cross-departmental collaboration mechanisms, the strengthening of personnel training and technical support, as well as the establishment of a data quality management system, to meet the needs of multiple fields and to realize the greater value of data.

Effective data quality management for electronic medical record data using SMART DATA

ObjectivesIn the medical field, we face many challenges, including the high cost of data collection and processing, difficult standards issues, and complex preprocessing techniques. It is necessary to establish an objective and systematic data quality management system that ensures data reliability, mitigates risks caused by incorrect data, reduces data management costs, and increases data utilization. We introduce the concept of SMART data in a data quality management system and conducted a case study using real-world data on colorectal cancer. MethodsWe defined the data quality management system from three aspects (Construction - Operation - Utilization) based on the life cycle of medical data. Based on this, we proposed the “SMART DATA” concept and tested it on colorectal cancer data, which is actual real-world data. ResultsWe define “SMART DATA” as systematized, high-quality data collected based on the life cycle of data construction, operation, and utilization through quality control activities for medical data. In this study, we selected a scenario using data on colorectal cancer patients from a single medical institution provided by the Clinical Oncology Network (CONNECT). As SMART DATA, we curated 1,724 learning data and 27 Clinically Critical Set (CCS) data for colorectal cancer prediction. These datasets contributed to the development and fine-tuning of the colorectal cancer prediction model, and it was determined that CCS cases had unique characteristics and patterns that warranted additional clinical review and consideration in the context of colorectal cancer prediction. ConclusionsIn this study, we conducted primary research to develop a medical data quality management system. This will standardize medical data extraction and quality control methods and increase the utilization of medical data. Ultimately, we aim to provide an opportunity to develop a medical data quality management methodology and contribute to the establishment of a medical data quality management system.

The impacts of poor data quality on business performance

In the digital era, data plays a pivotal role in the operational transactions of businesses across the globe. Along with human and capital resources, data is considered an invaluable asset for enterprises driving the success of business performance. Nevertheless, poor-quality data as inputs will negatively impact on the quality of the output and outcomes are the quality of decisions at individual, organizational, and corporate levels. Poor data quality will be a barrier for an enterprise to build an effective strategy and efficient management across its business function. Since the 1990s, businesses, and academic institutions in advanced economies such as the USA, France, Switzerland, Australia, and South Korea have focused on developing, implementing, and using data quality management systems to improve business operations. However, Vietnam has given less attention to data quality issues. This paper aims to summarize the introduction of data quality, the problems with data quality, and the impacts of poor data quality on operations at functional and strategic levels using the qualitative approach – literature synthesis and secondary data analysis. Thus, we propose some policy prescriptions for Vietnam’s industries and institutions alike.

Big Data between Quality and Security: Dynamic Access Control for Collaborative Platforms

Big Data often refers to a set of technologies dedicated to deal with large volumes of data. Data Quality and Data Security are two essential aspects for any Big Data project. While Data Quality Management Systems are about putting in place a set of processes to assess and improve certain characteristics of data such as Accuracy, Consistency, Completeness, Timeliness, etc., Security Systems are designed to protect the Confidentiality, Integrity and Availability of data. In a Big Data environment, data quality processes can be blocked by data security mechanisms. Indeed, data is often collected from external sources that could impose their own security policies. In many research works, it has been recognized that merging and integrating access control policies are real challenges for Big Data projects. To address this issue, we suggest in this paper a framework to secure data collection in collaborative platforms. Our framework extends and combines two existing frameworks namely: PolyOrBAC and SLA- Framework. PolyOrBAC is a framework intended for the protection of collaborative environments. SLA-Framework, for its part, is an implementation of the WS-Agreement Specification, the standard for managing bilaterally negotiable SLAs (Service Level Agreements) in distributed systems; its integration into PolyOrBAC will automate the implementation and application of security rules. The resulting framework will then be incorporated into a data quality assessment system to create a secure and dynamic collaborative activity in the Big Data context.

DQDF

Data quality assessment is an essential process of any data analysis process including machine learning. The process is time-consuming as it involves multiple independent data quality checks that are performed iteratively at scale on evolving data resulting from exploratory data analysis (EDA). Existing solutions that provide computational optimizations for data quality assessment often separate the data structure from its data quality which then requires efforts from users to explicitly maintain state-like information. They demand a certain level of distributed system knowledge to ensure high-level pipeline optimizations from data analysts who should instead be focusing on analyzing the data. We, therefore, propose data-quality-aware dataframes, a data quality management system embedded as part of a data analyst's familiar data structure, such as a Python dataframe. The framework automatically detects changes in datasets' metadata and exploits the context of each of the quality checks to provide efficient data quality assessment on ever-changing data. We demonstrate in our experiment that our approach can reduce the overall data quality evaluation runtime by 40-80% in both local and distributed setups with less than 10% increase in memory usage.

Distribution Network Operation Control Algorithm for Distributed Data Quality Management System

Distribution Network Operation Control Algorithm for Distributed Data Quality Management System

Big Data: Trade-off between Data Quality and Data Security

The essence of an information system lies in the data; if it is not of good quality or not sufficiently protected, the consequences will undoubtedly be harmful. Quality and Security are two essential aspects that add value to data and their implementation has become a real need and must be adopted before any data exploitation. Due to the high volume of data, their diversity and their rapid generation, effective implementation of such systems requires well thought out mechanisms and strategies. This paper provides an overview of Data Quality and Data Security in a Big Data context. We want through this paper to highlight the conflicts that may exist during the implementation of data security and data quality management systems. Such a conflict makes the complexity greater and requires new adapted solutions.

ISO 9001 International Standard, a Tool to Enhance Data Quality in Durable Socio-Technical Systems

The quality of data is recognized to be a key issue for the assets management in enterprises as data is the foundation of any decision making process. Recent research work has established that the quality of data is highly dependent on the knowledge one has on the socio-technical system being considered. Three modes of knowledge have been identified: knowing what, knowing how and knowing why. In this paper we focus on how to manage these modes of knowledge in durable socio-technical systems to enhance the data quality face to technological progress and employees turnover. We believe that an organization based on ISO 9001 international standard can provide a valuable framework to provide the data quality needed to an efficient decision making process. This framework has been applied to design the data quality management system within a high education socio-technical system. The most important benefits that have been noticed are: 1) a shared vision on the external clients of the system with a positive impact on the definition of the strategy and the objectives of the system and 2) a deep understanding of the data client-supplier relationship inside the socio-technical system. A direct consequence of these achievements was the increasing knowledge on “know-what” data to collect, “know-why” to collect that data and “know-how” to collect it.

A Study on the Problem Analysis and Improvement Plan of the Data Quality Management System of National R&D Data

National Science and Technology Information Service (NTIS) is constructed and operated to provide a comprehensive data provider service of national R&D data for improving the performance of the national R&D projects and disseminating the co-utilization of the national R&D data nationwide in Korea. The national R&D data is collected from 17 government departments and agencies. With the spread of the Government 3.0 paradigm, the demand for data sharing and utilization is being increased. However, the immature of the data quality management process for the public data hinders the progress on the data openness. This research presents the problems of the data quality management via a study on the current data quality management process/system of the national R&D data and proposes an improvement plan for the data quality management. In addition, DQC-M (Database Quality Certification-Management), which is a representative evaluation and certification model for assessing the quality and maturity of data quality management, is used to find insufficiency through the evaluation of the quality and matureness of the data quality management of NTIS service.

An integrated organisation-wide data quality management and information governance framework: theoretical underpinnings.

Increasing investment in eHealth aims to improve cost effectiveness and safety of care. Data extraction and aggregation can create new data products to improve professional practice and provide feedback to improve the quality of source data. A previous systematic review concluded that locally relevant clinical indicators and use of clinical record systems could support clinical governance. We aimed to extend and update the review with a theoretical framework. We searched PubMed, Medline, Web of Science, ABI Inform (Proquest) and Business Source Premier (EBSCO) using the terms curation, information ecosystem, data quality management (DQM), data governance, information governance (IG) and data stewardship. We focused on and analysed the scope of DQM and IG processes, theoretical frameworks, and determinants of the processing, quality assurance, presentation and sharing of data across the enterprise. There are good theoretical reasons for integrated governance, but there is variable alignment of DQM, IG and health system objectives across the health enterprise. Ethical constraints exist that require health information ecosystems to process data in ways that are aligned with improving health and system efficiency and ensuring patient safety. Despite an increasingly 'big-data' environment, DQM and IG in health services are still fragmented across the data production cycle. We extend current work on DQM and IG with a theoretical framework for integrated IG across the data cycle. The dimensions of this theory-based framework would require testing with qualitative and quantitative studies to examine the applicability and utility, along with an evaluation of its impact on data quality across the health enterprise.

Long-term diagnostics of precipitation estimates and the development of radar hardware monitoring within a radar product data quality management system

Quality is key to ensuring that the potential offered by weather radar networks is realized. To achieve optimum quality, a comprehensive radar data quality management system, designed to monitor the end-to-end radar data processing chain and evaluate product quality, is being developed at the UK Met Office. Three contrasting elements of this system are described: monitoring of key radar hardware performance indicators; generation of long-term integrations of radar products; and monitoring of radar reflectivity factor using synthesized observations from numerical weather prediction model fields. Examples of each component are presented and ways in which the different types of monitoring information have been used to both identify issues with the radar product data quality and help formulate solutions are given.Editor Z.W. Kundzewicz; Guest editor R.J. MooreCitation Harrison, D., Georgiou, S., Gaussiat, N., and Curtis, A., 2013. Long-term diagnostics of precipitation estimates and the development of radar hardware monitoring within a radar product data quality management system. Hydrological Sciences Journal, 59 (7), 1327–1342. http://dx.doi.org/10.1080/02626667.2013.841316

Explorative TTF Perspective on ERP Systems and Data Quality Management Interdependencies

Data quality management (DQM) gains importance due to the increasing amount and diversity of data as well as data’s critical impact on organizational success. Most organizations use enterprise resource planning (ERP) systems, establishing an enterprise-wide platform to integrate data and processes. There is a lack of sector-specific research of how ERP systems are used for DQM tasks in the insurance sector despite the sector’s specific characteristics. DQM plays a considerable role in the information-intensive insurance sector, with its numerous regulations, complex products and distribution structure. Following the task-technology fit (TTF) theory, we conducted semi-structured expert interviews in order to examine ERP systems’ use for DQM tasks and the interdependency of ERP systems and DQM. First, this paper shows that in the insurance sector, ERP systems are used to fulfill regulations and provide a starting point for data analysis. However, cultural and managerial challenges need to be addressed in order to lever the potential of diverse data for analysis. Second, we present a qualitative study applying the TTF theory. By doing so, we show that a qualitative application of TTF theory allows for rich insights and identi-fies potential causes of context-specific TTF problems. Furthermore, we present areas for further research on DQM and ERP systems and provide guidance for improving future use of ERP systems with respect to DQM.

Integrated Storage and Querying of Spatially Varying Data Quality Information in a Relational Spatial Database

AbstractThis article investigates a new, integrated technique for storing and retrieving spatially varying data quality information in a relational spatial database. Rather than storing global data quality statements, the system enables data quality information to be referenced to a spatial framework, individual spatial objects, or even parts of spatial objects. The integrated model, called as RDBMS for Spatial Variation in Quality (RSVQ), allows flexible storage of spatially varying data quality information, and seamless querying irrespective of the underlying storage model. RSVQ is founded on a formal model of relational databases, defining a new derived, polymorphic query operator to join quality data with spatial data. The operator is implemented in an extension to SQL as a new WITHQUALITY keyword. A performance evaluation of RSVQ was conducted, using an Oracle Spatial database and a case study of cadastral data for parts of Victoria, Australia. The results of this evaluation demonstrated that the system is practical and efficient for a wide range of queries, as well as indicating the performance trade‐offs associated with the different data quality storage models. Using the integrated RSVQ approach provides the potential for a single, consistent, database engine for a wide range of existing and proposed spatial data quality management systems.