Data Management Requirements Research Articles

ETL vs ELT: Evolving Approaches to Data Integration

This article explores the evolution of data integration approaches, focusing on the transition from Extract, Transform, Load (ETL) to Extract, Load, Transform (ELT) methodologies. It examines the characteristics, advantages, and challenges of both approaches in the context of modern data management requirements. The article discusses the factors driving the shift towards ELT, including cloud adoption, big data growth, real-time analytics demands, and the rise of data lake architectures. Additionally, it presents best practices for implementing ELT, covering areas such as data cataloging, version control, data quality checks, performance optimization, and security compliance. Through analysis of industry trends and real-world use cases, the article provides insights into why organizations are increasingly adopting ELT frameworks and the considerations involved in this transition.

Research Data Management in German Academia from a Multiple Logic Perspective

The requirements for research data management (RDM) have increased. Due to academia’s neoliberalization, however, researchers already face a high workload within a hypercompetitive environment. The demand to integrate RDM as an additional task into academics’ day-today actions seems to be quixotic. To deepen our understanding of early career researchers’ (ECRs) daily work arbitrage, we need to know more about their behavior, actions, and decisions in relation to RDM. Drawing on a multiple institutional logics perspective at the micro-level, we conducted 40 semistructured interviews at German higher education institutions (HEIs) to investigate how ECRs respond to institutional logics in the context of RDM. Our findings revealed three profiles – the conformist, the waverer, and the resister – that make use of different response strategies to the state, market, professional, and community logic. We contribute to institutional logic research at the micro-level and, in addition, broaden prior research on HEIs and RDM by taking neoliberal academia into account.

Transforming reservoir management in petroleum processing: a synergistic approach with IoT and Blockchain for enhanced emissions monitoring and regulatory reporting

This article proposes an innovative method to transform reservoir management in petroleum processing through the integration of the Internet of Things (IoT) and Blockchain technologies. In response to increasing demands for strict emissions monitoring and regulatory compliance, our approach aims to increase the precision, transparency, and security of data collection and reporting. In our proposed framework, we leverage IoT technology to deploy sensor networks across reservoirs and processing plants. The collected data is integrated into a Blockchain. This creates an immutable and transparent ledger for emissions-related information. Blockchain’s decentralized and tamper-proof features preserve data integrity and facilitate secure information sharing between stakeholders, including regulators and operators. Considering the specific scalability and performance requirements for data management in the petroleum industry, we propose a 4-layer system architecture. Our study examines the practical implementation and benefits of this innovative synergy.

Enhancing the Functionalities of Personal Health Record Systems: Empirical Study Based on the HL7 Personal Health Record System Functional Model Release 1.

The increasing demand for personal health record (PHR) systems is driven by individuals' desire to actively manage their health care. However, the limited functionality of current PHR systems has affected users' willingness to adopt them, leading to lower-than-expected usage rates. The HL7 (Health Level Seven) PHR System Functional Model (PHR-S FM) was proposed to address this issue, outlining all possible functionalities in PHR systems. Although the PHR-S FM provides a comprehensive theoretical framework, its practical effectiveness and applicability have not been fully explored. This study aimed to design and develop a tethered PHR prototype in accordance with the guidelines of the PHR-S FM. It sought to explore the feasibility of applying the PHR-S FM in PHR systems by comparing the prototype with the results of previous research. The PHR-S FM profile was defined to meet broad clinical data management requirements based on previous research. We designed and developed a PHR prototype as a web application using the Fast Healthcare Interoperability Resources R4 (FHIR) and Logical Observation Identifiers Names and Codes (LOINC) coding system for interoperability and data consistency. We validated the prototype using the Synthea dataset, which provided realistic synthetic medical records. In addition, we compared the results produced by the prototype with those of previous studies to evaluate the feasibility and implementation of the PHR-S FM framework. The PHR prototype was developed based on the PHR-S FM profile. We verified its functionality by demonstrating its ability to synchronize data with the FHIR server, effectively managing and displaying various health data types. Validation using the Synthea dataset confirmed the prototype's accuracy, achieving 100% coverage across 1157 data items. A comparison with the findings of previous studies indicated the feasibility of implementing the PHR-S FM and highlighted areas for future research and improvements. The results of this study offer valuable insights into the potential for practical application and broad adoption of the PHR-S FM in real-world health care settings.

Genotype and Phenotype Association Analysis Based on Multi-omics Statistical Data

Background: When using clinical data for multi-omics analysis, there are issues such as the insufficient number of omics data types and relatively small sample size due to the protection of patients' privacy, the requirements of data management by various institutions, and the relatively large number of features of each omics data. This paper describes the analysis of multi-omics pathway relationships using statistical data in the absence of clinical data. Methods: We proposed a novel approach to exploit easily accessible statistics in public databases. This approach introduces phenotypic associations that are not included in the clinical data and uses these data to build a three-layer heterogeneous network. To simplify the analysis, we decomposed the three-layer network into double two-layer networks to predict the weights of the inter-layer associations. By adding a hyperparameter β, the weights of the two layers of the network were merged, and then k-fold cross-validation was used to evaluate the accuracy of this method. In calculating the weights of the two-layer networks, the RWR with fixed restart probability was combined with PBMDA and CIPHER to generate the PCRWR with biased weights and improved accuracy. Results: The area under the receiver operating characteristic curve was increased by approximately 7% in the case of the RWR with initial weights. Conclusion: Multi-omics statistical data were used to establish genotype and phenotype correlation networks for analysis, which was similar to the effect of clinical multi-omics analysis.

TDSQL: Tencent Distributed Database System

Distributed databases have become indispensable in contemporary computing and data processing, owing to their pivotal role in ensuring high availability and scalability. They effectively cater to the requirements of data management and high-concurrency access. However, developing a distributed database system that is well-suited for diverse application scenarios, particularly for large-scale applications, presents several challenges. These challenges include ensuring data consistency and achieving high levels of performance. This paper presents TDSQL, a distributed database system that prioritizes core design principles of distributed systems, including high availability, strong consistency, and scalability. In particular, TDSQL has achieved high performance through over a decade of practical experience and optimization in various modules, such as the kernel, synchronous replication, and transaction processing, in large-scale application scenarios. By conducting the TPC-C benchmark test, TDSQL demonstrated outstanding performance, achieving a throughput of 814 million tpmC across 1650 database nodes, with a jitter rate of less than 0.2%. This jitter rate is an order of magnitude lower than the standard required, showcasing the system's stability and reliability. During the 8-hour TPC-C standard stress test, TDSQL successfully completed over 860 billion transactions and processed 40 trillion order details, with zero forced rollbacks and zero data inconsistency.

INTEGRATION OF MACHINE LEARNING-BASED DETECTION SYSTEMS INTO AUTONOMOUS VEHICLES

The incorporation of machine learning (ML) driven detecting systems into autonomous vehicles (AVs) signifies a revolutionary advancement in contemporary transportation. The present study investigates the use of ML systems, namely in the domains of traffic sign identification, pedestrian detection, and obstacle resolution. The work deals with a review of the technical advances in neural networks concerning their capability for real-time data processing for accurate and reliable navigation. Further, this paper points out that the challenges to be faced during the implementation of these systems include requirements for data management, high-performance computing, and exploitation of cloud technologies for scalable solutions. The results indicate that machine learningbased detection strategies greatly enhance autonomous vehicle performance and safety, while emphasising persistent issues concerning security and legal frameworks. This paper highlights the significance of ongoing technological advancements in machine learning and its influence on the development of autonomous driving.

Integration of Airborne Laser Scanning data into forest ecosystem management in Canada: Current status and future directions

Airborne Laser Scanning (ALS) has been the subject of decades of applied research and development in forest management. ALS data are spatially explicit, capable of accurately characterizing vegetation structure and underlying terrain, and can be used to produce value-added products for terrestrial carbon assessments, hydrology, and biodiversity among others. Scientific support for ALS is robust, however its adoption within environmental decision-making frameworks remains inconsistent. Cost continues to be a principal barrier limiting adoption, especially in remote, forested regions, however added challenges such as the need for technical expertise, unfamiliarity of data capabilities and limitations, data management requirements, and processing logistics also contribute. This review examines the current status of the integration of ALS data into forest ecosystem management in a Canadian context. We advocate for continued inter-agency acquisitions leading to integration of ALS into existing natural resource management decision pathways. We gauge the level of uptake thus far, discuss the barriers to operational implementation at provincial scales, and highlight how we believe ALS can support multiple objectives of forest and environmental management in Canada. We speak to potential benefits for supporting inter-agency terrain generation, ecosystem mapping, biodiversity assessments, silvicultural planning, carbon and forest health evaluations, and riparian characterizations. We conclude by providing key considerations for developing capacity using ALS and discuss the technologies future in the context of Canadian forest and environmental management objectives.

Investigating the Roles and Responsibilities of Institutional Signing Officials After Data Sharing Policy Reform for Federally Funded Research in the United States: National Survey.

New federal policies along with rapid growth in data generation, storage, and analysis tools are together driving scientific data sharing in the United States. At the same, triangulating human research data from diverse sources can also create situations where data are used for future research in ways that individuals and communities may consider objectionable. Institutional gatekeepers, namely, signing officials (SOs), are therefore at the helm of compliant management and sharing of human data for research. Of those with data governance responsibilities, SOs most often serve as signatories for investigators who deposit, access, and share research data between institutions. Although SOs play important leadership roles in compliant data sharing, we know surprisingly little about their scope of work, roles, and oversight responsibilities. The purpose of this study was to describe existing institutional policies and practices of US SOs who manage human genomic data access, as well as how these may change in the wake of new Data Management and Sharing requirements for National Institutes of Health-funded research in the United States. We administered an anonymous survey to institutional SOs recruited from biomedical research institutions across the United States. Survey items probed where data generated from extramurally funded research are deposited, how researchers outside the institution access these data, and what happens to these data after extramural funding ends. In total, 56 institutional SOs participated in the survey. We found that SOs frequently approve duplicate data deposits and impose stricter access controls when data use limitations are unclear or unspecified. In addition, 21% (n=12) of SOs knew where data from federally funded projects are deposited after project funding sunsets. As a consequence, most investigators deposit their scientific data into "a National Institutes of Health-funded repository" to meet the Data Management and Sharing requirements but also within the "institution's own repository" or a third-party repository. Our findings inform 5 policy recommendations and best practices for US SOs to improve coordination and develop comprehensive and consistent data governance policies that balance the need for scientific progress with effective human data protections.

A Common Data Dictionary and Common Data Model for Additive Manufacturing

Additive manufacturing (AM) leverages emerging technologies and well-adopted processes to produce near-net-shape products. The advancement of AM technology requires data management tools to collect, store, and share information through the product development lifecycle and across the material and machine value chain. To address the need for sharing data among AM developers and practitioners, an AM common data dictionary (AM-CDD) was first developed based on community consensus to provide a common lexicon for AM, and later standardized by ASTM International. Following the AM-CDD work, the development of a common data model (AM-CDM) defining the structure and relationships of the key concepts, and terms in the AM-CDD is being developed. These efforts have greatly facilitated system integrations and AM data exchanges among various organizations. This work outlines the effort to create the AM-CDD and AM-CDM, with a focus on the design of the AM-CDM. Two use cases are provided to demonstrate the adoption of these efforts and the interoperability enabled by the AM-CDM for different engineering applications managed by different types of database technology. In these case studies, the AM-CDM is implemented in two distinct formats to curate AM data from NIST—the first in XML from their additive manufacturing material database and the second in OWL from their 2022 AM bench database. These use cases present the power of the AM-CDM for data representation, querying, and seamless data exchange. Our implementation experiences and some challenges are highlighted that can assist others in future adoptions of the AM-CDM for data integration and data exchange applications.

Selecting an Ecological Momentary Assessment Platform: Tutorial for Researchers.

Although ecological momentary assessment (EMA) has been applied in psychological research for decades, delivery methods have evolved with the proliferation of digital technology. Technological advances have engendered opportunities for enhanced accessibility, convenience, measurement precision, and integration with wearable sensors. Notwithstanding, researchers must navigate novel complexities in EMA research design and implementation. In this paper, we aimed to provide guidance on platform selection for clinical scientists launching EMA studies. Our team includes diverse specialties in child and adolescent behavioral and mental health with varying expertise on EMA platforms (eg, users and developers). We (2 research sites) evaluated EMA platforms with the goal of identifying the platform or platforms with the best fit for our research. We created a list of extant EMA platforms; conducted a web-based review; considered institutional security, privacy, and data management requirements; met with developers; and evaluated each of the candidate EMA platforms for 1 week. We selected 2 different EMA platforms, rather than a single platform, for use at our 2 research sites. Our results underscore the importance of platform selection driven by individualized and prioritized laboratory needs; there is no single, ideal platform for EMA researchers. In addition, our project generated 11 considerations for researchers in selecting an EMA platform: (1) location; (2) developer involvement; (3) sample characteristics; (4) onboarding; (5) survey design features; (6) sampling scheme and scheduling; (7) viewing results; (8) dashboards; (9) security, privacy, and data management; (10) pricing and cost structure; and (11) future directions. Furthermore, our project yielded a suggested timeline for the EMA platform selection process. This study will guide scientists initiating studies using EMA, an in vivo, real-time research tool with tremendous promise for facilitating advances in psychological assessment and intervention.

Web-based information system framework for the digitization of historical databases and endowments

With the digitization of historical databases and endowments, care must be taken when designing the framework for an information system on the web. Because conflicts arise frequently in reality, different data management requirements are necessary for the preservation of waqf property. For the purpose of creating and putting into place historical information systems and endowments for this inheritance, it is necessary to develop an acceptable management plan. An inheritance that is thought to be distinct from customary ones since it is governed by its own law is referred to as waqf, as an example. They typically comprise histories and endowments that need to be protected to ensure sustenance among the population and to ensure they live up to the standards of the community and country. This research was compiled and analyzed to support stakeholders in producing a more practical, focused, and value-delivery framework. The datasets were mapped based on relationships, graph databases, and semantic networks. Moreover, the framework was developed using several data representation models to ensure easier, faster, and more accurate methods of displaying the data. Relationships, graph databases, and semantic networks were used to map the datasets. The design was made available to users, administrators, and managers, with the latter group being in charge of maintaining data control over each entity. The case study was conducted using historical information and waqf from the Nadzir Pangeran Sumedang Indonesia Waqf Foundation (YNWPS) in the Kingdom of Sumedang Larang Indonesia (KSL).` The creation of a web-based information system to keep track of the data in each entity and ensure better preservation of historical genealogical databases and endowments was made simpler by the structured framework design.

Data Management across Geographically-Distributed Autonomous Systems: Architecture, Implementation, and Performance Evaluation

Supporting efficient geographically-distributed autonomous data management is one of the critical obstacles for opening up a big data era. It creates the need for investigating such kind of systems in big data environments. In this paper, a distributed autonomous data management system is put forward, exhibiting the following features. (1) A distributed architecture designed to meet the requirements of autonomous data management by allowing interconnection, intercommunication, and interoperation of multiple sites over the Internet. (2) An autonomous, multi-level, unstructured data storage system to meet high-efficiency storage needs, with reference to the distributed heterogeneous data storage theories. (3) A distributed autonomous data indexing and retrieval system to support metadata search & full-text searching, fast loading, remote access, and unified view. Experimental and industrial application results demonstrate that the proposed system has high potential to reduce access time and improve storage efficiency, while maintaining satisfactory availability and scalability.

An innovative technological infrastructure for managing SARS-CoV-2 data across different cohorts in compliance with General Data Protection Regulation.

The ORCHESTRA project, funded by the European Commission, aims to create a pan-European cohort built on existing and new large-scale population cohorts to help rapidly advance the knowledge related to the prevention of the SARS-CoV-2 infection and the management of COVID-19 and its long-term sequelae. The integration and analysis of the very heterogeneous health data pose the challenge of building an innovative technological infrastructure as the foundation of a dedicated framework for data management that should address the regulatory requirements such as the General Data Protection Regulation (GDPR). The three participating Supercomputing European Centres (CINECA - Italy, CINES - France and HLRS - Germany) designed and deployed a dedicated infrastructure to fulfil the functional requirements for data management to ensure sensitive biomedical data confidentiality/privacy, integrity, and security. Besides the technological issues, many methodological aspects have been considered: Berlin Institute of Health (BIH), Charité provided its expertise both for data protection, information security, and data harmonisation/standardisation. The resulting infrastructure is based on a multi-layer approach that integrates several security measures to ensure data protection. A centralised Data Collection Platform has been established in the Italian National Hub while, for the use cases in which data sharing is not possible due to privacy restrictions, a distributed approach for Federated Analysis has been considered. A Data Portal is available as a centralised point of access for non-sensitive data and results, according to findability, accessibility, interoperability, and reusability (FAIR) data principles. This technological infrastructure has been used to support significative data exchange between population cohorts and to publish important scientific results related to SARS-CoV-2. Considering the increasing demand for data usage in accordance with the requirements of the GDPR regulations, the experience gained in the project and the infrastructure released for the ORCHESTRA project can act as a model to manage future public health threats. Other projects could benefit from the results achieved by ORCHESTRA by building upon the available standardisation of variables, design of the architecture, and process used for GDPR compliance.

Data privacy and DNA data

The letter to the Editor is in response to the manuscript by Hertzog et al. (2023) titled "Data management instruments to Protect the personal information of Children and Adolescents in sub-Saharan Africa." The letter elaborates on personal data protection, particularly the POPI Act's data management requirements; the DNA Act mandates specific measures to ensure the data integrity and security of the NFDD's information. In addition, it criminalises the misuse or compromise of the data's integrity within the NFDD. In addition, the DNA Act established the National Forensic Oversight and Ethical Board (NFOEB), which is responsible for overseeing ethical compliance, implementing the Act, and preserving data integrity within the NFDD. The NFOEB is also responsible for investigating any complaints regarding DNA forensics and the management of the NFDD.

What about Model Data? Best Practices for Preservation and Replicability

Abstract It has become common for researchers to make their data publicly available to meet the data management and accessibility requirements of funding agencies and scientific publishers. However, many researchers face the challenge of determining what data to preserve and share and where to preserve and share those data. This can be especially challenging for those who run dynamical models, which can produce complex, voluminous data outputs, and have not considered what outputs may need to be preserved and shared as part of the project design. This manuscript presents findings from the NSF EarthCube Research Coordination Network project titled “What About Model Data? Best Practices for Preservation and Replicability” (https://modeldatarcn.github.io/). These findings suggest that if the primary goal of sharing data are to communicate knowledge, most simulation-based research projects only need to preserve and share selected model outputs along with the full simulation experiment workflow. One major result of this project has been the development of a rubric, designed to provide guidance for making decisions on what simulation output needs to be preserved and shared in trusted community repositories to achieve the goal of knowledge communication. This rubric, along with use cases for selected projects, provide scientists with guidance on data accessibility requirements in the planning process of research, allowing for more thoughtful development of data management plans and funding requests. Additionally, this rubric can be referred to by publishers for what is expected in terms of data accessibility for publication.

The New Information Retrieval Problem: Data Availability

ABSTRACTThe goals of open science are driven by policies requiring data management, sharing, and accessibility. One way of measuring the impact of open science policies on scientific knowledge is to access data that has been prepared for re‐use. But how accessible/available are data resources? In this paper, we discuss a method for exploring and locating datasets made available by scientists from federally funded projects in the US. The data pathways method was tested on federal awards. Here we describe the method and the results from analyzing fifty federal awards granted by the National Science Foundation to pursue data resources and their availability in publications, data repositories, or institutional repositories. The data pathways approach contributes to the development of a practical approach on availability that captures the current ways in which data are accessible from federally funded science projects –ranging from institutional repositories, journal data deposit, PI and project web pages, and science data platforms, among other found possibilities. This paper discusses some background and motivations for such a method, the method, research design, barriers encountered when searching for data resources from projects, and how this method can be useful to future studies of data availability.

Opportunities and Limits of a Disciplinary Repository Using the Example MO|RE data (eResearch Infrastructure for Motor Research Data)

Out of two funding periods by the German Research Foundation resulted the first disciplinary repository for sports science motor activity research data MO|RE data. MO|RE data addresses sports scientists, researchers from related disciplines and practitioners such as teachers and educators, which work with or generate human motor performance test data as well. It has five main functions: publishing, storing, searching, citing and mapping. There are still some limitations as well as not exploited opportunities. Opportunities include among others the international expansion. Limitations are e.g., that linking health and motor test data is difficult due to data protection law. The sharing of sensitive data is not possible with the current concept and needs further solutions as e.g., remote access or workstations for guest researchers. In future, those data sets must be kept in mind to cover the requirements of research data management in sports science. Overall, the need for further development and optimization of MO|RE data repository in sports science becomes apparent to maximize its potential and ensure that it meets the evolving needs of researchers in the field.

Automated Documentation of Research Processes Using RDM

Published research results usually represent only a fraction of the data generated at a research institute. The unpublished data created in the process of producing the final result, however, often contain valuable information that can be reused. Through research data management, all these data should be stored centrally according to the FAIR principles (Findable, Accessible, Interoperable, Reusable). However, a significant part of knowledge is often not found in the data, but in the processes that led to their generation. It is therefore important to map these processes to archive and document this knowledge in a structured way. Procedures for documenting scientific processes already exist and are actively used at research institutes. However, these are often analogue or paper-based and hence do not meet the requirements for FAIR data management. At the Institute for Microstructure Technology of the KIT, such a paper-based procedure is used to document the production of microstructure components. During their manufacturing, it is essential to adhere to the correct process parameters in order to enable error-free production. Therefore, a so-called job ticket always accompanies the production of components. On this job ticket, the correct process sequence is listed and a detailed description of the respective process step is given. Depending on the component to be produced, a distinction is made between different types of job tickets according to internal conventions. On the one hand, there are so-called green job tickets, which describe a standardised process sequence, and on the other hand, blue job tickets, which are intended to document experimental manufacturing processes. The process sequence on the blue job tickets is initially empty and is filled in during the manufacturing process. Common to both types of job tickets is that they are stored in the institute's archive after completion of the component production. However, since the job tickets are paper-based, the corresponding archive of job tickets cannot be searched quickly and, given the sheer volume of archived job tickets, represents an unmanageable collection of data. The existing system for process documentation is therefore to be implemented with the help of the research data infrastructure Kadi4Mat [1] in accordance with FAIR principles, thereby making the available process knowledge more accessible.

Mpitb: A toolbox for multidimensional poverty indices

In this article, I present mpitb, a toolbox for multidimensional poverty indices (MPIs). The package mpitb comprises several subcommands to facilitate specification, estimation, and analyses of MPIs and supports the popular Alkire– Foster framework to multidimensional poverty measurement. mpitb offers several benefits to researchers, analysts, and practitioners working on MPIs, including substantial time savings (for example, because of lower data management and programming requirements) while allowing for a more comprehensive analysis at the same time. Aside from various convenience functions, mpitb also provides low-level tools for advanced users and programmers.