Data Management Challenges Research Articles

Challenges in clinical interpretation of next-generation sequencing data: Advantages and Pitfalls

Next-generation sequencing (NGS) technologies have completely changed the field of genomics by making it possible to rapidly and inexpensively produce genome-scale sequence data with higher precision and resolution. In recent years, rapid technological advancements driven by companies and academic institutions have expanded the range of NGS applications, from research to the clinic. Latest scientific developments have highlighted the effects of NGS technology on complex and Mendelian diseases, particularly cancer. However, the rapid adoption of NGS poses significant challenges in data processing, storage, management, and interpretation, as well as sequencing quality control, making it challenging to integrate the sequence data into clinical practice. Here, we first outline the technical features and performance of the most recent NGS platforms. Furthermore, we summarize the applications of NGS technologies in clinical diagnostics, data interpretation, and valuable bioinformatic tools and databases for analysis. Common issues in NGS processes are also discussed.

Improving Biological Collections Data through Human-AI Collaboration

Biological collections play a crucial role in our understanding of biodiversity and inform research in areas such as biosecurity, conservation, human health and climate change. In recent years, the digitisation of biological specimen collections has emerged as a vital mechanism for preserving and facilitating access to these invaluable scientific datasets. However, the growing volume of specimens and associated data presents significant challenges for curation and data management. By leveraging human-Artificial Intelligence (AI) collaborations, we aim to transform the way biological collections are curated and managed, unlocking their full potential in addressing global challenges. We present our initial contribution to this field through the development of a software prototype to improve metadata extraction from digital specimen images in biological collections. The prototype provides an easy-to-use platform for collaborating with web-based AI services, such as Google Vision and OpenAI Generative Pre-trained Transformer (GPT) Large Language Models (LLM). We demonstrate its effectiveness when applied to herbarium and insect specimen images. Machine-human collaboration may occur at various points within the workflows and can significantly affect outcomes. Initial trials suggest that the visual display of AI model uncertainty could be useful during expert data curation. While much work remains to be done, our results indicate that collaboration between humans and AI models can significantly improve the digitisation rate of biological specimens and thereby enable faster global access to this vital data. Finally, we introduce our broader vision for improving biological collection curation and management using human-AI collaborative methods. We explore the rationale behind this approach and the potential benefits of adding AI-based assistants to collection teams. We also examine future possibilities and the concept of creating 'digital colleagues' for seamless collaboration between human and digital curators. This ‘collaborative intelligence’ will enable us to make better use of both human and machine capabilities to achieve the goal of unlocking and improving our use of these vital biodiversity data to tackle real-world problems.

DAPHNE4NFDI and PUNCH4NFDI Consortia in the NFDI

The DAPHNE4NFDI and PUNCH4NFDI consortia represent the large scale facilities in the German physical sciences community. Work in DAPHNE4NFDI and PUNCH4NFDI is characterised by the use of large-scale research infrastructures – reactors, light sources, accelerators, telescopes, observatories, satellites – that serve international research communities of up to several thousand users and produce data in the terrabyte, often petabyte and in future exabyte range. The communities will have to master massive challenges in data management, building on and extending their leadership in “big data” management, distributed computing, multi-user management, and data loss / data irreversibility issues.

Machine Learning Applications in Sarcopenia Detection and Management: A Comprehensive Survey.

This extensive review examines sarcopenia, a condition characterized by a loss of muscle mass, stamina, and physical performance, with a particular emphasis on its detection and management using contemporary technologies. It highlights the lack of global agreement or standardization regarding the definition of sarcopenia and the various techniques used to measure muscle mass, stamina, and physical performance. The distinctive criteria employed by the European Working Group on Sarcopenia in Older People (EWGSOP) and the Asian Working Group for Sarcopenia (AWGSOP) for diagnosing sarcopenia are examined, emphasizing potential obstacles in comparing research results across studies. The paper delves into the use of machine learning techniques in sarcopenia detection and diagnosis, noting challenges such as data accessibility, data imbalance, and feature selection. It suggests that wearable devices, like activity trackers and smartwatches, could offer valuable insights into sarcopenia progression and aid individuals in monitoring and managing their condition. Additionally, the paper investigates the potential of blockchain technology and edge computing in healthcare data storage, discussing models and systems that leverage these technologies to secure patient data privacy and enhance personal health information management. However, it acknowledges the limitations of these models and systems, including inefficiencies in handling large volumes of medical data and the lack of dynamic selection capability. In conclusion, the paper provides a comprehensive summary of current sarcopenia research, emphasizing the potential of modern technologies in enhancing the detection and management of the condition while also highlighting the need for further research to address challenges in standardization, data management, and effective technology use.

Adaptive error bounded piecewise linear approximation for time-series representation

Error-bounded piecewise linear approximation (l∞-PLA) has been proven effective in addressing challenges in data management and analytics. It works by approximating the original time series with linear segments such that the approximation error on each data point does not exceed a pre-defined threshold. To achieve this goal, most prior works on l∞-PLA use a fixed error threshold during the entire approximation process, assuming a stable time series. However, in many real-world applications, the distribution of values on the time series undergoes substantial changes in different temporal stages. This introduces a need to adaptively tune the error threshold based on the properties of the time series while considering the trade-off between representation error and storage resources. In this work, we propose a general framework for constructing l∞-PLA with adaptive error bounds (AEPLA). It works by dividing the original time series into a set of intervals and assigns adaptive error bounds to these sub-sequences based on their fluctuation levels. Next, these sub-sequences are approximated using a user-defined l∞-PLA method. We implement this framework by employing three different types of l∞-PLA methods and evaluate the performance of our approach on an extensive set of real-world time-series datasets from industrial and scientific domains. Our experiments show that constructing l∞-PLA using the AEPLA framework can provide better a trade-off between representation error and storage resources and achieves lower time and space costs than applying the original l∞-PLA methods.

Perceptions of chief clinical information officers on the state of electronic health records systems interoperability in NHS England: a qualitative interview study

BackgroundIn the era of electronic health records (EHR), the ability to share clinical data is a key facilitator of healthcare delivery. Since the introduction of EHRs, this aspect has been extensively studied from the perspective of healthcare providers. Less often explored are the day-to-day challenges surrounding the procurement, deployment, maintenance, and use of interoperable EHR systems, from the perspective of healthcare administrators, such as chief clinical information officers (CCIOs).ObjectiveOur study aims to capture the perceptions of CCIOs on the current state of EHR interoperability in the NHS, its impact on patient safety, the perceived facilitators and barriers to improving EHR interoperability, and what the future of EHR development in the NHS may entail.MethodsSemi-structured interviews were conducted between November 2020 – October 2021. Convenience sampling was employed to recruit NHS England CCIOs. Interviews were digitally recorded and transcribed verbatim. A thematic analysis was performed by two independent researchers to identify emerging themes.ResultsFifteen CCIOs participated in the study. Participants reported that limited EHR interoperability contributed to the inability to easily access and transfer data into a unified source, thus resulting in data fragmentation. The resulting lack of clarity on patients' health status negatively impacts patient safety through suboptimal care coordination, duplication of efforts, and more defensive practice. Facilitators to improving interoperability included the recognition of the need by clinicians, patient expectations, and the inherent centralised nature of the NHS. Barriers included systems usability difficulties, and institutional, data management, and financial-related challenges. Looking ahead, participants acknowledged that realising that vision across the NHS would require a renewed focus on mandating data standards, user-centred design, greater patient involvement, and encouraging inter-organisational collaboration.ConclusionTackling poor interoperability will require solutions both at the technical level and in the wider policy context. This will involve demanding interoperability functionalities from the outset in procurement contracts, fostering greater inter-organisation cooperation on implementation strategies, and encouraging systems vendors to prioritise interoperability in their products. Only by comprehensively addressing these challenges would the full potential promised by the use of fully interoperable EHRs be realised.

Showcasing Data Management Challenges for Future IoT Applications with NebulaStream

Data management systems will face several new challenges in supporting IoT applications during the coming years. These challenges arise from managing large numbers of heterogeneous IoT devices and require combining elastic cloud and fog resources in unified fog-cloud environments. In this demonstration, we introduce a smart city simulation called IoTropolis and use it to create interactive eHealth and Smart Grid application scenarios. We use these scenarios to showcase three key challenges of unified fog-cloud environments. Furthermore, we demonstrate how our recently proposed data management system for the IoT NebulaStream addresses these challenges. Visitors to our demonstration can configure and interact with the scenarios to manage electricity usage in IoTropolis or to distribute patients across different hospitals. Thereby, visitors can actively engage with the challenges showcased by IoTropolis and utilize NebulaStream to address them. As a result, our demonstration enables visitors to experience data management for future IoT applications.

The dark side of FinTech in financial services: a qualitative enquiry into FinTech developers’ perspective

Purpose FinTech offers numerous prospects for significant enhancements and fundamental changes in financial services. However, along with the myriad of benefits, it also has the potential to induce risks to individuals, organisations and society. This study focuses on understanding FinTech developers’ perspective of the dark side of FinTech.Design/methodology/approach This study conducted semi-structured interviews with 23 Nigerian FinTech developers using an exploratory, inductive methodology The data were transcribed and then thematically analysed using NVivo.Findings Three themes – customer vulnerability, technical inability and regulatory irresponsibility – arose from the thematic analysis. The poor existing technological infrastructure, data management challenges, limited access to data and smartphone adoption pose challenges to a speedy integration of FinTech in the country, making customers vulnerable. The lack of privacy control leads to ethical issues. The lack of skilled developers and the brain drain of good developers present additional obstacles to the development of FinTech in Nigeria.Research limitations/implications FinTech operation in a developing country differs from that in developed countries with better technological infrastructure and institutional acceptance. This study recognises that basic banking operations through FinTech are still not well adopted, necessitating the need to be more open-minded about the global practicalities of FinTech.Practical implications FinTech managers, banks and policymakers can ethically collect consumer data that can help influence customer credit decisions, product development and recommendations using the mobile app and transaction history. There should be strict penalties on FinTech for selling customers’ data, sending unsolicited messages or gaining unnecessary access to the customer’s contact list. FinTech can offer to educate consumers about their financial management skills.Originality/valueWhereas other studies have focused on the positive aspects of FinTech to understand client perceptions, this study offers new insights into the dark side of FinTech by analysing the viewpoints of FinTech developers. Furthermore, the study is based in Nigeria, an emerging economy adopting FinTech, adding a new dimension to the body of knowledge.

Addressing Policy Challenges of Disruptive Technologies

ABSTRACT This special issue examines the policy challenges and government responses to disruptive technologies. It explores the risks, benefits, and trade-offs of deploying disruptive technologies, and examines the efficacy of traditional governance approaches and the need for new regulatory and governance frameworks. Key themes include the need for government stewardship, taking adaptive and proactive approaches, developing comprehensive policies accounting for technical, social, economic, and political dimensions, conducting interdisciplinary research, and addressing data management and privacy challenges. The findings enhance understanding of how governments can navigate the complexities of disruptive technologies and develop policies to maximize benefits and mitigate risks.

Key Aspects and Challenges in the Implementation of Energy Communities

Energy communities (ECs) are an important tool towards a fair energy transition. Hence, the European Union (EU) has positioned ECs at the centre of its energy strategy and the foreseen transformation of its energy system. This paper aims to give an overview of key aspects and challenges for the implementation of the EC concept. Firstly, the regulatory framework is examined with a focus on the new definitions for ECs introduced by the EU, Renewable Energy Communities (RECs) and Citizen Energy Communities (CECs). Secondly, examples of established ECs and their main objectives are mentioned. Additionally, based on the identified challenges and requirements of establishing ECs, the key technologies that are implemented or have the potential to be deployed in an EC are examined, as well as innovative cross-cutting services that are optimally suited to be integrated in an EC. Moreover, the data management challenges linked to some of these technologies are considered. Finally, an overview of actual or potential financing schemes to support the EC development is given. Overall, the analysis highlighted the regulatory, technical and financial aspects and challenges that ECs are facing and the need to address them so that the EC concept is effective and successful. The main challenges identified for each of these aspects are the regulatory compliance with the legal framework, the data management dimension when innovative technological concepts are adopted and the financing of new projects.

Partnering with health sciences libraries to address challenges in bioimaging data management and sharing.

Federal mandates, publishing requirements, and an interest in open science have all generated renewed attention on research data management and, in particular, data sharing practices. Due to the size and types of data they produce, bioimaging researchers confront specific challenges in aligning their data with FAIR principles, ensuring that it is findable, accessible, interoperable, and reusable. Although not always recognized by researchers, libraries can, and have been, offering support for data throughout its lifecycle by assisting with data management planning, acquisition, processing and analysis, and sharing and reuse of data. Libraries can educate researchers on best practices for research data management and sharing, facilitate connections to experts by coordinating sessions using peer educators and appropriate vendors, help assess the needs of different researcher groups to identify challenges or gaps, recommend appropriate repositories to make data as accessible as possible, and comply with funder and publisher requirements. As a centralized service within an institution, health sciences libraries have the capability to bridge silos and connect bioimaging researchers with specialized data support across campus and beyond.

FAIR for digital twins

The continuing drive towards digitization in manufacturing leads to an increasing number of digital twins for monitoring and controlling all kinds of processes. While these capture crucial data of all individual steps and allow for analysis and optimization, more often than not the underlying models are confined to individual systems or organizations. This hinders data exchange, especially across institutional borders and thus represents an important barrier for economic success. Similar challenges in the scientific community led to the emergence of the FAIR principles (Findable, Accessible, Interoperable, and Reusable) as guidelines towards a sustainable data landscape. Despite the growing presence within academia, their transfer to industry has not yet received similar attention. We argue that the existing efforts and experiences in science can be exploited to address current data management challenges in industry as well. An improved data exchange within organizations and beyond can not just lower costs, but also opens up new opportunities ranging from discovering new suppliers or partners to improving existing value chains.

Big data in contemporary electron microscopy: challenges and opportunities in data transfer, compute and management.

The second decade of the twenty-first century witnessed a new challenge in the handling of microscopy data. Big data, data deluge, large data, data compliance, data analytics, data integrity, data interoperability, data retention and data lifecycle are terms that have introduced themselves to the electron microscopy sciences. This is largely attributed to the booming development of new microscopy hardware tools. As a result, large digital image files with an average size of one terabyte within one single acquisition session is not uncommon nowadays, especially in the field of cryogenic electron microscopy. This brings along numerous challenges in data transfer, compute and management. In this review, we will discuss in detail the current state of international knowledge on big data in contemporary electron microscopy and how big data can be transferred, computed and managed efficiently and sustainably. Workflows, solutions, approaches and suggestions will be provided, with the example of the latest experiences in Australia. Finally, important principles such as data integrity, data lifetime and the FAIR and CARE principles will be considered.

Blockchain-Based Framework for Interoperable Electronic Health Records for an Improved Healthcare System

The healthcare industry has been transitioning from paper-based medical records to electronic health records (EHRs) in most healthcare facilities. However, the current EHR frameworks face challenges in secure data storage, credibility, and management. Interoperability and user control of personal data are also significant concerns in the healthcare sector. Although block chain technology has emerged as a powerful solution that can offer the properties of immutability, security, and user control on stored records, its potential application in EHR frameworks is not yet fully understood. To address this gap in knowledge, this research aims to provide an interoperable blockchain-based EHR framework that can fulfill the requirements defined by various national and international EHR standards such as HIPAA and HL7. The research method employed is a systematic literature review to explore the current state of the art in the field of EHRs, including blockchain-based implementations of EHRs. The study defines the interoperability issues in the existing blockchain-based EHR frameworks, reviews various national and international standards of EHR, and further defines the interoperability requirements based on these standards. The proposed framework can offer safer methods to interchange health information for the healthcare sector and can provide the properties of immutability, security, and user control on stored records without the need for centralized storage. The contributions of this work include enhancing the understanding of the potential application of blockchain technology in EHR frameworks and proposing an interoperable blockchain-based EHR framework that can fulfill the requirements defined by various national and international EHR standards. Overall, this study has significant implications for the healthcare sector, as it can enhance the secure sharing and storage of electronic health data while ensuring the confidentiality, privacy, and integrity of medical records.

An Automated Bioinformatics Pipeline Informing Near-Real-Time Public Health Responses to New HIV Diagnoses in a Statewide HIV Epidemic.

Molecular HIV cluster data can guide public health responses towards ending the HIV epidemic. Currently, real-time data integration, analysis, and interpretation are challenging, leading to a delayed public health response. We present a comprehensive methodology for addressing these challenges through data integration, analysis, and reporting. We integrated heterogeneous data sources across systems and developed an open-source, automatic bioinformatics pipeline that provides molecular HIV cluster data to inform public health responses to new statewide HIV-1 diagnoses, overcoming data management, computational, and analytical challenges. We demonstrate implementation of this pipeline in a statewide HIV epidemic and use it to compare the impact of specific phylogenetic and distance-only methods and datasets on molecular HIV cluster analyses. The pipeline was applied to 18 monthly datasets generated between January 2020 and June 2022 in Rhode Island, USA, that provide statewide molecular HIV data to support routine public health case management by a multi-disciplinary team. The resulting cluster analyses and near-real-time reporting guided public health actions in 37 phylogenetically clustered cases out of 57 new HIV-1 diagnoses. Of the 37, only 21 (57%) clustered by distance-only methods. Through a unique academic-public health partnership, an automated open-source pipeline was developed and applied to prospective, routine analysis of statewide molecular HIV data in near-real-time. This collaboration informed public health actions to optimize disruption of HIV transmission.

Data Management Challenges and Resolutions in the Telecommunication Industry

This paper addresses data management threats, challenges, and measures for mitigation of such data threats in the telecommunication industry. The telecommunications industry handles tonnes of essential, personal and sensitive data on staff, clients and customers. This data is critical in developing product lines and marketing strategies for telecom companies, opening them up to opportunities that otherwise would have been beyond their reach. As a result, telecom companies’ databases are rendered vulnerable to violation and unauthorised access from insiders, third-party entities, as well as hackers. They have become premium target for cyberattacks, malware, interference, data theft and loss of vital data. While systems for the management of data in the telecom industry are crucial in gathering, storing, and analysing data, it is equally important for such systems to be efficient enough to provide a safe and secure mode of accessing, analysing and storing data assets. This is to prevent illegitimate use of sensitive data, cyberattacks, malware and data infringements on telecom data assets.

A framework for implementing machine learning in healthcare based on the concepts of preconditions and postconditions

Machine learning is a powerful tool that can be used to solve a wide range of problems in various applications and industries. The healthcare sector has faced specific challenges that have kept machine learning algorithms from becoming as widely and quickly adopted as in other industries. Data access and management challenges, ethical considerations, safety, and physician and patient perception present bigger barriers to implementation than model performance. In this paper, we propose adapting and customizing the concept of preconditions and postconditions from software engineering to develop a framework based on required clinical parameters and expected clinical output that will help bridge identified gaps in the implementation of machine learning tools in health care.

Barriers that prevent adults living with HBV infection from participating in clinical research: experience from South Africa

High profile international goals have been set for the elimination of hepatitis B virus (HBV) infection as a public health threat by the year 2030. Developing and expanding equitable, accessible translational HBV research programmes that represent real-world populations are therefore an urgent priority for clinical and academic communities. We present experiences and insights by an expert interdisciplinary group focusing on barriers that impede adults living with HBV infection from participating in clinical studies. Our viewpoint describes barriers we have identified through working in a variety of settings across South Africa, including lack of education and awareness, experiences of stigma and discrimination, challenges for governance and data management, and a burden of complex morbidity. Through identifying these challenges, we propose solutions and interventions, highlight new approaches, and provide a framework for future research.

An Expectation-Maximization Algorithm for Combining a Sample of Partially Overlapping Covariance Matrices.

The generation of unprecedented amounts of data brings new challenges in data management, but also an opportunity to accelerate the identification of processes of multiple science disciplines. One of these challenges is the harmonization of high-dimensional unbalanced and heterogeneous data. In this manuscript, we propose a statistical approach to combine incomplete and partially-overlapping pieces of covariance matrices that come from independent experiments. We assume that the data are a random sample of partial covariance matrices sampled from Wishart distributions and we derive an expectation-maximization algorithm for parameter estimation. We demonstrate the properties of our method by (i) using simulation studies and (ii) using empirical datasets. In general, being able to make inferences about the covariance of variables not observed in the same experiment is a valuable tool for data analysis since covariance estimation is an important step in many statistical applications, such as multivariate analysis, principal component analysis, factor analysis, and structural equation modeling.

Comparative Analysis of Ciphertext Enlargement on Generalization of the ElGamal and Multi-factor RSA

Information and communication security has become more crucial and has become a new problem in relation to security, accessibility, data management, and other information policy challenges as a result of how easy it is for all users to use communication media. One of the fields of science that has a technique or art for disguising the data sent by the sender to the recipient with the aim of maintaining the confidentiality of the data is called cryptography. In determining better cryptographic algorithms for data security systems, in addition to considering strength, key length and ciphertext enlargement are also important factors to consider. Therefore, in this study, we attempted to compare the ciphertext magnification of the generalization of the ElGamal and multi-factor RSA algorithms by utilizing the same key length. Generalization of the ElGamal and Multi-factor RSA are both asymmetric algorithms that have public and private key pairs for encryption and decryption. However, at the level of security, the RSA algorithm is based on the difficulty of finding large integer factors into two prime factors. In contrast to the ElGamal algorithm, security is based on the difficulty of calculating the discrete logarithm of a large prime modulus. The results of the comparison algorithm carried out are represented in the form of a table containing the plaintext, key length, and size of the data.