Data Ecosystem Research Articles

A Framework for the Multi-Dimensional Assessment of Interoperability for Open Data Ecosystems Development

The generation of social and economic value from the vast quantities of data opened by governments has been much lower than the initial expectations. It requires the development and cooperation of ecosystems composed of various public, private, and non-profit actors. For the development of these ecosystems in order to generate social and economic value from open government data (OGD), necessary conditions are, on the one hand, the availability of a critical mass of OGD and, on the other hand, the existence of some level of homogeneity and interoperability among the numerous portals and websites providing them, which enables their combined exploitation that increases the potential for value generation from them. This paper focuses on the latter and develops the Technical, Semantic, Legal, and Organizational (TSLO) Interoperability Framework, a conceptual tool designed to provide a structured multi-dimensional assessment of the interoperability capabilities of OGD infrastructures, based on the ‘European Interoperability Framework’ (EIF) as well as relevant OGD literature. This TSLO framework has been applied for the assessment of the above main dimensions of the interoperability of the OGD infrastructures of the Greek local government administration institutions of all three layers (decentralized, regional, and municipal). After this application, nine interviews were conducted with nine municipalities that do not provide OGD in order to gain a deeper understanding of the reasons for this. The results of this first application of TSLO revealed not only the low participation of Greek local government institutions, and especially of the municipalities, in the OGD movement (with only 8 out of the 332 municipalities providing open data on their portals or websites), but also the inconsistent implementation of this, especially with respect to the use of semantic technologies as well as the legal frameworks under which the open data are provided, resulting in low levels of interoperability among them. The interviews that were conducted revealed that the opening of some of their data was not a priority for them, as their limited resources had to be used for the development of internal information systems for increasing their efficiency, as well as e-services for the citizens, enabling them to conduct transactions electronically (through the Internet). Our study makes a contribution to the still limited body of research on OGD ecosystems, examining an important precondition for their development that has not been researched in previous literature; also, our framework can contribute to the increase of interoperability of OGD infrastructures and therefore to the increase of social and economic value generations for the OGD they provide.

Towards sustainable public and open data ecosystems: An introduction to a special section

Towards sustainable public and open data ecosystems: An introduction to a special section

Data Modeling Essentials: Techniques, Best Practices, and Future Trends

Data modeling, in essence, is based on the process of data structuring and organization to effectively handle, analyze, and make informed decisions across organizations. This paper reviews the major concepts and methods significant for modern data architecture in "Data Modeling Essentials: Techniques, Best Practices, and Future Trends.". We start by considering the base techniques, namely Entity-Relationship and Dimensional modeling, and look into their applications in operational and analytical data systems. The paper also covers more advanced topics that include Data Vault modeling for agile data warehousing and NoSQL modeling to accommodate unstructured and semi-structured data, increasingly important in today's big data era. The paper also gives an overview of best practices in developing scalable, flexible data models: stakeholder communication, documentation, and the use of automation tools in accelerating model development and consistency. Data quality, adherence to regulations, and alignment of models with business objectives are among the practices identified as crucial in today's fast-moving data environments. New trends are taking precedence, and data modeling is continuously evolving. We will review the rise of AI- assisted data modeling, graph database structures for highly interconnected data, and model-driven development to accelerate system deployments. This paper examines actionable insights for data professionals, IT teams, and business leaders on how to design robust, agile models that improve the quality of data, enhance decision-making, and foster innovation in an ever-changing data ecosystem. Keywords Data Modeling, Entity-Relationship (ER) Modeling, Dimensional Modeling, Data Quality, Scalable Data Models, Model Driven Development (MDD), Data Management, Structured & Unstructured Data, Future Trends in Data Modeling.

Leveraging Big Data through NoSQL Databases-Architecture Methodologies, and Strategic Applications

The rise of Big Data has necessitated the evolution of databases beyond traditional relational models, paving the way for NoSQL databases to manage unstructured and semi-structured data at scale. This paper explores the architecture, methodologies, and strategic applications of NoSQL databases in the Big Data ecosystem. We will examine key challenges, tools, and technologies supporting NoSQL adoption and discuss real-world use cases across industries. Additionally, we will outline general activation steps, dependencies, and the risks associated with deploying NoSQL databases in enterprise settings.

Building a FAIR data ecosystem for incorporating single-cell transcriptomics data into agricultural genome to phenome research

Building a FAIR data ecosystem for incorporating single-cell transcriptomics data into agricultural genome to phenome research

Big Data and Blockchain Integration Enhancing Data Management and Security

Big Data and blockchain are two of the most transformative forces shaping the faces of industries around the world in this digital age. Where Big Data allows organizations to exploit vast pools of information to make informed decisions, blockchain gives one a decentralized and highly secure way of handling and sharing data. Based on the integration of the two technologies, several of the existing challenges in data security and transparency together with its management could be resolved. This paper looks at the advantages, technical details, practical applications, and challenges when big data meets blockchain and through this synergy, a more efficient, secure, and transparent data ecosystem can be developed.

The Intelligent Data Ecosystem: Uniting AI and Data Integration to Revolutionize Data Science

The Rise of Big Data: Today, the scale and domain of massive data management has prolonged to an unmatched state. In this paper, we will examine the process of data science developed over the recent past and argue that artificial intelligence (AI) and data integration converge to create intelligent data ecosystem reaching new heights. This paper combines AI's strength to automate analytics and data integration, which is able to harmonize different data sources and shows how businesses can benefit from a unified approach for innovation, operational efficiency, decision-making. We also cover some of the key use cases in the industry, and ways these technologies are gaining momentum and reshaping the future of data science.

Designing for Researcher Access in the U.S. Mortality Data Ecosystem

Access to mortality data is critical for researchers for various reasons, including the development of life-saving medical interventions. However, high-quality mortality data is typically unavailable for researchers to access in a timely manner due to bottlenecks and inefficiencies in the coordination of managing mortality data across a vast information ecosystem. To identify opportunities to improve access to mortality data, we conducted a qualitative interview study with 20 experts with direct knowledge and experience with mortality data systems. From these interviews, we mapped the mortality data ecosystem and found that policy constraints are the main underlying cause of bottlenecks and inefficiencies. We argue that policy intervention is required to optimize the coordination of mortality data between databases and between organizations. To support researcher access to mortality data, we present guiding principles for designers seeking to improve the mortality data ecosystem. These principles contribute to CSCW scholarship focused on the challenges of coordination across large information ecosystems and the tradeoffs between technology and policy when designing such systems.

Pioneering Museum Data Transformation: Unifying Legacy Systems and Enhancing Data Integrity

In the digital era, museums confront the challenge of modernising legacy data systems to align with current standards. Part of the RECODE (Rethinking Collections Data Ecosystems) Programme (Dupont et al. 2022) examines the complex transition from disparate departmental object models to a unified system, reflecting the broader museum community's struggle with data standardisation. The case study centres on consolidating five distinct data models accumulated over two decades, each tailored to different departmental needs under various constraints. The data mapping process identifies and aligns similar fields across models, facilitating the integration of analogous data points and exposing redundancies and unique practices embedded over years. Major challenges include data complexity, diversity, and quality issues (cleaning, standardizing, and deduplicating), compounded by the massive scale of the data: currently, the Natural History Museum London (NHM) has 262,831,590 records across 8,175 fields, and a total volume of 12 TB of data, which define both the ongoing data modelling work and our future data migration. A key innovation in our process is the immediate visibility of data issues that became apparent upon migrating to Amazon Web Services (AWS). AWS serves as the staging environment for our transition to the new NHM Collections Management System (CMS) and offers an unprecedented platform for directly addressing these challenges. It enables us to query all our data—across all departments and fields—in just seconds. This capability provides a comprehensive view that was previously unattainable. The core of this presentation is sharing "lessons learned" from navigating the intricacies of an ongoing CMS transition within a museum. The endeavor to untangle and unify diverse data models is a common challenge (IEEE Big Data Governance and Metadata Management Industry Connections Activity 2020,Wu et al. 2022, Wu et al. 2021,Little et al. 2022,Woodburn et al. 2022) highlighting the importance of community engagement and knowledge exchange. Our findings underscore the necessity of cross-departmental collaboration and the benefits of a data-driven data modelling approach. By sharing our journey and the developed comprehensive object model (Collier and Woodburn 2022), we aim to contribute valuable insights to museums undergoing similar transformations. The RECODE Programme sheds light on the practical aspects of CMS modernisation through the analytics and knowledge derived from over two decades of collections data and data use.

Data Organization Patterns in Data Mesh: Optimizing Data Management for the Modern Enterprise

This article explores the implementation of Data Mesh principles through a structured layered approach to data organization, addressing the challenges of managing large-scale, diverse data ecosystems in modern enterprises. As global data creation surges towards 181 zettabytes by 2025, traditional centralized data architectures are proving inadequate. We propose a four-layer methodology—Raw, Work, Gold, and Publish—that aligns with Data Mesh philosophy while providing clear delineation of data processing stages. Through case studies across various industries, we demonstrate how this approach leads to significant improvements in data quality, time-to-insight, and incident resolution. A large financial institution achieved a 40% reduction in time-to-insight, while a global manufacturer reported a 30% improvement in data quality. The article also discusses the challenges of implementation, including cultural shifts and skill gaps. Our findings suggest that combining Data Mesh principles with a layered organizational structure can transform enterprise data landscapes, enhancing agility, cross-functional collaboration, and alignment of data assets with business objectives. This article contributes to the evolving field of decentralized data architectures and provides practical insights for organizations seeking to optimize their data management strategies in the face of exponential data growth.

Remotely Sensed Variables Predict Grassland Diversity Better at Scales Below 1,000 km as Opposed to Abiotic Variables That Predict It Better at Larger Scales

AbstractGlobal spatial patterns of vascular plant diversity have been mapped at coarse grain based on climate‐dominated environment–diversity relationships and, where possible, at finer grain using remote sensing. However, for grasslands with their small plant sizes, the limited availability of vegetation plot data has caused large uncertainties in fine‐grained mapping of species diversity. Here we used vegetation survey data from 1,609 field sites (>4,000 plots of 1 m2), remotely sensed data (ecosystem productivity and phenology, habitat heterogeneity, functional traits and spectral diversity), and abiotic data (water‐ and energy‐related, characterizing climate‐dominated environment) together with machine learning and spatial autoregressive models to predict and map grassland species richness per 100 m2 across the Mongolian Plateau at 500 m resolution. Combining all variables yielded a predictive accuracy of 69% compared with 64% using remotely sensed variables or 65% using abiotic variables alone. Among remotely sensed variables, functional traits showed the highest predictive power (55%) in species richness estimation, followed by productivity and phenology (48%), spectral diversity (48%) and habitat heterogeneity (48%). When considering spatial autocorrelation, remotely sensed variables explained 52% and abiotic variables explained 41%. Moreover, Remotely sensed variables provided better prediction at smaller grain size (<∼1,000 km), while water‐ and energy‐dominated macro‐environment variables were the most important drivers and dominated the effects of remotely sensed variables on diversity patterns at macro‐scale (>∼1,000 km). These findings indicate that while remotely sensed vegetation characteristics and climate‐dominated macro‐environment provide similar predictions for mapping grassland plant species richness, they offer complementary explanations across broad spatial scales.

Developing a Competitive Edge: Building an Effective Portfolio in Snowflake and Teradata Data Engineering

This article examines the critical components of an effective portfolio for data engineering professionals specializing in Snowflake and Teradata platforms. As the data landscape evolves, the ability to showcase practical skills alongside theoretical knowledge has become paramount for career advancement. Through an analysis of industry trends and expert interviews, we identify key elements that contribute to a compelling portfolio, including project showcases demonstrating complex problem-solving, relevant certifications, and evidence of continuous learning. The article highlights the importance of balancing technical proficiency in SQL, data warehousing, and cloud technologies with strong documentation and communication skills. Our findings suggest that a well-curated portfolio not only validates a candidate's expertise but also provides tangible evidence of their ability to navigate the challenges of modern data ecosystems. This article provides a framework for data engineers to construct portfolios that effectively demonstrate their capabilities, potentially increasing their competitiveness in the job market and facilitating career growth in the rapidly evolving field of data engineering.

Usage of the National Cancer Institute Cancer Research Data Commons by Researchers: A Scoping Review of the Literature.

Over the past decade, significant surges in cancer data of all types have happened. To promote sharing and use of these rich data, the National Cancer Institute's Cancer Research Data Commons (CRDC) was developed as a cloud-based infrastructure that provides a large, comprehensive, and expanding collection of cancer data with tools for analysis. We conducted this scoping review of articles to provide an overview of how CRDC resources are being used by cancer researchers. A thorough literature search was conducted to identify all relevant publications. We included publications that directly cited CRDC resources to specifically examine the impact and contributions of CRDC by itself. We summarized the distributions and trends of how CRDC components were used by the research community and discussed current research gaps and future opportunities. In terms of CRDC resources used by the research community, encouraging trends in utilization were observed, suggesting that CRDC has become an important building block for fostering a wide range of cancer research. We also noted a few areas where current applications are rather lacking and provided insights on how improvements can be made by CRDC and research community. CRDC, as the foundation of a National Cancer Data Ecosystem, will continue empowering the research community to effectively leverage cancer-related data, uncover novel strategies, and address the needs of patients with cancer, ultimately combatting this disease more effectively.

The Privacy-Centric Data Pipeline : Strategies for Implementing Robust Security Measures in Data Engineering

This article presents a comprehensive framework for addressing data privacy and security challenges in modern data engineering environments. As organizations grapple with ever-increasing volumes of sensitive data and an evolving regulatory landscape, data engineers face complex challenges in safeguarding information throughout its lifecycle. We explore key components of a robust data protection strategy, including regulatory compliance, technical safeguards, and organizational practices. Our analysis covers critical areas such as encryption methodologies, access control mechanisms, data masking techniques, and lifecycle management policies. We also examine the role of data governance frameworks and the importance of continuous employee training in maintaining a secure data ecosystem. By synthesizing current best practices and emerging trends, this article provides data engineers and organizations with actionable insights for implementing effective privacy and security measures. Our findings underscore the need for a holistic approach that integrates technical solutions with strong governance practices to ensure data integrity, confidentiality, and compliance in an increasingly data-driven world.

SSBD: an ecosystem for enhanced sharing and reuse of bioimaging data.

SSBD (https://ssbd.riken.jp) is a platform for the sharing and reuse of bioimaging data. As part of efforts to build a bioimaging data ecosystem, SSBD has recently been updated to a two-tiered data resource comprising SSBD:repository, a public repository for the sharing of all types of bioimaging data reported in journals, and SSBD:database, an added-value database for the sharing of curated, highly reusable, metadata-rich data. This update addresses the conflicting demands of rapid data publication and sharing of richly annotated data, thereby promoting bioimaging data sharing and reuse. With this update, SSBD is now positioned as a core repository and database within the foundingGIDE, an international consortium working to establish a global image data ecosystem. Harmonizing metadata between data resources enables cross-searching and data exchange with data resources from other countries and regions.

NoSQL Technologies in Big Data Ecosystems: a Comprehensive Review of Architectural Paradigms and Performance Metrics

The exponential growth of data in volume, velocity, and variety has challenged traditional relational database management systems, leading to the emergence and widespread adoption of NoSQL and other innovative database technologies. This article provides a comprehensive review of NoSQL databases and their role in modern Big Data infrastructure stacks. Through a systematic analysis of current literature and industry case studies, we explore the architectural paradigms, performance metrics, and use cases of various NoSQL database types, including document-based, key-value, column-family, and graph databases. Our findings indicate that NoSQL solutions offer significant advantages in scalability, flexibility, and real-time processing capabilities, particularly for unstructured and semi-structured data. However, challenges persist in areas such as data consistency, security, and interoperability with existing systems. We also examine emerging trends, including NewSQL, time-series databases, and the integration of artificial intelligence in database management. This article contributes to the understanding of how organizations can leverage NoSQL technologies to optimize their Big Data infrastructure, highlighting both the opportunities and considerations for implementation. Our conclusions underscore the transformative impact of NoSQL on data management practices and provide directions for future research in this rapidly evolving field.

Mindat.org – the open access mineralogy database to accelerate data-intensive geoscience research

Abstract The mindat.org website (Mindat) has been operating since October 2000 as a free, crowd-sourced, and expert-curated database particularly focused on mineral species and their occurrences worldwide. The project has transformed from a hobbyist site in the beginning into a resource that has found use in various scientific research projects and educational programs. Together with other open data resources, Mindat has helped accelerate scientific discoveries in many fields, such as mineral evolution, mineral ecology, and the co-evolution of the geosphere and biosphere. Recently, through open data efforts, machine interfaces and software packages have been established to enable flexible data discovery and download from Mindat. We assume the data access and usage will further scale up in the next years. Although Mindat is curated by a team of geoscience and database experts across the world, the crowd-sourced records in Mindat still pose some bias. In this paper we first present an overview of the primary data subjects in Mindat, and then we give extensive details about the characteristics and bias of three most popular data subjects: locality, mineral species, and mineral occurrence. In the discussion we also give an outlook on appropriate data usage and future extension of data records. We hope users can obtain a more comprehensive view of the Mindat database through this paper and thus better plan their data use. We also hope more people will be inspired to contribute to the data curation work to make Mindat a sustained data ecosystem for geoscience research.

Synthetic Data, Common Data Models and Federation: Holy Trinity or unholy mess?

The healthcare sector's adoption of data and digital technologies is hindered by stringent data privacy regulations. Synthetic data, common data models (CDMs) and federated data ecosystems present promising solutions to these challenges. This workshop explored the potential of synthetic data to revolutionise health research, education and innovation (REI). By discussing applications, benefits, and ethical implications, the workshop addressed the role of synthetic data in enhancing data interoperability, scalability, and privacy preservation within healthcare REI systems. It also examined the relationship between synthetic data and CDMs, such as the Observational Medical Outcomes Partnership (OMOP) CDM, and the potential of federated data ecosystems to enable collaborative analysis while maintaining data ownership and control. The workshop delved into specific use cases to assess the feasibility, benefits, and limitations of synthetic data in various healthcare contexts. Through case studies and discussions, participants explored and developed best practices for effective utilisation. Ultimately, the workshop aimed to explore whether the combination of synthetic data, CDMs, and federation could facilitate privacy-preserving data sharing, seamless data integration, and scalable analysis across distributed data sources. By addressing privacy, interoperability, and scalability challenges, these approaches can drive innovation and insights within the healthcare sector.

Data ecosystem using smart home technology to support healthy aging

Abstract Background Smart home technology has been widely used to mitigate the challenges that population aging brings. However, the lack of a centralized data ecosystem has prevented these technologies from realizing their full potential. This project aimed to construct a data ecosystem compatible with smart home sensors to monitor healthcare for seniors, improving their quality of life. Methods The infrastructure was constructed using a user-centered design approach, allowing smart devices such as motion, temperature sensors to operate independently and communicate through a Wi-Fi mesh network. API technology provides a reliable channel for real-time data collection. To ensure data privacy and security, extracted data is hosted in a cloud environment that provides comprehensive data protection, and cloud computing is utilized for high-performance computation tasks. Analytical models such as Machine Learning can be implemented to recognize indoor activities, achieving the goal of remote monitoring. Results This data ecosystem advances in privacy and security, interoperability, accessibility, and data-sharing. Unified authentication mechanism ensures data is transmitted among heterogeneous devices with minimal vulnerability. Additionally, the common communication protocol provides the foundation of interoperability between devices. With unobtrusive devices, inhabitants are not burdened with operation, ensuring their accessibility and improving their user experience. Lastly, the data is exchanged in a standardized JSON data format, enabling efficient data-sharing board data integration. Conclusions This project developed a data ecosystem based on existing smart home environments. The ecosystem is compatible with smart devices that families already have, effectively reducing additional healthcare costs. Implications of this study include helping health policy and decision makers to adopt more smart home ecosystems to keep older people healthy at home. Key messages • Study constructed a data ecosystem compatible with smart home sensors, achieving remote monitoring of indoor activities and improving the quality of life for seniors. • The developed ecosystem offers promotion in privacy and security, interoperability, accessibility, and efficient data-sharing, reducing healthcare costs.

Co-creation of the Digital Democracy and Data Commons Manifesto: alternative sociotechnical visions of data

Amid public concern surrounding the proprietary and exploitative use of personal data by corporations and public institutions, and its consequences from a sociotechnical perspective, narratives around digital commons have recently emerged, framing potential alternatives. This paper presents the co-creation of the Digital Democracy and Data Commons Manifesto through a collaborative writing sprint, drawing on principles of openness, diversity, and inclusivity. The manifesto articulates a technopolitical vision for data governance that prioritizes community control over data. We analyze the manifesto's evolution throughout the process, demonstrating its capacity to address contemporary concerns such as data extractivism and algorithmic governance. Our approach is based on participatory design methods, more concretely on a collaborative writing sprint, to co-create a manifesto on alternatives to current datafication, digital inequalities, and lack of citizen control over personal data. On the one hand, we describe the process of implementing a sprint approach for collaboratively writing a topic-specific manifesto, in the context of the broader EU project DECODE (Decentralised Citizen Owned Data Ecosystems). On the other hand, we present and analyse the main results from the content structure of the manifesto over its initial and final versions, which moved progressively as a cohesive text away from a scholarly and policy-oriented tone.