Complex Data Integration Research Articles

The Future of Data Warehousing: Trends, Technologies, and Challenges in the Era of Big Data, Cloud Computing, and Artificial Intelligence

This article explores the future of data warehousing, discussing emerging trends, technologies, and challenges shaping the landscape of data management and analytics in the era of big data, cloud computing, and artificial intelligence. It delves into topics such as the rise of cloud data warehouses, the increasing adoption of artificial intelligence and machine learning in data warehousing for advanced analytics and automation, the growing importance of data governance, privacy, and security in the face of evolving regulations, and the need for real-time data processing and analytics to support agile decision-making. The article also discusses the challenges faced by organizations in adopting and implementing these new technologies, such as the need for skilled personnel, the complexity of data integration from diverse sources, the cost of infrastructure and migration, and the ethical considerations surrounding the use of AI and ML in data warehousing. It concludes by highlighting the potential of data warehousing to drive innovation, improve operational efficiency, and gain a competitive advantage in the digital age.

Exploring the impact of big data on companies' business intelligence strategies in the digital era

Using a mixed-method approach, this research examines how big data affects business intelligence strategies in the digital age. It points out the importance of the big data analytic process in decision-making, strategic planning, and identifying growth opportunities. Nevertheless, incorporating big data into business intelligence strategies has challenges: complex data integration, privacy concerns, and the availability of skilled analysts. Nonetheless, its potential to drive innovation and growth is non-negotiable. The study findings contribute valuable insights to organizations looking to effectively navigate the embrace of big data and thrive in a data-rich digital age as more businesses embrace data-driven decision-making, the implications of this research extend beyond individual organizations This study adds to existing knowledge in the field by shedding light on the evolving landscape of big data in business science approaches that shape industry, both economic and societal. Organizations are recommended to take the practical advice of this survey to gain the full benefits from big data and unleash its transformative power. As the digital age evolves, businesses must embrace big data to remain competitive and flexible amid dynamic market forces.

Simona: a comprehensive R package for semantic similarity analysis on bio-ontologies

BackgroundBio-ontologies are keys in structuring complex biological information for effective data integration and knowledge representation. Semantic similarity analysis on bio-ontologies quantitatively assesses the degree of similarity between biological concepts based on the semantics encoded in ontologies. It plays an important role in structured and meaningful interpretations and integration of complex data from multiple biological domains.ResultsWe present simona, a novel R package for semantic similarity analysis on general bio-ontologies. Simona implements infrastructures for ontology analysis by offering efficient data structures, fast ontology traversal methods, and elegant visualizations. Moreover, it provides a robust toolbox supporting over 70 methods for semantic similarity analysis. With simona, we conducted a benchmark against current semantic similarity methods. The results demonstrate methods are clustered based on their mathematical methodologies, thus guiding researchers in the selection of appropriate methods. Additionally, we explored annotation-based versus topology-based methods, revealing that semantic similarities solely based on ontology topology can efficiently reveal semantic similarity structures, facilitating analysis on less-studied organisms and other ontologies.ConclusionsSimona offers a versatile interface and efficient implementation for processing, visualization, and semantic similarity analysis on bio-ontologies. We believe that simona will serve as a robust tool for uncovering relationships and enhancing the interoperability of biological knowledge systems.

Nonlinear elasticity with the Shifted Boundary Method

We propose a new unfitted/immersed computational framework for nonlinear solid mechanics, which bypasses the complexities associated with the generation of CAD representations and subsequent body-fitted meshing. This approach allows to speed up the cycle of design and analysis in complex geometry and requires relatively simple computer graphics representations of the surface geometries to be simulated, such as the Standard Tessellation Language (STL format). Complex data structures and integration on cut elements are avoided by means of an approximate boundary representation and a modification (shifting) of the boundary conditions to maintain optimal accuracy. An extensive set of computational experiments in two and three dimensions is included.

Big Data Analytics For Decision Support In Healthcare Information Systems Opportunities And Challenges

This research discusses the use of Big Data analytics to support decision making in Health Information Systems (GIS): associated opportunities and challenges. The aim of this research is to identify the opportunities offered by Big Data analytics in the context of GIS as well as identify the challenges that need to be overcome to effectively implement Big Data analytics in GIS. The research method used in this study involved a literature review of relevant scientific publications on Big Data analytics in CIS. The information collected from this literature review is then analyzed systematically to identify existing opportunities and challenges. The research results show that Big Data analytics has great potential to improve decision making in CIS. Opportunities identified include improving understanding of disease patterns, epidemic forecasting, developing predictive models for diagnosis and treatment, and improving real-time patient monitoring. However, this research also revealed several challenges that need to be overcome, including data privacy and security issues, complex data integration, and limited existing resources. Based on the results of this research, it is concluded that Big Data analytics has great potential to support decision making in CIS. However, successful implementation requires effective addressing of the associated challenges. In facing these challenges, collaboration between health stakeholders and Big Data analytics experts is essential.

Applications of Digital Twins in Reinforced and Prestressed Concrete Bridge Infrastructure

This study investigates into the innovative application of Digital Twins to monitor the structural health of reinforced and prestressed concrete bridges. Digital Twins, precise digital representations of real structures, emerge as an advanced solution, integrating data collected through drones and sensors to simulate and analyze structural behavior. This approach is decisive for anticipating pathological manifestations, optimizing maintenance, and preventing failures, holding significant implications for the future of structural engineering. The research investigates the application of Digital Twins in monitoring the structural health of concrete bridges, emphasizing their potential in early identification of pathologies, efficient maintenance, and prevention of structural failures. Two case studies exemplify the practical application of this technology, highlighting challenges such as complex data integration and associated costs. Digital Twins, capable of simulating structural behavior, have proven effective in monitoring the health of concrete bridges, addressing pathological manifestations, and optimizing maintenance. Despite technological challenges and associated costs, their application is necessary to ensure structural safety, reducing both human and economic impacts.

Overview of Apache NiFi

Apache NiFi is an open-source data integration tool designed to automate the flow of data between systems. It provides a user-friendly interface to design data flows and move data between different systems, making it easier to collect, distribute, and manage data in real-time. NiFi supports the concept of data flow, allowing users to define and visualize the movement of data from various sources to various destinations.At its core, Apache NiFi is a data integration tool that facilitates the automated flow of data between systems. It was developed by the National Security Agency (NSA) and later open-sourced as part of the Apache Software Foundation. NiFi is designed to be scalable, flexible, and easy to use, making it suitable for a variety of data integration scenarios.Providing a user-friendly interface for designing, managing, and monitoring data flows, NiFi facilitates the seamless movement of data across different environments. Utilizing a visual design paradigm, users can construct data flows by connecting a variety of processors that perform tasks such as data ingestion, transformation, and routing. NiFi's scalability, provenance tracking, and security features make it well-suited for handling complex data integration scenarios. With a robust community and ongoing development, Apache NiFi continues to play a crucial role in simplifying and optimizing data movement and processing workflows

NextGen Training for Medical First Responders: Advancing Mass-Casualty Incident Preparedness through Mixed Reality Technology

Mixed reality (MR) technology has the potential to enhance the disaster preparedness of medical first responders in mass-casualty incidents through new training methods. In this manuscript, we present an MR training solution based on requirements collected from experienced medical first responders and technical experts, regular end-user feedback received through the iterative design process used to develop a prototype and feedback from two initial field trials. We discuss key features essential for an effective MR training system, including flexible scenario design, added realism through patient simulator manikins and objective performance assessment. Current technological challenges such as the responsiveness of avatars and the complexity of smart scenario control are also addressed, along with the future potential for integrating artificial intelligence. Furthermore, an advanced analytics and statistics tool that incorporates complex data integration, machine learning for data analysis and visualization techniques for performance evaluation is presented.

Integrating Heterogeneous Stream and Historical Data Sources using SQL

Applications capable of integrating data from historical and streaming sources can make the most contextualized and enriched decision-making. However, the complexity of data integration over heterogeneous data sources can be a hard task for querying in this context. Approaches that facilitate data integration, abstracting details and formats of the primary sources can meet these needs. This work presents a framework that allows the integration of streaming and historical data in real-time, abstracting syntactic aspects of queries through the use of SQL as a standard language for querying heterogeneous sources. The framework was evaluated through an experiment using relational datasets and real data produced by sensors. The results point to the feasibility of the approach.

Combining Molecular, Imaging, and Clinical Data Analysis for Predicting Cancer Prognosis.

Simple SummaryThe rise of Big Data, the widespread use of Machine Learning, and the cheapening of omics techniques have allowed for the creation of more sophisticated and accurate models in biomedical research. This article presents the state-of-the-art predictive models of cancer prognosis that use multimodal data, considering clinical, molecular (omics and non-omics), and image data. The subject of study, the data modalities used, the data processing and modelling methods applied, the validation strategies involved, the integration strategies encompassed, and the evolution of prognostic predictive models are discussed. Finally, we discuss challenges and opportunities in this field of cancer research, with great potential impact on the clinical management of patients and, by extension, on the implementation of personalised and precision medicine.Cancer is one of the most detrimental diseases globally. Accordingly, the prognosis prediction of cancer patients has become a field of interest. In this review, we have gathered 43 state-of-the-art scientific papers published in the last 6 years that built cancer prognosis predictive models using multimodal data. We have defined the multimodality of data as four main types: clinical, anatomopathological, molecular, and medical imaging; and we have expanded on the information that each modality provides. The 43 studies were divided into three categories based on the modelling approach taken, and their characteristics were further discussed together with current issues and future trends. Research in this area has evolved from survival analysis through statistical modelling using mainly clinical and anatomopathological data to the prediction of cancer prognosis through a multi-faceted data-driven approach by the integration of complex, multimodal, and high-dimensional data containing multi-omics and medical imaging information and by applying Machine Learning and, more recently, Deep Learning techniques. This review concludes that cancer prognosis predictive multimodal models are capable of better stratifying patients, which can improve clinical management and contribute to the implementation of personalised medicine as well as provide new and valuable knowledge on cancer biology and its progression.

BioSamples database: FAIRer samples metadata to accelerate research data management.

The BioSamples database at EMBL-EBI is the central institutional repository for sample metadata storage and connection to EMBL-EBI archives and other resources. The technical improvements to our infrastructure described in our last update have enabled us to scale and accommodate an increasing number of communities, resulting in a higher number of submissions and more heterogeneous data. The BioSamples database now has a valuable set of features and processes to improve data quality in BioSamples, and in particular enriching metadata content and following FAIR principles. In this manuscript, we describe how BioSamples in 2021 handles requirements from our community of users through exemplar use cases: increased findability of samples and improved data management practices support the goals of the ReSOLUTE project, how the plant community benefits from being able to link genotypic to phenotypic information, and we highlight how cumulatively those improvements contribute to more complex multi-omics data integration supporting COVID-19 research. Finally, we present underlying technical features used as pillars throughout those use cases and how they are reused for expanded engagement with communities such as FAIRplus and the Global Alliance for Genomics and Health. Availability: The BioSamples database is freely available at http://www.ebi.ac.uk/biosamples. Content is distributed under the EMBL-EBI Terms of Use available at https://www.ebi.ac.uk/about/terms-of-use. The BioSamples code is available at https://github.com/EBIBioSamples/biosamples-v4 and distributed under the Apache 2.0 license.

FIRED (Fire Events Delineation): An Open, Flexible Algorithm and Database of US Fire Events Derived from the MODIS Burned Area Product (2001–2019)

Harnessing the fire data revolution, i.e., the abundance of information from satellites, government records, social media, and human health sources, now requires complex and challenging data integration approaches. Defining fire events is key to that effort. In order to understand the spatial and temporal characteristics of fire, or the classic fire regime concept, we need to critically define fire events from remote sensing data. Events, fundamentally a geographic concept with delineated spatial and temporal boundaries around a specific phenomenon that is homogenous in some property, are key to understanding fire regimes and more importantly how they are changing. Here, we describe Fire Events Delineation (FIRED), an event-delineation algorithm, that has been used to derive fire events (N = 51,871) from the MODIS MCD64 burned area product for the coterminous US (CONUS) from January 2001 to May 2019. The optimized spatial and temporal parameters to cluster burned area pixels into events were an 11-day window and a 5-pixel (2315 m) distance, when optimized against 13,741 wildfire perimeters in the CONUS from the Monitoring Trends in Burn Severity record. The linear relationship between the size of individual FIRED and Monitoring Trends in Burn Severity (MTBS) events for the CONUS was strong (R2 = 0.92 for all events). Importantly, this algorithm is open-source and flexible, allowing the end user to modify the spatio-temporal threshold or even the underlying algorithm approach as they see fit. We expect the optimized criteria to vary across regions, based on regional distributions of fire event size and rate of spread. We describe the derived metrics provided in a new national database and how they can be used to better understand US fire regimes. The open, flexible FIRED algorithm could be utilized to derive events in any satellite product. We hope that this open science effort will help catalyze a community-driven, data-integration effort (termed OneFire) to build a more complete picture of fire.

Nursing Practice in the Digital Age

An ongoing influx of technological breakthroughs and innovations will cause nurses to practice in an ever-changing landscape. These technological resources and societal changes will necessitate that nurses reexamine and redesign their practice. Nurses can embrace the impending changes and leverage technology to automate the routine activities of their practice and to support the complex data aggregation and integration necessary to understand their patients and communities. In that way, nurses can focus more of their cognitive expertise on the coordination, integration, and facilitation of care to promote healing and improved wellness of their patients and communities.

An approach for semantic integration of heterogeneous data sources.

Integrating data from multiple heterogeneous data sources entails dealing with data distributed among heterogeneous information sources, which can be structured, semi-structured or unstructured, and providing the user with a unified view of these data. Thus, in general, gathering information is challenging, and one of the main reasons is that data sources are designed to support specific applications. Very often their structure is unknown to the large part of users. Moreover, the stored data is often redundant, mixed with information only needed to support enterprise processes, and incomplete with respect to the business domain. Collecting, integrating, reconciling and efficiently extracting information from heterogeneous and autonomous data sources is regarded as a major challenge. In this paper, we present an approach for the semantic integration of heterogeneous data sources, DIF (Data Integration Framework), and a software prototype to support all aspects of a complex data integration process. The proposed approach is an ontology-based generalization of both Global-as-View and Local-as-View approaches. In particular, to overcome problems due to semantic heterogeneity and to support interoperability with external systems, ontologies are used as a conceptual schema to represent both data sources to be integrated and the global view.

Entwicklung eines Frühwarnsystems für Rutschungen in den tropischen Anden (Medellín, Kolumbien)

AbstractThe Inform@Risk project aims to develop a cost‐effective but sufficiently accurate, easy‐to‐maintain early warning system (EWS) for informal settlements on the margins of large cities, adapted to tropical climatic conditions in South American mountain regions. This EWS will be implemented on the outskirts of the Bello Oriente district in Medellín. The area is characterized by an elevation of around 2,000 m, a medium slope inclination of 20° to 30° and deeply weathered crystalline rocks, which are particularly sensitive to shallow landslides during heavy precipitation. In addition to the development of a geo‐sensor network with complex data integration and real‐time evaluation, the main focus of the project is the social integration of the EWS both with the municipal authorities, which are going to take over the system after completion of the three‐year project, and with the population living in the settlement. This report describes the project and the specifications of the EWS and presents first results from the field work.

Engineering complex data integration, harmonization and visualization systems

Abstract Complex data transformation, aggregation and visualization problems are becoming increasingly common. These are needed in order to support improved business intelligence and end-user access to data. However, most such applications present very challenging software engineering problems including noisy data, diverse data formats and APIs, challenging data modeling and increasing demand for sophisticated visualization support. This paper describes a data integration, harmonization and visualization process and framework that we have been developing. We discuss our approach used to tackle complex data aggregation and harmonization problems and we demonstrate a set of information visualizations that can be developed from the harmonized data to make it usable for its target audience. We use a case study of Household Travel Survey data mapping, harmonization, aggregation and visualization to illustrate our approach. We summarize a set of lessons that we have learned from this industry-based software engineering experience. We hope these will be useful for others embarking on challenging data harmonization and integration problems. We also identify several key directions and needs for future research and practical support in this area.

The road map of cancer precision medicine with the innovation of advanced cancer detection technology and personalized immunotherapy.

The advancement of cancer genomics research due to the development of next generation sequencing technologies is going to bring the promise of cancer precision medicine, in turn revolutionizing cancer detection and treatment. In this review, we will discuss the possible road map for implementation of cancer precision medicine into the clinical practice by mainly focusing on the role of liquid biopsy, particularly circulating tumor DNA, as a potential tool for cancer screening, selection of an appropriate drug(s), surveillance of minimal residual diseases, and early detection of recurrence. We will also review the current status of genome-driven oncology and emerging field of immunotherapies that could be provided to patients to improve their clinical outcome and quality of life. Lastly, we will discuss the usefulness of artificial intelligence that facilitate complex data integration in our health care/medical care system.

Integration of macromolecular complex data into the Saccharomyces Genome Database.

Proteins seldom function individually. Instead, they interact with other proteins or nucleic acids to form stable macromolecular complexes that play key roles in important cellular processes and pathways. One of the goals of Saccharomyces Genome Database (SGD; www.yeastgenome.org) is to provide a complete picture of budding yeast biological processes. To this end, we have collaborated with the Molecular Interactions team that provides the Complex Portal database at EMBL-EBI to manually curate the complete yeast complexome. These data, from a total of 589 complexes, were previously available only in SGD’s YeastMine data warehouse (yeastmine.yeastgenome.org) and the Complex Portal (www.ebi.ac.uk/complexportal). We have now incorporated these macromolecular complex data into the SGD core database and designed complex-specific reports to make these data easily available to researchers. These web pages contain referenced summaries focused on the composition and function of individual complexes. In addition, detailed information about how subunits interact within the complex, their stoichiometry and the physical structure are displayed when such information is available. Finally, we generate network diagrams displaying subunits and Gene Ontology annotations that are shared between complexes. Information on macromolecular complexes will continue to be updated in collaboration with the Complex Portal team and curated as more data become available.

Expert Knowledge Content Database Design Suggestions from Validation Aspect

Expert Knowledge Content Database (EKCD) contains small or medium amount of data but its data content is valuable, which can be determined by expert work. Expert work can be a complex measurement, estimation, or data integration. Database design and validation was analyzed in a case study with EKCD Eulakes – Phytoplankton database of Lake Stechlin. Different data quality metrics was calculated and evaluated, and the content based inconsistency was evaluated. In the conclusion a schema is given for helping the design and validation.

Rise of Deep Learning for Genomic, Proteomic, and Metabolomic Data Integration in Precision Medicine.

Machine learning (ML) is being ubiquitously incorporated into everyday products such as Internet search, email spam filters, product recommendations, image classification, and speech recognition. New approaches for highly integrated manufacturing and automation such as the Industry 4.0 and the Internet of things are also converging with ML methodologies. Many approaches incorporate complex artificial neural network architectures and are collectively referred to as deep learning (DL) applications. These methods have been shown capable of representing and learning predictable relationships in many diverse forms of data and hold promise for transforming the future of omics research and applications in precision medicine. Omics and electronic health record data pose considerable challenges for DL. This is due to many factors such as low signal to noise, analytical variance, and complex data integration requirements. However, DL models have already been shown capable of both improving the ease of data encoding and predictive model performance over alternative approaches. It may not be surprising that concepts encountered in DL share similarities with those observed in biological message relay systems such as gene, protein, and metabolite networks. This expert review examines the challenges and opportunities for DL at a systems and biological scale for a precision medicine readership.