Data Integration Solutions Research Articles

Rank Aggregation Methods and Tools in Genomic Data Analysis

Rank aggregation (RA) is the process of consolidating disparate rankings into a single unified ranking. It holds immense potential in the field of genomics. RA has applications in diverse research areas, such as gene expression analysis, meta-analysis, gene prioritization, and biomarker discovery. However, there are many challenges in the application of the RA approach to biological data, such as dealing with heterogeneous data sources, rankings of mixed quality, and evaluating the consolidated rankings. In this review, we present an overview of the existing RA methods with an emphasis on those that have been tailored to the complexities of genomics research. These encompass a broad range of approaches, from distributional and heuristic methods to Bayesian and stochastic optimization algorithms. By examining these techniques, we aim to equip researchers with the background knowledge needed to navigate the intricacies of RA in genomics data integration effectively. We review the practical applications to highlight the relevance and impact of RA methods in advancing genomics research. As the field continues to evolve, we identify open problems and suggest future directions to enhance the effectiveness of rank aggregation in genomics, by addressing the challenges related to data heterogeneity, single-cell omics and spatial transcriptomics data, and the development of clear and consistent evaluation methods. In summary, RA stands as a powerful tool in genomics research, which can offer deeper insights and more comprehensive data integration solutions.

Advancing Healthcare Through Interoperability: Implementing Scalable Solutions for Patient Data Integration.

Healthcare faces significant challenges in exchanging and utilizing health information across diverse providers, necessitating innovative solutions for improved interoperability. This study presents a comprehensive exploration of scalable technical and semantic solutions for patient care integration, emphasizing the implementation of these solutions within the framework of the Fast Healthcare Interoperability Resources (FHIR) standard. Our approach revolves around the development and deployment of Technical Interoperability Suite (TIS) and Semantic Interoperability Suite (SIS) technology solutions to disparate health information systems, predominantly Electronic Health Records (EHRs) into a unified Patient Care Platform, fostering comprehensive data exchange and utilization. The integration process involves importing data from various EHR systems and transforming imported patient data into FHIR-standardized formats. The provided solution supports various functionalities, including automatic and manual importation of patient data, through standard computer-readable templates. The integration of TIS and SIS solutions is underpinned by a robust technological framework, incorporating technologies such as Typescript, Deno, and document-oriented databases such as MongoDB. The effectiveness of our interoperability solutions was validated through deployment in multinational EU projects: ADLIFE and CAREPATH. The scalability and generalizability of our approach underscore its potential for diverse healthcare settings.

Linked Data Generation Methodology and the Geospatial Cross-Sectional Buildings Energy Benchmarking Use Case

Cross-sectional energy benchmarking in the building domain has become crucial for policymakers, energy managers and property owners as they can compare an immovable property performance against its closest peers. For this, Key Performance Indicators (KPIs) are formulated, often relying on multiple and heterogeneous data sources which, combined, can be used to set benchmarks following normalization criteria. Geographically delimited parameters are important among these criteria because they enclose entities sharing key common characteristics the geometrical boundaries represent. Linking georeferenced heterogeneous data is not trivial, for it requires geographical aggregation, which is often taken for granted or hidden within a pre-processing activity in most energy benchmarking studies. In this article, a novel approach for Linked Data (LD) generation is presented as a methodological solution for data integration together with its application in the energy benchmarking use case. The methodology consists of eight phases that follow the best principles and recommend standards including the well-known GeoSPARQL Open Geospatial Consortium (OGC) for leveraging the geographical aggregation. Its feasibility is demonstrated by the integrated exploitation of INSPIRE-formatted cadastral data and the Buildings Performance Certifications (BPCs) available for the Catalonia region in Spain. The outcomes of this research support the adoption of the proposed methodology and provide the means for generating cross-sectional building energy benchmarking histograms from any-scale geographical aggregations on the fly.

Integration of product development data for further ontological utilization

AbstractThe amount of data within the product development process requires a structured approach to coordinate them. Knowledge management solutions, such as ontologies, are a suitable way of linking data and representing semantic relationships. However, making all relevant data usable to ensure their target-oriented application is still a challenge. Thus, this contribution presents an approach to identify and classify heterogeneous data in product development. Besides this single ontology approach, interface solutions for data integration into an ontology are proposed.

Automating External Flat File Integration in SAP BW/4HANA: A Cost-Effective ABAP Approach

With SAP BW/4HANA as their enterprise data warehouse, organizations face challenges loading external data like CSV files into the system efficiently. Manual methods using GUIs are slow, error-prone, and lack scalability. This paper presents an automation solution using custom ABAP programming and standard BW/4HANA capabilities like process chains and advanced datastore objects (ADSOs) to orchestrate and load files. An ABAP program handles file discovery, logging, triggering process chains, monitoring status, and error handling. Benefits seen in production deployments include reduced load times and manual efforts, and improved data quality. While requiring upfront development effort, this approach provides a flexible, cost-effective data integration solution leveraging SAP's native tools. Key technical aspects include modular program design, use of SAP standard APIs, logging/status tracking, and BW process chain integration. Limitations are scalability for large volumes and complex transformations. The solution offers organizations an automation template for their BW/4HANA loading needs using in-house resources.

Developing a goal-driven data integration framework for effective data analytics

Data integration plays a crucial role in business intelligence, aiding decision-makers by consolidating data from heterogeneous sources to provide deep insights into business operations and performance. In the big data era, automated data integration solutions need to process high volumes of disparate data robustly and seamlessly for various analytical needs or operational actions. Existing data integration solutions exhibit limited capabilities for capturing and modeling users' needs to execute on-demand data integration. This study, underpinned by affordance theory and the goal definition principles from the Goal-Question-Metric approach, designs and instantiates a goal-driven data integration framework for data analytics. The proposed innovative design automates data integration for non-technical data users. Specifically, it demonstrates how to elicit and ontologize users' data-analytic goals and addresses semantic heterogeneity, thereby recognizing goal-relevant datasets. In a structured evaluation using the context of counter-terrorism analytics, our design artifact shows promising performance in capturing diverse and dynamic user goals for data analytics and in generating integrated data tailored to these goals. Our research establishes a theoretical framework to guide future scholars and practitioners in building smart, goal-driven data integration.

An Overview of ETL Cloud Services: An Empirical Study Based on User’s Experience

Abstract Cloud technologies are being used more and more by companies in solution development and are able to replace traditional on-premises tools. The purpose of this paper is to provide the approach of IT employees to using different cloud technologies for developing data solutions. The objectives presented are based on two research questions – to determine the main challenges faced in data development and to see which are the most used tools and technologies by employees. The case study created to answer the questions mentioned above, consists of a survey addressed to employees from different companies that work on data development, including data engineers and data architects, and see their preferences for using services from key players in the cloud computing industry. After all responses were analyzed, it was identified that companies prefer to use cloud services instead of on-premises tools. Regarding which cloud provider they want to use, it is subjective as all cloud providers offer different advantages or disadvantages for the data integration solutions they are offering. However, even with all the cloud options and resources available, there are still challenges for big data development around the infrastructure available, data security and governance.

Logical big data integration and near real-time data analytics

In the context of decision-making, there is a growing demand for near real-time data that traditional solutions, like data warehousing based on long-running ETL processes, cannot fully meet. On the other hand, existing logical data integration solutions are challenging because users must focus on data location and distribution details rather than on data analytics and decision-making.EasyBDI is an open-source system that provides logical integration of data and high-level business-oriented abstractions. It uses schema matching, integration, and mapping techniques, to automatically identify partitioned data and propose a global schema. Users can then specify star schemas based on global entities and submit analytical queries to retrieve data from distributed data sources without knowing the organization and other technical details of the underlying systems.This work presents the algorithms and methods for global schema creation and query execution. Experimental results show that the overhead imposed by logical integration layers is relatively small compared to the execution times of distributed queries.

An approach towards removal of data heterogeneity in SDN-based IoT framework

IoT is an emerging technology that provides innovative solutions to our day-to-day life challenges. With billions of diverse smart objects, IoT produces gigantic amount of heterogeneous data that mostly need to be processed in real time and shared across systems and applications. For IoT data to be truly accessable and interpretable, the data heterogeneity problem needs to be resolved first. Semantic technologies is one of the most used solutions for data integration across systems. Data represented as knowledge graphs enables easier querying and matching. To address this, we propose an SDN based IoT architectural framework utilizing semantics for data integration and sharing using Kafka and Spark Streaming for real-time data handling. One main contribution of this work is that any consistent IoT ontology, defined using Web Ontology Language(OWL) which explicitly specify its classes, objects and data properties, can be used to convert the stream data into RDF format automatically using Natural Language Processing endeavor. SDN helps in dynamically locating the data producing devices such as sensors for providing flexible data routing. We have presented a testbed experimentation framework and evaluated the results with simulated virtual sensors. The homogeneous results from diverse datasets was generated in acceptable minimal time. A use-case for Air Quality Monitoring is given and SPARQL queries executed have shown satisfactory response time with over thousands of triples.

Guiding the choice of informatics software and tools for lipidomics research applications.

Progress in mass spectrometry lipidomics has led to a rapid proliferation of studies across biology and biomedicine. These generate extremely large raw datasets requiring sophisticated solutions to support automated data processing. To address this, numerous software tools have been developed and tailored for specific tasks. However, for researchers, deciding which approach best suits their application relies on ad hoc testing, which is inefficient and time consuming. Here we first review the data processing pipeline, summarizing the scope of available tools. Next, to support researchers, LIPID MAPS provides an interactive online portal listing open-access tools with a graphical user interface. This guides users towards appropriate solutions within major areas in data processing, including (1) lipid-oriented databases, (2) mass spectrometry data repositories, (3) analysis of targeted lipidomics datasets, (4) lipid identification and (5) quantification from untargeted lipidomics datasets, (6) statistical analysis and visualization, and (7) data integration solutions. Detailed descriptions of functions and requirements are provided to guide customized data analysis workflows.

DESIGN OF ENTERPRISE ARCHITECTURE IN E-GOVERNMENT DEVELOPMENT USING TOGAF ADM 9.2 FRAMEWORK IN ELECTRICITY SECTOR OF DEPARTMENT ENERGY AND MINERAL RESOURCE OF WEST JAVA PROVINCE

Over the past few years, the Government has used information and communication technology to integrate its internal functions and improve service delivery. As part of the West Java Provincial Government, the Department of Energy and Mineral Resources has an active role in the West Java provincial government. The West Java Government's target is to improve electrical energy infrastructure that supports economic growth and access to electricity for households in remote areas so that they can contribute to improving electronic-based government system services in West Java Province. This research will be aimed at helping the effectiveness and efficiency of public services through the design of Enterprise Architecture (EA). This research will focus on business processes, data and information, applications, and services (excluding infrastructure and security). The final result of this research is an EA design blueprint that contains proposed solutions for business process improvements, data integration, application development, and fulfillment of public service needs concerning the Electricity Sector of the West Java Province ESDM Department in the hope of facilitating access to electricity services in West Java.

A Scalable Data Integration Architecture for Smart Cities: Implementation and Evaluation

The collection, processing, and analysis of data generated by varied sources can help us better understand the functioning and demands of the cities. However, developing efficient solutions to explore urban data is challenging due to the large volume, heterogeneity, and lack of accessibility and integration of this kind of data. In this work, we identify the main requirements of a data integration system to support decision-making in cities, focusing on its challenges. We analyze some existing data integration solutions, to uncover their features and limitations. Based on these results, we propose a new microservice architecture to support the development of software platforms for integrating smart cities’ heterogeneous data and a guideline to assess their performance. We also present details of a proof-of-concept implementation of the proposed architecture and its performance evaluation. The results demonstrate that the platform can scale horizontally to handle the highly dynamic demands of a smart city while maintaining low response times.

UHF-HF Integrated RFID Sensor: A Novel Information Interface for Industrial Integration

The itemized product identification, sensing and data integration are critical building blocks for the efficiency, flexibility, and potential intelligence of the Industrial Internet of Things (IIoT). The product sensing and data integration solutions usually suffer from the sophisticated sensor devices and their inflexible integration with the industrial processes, which may restrict the performance of industrial systems. To maintain an information link for convenient product data integration, this investigation presents a surface mountable dual-mode Radio Frequency Identification (RFID) sensor as a wirelessly powered information interface for simultaneous product identification, sensing, and data integration. The focus of this investigation includes: (1) the theory and hardware/software scheme for sensor-augmented RFID integrating Ultra-High Frequency (UHF) and High Frequency (HF) with Commercial-off-the-Shelf (COTS) devices; (2) integration of the UHF and HF energy and data interfaces to support both UHF and HF powering and sensing; (3) a data integration method supporting sensor data transmission using common UHF and HF readers without extra hardware or software cost; (4) proposal of a model for industrial integration with the proposed dual-mode RFID sensor. The experimental studies and exemplar application demonstrate the feasibility of the presented techniques for industrial integration.

The seer and the seen: Surveying Palantir’s surveillance platform

Palantir is among the most secretive and understudied surveillance firms globally. The company supplies information technology solutions for data integration and tracking to police and government agencies, humanitarian organizations, and corporations. To illuminate and learn more about Palantir’s opaque surveillance practices, we begin by sketching Palantir’s company history and contract network, followed by an explanation of key terms associated with Palantir’s technology area and a description of the firm’s platform ecosystem. We then summarize current scholarship on Palantir’s continuing role in policing, intelligence, and security operations. Our primary contribution and analysis are a computational topic modeling of Palantir’s surveillance patents (n = 155), including their topics and themes. We end by discussing the concept of infrastructuring to understand Palantir as a surveillance platform, where we theorize information standards like administrative metadata as phenomena for structuring social worlds in and through access to digital information.

THE OPEN DATABASE OF REGIONAL MODELS OF THE INTERNATIONAL SERVICE FOR THE GEOID

Abstract. This paper describes the development of a web application to facilitate the management and processing of geoid models in the ISG collection as well as of additional information available in the ISG website (mainly projects, services and publications). The ISG service pays significant attention to metadata and data interoperability from published geoid models. As a result, the possibility of creating a database with all necessary information for describing geoid models has been considered. PostgreSQL was chosen as the database for the implementation to take advantage of PostGIS geographic processing functions for developing a Database Management System for the resources available inside the ISG service catalogue (including geoid products, software, publications, etc.). The project's web development environment is Django, a web framework which is supported by the PostgreSQL database back-end and which makes it easier to integrate new applications or processing services into the website. A model solution for raster data integration using a PostGIS database back-end named Django-raster has also been investigated as part of the implementation, provided also that it is possible to take advantage of the vast Django community. The Django-raster model allows for the storage of datasets as tiles in the database, their exposure as a Tiled Map Services (TMS), and allows as well raster map algebra operations. TMS services will be utilized to share the geoid models and metadata included in the open access archive using a WebGIS, created with OpenLayers.

Clustering of Template-Generated Webpages Using DOM Tree Paths of URLs

Web templates are layouts for webpages that enable rapid and easy access to web content. Web data integration solutions use template based wrapper tools to extract product information from e-commerce websites. Given a collection of webpages, wrapper tools are used to discover the template portion of a webpage and extract data from it. These wrapper based data extraction techniques require pages created with the same template belong to the same cluster. Clustering these webpages based on their template is a significant challenge. While there are algorithms for clustering webpages based on their template, they are computationally intensive to be applied at web scale. By examining the DOM tree paths of URLs on a webpage, the proposed work presents a highly scalable methodology for clustering template-generated webpages. Further, the locality sensitive hashing (LSH) technique is used to reduce the cost of clustering. The proposed technique is found to be more precise and cost effective than the existing baseline methods when tested on three separate real-time data sets.

Correlation-Aware Sport Training Evaluation for Players With Trust Based on Mahalanobis Distance

With the widely-adopted idea of health and longevity, sports have been becoming one of the most popular entertainment ways of the public. For the majority of sports, players need to know about their concrete physical conditions in a real time manner so as to pursue a good sport score or ranking in a competition or a race. Generally, we can achieve the above goal through analyzing and evaluating the daily training scores of each player. However, there are often multiple physical trainings for players and various correlations are existent among them, which significantly decrease the fairness and trust of player training score evaluation and ranking since traditional multi-dimensional data integration solutions are often based on a strong hypothesis, i.e., the involved multiple dimensions are independent with each other. In view of this shortcoming, we introduce the Mahalanobis Distance into the multi-dimensional player training score evaluation and further propose a <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</u> orrelation-aware <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</u> layer training score <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</u> valuation method with trust (abbreviated as CPE <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MD</sub> ) based on <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</u> ahalanobis <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</u> istance. As Mahalanobis Distance can eliminate the hidden linear correlations among the involved multiple dimensions, we can guarantee the fairness and trust of Mahalanobis Distance-based player training score evaluation and ranking results. At last, we use a case study to show the feasibility of CPE <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MD</sub> in this paper.

Scalable analysis of multi-modal biomedical data.

BackgroundTargeted diagnosis and treatment options are dependent on insights drawn from multi-modal analysis of large-scale biomedical datasets. Advances in genomics sequencing, image processing, and medical data management have supported data collection and management within medical institutions. These efforts have produced large-scale datasets and have enabled integrative analyses that provide a more thorough look of the impact of a disease on the underlying system. The integration of large-scale biomedical data commonly involves several complex data transformation steps, such as combining datasets to build feature vectors for learning analysis. Thus, scalable data integration solutions play a key role in the future of targeted medicine. Though large-scale data processing frameworks have shown promising performance for many domains, they fail to support scalable processing of complex datatypes.SolutionTo address these issues and achieve scalable processing of multi-modal biomedical data, we present TraNCE, a framework that automates the difficulties of designing distributed analyses with complex biomedical data types.PerformanceWe outline research and clinical applications for the platform, including data integration support for building feature sets for classification. We show that the system is capable of outperforming the common alternative, based on “flattening” complex data structures, and runs efficiently when alternative approaches are unable to perform at all.

IOAT: an interactive tool for statistical analysis of omics data and clinical data

BackgroundWith the development of high-throughput sequencing technology, a huge amount of multi-omics data has been accumulated. Although there are many software tools for statistical analysis and visual development of omics data, these tools are not suitable for private data and non-technical users. Besides, most of these tools have specialized in only one or perhaps a few data typesare, without combining clinical information. What’s more, users could not choose data processing and model selection flexibly when using these tools.ResultsTo help non-technical users to understand and analyze private multi-omics data and ensure data security, we developed an interactive desk tool for statistical analysis and visualization of omics and clinical data (shortly IOAT). Our mainly targets csv format data, and combines clinical data with high-dimensional multi-omics data. It also contains various operations, such as data preprocessing, feature selection, risk assessment, clustering, and survival analysis. By using this tool, users can safely and conveniently try a combination of various methods on their private multi-omics data to find a model suitable for their data, conduct risk assessment and determine their cancer subtypes. At the same time, the tool can also provide them with references to genes that are closely related to tumor staging, facilitating the development of precision oncology. We review IOAT’s main features and demonstrate its analysis capabilities on a lung from TCGA.ConclusionsIOAT is a local desktop tool, which provides a set of multi-omics data integration solutions. It can quickly perform a complete analysis of cancer genome data for subtype discovery and biomarker identification without security issues and writing any code. Thus, our tool can enable cancer biologists and biomedicine researchers to analyze their data more easily and safely. IOAT can be downloaded for free from https://github.com/WlSunshine/IOAT-software.

Evaluation of augmented reality supported approaches for product design and production processes

Augmented Reality (AR) has been a focus of research in the field of production for many years with the goal to improve e.g. monitoring and maintenance processes. Although the benefits have been well researched, the AR technology has not been widespread yet in the industry. This article discusses reasons why existing approaches have not become established outside the research context. Following this, solutions are examined for three identified problematic areas. First, this article presents a theoretical method to create use cases for AR applications in the product design process based on the Double Diamond design process model. The second main part discusses which device properties are relevant for a production environment. A catalogue of criteria is presented, weighting properties according to their importance. Third, a discussion is described regarding data integration concepts, problems and solutions for AR design and production use cases.